Hello everyone!

I'll be taking the site down for two maintenance windows this week to complete our server migration.

• Weds Jan 29th - 09:00 - 11:00 PT (12:00 - 14:00 ET)
• Thurs Jan 30th - 09:00 - 11:00 PT (12:00 - 14:00 ET)

During the first window I'll be migrating us from OVH to our new dedicated hardware. After this migration there will likely be some temporarily broken images, as it takes approximately 8 hours to resync our object storage from OVH.

This is a major change and despite my testing, may have some unintended side effects. If you run into any problems that aren't just a broken image, please let us know.

The second maintenance window is to migrate our pict-rs database from it's local sled-db into our primary postgres DB. This is a much smaller change but since pict-rs checks every image as it goes through them, it takes about 1.5 hours.

As usual, you can check https://status.lemmy.ca/ for updates.

Translation: To stay on X or not? Some elected officials think of leaving Elon Musk's platform.

What the article is saying is basically that a lot of the provincial MPs in Québec are thinking of leaving for something else. One of which, the health minister Christian Dubé, has already created an account on BlueSky.

We need to promote fediverse alternatives to our elected officials right now since everyone is worried about the control of information on Meta/X & Co.

Hello all! I am new to Lemmy.ca and really new to this whole concept. I was poking around the community list and thought of a couple that I would have liked to see (Canada related). Is there a procedure specifically for creating new communities here, or do I just go ahead if I'm sure it doesn't exist? I don't want to step on any toes.

Hello everyone, we're long overdue for an update on how things have been going!

Finances

Since we started accepting donations back in July we've received a total of $1350, as well as $1707 in older donations from smorks. We haven't had any expenses other than OVH (approx $155/mo) since then, leaving us $2152 in the bank.

We still owe TruckBC $1980 for the period he was covering hosting, and I've contributed $525 as well (mostly non-profit registration related stuff, plus domain renewals). We haven't yet discussed reimbursing either of us, we're both happy to build up a contingency fund for a while.

New Server

A few weeks ago, we experienced a ~26-hour outage due to a failed power supply and extremely slow response times from OVH support. This was followed by an unexplained outage the next morning at the same time. To ensure Lemmy’s growth remains sustainable for the long term and to support other federated applications, I’ve donated a new physical server. This will give us a significant boost in resources while keeping the monthly cost increase minimal.

Our system specs today:

• Undoubtedly the cheapest hardware OVH could buy
• Intel Xeon E-2386G (6 cores @ 3.5ghz)
• 32gb of ram
• 2x 512gb Samsung nvme in raid 1
• 1gb network
• $155/month

The new system:

• Dell R7525
• AMD EPYC 7763 (64 cores @ 2.45ghz)
• 1tb of ram
• 3x 120gb sata ssd (hw raid 1 with a hot spare, for proxmox)
• 4x 6.4tb nvme (zfs mirrored + striped, for data)
• 1gb network with a 50mbit commit (See 95th percentile billing)
• Redundant power supplies
• Next day hardware support until Aug 2027
• $166/month + tax

This means instead of renting an entire server and having them be responsible for the hardware, we'll be renting co-location space at a Vancouver datacenter PDF via a 3rd party service provider I know.

These servers are extremely reliable but if there is a failure, either Otter or myself will be able to get access reasonably quickly. We also have full OOB access via idrac, so it's pretty unlikely we'll ever need to go on site.

Server Migration

Phase 1 is currently planned for Jan 29th or 30th and will completely move us out of OVH and onto our own hardware. I'm expecting probably a 2-3 hour outage, followed by an 6-8 hour window where some images may be missing as the object store resyncs. I'll make another follow up post in a week with specifics.

Phases 2+ I'm not 100% decided on yet and have not planned a timeline around. It would get us into a fully redundant (excluding hardware) setup that's easier to scale and manage down the road, but it does add a little bit of complexity.

• Dark mode diagram
• Light mode diagram

Let me know if you have any questions or comments, or feedback on the architecture!

Morning all!

I'm going to be taking the site down for about 5 minutes, so that I can get a consistent copy of our databases (postgres + pict-rs sled).

Will do it at about 10am PST.

Hope nobody missed anything during the outage

Hey everyone, and happy new year!

Sorry about that super long downtime there. Yesterday (Sunday) morning at 10:03AM PST our server suffered a physical hardware failure, apparently a power supply failure. Unfortunately despite opening a ticket with our hosting vendor (OVH) a few minutes later and them claiming to have 24/7 support, nobody looked at our ticket until this morning when their phone support lines opened and I called them.

They've now replaced a defective power supply and we're back online, after ~26 hours of being offline. Some pretty disappointing response times, to put it nicely.

We're planning to move away from OVH at the end of this month, onto proper enterprise grade hardware that we own and control. This will give us a HUGE boost in server resources and allow us to scale for the foreseeable future, while also giving us the control to resolve problems like this much quicker. Expect another follow up post about this in the next couple weeks once I've put together the migration plan.

Timeline:

• Jan 5th 10:03am PST - We get alerts to the server being non-responsive.
• Jan 5th 10:05am PST - I pull up the console via IPMI and it's completely non-responsive. Attempting to power off / on the server or do anything, does not work.
• Jan 5th 10:15am PST - Initial support ticket created with OVH. I followed up a couple times over the next few hours, and got no response.
• Jan 6th 6:32am PST - Called OVH, gave them the case number and asked them to investigate
• Jan 6th 7:34am PST - I get notified they'll start their "intervention" in 15 minutes.
• Jan 6th 11:04am PST - Call them again, the tech is still working on it and they'll get back to me with an update
• Jan 6th 11:34am PST - "I was informed by our data centre technician that there is an issue with the power supply unit for the rack on which your server resides. Your server will come back online once they have replaced the power supply."
• Jan 6th 12:17pm PST - We're back up finally!

Edit on Jan 7th @ 8:40am PST: We just had another outage of about an hour. Investigating with OVH.

Hi everyone!

Last year lemmy.ca ran a census for our instance, and this year we wanted to solicit feedback ahead of time. Since other instances expressed interest in running similar censuses, having some agreed upon questions would make it easier to do comparisons. (previous: lemmy.ca results, lemmy.nz results).

Timeline wise, for our instance, we are thinking of gathering feedback for the rest of the month and then running the census early in the new year. We are open to adjusting when we run it, to align with other instances.

General areas of feedback:

Self Identification / Categorization

With some questions, it is difficult to have a predetermined list of options. Last year we included a free form entry option, and then created word clouds with the full list. However, that is not as insightful, and it takes a lot of time to process. For this year, do you prefer:

• A set of options + a free form option (same as last year)
• A set of options only

Please share if you know of a resource that has a good/inclusive set of options for the question types. Even if a free form option is included, a better set of options as a base will lead to better data.

For context, these were the questions where this happened:

• What is your ethnicity?
• How do you identify? (sexual orientation)
• What is your occupation / field of Work?

Data collection & Results

Last year we used Google forms for collection, and a spreadsheet for calculations and analysis. Visualization was done with google sheets, an open source word cloud generator (github link), and DataWrapper (link) for an interactive display on our website.

This year the plan is similar, although we’re interested to try chart.js for the visualizations. We had some issues with DataWrapper displays, and having the code be open source would make it easier for others to replicate. One consideration is data privacy (do we commit anonymized data to the repo, host it elsewhere and import it, etc.).

Sometime in the future, it might be nice to have a template and/or pipeline that everyone can use, for consistency and to minimize work. In the meantime, do you have any thoughts or recommendations?

Fun Question

Last year we asked about people's favourite dessert. It would be cool to do something similar this year. Do you have any suggestions for a fun question? Some ideas:

• Favourite comfort food
• Favourite fruit
• Favourite color (with a hex code input)

Proposed Questions

Where applicable, I’ve included changes from last year. Depending on the type of instance you run (ex. regional or not), you may want to skip, add, or modify questions.

Section 1: Where is everyone from?

Where are you from?

• Canada? (yes / no)
  • If in Canada, which province or territory?
  • If not, which continent?

What is the size of your community? (rural / urban / suburban / other)

• suburban and other are new options

In your opinion, what is the quality of your internet connection? (poor, reasonable, excellent)

What is your internet speed?

• new question inspired by lemmy.nz
• we may include a link to a speed test site for convenience, and have a free form entry

Section 2: Who are you?

What age range do you fall into? (0-9, 10-19, etc.)

Ethnicity? (options uncertain, see above)

Gender Identity? (Man, Woman, Non-binary person, Genderfluid)

• Optionally, we could include a free form entry 'other' option, and list the input in a separate list.

Are you someone with trans experience (yes / no)

How do you identify? (options uncertain, see above)

Disability Status (yes / no)

• New separate question inspired by lemmy.nz

Disability Status (free form entry)

• Last year we manually processed the text inputs and displayed them in a list. We're open to hearing suggestions on better ways to run this question!

Educational Background

• Last year we had multi-select, which didn't make sense. This year we can split it into a few questions:
  • Are you currently in school? (yes / no)
    • What are you studying? (single selection, see list below)
  • Highest level of education completed (single selection, see list below)
  • All past education (multi-select, see list below)
  • List of options:
    • No formal education
    • Some secondary school (high school)
    • Secondary school (high school) diploma
    • GED
    • Trade / technical / vocational training
    • Some Post-secondary (college, university, CÉGEP)
    • College diploma
    • Associate degree
    • Bachelor's degree
    • Master's degree
    • Professional degree (MD, JD, MBA, etc.)
    • Doctorate (PhD, etc.)
    • Other

Employment Status (full-time, part-time, self-employed, student, retired, unemployed, other)

• Last year we also got responses for 'homemaker', 'disabled', 'active duty military'. We could add these as options, or include a free form 'other' entry for them.

Field of Work (options uncertain, see above)

• This question was a free form entry last year, and it was very difficult to process. Having a solid list of options would be ideal in this case.
• We could include an 'other' option with a free form entry, with a note that any response will not be included in the main results and will be displayed as-is in a separate list.

Section 3: Instance Usage

How many {instance name} communities do you participate in? (0, 1-9, 10-19, 20+)

If you moderate any communities, how many? (number input)

• Last year this was a multi-select, but a number input would be more straightforward

What community do you want to see on {instance name}? (free form entry)

What existing community do you want to see more people using? (free form entry)

• This is a new question.

On average, how much time do you spend on Lemmy each day? (<1 hour, 1-2 hours, 2-3, 3-5, 5-8, 8-13, 13-21, 21+)

How often do you visit {instance name}? (multiple times a day, daily, weekly, monthly, rarely)

• This is a new question.

How do you access Lemmy most often? (desktop, mobile (including tablet), both about equally)

• (including tablet) is a new addition
• both about equally is a new option

If you use desktop, what operating system do you use? (Windows, MacOS, Linux, I don't use desktop, other)

• New question

On desktop, which interfaces do you use? (default, Mlymym, Photon, Voyager, Alexandrite, I don't use desktop)

• I don't use desktop is a new option, and will be separate from the 'skip this question' option

Is there another interface you would like us to add? (free form entry)

• New question

If you use mobile, what operating system do you use? (Android, iOS, I don't use mobile, other)

• I don't use mobile and other are new options

On mobile, how do you access Lemmy?

• We will include an updated list of apps, and a free form entry for any new ones
• We will include an option for I don't use mobile, web browser, and other, in addition to the skip this question option

If you primarily access lemmy.ca through different Lemmy/Kbin instance, please list it below (optional) (free form entry)

What other decentralized / federated platforms do you use?

• Some potential options: none, Mbin, Kbin, PieFed, Mastodon, Bluesky, Pixelfed, Peertube, Loop, Bookwyrm, WriteFreely, Pleroma, Friendica, Misskey, Writefreely, + free form entry

Fun Question (yet to be determined)

Feedback (free form entry)

General Changes:

• All questions will have a 'skip this question' option
• Fewer random comments from me when reporting the results. I looked back at them afterwards and thought they might be a bit much.

One of the drives in our server has failed. =( Even though it should be a 10 minute job OVH needs a 2 hour window to replace it.

I've requested they schedule it for Tuesday from 8am - 10AM PST. Hopefully it'll be reasonably quick, but expect cloudflare tunnel errors while they perform the work.

Hey All!

I'm going to upgrade us to 0.19.7 tomorrow (Sunday Nov 24th) around 10am PST. I don't expect significant downtime, but expect a few minutes at least.

Update: All done!

Hi, we've noticed some server instabilities around 9:20AM (Eastern Time) this morning and it seems our hosting provider had some issues, which lasted for about 40 minutes.

We also took the time to reboot the server after the event just to make sure we're not running into other issues down the road, sorry for the downtime everyone.

Recently I was asked if we'd be willing to reconsider federation with Hexbear, followed by CARCOSA reaching out to me to see if it was something we'd be open to.

They made a post asking if their community was interested in refederating with us, similarly I'd like the opinion of our local users.

Why refederate?

Hexbear has a decent number of sizable active communities:

• https://hexbear.net/c/games
• https://hexbear.net/c/technology
• https://hexbear.net/c/movies
• https://hexbear.net/c/food
• https://hexbear.net/c/vegan

These all seem fairly well moderated, giving us access to more diverse posts and viewpoints. A lot of our feed right now is dominated by US centric communities, and it'd be nice to get some more variety.

Why not?

They love trolling and I don't expect them to follow our rules when commenting (primarily "be civil").

Just looking at their thread about us, there's a significant number of immature replies slandering us / Canada. I'm not confident their users will engage in actual discussion in our threads, rather than just showing up to throw out insults and memes.

We can try to handle this by banning users / blocking specific communities, but that's going to increase moderation workload for both admins and mods. Hexbear is self admittedly quick to ban users on their side, but they also don't mind ban evasion - not ideal for us.

So what's next?

This is where we solicit your opinions, how does our community feel?

Edit: Well, it looks like the community is overwhelmingly against refederation. Decision made, thanks everyone for your feedback!

Currently the instance is the source of unmoderated trolls, and the instance admin seems unwilling to take action despite multiple reports sent.

We're therefore taking the action of defederating. We'd rather work with the instance admin to keep those trolls in check and minimize the disruption before taking a drastic action like this.

For transparency purpose, you can see which instances we are defederating from by accessing the https://lemmy.ca/instances page and going to the Blocked instances section.

We recently crossed the first anniversary for many of us on Lemmy and the Fediverse, and we are thrilled to announce an important milestone: we have started a new registered not-for-profit organization! The organization is registered as ‘Federated Open Communications Canada’, and we will go by 'Fedecan' in our day-to-day operations.

Over the past few months, we have been working behind the scenes on the necessary paperwork, bylaws, and other administrative tasks to get the organization rolling. Thanks to the invaluable feedback from our previous check-in, we decided to take the leap and register federally, which will allow us to operate across Canada more easily. For more details, you can check out the FAQs on our new website.

Resuming Donations

With this important step completed, we are now ready to resume accepting donations. We have set up several donation options to accommodate different preferences on convenience and fees. Our first priority is to reimburse those who have been covering server and other costs, and to set up regular financial updates. We are open to adding more donation options, so please let us know if there is an option you prefer.

• Donation Page: (fedecan.ca/donate)

• Alternate Donation page without embedded elements: (fedecan.ca/donate-private)

Our current costs

• Server hosting - $150/month
• Domains registrations - $50/year
• Non profit requirements - $500 one time

Our Next Steps:

• Reimburse individuals who have covered server and operational costs and to set up more regular financial updates
• Collaboratively creating guides and learning resources about Lemmy and the Fediverse (ex. how things work, data/privacy details, and best practices for moderating or running instances, etc.) + translations
• Exploring other platforms to run. Some that are on our radar include Sublinks, Pixelfed, and Matrix
• Developing logos, colours, and other branding details based on ideas from the community

What we want to know:

• Do you have any questions, comments, or concerns? This will also help us expand our FAQs
• Which other platforms would you like us to explore?
• Do you prefer any other donation method?

We want to help build a better online experience for everyone, and that requires an organization with a strong foundation. This should be the first step in that process.

Thank you for everything so far, and we look forward to seeing you around! 🥳

The Fedecan Team

Thank you to everyone that participated, it was fun :)

Link to the location:

https://canvas.fediverse.events/#x=-54&y=-185&zoom=11

LINK: https://canvas.fediverse.events/

Feel free to use this thread to discuss ideas and otherwise coordinate.

---

What is it?

You can sign in with almost any Fediverse account (ex. your lemmy account, mastodon account, etc.). Afterwards, you will be able to place a pixel every 30 seconds or so. By working together, people make (and break) different designs.

You shouldn't need to give away your login information to authenticate. For me, the system asked for my instance and username, then sent me a message with the verification code.

---

Timeline: 72 hours

• Opens on July 12th, 2024 @ 4am UTC
• Closes on July 15th 2024 @ 4am UTC

---

What should we do?

I don't think we have any templates yet, so you're open to suggest a Canada wide design, or smaller specific ones. In the meantime, just have fun :)

You can see what other people are up to here: https://programming.dev/post/16720673

You can see last year's design here: https://toast.ooo/post/288913

You can use this post as well, but it might be helpful to have everything in one place:

https://lemmy.ca/post/24637498

Hi everyone,

We'll be undergoing maintenance on Saturday morning from about 11am - noon PST.

We've been having some stability issues over the past few days that I've tracked down to database performance, so I'll be doing some tuning and also applying all our needed system updates at the same time.

Cheers!

Update: This is all done now!

Happy Canada Day everyone.

Related to the outage that happened last night, we rebooted the Lemmy services but we're still trying to figure out the root cause, which seems to point to an out of memory issue in the logs. However it's not what we see in our monitoring console.

In the meantime, we will monitor the service more closely until we are confident the issue is resolved, and we will improve our tools to detect such a problem faster.

EDIT: Also happened at night on July 2nd, still trying to find the root cause..

Apologies for the extended downtime.

It may be a few hours till things are up, please bear with us in the meantime.

See status here: https://status.lemmy.ca/

It looks like the alternative frontends are (partially) working:

• https://old.lemmy.ca/
• https://photon.lemmy.ca/
• https://alex.lemmy.ca

---

Also happy Canada Day!

Back when I joined lemmy.ca, the front page was full of bicycles, city/province-specific community posts, and others like woodworking This morning, four or five of the front page posts were making fun of some crazy bikini lady(?). I read their “read this first” post and understand why they’re doing what they’re doing, but is there a way to get our default front page to be a bit more…friendly and community-oriented?

Sorry for the brief bits of downtime here and there, I snuck in a quick upgrade of us to the latest lemmy 0.19.4 now that a few bugs have been patched out.

More details here! https://join-lemmy.org/news/2024-06-07_-_Lemmy_Release_v0.19.4_-_Image_Proxying_and_Federation_improvements

The sole moderator doesn't even follow their own rules: https://lemmy.ca/post/22741340?scrollToComments=true

I'll just say it - it's a Russian propaganda community. Is there any reason this community needs to exist on Lemmy.ca? Is there a rule against blatant astroturfing / propaganda / misinformation? I don't think the 5 rules in the sidebar are going to be enough to stop an army of trolls:

No bigotry - including racism, sexism, ableism, homophobia, transphobia, > or xenophobia. Be respectful. Everyone should feel welcome here. No porn. Use the NSFW tag when needed. No Ads / Spamming. Bot accounts need to be flagged as such in their settings.

Maybe time to get ahead of it?