Server Rundown Part 7

Hey all,

I didn’t have chance to do one of these last week because of the upgrade/database issue, and I won’t have chance to do one this weekend because I’m visiting family and won’t be near anything but my phone. So thought I would do a midweek one to try and keep to some regularity :)

Part 1 / Part 2 / Part 3 / Part 4 / Part 5 / Part 6

TL;DR

Lemmy update - We’re on 0.18.3 but we did have issues!
Small images change - Everything is .webp!
Front-end clients - They’ve all been updated!
Canvas event - Our first Lemmy event!
Funding - Thank you to those who’ve donated!
Interaction - Lurking = bad / Interacting = good!
Server update - Some graphs and pretty pictures!

Lemmy updated to 0.18.3

We did, after a couple of stressful days, update to 0.18.3. Our first attempt instantly failed prompting our first rollback in a while, which all happened in the space of a couple of minutes. Our second attempt appeared at first glance to have worked, but very quickly became apparent something had gone wrong during the upgrade.

Suddenly communities stopped loading, people’s accounts had vanished and federation slowed to a crawl. The DB was spewing errors left, right, and centre. So we had no other choice but to roll back again and see if we could replicate the error in a testing environment, which resulted in the loss of about 12 hours of data.

Unfortunately spinning up an exact replica of an instance is not an easy or quick task, and even then federation won’t work because you can’t duplicate the encrypted keys of a running instance. However we got the testing environment to a state where we could confidently apply the upgrade and see what happened. We were able to use this data to ensure the steps we then took on the live site would keep the database intact during upgrade and thankfully we were able to then get the live update to work.

I do apologise for the disruption and extended downtime, its never something we want to do but upgrades are never quite as simple as the documentation would make out!

Just a reminder that if you can’t access the site, you can visit status.lemmy.zip for updates, or join the Matrix chat where no doubt I’ll be swearing at the server.

Images change

Just a heads up that we changed the config so that all images are stored in .webp format, as per best practice suggestions. This should decrease storage space slightly with higher resolution pictures, but my eyesight is so crap I can’t tell the difference.

Front end clients update

For those that use them, the front-end clients have been updated to their latest versions, something I do each week. Currently we have the following front-ends:

m.lemmy.zip - our mobile site
beta.lemmy.zip - a beta front-end called Photon that I personally use quite a bit for either desktop or mobile.
old.lemmy.zip - err, yeah, that site replica if its your sort of thing!

If anyone has any other suggestions for a front end they’d like to see hosted please let me know and I’ll happily check it out and see if we can get it hosted.

Canvas event

This weekend the first Canvas Event will be running, and we were one of the first instances to say we’d jump on board with it. In order to participate you’ll need to verify your account, which you can do with a code you’ll be pm’d and enter into the website when it goes live. It would great to see a good turn out (and if anyone wants to recreate the lemmy.zip logo I’d fully support that 😉)

More detail on the event will be posted here closer to the time.

Funding

We continue to have a good level of funding from our supporters, which helps massively in keeping this instance running. I continue to keep the OpenCollective website updated, and if you want to donate you’ll get your name in the Thank You Thread. If you use your real name and want me to put your Lemmy.zip username instead, please send me a PM and i’ll update it ASAP.

If you’re enjoying your time here, please do consider it. 100% of funds go to Lemmy.zip’s upkeep and if we get enough funding we’ll be able to improve the performance of the server.

Interaction

Welcome to all new Lemmy.zip users and I hope you’re enjoying your time here!

This is a weekly reminder than interacting with the instance (up-voting, commenting, posting etc) goes incredibly far to helping to promote this instance. If you’re usually a lurker that is fine, but please consider posting a comment or making a post in a community if you’re enjoying the content there.

Server update

Pretty graph time! Unfortunately after a roll back we lose the last 7 days of CPU usage but hopefully in the future that won’t be an issue. So i’ll post what graphs I have for you:

Hetzner usage over the last 24 hours: