Hey everyone,
We’ve started to hit the upper limit of what our current server is able to handle in terms of performance, and we’ve noticed its having a performance impact on the site itself.
For example, this is the CPU usage as I type this:
We’re going to make the move to a much more powerful server, as we’ve done all we can on this one to control performance.
If all goes smoothly, I predict a couple of hours downtime while we migrate the database between servers and then move the surrounding services over.
The window is going to be 5 hours, so any posts made in this time will most likely be lost, especially if we have to roll back to the old server.
Once the update is done, Cloudflare should handle most of the DNS issues but you might need to hard refresh the site.
You can keep an eye on status.lemmy.zip for more information.
Please let me know if you have questions in the comments, but I will be doing a full breakdown in the next server update in a couple of days.
Thanks
Demigodrick
You must log in or # to comment.
Update:
- Main site is back up - let me know of anything not working!
- images should be working
- site should be faster with any luck
- work is still ongoing in the window around secondary services like the mobile site, please be aware that something could still go wrong.
Window is open until 18:00 UTC - until then we could still need to restore the site to an older version, any posts might go missing if that is the case.
I’ll make a final post when we’re done.
I was on the fence about donating but now you got a new donator. Thank you for running this site. I use it every day. Lemmy has pretty much replaced the old alien site for me now and I really like this instance after bouncing around between a few and even hosting my own. Wow, almost a year already (9 months~)
Same boat, I get paid tomorrow and will have a few spare bucks after that, lemmy.zip deserves it!
Thats very kind, thank you :)
Thank you! It makes me really happy to know people enjoy using the site! :)
I barely ever go there either now too, can’t stand all the adverts.
Goated admin
I’m no numbers guy but 98% usage seems like a lot. I guess its time to upgrade.
Oh dang, what’s caused the uptick on server usage? More people joining?
Yeah pretty much, we’re still on an upwards trend of active users so thats taking some of the resource.
Part of the issue is all of the federation traffic as the rest of the fediverse grows, and processing all of that. Some sites have struggled with lemmy.world for example due to their size.
The backend services (such as pictrs to serve images) are getting more advanced but that also requires more resources.
Interesting. One advantage I could see to federation would be the ability to process some content on a delay at times of high load. I.e. prioritize local content then catch-up on federated content as possible. I assume that isn’t how it works rn?
That could help alleviate federation forcing smaller instances to have to keep up with at least similarly big hardware requirements.
I believe there are discussions around “batching” updates together rather than sending each update as it happens. Pros and cons both ways (slower federation updates, but less resource) so the very clever people and devs will hopefully be able to come up with a good solution.
I’ve noticed we are growing but damn that’s a big jump from last time, good luck with all the upgrading and moving of things, goated instance.
Thank you for the work you guys do.
