Hey everyone,

We’ve started to hit the upper limit of what our current server is able to handle in terms of performance, and we’ve noticed its having a performance impact on the site itself.

For example, this is the CPU usage as I type this:

We’re going to make the move to a much more powerful server, as we’ve done all we can on this one to control performance.

If all goes smoothly, I predict a couple of hours downtime while we migrate the database between servers and then move the surrounding services over.

The window is going to be 5 hours, so any posts made in this time will most likely be lost, especially if we have to roll back to the old server.

Once the update is done, Cloudflare should handle most of the DNS issues but you might need to hard refresh the site.

You can keep an eye on status.lemmy.zip for more information.

Please let me know if you have questions in the comments, but I will be doing a full breakdown in the next server update in a couple of days.

Thanks

Demigodrick

  • DemigodrickOPMAEnglish
    5·
    3 months ago

    Update:

    • Main site is back up - let me know of anything not working!
    • images should be working
    • site should be faster with any luck
    • work is still ongoing in the window around secondary services like the mobile site, please be aware that something could still go wrong.

    Window is open until 18:00 UTC - until then we could still need to restore the site to an older version, any posts might go missing if that is the case.

    I’ll make a final post when we’re done.

  • mynamesnotrickEnglish
    14·
    3 months ago

    I was on the fence about donating but now you got a new donator. Thank you for running this site. I use it every day. Lemmy has pretty much replaced the old alien site for me now and I really like this instance after bouncing around between a few and even hosting my own. Wow, almost a year already (9 months~)

    • rootEnglish
      8·
      3 months ago

      Same boat, I get paid tomorrow and will have a few spare bucks after that, lemmy.zip deserves it!

    • DemigodrickOPMAEnglish
      7·
      3 months ago

      Thank you! It makes me really happy to know people enjoy using the site! :)

      I barely ever go there either now too, can’t stand all the adverts.

  • Possibly linux
    English
    8·
    3 months ago

    I’m no numbers guy but 98% usage seems like a lot. I guess its time to upgrade.

  • Altima NEOEnglish
    3·
    3 months ago

    Oh dang, what’s caused the uptick on server usage? More people joining?

    • DemigodrickOPMAEnglish
      10·
      3 months ago

      Yeah pretty much, we’re still on an upwards trend of active users so thats taking some of the resource.

      Part of the issue is all of the federation traffic as the rest of the fediverse grows, and processing all of that. Some sites have struggled with lemmy.world for example due to their size.

      The backend services (such as pictrs to serve images) are getting more advanced but that also requires more resources.

      • MrTolkinghoenEnglish
        2·
        3 months ago

        Interesting. One advantage I could see to federation would be the ability to process some content on a delay at times of high load. I.e. prioritize local content then catch-up on federated content as possible. I assume that isn’t how it works rn?

        That could help alleviate federation forcing smaller instances to have to keep up with at least similarly big hardware requirements.

        • DemigodrickOPMAEnglish
          3·
          3 months ago

          I believe there are discussions around “batching” updates together rather than sending each update as it happens. Pros and cons both ways (slower federation updates, but less resource) so the very clever people and devs will hopefully be able to come up with a good solution.

  • CluelessDudeEnglish
    2·
    3 months ago

    I’ve noticed we are growing but damn that’s a big jump from last time, good luck with all the upgrading and moving of things, goated instance.

    Thank you for the work you guys do.