After updating from 4.3 to 4.4.1, I noticed that my remote groups stopped updating. I double checked them here, and they’re even further behind the group hosts, with the last received topics dating back to 16 May.

d31b2edc-3c5f-46a5-a650-ed39b5699870-image.png

These are all Lemmy-hosted groups, so I’m not sure if it’s a Lemmy-specific thing, or group actors more generally.

  • julian@community.nodebb.org
    1·
    5 个月前

    After more debugging today I am less convinced that this is a NodeBB issue.

    I am finding Lemmy communities where federation continues without issue, and others where there is no activity at all.

    The sequence of events is:

    1. NodeBB sends out a Follow
    2. Lemmy responds with a 200 or equivalent
    3. Lemmy later sends me an Accept
    4. Lemmy later sends content (posts, upvotes, etc.)

    startrek.website and ani.social both do everything ok.

    lemmy.world and mander.xyz fail at step 3, and existing connections (where step 3 completed in the past) do not execute step 4. I literally am not seeing anything besides Like activities coming from those servers.

    Existing PieFed and NodeBB integrations continue to work without issue.

    • arachnibot@nodebb.arachnibot.com
      1·
      5 个月前

      [email protected] I’m actually having trouble getting new ani.social posts into my instance. I’m not sure if [email protected] is having the same problem, but it seems that it grabbed posts from up to 9 days ago, and then stopped federating them in.

      It says Asserting 0 group(s) when I check the remote category on my instance directly though, so perhaps this is related?

    • Kichae@community.nodebb.orgOP
      1·
      5 个月前

      Wild. What’s really weird is that I know I stopped receiving topics and posts from all of the Lemmy servers I follow communities on on the day I graded from 4.3 to 4.4.1, which was several days after this forum was updated to 4.4.1. Meanwhile, those communities here stopped receiving topics and posts a few days earlier than my forum did (though, I can’t say for certain that that was on the day y’all updated to 4.4.1).

      My follow relationships are rather mature at this point, so it’s not like this is an issue with establishing new follow relationships. Old ones are broken, too. Very strange. And also, FWIW, very unidirectional: I can still post into these groups and have my content federated. I’m just not processing or receiving anything coming back my way.

      I’m up to my eyeballs right now, but I’ll try to look more closely into logs this weekend to see if I can identify any paradigm shifts.

      • julian@community.nodebb.org
        1·
        5 个月前

        Kichae I suppose it’s worth rolling back to 4.3 and figuring out what the change was…! I just wish testing federation wasn’t such a long process.

        • Kichae@community.nodebb.orgOP
          1·
          5 个月前

          It is rather silly that it inter-platform federation so damn fragile. I can’t wait until there’s a bigger network of nodeBB sites federating.

          • julian@community.nodebb.org
            1·
            5 个月前

            Confirming that follow-unfollow-follow-unfollow logic is working as expected against ani.social which means the plumbing actually works.

            • julian@community.nodebb.org
              1·
              5 个月前

              With the Lemmy developers’ help, we figured it out! It was NodeBB’s spam protection mechanism. One of our configurable options is to limit the number of upvotes one can make per day.

              1. Someone (or someones) on Lemmy communities was sending lots of upvotes and running into this limitation when those activities were propagated to NodeBB.
              2. NodeBB dutifully noted this and sent back an HTTP 500.
              3. Lemmy dutifully noted this and retried sending the activity using an exponential backoff algorithm
              4. Eventually enough 500s were recorded that Lemmy decided that our server needed a time-out of sorts :smile:

              I updated the code to handle this case better, and now the activities are coming in at full speed. It’ll take awhile to get back up to sync though :sweat: