I'm not sure what the cause is, but the last few days it seems more of my comments are failing to federate than are succeeding.
No1 - 1.9yr
I'm claiming federation issues as the reason my posts aren't getting thousands of upvotes.
6
maniacalmanicmania - 1.9yr
I still can't believe you're not president of the world yet. Like wtf?
3
No1 - 1.9yr
Careful, or I'll turn to the dark side and federation will become a dictatorship so fast it'll make Palpatine spin in his grave. Darth No1orious đ¤Ŗ
3
Lodion đĻđē - 1.9yr
âšī¸
I'll see what I can see...
6
Lodion đĻđē - 1.9yr
Ok, for some reason practically all instances were flagged as "dead" in the database. I've manually set them all to be requeued... server is now smashed as it attempts to updated the ~4000 instances I've told it are no longer dead. See how this goes...
7
Lodion đĻđē - 1.9yr
Replying from my lemmy.world account, seems to have caught up eventually.
6
Lodion đĻđē - 1.9yr
test
1
Lodion đĻđē - 1.9yr
reply
1
Lodion đĻđē - 1.9yr
Seems to have sorted it for the most part... not sure what caused it, will do some more digging.
6
Zagorath - 1.9yr
Boss!
4
Lodion đĻđē - 1.9yr
Ok, something is busted with the lemmy API endpoint that shows current federation state. It is currently showing nearly all remote instances as dead:
But "dead" instances are still successfully receiving content from AZ, and sending back to us.
5
Lodion đĻđē - 1.9yr
Ok, there's more to this than I first thought. It seems there is a back end task set to run at a set time every day, if the instance is restarting at that time the task doesn't run... this task updates the instances table to show remote instances as "seen" by AZ. With the memory leaks in 0.19.1, the instance has been restarting when this task is running... leading to this situation.
I've updated the server restart cronjob to not run around the time this task runs... and I've again manually updated the DB to flag all known instances as alive rather than dead.
Excellent, hopefully that is included in 0.19.3 that came out overnight... will upgrade shortly.
2
Zagorath - 1.9yr
It is. I found the page I linked from the release notes.
1
Lodion đĻđē - 1.9yr
Cool, we're on 0.19.3 now.
2
maniacalmanicmania - 1.9yr
How do things look after the past week?
1
Lodion đĻđē - 1.9yr
0.19.3 still has the memory leak, I've left Lemmy to restart once a day to mitigate this. Federation seems to be fine for the last week after some minor changes and the upgrade.
2
Zagorath - 1.9yr
Sweet
1
maniacalmanicmania - 1.9yr
I found that this post's body text is best viewed on desk/laptop for anyone trying to make heads or tails of all the links on mobile. Edit: Looks like my idea that this issue was due to some instances not being on 0.19.2 has been shot out of the water.
4
Zagorath - 1.9yr
Yeah fair call. I've edited it into a table which should hopefully be easier to read.
4
maniacalmanicmania - 1.9yr
Ooo yeah scrollable table, nice work, much better at least on voyager đ
Zagorath in meta
Ongoing federation issues
I'm still seeing cases where comments don't seem to be federating correctly. For example:
I'm not sure what the cause is, but the last few days it seems more of my comments are failing to federate than are succeeding.
I'm claiming federation issues as the reason my posts aren't getting thousands of upvotes.
I still can't believe you're not president of the world yet. Like wtf?
Careful, or I'll turn to the dark side and federation will become a dictatorship so fast it'll make Palpatine spin in his grave. Darth No1orious đ¤Ŗ
âšī¸
I'll see what I can see...
Ok, for some reason practically all instances were flagged as "dead" in the database. I've manually set them all to be requeued... server is now smashed as it attempts to updated the ~4000 instances I've told it are no longer dead. See how this goes...
Replying from my lemmy.world account, seems to have caught up eventually.
test
reply
Seems to have sorted it for the most part... not sure what caused it, will do some more digging.
Boss!
Ok, something is busted with the lemmy API endpoint that shows current federation state. It is currently showing nearly all remote instances as dead:
But "dead" instances are still successfully receiving content from AZ, and sending back to us.
Ok, there's more to this than I first thought. It seems there is a back end task set to run at a set time every day, if the instance is restarting at that time the task doesn't run... this task updates the instances table to show remote instances as "seen" by AZ. With the memory leaks in 0.19.1, the instance has been restarting when this task is running... leading to this situation.
I've updated the server restart cronjob to not run around the time this task runs... and I've again manually updated the DB to flag all known instances as alive rather than dead.
Will keep an eye on it some more...
For anyone curious, two of the bugs that are related to this:
https://github.com/LemmyNet/lemmy/issues/4288
https://github.com/LemmyNet/lemmy/issues/4039
Thanks for the update!
Looks like what caused this might have been resolved? https://github.com/LemmyNet/lemmy/pull/4377
Excellent, hopefully that is included in 0.19.3 that came out overnight... will upgrade shortly.
It is. I found the page I linked from the release notes.
Cool, we're on 0.19.3 now.
How do things look after the past week?
0.19.3 still has the memory leak, I've left Lemmy to restart once a day to mitigate this. Federation seems to be fine for the last week after some minor changes and the upgrade.
Sweet
I found that this post's body text is best viewed on desk/laptop for anyone trying to make heads or tails of all the links on mobile. Edit: Looks like my idea that this issue was due to some instances not being on 0.19.2 has been shot out of the water.
Yeah fair call. I've edited it into a table which should hopefully be easier to read.
Ooo yeah scrollable table, nice work, much better at least on voyager đ
Cant argue with any of those examples.
In a delicious cherry on top, this thread is also not federating (ex: https://lemmy.world/c/meta@aussie.zone)