We just deployed the latest version of Lemmy 0.19.2 where the github code includes a possible fix for the outgoing federation issues we have been having.
But lets see before we celebrate. Help us test if outgoing federation seems to work now by making comments, posts and upvotes and see if they appear on other instances.
Of course if the other instances are on Lemmy 0.19.0 or 0.19.1, they could have issues with outgoing federation still until they update.
Release notes for 0.19.2: https://join-lemmy.org/news/2024-01-10_-_Lemmy_Release_v0.19.2_-_More_Federation_Fixes
I haven’t seen outbound federation issues (that I’ve noticed) since the 0.19.2 update. However…not an outbound federation problem, but I’ve been seeing a number of problems with not having comments from other instances show up in posts when viewed via lemmy.today.
In Jerboa, this manifests itself as a “there is no record of this comment” errors, with visible child comments beneath. In the Web UI, I don’t see either the parent or child comment. I don’t believe I saw errors like this prior to the 0.19.2 upgrade.
An example of one such comment (on a lemm.ee (0.19.2) community, !Wales@lemm.ee:
This appears fine in the lemm.ee Web UI for me:
https://lemm.ee/comment/8315979
There are presently two comments (a top-level comment from TWeaK@lemm.ee, and a child from Navarian@lemm.ee). Both comments are from users local to the instance. Both are over two hours old as of this writing.
In the lemmy.today Web UI, no comments are visible (though the post does have a comment count of two):
https://lemmy.today/post/5249421?scrollToComments=true
In Jerboa, when logged into lemmy.today, I can see the child comment, but not the parent comment. In place of the parent comment, there is italicized text reading “there is no record of this comment”.
I’m running Jerboa 0.0.55, the latest on F-Droid as of this writing.
I’m assuming that the same problem is producing the output on both, and Jerboa isn’t actually a factor in the problems, that the lemmy Web UI and Jerboa just act differently when a comment is “unreadable” (the Web UI hides the whole tree, Jerboa shows an error placeholder specifically for the comments it can’t see and then tries to show the child comments), but I do not know this to be true for sure.
EDIT: A second example from another instance (feddit.de, also running 0.19.2):
Nine comments as of this writing, and are visible via the feddit.de Web UI:
https://feddit.de/post/7871940?scrollToComments=true
They’re all visible on lemmy.world’s Web UI:
https://lemmy.world/post/10751010?scrollToComments=true
But on lemmy.today’s Web UI, no comments at all are visible:
https://lemmy.today/post/5235270?scrollToComments=true
In Jerboa, when talking to lemmy.today, I can see two comments, one from NoneOfUrBusiness, and one from DarkThoughts:
https://lemmy.world/comment/6715548
https://lemmy.world/comment/6716235
I get the “no record” error for several comments (hard to know which ones, since one can’t see usernames, but due to the comment tree structure, one can infer that the top-level comment from ShroOmeric and the child from RidderSport are among those). Jerboa only actually attempts to display six comments for this post.
OK thanks for the detailed information! I really hate to see that Lemmy still has federation issues. It seems like one of those core things that would just work by now. But any info and help about remaining problems is very valuable for the developers. I will read through what you said here and see if I can create a github issue from it.
Well, IRC had netsplits decades in, so… shrugs
But yeah, I agree, it’s a pain.
Thanks for all your work on this and the earlier problems, though.
Yeah true, it’s always something. :) I used to be on irc too back in the day, but sadly not anymore. But the worse big tech becomes, the more we will want to go back to other tech that didn’t bother us and felt good to use.
Just a wild thought, but are you still running that “restart lemmy periodically” cron job that you put in to deal with the earlier outbound federation problems? I am thinking that maybe the lemmy server software does not deal well with being shut down when it’s trying to fetch a comment, and after it comes back up, never actually tries getting the comment a second time. That might explain why lemmy.today is seeing random comments not being pulled over, but it’s not (at least immediately) apparent on other instances. I don’t know what else would be special about lemmy.today.
I mean, it’d still be a bug in the lemmy server software, just something that maybe lemmy.today could be tickling more than other instances are if it’s frequently restarting.
EDIT: Though if that’s the cause, if it’s also in earlier versions, that wouldn’t explain why I hadn’t noticed this earlier, when lemmy.today was running 0.19.1. I could have missed it in the Web UI, maybe, but it’s pretty obvious when using Jerboa – all those errors in place of comments – and I think I would have seen it.
We dont have anything restarting the server automatically, but you are on to something.
There have been occurances of the lemmy software and postgres restarting now and then since the update (several times per day). Postgres memory usage has been enormous, and we turned off the extra web user interfaces for a while to see if its still happening. So far its actually looking a lot better and memory usage looks stable, so maybe our hunch is right…
There’s some other issue with posts not propagating to lemmy.today. I’m not sure at what point this arose; I’ve only just noticed it.
I’m currently subscribed to !retrogaming@lemmy.world.
This is lemmy.today’s view of that community:
https://lemmy.today/c/retrogaming@lemmy.world
The newest post to be visible is “Death Knights of Krynn [MS DOS] Intro on Creative Music System/Game Blaster”, visible here:
https://lemmy.today/post/5957815
As of this writing, that post was five days old.
However, actually looking at the community on lemmy.world, there have been newer posts there. These have not propagated.
https://lemmy.world/c/retrogaming?dataType=Post&page=1&sort=New
As of this writing, there are four, the oldest of which is also five days old (so presumably, whatever caused the issue happened at least five days back, possibly exactly five days back). None of these have propagated to lemmy.today via federation.
https://lemmy.world/post/11464649
https://lemmy.world/post/11489415
https://lemmy.world/post/11526052
https://lemmy.world/post/11531592
The newer posts have propagated to some other instances that I have looked at, including lemmy.ml (0.19.3):
retrogaming@lemmy.world on lemmy.ml
feddit.de (0.19.2):
retrogaming@lemmy.world on feddit.de
feddit.uk (0.19.2):
retrogaming@lemmy.world on feddit.uk
Both lemmy.today and lemmy.world are reporting that they are federated with each other:
https://lemmy.today/instances
https://lemmy.world/instances
That is making me a little suspicious that whatever is broken is somehow specific to lemmy.today (and apparently an ongoing problem, not a one-off, since it’s not just a single missing post).
I also have one request. Would it be possible to create a “lemmy.today community”? This could serve as a point for people to discuss or report issues on the lemmy.today instance (like this one). The closest thing we have right now is the “announcements” community, but post creation there is (understandably) locked for non-moderators. I’d do it myself, but I think that it’d probably be more appropriate were an instance admin the moderator. And while adding comments to this post is probably the best existing way to both notify you that there is an issue and make anyone else on the instance who might be experiencing the problem aware of it, I feel like most people aren’t likely to go hunting into a recent update announcement to see if there are any issues on the instance or try to troubleshoot or help each other. I’d just message you directly with issues, but then other people can’t see discussion or help or even be made aware that there is an issue.
On lemm.ee, there is such a community, with the name of !meta@lemm.ee.
I checked several other large instances, but wasn’t able to find a similar-such community (I was looking to see if any convention had emerged as to what name to use for “discussion about the local instance”). I’m not sure that “meta” is the best term to use (I think it might be clearer to use the instance name or something, like !lemmytoday@lemmy.today, to make it clear that it’s about the instance) but “meta” is short, and it has uptake from at least one instance.
Thanks.
Hi!
I looked and I can see the posts if I sort by New (default sort is Top Day I think):
https://lemmy.today/c/retrogaming@lemmy.world?dataType=Post&sort=New
That page has all the posts you mention above from the looks of it? Correct me if im wrong or if Im misunderstanding something.
And of course we can create a
lemmytodaycommunity, sounds like a great idea. A place to bring up issues about the instance. I agree with you about the name also. Meta doesnt sound good, and also its related to that Zuckerberg place… :)Hmm. I’m sure that I checked sorted by New.
investigates further
Okay, this is more confusing. It looks like what happens is that when I am logged in to lemmy.today, I cannot see the newer posts via lemmy.today. When I am not logged in, I can. That means that it can’t be a federation problem, that something else is going on.
I’ve got no idea what would cause that. I don’t have my account set up to block anything at all. I don’t think that there’s some kind of remote per-user blacklist or even if there were, any idea why I’d be on it.
considers
If there is some sort of blacklist, I assume that I wouldn’t be able to comment there.
checks
It looks like I can comment in the community there just fine, and that a comment made on lemmy.today promptly propagates to lemmy.world:
https://lemmy.world/comment/7332440
Hmm. Well, now I am definitely puzzled.
If you don’t mind me asking, when you viewed the page and were successfully able to see the newer posts, were you logged in to lemmy.today? I’m half-inclined to go create a second account on lemmy.today to see whether it is also affected, but if your account is impacted as well, that’s at least one more data point, and then I’m not just creating throwaway accounts.
My understanding is that lemmy very recently got per-user instance blocking added as a feature – kbin has had this for some time – and I am wondering if maybe something might not be working correctly related to that. I can’t imagine what else would cause the user account involved to be an input.
Thank you for looking at this.
EDIT: Maybe if there is something in the stack that caches generated webpages and those aren’t being invalidated, since user might be an input there, but then…no, that can’t be it, because I also don’t see the newer posts in Jerboa.