Hello World,
as many of you know, several newer Lemmy versions have been released since the once we are currently using.
As this is a rather long post, the TLDR is that we’re currently planning for late January/early February to update Lemmy.World to a newer Lemmy release.
We’re currently running Lemmy 0.19.3 with a couple patches on top to address some security or functionality issues.
As new Lemmy versions have been released, we’ve been keeping an eye on other instances’ experiences with the newer versions, as well as tracking certain issues on GitHub, which might impact stability or moderation experience.
We updated to Lemmy 0.19.3 back in March this year. At that point, 0.19.3 had been released for a little over a month already and at that point all the major issues that troubled the earlier 0.19 releases had been addressed.
Several months later, in June, Lemmy 0.19.4 was released with several new features. This was a rather big release, as a lot of changes had happened since the last release. Only 12 days later 0.19.5 was released, which fixed a few important issues with the 0.19.4 release. Unfortunately, Lemmy 0.19.5 also introduced some changes that were, and to some part are still not fully addressed.
Prior to Lemmy 0.19.4, regular users may see contents of removed or deleted comments in some situations, primarily when using third party apps. Ideally, this would have been fixed by restricting access to contents of removed comments to community moderators in the communities they moderate, as well as admins on each instance. Deleted comments will be overwritten in the database after some delay, but they might still be visible prior to that. This is especially a problem when moderators want to review previously removed comments to either potentially restore them or to understand context in a thread with multiple removed comments. Lemmy modlog does not always record individual modlog entries for bulk-removed items, such as banning a user while also removing their content would only log their ban but not the individual posts or comments that were removed.
We were considering writing a patch to restore this functionality for moderators in their communities, but this is unfortunately a rather complex task, which also explains why this isn’t a core Lemmy feature yet.
While admins can currently filter modlog for actions by a specific moderator, this functionality was lost somewhere in 0.19.4. While this isn’t something our admin team is using very frequently, it is still an important feature to have available for us for the times we need it.
This also included a few security changes for ActivityPub handling, which resulted in breaking the ability to find e.g. Mastodon posts in Lemmy communities by entering the post URL in the search. It also caused issues with changes to communities by remote moderators.
The 0.19.4 release also broke marking posts as read in Sync for Lemmy. Although this isn’t really something we consider a blocker, it’s still worth mentioning, as there are still a lot of Sync for Lemmy users out there that haven’t noticed this issue yet if they’re only active on Lemmy.World. Over the last 2 weeks we’ve had nearly 5k active Sync for Lemmy users . This is unfortunately something that will break during the upgrade, as the API has changed in upstream Lemmy.
There are also additional issues with viewing comments on posts in local communities that appear to be related to the 0.19.4/0.19.5 release, appear to be a lot more serious. There have been various reports of posts showing with zero comments in Sync, while viewing them in a browser or another client will show various comments. It’s not entirely clear to us right now what the full impact is and to what extent it can be mitigated by user actions, such as subscribing to communities. If anyone wants to research what is needed to restore compatibility and potentially even propose a patch for compatibility with both the updated and the previous API version we’ll consider applying it as a custom patch on top of the regular Lemmy release.
If there won’t be a Sync update in time for our update and we won’t have a viable workaround available, you may want to check out !lemmyapps@lemmy.world to find potential alternatives.
There were also several instances reporting performance issues after their upgrades, although they seemed to mostly have been only for a relatively short time after the upgrades and not persistent.
Lemmy 0.19.6 ended up getting released in November and introduced quite a few bug fixes and changes again, including filtering the modlog by moderator. Due to a bug breaking some DB queries, 0.19.7 was released just 7 days later to address that.
Among the issues fixed in this release were being able to resolve Mastodon URLs in the search again and remote moderators being able to update communities again.
0.19.6 also changed the way post thumbnails generated, which resulted thumbnails missing on various posts.
A month later, now we’re in December, 0.19.8 was released.
One of the issues addressed by 0.19.8 was Lemmy returning content of removed comments again for admins. For community moderators this functionality is not yet restored due to the complexity of having to check mod status in every community present in the comment listing.
At this point it seems that most of the issues have been addressed, although there seem to still be some remaining issues relating to thumbnails not reliably being created in some cases. We’ll keep an eye on any updates on that topic to see if it might be worth waiting a little longer for another fix or possibly deploying an additional patch even if it may not be part of an official Lemmy release yet at the time.
While we were backporting some security/stability related changes, including a fix for a bug that can break federation in some circumstances when a community is removed, we accidentally reverted this patch while applying another backport, which resulted in our federation with lemmy.ml breaking back in November. This issue was already addressed upstream a while back, so other instances running more recent Lemmy versions were not affected by this.
Among the new features released in the Lemmy versions we have missed out on so far, here are a couple highlights:
- Users will be able to see and delete their uploads on their profile. This will include all uploads since we updated to 0.19.3, which is the Lemmy version that started tracking which user uploaded media.
- Several improvements to federation code, which improve compatibility with wordpress, discourse, nodebb.
- Fixing signed fetch for federation, enabling federation with instances that require linked instances to authenticate themselves when fetching remote resources. Not having this is something we’ve seen cause issues with a small number of mastodon instances that require this.
- Site bans will automatically issue community bans, which means they’re more reliable to federate.
- Deleted and removed posts and comments will no longer show up in search results.
- Bot replies and mentions will no longer be included in notification counts when a user has blocked all bots.
- Saved posts and comments will now be returned in the reverse order of saving them rather than the reverse order of them being created.
- The image proxying feature has evolved to a more mature state. This feature intends to improve user privacy by reducing requests to third party websites when browsing Lemmy. We do not currently plan on enabling it with the update, but we will evaluate it later on.
- Local only communities. We don’t currently see a good use for these, as they will prevent federation of such communities. This cuts off users on all other instances, so we don’t recommend using them unless you really want that.
- Parallel sending of federated activities to other instances. This can be especially useful for instances on the other side of the world, where latency introduces serious bottlenecks when only sending one activity at a time. A few instances have already been using intermediate software to batch activities together, which is not standard ActivityPub behavior, but it allows them to eliminate most of the delays introduced by latency. This mostly affects instances in Australia and New Zealand, but we’ve also seen federation delays with instances in US from time to time. This will likely not be enabled immediately after the upgrade, but we’re planning to enable this shortly after.
edit: added information about sync not showing comments on posts in local communities
Wait, mods can undelete and revert edits? That’s not scary at all…
not at all. nobody can revert edits, only the creator can edit it again.
moderators can restore removed (mod action) posts and comments, but not deleted (creator action) posts and comments.
Ah, that makes sense.
Thank you for all you do to make this place so great. I have promised myself to sign up to Patreon specifically to support the server as soon as I have a job. You provide all of this for us for free and I would like to give back.
Obviously going to do that $8 tier. I need that non-offer!
Can you link the patreon please?
Sure! It’s also linked to in the sidebar on the front page along with other donation options:
https://www.patreon.com/mastodonworld
(I know that says “mastodonworld,” but it supports three .world servers including lemmy.world.)
My mistake. I thought it said three on the Patreon.
it does indeed. i’ve let @ruud@lemmy.world know.
Appreciate the communication. It’s nice that a large instance works to be stable, both in tech and in your organization (a non-profit foundation based in the Netherlands). It sure beats hosting it out of my basement server as my personal pet project.
I like the stable pace of updates. Good work!
Thank you for the update and it’s good to hear your upcoming plans. Being one of those people in Australia (Reddthat) it will be good to see if it actually works as it’s designed too!
I’d love to save $7/m to not have a server dedicated to batching the federation traffic 😅When you lay out the timelines for 0.19.3 onwards no time at all has gone by, and having to deal with the issues after .3 has certainly not been fun as an admin. (And I’m only a small server compared!)
Being such a huge player in our Lemmyverse, thanks for taking the time to plan this out as I know how much testing has been done to get us this far.It’s always a nice experience chatting to the LW team!
Hope your updates go smoothly!Thank you for taking updates as seriously as you do. Making sure to only update when it is overall a good idea to do so. I’m sure you get no shortage of flak for being out of date. Thank you for being able to stand up to that.
Your well thought out decision process on everything is what reassures me I found a good home. And your open and honest posts about every major decision are nice to read when I initially disagree with a decision made, until I see why it was made.
There seems to be an issue with Youtube links posted from .world accounts not showing thumbnails, is this related to the thumbnail bug in the post?
this is a completely different issue than the one seen on the newer lemmy versions unfortunately. i believe that issue in newer lemmy versions is already prepared to be addressed in the next lemmy release.
for some reason youtube/google decided that hetzner customers don’t get any opengraph metadata anymore, so it doesn’t really matter which lemmy or even client in general we use. fetching youtube pages with curl yields the same result where the necessary metadata isn’t included.
this also affects other instances hosted on hetzner.
Ouch. Thanks for the answer.
So I’m guessing we shouldn’t expect it to work in the near future unless the folks at YT change their metadata output, correct?
Unfortunately at this point I don’t see another option.
Dang, I found nothing on c/lemmysupport but didn’t think of searching on c/fediverse, thanks.
…though I’m still kinda lost, is the TL;DR just that posts from non-.world accounts are all using the 0.19.4 function to manually add thumbnails?
Not an expert, but that’s my understanding
Re: Sync users
I haven’t been able to view comments on posts on any local communities since the update unless I’m the poster or mod of the community.
No idea why.
Edit: there’s no outrage about this, so I may be alone, which is more frustrating, but maybe there’s just that many of you still on LW
Edit: each Lemmy.world user replying proved that point lol.
I was surprised to see such a large part of our active users using Sync, although given that there aren’t really any bad breakages yet it shouldn’t be that unexpected.
The comments in local communities issue seems a lot more serious than “only” not marking posts as read, I’ve read about that before and totally forgot about it since. There were quite a few threads in !syncforlemmy@lemmy.world about that.
If anyone wants to spend some time on figuring out which change exactly broke this for Sync we might consider a patch to restore compatibility with the old API if it’s simple enough to do. I probably won’t have time to look into this much until somewhere in January, and my time will probably be better spent with migration preparation and testing of Lemmy itself there.
People looking for alternatives might find some inspiration in !lemmyapps@lemmy.world.
The only issues i run across using Sync are glide ocassinaly failing to grab an image which i dont think is an exclusively Sync related problem.
I am running the most current version of Sync available on the Play Store via android.
It’s possible it’s only me and I’m just unlucky tbh.
The app is beyond broken for me since it stopped supporting mark read on scroll… The worst thing is that AFAIK this “new” change is just a minor code fix… But the dev is missing.
What do you mean?
What is “mark read on scroll”?
You scroll, the scrolled posts get marked as read, you can then hit that pretty button to clean up said read posts.
Boom, you have the cleanest newer feed baby! (Unless there is an ongoing popular news lmao)
This is how I browsed Reddit and now Lemmy all the way.
My replacements for Sync that do this are:
-
Summit
-
Voyager
-
Boost
-
Eternity
-
Sync is working mostly fine for me.
Not having this issue using sync currently
You Sync-using guys discussing this might also want to list the version of Sync that you’re using. Could be that the Sync version is a factor.
Most of the commenters are on lemmy.world, which is the bigger factor here.
Also a Sync user. I haven’t noticed this, but I’ll be on the lookout for it, now.
Nice to see this post !
I’m not sure I understood the update schedule tho😅. Are you planning for an update ? Is there an ETA?:))
They are waiting a bit more to see if some fix arrives or not:
we’re currently planning for late January/early February to update Lemmy.World to a newer Lemmy release.
A bit more detail:
We’ll keep an eye on any updates on that topic to see if it might be worth waiting a little longer for another fix or possibly deploying an additional patch even if it may not be part of an official Lemmy release yet at the time.
Oh, I guess I missed them. Thanks a lot! I hope devs have the time to update their apps until then too
Those new versions have been around for a while, Voyager and other apps updated a while ago for other instances
Some others haven’t like sync boost and eternity. I think
Sync seems okay: https://feddit.org/post/6031775/3789768
Boost and Eternity not sure, their communities are not that active
Maybe you can post on !lemmyapps@lemmy.world to have people from other instances using those apps to share their experiences?
there’s another sync issue where posts in local communities don’t show any comments, i had forgotten about that. there are several posts in !syncforlemmy@lemmy.world about that issue, as well as 2 issues on the github issue tracker.
Ah, good to know!
That looks like horrible semver
Semver isn’t a thing for pre v1 versions
Sure it is.
https://semver.org/#spec-item-4
Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.
that isn’t saying that you should change things, but it’s technically not against semver.
given that the lemmy api is versioned separately though and it’s currently at v3, with v4 in active development, that would be good to avoid breaking changes there. I don’t know if that one is supposed to be sever though.
deleted by creator
Can someone please recommend a US-based instance to replace “.world”?
midwest.social is the only US-based instance I know of
https://lemm.ee/post/49433090/16770726
https://discuss.online/ is US based as well
And they’re not without their own drama
I’ve heard people say discuss.online is a good one
Thanks!
I’m a little biased, but I like it here 😁 Going to plug !AskUSA@discuss.online, for casual conversation about US-related topics.
Ew, tankie instance 🤢
I thought communists were starving in the winter 😊
.ml is a “communist/authoritarian” instance so if that’s your thing…
Do people have backup accounts , e.g. on several servers, for when one is down?
Pretty sure they’re based out of Mali
The ml stands for “Marxism-Leninism” as a tongue-in-cheek joke rather than anything to do with Mali. Similar to how a lot of domains ending in .ly (i.e. bit.ly) has nothing to do with Libya.
The “Marxism-Leninism” reference isn’t tongue-in-cheek.
It’s the whole reason lemmy exists. The .ml creators got mad reddit mods pushed back on extreme disinfo and created the thing were using now.
I don’t want to get pedantic about this. There’s two reasons the “original” lemmy uses ml as its TLD:
- It’s cheap
- They can claim it stands for what we’ve been talking about.
Perhaps “tongue-in-cheek” was the wrong descriptor. More pointing out it has nothing to do with Mali, but sites haven’t been using TLDs properly for ages now.
Well that’s the first I’m hearing of this
Man, of all people I thought you’d catch the sarcasm lol, thanks though.
To be fair, I was really surprised about your comment, it makes more sense now 😄
Sometimes they deny it, but if you go on a .ml community and comment anything negative about China or Russia, you’ll find out real fast.