Outgoing federation problem · Issue #4288 · LemmyNet/lemmy
github.com
Requirements Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support Did you check to see if this issue already exists? Is this only a single bug? Do not put multipl...
    • Aurix@lemmy.worldEnglish
      439·
      11 months ago

      I am not sure what you imply. Like Early Access it might be a descriptive factor, but it matters very little. Some use that label for finished products with full support adding new features, while others use it as a means they skipped quality control entirely or are prototyping heavily. In the end it still is a paid for software. In Lemmy’s case it is the first with its own financ model. It is highly usable and hopefully this bug sees the improvement in testing to avoid.

      LibreOffice has a much higher version number, but as my small rant post on it, if I was responsible for QA I wouldn’t have let it release at all in the current state and am baffled what goes on internally to cause that.

        • fmstrat@lemmy.nowsci.comEnglish
          10·
          11 months ago

          I think it’s more about the fact that Lemmy is going to be used as, and have expectations of, a finished product, whether they say it is or not. Any project, FOSS or not, that grows this quick goes through this stage.

          I’m not saying those expectations are valid, I’m just saying the vast majority of people will not be able to comprehend how an issue like this can make it’s way through given the maturity that Lemmy has, since it works so well most of the time despite being alpha.

          Edit: Clarity: I think the 0 should be in the title regardless.

  • Levsgetso@lemmy.zipEnglish
    403·
    11 months ago

    Lemmy has been feeling pretty empty the past couple of days. That just really goes to show how important federation really is.

    • ProdigalFrog@slrpnk.netEnglish
      7·
      11 months ago

      I noticed that as well, thought maybe things were just slowing down, but it’s good to know it was the software and not a sudden drop in community involvement. Since admins became aware of the bug and began restarting their servers as an ad-hoc fix, I’ve been seeing way more engagement on my instance.

    • RBG@discuss.tchncs.deEnglish
      21·
      11 months ago

      Yes and no, if you have enough users you do not need federation for a server to not feel empty. But overall many servers do not have that many users, so federation with larger ones is essential.

    • sir_reginald@lemmy.worldEnglish
      51·
      11 months ago

      for the most part federation works just fine. there are occasional issues, yes, but it’s not like it can’t federate at all.

  • 1984English
    15·
    11 months ago

    I would comment on this but since federation is broken…this won’t be seen until the instance admin restarts Lemmy and federation works once again.

    • BlueÆther@no.lastname.nzEnglish
      131·
      11 months ago

      Federation seems to be working fine from where I stand?

      Edit:

      Edit 2:

      Even the edit federated out to your instance fine…

      • Skull giver@popplesburger.hilciferous.nlEnglish
        5·
        11 months ago

        I have mh suspicions that this is some kind of deadlock issue. After fucking up my config during am upgrade (too many database connections, too many workers) the new multithreaded federation thingy seemed to become quite unstable. From what I could tell, the threads that are supposed to deal with federation all broke/shut down, but didn’t get automatically restarted again, leaving some messages and instances in limbo until the entire server got restarted.

        After fixing my database connection issue and reducing the amount of workers, I don’t think I’ve encountered many issues since. I occasionally click the little icon above a post to check if my response had reached the community server, and usually it does, pretty quickly.

        This issue sounds awful to have to debug, best of luck to the Lemmy devs in finding a fix!

      • 1984English
        5·
        11 months ago

        I think it works for a while after restart of the server, and then stops working. Yesterday I was making comments that didn’t federate until 12 hours later when I assume the instance admin restarted the server.

        • BlueÆther@no.lastname.nzEnglish
          52·
          11 months ago

          My instance has been uo for over 3 days since 0.19.1 update with no issues - very little load though

          • 1984English
            2·
            11 months ago

            Yeah it seems to work for some instances with 0.19.1 but Lemmy.today has issues and feddit.de as well, just from what I’ve heard. Probably plenty more.

  • RandomVideos@programming.devEnglish
    141·
    11 months ago

    A while ago, programming.dev had a problem where nothing from the instance federated for a week. Lemmy felt so empty

  • thisisawayoflife@lemmy.worldEnglish
    121·
    11 months ago

    Am I little naive in wondering how this isn’t caught in unit, integration and E2E tests?

    Edit: looks more involved than that, but it forced me to dig into how some of the components are tested. Educational.

    • kattenluik@feddit.nlEnglish
      2016·
      11 months ago

      The main Lemmy developers very often don’t have the biggest clue of what they’re doing, and they’ve proven that time and time again. A lot of times they forget normal web practices and rules, and a lot of side things like Jerboa are also just broken.

      They also don’t know how to interact with the community or how to be normal people. Let’s not forget they have been working on Lemmy full-time and this is the state we’ve come to, ignoring all the database and security and everything else issues they already had.

        • kattenluik@feddit.nlEnglish
          244·
          11 months ago

          Lemmy IS awesome, that doesn’t mean we can’t want it to be better or criticize it. No one said Lemmy isn’t great.

          • sir_reginald@lemmy.worldEnglish
            173·
            11 months ago

            The main Lemmy developers very often don’t have the biggest clue of what they’re doing

            this doesn’t sound constructive criticism to me.

            that commenter has something personal against the devs. Lemmy is far from perfect but it’s functional and improving everyday.

        • linearchaos@lemmy.worldEnglish
          6·
          11 months ago

          I love writing tests, It’s all the shit that comes after that that sucks.

          Those first few pushes that all come up green feel like magic. That first red that points out something you missed, you go back and make a quick change and it’s now green and it’s the best thing you’ve ever seen.

          It’s sooner or later, you throw a couple big red bois on a production build that don’t make any sense. You start digging through the code of some guy that only writes comments in haiku and has the impression he gets paid by the number of layers deep he can nest a ternary.

          Sooner or later you figure out it’s just an edge case there’s nothing actually wrong. You’ll need to refactor one of the systems but you still have production to push to fix a critical bug, so you hotwire the test and write it off as P1 tech debt.

          Eventually, you end up with unit tests that aren’t P1 and they fail. If you’re understaffed, or overscoped, sooner or later you just have a bunch of half-assed zombie test sitting around. Unless you can convince production to let you go back and clear up your tech debt it’s just a unit test graveyard. It still has the big bumpers in place so something serious can’t fail. But you never seem to be able to get back to make everything bright new and shiny again.

          • thisisawayoflife@lemmy.worldEnglish
            1·
            11 months ago

            That does sound like a nightmare. I’m assuming you mean failed test when you say “red boy”, and that made me wonder about PR practices. I’m used to a very strict review environment and fairly quick review turnaround or requests to go over the code. I’ve heard horror stories about people not getting PRs reviewed for days or weeks or some people just plain refusing to review code. I work on microservices that are all usually less than 10,000 lines though, not something with over a million lines of legacy code.

        • fuckwit_mcbumcrumble@lemmy.worldEnglish
          3·
          11 months ago

          I wish more developers would do QA. After working with QA my code improved so much because I was proactively thinking about how things might break or potential issues that I never would have thought of.

          • thisisawayoflife@lemmy.worldEnglish
            2·
            11 months ago

            Yep. When I was still doing QA, I saw some pretty terrible practices and tested code that barely built. Now as a software engineer, I have no QA and rely heavily on my own testing practices, namely, unit testing first, integration testing and system/e2e testing. I can’t guarantee the code is bug free and there’s parts I know that could be refactored (tech debt), but I know each piece is tested and does what I expect it to. As corny as it sounds, I’m a big fan of TDD. Unit/IT/E2E don’t replace QA in my opinion, but better set QA up to focus on the bugs that matter and not basic stuff.

        • ExLisper@linux.communityEnglish
          1·
          11 months ago

          Personally I dislike writing useless tests. I use test as a development tool (it’s easier to implement some DB operations for example by writing tests than performing some actions manually) and to test logic that can actually fail because of changes in other part of the code withouts me noticing. Testing thinks like “button calls click() method when clicked” is IMHO pointless. If someone can change this code and push to prod without testings manually or doing code review they can also disable the test without anyone noticing.

          • thisisawayoflife@lemmy.worldEnglish
            2·
            11 months ago

            I agree, writing meaningless tests helps nobody and just creates extra work everyone. Unit tests should prove functionality and integration tests act as a vise. Much like you said, if a test breaks in that scenario, then you know something in another class has violated that contract. Good tests will have meaningful names and prove functionality, especially in the backend where it is especially important…

            You mention (what I would consider) a bad practice of allowing merges without review. While that should be possible on personal projects with only one dev, strict review guidelines should exist so that nobody can just “push to prod”. CICD is your friend - use it so that staging and prod never break. Again, I’m used to working on systems used by scores of millions of users so I appreciate forced automated validation. Nobody likes dumb breaks on a Friday before vacation.

  • walden@sub.wetshaving.socialEnglish
    9·
    11 months ago

    I’m going to set up a cronjob to restart the Lemmy docker container every 6 hours. That seems to be around when stuff stops working usually.

  • thegiddystitcher@lemm.eeEnglish
    91·
    11 months ago

    Yup, absolutely nothing I posted from lemm.ee yesterday made it out into the wider world. In fact I’ll probably have to manually federate this comment too.

    Hopefully they can get to the bottom of it. Maybe next year we need a rule about releasing anything in December when everyone’s too busy to bugfix 😅

      • thegiddystitcher@lemm.eeEnglish
        8·
        11 months ago

        My other account is on .world so I can use search to forcibly pull things onto at least that server (search for the original URL of the unfederated post or comment and then wait a few seconds). Sometimes doing that will also encourage whatever it was to federate more widely to more servers, but that doesn’t seem to be happening as reliably either since the upgrade 🤷‍♀️

    • 1984English
      3·
      11 months ago

      A restart of Lemmy seems to fix it temporarily according to admin of Lemmy.today.

      But it’s inconsistent and stops working after a while it seems.

  • maegul (he/they)@lemmy.mlEnglish
    55·
    11 months ago

    I’d say the bug seems to be slightly different from my observations, in that things are leaky rather than broken. Could be wrong though.

    Edit: previously post was mobile keyboard mince meat:

    I’d say the big seems to be slightly different from my observations, in that things are meant rather than broken. Could be wrong though.