We just had another unscheduled downtime due to a linux kernel bug.

Yesterday we noticed some issues with the server - mainly that we couldnt stop some docker containers. As you know, we are running some extra user web interfaces for Lemmy and we noticed that they started acting weirdly and had one cpu running at 100% constantly.

I wanted to restart those containers but I couldnt stop them. Found some posts online that this is a bug in the Ubuntu linux kernel: https://github.com/moby/moby/issues/43094.

Our stop attempts caused the docker platform to behave weirdly and it started to affect the main Lemmy software, so we did a reboot of the server and installed the latest updates.

We are very sorry for this unscheduled downtime. :/ Did you guys notice weirdness with Lemmy in the last 8 hours or so?

  • megalow
    8·
    1 year ago

    I didn’t notice anything strange besides the downtime. Thanks for the update and information!

  • 1984
    4·
    1 year ago

    Thanks for information, also didn’t notice anything special really except a few minutes of downtime just now.

  • Rivalarrival
    3·
    1 year ago

    I’ve noticed some connectivity issues for the past couple days. Seemed like my client (Thunder) wasn’t getting a response from lemmy.today, and eventually timed out. Most times I was able to sign out and back in immediately.

    Shit happens, you’re doing great.

    • mrmanagerOPMA
      1·
      1 year ago

      Yeah it’s been a rocky ride since the update, and we had to restart the server a number of times to diagnose issues with outgoing federation that came with Lemmy 0.19.1.

      Very sorry about that, and hopefully there is a fix for this coming soon from the developers. Thanks for being so nice about it. :)