Edit : OK so the outage was less brief than expected. The upgrade of the 3GB pictrs database took over 1 hour, and the version 0.4 database is now 14.5 GB… But anyway, it seems to be working alright now!
We will upgrade pictrs today, from 0.3.1 to 0.4.4. This will enable us to switch to S3 storage later on, and is needed for more anti-CSAM tools. Outage should only involve picture uploads, and should be brief.
Awesome that pict-rs has postgres and full s3 support now. Lemmy is getting really scalable!
I think the postgres support will come in 0.5?
Ah, right. I was aware of the ability to use S3 for 0.4, but this post made me go check out Asonix’s Forgejo. The fact that you can get rid of sled in 0.5 and go fully stateless/scalable is a game changer.
How will the postgres support look like? They are not storing actual images in the database I hope?
No. Just like now , they store the metadata there, the images are on disk or S3
Very good, thank you.
Yeah we can upload pictures again!
OK, the pictrs database upgrade is taking it’s time… please wait… ;-)
pictrs_1 | {"timestamp":"2023-10-02T16:31:44.746467Z","level":"WARN","fields":{"message":"new"},"target":"pict_rs::repo","span":{"name":"Migrating Database from 0.3 layout to 0.4 layout"},"spans":[]}how’s it faring? 👀
It’s building a 0.4 database, which is already twice as big as the 0.3 one
It’s grown from 3.1 GB to over 8GB and still running…
thats a lot of pictures
And that’s just his personal collection of feet pics
Don’t tell them about the feetish
That’s not the pictures… The picture take up 1.2 TB… it’s just the database with the metadata about the pictures. (11.8 GB now…)
I’m curious, how much storage in total is needed for a Lemmy instance like this one currently?
1.2 TB for pictures and 100GB for Postgres
For reference I run a pretty small instance. I’m at 34GB for the main database, and about 120GB of media. (Doesn’t count backups)
My instance is much, much smaller though. A few hundred users, and probably only a couple dozen actually active.
Who pays for that s3 storage?
The people that donate to lemmy.world
But it’s not in S3 yet. Still on disk.
I know. Giant database = Giant × 10² amount of Pictures
Are you manually updating Pictrs or is this included in the new lemmy update?
Newest update defaults to 0.4, but the latest version is 0.4.4. I don’t think there’s a significant difference.
But it looks like ruud stayed on 0.3.x until now because there’s a pretty serious migration between 0.3.x and 0.4.x
deleted by creator
deleted by creator
thanks, glad you do the homework https://www.google.com/search?q=pict-rs
Here is an alternative Piped link(s):
https://www.piped.video/watch?v=uVXR2LYeFBI
Piped is a privacy-respecting open-source alternative frontend to YouTube.
I’m open-source; check me out at GitHub.
