Good post; kinda surprised sshfs is outperforming cifs and makes me need to take a second look at that because, boy, do I ever not like how samba performs, though I’m willing to chalk some of that up to configuration weirdness on my end since I have samba configured to allow any version of Windows that could ever connect to smb/cifs shares to be able to. (Retro computing yay.)
Also, I’d also like to toss in iDrive e2 as a cheap S3 blob storage provider.
I’m paying ~$30 a year for 1tb, with “free” egress. (They operate on the IT’S ON SALE! pricing nonsense so your price will certainly vary because well, it’s always on sale, but always different amounts but $30 is the usualish price.)
You get zero useful support, less than the best performance I’ve ever seen, but it’s shockingly cheap and in the last ~2 years (out of the VA datacenter) I’ve had exactly ONE downtime where it wasn’t working, for about three hours.
Good enough to stuff server backups and object storage for a couple of websites.
Oh, and “free” egress means up to 3x the amount you have stored, so it’s probably bad if your majority use is going to be public downloads, but if it’s not, it’ll probably never be an issue; I have like 600gb of backups sitting there so lots of buffer.
CIFS supports leases. That is, hosts will try to ask for exclusive access to a file, so that they can assume that it hasn’t changed.
IIRC sshfs just doesn’t care much about cache coherency across hosts and just kind of assumes that things haven’t changed underfoot, uses a timer to expire the cache.
considers
Honestly, with inotify, it’d probably be possible to make a newer sshfs that does support leases.
I suspect that the Unixy thing to do is to use NFSv4 which also does cache coherency correctly.
It is easy to deploy sshfs, though, so I do appreciate why people use it; I do so myself.
Here are some 2019 benchmarks that show NFSv4 to generally be the most-performant.
The really obnoxious thing about NFSv4, IMHO, is that ssh is pretty trivial to set up, and sshfs just requires a working ssh connection and sshfs software installed, whereas if you want secure NFSv4, you need to set up Kerberos. Setting up Kerberos is a pain. It’s great for large organizations, but for “I have three computers that I want to make talk together”, it’s just overkill.
EDIT: I’d also add that I kind of wish that Linux authentication were somewhat more-unified in general in 2024. You’ve got:
/etc/shadow passwords (the above with ssh, plus plenty of other services like CUPS).
Wireguard keys
GPG keys (email, git commits)
X.509 certs (email, TLS, smartcard applications)
Kerberos (NFSv4, CIFS at least optionally)
Then you’ve got various keyrings and credential caches, like ssh-agent, gpg-agent, Gnome has some keyring that can wrap ssh-agent, web browsers have a keyring…
I mean, there’s kind of a lot of overlap among all these. Maybe one system would be too far, but I’d kind of like to have things more-unified than they are today.
EDIT2: Apparently inotify() doesn’t let one block the operation that one is monitoring, so probably can’t use it to implement leases.
I’m an idiot. I do have NFS setup on the NAS (I mean, because why not?) but I always forget it’s there, since one client OS (Mac OS) doesn’t support it basically at all, and the other (Windows) does, but it’s not really integrated into the GUI at all, and I’m lazy. I should see what the performance looks like between Windows SMB and NFS implementations are.
As for your key storage, I bloody love my (pair of) Yubikey 5s. I’ve stuffed a giant pile of keys and certs in there and basically don’t think about managing them anymore because, well, it’s just there and just works*.
*Okay the setup was a fuck and a half, but I mean, that does technically qualify as works.
Good post; kinda surprised sshfs is outperforming cifs and makes me need to take a second look at that because, boy, do I ever not like how samba performs, though I’m willing to chalk some of that up to configuration weirdness on my end since I have samba configured to allow any version of Windows that could ever connect to smb/cifs shares to be able to. (Retro computing yay.)
Also, I’d also like to toss in iDrive e2 as a cheap S3 blob storage provider.
I’m paying ~$30 a year for 1tb, with “free” egress. (They operate on the IT’S ON SALE! pricing nonsense so your price will certainly vary because well, it’s always on sale, but always different amounts but $30 is the usualish price.)
You get zero useful support, less than the best performance I’ve ever seen, but it’s shockingly cheap and in the last ~2 years (out of the VA datacenter) I’ve had exactly ONE downtime where it wasn’t working, for about three hours.
Good enough to stuff server backups and object storage for a couple of websites.
Oh, and “free” egress means up to 3x the amount you have stored, so it’s probably bad if your majority use is going to be public downloads, but if it’s not, it’ll probably never be an issue; I have like 600gb of backups sitting there so lots of buffer.
I really need to move my CIFS shares to NFS now that I’ve migrated to linux for everything. It’d probably fix half the errors I regularly have tbh.
Ah NFS… It’s so good when it works! When it doesn’t though, figuring out why is like trying to navigate someone else’s house in pitch dark.
CIFS supports leases. That is, hosts will try to ask for exclusive access to a file, so that they can assume that it hasn’t changed.
IIRC sshfs just doesn’t care much about cache coherency across hosts and just kind of assumes that things haven’t changed underfoot, uses a timer to expire the cache.
considers
Honestly, with inotify, it’d probably be possible to make a newer sshfs that does support leases.
I suspect that the Unixy thing to do is to use NFSv4 which also does cache coherency correctly.
It is easy to deploy sshfs, though, so I do appreciate why people use it; I do so myself.
kagis to see if anyone has benchmarks
https://blog.ja-ke.tech/2019/08/27/nas-performance-sshfs-nfs-smb.html
Here are some 2019 benchmarks that show NFSv4 to generally be the most-performant.
The really obnoxious thing about NFSv4, IMHO, is that ssh is pretty trivial to set up, and sshfs just requires a working ssh connection and sshfs software installed, whereas if you want secure NFSv4, you need to set up Kerberos. Setting up Kerberos is a pain. It’s great for large organizations, but for “I have three computers that I want to make talk together”, it’s just overkill.
EDIT: I’d also add that I kind of wish that Linux authentication were somewhat more-unified in general in 2024. You’ve got:
SSH keys (ssh, sshfs, mosh, tunneling network traffic over ssh connections).
/etc/shadow passwords (the above with ssh, plus plenty of other services like CUPS).
Wireguard keys
GPG keys (email, git commits)
X.509 certs (email, TLS, smartcard applications)
Kerberos (NFSv4, CIFS at least optionally)
Then you’ve got various keyrings and credential caches, like
ssh-agent,gpg-agent, Gnome has some keyring that can wrap ssh-agent, web browsers have a keyring…I mean, there’s kind of a lot of overlap among all these. Maybe one system would be too far, but I’d kind of like to have things more-unified than they are today.
EDIT2: Apparently inotify() doesn’t let one block the operation that one is monitoring, so probably can’t use it to implement leases.
I’m an idiot. I do have NFS setup on the NAS (I mean, because why not?) but I always forget it’s there, since one client OS (Mac OS) doesn’t support it basically at all, and the other (Windows) does, but it’s not really integrated into the GUI at all, and I’m lazy. I should see what the performance looks like between Windows SMB and NFS implementations are.
As for your key storage, I bloody love my (pair of) Yubikey 5s. I’ve stuffed a giant pile of keys and certs in there and basically don’t think about managing them anymore because, well, it’s just there and just works*.
*Okay the setup was a fuck and a half, but I mean, that does technically qualify as works.