What’s up, what’s down and what are you not sure about?
Let us know what you set up lately, what kind of problems you currently think about or are running into, what new device you added to your homelab or what interesting service or article you found.
I finally finished my first iteration of my Minilab including a very smooth migration from the old server yesterday so I can go to the service side of things again. I plan to get some kind of selfhosters VPN for external access to stuff that’s not exposed to the internet, I’ll have to investigate which one.
Configured changedetection.io to notify me when my usual bus is delayed or canceled.
Latest thing is my server was hard locking up randomly every couple days. Finally thought to check IPMI and it was triggering a correctable ECC error on a specific stick of RAM.
I figured maybe the first couple errors were correctable by the ECC RAM but then they just got worse and caused the lock up.
Pulled the 2 sticks in that pair and so far so good. I’ll survive just fine with the remaining 192GB of RAM lol.
Also switched from my old Dell box with Opnsense to a Linksys MX4300 running OpenWRT, saves me about 20W and its fun to try something different.
Hello! I recently deployed GPUStack, a self-hosted GPU resource manager.
It helps you deploy AI models across clusters of GPUs, regardless of network or device. Got a Mac? It can toss a model on there and route it into an interface. Got a VM on a sever somewhere? Same. How about your home PC, with that beefy gaming GPU? No prob. GPUStack is great at scaling what you have on hand, without having to deploy a bunch of independent instances of ollama, llama.ccp, etc.
I use it to route pre-run LLMs into Open WebUI, another self-hosted interface for AI interactions, via the OpenAI API that both GPUStack and Open WebUI support!
Oh that’s dope. How many hours are you running? Do you also use them for things like encoding or something like that?
This week moved all my vps’s to nixos, so am now able to use one flake for my desktop and all my vps’s which significantly lowers down the time I need to manage my vps’s.
Nowto move my proxmox homelab server ( an old desktop pc I bought recently ) and all my server’s/devices witll be running nixos.
EDIT: An issue I’m thinking about is getting a “proper” server. Not a server like a server rack server, but a mini pc or something along those sides wbich would be a lot stronger and a lot more power efficient than the current 10+ y/o desktop pc I’m using currently.
So would like some reccomendations on that front, like what are some good mini pc brands and mini pc’s that I could have raid seted up on for nas or good budget parts and case to make one myself.
Hey! Another nixos user 😁 What are you using for your VPS? nixos-infect? nixos-everywhere?
As for mini PCs, a friend bought one from Minis Forum and quite likes it. But if you want to support the opensource ecosystem, there are tuxedo computers and slimbook. There’s also starlabs byte.
Take your pick :)
I’m using nixos-anywhere to install and then deploy-rs to deploy updates to nixos vps’s.
Also using agenix for managing secrects for the services so that I can easilly have them all in a public repo, so that other’s could take a look and take inspiration.
My nixos flake url if you wan’t to take a look.
Today I learned that for some reason some DNS servers don’t like SRV records, so had to troubleshoot it when people were unable to log onto my Minecraft server that is on a non-default port.
I’m working on self-hosting my own LLMs.
I realized there are things I wanna talk about and research but I don’t want to send it to open AI. Frankly I feel gross about how much I’ve sent to open AI. My desktop is a beefy gaming rig that I don’t use for gaming much. I have a 20thread core, 64gb ram, an Nvidia gtx 3060 and 5 spare TB so why not.
- I keep a few ollama models downloaded and I’m slowly getting to know them and what they can do. Gemma seems to answer the fastest so I’ve been using that. Deepseek is like the reasoning button on chatgpt.
- I use openai-whisper to transcribe meetings I record using OBS. It’s really slow so I have a cronjob transcribe all my meetings for that day overnight.
- Open Web UI is a fantastic LLM frontend. It provides tools, rags, web searching, and model ranking all as a simple to use UI.
- My desktop has a Wireguard server which makes it easy to use my OpenWebUI on my phone.
Now I want to work on giving the LLM access to my Google calendar so it can create reminders for me. I’m sick of forgetting to think about remembering to do things so I hope if I can just ramble at the LLM about what I’m doing or what’s on my mind it can organize my thoughts. What else are these LLM actually for?
Having electric stability issues this week in Bangkok - several 2-3 hour outages, which are too long for a UPS to cover the gap. I have several mid range but older PCs running docker, virtualbox, etc for various things including a postfix server for the family email, immich, QBittorrent, pihole, paperless, huly, postiz, a Minecraft bedrock server, a flightradar24 ads-b collector, and a variety of other homegrown projects.
Thinking about getting some or most of this over to a service like hetzner, perhaps even splurging on a baremetal dedicated system.
Recently I’ve been reading about/trying to learn qemu and proxmox, but don’t understand them yet. Is that where it’s at for managing a bunch of your own VMs? Or kubernetes/k8s?
I’ve been a little out of the loop for a few years and of course coming back up to speed IT wise judge take weeks. Looking for recommendations on offloading my home stuff to a cloud that I control.
Proxmox runs Qemu under the hood. It’s the current favorite for VM management.
I wouldn’t bother with k8s unless you’re deploying services in high availability, or groups of related containers.
proxmox
You will enjoy Proxmox. When you get it all jammy, check out the Proxmox Helper Scripts: https://community-scripts.github.io/ProxmoxVE/
Hey that’s awesome! thank you for the share. Planning to install proxmox this weekend and give it a try.
Thinking about getting some or most of this over to a service like hetzner, perhaps even splurging on a baremetal dedicated system.
If I may, I find LUXVPS to be quite capable and responsive hosts.
Black Luxury Deal #1
4 vCores (Xeon Gold 6150) 26 GB DDR4 RAM 150 GB Raid 1 NVMe 1 Gbit internet speed | 40 TB Traffic 1x IPv4 1x /64 IPv6 3.2Tbit Premium DDoS Protection 24/7 Ticket Support 4 Backups For ONLY 10€/Mo (recurring)I’ve never used Hetzner, and I don’t know what you are hosting, but I’m sold on LuxVPS. I also use Contabo, and Ethernet Services. The latter would indeed be bare-bare-metal as there are no frills. However, for a test server and for $35 a year, it works.
K8S is a whole different approach and I find it to be a lot more complex, but you would not need virtual machines. If all your applications are running in containers anyways, you could consider it. Finding a good solution for persistent storage is probably the most important design decision.
I’ve used a RV/Marine deep cycle battery attached to a UPS before, that would certainly give you enough for 2-3 hours on most setups.
For some reason Grafana started to sync roles with my IdP (google) and now my own user keeps getting a read only role, so I decided to take this opportunity to finally move away from google and start hosting keycloak instead.
It was a busy week so I could not get the time to finish it yet.
You may also have a migration path by hosting keycloak and add Google as an Identity Provider. Gives you much more flexibility and control this way
Thanks for the tip! I didn’t know that setup was possible.
I’ve installed coraza web app firewall with OWASP ruleset this weekend. I must admit that it wasn’t as easy as I expected it, but it now (mostly) works. I had to give up with nextcloud though.
Not really self-hosted, but I set up obsidian with syncthing and am going to transfer all of my notes from book stack to it and let bookstack be more organized documentation and obsidian to be a big scattering of notes and tags and such. I tried it with bookstack, but the flow was too much of a barrier for me to use it consistantly
I’m currently trying to figure out why my email server got blocked by Proofpoint and they refuse to talk to me. Really about ready to give up on email after self-hosting it for a decade with few problems.
Oh that sucks! One would think that after that long, it’d be somewhat established.
RIGHT?!
Check RBLs a lot of times services just use one of those, and they can be flaky. Usually, you can fill out a form and get reinstated.
I’m not on any of those blacklists, luckily. I guess Proofpoint doesn’t publish theirs. At least iCloud and Gmail both use them. I saw one hint that they may require mail servers to literally have the word “mail” as the subdomain, so I’m working up the courage to mess around with my perfectly working DNS.
There is still the relay through the cloud route (SES, but also at least Scaleway)
Part of me thinks if I have to pay for a relay service, I should just pay for hosted email. But I’ve definitely been considering it!
I need to get a new VPN setup. Been using OpenVPN through OPNsense for years but I’m fed up with the abysmal performance of the OpenVPN client on iOS. Open to suggestions but it has to be fully self hosted.
Wireguard is where it’s at.
Good on iOS too, albeit a bit battery hungry if you route 0.0.0.0/0 and ::/0
I dunno if there’s an iOS equivalent but on my Android phone I use the WG Auto Connect app so it’s only active when not on my home wifi.
The iOS app has this, based on SSIDs
wg-easy is what you want
I have running OPNsense as well and was looking for OPEN VPN on it as well, but I’m not decided yet. I wonder if Android clients are any better.
I’m running OpenVPN on pfsense and am using the android app.
I’ve got a stable 150mbit/s, depending on carrier coverage.
I’m still trying to get a good backup strategy. I am currently using Duplicati but I cannot get the before script execution to work. I will eventually look at Kopia.
What kind of hardware are you using for a mini lab? I want to switch from a raspberry pi 5 to a small form factor Intel based system so I can run Proxmox. I was looking at the Lenovo m920q or an Optiplex 79xx series machine.
Do you have any recommendations for backups or the hardware switch I mentioned?
Have a look at Backrest for Restic. It works great with pre/post scripting and supports healthchecks for monitoring status and stats.
Also is a nice easy to use WebUI which is great for servers.
I’ll look at this again. I had it before and did not stick with it though I don’t remeber why now.
I use cron schedules to run scripts that backup my important stuff to s dedicated backup drive, then copies the backups to a different external drive, then upload the backups to a dedicated backup cloud storage account. Then it deletes any backups older than a month.
I don’t know Duplicati or Kopia, im mostly just using VM snapshots as backups. I store them in an NFS Share on my NAS.
I just posted my Minilab, check my history - I’m also using tiny Lenovos. m920q should be able to do anything you want it!
If you do make a switch to Proxmox, then Proxmox Backup Server is where it’s at for backups. Its de-duplication feature is incredible. I backup all my Proxmox VMs/LXCs with it, as well as my non-Proxmox hosts (laptop, etc.), with
proxmox-backup-client.Personally, I’m using a few of those tiny Beelink PCs (a couple Mini S12 and an EQ12) with the N100 processor, as well as a couple larger rackmount PCs I built for situations where I needed to add an HBA or some other PCI-Ex device. I do recommend something like a Beelink before building, though - they run Proxmox fine, they’re inexpensive, efficient, quiet, and each one can run a handful of VMs.
Yeah, I heard about Proxmox backup and that sounds really nice. Love the idea of being able to take a snapshot before any major changes to a VM and then if it goes south restore from snapshot very quickly.
Sweet!
What’s up is everything I’ve been running and down is what I haven’t.
not working
I haven’t been able to get friendica to connect to Maria DB, so I’ll eventually try just MySql. Grafana isn’t running bc I would need to change a lot of things to get an exporter into each container and the truenas apps don’t really allow that configuration - fine if you have docker compose though, which I’ve started doing more and more.
new
I just got up and running with Stirling pdf, a free (and paid) PDF editor. That looks pretty sweet.
But I’m now also using 15GB of the 32 on the system, which is still plenty for Arc cache for me
what I want
I want to rent a VPS to host various fediverse apps, probably Lemmy, pixelfed, and write freely to start, for the nomad/expect communities. I’ve been looking at netcup and they have some decent arm offerings.
I’d like to put Talos Linux on it so I can get some kubernetes experience. They have a good sized server for €10, so I could expand to add a DB server or one specifically for logging and metrics.
I was looking at Hetzner, but I’ve read that their block storage is super slow and causes timeouts on DB.
Of course, can I even run these apps on arm? I guess I gotta find that out.
One thing I’d like to do is make a web page that makes signups super easy and would create an account on all services, ideally. Not a huge deal of that isn’t reasonable, but it’d be nice to allow doing it once rather than multiple times. If I could get sso, that’d be good, but I don’t know how supported that is.
Up: My unraid server with media library, emby and my kids Down: the fiber internet line into the house that the contractor working on our siding snapped. No one is upset so the system is working.
