Hi selfhosting community! At the moment I have my services set up with nix. They store their data on a btrfs subvolume which I can easily snapshot.
For backup I configured btrbk to send the snapshots to a target pc. However I feel uneasy not knowing if the backups succeeded or not. Right now I would need to check manually on the target machine every so often.
How are you guys handling this problem? Custom scripts for btrfs send/receive? Is there a function in btrbk I overlooked where I can verify the state and get notified of problems?
I was looking into ntfy-sh, which looks optimal, but how can I know if the send/receive was successful?
I’m not familiar with btrbk specifically, but my backups all send a ping to https://healthchecks.io/ when they start and when they finish. healthchecks.io works by getting a simple curl request, similar to ntfy, but if that request doesn’t happen after a certain amount of time (or if you do a start request and a succeed/fail request and the job is taking too long) then it will notify you. It can use ntfy for notifications, but it can also send emails or use any of a number other services.
Thank you for the nod to healthchecks.io, looks great!
I use the “||” operator in bash scripts to send a curl command to ntfy if the backup fails. But the healthchecks option is a little bit more robust since you wouldn’t get a notification if your system crashed and ntfy dies too. I believe you can also do this with Uptime Kuma if you want to go self-hosted on a remote server.
This sounds like an elegant solution, will try it out, thanks!
btrbk ... && curl https://uptime.my.domain/api/push/...is exactly what I do in a systemd service with nightly timer. Uptime Kuma sends a matrix message (via a bot account on matrix.org) if it doesn’t get a success notification in 25h. I have two servers in different locations that do mutual backups and mutual uptime kuma monitoring. Should both servers go down at the same time, there’s also some basic and free healthcheck from my dynamic-ipv6 provider https://ipv64.net/, so I also get an email if any of the two uptime kumas cannot be reached anymore.
ok i’m not saying do this
i recently setup an API proxy, C&C server, Grafana and Prometheus, and Discord bot. now i can send pings via Grafana or with a simple request (provided it’s authed via VPN or proxy) and have my Discord bot use a local LLM on my network to deliver the alert to a Discord channel in the voice of Ultron.
Well the local LLM is definitely a stretch goal of mine hahahah
Hmm, good question. I don’t have a setup for that either, but I don’t think there is a built in function for that.
Your best bet is probably to find some tool that notifies you if new files get created in a directory, as AFAIK new snapshots are only created if they were successfully transferred.
