Yes, I also suspect that at a certain scale it would make a difference, but with a single Postgres database and no advanced clustering or so, the real-life performance metrics of a medium sized Lemmy instance strongly point to Postgres being the bottleneck and not the Python or Rust codebase of the rest of the software.

For regular users there is not a huge difference, but the web-interface is significantly faster and you can subscribe to topics that combine multiple communities. The disadvantage is less mobile app support. Right now only Interstellar supports Piefed.

The really advantage is for moderators and admins, as the Piefed developers actually listened to community feedback and implemented a lot of nice moderation features that require bots or annoying work-arounds on Lemmy.

And there are a few technical differences that make Piefed easier to administrate and troubleshoot from a sysadmin perspective.

I am genuinely confused. What makes you think you were ignored? We do have additional admins, the problem is physical access to the servers to replace hardware that seems to have broken.

We plan to add additional safeguards to such scenarios, but in the end it is a trade-off between reliability and costs. Running a Lemmy instance of our size on a fully managed rented server would easily triple the monthly costs. This would be beyond what I am willing to cover out of my own private pocket, thus making the instance dependent on donations. And that IMHO makes it more likely to be shut down in the future as donations are very variable.

I am not on vacations, but on a work deployment abroad, which further limits the time I can spend on this. Not that it would have made much of a difference, but at least the domain redirection and placeholder text was only possible to do on the weekend.

Sure, but they will not replace hardware and set up remote access for you, which seems like this would have been required in this case.

It is run from a private residence in the DIY punk spririt (and this also allows us to run of a local solar PV system), but more or less the same would happen if you rent rack-space in a “real” data-center. Only if you rent a managed server or VPS someone else will be responsible to fix such issue and this comes at a significantly higher cost at the scale we operate at (slrpnk is part of a bigger project that also hosts other services).

Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.

Thanks.

Yeah at the very least this will result in some changes in the hardware setup to avoid similar situations. But of course it is impossible to forsee all possible problems.

I think now that Piefed has an API for apps, we will see some of them adding support soon. Overall I think the benefits of a Piefed migration outweight the disadvantages, but it remains to be seen if it is doable.

Yeah, the xmpp server is down too. That is something that bugs me quite a bit and I will probably move that one to an external small VPS to retain a more secure backup communication channel.

Lets see. I think a relaunch on Piefed might interest some people to come back, and most slrpnk communities are rather niche and will probably stay. /c/climate might move though.

Slrpnk.net admin here.

The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.

There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time…

An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.

I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I’ll know more when I have physical access again.

Not necessarily, no. We aim to preserve users, communities and posts/comments. Image uploads might get lost though.

Such an in-place migration will need extensive database operations and likely some support by the Piefed developer (to add support for bcrypt hashed passwords), but we are hopeful to make it happen and maybe this will result in a database migration script other Lemmy instances could also use.

If this turns out to be infeasible, we will stay with Lemmy rather than reset everything.

As email in profile is an optional feature on our instance about 50% of the userbase opted to not share one so we would only reach some people. But yes, getting spam filtered would be another problem of mass-mailing.

But I found a way to put up a simple html error page and will add this later to inform our members and link to this thread.

Lemmy has a lot of individual parts that don’t interact very well with each other, especially the image host part. Futhermore the main UI is quite a mess and we were thinking of switching to an alternative already, but this would further increase the “too many moving parts” issue. Piefed on the otherhand has an integrated and very lightweight UI, which also has some nice additional filtering and moderation features Lemmy currently lacks.

And I personally feel more at home with the Python codebase, as it allows better troubleshooting and more standartized (Flask) tooling. The Rust codebase of Lemmy has a lot of obscure custom stuff and the error messages are extremely obstruse from a sysadmin perspective.

And looking at the performance metrics of Lemmy, the main limiting factor seems to be the Postgres database anyways, so the theoretically slower Python codebase of Piefed should not have much impact.

We have a small write up about the hardware on our wiki, but it is also down right now.

I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.

One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.

Already thinking about how to make the best of it 🤷 Maybe we can use this opportunity to try and migrate to Piefed? I had this in the back of my mind for some time already and Rimu seems optimistic that it is possible.

Glad that at least someone sees a positive side of it, because I sure don’t… well at least it gives me a bunch of ideas how to avoid such situations in the future 🤷

Thanks, I am trying, but it really is quite annoying to lose access like that.

Sort of yes. It is a homelab a few people have access to, but currently they are all either on holiday or on extended work related travel. It seemed like an acceptable risk as everything worked smoothly for years.

Yeah I had plans to set up something like that, but always other priorities and in this specific case I could maybe access other internal servers but i would need KVM access to reboot the firewall or some other way to cut physical power. And exfiltrating hundreds of GBs of lemmy database wouldn’t work over such a small pipe either.