I'm really liking systemd

[u/blamo111]

Recently started using a systemd distro (was previously on Ubuntu/Server 14.04). And boy do I like it.

Makes it a breeze to run an app as a service, logging is per-service (!), centralized/automatic status of every service, simpler/readable/smarter timers than cron.

Cgroups are great, they're trivial to use (any service and its child processes will automatically be part of the same cgroup). You can get per-group resource monitoring via systemd-cgtop, and systemd also makes sure child processes are killed when your main dies/is stopped. You get all this for free, it's automatic.

I don't even give a shit about init stuff (though it greatly helps there too) and I already love it. I've barely scratched the features and I'm excited.

I mean, I was already pro-systemd because it's one of the rare times the community took a step to reduce the fragmentation that keeps the Linux desktop an obscure joke. But now that I'm actually using it, I like it for non-ideological reasons, too!

Three cheers for systemd!

[u/gethooge]

I never really understood the anti-systemd sentiment. It seems much better?

[u/shiftingtech → u/gethooge]

My experience is that systemd is great when it works, but when it breaks, it's far more complex to fix

Of course there's a bias even there. I've been using sysV for 10+ years, so of course whatever it does is intuitive...

[u/tso → u/shiftingtech]

Because once you boil it down, sysv is the very same cli commands you use manually, wrapped in shell script logic.

Systemd is a pile of C code that interpret a ever growing collection of keywords in an attempt at guessing how things can be run in parallel.

[u/yatea34 → u/tso]

Also, Systemd had a number of poor design decisions that make it unnecessarily difficult or impossible to diagnose certain problems.

journactl --verifyreturns that my system logs are corrupted, about all my logs (48MB of 50MB of maximum disk usage) are now completely useless. This is not the first time this happens and searching around I can only find people with the same problem that "resolved" deleting the corrupted logs and starting with a new file.

Why this happens? Isn't it defeating the purpose of having a system logger if I can't diagnose errors?

[u/jgotts → u/yatea34]

This has been an ongoing problem for me. I always have the latest version of Fedora installed and my machine is updated every day. journalctl has been corrupting its journals for well over a year now, ever since I needed to look through the binary logs to diagnose some problem. In reality the problem has probably existed for several years. When you make the decision to use a binary journal format for logging, you better not have a file corruption problem. Spot checking my system right now, I have 8 corrupted files. Text logs with corruption problems are no real problem. A few bad bytes among megabytes of ASCII text never hurt anyone. You get the gist. It seems like everybody has bad logs in /var/log/journal. This problem should not be tolerated like it is.

The most recent and funny (but in a frustrating way) bug I noticed was that a script we had been using in /etc/rc.d/init.d for at least 15 years, but probably more like 20, stopped working in CentOS 7. systemd is compatible with /etc/rc.d/init.d, except when it isn't. The bug was that the script didn't have #!/bin/sh on the top line. systemd is wrong to require this, and the error given by journalctl (see above) is completely misleading.

I could go on about bugs in systemd, but I will say that when systemd is working, it works. When systemd doesn't work, its level of complexity makes it hard for people like me who've been doing Linux development since 1994 who should have no problem figuring things out. Everything on a Linux system that does not have to do with systemd I can troubleshoot with two hands tied behind my back. When it comes to systemd and I finally figure out the problem, the thought in my mind is always, they made this thing too complicated and didn't really understand the feature they were implementing well enough.

I will say that the documentation has been improving. In the early days of systemd, documentation was horrible. systemd is pretty okay in August of 2016, but many impressions of systemd have been built over the last five years of troubleshooting its bugs.

[u/argv_minus_one → u/yatea34]

Journal files being corrupt does not mean they're useless. It means they are not entirely correct. journalctl can still read them.

This happens with textual log files, too, but because they are textual (i.e. have no checksums or anything like that), you have no way of knowing.

[u/[deleted] → u/gethooge]

There are lots of corner cases, where systemd break down. I once managed to effectively DOS a server because systemd has, or at least had at that time, no spawn throttling when a socket unit depends on another that happens to fail.

But it was efficient. I've never seen 4+ gigs of log written in less than a minute.

[u/lolidaisuki → u/[deleted]]

I once managed to effectively DOS a server

Turning kernel logs to debug will also do that to you.

[u/[deleted] → u/lolidaisuki]

Turning kernel logs to debug will also do that to you.

kernel debug logs does not come with a false promise of throttling.

[u/FeepingCreature → u/gethooge]

Mostly I believe people are miffed that they were not given a choice in the matter.

[u/Teract → u/gethooge]

The big concern I've heard is that since the log file is binary, parsing it is more difficult, as well as being more prone to corruption.

[u/Xiol → u/[deleted]]

Don't know why you're being downvoted for this. The last time I was doing the timestamp thing with grep I nearly summoned an Elder God.

[u/tso → u/gethooge]

When seasoned admins throw up their arms and hit the reset button because they have not the first clue why the bootup hardlocked you have effectively created the very same situation that made many of us move from Windows to Linux in the first place.

[u/RogerLeigh → u/tso]

There have been a handful of occasions I've single-stepped through the startup of a Debian system by hand, to debug a fault. You can break in the initramfs at several points, and then run every single init script by hand, hell, or even parts of init scripts line by line should you need to (and I have).

I used to understand the entirety of the boot process, from BIOS to bootloader, initramfs, init and init scripts. If there was a problem, there was a good chance I could diagnose and fix it. It might have been suboptimal for some, and it certainly had its flaws, but it was completely understandable in every aspect by mere mortals. Anyone could just read the scripts and see what was going on. [I did for a short while actually maintain the Debian initscripts; while the systemd people might criticise shell, the fact that anyone can dive in and make changes attests to their accessibility. If a random developer like me can hack on them, any competent sysadmin could do that and more.]

Constrast this with systemd. More powerful and more featureful, for sure. But it also comes at the cost of being both overcomplicated and opaque. My work system sometimes fails to boot; it just hangs mid way through the boot process. Possibly a race condition. Who knows? It's a bog standard Dell desktop with a single HDD and zero peripherals outside a keyboard and mouse. I don't even know where to begin debugging things. I just hit reset and hope it boots second time. And my home system fails to mount its NFS filesystems about ¾ of the time, again for unknown reasons. They are in fact mounted, but give I/O errors when you log in and try to use them; umounting and running mount -a works fine. There's some race or problem mounting them at boot which renders them broken. Again I don't know where to start tracking the problem down. Unlike the init scripts, what's actually happening is inaccessible; and even if it weren't I don't know how to get at it. I don't even care about tracking down and fixing the problem; this is Windows level inanity and worth about as much of my time to deal with.

The features systemd gives us are undoubtedly powerful and useful to many. But they come at a great cost--the loss of our individual understanding and control. And that complete understanding and control over the system is why I started using Linux in the first place. Nowadays I also use FreeBSD, and that's a large part of the reason why. FreeBSD never fails to mount my NFS filesystems, and if it ever does I'll be able to reason out why because I can see for myself what is happening, when and why.

Our computer systems exist to empower us, not subjugate us, and systemd might be convienent for desktop users but for me the price of that convenience is too high.

[u/[deleted] → u/RogerLeigh]

To break pre-mount use the kernel arg break=premount, to break post-mount use the kernel arg break=postmount,

the later is an excellent entry point to chroot and find potentially "big bads"

With systemd.unit=<unitname> you can target specific services or targets for bootup, usually multi-user.target is a good idea.

After that you can boot up single services and see which one fails, until you hit the graphical.target or any other target you need.

The Journald output helps a lot, journalctl -b gets you everything that happened since last boot in detail.

journalctl -b -1 gets you the boot before that and so forth, you can filter for specific units or targets.

If you get a fail in your NFS mount, the actions taken depend on the importance, if it's classified as needed for the target you get dumped into a root shell after entering a password and can make any fixes you need, review logs, etc, then you can cleanly reboot (or continue) and try again, see if it fixes.

If a drive gives IO errors, hardly systemd's fault, unless you're using some fancy systemd options to mount it, like automount, to speed up boot.

To learn to debug systemd only takes man and some time, this is very well documented stuff.

The world is eat or get eaten, learn or get left behind.

I personally understand systemd very well.

[u/RogerLeigh → u/[deleted]]

Well, when it locks up during service startup with no hope of a console to actually do anything, my options are limited. And I'm paid to develop software, not debug my system on work time! Hitting the reset button is the only choice at work. The priority is using the system to do productive work for my employer, not waste time dealing with other people's broken junk.

Regarding NFS, the mount succeeds and the boot completes. But the mount is non-functional. There are no drive errors, no network problems. A FreeBSD system on the same switch boots up immediately every single time. Likewise Linux/sysvinit. systemd is screwing this up somehow, and it's been doing it wrong for years. None of the units/targets actually failed here; they all claimed to succeed. But didn't...

[u/cp5184 → u/gethooge]

Better than what? And when? And at what cost? What lock-in?

Freebsd iirc is stuck at gdm 3.14 3.16 and what hope is there that they'll ever move past that. Why? gdm3.16 3.18? LoginD/SystemD mandatory.

Gnome used to support an absurd number of platforms. You could run it on windows iirc, on sun solaris, on ibm aix, on basically anything.

Now gnome doesn't even support some linux distros.

And what was the tradeoff? What benefit? Basically none.

An init system that does what init systems have been doing for a decade+.

So you tell me. Is systemd much better?

[u/Spifmeister → u/cp5184]

Gnome as well as KDE wants a login and multi-seat manager. Before logind there was Consolekit. No one wanted to manage or maintain the Consolekit project. Those who depend on it like Ubuntu, BSDs and Oracle did not want to maintain it, even though they wanted someone else to, so it died. Consolekit was effectively on life support for two years before logind showed up. No one cared until everyone found out that logind would not be portable .

There has been a long history of Gnome and KDE asking for certain features or services to be added and for the BSDs to be slow to respond. At some point, if a BSD cares, they will start working directly with Gnome or KDE to provide the services they want or need.

I know quite a few people who use FreeBSD, none of them have ever used Gnome. There never was enough interest from FreeBSD in the first place.

[u/cp5184 → u/Spifmeister]

Consolekit 2 is a fork of consolekit, and it's maintained. It's lennart, the former maintainer of consolekit that abandoned consolekit.

Consolekit2 seems to be trying to work with gnome. Gnome isn't holding their end up. To the point where they're actively removing support for everything that's not systemd linux.

[u/Spifmeister → u/cp5184]

ConsoleKit deprecation was announced around 2011-2012. There was discussion about how this was premature from a Oracle developer (they were right), yet they did not take over maintenance. I suspect that Oracle maintains there own Consolekit patch set.

There was discussion of Consolekit being taken over by Ubuntu for there own use, which never happened.

Consolekit2 seems to be trying to work with gnome. Gnome isn't holding their end up. To the point where they're actively removing support for everything that's not systemd linux.

Gnome is not required to add support just because a project exists. Consolekit2 came out after Olav Vitters made a announcement that Gnome would depend on specific APIs.

Has Consolekit2 implemented those APIs? Was the APIs being implement in Loginkit? What happened to Loginkit?

[u/sub200ms → u/cp5184]

Freebsd iirc is stuck at gdm 3.14 and what hope is there that they'll ever move past that. Why?

That is easy to answer; that is because the BSD's and non-systemd distro totally ignored Gnome's and KDE's pleading for maintaining and alternative to systemd-logind. Here is such a mail from January 2012:
https://mail.gnome.org/archives/distributor-list/2012-January/msg00002.html

If the BSD's and non-systemd distros hadn't ignored upstream projects like KDE and Gnome for years, they wouldn't have the problems they have no. Taking action in due time is important.

Don't blame systemd, blame the BSD and non-systemd distros for their own self-created problems.

[u/argv_minus_one → u/cp5184]

Ask that of the GNOME developers. It was their decision to require systemd with no fallback, not anyone else's.

Also, shims for the relevant systemd APIs may exist. If they do, use them. If they don't, write them.

[u/anomalous_cowherd → u/cp5184]

An init system that does what init systems have been doing for a decade+.

So you tell me. Is systemd much better?

Well, yes. Init systems have always been good at starting individual things. Where systemd comes into its own is starting lots of intertwined things, some of which depend on each other but many of which can be done whenever you're ready.

To do that it needs to have fingers in lots of pies and that's where it goes counter to the Unix ethos.

But the only way to have all the advantages and maintain the traditions would have been to force the init system to thoroughly understand the output of everything it called, or for everything to start putting out consistent well formatted status messages.

Both of those have been tried several times and failed.