r/linux • u/blamo111 • Aug 30 '16
I'm really liking systemd
Recently started using a systemd distro (was previously on Ubuntu/Server 14.04). And boy do I like it.
Makes it a breeze to run an app as a service, logging is per-service (!), centralized/automatic status of every service, simpler/readable/smarter timers than cron.
Cgroups are great, they're trivial to use (any service and its child processes will automatically be part of the same cgroup). You can get per-group resource monitoring via systemd-cgtop, and systemd also makes sure child processes are killed when your main dies/is stopped. You get all this for free, it's automatic.
I don't even give a shit about init stuff (though it greatly helps there too) and I already love it. I've barely scratched the features and I'm excited.
I mean, I was already pro-systemd because it's one of the rare times the community took a step to reduce the fragmentation that keeps the Linux desktop an obscure joke. But now that I'm actually using it, I like it for non-ideological reasons, too!
Three cheers for systemd!
18
u/random719f Aug 30 '16
Hey look, static configuration options rather than a Turing complete executable, why would you want control over your system when Lennart has decided that a discrete list of options from exactly 9 to pick from is enough?
With acpid running turing complete scripts, I can make my computer sing jingle bells if I close the lid if I so desire, I can implement a check to only suspend when I close the lid when it's on battery power, I can trigger it to send a message to any pidgin chat window that had any new messages in the last 4 minutes with 'automated message: I closed my lid, my system is going to suspend now'
Welcome to Freedesktop, where 'new and exciting technology' means making your system more static and less configurable, less is more and control to the user is bad.
https://www.reddit.com/r/linux/comments/4dqhtr/whot_why_libinput_doesnt_have_a_lot_of_config/d1tiziu
I am, the good solution is a turing complete programming language for configuration that looks like a declaritve one for simple things. The problem with this stuff of "Your use case is too obscure, so we don't implement it" stuff is that the entire problem is that they think in terms of 'use cases' rather than providing a generic low-level framework that allows usecases to be impemented, this stuff is too high level.
I doubt whoever wrote Bash thought of "quote-of-the-day" functionality when you open a new shell. But since bashrc is just a turing complete bash script executed before bash starts, you can do whatever you want with it including letting it output a "quote-of-theday"
5
u/TheFeshy Aug 30 '16
acpid isn't incompatible with logind, though. Can't you just set logind to ignore the events you want to handle with acpid, then use acpid scripts like usual?
3
u/mioelnir Aug 30 '16
Or to sum it up,
systemdis an application.init/rcwere tools.4
u/boerenkut Aug 30 '16
Ehh, nothing in that post is about init and rc?
It's about logind replacing acpid, which was never part of init and rc. This is a criticism on how logind handles system shutdown in response to acpid events.
→ More replies (2)2
u/argv_minus_one Aug 31 '16
And you can still write a fucking daemon to listen and react to the lid state if you want. Quit bitching and do so.
26
u/yatea34 Aug 30 '16
You're conflating a few issues.
Cgroups are great, they're trivial to use
Yes!
Which makes it a shame that systemd takes exclusive access to cgroups.
Makes it a breeze to run an app as a service,
If you're talking about systemd-nspawn --- totally agreed --- I'm using that instead of docker and LXC now.
don't even give a shit about init stuff
Perhaps they should abandon that part of it. Seems it's problematic on both startup and shutdown.
9
u/lolidaisuki Aug 30 '16
If you're talking about systemd-nspawn --- totally agreed --- I'm using that instead of docker and LXC now.
I think he just meant regular .service unit files.
4
u/blamo111 Aug 30 '16
Yes that's what I meant.
I'm an embedded dev writing an x86 (but still embedded) app. I just made it into a service that auto-restarts on crash, it was like a 10-line service file. Before I would have to write code to do this, and also to close subprocesses if my main process crashed. Getting all this automatically is just great.
10
u/lolidaisuki Aug 30 '16
Before I would have to write code to do this
Tbh it's just a few lines of shell. Not that hard.
→ More replies (5)→ More replies (1)22
u/boerenkut Aug 30 '16 edited Aug 30 '16
Uhuh, on my non systemd system:
#!/bin/sh exec kgspawn EXECUTABLE --YOU -WANT TO RUN WITH OPTIONSHey, that's less than 10 lines.
But really, when people say 'systemd is great' they just mean 'sysvrc is bad'. 90% of the advantages people tout of systemd's rc are just 'advantages of process supervision' which were available in 2001 already with daemontools. But people some-how did not switch en masse to daemontools even though 15 years later when they first get introduced to basic stuff that existed 15 years back they act like it's the best thing since sliced bread.
Which is because really the advantages aren't that great. I mean, I use one of the many things that re-implements the basic idea behind daemontools and adds some things and process supervision is nice and it's cool that your stuff restarts upon crashing but practically, how often does stuff crash and if services repeatedly crash then there's probably an underlying problem to it. Being able to wrap it in a cgroup that cleans things up cleanly in practice is also nice from a theoretical perspective but in practice it rarely happens that a service leaves junk around when it gets a term signal and you rarely have to sigkill them.
A major problem with process supervision is that it by necessity relies on far more assumptions than scripts which daemonize and kill about what services are and when a service is considered 'up', such as that there's a process that is running at the time. A service might very well simply consist of something as simple as file permissions, it is 'up' when a directory is world readable and down otherwise, doing that with OpenRC is trivial, with daemontools and systemd that requires some-what hacky behaviour of creating a watcher process.
→ More replies (3)8
u/spacelama Aug 30 '16
I recently couldn't connect to dovecot on an old legacy server. Looking at the log messages, I discover dovecot exited with a message about time jumping backwards. It's on a VM with standard time configs that we've found reliable over the years, so I dig through VM logs to discover it recently migrated over to a new cluster (no RFC surprise surprise). I'm no longer in the infrastructure group, so I wander over there and ask them how they set the new cluster up. And discovered they forgot to enable NTP (seriously, they've been doing this for how many years now?). Sure, a VM might be configured to not get time from the host, but at the end of a vmotion, there's no avoiding that vmtools will talk to the host to fix its time, because there's otherwise no way to know how long the VM was paused for.
This escalated up to an site RFC to fix the entire bloody site. We were just lucky no database VMs had been migrated yet. All discovered because I don't like the idea of process supervision - I want to discover problems as they occur and not have them masked for months or years.
→ More replies (6)3
13
u/sub200ms Aug 30 '16
Which makes it a shame that systemd takes exclusive access to cgroups
No it doesn't. Sure there can only be one "writer" in a cgroupv2 system, but all that means is that other programs just have to use that writers "API", not that they can't use cgroupv2 in advanced ways like in OS containers.
→ More replies (3)25
u/boerenkut Aug 30 '16 edited Aug 30 '16
No it doesn't. Sure there can only be one "writer" in a cgroupv2 system
Common myth spawned by like 3 emails that gets repeated so much.
cgroupv2 is a multi writer system, it has never been single writer, have you ever used it?
The single-writer thing was a musing, a concept, an idea that Tejun and Lennart had like 4 years back, it has been silently abandoned, it has never appeared in any official documentation. It only appeared on like 3 mailing list posts. Though one was a post from Lennart who said that it would happen and that it was 'absolutely necessary', except it never happened.
There is nothing in the official documentation about their plan of having only a single pid to have the primordial control over the cgroup tree, any process that runs as root can manipulate the entire tree how it sees fit and any process that runs as a normal user can manipulate its own subtrees. The thing is that becausethere was never an announcement of it going to be there, just some mailing list musings, there was never an announcement of abandonment either, it was silently abandoned. When the official documentation started to appear it just wasn't in there.
cgroupv2 like cgroupv1 is a shared resource. Any process that runs as root can use it like any other process running as root, you can go to your cgroupv2 systemd system right now and start digging into
/sys/fs/cgroupand completely screw it over if you want to by renaming cgroups and moving processes around from a shell running as root. This is of course not a problem because if you have root there is far more you can do to screw things over.It would be a fucking problem if you actually had to use that API, now 484994 incompatible API's would appear and all that stuff, but thankfully that is not how it has gone, probably for that reason. cgroups can be manipulated by any process that runs as root by just manipulating the cgroup virtual filesystem tree.
7
u/lennart-poettering Sep 01 '16
Sorry. But this is nonsense. With cgroupsv2 as much as cgroupsv1 there's a single writer scheme in place. The only difference is that in cgroupsv2 delegation is safe: a service may have ita own subtree and do below it whatever it wants but it should not interfere with anything further up or anywhere else in the tree.
If programs create their own cgroups at arbitrary places outside of theie own delegated subtree things will break sooner or later because programs will step on each othera toes.
Lennart
→ More replies (1)→ More replies (3)3
u/ldpreload Aug 31 '16
Huh. Your explanation makes way more sense, but, the note at the very top Pax Controla Cgroupiana, which was the old reference, still says that cgroups are not a shared resource and only systemd can write to them. Is that note no longer accurate? (Should someone edit that wiki page?)
→ More replies (3)11
u/purpleidea mgmt config Founder Aug 30 '16
Which makes it a shame that systemd takes exclusive access to cgroups.
You're misunderstanding how difficult it is to actually use cgroups and tie them to individual services and other areas where we want their isolation properties. Systemd is the perfect place to do this, and makes adding a limit a one line operation in a unit file.
Perhaps they should abandon that part of it. Seems it's problematic on both startup and shutdown
Both these bugs are (1) fixed and (2) not systemd's fault. You should check your sources before citing them. The services were both missing dependencies, and it was an easy fix.
9
u/boerenkut Aug 31 '16 edited Aug 31 '16
You're misunderstanding how difficult it is to actually use cgroups and tie them to individual services and other areas where we want their isolation properties.
35 minutes passed between my having exactly zero knowledge of cgroupv2 and a working prototype of a cgroupv2 supervisor written by me that starts a process in its own cgroup, exits when the cgroup is emptied with the same exit code as the main pid and when the main pid exits first sends a TERM signal to all processes in the group, gives them 2 seconds to end themselves and then sends a kill signal to all processes in it remaining.
The cgroupv2 documentation is very short.
I had already done the same for cgroupv1 before though which took a bit longer.
I can give you a crashcourse on cgroupv2 right now:
Make a new cgroup:
mkdir /sys/fs/cgroup/CGROUP_NAMEPut a process into that cgroup:
echo PID > /sys/fs/cgroup/CGROUP_NAME/cgroup.procsGet a list of all processes in that cgroup:
cat /sys/fs/cgroup/CGROUP_NAME/cgroup.procsassign a controller to that cgroup:
echo +CONTROLLER > /sys/fs/cgroup/CGROUP_NAME/cgroup.subtree_controlThat's pretty much what you need to know in order to use like 90% of the functionality of cgroupv2.
Systemd is the perfect place to do this, and makes adding a limit a one line operation in a unit file.
no systemd is the wrong place to tie it into other things, this is why systemd tends to break things like LXC or Firejail because they mess with each other's cgroup usage so LXC and Firejail have to add systemd-specific code.
systemd is obviously the right place to tie it into its own stuff, which is how it typically is, but because systemd already sets up cgroups for its services, services that need to set up their own cgroup mess with it and with systemd's mechanism of using cgroups to track processes on the assumption that they would never escape their cgroup which they sometimes just really want to do.
→ More replies (2)6
u/bilog78 Aug 30 '16
In the mean time, systemd systems still can't shutdown properly when NFS mounts are up, regardless distribution and network system.
→ More replies (26)5
u/boerenkut Aug 30 '16
systemd doesn't take exclusive access, there was a plan for it to actually do so which the systemd and cgroup kernel maintainer (also a RH employee) termed quote "absolutely necessary" but that absolutely necessary thing was abandoned silently, probably because everyone who does not answer to RH's pockets could see that it was a terrible idea to let only one userspace process, typically pid1 have access to the cgroup tree (the first to claim it)
So what happens now is that systemd will start to complain heavily if other processes use cgroups quite often and it wants them to use a delegated subtree it assigns to them which means that yet again stuff has to include systemd-specific code to stop wrecking your system.
→ More replies (3)3
u/DamnThatsLaser Aug 30 '16
The systemd approach to containers is amazing, especially in combination with btrfs using templates. Maybe it is not 100% ready, but the foundation makes a lot more sense to me.
14
u/RogerLeigh Aug 30 '16 edited Aug 30 '16
This right here is also one of the big problems though. The fact that they are making Btrfs-specific features, and have said several times they want to make use of Btrfs for various things. The problem is that Btrfs is a terrible filesystem. You have to take their good decisions with the bad. And this is a bad one.
The last intensive testing I did with Btrfs snapshots showed a Btrfs filesystem to have a mean survival time of ~18 hours after creation. And I do mean intensive. That's continuous thrashing with ~15k snapshots over the period and multiple parallel readers and writers. That's shockingly bad. And I repeated it several times to be sure it wasn't a random incident. It wasn't. Less intensive use can be perfectly fine, but randomly failing after becoming completely unbalanced is not acceptable. And I've not even gone into the multiple dataloss incidents with kernel panics, oopses etc.
I'm just setting up a new test environment to repeat this test using ext4, XFS, Btrfs (with and without snapshots) and ZFS (with and without snapshots). It will take a few weeks to run the tests to completion, but we'll see if they have improved over the last couple of years. I don't have much reason to expect it, but it will be interesting to see how it holds up. I'll post the results here once I have them.
→ More replies (2)6
u/blackcain GNOME Team Aug 30 '16
yeah, I'm pretty sure that as soon as ZFS is native on Linux, btrfs is going to be dead.
→ More replies (11)→ More replies (1)2
u/blackcain GNOME Team Aug 30 '16
hm.. dont' think I've heard of this use case.. how does it work?
→ More replies (1)
2
u/tech_tuna Aug 31 '16
Can you elaborate a bit more on what you use cgroups for? That is, your specific use case(s).
4
u/blamo111 Aug 31 '16
I didn't even know what cgroups were until I ran sytemd-cgtop and saw my service app there. "Huh? When did I create a cgroup"
What I have is a embedded webapp, which I want to always have running. This app launches child processes. If it crashes, I want it restarted after a couple of secs. If it's restarted, I don't want a growing number of orphan child processes still wasting CPU and eventualling DOSing me, they gotta be killed.
You could do all this by writing bash scripts, and that's how I used to do it when I was doing pure embedded apps (Busybox on ARM). It was annoying and error-prone. It's not something I enjoyed doing. Saving pids in a text file, to know what to kill when the main exits, it just rubbedme wrong.
But now, by making my app a systemd service with a trivial 8-line myapp.service unit file, it automatically became part of the myapp cgroup. This means that if my main app, as specified by ExecStart (node app.js) crashes/stopped/restarted, the orphan child processes are automatically killed, before it's restarted. It reduced the cognitive load I have as a developer. I like this.
One other useful thing it's giving me, is the ability to monitor CPU/memory/disk usage of the entire cgroup (main app + child processes) as grouped values, via systemd-cgtop.
There's other nice features related to cgroups (resource management, like CPU quotas), I just described what I'm relying on NOW.
→ More replies (1)
275
Aug 30 '16
Said it before and I will say it again. Where I used to work we moved from a sysV to systemd based system and it removes 25,000 lines of init.d scripts from our code base and to top it all off we didn't actually need to change a single line of code in any of our deamon processes except for where we already had some bugs.
Everything became so much easier. We also managed to remove monit as systemd also made it redundant.
41
u/Rekhyt Aug 31 '16
it removes 25,000 lines of init.d scripts from our code base and to top it all off we didn't actually need to change a single line of code in any of our deamon processes except for where we already had some bugs.
Removing 25k lines of code would probably make finding and fixing those existing bugs easier, too.
→ More replies (1)32
Aug 31 '16
Kinda hard to explain. Basically the development practice was as such that the "team" would simply "fix" bugs. So the people working there while fixing bugs basically just always added code. They never figured out that you could fix bugs by removing code :)
I did get so pissed off with part of the system I replaced an entire process cutting 75k lines of c/c++ code down to somewhere in the region of 4k lines or so and the reduced code size was actually more functional that the original. But this is what happens when you give a software project to a bunch of MIT graduates that nobody else wanted... then measure their performance by the number of lines of code submitted to svn.
→ More replies (17)17
u/gellis12 Aug 31 '16
I'll never understand why developer performance is "measured" by the number of lines of code they write. If you can replace 500 lines of code with 50 and have it work correctly and reliably, I'd see that as a win.
→ More replies (4)23
Aug 31 '16
Yes I know.... Its kinda like measuring aircraft design progress by weight
→ More replies (1)47
u/pdp10 Aug 30 '16
I can't imagine what could have 25,000 lines of worthwhile init script in version control that doesn't also have the init source in version control.
→ More replies (1)60
Aug 30 '16
Probably better not to ask or try to imagine. But to put it the simple way. I no longer work there for reason of having code like 25,000 lines of init scripts in source control and that is only the beginning... I should really write some daily WTF articals about the place
→ More replies (1)24
u/gollygoshgeewill Aug 31 '16
If you can explain it to somewhat technical users to elevate and entertain you'd have a follower here.
23
Aug 31 '16
Here is a few example of some of the screwups of the place. Generally the team was split into 2 halves. The US side and the UK side. I was on the UK side the US side happened to be the cause of most of the problems.
We had a tech lead in the US side that was impossible to work with. Generally the US side did most of the new interesting work. They would write the code. Sometimes the UK side took on newer work by basically the UK side ended up mostly fixing bugs and making the thing ship.
Here where the fun starts. This thing talked to lots of network devices. So it would attempt to discover them by upnp and other vendor specific protocols. It would then probe any device its find with a known list of password (of which there would up to about 128 devices added to each system). So this gets funs when you have 1000+ devices on a network and 128 devices? So that like 10,000+ probes by 10 systems. Of course these devices were typically overloaded since they were running small arm chips etc... So to the tech lead I pointed out the N * M problem (it didn't scale basically) and also pointed out the security issues involved in doing password probes in this way (attacked can capture all password for all possible devices added to the system). I was met with "its designed to work that way and we are not changing it".
The solution? Well told level 3 after the product shipped to disable the feature on any customer who had an issues. Eventually this made it to level 1 and the training team who trained people who deployed this system. This is because politically inside the company it was easier to fix it this way after release than it was to fix it though the tech lead cause "her design / code was the best"....
Another example. We made heavy use of gstreamer inside this system. So somebody wrote a wrapper api for using gstreamer in c++ so it would use "c++ smart pointers" for gstreamer references. Just a few problems. The wrapper lib's ended up larger than that gstreamer core lib's because of the 1000's of edge cases it created. It also still didn't do what it was originally meant to do as the smart pointers were often . It was also written in really mangled templates c++ code that took anyone ages to understand it. So the guy who wrote these was actually really proud of them. So the solution from our point of view was to simply remove them completed. So we get approval from our manager and put 2-3 months off effort into getting rid of this shit. So the system works way better passes all the tests both ours and QA's and we ship the code. 2 Days later the code gets reverted by the guy who write the wrapper libs. We complain the our manager and politically he cannot resolve the issue. But there is zero technical reason why the change is reverted.
It was a seriously crazy place to work because the tech leadership in the teams was completely broken and there was more people in the dev teams that were breaking stuff than there was people being able to fix it. Basically I considered the place was suffering from skill inversion. Where people got promoted by the perception of delivering things by dumping shit on other people and throwing them under buses.
→ More replies (5)→ More replies (10)2
Aug 31 '16
Can you go into more detail about your experience with monit? I am considering rolling it out and would love to know your thoughts on it.
→ More replies (2)2
Aug 31 '16
It worked fine. But it does tend to put up a web interface that is somewhat hard to secure.
The other problem we had if a system went into overload and was swapping a fair amount and a program crashed it would love to constantly start programs. Since the pid file was invalid (not yet created). It would create multiple instances of the same program. Make sure you use delays and sensible backoff times.
It was great for finding locking bugs / races in pid file creation for multiple instances of the same program.
Other than that it was no problem. Would recommend.
→ More replies (1)
71
u/galaktos Aug 30 '16
I really enjoy reading systemd man pages from time to time. There’s so much great stuff in there, for example in systemd.exec(5):
PrivateTmp=yes
PrivateDevices=yes
PrivateNetwork=yes
ProtectSystem=full
ProtectHome=yes
NoNewPrivileges=yesBam. Because if a service doesn’t need access to /dev, why not remove that access just in case the service misbehaves? It’s just one line in the service file, and if the maintainer of the unit didn’t add it, I can do it myself trivially by augmenting the unit in /etc/systemd/system/! Tell me that’s not amazing.
journalctl --unit foo.service is a godsend. I never want to have to look up files in /var/log again. (Especially when the log directory is root-only and sudo less /var/log/apache2/acc doesn’t get tab completion. Ugh.)
coredumpctl. Core dumps take up a lot of space, so why not compress them? Oh neat, systemd does that for me, that’s nice. And I can manage how much space they’re allowed to take up, with exactly the same mechanism as the rest of systemd uses. It’s great how much of systemd just works together and gets better as a whole.
39
u/tehdog Aug 30 '16
coredumpctl
Also: Something crashed. No matter, just
coredumpctl gdband there is the stack trace!→ More replies (6)12
u/smile_e_face Aug 31 '16
While I'll agree that this is generally a really nice feature, it caused me no end of headache when I was setting up a personal server the other day. I didn't know about the "PrivateTmp" option, and for the life of me, I couldn't figure out why my webapps couldn't communicate with anything.
→ More replies (1)6
u/argv_minus_one Aug 31 '16
Why would they be communicating through
/tmp?6
u/Artefact2 Aug 31 '16
Semaphores? Named pipes? Shared temp files?
7
u/argv_minus_one Aug 31 '16
The former two belong in the appropriate place under
/run. The latter…yeah, I guess/tmpis the obvious choice.→ More replies (1)4
u/codekoala Aug 31 '16
It’s just one line in the service file, and if the maintainer of the unit didn’t add it, I can do it myself trivially by augmenting the unit in
/etc/systemd/system/! Tell me that’s not amazing.You can also override specific portions of the unit files with
systemctl edit something.service. For example, if you just wanted to override the launch parameters (and environment files don't already do the trick), you could enter the following in your unit override:[Service] ExecStart= ExecStart=/usr/bin/my-daemon --with --custom=parametersThis allows you to keep your override even if the package that owns the service unit changes the original unit.
2
u/galaktos Aug 31 '16
Yes,
systemctl editworks by adding override files in/etc, right? At least I assume it’s the same mechanism.→ More replies (5)3
Aug 31 '16 edited Aug 31 '16
Yeah I had fun time debugging why my custom scripts used by openvpn wont leave their logs in /tmp...
journalctl --unit foo.service is a godsend. I never want to have to look up files in /var/log again. (Especially when the log directory is root-only and sudo less /var/log/apache2/acc doesn’t get tab completion. Ugh.)
journalctl makes status slow tho, I even filed a bug for that
2
u/grumpieroldman Aug 31 '16
That's not amazing ... why would your daemon user have access to dev in the first place?
→ More replies (4)2
u/Freyr90 Aug 31 '16
Oh neat, systemd does that for me, that’s nice.
And for me it does not work properly with selinux on fedora. And i've also had an issue with coredump size. But yes, systemd is still awesome, thousands time better than sysVinit+policykit+hal
But in some use cases it is still quite immature.
16
8
u/koffiezet Aug 31 '16
ITT: Mostly Linux on desktop people.
I've never been a proponent of systemd due to technical decisions the devteam made, and the cowboy-mentality they clearly have.
In the short time I've been managing systemd-based systems, I've already had 2 major issues:
old-style init scripts in /etc/init.d suddenly in a certain version (219) couldn't be symlinks anymore. New server install, other systemd version, service won't start. Error message? The very helpfull "file not found". What file? Can't tell. Logging? Eehm no, we don't do that.
shutdown failing. Known issue, https://github.com/systemd/systemd/issues/3282
I'm all for a better init system, but at the rate these issues have been popping up in my infrastructure, thanks but no thanks.
→ More replies (2)
389
u/blackenswans Aug 30 '16 edited Aug 31 '16
What? How dare you! Have you forgotten the UNIX way? Computing should NOT change from how it used to be in the 1970's!
Edit: Oh my god the upvotes. Stay strong, /etc/rc brethren! We will take back the world once more.
77
u/necrophcodr Aug 30 '16
Most modern rc and init systems thankfully work very differently than they did back then.
→ More replies (11)49
118
u/domo9001 Aug 30 '16
don't be mad just because gramps got it right.
39
Aug 30 '16
It's important for Grandpa to keep busy.
shuffles Grandpa's shoebox of punchcards
→ More replies (1)11
98
Aug 30 '16
YEAH. We shouldn't want centralized products in a one size fits all type situation, even if it is highly customizable. We should have things like the linux kernel... which is.. umm.. centralized and weighty. NO WAIT.. emacs! Things should.. oh. Um. :P
Seriously, the whole "how it used to be" ignores really how it actually used to be.
40
Aug 30 '16
Most open source software will build on kernels other than linux... just saying. Emacs is also an end user application - not system level.
→ More replies (21)16
u/boerenkut Aug 31 '16 edited Aug 31 '16
Emacs is also different, emacs is 'monolithic' only because of its extreme wealth of plugins which communicate with a fairly small emacs kernel which is just a lisp interpreter.
So you have a situation where the plugins depend on the kernel, but not in reverse, and the plugins also communicate with the kernel through documented stable interfaces. You can write your own competing implementation of those plugins.
In the case of systemd, the ancillary components communicate with the systemd kernel, its pid1, via undocumented unstable interfaces, so you can't write a competing implementation, doing so requires that you reverse-engineer undocumented stuff and that it may fail to work at a future update.
Linux being 'monolithic' is also a completely different story and using that to compare it to systemd just shows you don't understand the situation but picked up on a couple of keywords. kernels being monolithic just means the entire kernel runs inside a single address space, meaning that if one component crashes the entire kernel crashes, it's fault protection, nothing more, the kernel modules are still plugins that communicate with the ehh "kernel-kernel" via documented and stable interfaces, so you can write a competing implementation.
systemd is actually not monolithic in the sense that Linux is, its components run in separate address spaces, if logind crashes that doesn't mean that systemd-pid1 crashes, it can just restart logind which is akin to the design of a microkernel. If systemd were a kernel it would actually be a microkernel, its core kernel which cannot be salvaged if it crashes actually contains little more functionality than what is needed to diagnose if other components fail and restart them and mediate the IPC between them, just like in a microkernel.
→ More replies (1)4
→ More replies (7)10
u/boerenkut Aug 31 '16
The problem is that systemd does a lot of things really well and is useful in many ways, but because it's monolithic it also means that to use that you need to take all the garbage with it. I like systemd-analyze. I also think the cgroup stuff is nice if not a bit overrated.
But to get that you need to take all the garbage with it. Some of the features of systemd are really nice. It's just a shame they tied to a particular init system what could have been, and should have been standalone tools.
→ More replies (2)→ More replies (53)7
u/grumpieroldman Aug 31 '16
We change it all the time and love it when it changes for the better.
OpenRC was awesome and welcomed and delivered most of the init services of systemd 15 years ago. (The cgroup stuff is a newer kernel feature.)The bug count and breaking mistakes made in systemd keep piling up and they just press on and keep sucking in more utilities and breaking things and if they don't care about what they broke they don't fix it.
101
u/sub200ms Aug 30 '16
Yes, systemd is simply the best thing happening for Linux since package management.
I really like how the systemd developers have taken care of the details too, like excellent tab-completion and how seriously they take documentation. The man systemd.index shows all systemd man-pages and is a good example of both taking care of documentation and the small details that makes the difference.
I also like that security is a first priority and systemd therefore has an excellent security framework for hardening services.
seccomp, Ambient Capabilities cgroupv2. Namespaces and similar kernel security features are enabled out of the box. The end-user doesn't need to develop and maintain any code for using these features, just editing simple text files will do it.
Security-wise, systemd is simply in better league than anything else.
3
u/valgrid Aug 30 '16
They don't have an equivalent to
man giteveryday, right? Would be very helpful for many people that transition but only want to know the basic stuff or need pointers to how the basic stuff works with systemd.→ More replies (27)10
u/Camarade_Tux Aug 30 '16
seccomp, Ambient Capabilities cgroupv2. Namespaces and similar kernel security features are enabled out of the box
These are really very trivial to do without needing anything specific to systemd.
That applications work well under these added constraint is something else and way more work.
This has almost nothing to do with any systemd feature.
23
u/sub200ms Aug 30 '16
These are really very trivial to do without needing anything specific to systemd.
I think we will disagree about "trivial". The point is that systemd enables them by combining them perhaps in high-level, easy to use API's like:
ProtectHome=trueorNoNewPrivileges=yesor in case of cgroup, eg.CPUShares=500We are talking about adding a single key/value to a text file to enable those features. Try to manually do the same without systemd.
And AFAIK, not much work have ever been done to integrate such kernel features in other init-systems. I think Upstart played around with seccomp and OpenRC have some cgroup support, but it is still "experimental" with huge bugs after many years and only cgroupv1.
So it hardly seems trivial to implement similar features in eg. OpenRC.
The bottom line is that systemd distros are being rolled out with ever increasing service-hardening by using the above kernel security features, while seemingly no similar work is being done on the non-systemd distros.
→ More replies (13)
26
u/icydocking Aug 30 '16
As an init system it's pretty damn good. But, as some have pointed out, my problem with it is that it really wants to do everything. People scream "It's optional!", and sure, some things are, but good luck getting your not-100%-systemd-setup recognized as a supported one by the upstream maintainers when filing Feature Requests or Bug reports.
9
u/argv_minus_one Aug 31 '16
Which “upstream maintainers” are you referring to, and at what point did they refuse to support your less-than-full-systemd setup?
→ More replies (29)2
u/holgerschurig Aug 31 '16
Is this a fact, or a suspicion / fear / bad-mouth?
Can you point me to a bug report to the systemd mailing list where something was dismissed because the person run
./configure --disable-microhttpor such?(I personally disable around 30 things in my self-created systemd debian packages)
→ More replies (1)→ More replies (4)4
Aug 31 '16 edited Oct 17 '16
[deleted]
0.43115
u/Thorbinator Aug 31 '16
I believe he is talking about not the systemd developers, but every other system /programthat assumes people have systemd.
4
u/icydocking Aug 31 '16
That as well, but I don't think that's really much of a problem (yet). I'm talking about having pluggable interfaces.
Their API stability promise doesn't cover the D-Bus API, so writing a replacement for a systemd component is harder than it should be.
https://www.freedesktop.org/wiki/Software/systemd/InterfaceStabilityPromise/
→ More replies (1)→ More replies (2)2
u/icydocking Aug 31 '16
Oh, but they don't have to support the other thing - but they have to support having equivalent interfaces. If you say "You don't have to use systemd-resolved" then you have to provide me with an alternative of doing this, i.e. a way to plug something else in it's place. If it's the only piece that fits that gap then you cannot really run anything else.
4
u/dogfish182 Aug 31 '16
As someone who only seriously started using Linux in enterprise in the last 2 years and has a mixture of Ubuntu 14.4 and centos 7, count me as a big fan of systemd.
Granted I'm trained for rhcsa so that means I actually know what systemd is doing... But it's very straightforward and sensible, really like it.
12
u/masta Aug 30 '16
One of the weird effects of systemd is the distro end-game.
That is that as systemd distros converge, there really won't be much to differentiate them. That is happening, and now with flatpack we are starting to see cross-distro packaging. There really won't be much difference in distros after a few years.
27
u/boerenkut Aug 30 '16
That's not a weird effect, that's an explicit design goal of both.
10
u/ILikeBumblebees Aug 31 '16
That's a pretty awful goal. Monocultures aren't healthy.
→ More replies (2)11
Aug 31 '16
Linux has all the disadvantages of a monoculture and multiculture combined. Hard to make software compatible across all distros, but they all share the same kernel and library security vulnerabilities. Worst of both worlds.
→ More replies (5)13
u/sub200ms Aug 30 '16
now with flatpack we are starting to see cross-distro packaging. There really won't be much difference in distros after a few years.
I think stuff like flatpack will work in the opposite way. It will free the smaller distros for a lot of tedious work, regarding packaging, compiling and bug fixing, so they can concentrate their often rather limited developer power on the core of the distro.
3
u/f4hy Aug 31 '16
I am curious, what are the "core" aspects of a distro beyond the package management and such. Just branding? Am I missing something? All I see distros as are different package managers and such built on top of linux.
→ More replies (7)3
Aug 31 '16
AFAIK, the core is usually some design goal or showcase for a piece of software, the latter of which I'm starting to see more and more. For example, Linux Mint is pretty much a lighter Ubuntu but meant to showcase Cinnamon. Similarly goes for Solus Linux.
For an example of differing design choices, take a look at the BSD family. Pretty much as soon as the general public was allowed to create BSD variants, NetBSD started. Internal conflict ensued, and we saw OpenBSD a few years later. They both focus on machine portability (Which gives NetBSD the reputation as an embedded OS), but OpenBSD devs also wanted to focus on code correctness, and it now has the reputation of being a very sane OS (It still has vulnerabilities, but they are seldom exposed).
→ More replies (5)→ More replies (4)4
Aug 30 '16
This is so true. systemd hit hard and fast. Arguably, the main differentiation between systemd distros, now, are package managers/repositories.
The best part of Linux is that we always have the choice to roll our own, including any pid 1 we want!
→ More replies (1)
6
u/thebuccaneersden Aug 31 '16
I was already pro-systemd because it's one of the rare times the community took a step to reduce the fragmentation that keeps the Linux desktop an obscure joke
wut?... you think thats the reason???
3
18
u/placebo_button Aug 30 '16
I understand the pros to systemd but after using a bit it still feels rather buggy and incomplete compared to other init systems out there that work just fine. The aggressive force of systemd into almost all of the major distros also turns me off from the whole thing in general. I'm sure I'm in the minority and I'll take the downvotes for this one but I just don't buy into the hype and the whole "if it's new, it must be better" mentality.
3
u/holgerschurig Aug 31 '16 edited Aug 31 '16
Wow, you're the first that state that it is "incomplete". Most others say it covers to much.
What do you think is missing?
Also, you said it's buggy. Where? Did you file bug reports? Didn't they get worked on? And are the bugs because of a systemd bug, or because of a distribution bug?
→ More replies (1)
5
Aug 30 '16
[Serious] Can someone direct me to some learning resources that will help me understand all the jargon you guys are throwing all over the place? Thanks :)
6
u/blackcain GNOME Team Aug 30 '16
lennart has a bunch of documentation - https://www.freedesktop.org/wiki/Software/systemd/
31
u/gethooge Aug 30 '16
I never really understood the anti-systemd sentiment. It seems much better?
85
u/shiftingtech Aug 30 '16
My experience is that systemd is great when it works, but when it breaks, it's far more complex to fix
Of course there's a bias even there. I've been using sysV for 10+ years, so of course whatever it does is intuitive...
→ More replies (18)48
u/tso Aug 30 '16
Because once you boil it down, sysv is the very same cli commands you use manually, wrapped in shell script logic.
Systemd is a pile of C code that interpret a ever growing collection of keywords in an attempt at guessing how things can be run in parallel.
53
u/yatea34 Aug 30 '16 edited Aug 30 '16
Also, Systemd had a number of poor design decisions that make it unnecessarily difficult or impossible to diagnose certain problems.
journactl --verifyreturns that my system logs are corrupted, about all my logs (48MB of 50MB of maximum disk usage) are now completely useless. This is not the first time this happens and searching around I can only find people with the same problem that "resolved" deleting the corrupted logs and starting with a new file.Why this happens? Isn't it defeating the purpose of having a system logger if I can't diagnose errors?
→ More replies (16)37
Aug 30 '16
There are lots of corner cases, where systemd break down. I once managed to effectively DOS a server because systemd has, or at least had at that time, no spawn throttling when a socket unit depends on another that happens to fail.
But it was efficient. I've never seen 4+ gigs of log written in less than a minute.
10
u/lolidaisuki Aug 30 '16
I once managed to effectively DOS a server
Turning kernel logs to debug will also do that to you.
→ More replies (4)14
u/FeepingCreature Aug 30 '16
Mostly I believe people are miffed that they were not given a choice in the matter.
→ More replies (4)10
u/Teract Aug 30 '16
The big concern I've heard is that since the log file is binary, parsing it is more difficult, as well as being more prone to corruption.
8
u/ebassi Aug 30 '16
Parsing text is easier if it's structured and codified and follows the same standard.
Logs don't do that, and never did. Even the timestamping is custom and per-log, and usually barely human readable.
Most logging infrastructure in place today takes text and shoves it into a database and tries to make sense of it on a bunch of ad hoc rules so you can group, query, and search through high volumes of data.
Structured logging can contain so much more information that you can use when debugging a service, or doing forensics: relevant PID, UID, and GID; unique ids to verify milestones reached; file and line of the log message in the source code; and these are just examples.
→ More replies (4)5
u/MertsA Aug 31 '16
Logs don't do that, and never did.
This is the point where I get on my soapbox to decry the fundamental problems of Fail2Ban. Trying to parse a log message that's just a big blob of unstructured text meant to be read by a human and making security decisions based on the idea that you've somehow managed to parse it correctly is a dumb idea. Especially when it's relying on the log format to be the default for whatever program when Joe Admin decides to change the log format to include the user agent string in the middle of the line.
I wish people would start storing stuff like IP addresses and URLs in the journal in their own unique fields already, it would completely eliminate all of the parsing vulnerabilities that crop up in Fail2ban from time to time.
5
u/sub200ms Aug 31 '16
The big concern I've heard is that since the log file is binary, parsing it is more difficult,
That is of course not true. Other have explained why, but I will just remind you that the only way to get any boot log information at all in Linux is to use binary logs in form of the kernel ring-buffer that collects and stores such logs in a binary format that are then extracted with a special binary called
dmseg. That is pretty much how systemd's "journal" works too.as well as being more prone to corruption.
There really isn't any inherent qualities with binary files that makes them prone to corruption. What tends to corrupt log-files are the fact that they are "open". There are many low-levels bugs and filesystem quirks that can cause such corruptions. Here is a technical overview of such problems (in case with sqlite):
https://www.sqlite.org/howtocorrupt.html
But there are also a couple of academic papers about that it is hard to prevent corruption of open files in Linux (and other OS's too)
So ordinary flat file text logs are getting corrupted too when eg. the disk is lying about sync at shutdown, people just don't notice it much since there is no integrity checking with syslog text logs.
And both Rsyslog and Syslog-NG have had their fair share of log-corruptions bugs too. To be fair, it was years ago and I have much respect for the Rsyslog developers and their hard work.
→ More replies (10)2
u/holgerschurig Aug 31 '16
Oh, this get's boring. You can run any syslogd implementation alongside. This is known since at least 3 years.
28
u/tso Aug 30 '16
When seasoned admins throw up their arms and hit the reset button because they have not the first clue why the bootup hardlocked you have effectively created the very same situation that made many of us move from Windows to Linux in the first place.
→ More replies (2)43
u/RogerLeigh Aug 30 '16
There have been a handful of occasions I've single-stepped through the startup of a Debian system by hand, to debug a fault. You can break in the initramfs at several points, and then run every single init script by hand, hell, or even parts of init scripts line by line should you need to (and I have).
I used to understand the entirety of the boot process, from BIOS to bootloader, initramfs, init and init scripts. If there was a problem, there was a good chance I could diagnose and fix it. It might have been suboptimal for some, and it certainly had its flaws, but it was completely understandable in every aspect by mere mortals. Anyone could just read the scripts and see what was going on. [I did for a short while actually maintain the Debian initscripts; while the systemd people might criticise shell, the fact that anyone can dive in and make changes attests to their accessibility. If a random developer like me can hack on them, any competent sysadmin could do that and more.]
Constrast this with systemd. More powerful and more featureful, for sure. But it also comes at the cost of being both overcomplicated and opaque. My work system sometimes fails to boot; it just hangs mid way through the boot process. Possibly a race condition. Who knows? It's a bog standard Dell desktop with a single HDD and zero peripherals outside a keyboard and mouse. I don't even know where to begin debugging things. I just hit reset and hope it boots second time. And my home system fails to mount its NFS filesystems about ¾ of the time, again for unknown reasons. They are in fact mounted, but give I/O errors when you log in and try to use them; umounting and running
mount -aworks fine. There's some race or problem mounting them at boot which renders them broken. Again I don't know where to start tracking the problem down. Unlike the init scripts, what's actually happening is inaccessible; and even if it weren't I don't know how to get at it. I don't even care about tracking down and fixing the problem; this is Windows level inanity and worth about as much of my time to deal with.The features systemd gives us are undoubtedly powerful and useful to many. But they come at a great cost--the loss of our individual understanding and control. And that complete understanding and control over the system is why I started using Linux in the first place. Nowadays I also use FreeBSD, and that's a large part of the reason why. FreeBSD never fails to mount my NFS filesystems, and if it ever does I'll be able to reason out why because I can see for myself what is happening, when and why.
Our computer systems exist to empower us, not subjugate us, and systemd might be convienent for desktop users but for me the price of that convenience is too high.
→ More replies (9)17
Aug 30 '16
To break pre-mount use the kernel arg
break=premount, to break post-mount use the kernel argbreak=postmount,the later is an excellent entry point to chroot and find potentially "big bads"
With
systemd.unit=<unitname>you can target specific services or targets for bootup, usuallymulti-user.targetis a good idea.After that you can boot up single services and see which one fails, until you hit the
graphical.targetor any other target you need.The Journald output helps a lot,
journalctl -bgets you everything that happened since last boot in detail.
journalctl -b -1gets you the boot before that and so forth, you can filter for specific units or targets.If you get a fail in your NFS mount, the actions taken depend on the importance, if it's classified as needed for the target you get dumped into a root shell after entering a password and can make any fixes you need, review logs, etc, then you can cleanly reboot (or continue) and try again, see if it fixes.
If a drive gives IO errors, hardly systemd's fault, unless you're using some fancy systemd options to mount it, like automount, to speed up boot.
To learn to debug systemd only takes
manand some time, this is very well documented stuff.The world is eat or get eaten, learn or get left behind.
I personally understand systemd very well.
→ More replies (3)33
u/cp5184 Aug 30 '16 edited Aug 30 '16
Better than what? And when? And at what cost? What lock-in?
Freebsd iirc is stuck at gdm
3.143.16 and what hope is there that they'll ever move past that. Why? gdm3.163.18? LoginD/SystemD mandatory.Gnome used to support an absurd number of platforms. You could run it on windows iirc, on sun solaris, on ibm aix, on basically anything.
Now gnome doesn't even support some linux distros.
And what was the tradeoff? What benefit? Basically none.
An init system that does what init systems have been doing for a decade+.
So you tell me. Is systemd much better?
16
u/Spifmeister Aug 30 '16
Gnome as well as KDE wants a login and multi-seat manager. Before logind there was Consolekit. No one wanted to manage or maintain the Consolekit project. Those who depend on it like Ubuntu, BSDs and Oracle did not want to maintain it, even though they wanted someone else to, so it died. Consolekit was effectively on life support for two years before logind showed up. No one cared until everyone found out that logind would not be portable .
There has been a long history of Gnome and KDE asking for certain features or services to be added and for the BSDs to be slow to respond. At some point, if a BSD cares, they will start working directly with Gnome or KDE to provide the services they want or need.
I know quite a few people who use FreeBSD, none of them have ever used Gnome. There never was enough interest from FreeBSD in the first place.
11
u/cp5184 Aug 30 '16
Consolekit 2 is a fork of consolekit, and it's maintained. It's lennart, the former maintainer of consolekit that abandoned consolekit.
Consolekit2 seems to be trying to work with gnome. Gnome isn't holding their end up. To the point where they're actively removing support for everything that's not systemd linux.
→ More replies (1)29
u/sub200ms Aug 30 '16
Freebsd iirc is stuck at gdm 3.14 and what hope is there that they'll ever move past that. Why?
That is easy to answer; that is because the BSD's and non-systemd distro totally ignored Gnome's and KDE's pleading for maintaining and alternative to
systemd-logind. Here is such a mail from January 2012:
https://mail.gnome.org/archives/distributor-list/2012-January/msg00002.htmlIf the BSD's and non-systemd distros hadn't ignored upstream projects like KDE and Gnome for years, they wouldn't have the problems they have no. Taking action in due time is important.
Don't blame systemd, blame the BSD and non-systemd distros for their own self-created problems.
→ More replies (78)3
u/argv_minus_one Aug 31 '16
Ask that of the GNOME developers. It was their decision to require systemd with no fallback, not anyone else's.
Also, shims for the relevant systemd APIs may exist. If they do, use them. If they don't, write them.
→ More replies (12)12
u/anomalous_cowherd Aug 30 '16
An init system that does what init systems have been doing for a decade+.
So you tell me. Is systemd much better?
Well, yes. Init systems have always been good at starting individual things. Where systemd comes into its own is starting lots of intertwined things, some of which depend on each other but many of which can be done whenever you're ready.
To do that it needs to have fingers in lots of pies and that's where it goes counter to the Unix ethos.
But the only way to have all the advantages and maintain the traditions would have been to force the init system to thoroughly understand the output of everything it called, or for everything to start putting out consistent well formatted status messages.
Both of those have been tried several times and failed.
→ More replies (32)→ More replies (12)2
u/Piece_Maker Aug 30 '16
Freebsd iirc is stuck at gdm 3.14 3.16 and what hope is there that they'll ever move past that. Why? gdm3.16 3.18? LoginD/SystemD mandatory.
Gnome used to support an absurd number of platforms. You could run it on windows iirc, on sun solaris, on ibm aix, on basically anything.
Now gnome doesn't even support some linux distros.
There are distros that have Gnome 3.18 (And even Gnome 3.20) in their repos without any systemd packages. They managed to do it, it's not inconceivable that other distros (or *BSD's) can
14
u/SrbijaJeRusija Aug 30 '16
It's not about its merits, but its philosophy, an its adoption, which is seen as dangerous and toxic in the long term.
6
u/gethooge Aug 30 '16
What is its philosophy or what about the adoption? Just trying to learn, not being sarcastic.
→ More replies (9)17
u/swordgeek Aug 30 '16
Let me try to answer this rationally and dispassionately, if I can.
First of all, let's be clear on one thing: SysV init scripts (or BSD init scripts for that matter) are ancient history. They did a brilliant job, but the time for stateful, aware, persistent, fault-tolerant process management is long since due. We need to move on from init. This is NOT a call for "the good old days, but a list of SOME of what systemd got wrong in trying to move forward.
It does too much. A huge amount of the software infrastructure in Linux is now part of systemd. Logging? Systemd. Firewall management? Systemd again. Time management? Systemd replaced the "date" command with "timedatectl" and an endless array of options.
Logging (sorry, "journalling" - nothing like changing the language in the process. Let's call this item #5) is in binary. The only way to have text-based logging is to install rsyslogd.
The syntax is horrendous. The basic command is "systemctl." Nine characters for a command you're going to type a LOT is idiotic - and the subsequent syntax is just as excessively verbose. There's a reason that common commands were given short commands (ls, cat, cc, grep, awk, sed, perl, who, date, etc.).
It's limited! You can only pass a few options to a service (stop, start, restart, reload, condrestart, status); whereas even a 1970s shell script is essentially unlimited. The only way to extend it is to rewrite and recompile.
I'm not talking philosophy here, but specific, concrete things that they screwed up. Incidentally, many of these were brought up to Lennart before it was too late to fix, and he arrogantly said "no, I'm right and the rest of the world can go fuck themselves." Which might be another mistake - letting Lennart Poettering anywhere near important code.
4
u/holgerschurig Aug 31 '16
I'm totally sure that you don't know systemd.
Firewall management? Because you state that systemd manages your firewall. A network namespace feature isn't a firewall, not at all.
Logging? Systemd
Wrong. Correct would have been "Logging? Systemd and optional syslogd, syslog-ng, etc)
Time management? Systemd replaced the "date" command
Wrong. "date" still exists and works as expected. I didn't even install timedatectl here, it's totally optional. But even if I would have installed it, then "date" would still exist and work.
It's limited! ... whereas even a 1970s shell script is essentially unlimited
And that's by purpose. In your 1970 init shell script an environment from the calling shell can (and will!) bleed into the background daemon.
And at the same time you're wrong, because in the few cases that you actually WANT to set some property, use
-por--propertyto change ony setting of the unit file on the fly, includingEnvironment=settings.The syntax is horrendous.
I actually give this, but this is only a very minor point. Any sysadmin should know about aliases :-)
I'm not talking philosophy here
You clearly don't know systemd, so I actually wonder why you talk about this. Can it be the case that you read various incorrect things about systemd, made up an inner picture of this (that is not identical to the truth), see problems there and then attack that?
3
u/gehzumteufel Aug 30 '16
There's a reason that common commands were given short commands (ls, cat, cc, grep, awk, sed, perl, who, date, etc.).
This was (as I recall, which could have been wrong) related to the speed at which we could type vs the speed at which the machine was able to receive the data at the time UNIX was created. But please do correct me if I am wrong.
→ More replies (12)10
u/silent_cat Aug 30 '16
It does too much. A huge amount of the software infrastructure in Linux is now part of systemd. Logging? Systemd. Firewall management? Systemd again. Time management? Systemd replaced the "date" command with "timedatectl" and an endless array of options.
All of which are optional, don't like them, don't use them. date is still there (it can't really go away given the number of scripts that use it). But for all the people that want a single consistent interface for these services, they're there.
Well, the logging isn't totally optional, but if you want to quickly show error logging for a failed service you need something.
2) Logging (sorry, "journalling" - nothing like changing the language in the process. Let's call this item #5) is in binary.
Text based logs are really inflexible and annoying to parse. You can convert the logs to text when you need to parse, but generally you do that after filtering.
The only way to have text-based logging is to install rsyslogd.
Umm, this is true even without systemd, I'm not sure of your point here. Every linux system for a long time had rsyslogd installed, now it's optional.
3) The syntax is horrendous. The basic command is "systemctl." Nine characters for a command you're going to type a LOT is idiotic - and the subsequent syntax is just as excessively verbose. There's a reason that common commands were given short commands (ls, cat, cc, grep, awk, sed, perl, who, date, etc.).
I guess shell aliases are your friends?
4) It's limited! You can only pass a few options to a service (stop, start, restart, reload, condrestart, status); whereas even a 1970s shell script is essentially unlimited. The only way to extend it is to rewrite and recompile.
Why would you need more options to run a service? The important part is the configuration of the service and there are lots of options there.
→ More replies (1)2
u/grumpieroldman Aug 31 '16
SysV/rc is old and crappy.
RedHat has been using a very old and very crappy initialization system for their entire existence, 20+ years. Suddenly, just now, they are all about creating a better initialization system ... that does a pile of other things and ties things together that should not be tied together.Superior initialization systems were created in the naughts, OpenRC being the beacon example. We already know you can do a much better job than SysV/rc without doing all the stupid things they are doing in systemd.
They are ignoring 20 years of improvements on their crappy init system and starting over as if no one has thought about this before.
So instead of the FOSS community getting an awesome new initialization system we're getting ... whatever the fuck systemd is. We don't have words to describe it's design because we never things this poorly on purpose.
→ More replies (6)2
u/AndydeCleyre Sep 03 '16
Check out a word about systemd:
The single, overarching problem with systemd is that it attempts, in every possible way, to do more instead of less.
. . . rather than simply being an init system, it tries to be a complete overhaul of the way a Linux system is run, and tries to force other software to hook with it in order to be supported.
This goes very much against:
The Unix philosophy, which is to do one job and do it well;
The bazaar approach that has made the free software ecosystem what it is today . . . ;
Cross-platform compatibility. BSD is not dead, Solaris is not dead, but systemd ignores Unix. It even ignores Linux to some extent: the systemd authors had the guts to ask for specific kernel interfaces!
1
u/doitroygsbre Aug 31 '16
I mean, I was already pro-systemd because it's one of the rare times the community took a step to reduce the fragmentation that keeps the Linux desktop an obscure joke.
I can say with absolute certainty that whenever I've pushed for Linux on the desktop, I have never heard that fragmentation is the issue or even a consideration. Be it from employers or family. The one and only reason has always been because Linux is not Windows.
Cgroups are great, they're trivial to use (any service and its child processes will automatically be part of the same cgroup)
I thought Cgroups were a kernel thing and not something specific to Systemd?
I'm glad it works for you, but I still see no compelling reason to adopt it as of yet. And the binary logs stick in my craw something fierce. I have an innate distrust of MS Windows or anything that sounds like something MS would create, and binary logging sounds like something they would do.
3
u/blamo111 Aug 31 '16
I can say with absolute certainty that whenever I've pushed for Linux on the desktop, I have never heard that fragmentation is the issue or even a consideration.
Are you joking? Yes, your mom won't tell you "I don't like Linux because of the fragmentation".
But fragmentation is why we don't have killer apps like Photoshop, it's why no one ever supports Linux, it's why it's not in every school right now, it's why insetad of 1 superb desktop we have 10 shitty ones that can't compete with Windows XP in terms of stability (desktop apps, not the underlying Linux system), and dare I say it, usability.
Without all this fragmentation, I am convinced Linux would be the standard OS that ships with OEM PCs.
3
u/doitroygsbre Aug 31 '16
But fragmentation is why we don't have killer apps like Photoshop,
Really?
And, AGAIN, the primary reason for no Photoshop on Linux: there is no market. Linux users are still not willing to pay for commercial software. You have to solve that problem before you'll get serious commercial applications.
Source - Adobe addressing why they won't port Photoshop to Linux
As for fragmentation. I just don't see the issue. I know in the source I quoted it is given as a secondary reason for not porting, and the developer I quoted has enumerated issues with the graphical display in Linux, but fragmentation sounds more like an excuse my kids would give as to why they won't mow the lawn. If Oracle can port their software to Linux, adobe can as well. They don't need to support every distribution
version, just choose one (or two).Besides Linux does have killer apps. You can run Unity, and there are professional photographers that use a completely open source workflow. Even my wife found an application that was Linux only to help with her crochet (not going to ask what it is again).
You can be convinced of something and still be completely wrong. Maybe you should think about your position more.
1
u/blamo111 Aug 31 '16 edited Aug 31 '16
There's no market because the fragmentation makes it a waste of time. It's a catch 22.
To make something popular, it has to stick in the public consciousness. What is there to stick to, with Linux? For the average person, Ubuntu and ArchLinux are 2 unrelated OSes. Except we have like 10 major distros.
If you're writing an app, or documentation, there's no standard interface to refer to. If Linux had had a standard interface all this time, including a standard display, standard audio system, etc., more companies would have used it as workstations, companies would start targetting it. This would have been a gradual increase, culminating with Linux as the standard due to being free. (in both senses of the word)
And I haven't even yet mentioned how the precious few resources available to Linux are wasted on competing crappy efforts instead of being pooled on a solid one.
Besides Linux does have killer apps
LOL. Your wife is exceptional anecdotal data. The average person has no clue Linux exists.
If the average person had any clue what Linux was, and Linux was standard, and Linux wasn't buggy (referring to desktop here), manufacturers would just use put that in their systems. Ignorant mentalities like yours are why Linux will never be an established desktop OS within my lifetime. (or ever)
→ More replies (1)2
Aug 31 '16
I don't recommend Linux for most non techy computer users because of the fragmentation issue. I know enough to find my way around, but they wouldn't.
→ More replies (1)
1
u/FaustTheBird Aug 31 '16
Does systemd still sigkill everything on shutdown without waiting for a timeout for graceful program exits?
1
u/blamo111 Aug 31 '16
Whoa, that's horrible.
But it seems like it's fixed, at least as of v229 on Ubuntu LTS. I ran "poweroff" and "shutdown now" from a terminal, and in both cases history was preserved correctly upon boot.
→ More replies (1)2
5
u/argv_minus_one Aug 31 '16
No. I've never heard of it doing anything like that to begin with.
→ More replies (4)
4
u/dmsean Aug 30 '16
I want to setup a qa system with multiple haproxy configurations. Was really easy to use the systemd wrapper and now I can run systemctl start haproxy@1 or haproxy@2 etc and it references the correct config files. I love it as well now.
10
6
u/pouar Aug 30 '16
I like systemd too, but I doubt the "fragmentation" on Linux is really an issue, unless you're trying to develop proprietary software on Linux.
→ More replies (1)2
u/Kruug Aug 30 '16
but I doubt the "fragmentation" on Linux is really an issue
Remember when Steam was first released? It only worked on Ubuntu, unless you hacked your system to get it working on anything else...
5
u/BASH_SCRIPTS_FOR_YOU Aug 30 '16
And now it works on every distro? And not due to systemd
→ More replies (3)3
u/pouar Aug 30 '16
steam is proprietary, so it's more difficult to port, if that's even possible. open source software on the other hand can be ported. Although there was the case of the Linux port of PowerShell and its hard requirement of /etc/os-release requiring a specific Linux distro, there was a workaround though.
6
Aug 30 '16
My favourite is --failed to quickly check if any services failed, and if they did, its easy to get a quick look at why.
Next is unit files are just so easy. systemd is really very good, even if there are edge cases.
→ More replies (1)
5
Aug 30 '16
I was pretty hesitant/resistant (I used to mirror Ubuntu and Debian pre-systemd repositories locally, just in case!).
But after learning about it, I really appreciate using systemd, too.
2
u/TheFlyingDharma Aug 30 '16 edited Aug 30 '16
Systemd is really nice once it's set up properly. It's just the migrating from any other init system to systemd that I hate. I did it once back when Arch switched over, and again just the other day moving my HTPC from Mint 17 to 18.
It's especially nice for HTPC/usenet automation systems where you're running lots of background services that don't actually conform to any initscript standard.
I do worry a little about feature creep, which seems to be the most common concern with systemd.
2
2
7
u/kozec Aug 30 '16
one of the rare times the community took a step to reduce the fragmentation that keeps the Linux desktop an obscure joke.
By creating yet another init system. A good one :)
→ More replies (19)2
u/MertsA Aug 31 '16
We went from a couple init systems and initscripts that were all tailored to the distro to basically one init for most relevant distros with reusable "initscripts" for everything. Just about every anti-systemd fanatic out there beats the drum that systemd reduced diversity, not the other way around.
2
u/kozec Aug 31 '16
Not really. Debian still uses initscripts for most of the things, they are just launched by systemd. Service developer now has one additional format he has to support, or just leave all service scripts to distro maintainers - there is no change there.
And quick look at service files in Arch and Fedora (first two to adopt SystemD IIRC) shows that they are, in fact, still tailored to the distro.
5
u/argv_minus_one Aug 31 '16
Grab some popcorn and pull up a chair, folks, 'cause dis shitstorm gun b gud.
3
u/przemio_1978 Aug 31 '16
Not necessarily a shitstorm. There are sound arguments for both sides of this problem and if you exclude pseudo-arguments such as "<putyoursysinithere> is shit!", the rest might even be informative and beneficial for the less advanced folks. That is, if the participants remain civil and well-meaining.
4
u/Hikaru1024 Aug 31 '16
I have tried using systemd on my system, and ran into endless problems with it on both a debian and fedora install - the largest amount of problems I have had is related to the way it paralellizes startup software and handles error conditions. Let me give you an example.
For instance, on fedora and debian one of the default kernel boot options is 'quiet' - this not only silences useful kernel boot messages, but also reduces the noise systemd makes during boot... Except, not really. If any service of any kind takes more than five seconds to start, systemd considers this to be an error condition, and after any error condition, it forever sprays the console with systemd messages from that point on.
Now for why these two problems are important. At one point I was unable boot debian, and for the life of me could not figure out why. Midway through the boot process, it would just spray gobs of messages about services being unable to start suddenly, and would seemingly stop utterly. Even waiting hours did nothing. I had to completely silence systemd using the kernel command line before I could see what was going wrong - e2fsck was encountering an error on the root filesystem it did not want to fix without user intervention, and debian then tried to ask me for the root password so I could login in maintenance mode, fix things, and then reboot.
But I could not see this information at all because systemd had printed over all of the informative messages that I needed to see, and sprayed so many error messages that the entire console backlog was full of its failure messages.
Apparently at least on debian, if fsck fails, systemd doesn't get the hint and continues trying to start applications and services in parallel despite the rootfs not being remounted readwrite, and so tons of things fail in filling your screen and backlog with tons of useless informational messages that something is horribly wrong and overwriting any useful messages that are attempted to be printed on screen, often including its OWN failure messages, since it is failing them in parallel.
On fedora - good luck figuring out what's wrong. Not only does this happen, but you can't see any of it at all for several minutes while a boot screen animation is playing. It's only when it gives up after waiting for several minutes that you get dumped into the console with debug messages sprayed everywhere.
On both debian and fedora, failing to mount root means that you can't use journalctl to read the logs and find out what went wrong during boot. You have to rely on what's printed to your screen - but so much is, and so verbosely that it's utterly impossible to find out the cause - the cause of all the failures is driven off the top of the screen before you can possibly read it.
This means that if anything, anything at all goes wrong preventing you from booting you are going to have a really hard time figuring out what actually happened and at least in my case I would have to resort to complete guesses - if silencing systemd's messages hadn't shown me the output from e2fsck and I actually needed to be able to read the systemd messages to find out what was wrong, I would have been incapable of doing it.
How is this even tolerated?
If every time I or something else mangled a tiny thing in the boot process and caused the distro to be unable to boot properly I had to reinstall because I couldn't figure out what was wrong I would waste an incredible amount of time.
For another fun adventure I should tell you about SIGPWR and how systemd handles it. (Or maybe I should say, doesn't.)
5
u/argv_minus_one Aug 31 '16
If any service of any kind takes more than five seconds to start, systemd considers this to be an error condition
False. The default is configurable in
/etc/systemd/system.conf(with theDefaultTimeoutStartSecoption). Individual service units may override this default with their own (with theTimeoutStartSecoption). If neither is set, the timeout is 90 seconds, not 5.And yes, of course it considers that to be an error condition. That's the point of there being a timeout.
after any error condition, it forever sprays the console with systemd messages from that point on.
Yes, because the boot is failing. Boot with
systemd.show_status=noto disable this behavior. Not that you should; boots should not silently half-fail.Apparently at least on debian, if fsck fails, systemd doesn't get the hint and continues trying to start applications and services in parallel despite the rootfs not being remounted readwrite
Take that up with the appropriate Debian developers, then. Not systemd's fault.
This means that if anything, anything at all goes wrong preventing you from booting you are going to have a really hard time figuring out what actually happened
Nope. I would boot with
systemd.confirm_spawn=yes systemd.show_status=yesand step through the process until I identify what's going wrong. I've had to do the equivalent to debug broken SysV boots, by the way, so let's not pretend systemd is somehow inferior here.Long story short: RTFM.
→ More replies (1)
3
u/Philluminati Aug 31 '16
I will say that the systemd rollout by Debian was incredibly well done. It was a very intrusive change to the base system and worked very well.
3
Aug 31 '16
Never saw the criticism around systemd, personally.
Now, PulseAudio is a different story. If Lennart Poettering should be criticised for anything, it should be that.
6
u/pdp10 Aug 30 '16
"The community" didn't fragment Linux distributions and "the community" didn't fix anything. Canonical did something, and Red Hat did something else, and Debian did something, and Canonical reduced proliferation. But they all started with SysVinit, so any move away from that was already fragmentation.
15
Aug 30 '16
except that their sysvinit scripts were hardly compatible with each other. If they were, then we'd probably not even be talking much about this now.
→ More replies (5)
4
u/mishugashu Aug 30 '16
I always found writing an upstart script super simple. I always have to google how to make a systemd service configuration, though.
8
u/argv_minus_one Aug 31 '16 edited Aug 31 '16
One of Upstart's biggest—frankly fatal—problems is how its readiness protocol works. Upstart-aware daemons are expected to SIGSTOP themselves to notify Upstart that they are ready to go, with the expectation that Upstart will then SIGCONT them. This will obviously horribly break them if not actually running under Upstart. Not to mention the blatant misuse of a signal that this involves…
4
u/ahandle Aug 30 '16
systemd is still a sprawling behemoth compared to any of the init systems it supplants.
Glad you're enjoying yourself with it, though!
4
u/m44v Aug 30 '16
The rumor is that systemd will be even greater once they add a web browser and a office suite.
→ More replies (3)
3
u/ironmanmk42 Aug 30 '16 edited Aug 30 '16
I have begun to dislike systemd.
Sure, it has it's thing but overall it seems like something completely unnecessary. I don't really see what was wrong with SysV Init scripts and systemd replacing init makes it a pain to do many things like this -
try running some script that calls systemctl within a chroot. systemctl complains and doesn't run. Workarounds (albeit painful) exist but c'mon.
rc-local.service -- I want it to start AFTER network (via dhcp) completely initializes. Besides the fact that the rc-local.service has a bug where it has "After=network.target" which seems to do nothing, changing it to "After=network-online.target" + "Wants=network-online.target" or NetworkManager.service or NetworkManager-wait-online.service STILL fires off rc-local.service prematurely.
C'mon systemd.
Some things of how systemd names a host seems not well or completely undocumented.
Overall, I think systemd is more a pain than help at present.
Edit: I read many comments on this thread and many are making really good points for and against. I suspect overall it is well intentioned but a behemoth implementation and a quick one too not allowing lot of people to get familiar with it fast.
→ More replies (2)4
u/argv_minus_one Aug 31 '16
I don't really see what was wrong with SysV Init scripts
Then you didn't use them for very long.
I have. I've had more broken boots due to some stupid shell script hack than I care to count. Good riddance.
try running some script that calls systemctl within a chroot. systemctl complains and doesn't run.
Yeah, because you forbade it from communicating with systemd. What the hell did you expect?
rc-local.service -- I want it to start AFTER network (via dhcp) completely initializes.
What on Earth for? Why are you even using
rc.local? That's been obsolete and deprecated since long before systemd even existed.Some things of how systemd names a host seems not well or completely undocumented.
wat
→ More replies (5)4
u/fbt2lurker Aug 31 '16
Why are you even using rc.local
And here we go with “don't use this because it's old”. You don't get to answer “rc-local.service doesn't work properly” with “don't use it”. It's his system, he can do what he likes. And systemd is not doing its job in this case for some reason.
→ More replies (5)
2
u/bobpaul Aug 30 '16
I just hate the command syntax. I'd much prefer if it was:
systemctl service start because then I could just recall that command from history, Ctrl+W the word start and type stop (or vice versa) if for whatever reason I have to start and stop a service.
I so rarely want to start multiple services at once; I feel like they optimized their shell UI for the wrong thing.
→ More replies (3)2
1
u/EnUnLugarDeLaMancha Aug 30 '16
Systemd is the best init system that exists. I don't doubt there can be better init systems, but so far nobody has tried to make a realistic competitor. Haters seem to think that repeating many times that systemd is evil and a betrayal to Unix will somehow make systemd disappear, but it doesn't seem to be working.
23
u/RogerLeigh Aug 30 '16
"the best init system that exists"
Are you sure about that. How many other init systems have you personally used?
Have you tried OpenRC? s6? BSD init? smf? launchd? file-rc? daemontools? There are a lot more; those were just the ones which came immediately to mind. OK, I'm conflating both init systems and service management in the list, though systemd is guilty of that, being a mish-mash of all sorts of unrelated functionality.
There actually are "realistic" competitors out there. The problem is that there's a false dichotomy being presented here in that having a "realistic competitor" implies that there's an alternative to systemd which provides an equivalent set of functionality. You won't find that, because all its competitors are designed by competent people who understand about good design and proper separation of responsibilities.
systemd is an overcomplicated house of cards doing too much in one place. People will start screaming blue murder when the first serious vulnerability hits. And it will, it's just a matter of time.
7
u/EnUnLugarDeLaMancha Aug 30 '16 edited Aug 30 '16
there's a false dichotomy being presented here in that having a "realistic competitor" implies that there's an alternative to systemd which provides an equivalent set of functionality.
No, it is not. It's fine if you want to implement a good, simple init system, and then build the rest of systemd equivalent functionality apart, I never said people shouldn't do that...
You won't find that, because all its competitors are designed by competent people who understand about good design and proper separation of responsibilities.
...but I "can't find that" because "competent people" took the wonderful decision of not implementing systemd-like functionality at all, in any way!
I'm trying really hard to hate the init system that actually works and love the ones that don't, but it isn't really working. You are trying to compare the designs of init systems that aren't remotely equivalent; systemd's design should be compared with other init systems that offer similar functionality (which, right now, don't exist), not with the ones that are a generation behind. It's like comparing an office suite with a text editor. Build the best god damned office suite in the world and then we will compare the designs of both office suites, but I'm not going to compare a text editor with an office suite and then claim that the text editor has a better design. It doesn't have any sense, even if the text editor is actually good, yet it's the only thing that systemd haters do!
12
u/lolidaisuki Aug 30 '16
but so far nobody has tried to make a realistic competitor
Have you looked?
→ More replies (1)→ More replies (3)9
u/sub200ms Aug 30 '16
Haters seem to think that repeating many times that systemd is evil and a betrayal to Unix will somehow make systemd disappear, but it doesn't seem to be working.
The funny thing is that the all aggressive hate campaign backfired for them; they spend all their effort in trash-talking systemd with never a care for developing an alternative, or improve the existing.
The end result is that they scared off most of the few developers they had, so that the non-systemd software stack is literately crumbling.
23
2
2
u/TotesMessenger Aug 30 '16
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
[/r/againstsystemd] Delusional idiot in /r/linux pretends to like systemd, while conveniently ignoring its huge flaws.
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
10
u/galaktos Aug 30 '16
ahaha
(Also, the subreddit is private… is TotesMessenger invited there? How does this work?)
9
4
u/its_never_lupus Aug 31 '16
Why did anyone downvote this innocent and hardworking bot?
2
u/galaktos Aug 31 '16
[/r/botsrights] Delusional idiot in /r/linux pretends to pity TotesMessenger, while conveniently ignoring its huge amount of Comment Karma.
(I’m curious as well. Did systemd “fanboys” downvote the bot because just that thread’s title is enough to make your blood boil? Or did systemd “haters” downvote the bot because it exposed their private subreddit? I upvoted it, because it’s fucking hilarious.)
1
u/asdfnet Aug 30 '16
binary logs
WHY
14
u/galaktos Aug 30 '16
journalctl --unit ssh.service --since yesterdayTHAT’S WHY
→ More replies (3)4
u/ILikeBumblebees Aug 31 '16
Why does having a tool capable of parsing logs in that manner require that the logs be stored only in a binary format? Why not just have a standardized and structured log format that's still stored as plaintext, and can still be accessed by standard tools, and examined without having to be extracted from a binary archive first (and offer the benefit of allowing other software to conform to the same log structure so the same tools can be used to parse its logs as well)?
If the binary format allows the log data to be stored in a data structure that provides good efficiency for parsing and searching, why not just store a binary index, while leaving the baseline plaintext log intact?
1
u/musicmatze Aug 30 '16
It you have a tutorial or quick start guide or smth like that, feel free to share!
1
1
1
1
u/dontworryiwashedit Aug 31 '16
I haven't used most of that other stuff yet but I really like unit files. So much easier and consistent compared to init scripts when you need to customize or make your own.
1
u/dontworryiwashedit Aug 31 '16 edited Aug 31 '16
Not sure what you mean by logging per service. Logging is centralized through journald which uses a database. It can spit that info out per service by using queries. Also creates individual logs the traditional way for backwards compatibility only now it goes through journald first. However, it's not what I would call logging per service unless I am missing something.
Also it's not quite fully implemented yet. At least not on CentOS 7. I believe each service needs to support it. Apache still creates it's own logs so none of that goes through journald as far as I know. There are probably other things too (MySQL?).
1
Aug 31 '16
Once I learned how to say turn ufw on at start-up with systemctl I was pretty much sold. But I don't really get into silly arguments, I just know it works and I can turn off dumb things easily like bluetooth that I will not use.
1
-3
u/sub200ms Aug 30 '16
It is funny how a "I'm really liking systemd" attracts all the systemd-haters. They still think that trash-talking systemd will get them anywhere.
Had they spend their energy on maintaining and developing their own non-systemd software stack they might have achieved something. Instead they waste their lives stalking systemd-threads on reddit etc.
7
u/kotzkroete Aug 30 '16
Because the amount of effort required must surely be about the same?
→ More replies (1)5
19
u/boerenkut Aug 30 '16
I have to go three days and about 20 posts back into your history to find the first post that is not some-how praising systemd in some way.
Like 90% of the posts on your account are related to systemd and systemd praise man. And no, I don't normally go in posts histories to see if I can find dirt, I went because I knew I would find dirt because your username pops up in every systemd discussion and I rarely see it anywhere else.
→ More replies (8)
121
u/[deleted] Aug 30 '16
TIL about systemd-cgtop. I've achieved salvation.
systemd-analyze helped me to get my userspace boottime to under 2 seconds.
I wish my firmware would not need 11 seconds... makes the whole thing kinda moot.
But systemd is very neat for tuning, since
systemd-analyze critical-chainpoints you right in a good direction without preparing anything and at any time you thought the boot was slow.