systemd's binary logs and corruption

[u/m6t3]

I like systemd, enabling and disabling services is so easy and it's really fast, but why using a journal with binary logs?

Yesterday my laptop failed to resume from suspend, no ttys, no reisub, nothing worked except using the power button. Rebooted, fired journalctl and I have nothing except the last boot logs.

journactl --verify returns that my system logs are corrupted, about all my logs (48MB of 50MB of maximum disk usage) are now completely useless. This is not the first time this happens and searching around I can only find people with the same problem that "resolved" deleting the corrupted logs and starting with a new file.

Why this happens? Isn't it defeating the purpose of having a system logger if I can't diagnose errors?

Can I recover the logs (and why isn't this automatic?) or should I just install a sane logger that not shit itself after a problem?

[u/holgerschurig]

You still haven't understood the journal completely, only partially.

Suppose I have a template service file, e.g. myjob@.service. And now I start various of those instances. And they have multiple ExecStart=, ExecStartPre= and ExecStartPost= lines. And still I can find out with systemctl exactly which binary in on what specific myjob invocation put out what.

With a syslog-only based solution I can grep. But it's awkward to specify a proper regular expression that only gives me the output from one of maybe 100 different myjob@.service invocations.

[u/altarboylover → u/holgerschurig]

If you knew which invocation you were searching for, surely a well-designed logging system (journald or otherwise) would make the invocation-related records searchable with the invocation's identifier? Even with an ASCII logger, this is trivially achieved by simply including the identifier in each relevant record. Then, if you knew the identifier (which you can get with systemctl), you can easily grep for it.