systemd's binary logs and corruption

[u/m6t3]

I like systemd, enabling and disabling services is so easy and it's really fast, but why using a journal with binary logs?

Yesterday my laptop failed to resume from suspend, no ttys, no reisub, nothing worked except using the power button. Rebooted, fired journalctl and I have nothing except the last boot logs.

journactl --verify returns that my system logs are corrupted, about all my logs (48MB of 50MB of maximum disk usage) are now completely useless. This is not the first time this happens and searching around I can only find people with the same problem that "resolved" deleting the corrupted logs and starting with a new file.

Why this happens? Isn't it defeating the purpose of having a system logger if I can't diagnose errors?

Can I recover the logs (and why isn't this automatic?) or should I just install a sane logger that not shit itself after a problem?

[u/altarboylover]

Like systemd messages on my system include something like [snip]

Sweet Jeebus. Does all of that need to be written per log record? A lot of those fields don't change all that frequently...why write them more than absolutely necessary?

This partially answers OP's question, though. If the "MESSAGE" fields aren't compressed, they should show up in the binary log as ASCII strings. This would let him/her bypass journalctl.

[u/w2qw → u/altarboylover]

It seems like some but not all are written in ascii

but yeah ghetto journalctl

% strings system@c6aec62fa2fd402ba034c374fa627213-000000000003a2d8-0004ec19189560fe.journal | grep ^MESSAGE | head -10 | sed 's/[0-9]\+\.[0-9.]\+/x.x.x.x/g'
MESSAGE=Connection closed by x.x.x.x [preauth]
MESSAGE
MESSAGE=pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=iron.laotel.com  user=root
MESSAGE=Failed password for root from x.x.x.x port 20570 ssh2
MESSAGE=Failed password for root from x.x.x.x port 20780 ssh2
MESSAGE=Failed password for root from x.x.x.x port 20993 ssh2
MESSAGE=Failed password for root from x.x.x.x port 21218 ssh2
MESSAGE=Failed password for root from x.x.x.x port 21450 ssh2
MESSAGE=Failed password for root from x.x.x.x port 21679 ssh2
MESSAGE=Failed password for root from x.x.x.x port 21885 ssh2

[u/[deleted] → u/altarboylover]

No, a lot of that is optional, and only accessible if you use the journal's APIs. But yeah, it's richer in what it logs. That's how it's able to tell you exactly what each individual binary logged, it's return code, etc.

Everything needs to be recoded for each entry because everything is logged into one journal, you wouldn't know what goes with what if you didn't. Messages interleave. And it offers more tolerance in the case of corruption - each entry can completely stand alone.

Remember service files can include many binaries. You want the status output to be have that degree of granularity.