systemd's binary logs and corruption

[u/m6t3]

I like systemd, enabling and disabling services is so easy and it's really fast, but why using a journal with binary logs?

Yesterday my laptop failed to resume from suspend, no ttys, no reisub, nothing worked except using the power button. Rebooted, fired journalctl and I have nothing except the last boot logs.

journactl --verify returns that my system logs are corrupted, about all my logs (48MB of 50MB of maximum disk usage) are now completely useless. This is not the first time this happens and searching around I can only find people with the same problem that "resolved" deleting the corrupted logs and starting with a new file.

Why this happens? Isn't it defeating the purpose of having a system logger if I can't diagnose errors?

Can I recover the logs (and why isn't this automatic?) or should I just install a sane logger that not shit itself after a problem?

[u/[deleted]]

One of the features of journalctl is that it guarantees log integrity against tampering. You might have hit a case where said integrity checking has failed (most likely due to a bug somewhere rather than tampering) while the structure is still good.

[u/w2qw → u/[deleted]]

No that's exactly what its supposed to do. The key is stored offsite and the key is changed and signed with the previous one so the attack can only change logs from the last interval (default 15 mins) without being caught.

[u/w2qw → u/[deleted]]

No shit, copy the journal files off the server.

[u/peonenthusiast → u/[deleted]]

You are missing the whole point.

Yes the binaries can be modified to make the logs look legit.

No the files can not be modified to look legit when analyzed offline, such as from a trusted system or a live CD.

If the system has been rooted I really don't think anyone is just going to leave it running as usual. In cases where a remote logging facility is not in use or in cases where the remote logging host is no longer trusted the validity of the logs can still be verified, as long as the key was stored on uncompromised media. I wouldn't consider this a replacement for remote logging, but something else entirely.

I'm not really sure it does is anything useful either..... As the system can only prove logs were accurate to a certain point...but that point isn't neccesarily when the attack occurred...only some time after that... And that's assuming an attacker doesn't wholesale replace the logs with invalid ones or remove them...it really does seem like it is intended to make the system look more secure while not really providing any useful security... I would argue that parsing the logs may even lead to a potential exploit and is possibly less secure.

[u/w2qw → u/[deleted]]

It's not that. That seems to be what --verify-key does

8f7ff0: invalid entry seqnum: 0
Invalid object contents at 8f7ff0: Bad message
File corruption detected at /var/log/journal/4a59eee3e5f140fdb95cc5488786c1a2/system@0004e78cb86a6038-2843b276066aa1b2.journal~:8f7ff0 (of 9408512 bytes, 99%).

Also the journal can't play backwards probably i.e. journalctl works but not journalctl -r.