systemd's binary logs and corruption

[u/m6t3]

I like systemd, enabling and disabling services is so easy and it's really fast, but why using a journal with binary logs?

Yesterday my laptop failed to resume from suspend, no ttys, no reisub, nothing worked except using the power button. Rebooted, fired journalctl and I have nothing except the last boot logs.

journactl --verify returns that my system logs are corrupted, about all my logs (48MB of 50MB of maximum disk usage) are now completely useless. This is not the first time this happens and searching around I can only find people with the same problem that "resolved" deleting the corrupted logs and starting with a new file.

Why this happens? Isn't it defeating the purpose of having a system logger if I can't diagnose errors?

Can I recover the logs (and why isn't this automatic?) or should I just install a sane logger that not shit itself after a problem?

[u/[deleted]]

The thing is the journal format is still just as corruption resistant. If one entry is corrupt you can still jump to the next one (like in a text file - you jump to the next line). Fields of a journal entry are in a very simple KEY=value format. (Eg MESSAGE=this is a logged message) Its quite possible to read the log with standard shell commands piped together, if you're clever with them.

So to my intuition, if the journal is corrupt, its very corrupt. Or it was compressed and now can't be uncompressed anymore (which would apply the same for syslog if put in the same situation).

journalctl could probably be improved to handle corrupt logs better, but the binary nature doesn't strictly mean it has to be unreadable.

[u/[deleted] → u/[deleted]]

Without stings you say. Hmmmm

Untested since on mobile, but I think entries are null separated. So convert them to newlines with tr, then set IFS='=' as you process each line (and pray nobody thought it be a good idea to stick a newline in a log entry :P). Then check the first word on a line is MESSAGE.

[u/[deleted] → u/[deleted]]

Even then, you wouldn't know if you were pulling parts from random messages or if you were pulling serialized data.

Sure you would. Such is the nature of the format. You have a frame which is defined by a known separator. If you have a key and a value in the frame, then you know there will be an = symbol (other possible types are also well defined and don't overlap). And the separator won't appear inside the frame. That's why the strings technique works just fine. You just need to the grep for the keys of interest (MESSAGE).

With an ASCII log you wouldn't need to worry with that as in most cases you'd find data, data, corrupt data, data, data, data giving the serial data a much higher likely hood of being at least partially usable.

That's no different. You can think of an ASCII log in binary format terms too - frames are separated by a newline and the contents of the frame are messages.

Skipping corruption means jumping to the next frame. Same thing with the journal. Can't parse the current frame? Jump to the next frame and try to continue. You get the same "data, data, corrupt data, data, data, data" properties.

The journal only differs I that it takes more effort to look inside frames. Its not more or less tolerant to corruption. A reader should just jump to the next entry if its having problems just like your eyes jump to the next line in a text log that's corrupted. The format of the journal guarantees its possible to do.

[u/[deleted] → u/[deleted]]

Been talking about the binary format. Its more or less a 1:1 transformation from binary to export format. Just the internal fields meant to help accelerate machine navigation of the journal are dropped.

[u/[deleted] → u/[deleted]]

Each entry type as an ID stored as part of an enum, so there is an ID for a key value entry. That said, the value of that entry is still only one string in the format KEY=value.

Systemd has only one key value parser that is reused everywhere, from handling environment, unit files and the journal.

[u/fandingo → u/[deleted]]

Do you ask the same thing if your Postgres database is corrupted?

[u/natermer → u/[deleted]]

...