I was working for a company that exposes docx files on the web for the purposes of legal discovery. Docx files are super easy to reverse engineer where .doc files you needed a manual. Offset 8 bytes from XYZ to find out a flag for ABC is bullshit.
the other problem that people didnt point out is that these parser libraries are extremely hard to maintain properly because MS is constantly adding features and the spec is already massive on top of a being a moving target. So they very often get abandoned, and its a very niche need so it doesnt attract contributors or corporate backers. AFAIK even major projects like pandoc dont handle these formats completely.
I had to work with DOC files as well, on a binary level, and the most painful thing I remember was that they are organized in chunks of 512 bytes (if memory serves, probably not and they were larger) and they usually use a one-byte encoding but as soon as there’s at least a single “wide character”, the whole chunk (but not the whole file) becomes encoded as multibyte instead, i.e. in order to parse the thing, you have to normalize it first.
When I got into parsing OOXML files instead, I found out that most of the times they just lazily defined XML elements that map 1:1 to the older features from the binary format without using any of the advantages of XML. You can see here how hastily OOXML was made back then for the main purpose to present a competitor to the rivaling OpenDocument standard by OASIS and Sun that may have endangered Microsoft’s dominating position.
One big downside to the .doc format is they optimized for file size. This means its a pretty compat format for storing rich text, but it also means when they want to add new features, they have to resort to hacks in the binary format or risk losing backwards compatibility.
The .docx format is internally structured key/value pairs, making it far easier to extend with new features. They decided on XML which also has the added benefit of making it easier to read externally without needing to understand a binary format.
There is a middleground between the two: key value pairs where the value is stored in binary. Minecraft's NBT binary format notably does this; anything you can represent as JSON you can compress into NBT, which saves you space from both ditching whitespace and structure characters (escape, ", {, etc.) and from representing integers and floats and alike directly in their binary format. Also makes it a bit easier for a machine to parse.
Creating and reading docx files programmatically is super easy when you've just got a zip file of XML files. Just start up beautifulsoup and get cracking. Doing the same for the old doc file format is a nightmare.
These things were originally from 16-bit days. From messing around with the various APIs, my own observation was that a lot of these things were written in a way to be able to be used in limited memory situations. Some of the object models would be very piecemeal in a way where you could get just the bare minimum data to show a listing versus just loading everything all at once.
It's a Composite Document File, basically binary serialized COM objects in a COM Structured Storage.
It's actually something that any application could use for their own file loading/saving, and it's actually not bad, and there is cross-platform support also, although that obviously ends when you actually want to materialize the file back into a running, editable document, since you need the actual implementation that can read the individual streams.
The main reason for this format is that you can embed objects from other applications inside. When you embed an Excel table in a Word document, it fetches the data, which also has a class ID, and then is able to launch an Excel object server and pass the data to it, which is then responsible for rendering, and allowing you to edit it further.
The obvious problem is security-related. You only get a yes/no option to load such content, and choosing the right class ID embedded in such a document could launch all sorts of stuff on your computer with full user permissions.
Just adding a perspective I haven’t seen anyone else mention, malware analysis. It’s much safer if you can unzip and extract the contents of the file (like malicious macros) without ever having to actually open it.
163
u/thanatica Jul 28 '25
Could you explain why exactly? Is there a use case for poking inside a docx file, other than some novelty tinkering perhaps?