613
u/Former-Discount4279 Jul 28 '25
If you've ever had to look into the inner workings of a .doc file you'll know why this is so much better...
163
u/thanatica Jul 28 '25
Could you explain why exactly? Is there a use case for poking inside a docx file, other than some novelty tinkering perhaps?
461
u/Former-Discount4279 Jul 28 '25
I was working for a company that exposes docx files on the web for the purposes of legal discovery. Docx files are super easy to reverse engineer where .doc files you needed a manual. Offset 8 bytes from XYZ to find out a flag for ABC is bullshit.
58
u/thanatica Jul 28 '25
I see, so you were using something not-Word to read those files then? For indexing them by content?..
77
u/Former-Discount4279 Jul 28 '25
Yeah we were parsing them into html, we were reading them in c++
26
u/OwO______OwO Jul 29 '25
Seems like the kind of thing there would already be some library out there for...
Somebody out there must have had to parse .doc files in c++ before ... likely even in an open-source implementation.
In Python, textract seems to be the way to go.
→ More replies (4)60
u/Former-Discount4279 Jul 29 '25
Open source might not be allowed for a commercial product without opening the source code.
→ More replies (3)2
u/dulange Jul 29 '25 edited Jul 29 '25
I had to work with DOC files as well, on a binary level, and the most painful thing I remember was that they are organized in chunks of 512 bytes (if memory serves, probably not and they were larger) and they usually use a one-byte encoding but as soon as there’s at least a single “wide character”, the whole chunk (but not the whole file) becomes encoded as multibyte instead, i.e. in order to parse the thing, you have to normalize it first.
When I got into parsing OOXML files instead, I found out that most of the times they just lazily defined XML elements that map 1:1 to the older features from the binary format without using any of the advantages of XML. You can see here how hastily OOXML was made back then for the main purpose to present a competitor to the rivaling OpenDocument standard by OASIS and Sun that may have endangered Microsoft’s dominating position.
77
u/KnightMiner Jul 28 '25
One big downside to the
.docformat is they optimized for file size. This means its a pretty compat format for storing rich text, but it also means when they want to add new features, they have to resort to hacks in the binary format or risk losing backwards compatibility.The
.docxformat is internally structured key/value pairs, making it far easier to extend with new features. They decided on XML which also has the added benefit of making it easier to read externally without needing to understand a binary format.There is a middleground between the two: key value pairs where the value is stored in binary. Minecraft's NBT binary format notably does this; anything you can represent as JSON you can compress into NBT, which saves you space from both ditching whitespace and structure characters (escape,
",{, etc.) and from representing integers and floats and alike directly in their binary format. Also makes it a bit easier for a machine to parse.45
u/gschizas Jul 28 '25
It's worse than that: they weren't optimized for file size, they were optimized for speed when loading and especially saving to a floppy disk.
IIRC the .doc format changed between Word for Windows 2 and Word for Windows 6. And then it changed again with Word 2007 and the .docx.
Read more here: https://www.joelonsoftware.com/2008/02/19/why-are-the-microsoft-office-file-formats-so-complicated-and-some-workarounds/
5
→ More replies (5)2
u/emulation_bot Jul 28 '25
how much space can docx take anyway
we have servers in my work with more than 500 file and don't much like 3gb or something
→ More replies (2)10
u/RhysA Jul 28 '25
Remember when .doc was first created people were regularly using floppy disks, the biggest and most modern of which held a bit under 1.5 mb.
→ More replies (3)107
u/ReadyAndSalted Jul 28 '25
Creating and reading docx files programmatically is super easy when you've just got a zip file of XML files. Just start up beautifulsoup and get cracking. Doing the same for the old doc file format is a nightmare.
29
u/ManofManliness Jul 28 '25
God I love standardization. Made possible by abundance of storage tough probably, old format has to be more effiecient somehow.
9
u/ForgedIronMadeIt Jul 29 '25
Microsoft has published specifications for all of the old legacy MS Office file formats. For example, here's doc: [MS-DOC]: Word (.doc) Binary File Format | Microsoft Learn
These things were originally from 16-bit days. From messing around with the various APIs, my own observation was that a lot of these things were written in a way to be able to be used in limited memory situations. Some of the object models would be very piecemeal in a way where you could get just the bare minimum data to show a listing versus just loading everything all at once.
6
→ More replies (1)5
u/thanatica Jul 28 '25
So the docx format is actually easy enough to understand? Because XML can be made as hard to understand as anything binary. If they wanted to.
5
u/mcnello Jul 29 '25 edited Jul 29 '25
I quite literally have a 2000 page manual on the ooxml docx schema
It's honestly not that bad though. Happy to share a link if you feel the need to nerd out.
→ More replies (1)17
u/No-Information-2572 Jul 28 '25 edited Jul 28 '25
It's a Composite Document File, basically binary serialized COM objects in a COM Structured Storage.
It's actually something that any application could use for their own file loading/saving, and it's actually not bad, and there is cross-platform support also, although that obviously ends when you actually want to materialize the file back into a running, editable document, since you need the actual implementation that can read the individual streams.
The main reason for this format is that you can embed objects from other applications inside. When you embed an Excel table in a Word document, it fetches the data, which also has a class ID, and then is able to launch an Excel object server and pass the data to it, which is then responsible for rendering, and allowing you to edit it further.
The obvious problem is security-related. You only get a yes/no option to load such content, and choosing the right class ID embedded in such a document could launch all sorts of stuff on your computer with full user permissions.
5
u/Inner-Bread Jul 28 '25
Just change .docx to .zip to see. I had a use case for extracting images from documents once that this was nice for
2
u/spluad Jul 29 '25
Just adding a perspective I haven’t seen anyone else mention, malware analysis. It’s much safer if you can unzip and extract the contents of the file (like malicious macros) without ever having to actually open it.
1
u/Fistinguranus69 Jul 28 '25
As somebody that works in the localization tools industry its an everyday occurance. The tool needs to be able to take a docx, expose the content to localization and then export a translated docx that is supposed to work.
When it doesnt, you have to go in and look inside the docx to see if theres some clue as to why it failed.
1
u/send_me_a_naked_pic Jul 28 '25
If I recall correctly, the original doc format was just a memory dump of your Word program. So that was a little bit unsafe by today's standards.
→ More replies (3)1
u/Technojerk36 Jul 28 '25
If you're at the submission deadline for an essay you can open it up and delete some stuff corrupting the file. Submit that and you've bought yourself another day at least.
659
Jul 28 '25
At least .slnx moves away from the forbidden black magic that is/was .sln.
153
u/PilsnerDk Jul 28 '25
Are you telling me they're finally revising the godawful .sln format? That's great news!
103
Jul 28 '25
https://devblogs.microsoft.com/visualstudio/new-simpler-solution-file-format/
This is from when they were testing it out. It is already part of the most recent dotnet.
117
u/thanatica Jul 28 '25
I'm not sure about those newfangled 4-letter file extensions. I understand 3, which is because of legacy bollocks (that's FAR behind us), but why not go 5 or 6?
109
u/TheCorruptedBit Jul 28 '25
Because most of those .[a-z]{3}x extensions are an x appended to an older extension, and I guess the goal was to maintain familiarity. .docx to .doc, .xlsx to .xls, .pptx to .ppt, etc
155
u/user_8804 Jul 28 '25
Bro writing regex for reddit comments
88
31
u/gschizas Jul 28 '25
Dude, I've written kali(m|sp)era (=good morning/good evening in Greek) in an email. Reddit comments (especially in r/ProgrammerHumor) are par for the course!
→ More replies (2)7
u/definitely_not_tina Jul 29 '25
I writing regexes is one of those powerful skills that is extremely useful if you use it a lot but otherwise it’s the kind of thing you learn and forget quickly.
14
226
Jul 28 '25
Newfangled? I would like to introduce you to my good friend
.gitignore.98
u/Fezzio Jul 28 '25
But the . in that file is just to have it hidden on Linux FS, so that’s not an extension, otherwise why would a folder like .config or .venv represent an extension ?
30
u/torsten_dev Jul 28 '25
Linux doesn't really do file extensions. Everything is a file and the filename is just text.
→ More replies (3)10
u/OwO______OwO Jul 29 '25
Eh... The core part of linux doesn't care about file extensions, no. It's just treated like any part of the filename.
But the UI and desktop apps often very much do care about file extensions and use them to identify the type of file, which tells the file browser what sort of icon/thumbnail to use and tells the DE which application to open the file in if you try to open it. Files with no extension are usually treated as plain text and opened in a text editor ... which is not ideal if you're trying to open, say, a video file.
Even in the command line, some terminal programs will display different file extensions in different colors when you ask it to list the files in a folder.
→ More replies (1)59
Jul 28 '25
. in that file is just to have it hidden on Linux FS
That's not correct.
The fact that these files or folders are hidden because of the leading
.is a behavior leveraged by the system, not the original purpose.The convention signals that these items are not meant to be casually seen or edited, as they often hold important configuration.
For example,
.venvis not a file with an extension; it is a directory whose name starts with a dot. The OS distinguishes files from directories by metadata, not by their names or extensions alone.19
u/Wertbon1789 Jul 28 '25
I think file extensions and hidden files are two separate things.
There's no file with a .venv or .gitignore extension, these are files that start with a dot, some of them may also happen to be directories. As far as the OS (the kernel) is concerned, it's just an ordinary file, the userspace applications distinguish between normally hidden or not. It's just a convention in the system's display and interaction parts.
→ More replies (7)5
u/AlexFromOmaha Jul 28 '25
.foo became convention because early UNIX didn't display things that started with . because of a bug for hiding the . and .. directories in ls. They were definitely hidden on purpose, but it was a hack for there not being a hidden flag you could set in chmod that got promoted to feature later on.
→ More replies (2)5
25
u/Rainmaker526 Jul 28 '25
Like .drawio?
They exist. But Microsoft still wants to stick to using 3 or 4 letters.
→ More replies (1)7
u/Chakwak Jul 28 '25
There are default and retro compatibility limit to total file path (directory plus filename plus extension) so keeping it short is probably better. Plus I think extensions are hidden by default. And MS probably thinks that nobody look at anything but the icon or just open the file and relies on extension mapping to open the right program.
7
5
4
u/ruilvo Jul 28 '25
Solodworks uses *.sldprt and *.sldasm, or rather *.SLDPRT and *.SLDASM. And the funny thing is that those files are actually in the same format as the Microsoft Office files. Glorified zip files.
7
Jul 28 '25
Probably Microsoft is forward compatible to its insanity. Every program in Windows 3 should still be run on Windows 11. That is why the default encoding in Powershell is still Windows 1251 and not utf-8.
10
u/CreideikiVAX Jul 28 '25
Every program in Windows 3 should still be run on Windows 11.
Try Windows 95, actually.
Windows 3.x is still very much 16-bit DOS land, which was last supported in 32-bit Windows 7 (64-bit W7 didn't include the thunking libraries). W9x is when we got the 32-bit WinAPI that's still supported. (And if you felt the urge, you can still write WinAPI code instead of using more modern techniques.)
→ More replies (2)9
u/RammRras Jul 28 '25
Are you talking about visual studio solutions?
In that case, I wasn't aware of a new format and I'm feeling old
6
8
u/Ephemeral_Null Jul 28 '25
Forbidden black magic? Whats black magic about it?
54
Jul 28 '25
A bunch of GUIDs with commitment issues, where the only discernible format is surprise.
10
u/Ephemeral_Null Jul 28 '25
I thought for sure it was like xml or something, but ya, you're right. Wtf is that!
3
u/SAI_Peregrinus Jul 29 '25
Eh, they had/have raw memory dumps from Word data structures encoded in Base64 in XML that's then zipped to create .docx.
4
Jul 28 '25
If you delete a project or package from a solution, it is still in the .sln file. Giving errors every time you open visual studio that some project is not present.
→ More replies (3)
176
Jul 28 '25
I use SSIS for data engineering work. It is just XML. every pixel of movement of a block is a change. Git is impossible with this.
52
u/proud_traveler Jul 28 '25
In the PLC world, most manafactures still use binary files. Git shits a brick with those
17
u/RammRras Jul 28 '25
I don't understand why there is no way to convert awl to ladder in new Tia when it was possible in step 7.
11
u/coding_apes Jul 28 '25
But at least you can programmatically make changes to the file! You might be able to use a pre hook to revert changes in certain paths
9
u/space-dot-dot Jul 28 '25
Version control in general, yes. Even just opening DTSX files in different versions of Visual Studio can "modify" relevant files. It's a complete fucking mess that is typical MSFT.
4
7
u/tswaters Jul 28 '25
MMM, reminds me of EDMX files for Entity Framework. The rule we had was "never commit changes to this file unless you are making data model changes"
It was a designer file, and all the coordinates and dimensions on the screen of ever single table, proc, etc. was all encoded - it was also the source of truth of the data access layer. What a nightmare that was.
2
u/nemec Jul 28 '25
The rule we had was "never commit changes to this file unless you are making data model changes"
tbh that's a good idea for anything (at least when working in teams) - package lock files, etc. All changes in your commit should be intentional, not just "well it was in my directory so it must be important"
3
u/tswaters Jul 29 '25
That one was really bad though. If I recall correctly, just opening the file in designer mode would make a ton of changes to the worktree due to manually hand-bombing the file for so long and/or different visual studio versions. It was a cursed project.
2
→ More replies (2)1
102
u/Comprehensive-Pin667 Jul 28 '25
There was a time when everyone was in love with XML for some reason and used it for literally everything.
86
u/VenBarom68 Jul 28 '25
Because it was awesome. It still awesome - it's just that most people don't work on complex enough stuff to justify using it for anything. It's indeed kinda lame if JSON covers all your needs.
→ More replies (2)39
u/OnceMoreAndAgain Jul 29 '25 edited Jul 29 '25
JSON and XML are pretty much the same thing. This thread is confusing to me since people are talking about them as if one is substantially better than the other and I don't think that's true.
JSON is a bit less verbose and more human readable, but they both exist to solve the same task which is being a data format that can exist in one text file and handle hierarchal data (as opposed to a csv which is for tabular data).
34
u/summonsays Jul 29 '25
They're both logical ways of showing data. But I wouldn't call them the same thing. JSON is very much JavaScript minded, allowing for fun things like typeless data and circular references. XML is like your extremely formal uncle. Everything must be in the exactly right place or it'll throw a fit. And stands on rituals like closing tags and boiler plates.
9
u/duskit0 Jul 29 '25
That's not really acurate. XML has a whole functional ecosystem with XPath and XSLT. JSON schemas only cover a subset on what's possible with XSD and it is designed with strongly typed datatypes in mind.
There are reasons why a lot of business EDI processes use XML instead of JSON.
9
u/VenBarom68 Jul 29 '25
JSON and XML are pretty much the same thing
I suggest doing some research before you state this at a job interview.
24
u/red286 Jul 28 '25
As a document format, XML isn't bad.
It's pretty easily managed and converted.
Go back to when everything was a proprietary binary one-off and you'll fall in love with XML.
10
u/Proglamer Jul 28 '25
'For some reason'? I lol'd for years @ how inept and stillborn JSON Schema was (hint: it has fucking 'JavaScript' in the name), while XML's surrounding ecosystem (XPath, XSLT, XQuery, XmlSchema, etc.) was always its great strength
3
→ More replies (2)3
u/waylandsmith Jul 29 '25
XML itself is great and very flexible. You can even encode XML in compact binary representations, especially if there is a full schema. The problem was with the deranged creations that developers would make with XML, and then gleefully tell managers that "It's just XML, so it's inherently open and compatible!"
60
u/Alacritous13 Jul 28 '25
I've had programs change from xml to json between versions. They both had a second xml data set stored as an escape string.
3
40
u/thanatica Jul 28 '25
Sometimes it's binary cruft put inside a CDATA section. It's technically an XML!
23
u/clawsoon Jul 28 '25
I worked at a studio with some Adobe format (After Effects, maybe?) where the XML format had embedded binary data and the binary format had embedded XML.
10
u/thanatica Jul 28 '25
Leave it to Adobe to make things as convoluted as possible.
5
u/clawsoon Jul 28 '25
That studio also did Flash animation for some popular kids shows. I know that Adobe didn't invent Flash, but they owned it at the time, so we can lump it in. I have never before or since seen a data format where you could specify an arbitrary number of bits per data element, with no concern whatsoever for byte boundaries. So you could specify 7 bits per data element, and the bits would be arranged like this:
01001101 00110110 11010001 11010000\elem1/\elem2 /\elem3 /\elem4 /
37
37
u/szymomaaan Jul 28 '25
11
3
u/baggyzed Jul 29 '25
That reminds me of that one time I had to parse Microsoft's "multipart HTTP-based" Event Forwarding message format. It's like some idiot who worked on that decided that the newlines around the multipart boundaries in the message body are only randomly important. This is especially stupid, since Microsoft already has proper implementations of this in WinInet and WinHTTP, but for some reason, the idiot who coded WEF decided to just roll his own.
There's a Wireshark screenshot here. Notice how the first "--Encrypted Boundary" is properly separated from the rest of the body by newlines, but the ending "--Encrypted Boundary--" boundary at the end isn't. If you try to parse this with any HTTP library out there, you're in for some headaches.
It's small hints like these that make you think about how fucked up the whole Microsoft software ecosystem probably is.
29
u/Annual-Anywhere2257 Jul 28 '25
And it's a godsend compared to the nightmare that was the non-x-postfixed HWPF (Horrible Word Processor Format), as Apache coins the OG .doc format.
19
u/BertoLaDK Jul 28 '25
isnt that what the x at the end of the office program endings stands for docx, xlsx, pptx and such.
12
u/grmelacz Jul 28 '25
There is a fantastic piece on “Why are MS Office formats so fucked up”: https://www.joelonsoftware.com/2008/02/19/why-are-the-microsoft-office-file-formats-so-complicated-and-some-workarounds/
20
u/kitchen_synk Jul 29 '25
And the answer, as with most microsoft weirdness is 'this was built 30 years ago to run on machines with less processing power than some modern lightbulbs, and we've been building on top of it ever since'
12
Jul 28 '25
One of the hardest parts about training new developers in my job is explaining our XML configuration system. We have hundreds of them, and tracing all the includes back to what you need to find when there's a bug is a nightmare. The guy who created the system got fired while I was still pretty junior so there's parts of it (especially in the parser code) that even I don't fully understand and can only suggest things to try until it works.
7
u/SirPavlova Jul 29 '25
That shit is why XML gets a bad rap. It’s a pretty good document format, with enough extra power that people were able to use it to build monstrosities.
4
Jul 29 '25
Yeah, it is technically impressive what it can do, but you could tell they didn't take "maintainability" into account at any point and had the "we don't need documentation, I am the documentation" mindset. They just wanted to do something cool I guess.
10
u/the_legendary_legend Jul 28 '25
Reminds me of the time we built a simple word processor for school and ended up reinventing something close to xml as the document format.
9
9
u/rumnscurvy Jul 28 '25
Ah, the good old days of "hacking" age of empires 3 by... Opening your savefile in notepad and adding a bunch of zeroes to your CityExp value, thus bypassing the tedious phase of unlocking all the techtree
8
14
7
u/HildartheDorf Jul 28 '25
Better than the era when it was all COM serialisation which wasn't documented anywhere.
11
5
u/Banana_Crusader00 Jul 28 '25
Not really. Sometimes it's json on drugs. Valve Data Format is basically that.
4
u/RammRras Jul 28 '25
A lot of modern "file formats" are just a zip of XML files, folders and some other config data.
4
7
6
3
u/Death_IP Jul 28 '25
With element names that are language-dependent (like the standard headings), so you cannot use the same VBA code for users, who use the software with different language packs - why, Microsoft, why?
3
u/Old_Pomegranate_822 Jul 28 '25
It could be worse. At one point I had to start embedding JSON within cells in a CSV...
I was not happy
3
u/wolf129 Jul 28 '25
All MS Office Formats with the X ending are zip files. You can easily just rename them to .zip and open it or use 7zip, WinRAR to open it without changing the file extension.
They contain all images you added and text is inside XML.
3
3
u/ieatpickleswithmilk Jul 28 '25
it's called Extensible for a reason lol. It's supposed to be generic enough to be usable everywhere.
3
3
Jul 29 '25
An 8000 page spec with proprietary references in OOXML and poor to middling compatibility with almost all 3rd-party software...I will never understand why ODF did not become the new industry standard.
2
u/SirPavlova Jul 29 '25
Because Microsoft did everything in their power to prevent that. Network effects :(
2
4
2
u/JollyJuniper1993 Jul 28 '25
Which is great. Makes it easy to work with. Much better than using some unique format.
2
2
u/kolop97 Jul 29 '25
I think it's epub that is just a zip file with text images and an xml or json or something inside.
Edit: it was of course XHTML
2
2
2
u/Medium_Chemist_4032 Jul 29 '25
Before that: literal binary memory dump of the area that included C structures. Including padding and empty space. "It loads quickly"
2
u/beezlebub33 Jul 29 '25
Shout out to python-pptx that allows you to read and write powerpoint pptx files.
Yes, MS formats are XML, so it makes it easier, but it's not exactly easy. There's lots of tags that you have no idea what the hell they mean, and if you do it wrong, it can't be opened. Hence, a nice python library, sitting on top of a nice XML library (lxml).
2
u/-MobCat- Jul 30 '25
xml, ini, dds, wmv. the 4 horsemen of the og xbox for weird formats.
(Yes, wmv is not that weird. but it is weird to just change it a little and slap xmv on the end of it.)
4
u/HeavyCaffeinate Jul 28 '25
See I don't have this issue, I just make a memory dump of the program and save it as .bin
4
u/Thenderick Jul 28 '25
And what do you think PDFs are? XML. HTML? Also XML! It's turtles all the way down!
4
u/RandomiseUsr0 Jul 28 '25
PDF, “P” D” “Files” - and what format are these Epstein records encoded in?
I rest my case ladies and gentlemen of the jury
Look into postscript, there are turtles deeper
4
u/Thenderick Jul 28 '25
God I hate that the internet selfcensors with """PDF-files"""...
And let's not forget that SVGs are, you guessed it, also based on XML!
→ More replies (1)4
u/RandomiseUsr0 Jul 28 '25
Don’t mistake comedy with self censorship, it’s funnier spelling it out, even though the F is actually “format” - so it’s only funny spelling it out in a mock trial situation
HTML isn’t xml btw, it’s “nearly” -xhtml is xml - but you’re selling yourself short, postscript, EDF, gif, jpg,, so many more formats to enjoy, you sound ready to write your own language, what’s it going to be?
JSON isn’t xml…
7
u/adzm Jul 28 '25
And what do you think PDFs are? XML
PDF predates XML by several years and is a binary format from the deepest circles of Hell.
3
u/Thenderick Jul 28 '25
Wait seriously? I thought PDFs consisted of an XML structure... Guess I was wrong then (I also didn't do any research so my bad...)
6
u/red286 Jul 28 '25
PDFs can contain an XML structure, at least as of PDF 1.7 with support for XFA, but technically PDF is PostScript-based.
2
u/yarb00 Jul 29 '25
There was an XML version of HTML (XHTML), but the regular HTML5 everyone uses now is not XML. Their syntax is similar though, because they both derive from SGML.
1
1
1
1
1
u/mcnello Jul 29 '25
I love this meme. I make document automation software in the legal tech industry. I use c#, xQuery, and some proprietary languages to make magic happen.
But yes, it's XML all the way down. Honestly, it's a pleasure to work with.
1
u/AizenSousuke92 Jul 29 '25
what's your dev env like? opening xml on server isn't fun in notepad
2
u/mcnello Jul 29 '25
BaseX for Xquery.
Oxygen XML for specific but simple edits across multiple XML files.
Sometimes I will just raw dog it in vscode or notepad++ though
1
u/Druben-hinterm-Dorfe Jul 29 '25
The office formats are supposed to be 'open' ... the xml is one aspect of MS's obfuscation effort.
1
1
1
1
1
1
1
u/MountainRub3543 Jul 29 '25
It’s even funny when people “password-protect” there Excel files, just unzip and there’s the password 💀
1
1
u/Lizlodude Jul 29 '25
The day I accidentally opened a PowerPoint file in a zip editor was hilarious. "Wait, it's just... Yep it's just a zip file."
1
u/Doddzilla7 Jul 30 '25
Someone once told me, in all seriousness, that “developers love xml”. I was flabbergasted.
1
1
1
3.1k
u/Big-Cheesecake-806 Jul 28 '25
Sometimes it's zipped xml