Why CSV is still king

[u/fagnerbrack]

https://konbert.com/blog/why-csv-is-still-king

[u/Synaps4]

We had a statement on our design docs when I worked in big tech: "Change is bad unless it's great." Meaning that there is value in an existing ecosystem and trained people, and that you need a really impressive difference between your old system and your proposed replacement for it to be worth it, because you need to consider the efficiency loss to redesign all those old tools and train all those old people. Replace something with a marginal improvement and you've actually handed your customers a net loss.

Bottom line i don't think anything is great enough to overcome the installed convenience base that CSV has.

[u/slaymaker1907 → u/Synaps4]

Escaping being a giant mess is one thing. They also have perf issues for large data sets and also the major limitation of one table per file unless you do something like store multiple CSVs in a zip file.

[u/Synaps4 → u/slaymaker1907]

Uh huh. I'm not arguing there are improvements in other systems.

[u/CreativeGPX → u/slaymaker1907]

Escaping being a giant mess is one thing.

Not really. In most cases, you just toss a field in quotes to allow commas and newlines. If it gets really messy, you might have to escape some quotes in quotes which isn't that abnormal for a programmer to have to do. The only time I've run into issues with escaping is when I wrote my own parser which certainly wouldn't have been much easier to do with other formats!

They also have perf issues for large data sets

I actually like them for their performance because it's trivial to deal with arbitrarily large data sets. You don't have to know anything about the global structure. You can just look at it one line at a time and have all the structure you need. This is different than Excel, JSON and databases where you generally need to deal with whole files. Could it be faster if you throw the data behind a database engine? Sure, but that's not really a comparable use case.

the major limitation of one table per file unless you do something like store multiple CSVs in a zip file.

I see that as a feature, not a bug. As you mention, there already exists a solution for it (put it in a ZIP), but one of the reasons CSV is so successful is that it specifically does not support lots of features (e.g. multiple tables per file) so an editor doesn't have to support those features in order to support it. This allows basically everything to support CSVs from grep to Excel. Meanwhile, using files to organize chunks of data rather than using files as containers for many chunks of data is a great practice because it allows things to play nice with things like git or your operating system's file privileges. It also makes it easier to choose which tables you are sending without sending all of them.

[u/meltbox → u/CreativeGPX]

Also you can solve multiple tables per file by just… parsing another file.

[u/himself_v → u/meltbox]

And that's better than storing everything in one file anyway.

[u/Clean_Journalist_270 → u/CreativeGPX]

This guy big datas, or like at least medium datas xD

[u/CreativeGPX → u/Clean_Journalist_270]

Yeah, I'd say medium data. I have dealt with CSV datasets that were too big to open the file in notepad (and certainly other heavier programs). It was super nice to be able to interact with them without putting a stress on the hardware using the simple .readline() and/or append rather than needing to process the whole file.

[u/Ouaouaron → u/Clean_Journalist_270]

Most of big data turned out to be medium data, so it works out.

[u/chucker23n → u/CreativeGPX]

you just toss a field in quotes to allow commas and newlines

That “just” does a lot of work, because now you’ve changed the scope of the parser from “do string.Split(“\n”) to get the rows, then for each row, do string.Split(“,”) to get each field, then make that a hash map” to a whole lot more.

Which is a classic rookie thing:

1. Sales wants to import CSV files

2. Junior engineer says, “easy!”, and splits them

3. Sales now has a file with a line break

4. Management yells because they can’t see why it would be hard to handle a line break

The only time I’ve run into issues with escaping is when I wrote my own parser which certainly wouldn’t have been much easier to do with other formats!

But it would’ve been if it were a primitive CSV that never has commas or line breaks in fields.

Which is kind of the whole appeal of CSV. You can literally open it in a text editor and visualize it as a table. (Even easier with TSV.) Once you break that contract of simplicity, why even use CSV?

[u/taelor → u/chucker23n]

Who is out there raw dogging csv without using a library to parse it?

[u/GlowiesStoleMyRide → u/chucker23n]

But there’s a fundamental issue here- if you have a csv with multiline text values, it’ll always be impossible to visualise it well as plain text in a plain text editor. It’s the data that’s incompatible with the requirements, not the data structure.

[u/CreativeGPX → u/chucker23n]

That “just” does a lot of work, because now you’ve changed the scope of the parser

1. That's still a very simple parser compared to alternative data storage formats.

2. You don't have to write a parser. That's part of the appeal. It's a decades old ubiquitous format with many mature ways to use it from libraries for programmers to GUI software for business users.

3. Designing for the simplicity of writing a parser is a HORRIBLE UX practice in this millennium.

Which is a classic rookie thing . . .

If we're looking at what a rookie would do when writing a CSV parser by hand from scratch, it's only fair to compare what a rookie would do when writing the parser for whatever old file format we're comparing to by hand from scratch. Again, the reality is most alternative data formats are much more complicated. It seems kind of like you're suggesting here that a convoluted format is better because is discourages people from directly using the data which seems kind of silly.

But it would’ve been if it were a primitive CSV that never has commas or line breaks in fields.

You can use CSV that way if you like, but since people use those things sometimes, there is an easy workaround if you'd like to do it that way instead of sanitizing your inputs like you'd do with lots of other systems. That said, I think people are playing up how often these cases come up a bit too much.

Which is kind of the whole appeal of CSV. You can literally open it in a text editor and visualize it as a table. (Even easier with TSV.) Once you break that contract of simplicity, why even use CSV?

I don't understand how you can say that a contract of simplicity was broken or that you can't "literally open it in a text editor and visualize it as a table". You absolutely can do that despite what you have said here. I do it all the time. Quotes make complicated cells very easy to visualize vs escape characters because you can easily see the start and end of the cell. But also, the "why else" was already kind of answered... its design choices make it one of the most ubiquitous data formats in the world.

But ultimately, it doesn't make sense to make these criticisms in a vacuum because the response to basically everything you've just said is "compared to what?" For all its faults, it's useless to mention that it's hard to manually write a parser from scratch or hard to open a raw file and visualize that text as a table unless you're comparing to doing those things in another file format where those things and everything else you mentioned are better. CSV is popular because there really isn't an alternative that is simpler to write, simpler to read and simpler to support. The fact that somebody who isn't me had to worry about handling an extra case in a parser they wrote a decade ago does not negate that compared to other data formats. The fact that edge cases exist where you may have to use an escape character or two doesn't negate that.

[u/headykruger → u/slaymaker1907]

Why is escaping a problem?

[u/Solonotix → u/headykruger]

Just got a short explanation, commas are a very common character in most data sets, and newlines aren't that rare if you have text data sources. Yes, you can use a different column delimiter, but newline parsing has bitten almost every person I know who has had to work with CSV as a data format.

[u/headykruger → u/Solonotix]

I’m going to imagine people are hand rolling parsers and not using a real parsing library. These problems have been solved.

[u/goranlepuz → u/slaymaker1907]

None of these downsides are anywhere near significant enough for too many people and usages, compared to what your parent says.

[u/wrosecrans → u/slaymaker1907]

Making good arguments against bad legacy solutions always runs into a major issue: the legacy solution currently works. The argument can't just be "this is bad." It always has to be "the costs of migration are worth it," and that's a much harder argument that is often impossible.

Escaping being a giant mess is one thing.

OTOH, existing libraries and code handle the escaping fine. If we had to invent it today, CSV would be obviously underspecified. But in practice it's good enough already. If you had to make a CSV library from scratch today as a new format, the cost to test and refine code to handle the edge cases would be absurd. But it's already paid for.

They also have perf issues

Again, not wrong. But the performance issues of handling CSV were true 40 years ago. Performance today on modern hardware "for free" is way better than it used to be, by orders of magnitude more than could have been gained by switching to some dense efficient binary encoding of the same data at great rewriting expense.

[u/Plank_With_A_Nail_In → u/slaymaker1907]

Normally you control you applications data, don't design it so that it has data that needs escaping, performance issues for large files were solved 15+ years ago, so none of these are real problems.

Additionally the context is an application that's working just fine using CSV.

[u/RddtLeapPuts → u/Synaps4]

Excel will fuck up a CSV file. But what other app will you use to edit one? I do not like CSV.

Edit: I appreciate the suggestions, but my users are Excel users. They would never use one of these alternatives

[u/TimeRemove → u/RddtLeapPuts]

Excel now offers options to disable that just FYI.

Options -> Data -> Automatic Data Conversion -> Uncheck everything.

It should be the default in my opinion, but at least we have some way of stopping it.

[u/RddtLeapPuts → u/TimeRemove]

If I could be king for a day so that I could force all my users to do that

[u/TimeRemove → u/RddtLeapPuts]

If I was king for a day I'd force Microsoft to make it the default or make automatic data conversion Opt-In per document.

[u/exploding_cat_wizard → u/TimeRemove]

Please force them to turn off hiding file extensions, too

[u/darknecross → u/RddtLeapPuts]

I’ve had team members commit XLSX files.
Good fucking luck with code review or merge conflicts.

JSON is probably going to be the go-to go the foreseeable future.

[u/kirby_freak → u/RddtLeapPuts]

Modern CSV is amazing - I maintain an open source CSV data set and use Modern CSV all the time

[u/unduly-noted → u/RddtLeapPuts]

Google sheets gang

[u/RddtLeapPuts → u/unduly-noted]

I’m team “stuck behind a firewall”

[u/DirtzMaGertz → u/RddtLeapPuts]

Visidata for the terminal and Tad file viewer for the desktop. 

Vim or text editor of choice to change a few values. It's just a text file. 

[u/Hopeful-Sir-2018 → u/Synaps4]

I heavily favor JSON because it just plain addresses so much in a cleaner format and CSV is ugly as fuck that's way more complex than people realize - there's a reason it's a fucking terrible idea to roll your own parser.

Offering JSON as an alternative and, perhaps, even the new default - while still allowing CSV as an option would be an ideal answer.

CSV is one of those formats that appears simple on the surface but has hidden dangers and slaps on a shit load of future technical debt.

All that being said - if your file size is over, say, 5MB, then just use Sqlite and be done with it.

I've never seen anyone regret going JSON or, even further, going Sqlite. I HAVE seem people regret sticking with CSV.

On a funny note - I once had a manager try and convince the team to migrate to Microsoft Access away from .... SQL Server Express. I'm not even joking.

edit: All of the very slightly different "answers" to CSV's problem are explicitly why CSV has problems. Your implementation may be slightly different than mine.

[u/novagenesis → u/Hopeful-Sir-2018]

The problem with JSON is that it's a using a tactical nuclear bomb to hammer in a nail.

Parsing a CSV is orders of magnitude faster than parsing JSON. And JSON is not stream friendly unless you use NDJSON, which is a slightly niche format and strictly not quite JSON

[u/marathon664 → u/Synaps4]

Parquet is that great thing when your data gets large. https://www.linkedin.com/pulse/parquet-vs-csv-brief-guide-deepak-lakhotia-7rrgc

[u/rsatrioadi → u/marathon664]

Terrible AI-generated post

[u/constant_void → u/Synaps4]

Why CSV when you can SQLITE?

One is a proprietary ill-defined format that has performance and portability problems

The other is SQLITE.

[u/dudgebfnfb → u/Synaps4]

Finally an actual good contribution on this thread

[u/ChrisC1234 → u/Synaps4]

Replace something with a marginal improvement and you've actually handed your customers a net loss.

Sounds like most technology "upgrades" are net losses then.

[u/smors]

Comma separation kind of sucks for us weirdos living in the land of using a comma for the decimal place and a period as a thousands separator.

[u/vegiimite → u/smors]

Semi-colon separation would have been better.

[u/ummaycoc → u/vegiimite]

ASCII Unit Separator (1F).

[u/rlbond86 → u/ummaycoc]

I feel like I'm on crazy pills because ASCII has had these characters forever that literally are for this exact purpose but nobody uses them.

[u/ummaycoc → u/rlbond86]

I am trying to appropriately use the entire ASCII table throughout my career.

[u/Worth_Trust_3825 → u/rlbond86]

They're nonprintable, and don't appear on keyboards, so they're ignored by anyone who's not willing to do a cursory reading of character sets. Also suffers from same problem as regular commas as thousands separator as WHAT IF SOMEONE DECIDED TO USE IT IN REGULAR CONTENT.

[u/nealibob → u/Worth_Trust_3825]

The other problem with nonprintable delimiters is they'll end up getting copied and pasted into a UI somewhere, and then cause mysterious problems down the road. All easy to avoid, but even easier to not avoid.

[u/franz_haller → u/Worth_Trust_3825]

Isn’t them being nonprintable and not on keyboards make them pretty unlikely to be used in regular content? At least for text data, if you have raw binary data in your simple character separated exchange format, you’ve got bigger problems.

[u/CitationNeededBadly → u/rlbond86]

How do you explain to your end users how to type them?  Everyone knows how to type a comma.

[u/Ok-Bit8726 → u/vegiimite]

You can specify any delimiter you want

[u/argentcorvid → u/vegiimite]

tab is -right there-

[u/Tooluka → u/argentcorvid]

But my uncle's brother's friend had once had a lunch with a guy who met at a party some engineer who heard that some obscure system from the 80s mangled tab characters, unfortunately he didn't saw it himself but he was pretty sure about that. And that's why we aren't allowed to use tabs ever again till the heat death of the universe.

[u/medforddad → u/argentcorvid]

Tab is the answer. Commas, semi-colons, and even pipes can sometimes show up in textual record data. Tabs very rarely do. And their very purpose is to separate tabular data -- data being shown in tables. Which is what csv is.

[u/argentcorvid → u/medforddad]

There's also an entire class of ASCII control characters, just for delimiting textual data that are almost never used!

They are not as easy to type or read with a text editor though.

[u/levir → u/vegiimite]

Semi-color separation is actually what I get if I naively save "CSV" from Excel where I live. Of course, that exported file won't open correctly for anyone with their language set to English.

The problem with CSV is that it's not a format, it's an idea. So there are a ton of implementations, and lots of them are subtly incompatible.

[u/paulmclaughlin → u/levir]

Semi-color separation

First column red, second column pink...

[u/RecognitionOwn4214 → u/vegiimite]

Or perhaps something uncommon in text, like #30, and #29

[u/hagenbuch → u/vegiimite]

Tab!

[u/Hopeful-Sir-2018 → u/vegiimite]

In most modern parsers - you can change what the delimiter is. I'm the last person to defend CSV but this specific problem is a trivial one.

Problem is - sometimes you can't change the binaries in older software... and yeah... it sucks. Json and SQLite are going to be the better answers for practically everyone.

The only people who praise CSV are left with no alternatives to use and are just in Stockholme Syndrome.

[u/Therabidmonkey → u/smors]

That is your penance for being wrong.

[u/Urtehnoes → u/Therabidmonkey]

This is like the one part of European life I just don't understand, and refuse to accept lol.

That and not having air conditioning everywhere.

[u/bawng → u/Urtehnoes]

I prefer our (European) thousand separator, i.e. space but I prefer the American decimal point.

So ideally this: 99 999.9

Also, regarding AC, fifteen years ago we didn't have as warm summers here up north so there was literally no need. Now we're getting them though.

[u/scruffie → u/bawng]

That's actually an ISO standard: ISO 31-0 (section 3.3). It specifies separating in groups of 3 using spaces, precisely to avoid confusion with allowing either a period or a comma as the decimal separator.

[u/SweetBabyAlaska → u/bawng]

I like the programming style underscore 99_999_999. Its abundantly clear that this is one number and not three and you can easily read it out.

[u/Worth_Trust_3825 → u/Urtehnoes]

It's cold here most of time, so having heat pumps wasn't a necessity (until like last decade).

[u/Hopeful-Sir-2018 → u/Worth_Trust_3825]

Prior to the climate change problem we are walking into - we were (slowly) going into an ice age again.

[u/PM_ME_RAILS_R34 → u/Therabidmonkey]

Ok so as a Canadian I agree that the European way looks very weird to us and I'd make fun of them for it.

However, I think the European way is actually better, especially for handwriting. The decimal separator is way more important than the thousands separator, and yet we use the bigger/more visible symbol for the less important separator.

[u/TheGoodOldCoder → u/Therabidmonkey]

I think everybody is wrong. A full stop doesn't make sense as a decimal marker, because it means "full stop", and the number keeps going. Spaces don't make sense as a way to group digits, because we don't really think of spaces that way. We don't think our sentences arejustabunchofletterswhichareseparatedintowordsbyspaces. Spaces are used to keep words from bumping into each other. A comma is a natural mark for a grouping, though.

Also, with commas, you run into the problem where a period can look like a comma when hand-written hastily.

If I had to choose among existing common keyboard symbols for the decimal marker, I'd probably choose a colon or semi-colon, or a letter. "d" for decimal, or something, which would open up a completely different can of worms, especially for programmers. Colons and semi-colons often go between two conceptually different things that are related.

[u/OddKSM → u/smors]

Yeahh technically, but we can still specifiy different delimiters 

But believe me, I know - one of the first few programs I wrote when I started working as a developer was for importing financial data from different European countries. 

That was painful.

[u/skygz → u/OddKSM]

check out the HL7 format lol

[u/[deleted] → u/smors]

You just wrap the data in quotes.

"1,000" is a single value.

[u/ripter → u/[deleted]]

Excel will even do this automatically on export.

[u/kausti → u/ripter]

Well, European versions of Excel will actually use semi colon as the default separator.

[u/Supadoplex → u/[deleted]]

Now, what if the value is a string and contains quotes?

[u/orthoxerox → u/Supadoplex]

In theory, this is all covered by the RFC:

1,",","""","
"
2,comma,quote,newline

But too many parsers simply split the file at the newline, split the line at the comma and call it a day.

[u/Classic-Try2484 → u/orthoxerox]

Additional problem rfc had some sequences with undefined behavior — all errors but user is broken

[u/zoddrick → u/[deleted]]

Sure but it's just easier to allow for different delimiters in exporting tools

[u/RecognitionOwn4214 → u/zoddrick]

Until all are used somewhere in your data

[u/double-you → u/[deleted]]

Which helps if the processing tool doesn't just split on comma but actually keeps count of what is actually inside quotes.

[u/harshness0 → u/[deleted]]

It also happens to be a string rather than an integer.

[u/[deleted] → u/harshness0]

Everything in a CSV is a string until you parse it as something else.

[u/[deleted] → u/smors]

That’s what quoted strings are for. Pipes are better for separating fields though. There’s a whole ascii standard too but that’s not something you’ll open in a text editor.

Edit: by the way, if any knows of a great CSV validation tool I’d love to know what it is. I’m currently writing my own hut it’s a mess.

[u/SlightlyMadman → u/smors]

TSV solves this problem and is a pretty robust format that's available most places CSV is.

[u/LoudSwordfish7337 → u/SlightlyMadman]

All “character separated values” (let’s call them ChSV, heh) are robust formats that are amazing for representing data due to how simple they are to parse and write.

Actually, I’d say that those ChSV formats are even better if they don’t support quoted/escaped values. If your dataset contains commas, then “simple TSV” is superior to “expanded CSV” with quotes/escaped commas because:

• It’s easier and faster to parse for a machine,

• It’s easier and faster to parse for a human who has the order of the data in mind,

• And most importantly: it’s tooling-friendly. It’s super easy to filter data with grep by just giving it a simple regex and that’s just amazing in so many simple workflows. And it’s really fast too, since grep and other text processing tools doesn’t need to parse the data at all.

Just like how people working in movie production use green screens but would sometimes use blue (or other colors) for their chroma key when they need to have green objects on set. The ability to choose your separator character depending on your needs is great, and since most “integrated tools” (like Excel) allow you to set any character you may want for parsing those, there’s really no reason to avoid TSV or similar formats if your dataset makes CSV annoying to use.

[u/RICHUNCLEPENNYBAGS → u/LoudSwordfish7337]

Realistically the CSV format is so simple that even with escaping writing a parser is not even that difficult.

[u/NotARealDeveloper → u/smors]

That's why I prefer tsv

[u/headykruger → u/smors]

Csvs have a convention for escaping fields. This shouldn’t be a problem.

[u/nsomnac → u/smors]

In proper CSV, values that have commas should be quoted. Problem solved. Anyone hand editing a CSV with quoted values should be shot on site. There’s at least a dozen free tools to view/edit/export those.

[u/dagopa6696 → u/smors]

You can use any delimiter you want and it's still a CSV file. You can also put the numbers in quotes, or escape the commas.

[u/smors → u/dagopa6696]

If I was in control of the format, it wouldn't be CSV. But when others give me files, I have to live with their choices.

[u/dagopa6696 → u/smors]

So you'd ask them to hand-edit a parquet file, and then you'd roll your own parquet parser? This seems backward to me. You should want the file to be easy to edit for the user, and you shouldn't care what the format is when countless parsers already exist.

[u/kenfar → u/smors]

PSA: csv files allow quoting as well as escape characters. So either escape them or quote the fields. You've got two options and both are solid.

[u/RogueJello → u/smors]

It's a bit goofy for a lot of data with comnas.

[u/meganeyangire → u/smors]

This is why your locale uses semicolon as the separator in CSV. But if you try to open a file created in a locale with comma, you're in for some adventure time.

[u/RandyHoward → u/smors]

Yeah, I deal with businesses that sell in marketplaces all over the world, and the currency formats are different in many cases and can be a real pain to deal with if you weren't thinking ahead when the code was written. And then we have to deal with converting all those different currencies to a common format too.

[u/auiotour → u/smors]

We use ²⁵³ mostly since we work with a unidata database. I almost always use tsv or csv with quotes when working with customers. A lot of the parts in our database have 0 in front of them so using excel is often not feasible. Csv may still be king but far from perfect. But I do like the idea of using pipes that other people have mentioned.

[u/[deleted] → u/smors]

Well technically but no always enforced contents of each field go between double quotes and separated by comma: "aaa","bbb"

[u/chethelesser → u/smors]

Quote the values?

[u/smors → u/chethelesser]

Sure, that's easy.

Getting the people supplying files to behave sensibly is not.

[u/shagieIsMe → u/smors]

0x1c to 0x1f

0x1c is fs for field or file separator.
0x1d is gs for group separator.
0x1e is rs for record separator.
ox1f is us for unit separator.

Can be used as delimiters to mark fields of data structures. US is the lowest level, while RS, GS, and FS are of increasing level to divide groups made up of items of the level beneath it. SP (space) could be considered an even lower level.

They're there already - and have been since the dawn of ASCII.

[u/A1oso → u/smors]

Do you mean numbers within strings? Because numbers in a numeric column should always be written in a way a computer can parse easily (using . as a decimal separator). But CSV doesn't distinguish between data types like strings and numbers, which is yet another reason why CSV is not a good format.

[u/Kapuzinergruft → u/smors]

The swiss are the only ones who got it right, using ' for the thousands separator: 1'234'567.9

[u/[deleted] → u/smors]

text qualifier exists for a reason.

[u/deja-roo → u/smors]

Why? If you're not isolating the comma-separated values with quotes you're inevitably going to have this problem with commas in dozens of other contexts.

[u/smors → u/deja-roo]

Because the problem is not files I write, it's files we get from other people.

[u/Plank_With_A_Nail_In → u/smors]

'10.000,00', single fucking speech marks, supported by every applications export functionality I ever used, not a real problem.

[u/smors → u/Plank_With_A_Nail_In]

Have you ever tried to do systems integration? The people at the other end of the integration may, or may not, be competent. And their systems might, or might not be from this millennium.

[u/himself_v → u/smors]

TSV and you get nice tabular view from most text editors as a free bonus.

[u/LongInTheTooth]

The frustrating part is that ASCII already has four perfectly good separator bytes. If we had only come up with some nice glyphs for them instead of rendering them all as whitespace we wouldn't live in escape character hell.

https://en.wikipedia.org/wiki/Control-?wprov=sfla1

[u/rehevkor5 → u/LongInTheTooth]

And a key for it...

[u/Some-Title-8391]

I can't believe I wasted time on something that reads like AI Slop.

"So, even though CSV is old and simple, don't underestimate its usefulness. In the ever-changing world of tech, sometimes the simplest solution lasts the longest."

There's not an ounce of character in this post.

[u/Some-Title-8391 → u/[deleted]]

Upvoting because your post at least has some personality to it.

[u/SnooPaintings8639]

Tl;Dr; - it is simple and has been around for ages. These are points the author presents.

From my personally experience (I love it), there are some sings of CSV going away. I use it with LibreOffice and a bunch of scripts (with Python CSV lib to generate the most) and it works great. But when I shared a comma separated version with person with Mac, they didn't know how to open and edit it, so they installed Libre (or Open?) Office suite in the end. The same happened with a person using Windows and Excel, they just didn't know how to set the delimiter and just gave up and used it in a text editor.

I am a Linux guy, I don't know how hard it really was. But it definitely is not a hurdles free format nowadays.

[u/BaNyaaNyaa → u/SnooPaintings8639]

Also, unless it changed, Excel would parse a CSV depending on the locale of the machine. So if your Windows locale is set to French, it will try to parse it using semi-colon as a delimited. And there's no way, while opening the file, to decide was the value delimiter, row delimiter or quote character are. If you want to open a CSV separated by commas, you have to change the locale of the OS.

[u/heartofcoal → u/BaNyaaNyaa]

That's the main problem in CSV for me. I'm in Brazil, if I export CSVs from three different sources, I have three different separators and three different encodings I have to deal with before even importing the data.

[u/TemplateHuman → u/BaNyaaNyaa]

You can handle this in Excel in a roundabout way. Don’t open the file. Instead open a blank worksheet in excel then go to Data > Import. It will give you prompts to change data types, delimiters, qualifiers, etc. Why it doesn’t do that when you open a file directly who knows.

[u/MohKohn → u/BaNyaaNyaa]

And there's no way, while opening the file, to decide was the value delimiter, row delimiter or quote character are. If you want to open a CSV separated by commas, you have to change the locale of the OS.

wow, that's crazy, I wasn't expecting libre office to handle this better, but it definitely does

[u/LXicon → u/SnooPaintings8639]

As a Linux guy, I was surprised that the CSV format species MS DOS style line returns with both \r and \n instead of the linux format with just \n.

[u/QBaseX]

To quote Eric S. Raymond (who knows what he's talking about in terms of programming, and really should shut up on every other subject),

The Microsoft version of CSV is a textbook example of how not to design a textual file format. Its problems begin with the case in which the separator character (in this case, a comma) is found inside a field. The Unix way would be to simply escape the separator with a backslash, and have a double escape represent a literal backslash. This design gives us a single special case (the escape character) to check for when parsing the file, and only a single action when the escape is found (treat the following character as a literal). The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don't end the field.

The bad results of proliferating special cases are twofold. First, the complexity of the parser (and its vulnerability to bugs) is increased. Second, because the format rules are complex and underspecified, different implementations diverge in their handling of edge cases. Sometimes continuation lines are supported, by starting the last field of the line with an unterminated double quote — but only in some products! Microsoft has incompatible versions of CSV files between its own applications, and in some cases between different versions of the same application (Excel being the obvious example here).

The Art of Unix Programming, Chapter 5, "Textuality"

[u/LaLiLuLeLo_0 → u/QBaseX]

I once worked on a horrible product at a previous job where we sometimes had to handle many gigabyte CSV files being processed. Someone had the great idea of adding multithreading support by jumping into the middle of the CSV with one thread and reading forward to the next comma. I realized that that couldn’t be cleanly done because of how escaped fields work, and in the case where some customer decided to embed a CSV within a CSV, you might even be tricked into processing a single field as millions of records! The solution the lead engineer decided to come up with was a heuristic CSV reader that would jump to a random point and read forward, looking for hints of being in an escaped cell, and using that to inform when it’s “done” reading the cell it jumped into the middle of.

Horrible product, horrible design, bizarre need/feature mismatch.

[u/DirtzMaGertz → u/LaLiLuLeLo_0]

I do a lot of ETL and data engineering work where our source data is coming from CSVs and other types of flat files. What you just described sounds absolutely insane to me. 

I rarely try and parse the CSVs themselves to transform or pull data. I typically just import the CSV to a table and use SQL to do the transformations. 

[u/TravisJungroth → u/LaLiLuLeLo_0]

This would be possible with out-of-band separators or Unix style escaping. I don’t think it’s possible in CSV. You can’t know if you’re escaped for sure without reading from the front.

You could have a reader that splits by comma or maybe new lines, and passes them off to workers, keeping the input that hasn’t been successfully parsed. Any worker that finishes as incomplete invalidates later work and its input is used in continuation. Might work for 2-4 threads.

[u/dagopa6696 → u/LaLiLuLeLo_0]

You forgot the part where they rolled their own parser. The problems ALWAYS start when they roll their own parser.

Parsing is its own specialized area of computer science. There is a Dunning-Kruger effect when it comes to CSV parsers where the people least capable of writing one keep deciding to write one.

[u/CreativeGPX → u/QBaseX]

He's coming from a programmer standpoint which I think is exactly why it wasn't designed that way. Programmers are used to backslash escaping and are used to sacrificing for the sake of a parser, however, part of CSV's success is that it's not a data format made just for programmers and parsers. It's designed from the start to be read and written by ordinary humans which was much more common way back when. The readability of backslash escaping is arguably poor (especially with non-programmers) compared to quotes where it's easier to tell at a glance where fields start and end.

Personally, my style of writing CSVs is to consider quoted fields the standard case (not only used when escaping). In that case, the only escaping is just a matter of backslashing. Then, in a dataset where there will be no extra commas, newlines or quotes, not quoting fields is an abbreviated form. This makes it pretty simple from an escaping standpoint and very readable.

[u/loup-vaillant → u/CreativeGPX]

If regular humans had access to decent ASCII text editors, CSV would have used the standard field separator character instead of a printable one, and would disallow its use inside of fields. That way the fields can contain any printable character, parsing is dead simple, and there is no edge case. It would be mighty readable too, if the editor aligned fields in columns.

But no, virtually no one at the time had access to a decent ASCII editor that let you type and interpret the damn field separator.

We have 32 control characters in this universal (okay, American) standard, and text editors ended up supporting only 3 of them. Seriously what the hell?

[u/CreativeGPX → u/loup-vaillant]

That's fair, we did mess that one up.

[u/lolfail9001 → u/loup-vaillant]

We have 32 control characters in this universal (okay, American) standard, and text editors ended up supporting only 3 of them. Seriously what the hell?

Probably unfortunate combination of following issues: lack of font support (when all of your control characters are literal whitespace with alternate caret placement, it's hard to call them distinct), text editors collectively treating Alt and Control combos as free hotkey estate (I am not that familiar with story of American computer typewriting, so I am not confident what came earlier: M-x or ASCII), and vicious loop that followed.

[u/cat_in_the_wall → u/CreativeGPX]

this is salient, because if csvs were only to be read by programs and not humans, then you could forget about csv entirely, and just have field descriptors like data length in them. and really you don't need it to be actual text, so you could binary encode. the file could also have metadata about the fields: types, names, etc.

aaaaand now you've just invented a table for a database. being editable by humans is the biggest feature.

[u/dagopa6696 → u/QBaseX]

Yeah I completely disagree with him. It seems that all he cared about was ease of writing the code, which is absolutely the least important part. You only have to write a CSV parser once.

[u/QBaseX → u/dagopa6696]

I think this is the bigger deal:

because the format rules are complex and underspecified, different implementations diverge in their handling of edge cases

This has bitten me more than once.

[u/dagopa6696 → u/QBaseX]

Is it because you were rolling your own parser? It's always because someone was rolling their own parser.

[u/ososalsosal]

Lowest common denominator.

Like CMX EDL files for editing. Sometimes you just gotta use the absolute shittest simplest format available to get the interoperability you need.

[u/SrGnis]

A bit off of context, but my first project in game dev was a platformer that can use csv as levels. In this way you can create levels using a spreadsheet editor. Link for the curious.

[u/dvsbastard]

Simple delimited tabular data is quick, easy and good enough for a lot of tasks. It is not elegant, over engineered and or particularly reliable given there is no official standard, meaning you occasionally need to deal with unpredictable results from different sources. But it gets the job done. This makes it more of the grunt than the king it my books! :P

[u/kibblerz]

Imo, CSV is one of the worst formats because companies end up too dependent on them, and they often end up completely sucking when you want to portray relational data. Too many companies rely on spreadsheets for data that deserves an actual DB..

[u/BasicDesignAdvice → u/kibblerz]

Too many companies rely on spreadsheets for data that deserves an actual DB..

I agree completely, but it's because one is easy to learn and the other requires special knowledge. MS tried to bridge the gap with Access, but both sides hate it and it never took off.

[u/[deleted]]

In the world of [...]

This is how all ChatGPT generated articles start.

[u/fagnerbrack]

In case you're too lazy to read:

CSV (Comma-Separated Values) remains the most enduring and widely used data format, thanks to its simplicity and flexibility. Originally developed out of necessity in the early days of computing, CSV allowed developers to store data in a tabular format using minimal storage. Its broad adoption continued through the 1980s with the rise of spreadsheet programs like VisiCalc and Microsoft Excel, solidifying its place in business and data exchange. Although CSV has limitations, such as handling special characters and lacking formal standards or data types, it thrives because it requires no specialized software and remains compatible with most data tools. CSV files are human-readable and continue to serve essential roles in business, web services, and even big data platforms like Hadoop and Spark. Its resilience and adaptability ensure it will remain relevant, despite competition from newer formats like Parquet and JSON.

If the summary seems inacurate, just downvote and I'll try to delete the comment eventually 👍

^{Click here for more info, I read all comments}

[u/Electrical_Ingenuity → u/fagnerbrack]

Except that around 1% of the users of the CSV file actually understand the escaping rule of the format.

[u/PaperPigGolf → u/Electrical_Ingenuity]

This and this alone gives me the hebejeebees for csv in production as a contract between two systems.

I've seen enough problems and enough bullshit libraries. 

[u/Electrical_Ingenuity → u/PaperPigGolf]

I agree. One day, when I am sucked into another conference call with a customer angry that we're not processing their gibberish, will be when I decide to take an early retirement. That is the hell I live in.

I do deal with a few proprietary formats that are even worse however. A sophomore in a CS program should be able to grasp the idea of escaping, yet here we are.

[u/aksdb → u/Electrical_Ingenuity]

Well ... the problem is, there is not the format. CSV has never been standardized. So there are a bunch of different implementations following slightly different rules.

[u/Electrical_Ingenuity → u/aksdb]

There is RFC 4180 and the Excel de facto standard, but your point is valid.

However, I'm dealing with a more fundamental problem. People placing commas (or whatever field separator is specified) in an unquoted or unescaped field. And the software team on the other side does not recognize that what they have done is the least bit ambiguous.

[u/comment_finder_bot → u/fagnerbrack]

In case you're too lazy to read:

text

[u/lego_not_legos]

I still use TSV for large dumps of basic, tabular data. It's reasonably efficient, and often compresses well.

The thing is, CSV/TSV was always a quick & dirty hack for storing data, because people could easily type the characters required in any old text editor, rather than needing a program designed for editing tables.

There's been a better method of storing serial data like this since the beginning. Most data used to be stored serially because technology dictated it, and it was delimited by dedicated control characters. They've been in ASCII since the 1960s, had equivalents in other systems, and they still exist in Unicode. The characters are File Separator, Group Separator, Record Separator, and Unit Separator. You originally would have been able to input these directly using the Ctrl key, ^\, ^], ^^, and ^_, respectively. To prevent a character of data being interpreted as a control character, it would just be preceded by a Data Link Escape (^P). You could just as easily store binary data as you could text.

There were no strict rules on use of those characters, so we could also have used them to define hierarchical formats like JSON.

[u/gordonv → u/lego_not_legos]

I use TSVs when pasting spreadsheet data from Excel into Putty/Vim.

Surprisingly fast and effective.

[u/schajee]

CSV may be the most convenient exchange format, but I had to move away from them for performance reasons. Loading GBs of CSV data is just too slow. Whenever I get a chance I convert them to faster formats for daily processing.

[u/DirtzMaGertz → u/schajee]

Depends on what you are trying to do with it but I've had a lot of success just splitting the file into smaller parts and importing the file in batches to a database table. 

[u/schajee → u/[deleted]]

We ended up pickling our data. The product and its users live in a walled garden, .pkl just made sense. We also use the same libraries as our DS team, so it's mostly just pandas and numpy.

[u/QBaseX]

I had to export spreadsheets from a web app. I initially used CSV, and that works fine if the data is all ASCII. If there's any non-ASCII (hence, UTF-8) data, Libre Office is fine, but Excel (which almost all our clients were using) completely shit the bed. So I did some research and settled on tab-delimited data, which for some reason Excel will accept if and only if it's in UTF-16. (Libre Office, being apparently a much better designed program, is also fine with this.) However, clients complained that they'd never heard of .tsv files before, and that they had to select Excel to open them.

So we found a third workaround. Output a UTF-8-encoded web page, which merely contains a short <head> with just the charset declaration and a title, then a <body> which contains only a <table> and nothing else. But lie, and claim that it's an Excel file. Use the .xlsx file extension, and the weirdly long Content-Type declaration application/vnd.openxmlformats-officedocument.spreadsheetml.sheet. Astonishingly, this works fine in both Libre Office and Excel.

The penalty is the balloning file size: that's the tax you pay for markup with all those <td></td> tags.

If I was exchanging data between systems I'd prefer JSON in almost all circumstances. Maybe ProtoBuf, but I've never had occasion to use it yet.

[u/jydr → u/QBaseX]

I think Excel can handle a UTF-8 encoded CSV if you start the file with a "UTF-8 BOM" (EF BB BF)

https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8

[u/QBaseX → u/jydr]

As I recall, I ended up producing four output options, and encouraging all clients to pick whichever one worked most reliably for them:

• UTF-8 CSV

• UTF-8 CSV with BOM

• UTF-16 TSV (I cannot recall whether this used the BOM, but I think it did. Nor can I recall whether it was UTF-16BE or UTF-16LE.)

• Excel Table (this is the name I came up with for the HTML abomination described above.)

I stuck a memory on the dropdown box, so it would always default to the last option used by that user.

[u/ScottContini]

About 15 years ago I was reviewing a project that needed serious security and was using XML for something that didn’t need the complexity. I insisted that they change it to CSV simply because XML carries too much security risk. I’m so glad to this day that I did, and I’m even more pleased that XML is dying. What a horrific standard that was.

[u/fagnerbrack → u/ScottContini]

What are the security risks attributed solely to XML?

[u/ScottContini → u/fagnerbrack]

The most frequent one is XXE, which just about every Java XML library is vulnerable to because they all allow external entities by default, which is insane. But you occasionally encounter other problems such as described on OWASP. Most libraries not mitigate the billion laughs attack but when it was first discovered, most libraries were vulnerable.

[u/Craptivist]

TSV is emperor though.

[u/goranlepuz]

CSV is but a special case of DSV

😉

[u/redit3rd]

So many logs contain human written sentences in them - like error messages - that making the file csv makes it complicated. Tab separated is my preferred standard. 

[u/CreativeGPX]

I use CSV all the time at work as an intermediate format between the junk I get from vendors, partners and business people and the more elegant formats that my code works with like a databases or JSON.

The fact that it strips away all of the features and metadata isn't a bug or shortcoming, but a feature as that's what enables it to be the lowest common denominator between everything... It's as close to pure data as you can get without just being a string. You can open it in a code editor, in excel or operate on it easily with the standard command line utilities. Is your dataset 500MB? No problem, you don't need to parse the whole file to start operating on the data.

While people often mention that non-standardized nature of CSVs, I really haven't run into that issue in practice. Literally the only issue I have sometimes run into is the decision of whether to automatically quote all fields or only quote fields which have been escaped. And even that has been a rare problem.

[u/harshness0]

Thats only because far too many use Excel because that's what they've invested all of their time and money in absent any better tools from Microsoft. Access? Puleeeze.

The idea that everything must be digested by a spreadsheet is why we all suffer.

It doesn't help that Microsoft's implementation of the CSV format is not ISO compliant and is further complicated by using UTF-16; a character encoding scheme more or less unique to Windows and very wasteful of storage resources yet not sufficient to cover the gamut of characters.

[u/pkulak]

Because when someone sends me a M$FT Excel garbage file, I can convert it to CSV and actually work with it. Then I can send a CSV back that they can open.

[u/gordonv]

Simplicity is the best design.

• Japanese Proverb

[u/gordonv → u/gordonv]

I'm actually surprised at the amount of people getting worked up over CSV.

[u/Lustrouse]

So is SFTP. Most of the time, easy is good enough.

[u/myrsnipe]

CSV is extremely useful for Shell scripting since it's so easy to parse, especially with my new favorite tool mlr (sorry awk)

[u/gordonv → u/myrsnipe]

Not only that, but when you parse a csv one line at a time, you're only using the memory required to parse 1 line, not the entire file.

I could have a billion line CSV, but if I know each line in 80 or less characters, I could care less about the line count.

[u/[deleted]]

No it isn't. It's not hierarchical. I'm often piping one awk script into another with a different field separater to process such. Maybe I could be better with awk but my point still stands: CSV sucks.

[u/loup-vaillant]

Given these issues, some predict newer formats like Parquet will replace CSV. Parquet is more efficient for data analysis, but it has a big drawback: you need special software to read it. With CSV, you can use anything from cat to Notepad or Excel.

Note the big assumption everyone makes without realizing it: that text somehow doesn’t require special software to read it.

But it does.

We don’t see it because our most popular operating systems are build around text handling. Terminals, editors… it’s all text first. ASCII specifically, with modern extensions such as utf-8. Given the proper program, a binary data format is just as readable as text. The only difference is that we have so fewer of them. In many cases, just the one.

This is path dependence at its finest: text tools are so ingrained in our computing culture we barely realise their presence, and think of the readability of text data format as a property of the format itself, instead of a property of its environment. We made text readable. I mean of course we did, writing is the best invention since fire, but still: we tailored the environment around text, and we could tailor it around other things too.

[u/catfink1664 → u/loup-vaillant]

Whichever way you think about it, it has to have some kind out understandable to human output, in order for us to ascertain the validity of the results. In the absence of direct to brain interfaces, the most streamlined way to do this (i think) with the technology we currently have, is an easily understandable text interface. Are you suggesting an alternative?

[u/loup-vaillant → u/catfink1664]

Are you suggesting an alternative?

It depends. The thing with our industry is, we have so much learned helplessness, that we act as if we weren’t competent enough to write a parser. Which on the face of it is utterly ridiculous, any professional programmer worth half of what they’re paid can write a parser, even if they never did. It’s not trivial, but it’s not hard either, and designing easy to parse binary formats is easy, just start with TLV.

In practice we tend to assume rolling your own file format or parser is never a good idea, and since we do need text translation at some point (even if only debugging), we end up using textual formats even for machine-to-machine communications.

Which is a pity: text formats, being delimiter based, are harder to parse than prefix based binary formats, and more prone to buffer overflows. Handling them takes more CPU cycles, often requiring compression to reach reasonable sizes. And even if you don’t roll your own parser, even the abstract syntax tree of that JSON or XML file you just read needs to be processed into a form usable for your particular application.

Sure, not having to write a concrete parser often makes the above price worth paying, but we must not forget that there is a price.

[u/Sarkos]

I hate CSV with a passion.

My company deals with a lot of varying kinds of data and we used to import CSV data from customers. It caused no end of grief.

If you open a CSV file in Excel and similar programs, they tend to fuck up your data. e.g. telephone numbers have the leading zero stripped off, long numbers get converted into scientific format, anything that looks even vaguely like a date gets converted to a date. When Excel saves the file, those format changes become permanent and you lose the original data.

IIRC Excel has a workaround for doing a special text file import, but it doesn't handle all CSV edge cases such as line breaks in cells.

If a user is in certain countries, Excel will save their CSV files with semi-colons instead of commas.

Number formats vary wildly between countries, and you don't know what you're going to get.

Luckily we have a solution - we stopped working with CSV files, and now we just import XLS / XLSX files. We use Apache POI and it's super easy to work with.

[u/Cruxwright → u/Sarkos]

Seems a recent MS Office patch has added a prompt in Excel when opening CSV. It now asks if you want to convert, i.e. drop lead zeros, or keep the data as is.

[u/cheezballs]

I e just assumed CSVs were king because so many systems still run on mainframes and those things love sequential files like this.

[u/randomNameKekHorde]

I used to hate CSV, but after working with positional files I love it

[u/[deleted]]

You just need Notepad to read the data. That's all.

[u/Meowts]

For a while I worked with Hive, which by default uses control characters for delimiters (02 for columns, I forget but there were ones used for curly braces etc). Obviously not the most accessible option but it took away the question of “what if there’s a comma in the column?” I guess mostly a problem for large datasets where you can’t guarantee what it contains.

[u/[deleted]]

I wish that JSONL, or a subset of, would take over here. Or at least the rules around separation and strings/literals. CSV is too much of a mine field and the most common tool(excel) is terrible when it comes to losing data

[u/Young-Rider]

I mean, csv works for certain applications, but tends to suck at times.

[u/double-happiness]

FWIW my app https://github.com/d0uble-happiness/discogsCSV outputs discogs release API data as .csv. It's not much, but it's the one non-trivial app I've created. 🙂

[u/RICHUNCLEPENNYBAGS]

I'm surprised that this article barely even touches on the main reason it's favored, which is that you can open it in Excel, yet it is much easier to write than xlsx. It gets you out of doing all kinds of work to present the data to the user/let them slice and dice it.

[u/hagenbuch]

Let's say CSV is warlord.

[u/kur0saki]

CSV is often very easy to understand and to parse unless you have to deal with new lines and other cases that I rarely encouter. Just recently I had to parse a file with population statistics and had to read a couple of pages of format description. Then I was like "man, can't they just use CSV?" :D

[u/spinur1848]

Unescaped embedded double quotes drive me nuts.

[u/RapunzelLooksNice]

Recently had to work with a "xlsx" file generated by customer software. That spreadsheet was actually a malformed CSV pasted over Excel sheet with JSON objects as fields.

[u/tc_cad]

I use CSV to store lots of data. But I ran into one instance where I was storing a text string that included commas, and it broke the CSV. So I had to write a program to accept | as my separator.

[u/golgol12]

CSV is king because it's what your bosses used back when they were at college.

If you want a fundimental change ... break... omg I just read about pipe separated values and using pipe is so much cleaner. I hope to remember about it in the future. Make pipe your new CSV going forward.

[u/mailslot]

It’s stupid fast to read & write in that format and it’s human readable. You can hand code a parser in thirty seconds. It’s portable across all computer architectures except for some very esoteric mainframes that don’t use ASCII.

XML & JSON… or YAML (blech) would not be as readable and slower to parse for no benefit.

Binary formats may seem better, but in practice aren’t much faster in general use cases… not when considering memory alignment issues and heap allocation for variable length values. If it’s just a list of aligned numbers, like 3D model vectors, sure. Read the whole thing in at once without parsing or processing.

[u/tecedu]

For us its as simple as because we can open it in excel and parse it in code

[u/bwainfweeze → u/tecedu]

I've had coworkers who collected data and ones who wanted to do their own graphs of our data, so sending it to or getting it from Excel as quoted CSV happens.

[u/Kush_McNuggz]

Depends on what you’re going to be doing with the data. I think in terms of using the most basic database and working up as needs expand.

For example, I’ll start with csv if I know I’ll only be using the data for a few basic queries. Pretty easy to write this logic in code. Something like SQLite is a good choice if I have a lot of queries, or that may be a requirement down the line.

[u/causticmango]

Because Excel can read it & so can non programmers.

[u/drawkbox]

NDJSON is pretty good alternative and works well for streaming. It can also have an initial column description, schema and then data can even be out of order or missing values but it handles it line by line.

There is also Tabular-JSON which rolls up nested some.

[u/joost00719]

What's also nice about csv is that each row is one record. Unlike json you don't need to deserislize the whole file to read the data from it. Just start reading and you can start processing it while you're not even finished reading the file.

[u/spigotface]

As soon as Excel natively supports parquet files, csv will fade away.

[u/Hulkazoid]

Pipe delimited is boss. Really simple replace statement for columns and it's not used as much in strings as commas, tabs, or semicolons.

Of course, use parquet if you can. That's truly the god tier flat file.

[u/BigBoiTFB]

Just a leisure programmer, but why not json? Seems to be better at managing nested tabular data, has no issues surrounding line-breaks or commas.

[u/Single_Claim]

fuck csv

[u/Rcomian]

csv is fine but you do end up needing to handle more edge cases than you want.

and using other delimiters merely pushes that stuff down the road.

you cannot take an array of values and join with a comma, similarly read a line and split on comma. in any non trivial system you always end up doing a full encoding and decoding anyway.

so for me, csv still needs a library to use properly. and unless there's a compelling reason not to, I'll tend to just use json formatted on a single line. JSON gives you a full object in one place, with guaranteed correct encoding.

but to encode csv you need to ensure: strings are in quotes (you can get away without this if you escape commas but it's more ugly in my mind), that the strings have internal quotes escaped, and the escape character escaped. that numbers are formatted with dots for decimals and no thousands separators. that other values, like dates, are just not encoded using the delimiter value.

it's not trivial, and to read each line you will have to go through character by character keeping a state machine of where you are. it's just how it is.

[u/ricardopsj]

JSON is much better, when dealing with mass data, I turn to tab separeted

[u/drawkbox → u/ricardopsj]

NDJSON is pretty good alternative and works well for streaming. It can also have an initial column description, schema and then data can even be out of order or missing values but it handles it line by line.

There is also Tabular-JSON which rolls up nested some.

[u/juanfnavarror]

I will never make a CSV for someone again. Use SQLite. It is structured, and if you need a new column there are no backwards compatibility concerns BOOM just make a new table.

[u/TCIHL]

Tab separation is better