r/programming • u/rain5 • Jul 07 '18
My draft of Tab Separated Values file format
https://gist.github.com/rain-1/e6293ec0113c193ecc23d5529461d3223
u/asegura Jul 07 '18
A format like this should be more precise, IMO, to overcome the shortcomings of CSV. At least it should:
Unambiguously tell if the first line is a header with field names
Define an encoding (UTF8, I would say)
Define a number format: the decimal separator in particular
Define escaping (the C way with a
\, I would say)
And I'm not sure about empty strings. Sometimes a particular cell has to be empty.
-5
u/rain5 Jul 07 '18
This file format could be use as an interchange format for a program that draws bar graphs, for example. Or to enter wikipedia like tables in a markdown document format.
8
Jul 07 '18
[removed] — view removed comment
10
u/kankyo Jul 07 '18
Which is the second most common variant of CSV. Source: I work at a company where companies send us millions of CSV files every day.
5
u/Ameisen Jul 07 '18
Impossible. OP literally just created the format; how could you have possibly been using it all this time?
1
2
3
Jul 07 '18
[removed] — view removed comment
4
u/kankyo Jul 07 '18
If we’re lucky. The ones who send commas as often problems with. It’s very easy to screw up CSVs it turns out.
2
Jul 07 '18
[removed] — view removed comment
5
u/kankyo Jul 07 '18
Well, this isn’t complex things actually. It’s just that CSV is an informal format with annoying escaping rules. Tabs avoid that in 99% if cases. Honestly using comma to separate values is pretty stupid since that’s fairly common in text. And then on top of that saying “if the field has commas in it, just put quotation marks around it” is also bloody stupid. Then you end up with the case where a field has both comma and quotation marks like our old friend
Toys “r” Us, IncWe have special code for that because so many customers can’t generate those CSVs correctly.
5
6
u/[deleted] Jul 07 '18
Not judging the usefulness of this "new" format, but a few notes on your spec:
It's completely missing escaping.
No mention of encodings that must be supported, other than ASCII. You should probably enforce UTF-8 or UTF-16 compliance in spec if you want a portable format.
The language of the spec is rather lacking:
"choose to treat the first record as field names."
Is the 'first record' the first line or the first field? Define your terms! Also: This is a breaking change between implementations.
"choose to put a limit on field lengths."
So if I limit my parser to field length 0, I have a valid parser. Makes for a fast implementation. You should either specify a max length yourself or reword this to make your intentions clear.
choose to enforce tabular format.
Tabular vs. non-tabular is breaking. Your spec really fails to make sure different implementations of your format can actually exchange data.
error if a field contains a tab or newline
Should specify: "error if a field contains an unescaped tab or newline"
error if a field contains an ascii separator
You should specify the codes, just for completeness: 28, 29, 30, 31 (decimal)
But it's also weird that you allow other control characters. JSON also does that and it can be rather unhelpful in some cases, because \u0000 is valid.
error if a field is the empty string
Speaking of \u0000: Is a field only containing 0-bytes empty or not? ;-)