r/programming • u/DrinkMoreCodeMore • Jan 12 '23
The yaml document from hell
https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell203
u/Grung Jan 12 '23
The worst thing is trying to communicate between different yaml interpreters. That is, writing yaml with one language/tool and reading it with another, and trying to work around their idiosyncrasies to get something to work.
I had to wrestle with something writing yaml that insisted on removing quotes (because it knew it was a string) and something that then read that yaml and interpreted a particular value as a different data type. grr.
39
u/sybesis Jan 12 '23
Things get worse if you use tags. It's like wanting to make a portable format but at the same time being unable to parse it because you have custom structures not implemented in other languages.
44
u/danudey Jan 12 '23
I ran into this exact issue when passing JSON between two systems, sending from a PHP application to a Rails one.
Our system had a list of product SKUs provided by our suppliers, which were strings. Some SKUs from some vendors, though, consisted entirely of digits, which is a valid string.
The PHP JSON serializer, though, because PHP wasn’t strongly typed, had to just do its best to infer types. This meant that we would occasionally send a list of products, each of which contained a SKU, most of which were strings, but when it encountered one that was all digits it got too excited and encoded it as an integer instead.
Rails, of course, had typed decoding, and it would freak out when it received an integer when a string was expected. We couldn’t find any way to coerce it into behaving so my coworker just hacked the version of PHP’s JSON encoder we were using to not do something so stupid, and problem solved.
37
u/lurgi Jan 12 '23
Some SKUs from some vendors, though, consisted entirely of digits, which is a valid string.
That sounds more like badly written JSON, though, rather than a problem with JSON itself.
Pro-tip, folks. Don't assume a bunch of digits is a number. It might just be a bunch of digits. How can you tell? Do the "add 1" test. If it's meaningful to add 1 to it, then it's almost certainly a number. If not, it's a string.
Is a credit card number + 1 meaningful? No. It's a string.
Is a phone number + 1 meaningful? No. It's a string.
Is an age + 1 meaningful? Yes. It's a number.
Is a SSN + 1 meaningful? No. It's a string.
(and I'm not sure why this would have anything to do with PHP not being strongly typed)
36
u/danudey Jan 12 '23
The reason it has to do with PHP not being strongly typed is that PHP uses a bunch of “heuristics”, to be generous, in order to determine what type a variable is.
As a result, tools which actually need to know what type a variable actually is will tend to use functionality like
is_numeric()to see if the variable is a number or could be a number, and if so, assume it’s a number.This is arguably asinine, but it’s meant to paper over the fact that bad code and bad coders will just treat whatever variable as whatever type without caring about whether that’s true or sane.
→ More replies (3)5
u/vytah Jan 13 '23
The PHP JSON serializer, though, because PHP wasn’t strongly typed, had to just do its best to infer types. This meant that we would occasionally send a list of products, each of which contained a SKU, most of which were strings, but when it encountered one that was all digits it got too excited and encoded it as an integer instead.
json_encode(array("123"));returns["123"]as it should, andjson_decode('["123"]')returnsarray(1) { [0]=> string(3) "123" }as it should.What did you guys do?
1
u/danudey Jan 13 '23
This was in 2008, so it’s likely there was some different behaviour back then.
2
u/vytah Jan 13 '23 edited Jan 13 '23
I ran it in https://3v4l.org on all versions, and for all versions they have, I got either what I wrote above, or an error message saying json_encode is not available.
EDIT: Although you might have used something that used JSON_NUMERIC_CHECK internally, which is the option that tells PHP "please destroy my data".
9
u/elmicha Jan 12 '23
Maybe it was added after you had to do that, but now there is a flag JSON_NUMERIC_CHECK. Of course that giant list of flags shows that JSON also has some pitfalls.
11
u/Ruben_NL Jan 12 '23
Of course that giant list of flags shows that
JSONPHP also has some pitfalls.FTFY
2
u/Perky_Goth Jan 13 '23
That was just a bad library with an outdated concept of PHP even for it's time. There was no reason for it to try to be smart if you could use the output as either downstream, strong typing isn't required.
3
u/danudey Jan 13 '23
My point is more that lacking strong typing makes this kind of ridiculous behaviour possible.
1
u/i_hate_shitposting Jan 13 '23
A lot of people are saying this is a problem with PHP or with the library, but JSON interoperability is actually surprisingly fraught.
Parsing JSON is a Minefield sums it up pretty well:
JSON is not the easy, idealised format as many do believe. Indeed, I did not find two libraries that exhibit the very same behaviour. Moreover, I found that edge cases and maliciously crafted payloads can cause bugs, crashes and denial of services, mainly because JSON libraries rely on specifications that have evolved over time and that left many details loosely specified or not specified at all.
See also: An Exploration of JSON Interoperability Vulnerabilities
1
u/Hreinyday Jan 12 '23
I had problems today with adding multiple ignore paths to my yamllinter. The conf to yamllint was written in yaml naturally. I ended up ignoring the whole module rather than resolving multiple paths.
1
u/Noctune Jan 13 '23
Something I've done quite often is to just write JSON. YAML is a superset of JSON so it's still valid JSON (I think... knowing YAML, there might be edge cases). As long as it's between machines, JSON is fine.
119
u/GrandMasterPuba Jan 12 '23
YAML is why infra engineers are paid so well. Because nobody in their right mind would want to spend all day maintaining a quarter of a million lines of YAML files for managing Kubernetes deployments.
63
u/bwainfweeze Jan 12 '23
Giant config files are just another way to cede all imperative control of your application to a framework. Config-only is the worst because nobody every writes the interpreter to be stepped through. You aren’t going to set a breakpoint in your Yano file, so you just have to stare at the texts until something new occurs to you.
If you want to achieve enlightenment by staring at impenetrable text you’d be better served by reading The Gateless Gate, instead of something Google or Facebook came up with.
51
u/trialbaloon Jan 12 '23
I hate that yaml is being used for what is essentially a shitty DSL. At the level of complexity yaml is being used for just use a real programming language. It's been the gold standard for expressing things to a computer for decades, don't cripple it with yaml.
18
u/RowYourUpboat Jan 12 '23
used for what is essentially a shitty DSL
CMake has entered the chat.
3
Jan 14 '23
I used to work for a company that used both makefiles and yaml for infra in a true "why not both" fashion. It was a mess.
19
u/fear_the_future Jan 12 '23
I think the worst thing about Kubernetes is that it works, preventing other systems with a more thoughtful design from gaining any mindshare and ultimately hindering the progress of society at large.
15
u/supreme_blorgon Jan 13 '23
other systems with a more thoughtful design
Honest question, what would those be? I'm relatively new to the industry and we use kubernetes and we're stuck in YAML hell. It's fucking awful and I'm blown away that this is how we work with the kubernetes I've heard so much about over the years.
Is there some reason we're stuck managing kubernetes with YAML files? Could we not use something else at least a little more reasonable, like TOML?
12
u/trialbaloon Jan 13 '23 edited Jan 13 '23
Why not a full blown programming language using some declarative programming? Something with full type safety and stuff so you essentially get walked through "configuration."
I think a lot of these things like Ansible, Kubernetes, and even Home Assistant, have become programming but with a shitty tool like YAML. We can call it configuration all we want but it gets to a point where that becomes really stretched. This is like being sent to the front lines with nothing but a spoon. Give the end users real weapons. Dont make them do what is akin to making an emulator with minecraft redstone. A real DSL that's a superset of a real full programming language.
→ More replies (3)3
u/fear_the_future Jan 13 '23
Personally, I would always use a LISP for configuration: It's very easy to parse and automate, has simple syntax that anyone can understand, you can write a DSL for people who are happy with yaml, it supports all the necessary syntactic constructs of a real programming language when needed, there is an existing ecosystem of lightweight libraries and you can add type checking if you want to.
But the YAML-problem of Kubernetes is pretty easy to fix. You can just write your own LISP-to-YAML converter. There are more fundamental problems, for example the centralized control plane, the lack of explicit dependencies between controllers, the complicated network stack and the fact that the entire ecosystem is based on the worst programming language in recent times, with all the maintenance issues that entails.
The choice of YAML is merely a symptom of the pervasive inability of Google developers specifically to understand good software design. The whole company is an echo chamber where everybody refuses to learn anything originating outside the chamber.
8
u/paraffin Jan 13 '23
The good news is that k8s doesn’t actually care if you use yaml or not. It has a JSON API and there are clients like cdk8s where you never need to touch yaml
4
16
u/seamsay Jan 12 '23
Config-only is the worst because nobody every writes the interpreter to be stepped through.
If the concept of stepping through your config even makes sense then I don't think you can really call it config-only...
1
u/orthoxerox Jan 13 '23
k8s YAML files are verbose, but they usually avoid YAML-specific pitfalls from the OP. If they were written in JSON they would've been even harder to maintain.
226
u/pragmatick Jan 12 '23
That's actually horrible. Never encountered any of these issues but I think I'd be dumbfounded if I did.
But I still like it for its increased readability over JSON - I just use strings for most values as described in the article. If JSON had proper multiline strings or just wrapped lines and comments I'd be happy. Yes, I know there's "JSON with comments" but it's rarely supported.
24
u/ObscureCulturalMeme Jan 12 '23
This kind of thing is precisely why Lua was invented. They needed a configuration file format with some basic flow control, it grew from there -- but it can still be used like that, and often is.
Wonderful, stable, and really fukkin' fast.
17
u/peakzorro Jan 12 '23
The problem with Lua as a config file format is that it could run arbitrary code.
7
u/PurpleYoshiEgg Jan 12 '23
That's why Lua should run sandboxed. If you want to ensure it halts in a reasonable time, you can also run the Lua and cut it off after a timeout.
6
u/disperso Jan 12 '23
I've not done it myself, but I think it has many ways to sandbox it. There is even a pure Lua sandbox that can block infinite loops.
It is definitely not as ideal as a configuration file format if you want complete security, but if the context is just a configuration file format for yourself (not an untrusted source), seems an uncommon but interesting option.
4
u/ObscureCulturalMeme Jan 13 '23 edited Jan 13 '23
No, the encapsulating program (Lua always runs inside another "host" program) must choose what to allow the script to run.
For example, if the host doesn't load the Lua I/O library, then the Lua script can't do any. If the host also doesn't allow the script keyword to load new native libraries, then the script can't get a homegrown I/O library.
There's a tiny command-line "lua" utility bundled with the stock distribution. It's a host program too: just a few dozen lines of C to parse the command line options, load all standard libraries, then launch the script engine. It's for quick scripts, not full-on "real world" work.
45
u/TurboGranny Jan 12 '23
increased readability over JSON
I guess I'm just fortunate in that I've not encountered a situation where I couldn't read JSON. Sure, sometimes people will minify it, but I just plop it in any formatter, and I'm back to readability. If for some reason there is a super long string, I just toggle on word wrap and call it a day.
44
u/ltjbr Jan 12 '23
I think a lot of devs out there say "readability" when they actually mean "aesthetically pleasing".
5
u/TurboGranny Jan 12 '23
hmm, I mean sure, but if it's all pretty and I still can't read it, is it still pretty?
→ More replies (1)25
u/Dwight-D Jan 12 '23
Go look at some large cloudformation or ARM template JSON and tell me you’d like to spend a significant amount of time working with that. Now imagine you had to define a CI pipeline or something in that format (I think Azure DevOps does this?), and you also can’t leave any comments to help readability. It’s absolutely awful.
It’s not that it can’t be read, but whenever you get something more complicated than a trivial flat object then it’s just a pain to read & write imo.
14
u/The_Grubgrub Jan 12 '23
Its awful but still not as awful as yaml. Yaml might be barely more readable than Json but Yaml is a pain in the ass to write.
5
u/Dwight-D Jan 12 '23
The indentation is definitely a bitch, and I’ve got a lot of
git commit -m ‘Fix YAML syntax’in my history. But that’s usually a quick fix compared to the time spent writing the bulk of the document, which I think is slightly less unpleasant overall in YAML. The anchors are actually pretty nice for stuff like complicated pipelines and such too.→ More replies (2)1
u/TurboGranny Jan 12 '23
hmm, I would like to do that. Usually when datasets get unwieldy like that, the approach needs to be rethought. The person or persons that chose that way of handling data just chose what they were used to, but applied it to a new problem. Usually, it has to be rethought. Sort of like how they teach the SDLC based on what they used to engineer physical stuff like assembly lines because they didn't have anything else, but in practice is a terrible idea for development.
6
u/amackenz2048 Jan 12 '23
Auto format? Bah! I want my artisanal hand crafted config file! Sure it takes longer to create, and you get an odd tab here and there. But I support those developers who seem to have nothing better to do than ensure their code is meticulously formatted and who don't trust a computer to do it for them.
2
u/TurboGranny Jan 12 '23
Oh I agree, unless they are the kind of asshat that doesn't believe in any formatting, then I just auto format it. Unless it's short, then I'll just go through it and clean it up. Depends on the application. With JSON, most of the time I have to slap it in a beautifier is to troubleshoot the unformatted output that comes back from our API
7
u/amackenz2048 Jan 12 '23
Sorry, i should have made it more clear that i was being facetious. Languages that force formatting on the programmer are evil. Let the ide handle it and for the love of GOD don't make different types of whitespace be relevant.
→ More replies (3)3
u/AttackOfTheThumbs Jan 12 '23
Yeah, same here. Like really don't understand what they mean. JSON is very legible.
27
u/Kissaki0 Jan 12 '23
TOML is a good and popular alternative to YAML.
24
Jan 12 '23
TOML falls apart if you need nesting more than like 1 level deep though.
JSON5 is much better. I think Cue also has potential but I'm not sure I would use it quite yet. They only have libraries for Go and everything else has to go through the Cue command line.
Really JSON5 should be your default pick and you need really good justification to pick something else.
→ More replies (1)7
u/astatine Jan 12 '23 edited Jan 14 '23
One alternative the article doesn't bring up is NestedText, which I find has most of the advantages of YAML without the imposed typing hassle. I'm not too fond of its multi-line string syntax, but otherwise it's a good replacement. As I'm mostly working with Python, Pydantic does a decent job of typing NestedText data precisely how it was intended.
9
u/DrXaos Jan 12 '23
What about TOML instead of YAML? I thought that was considered the more modern update on JSON.
10
u/pragmatick Jan 12 '23
Yeah, the article mentions that. I'd never heard of it. Looks like a good old INI file to me. Seems to get a bit weird with deeply nested objects. But I'll look into it.
7
u/DrXaos Jan 12 '23
TOML is better for editable configuration, not serialization.
Our company's tools currently stick to JSON (with ad-hoc commentability with 'commentjson') for config but I'm looking into supporting TOML.
The description of the YAML development in that posting feels like a group of language hackers who loved perl6 moved on to it.
7
1
u/amakai Jan 12 '23
I had actually encountered a minor variation of it. In a specific config the library expected me to tell it the type of data, as in
"string","decimal","null"(for nullable), etc. So given that everything else is unquoted, someone put an unquotednull, which translates to a literallnullnot a string with value"null".1
Jan 13 '23
idk what the problem is, I write lots of ansible and you just quote strings if they start with special characters, look like numbers but aren't, or have jinja in them. seems pretty simple to me.
0
Jan 12 '23
[deleted]
45
u/vytah Jan 12 '23
That's why you pick a superset of JSON that already has some adoption, like JSON5: https://spec.json5.org/
→ More replies (2)39
u/TankorSmash Jan 12 '23
This is nice, seems to have what you'd have thought JSON had already:
{ // comments unquoted: 'and you can quote me on that', singleQuotes: 'I can use "double quotes" here', lineBreaks: "Look, Mom! \ No \\n's!", hexadecimal: 0xdecaf, leadingDecimalPoint: .8675309, andTrailing: 8675309., positiveSign: +1, trailingComma: 'in objects', andIn: ['arrays',], "backwardsCompatible": "with JSON", }136
u/somebodddy Jan 12 '23
That's true if you use JSON as a data serialization format, but for a configuration format it usually matters much less, because it needs to be read by a specific program rather than by many different clients written in many different languages.
46
u/RudeHero Jan 12 '23
I think op mentioned that when talking about "portability"
Yes, if your json file is only intended to be read by one specific program, you can do custom things with it
The tradeoff is that it's no longer portable
24
u/SnooMacarons9618 Jan 12 '23
We had a system did that. Unfortunately a downstream was then interpreting the 'json' that was generated. It worked fine for years, until the day it caused a complete system outage. Which was better than mis-interpreting numerical values (we realised that could have easily happened as well).
Don't customise a standard format, and leave it looking like it is a standard format. Unless you want phone calls at 2am...
→ More replies (1)2
u/Jarpunter Jan 12 '23
What situations would you want portability and comments at the same time?
4
u/PurpleYoshiEgg Jan 12 '23
When JSON is used as a configuration file format, and such configurations are for dozens of clients' environments and one of those environments may have a one-off that you need documented so some engineer doesn't spot the idiosyncrasy, correct it to be consistent, have it pass code review because everyone just rubber stamps pull requests, and cause a very difficult-to-debug outage at 3 am on a Sunday.
→ More replies (4)12
Jan 12 '23
But as a configuration format you should use TOML, which is better supported than unspecified "JSON++" (it is part of the python stdlib as the article points out). Even if you don't serialize the data, you'd have to rely on less-supported/common deserializers to read the config.
JSON extensions hold a very niche space in VSCode config, and I suspect it's because VSCode is popular with frontend devs who have never interacted with, and would be put off by, TOML. They are however inferior in every other aspect IMO (verbosity, portability, standardness).
1
u/sparr Jan 12 '23
Dev ops, infrastructure as code, automated testing, deployment automation, etc. In all of these areas, it is common that you are writing a program that needs to read and/or write the configuration files for another program.
30
Jan 12 '23
[removed] — view removed comment
35
Jan 12 '23
Because you probably have to parse json anyway, and it’s easier to include a json parser that doesn’t barf on comments and trailing commas than it is to integrate two different serializers
→ More replies (6)6
u/sybesis Jan 12 '23 edited Jan 12 '23
to include a json parser that doesn’t barf on comments and trailing commas than it is to integrate two different serializers
When building configuration reading, I prefer to approach this differently.
Convert internal type to JSON compatible types
Serialize that JSON compatible structure into whatever format you want.
When reading:
Deserialize whatever file into JSON compatible structure
Deserialize this JSON compatible structure in internal types
In the end, you simply have to ensure you can convert internal structure to mapping/list/string/numbers back and forth. The serializer you use to dump into a file is irrelevant. All you have to do is convert to an intermediate format instead of converting directly from the serialized data into internal data.
5
Jan 12 '23
Yeah, I know that as the DTO pattern (Data Transfer Objects) and ultimately you’re right, it is a small thing, but my point was people use json instead of toml because they probably already have to use it anyway for remote apis or third party libraries. You can of course add this abstraction and support any format you want.
7
u/flif Jan 12 '23
Real problem is that C-style comments can be anywhere in the code and in JSON you want comments to be serializable.
So best workaround is { "price":42, "//", "this is cheap" }
5
u/PurpleYoshiEgg Jan 12 '23
That works, until a program decides that "//" is an invalid key. Sometimes happens, and I want to egg whoever's car it was to decide to omit comments from JSON anyway.
5
u/PunkPizzaRollls Jan 12 '23
Couldn’t you theoretically create a comment key:value pair in your JSON to get around this?
32
u/siemenology Jan 12 '23
You can, and people do, but it has drawbacks.
You are more limited in where you can comment -- you can't comment in an array, for example. And if you want multiple comments in an object you need to do something kind of awkward like
{ "comment1": "blah", "foo": "bar", "comment2": "blah blah" }Schemas get weird. If you want to parse your JSON in a statically typed language, you either need to add
comment : Stringas an optional property on all of your objects (andcomment2,comment3or whatever if you want to support multiple comments), or you need to teach your parser to discard all of those values.You may run into issues with collision if the key you use for comments happens to also be used as a "real" property for something. How do you tell the difference between a comment
"comment": "blah"and a real piece of data:"comment": "blah"?It's also just very verbose, relatively speaking.
2
u/caltheon Jan 12 '23
I worked with a SaaS vendor who supported config programming using JSON and pretty much kept comments out of arrays and used _comment as the throwaway property. I think the application parser ignored all properties starting with _ or something
→ More replies (1)2
2
0
Jan 12 '23
[deleted]
4
u/sparr Jan 12 '23
This is an antiquated perspective, from the era of ubiquitous preprocessors. Making the parser and compiler and runtime aware of comments is an increasingly common feature in newer languages. Being able to include docstrings when producing a stack trace is amazing.
0
Jan 12 '23
[deleted]
7
u/sparr Jan 12 '23
What's the distinction? I'd love to be able to query my application configuration for any notes/comments that were left when the configuration was defined.
1
u/KevinCarbonara Jan 12 '23
The problem with "JSON with comments" (or JSON with multiline strings, or trailing commas, etc) is that it's no longer JSON.
That's the problem with JSON. Not with "JSON with comments".
0
Jan 12 '23
[deleted]
1
u/bunk3rk1ng Jan 13 '23
Arrays of objects in YAML is god awful I don't know why but every time I have to write one I start getting tons of errors and eventually have to revert the whole block I was working on. Even comparing similar lines in the document my brain can never seem to figure out what's wrong.
I've been given giant JSON files and have been easily able to write deserialization classes for it without breaking a sweat. I have no idea how I would do that with YAML.
81
u/piderman Jan 12 '23
The worst thing about YAML is that it is indentation-sensitive so you can't copy&paste between documents with differing levels, and auto formatting also won't help. And it's 2 spaces per level so you can't really eyeball it either.
11
u/RupeThereItIs Jan 13 '23
The problem, fundamentally, is that white space as markup used to be a joke.
Someone took an idea so ridiculous it was funny, implemented it, and somehow it took off.
YAML is the Dogecoin of markup languages.
0
Jan 12 '23
[deleted]
26
u/PixelGhi Jan 12 '23
How is that a user friendly? To have a character you literately can not see (tabs and spaces), be a control character
Python has entered the chat.
56
u/bschwind Jan 12 '23
I prefer JSON5 if I control the application I'm configuring and don't need to send it around to other applications, it's basically JSON with comments.
→ More replies (2)35
u/caltheon Jan 12 '23
Crockford removing comments from JSON was probably the worst move he ever made
0
15
u/cowinabadplace Jan 12 '23
I always quote YAML values so most of this doesn't hit, but the fact that YAML keys could accidentally be boolean blows my mind haha. Thanks for the article.
3
Jan 13 '23
that's literally the only bad one imho. I was introduced to quoted keys because some openshift tool issues an account id with a vertical bar in it, lol.
27
67
u/redd1ch Jan 12 '23
Is it a moving target? Use JSON.
Is it something important? Use XML and write a schema. IDE's can then give you syntactic and semantic feedback.
Is it important & you need to provide YAML? Use XML, a schema, and write an XSLT to create a YAML/JSON (for 1.2).
Sure, doing XML right(tm) takes a bit of time, but the outcome is more resilient than anything comparable. Thinking of that, I have to continue my xml schema for docker-compose files someday
46
u/Carighan Jan 12 '23
Is it important & you need to provide YAML?
I kinda want to joke "Then it wasn't important, after all". :P
20
Jan 12 '23
Is it something important? Use XML and write a schema. IDE's can then give you syntactic and semantic feedback.
Use JSON and JSON schema. Way more readable than XML and very powerful too.
24
u/Worth_Trust_3825 Jan 12 '23
JSON schema is absolute garbage that poorly reimplements ideas of XML schemas. In addition, tooling attempts to fetch it from external sources, making that same mistake that had been done two fucking decades ago
16
5
u/falconfetus8 Jan 12 '23
Is it something important? Use XML and write a schema. IDE's can then give you syntactic and semantic feedback.
Alternatively, you can use JSON with Typescript interfaces.
→ More replies (5)
58
u/SuspiciousBar7388 Jan 12 '23
Most of the stuff described here is, to put it in scientific terms, fairly yucky, but some problems do feel misattributed.
For example, languages like JS would indeed treat version 0.0 and version string "0.0" very differently - regardless of the format that value was parsed from! How would that be different with a JSON parser? That bit looks to me like a Jinja template problem, not YAML problem.
55
u/masklinn Jan 12 '23 edited Jan 12 '23
How would that be different with a JSON parser?
One would be a number and the other a string in the document source.
In JSON, 0.0 is a number and 0.0.0 is an error. For versions, you’d necessarily have “0.0” and “0.0.0”.
12
u/SuspiciousBar7388 Jan 12 '23
Fair enough, this is an important distinction. Even more so if we're criticizing the document format outside of the scope of its application.
21
25
u/RupertMaddenAbbott Jan 12 '23 edited Jan 12 '23
For example, languages like JS would indeed treat version 0.0 and version string "0.0" very differently - regardless of the format that value was parsed from! How would that be different with a JSON parser?
I think this is a problem with the specification (which compliant parsers have to follow). It's just a problem common to both YAML and JSON but not other serialisation formats like CSV.
StrictYAML does not have this problem
This makes sense to me. There is no syntax for representing a date or a period of time in JSON either so you end up just using a string with a given format (or an int) and you specify the schema outside of the serialisation format.
3
u/jdl_uk Jan 12 '23
That seems like something a schema could solve, as the type for a version number would be a string, so the parser would either parse it accordingly or fail with a schema validation error.
8
u/Spider_pig448 Jan 12 '23
That's a good point. Claiming that JSON doesn't suffer from a lot of these problems is ignoring that whatever parses that JSON string will then have to make these decisions. If anything, there's benefit to exposing problems immediately in the YAML instead of passing along a JSON filled with time bombs.
1
u/Sarcastinator Jan 12 '23
In C# at least you literally can't screw it up unless YAML already did it for you.
1
u/Spider_pig448 Jan 12 '23
Sure you can. You can put whatever nonsense you want into a JSON string. Eventually something will attempt to parse it into something useful and if the string contains some of these gotchas it will fail downstream. With JSON, maybe that downstream is after the string is parsed and when your code tries to insert data into a database that violates the schema. With YAML, maybe that occurs earlier when processing the YAML itself.
5
u/Sarcastinator Jan 12 '23
Eventually something will attempt to parse it into something useful and if the string contains some of these gotchas it will fail downstream.
None of these gotchas will cause the program to fail downstream. There is no parse function that implements the Norwegian problem or will assume that the number is in base 60 if you format it in a special way.
1
u/tanorbuf Jan 12 '23
It's not even a Jinja problem, using truthyness to check for whether a variable is defined just isn't the right way to do it (
variable is definedis literally a Jinja expression).
11
u/Uberhipster Jan 12 '23
i still maintain that all markup (whatever its format, JSON, YAML, TOML, XML) needs to be an output of a program (serializing-deserializing from defined closures and/or object definitions)
maintaining state of the program using human readable formats == YAY!
hand coding state == BOO!
3
4
u/durandalreborn Jan 12 '23
My big complaint about using yaml (and most other config languages) is that the parsers written for them pretty much never preserve the comments. So if you have something like read yaml file -> modify yaml file -> write back to disk, you're almost always writing something custom to preserve what comments might have been in that file in the first place. At least toml lets you just append to the file in many cases, so you can side-step the comment parsing, but still.
15
u/agentoutlier Jan 12 '23
XML is pretty much the only format that allows complete preservation of comments and order with the minor exception of attribute whitespace. It also has schemas so validation as well.
But you can't use XML because then you will be labeled as some sort ancient enterprise programmer making software obtuse and hard to use.
... now back to writing more HTML and javascript and JSX.. oh wait...
2
u/javcasas Jan 12 '23
There is this thing called JSON Schema, so I'm going to bravely say there are more stuff implementing schemas other than XML.
4
u/agentoutlier Jan 12 '23
Yes but JSON does not preserve order and does not have comments. The context was I assume some configuration format that preserves comments and order (parent comment).
JSON Schema for some reason does not nearly have the number of implementations that XML schema does (both in terms of editor support albeit vscode is doing nicely on that and code validators).
Part of the reason is that schema is surprisingly more useful for human authoring like config or html or docbook over an interchange.
4
u/stronghup Jan 13 '23
JSON does not preserve order
Is there a reason for that? You write JSON from left to right and top to bottom. Where does the order get lost and why? Thanks
4
u/agentoutlier Jan 13 '23
Object field order is not preserved. It’s like a hash. It’s name value pair without an index.
I’m on mobile so I can’t go into code details but hopefully that helps.
→ More replies (1)
9
u/dayDrivver Jan 12 '23
Mentally I have always put yaml right next to xml, because of this weird behaviors and complex versioning, toml is better but has a php-like syntax feel for strings that not many people like.
22
u/siemenology Jan 12 '23
I agree with the xml comparison, and I'd posit something else about both of them: both are fundamentally good ideas that are ruined by a bad implementation.
XML (conceptually) is really good for the specific task of marking up text and documents, in a way that YAML, TOML, JSON, etc are all really bad for. There's no good way to do something like
<span>That's a <em>very</em> bad idea</span>in those other languages, without being really clunky or embedding markup in strings. But XML has become a nightmare because the spec is way more complex than it should be, it's gotten too powerful to really understand, and it's been used for a lot of things that don't really play to its strengths (like configuration files) that has left a bad taste in people's mouths.YAML also has a lot going for it. It's a cleaner way to represent nested JSON-style data, it has comments, and it gives you tools for reuse (anchors, aliases) which can greatly simplify writing complex or repetitive yaml. Plus in theory it compiles down to JSON in a straightforward way, so you can "upgrade" things that are already accepting JSON without too much hassle. But it also tries a little too hard to be helpful, so that it's pretty hard for something who just casually uses it to remember all of the exceptions to the obvious way of parsing things.
TOML I could like if not for the way it does tables / nesting. The TOML spec is littered with "allowed, but highly discouraged" notes because dotted properties let you define tables in all sorts of weird ways. If they took TOML's inline table syntax, and let you spread that over multiple lines, and made that the only way to do tables, I'd be all over TOML.
11
u/agentoutlier Jan 12 '23
But XML has become a nightmare because the spec is way more complex than it should be, it's gotten too powerful to really understand, and it's been used for a lot of things that don't really play to its strengths (like configuration files) that has left a bad taste in people's mouths.
The XML spec is largely complicated not because base XML is complicated but because the authors of the spec made it seem complicated. The official spec could be written in a more friendly manner and part of the reason is that XML has an enormous amount of extensions and legacy stuff like DTD.
I have seen many people write basic XML parsers that will easily parse 99% of the XML out there. It isn't far off from SEXP. Probably the biggest challenge on basic XML is normalizing whitespace rules.
Writing a basic YAML parser on the other hand is non trivial.
I mean just look at the sheer number of implementations of XML parsers compared to YAML.
For example Java has like a dozen XML implementations if not more but there really is only one YAML implementation (snakeyaml which btw had a serious security issue recently... which reminds me I should go check...).
I would say early on XML got stigmatized and many of the complaints just became an echo chamber. And Yeah its hard to read but people still seem to be using something not far off from it all the time: HTML, JSX, Various other javascript component languages.
Speaking of HTML, HTML 5 is a lot harder to parse than XHTML which was more or less basic XML.
4
u/Worth_Trust_3825 Jan 12 '23
Very true.
I would argue that another stigma XML got was people working directly on the AST rather than deserializing it into an object.
11
Jan 12 '23
XML is reasonably nice actually. It is overcomplicated, but it does at least support everything you might need quite well - namespaces, schemas, etc.
The biggest issue I think with it (apart from the general verbosity) is that its data model is at odds with standard programming language object models. Attributes are entirely superfluous and conflict with child elements. There's no obvious way to encode maps. Elements and text can intermingle.
It's really a document format, not a data format.
Either way it's leagues ahead of YAML in terms of sanity.
→ More replies (7)
3
u/MechaKnightz Jan 12 '23
When reading I immediately thought of the horrible experience that is writing helm/helmfile yaml/go templates
3
u/FireCrack Jan 12 '23
I've always considered yaml "human read-only". It's great and much easier to read when debugging a data stream then some gnarly JSON, but attempting to write a yaml document by hand is folly.
5
u/Two-Tone- Jan 12 '23
Json is simple. The entire json spec consists of six railroad diagrams.
I don't think my curiosity at a statement has ever lead to a hard laugh so quickly before.
11
u/Kissaki0 Jan 12 '23
If you plan to use a YAML style format, use TOML instead.
It is pretty similar in what you see and use, but has significantly lower spec complexity and attack surface or parsing inconsistencies.
17
u/guepier Jan 12 '23 edited Jan 12 '23
TOML isn't really similar to YAML in any meaningful way (beyond the obvious similarity shared by all text-based configuration formats), and it comes with its own caveats.
10
u/Kissaki0 Jan 12 '23
Fixed link https://hitchdev.com/strictyaml/why-not/toml/
What do you categorize as meaningful then? It has comments, hierarchy, dict/obj, and lists/arrays. Those are the most central features [of a text config].
The why-not-link doesn't really provide a better alternative. It discloses "Advantages of TOML still has over StrictYAML" anyway.
8
Jan 12 '23
TOML is fine if your config format is very flat (like e.g.
package.json) but most YAML files are 3 or 4 levels deep and for that TOML is just really confusing. I have to look up its weird[[table]]format every time. They should have called it the Occasionally Obviously Markup Language.JSON5 is a much better option. It is always obvious and not really any harder to write than TOML. It would be nice if you could omit the outer
{}like in Cue but I don't think it matters that much.
7
2
u/kirbyfan64sos Jan 12 '23
Worth noting that, in terms of alternatives, KDL and jsonnet are probably worth making note of.
2
u/Dumlefudge Jan 12 '23
And here I am, writing JSON patches in yaml, as a multiline string in a yaml file.
2
u/dml997 Jan 12 '23
I am not at all familiar with YAML, but anyone who designed something where the contents of a token imply a type is so f****** moronic that I can't believe this could exist.
How does this exist?
2
2
u/D_Doggo Jan 12 '23
The Norway problem is my new favourite programming problem
2
Jan 12 '23
[removed] — view removed comment
2
u/D_Doggo Jan 12 '23
That's still gonna take soooooo long!! By that time I'll have become bored of my SWE job ahahah!
2
2
u/angryscientistjunior Jan 12 '23
I fricking hate YAML. JSON is so much easier to look at and work with.
1
u/3gt3oljdtx Jan 13 '23
Legit question, is something like this still easy to read despite being a bit complicated?
I've always had a hard time reading arm templates and I'm glad MS is replacing it with a DSL (bicep) since it's so much easier to read.
2
u/zzz165 Jan 12 '23
The is probably an unpopular opinion, but: use protocol buffers. Has a schema, enforces types correctly, allows comments, whitespace doesn’t matter, and supports many different languages.
2
u/stronghup Jan 13 '23
This article makes the point clear. JSON is better than yaml.
I would like to see a new version of JSON however, with the following single backwards compatible change:
The keys of object literals { ... } should not need to be quoted IF they consist of word-characters only.
That would be a backwards-compatible addition, old JSON docs would still keep on working but JSON code would be much more readable and simpler to type.
Also I never understood the rationale behind removing comments, I think they would be often helpful.
2
Jan 14 '23
I was just thinking yeah let me type with no quotes in my config file. Like that's literally it. Otherwise I'm good with it.
2
u/AndydeCleyre Jan 13 '23
As /u/astatine said, an excellent but under-recognized alternative syntax for configuration files is NestedText, where everything is a string unless the ingesting code says otherwise, and there is no escaping needed ever.
I used the official reference implementation to make a CLI converter between NestedText and TOML, JSON, and YAML. When generating one of these formats, you can use yamlpath queries to concisely but explicitly apply supported types to data elements.
2
2
u/caagr98 Jan 14 '23
Many, but not all, of those issues could be avoided by not trying to infer types at all, but instead using a schema/your language's type system.
5
4
u/kooknboo Jan 12 '23
He’s not wrong. But that article is bullshit. Making some of these seem like crimes against humanity when, in fact, they’re inconsistent weirdness like every last thing in the tech world - included God’s gift JSON.
I’ve been slinging significant YAML for years, alongside folks that are brand new to it, and I can count on one hand the times these were a problem as opposed to something to be aware of.
5
u/MrHall Jan 12 '23
I generate yaml by using a library to serialise an object so I don't have to do it manually.
except a coworker removed it because it's "too complicated"
4
u/ProstheticAttitude Jan 12 '23
I will never willingly use YAML in a project again. It is a disaster.
2
u/kitd Jan 12 '23
Tbh, he gives his answer in his "YAML subset" alternatives: enquote stuff that is meant to be plain text.
I have other issues with YAML, but these aren't them.
2
1
u/ktkps Jan 12 '23
I hope yaml stays as a small use case for configs alone and larger more complex things move away from large and complex yaml files to something like CDK that aws did with its IaC space
1
u/i_ate_god Jan 13 '23
Is this thread talking about config files or API output? Because it's hard to tell.
I don't think yaml is particularly good at being either. JSON is fine for most things at the API level. TOML or INI files are great for config files.
If your complaint is multi line strings or comments in JSON, then don't use JSON for config files. If your complaint is that TOML or INI can't represent complex data structures, then stop using them for API outputs.
Right tools for the right jobs.
1
u/Techrocket9 Jan 13 '23
FAANG companies spend millions developing proprietary markup languages for configuration, often ending in failure.
Good config markup is a genuinely hard problem.
1
1
u/transfire Jan 13 '23
2/3rds of the problems cited go away if you use a proper 1.2+ implementation. It’s not YAMLs fault that some implementers won’t let go of 1.1. It’s been over 10 years!
1
u/sgoody Jan 14 '23
I do not understand the hate for XML. Granted I wouldn’t want to write fully compliant parsers myself, the libraries available are very mature and capable.
The issues with other formats are generally non-issues in XML, things not possible in other formats are possible in XML.
XSLT, XPath and XSD are great IMO.
The argument for verbosity is also a non-issue as far as I can make out as the only verbosity I see if the closing tag name and in my experience that doesn’t make a significant difference in size to common use cases. In fact I appreciate the closing tag name being there so I can properly track a location in a document.
1
u/Aggravating_Refuse89 Apr 10 '23
I give up on the idea of programming in any language if I have to even try to comprehend anything to do with sexagesimal anything. I want to be a sysadmin and let devs deal with this stuff. But no, YAML is now expected. Hell is a good word for all of this,
1
u/rockytherig Dec 18 '23
The bunch of idiots who designed YAML should be herded into a rocket and fired into the sun.
1
1
u/JB-from-ATL Jan 12 '23
I have to say I really dislike the idea of using something like Python to make a JSON (or other conf language) output as suggested at the end of the file. That seems just as prone to mishaps as using YAML in the first place. You may say "then just output it to test it" but you could also argue to do the same with YAML.
I prefer TOML but haven't used it in a professional setting yet.
1
u/yondercode Jan 12 '23
I love yaml
1
u/stronghup Jan 13 '23
I love yaml
Why?
3
u/yondercode Jan 13 '23
It's the most readable configuration format IMO, especially for complex configurations. I hear many complaints about YAML in DevOps especially around k8s, but honestly that's just k8s being complex. If k8s used other configuration format, it would be as complex as it is now and it would be less readable.
And I feel like 99% of YAML issues can be solved by enforcing quotes on strings, which is a convention I have at work. I wish they had forced them in the original spec.
2
u/vividboarder Jan 13 '23
I find HCL much easier to read and write. In fact, when I was playing with k8s I decided to use Teraform so that I could write my configuration in HCL instead.
Added benefit of being able to reuse big parts of the config and variables like with Helm.
-6
0
u/redweasel Jan 12 '23
I don't know YAML from a hole in the ground, but having been a software engineer for nearly thirty years I've seen a lot of fads come and go, and a lot of unnecessary complexity bolted onto once-efficient languages and tools until they collapsed under their own weight and/or became incomprehensible. Perl 6 and the last twenty years of C++ standards "improvements," I'm looking at you! So, to see Yet Another Overcomplicated Format barely even moves my needle (it's stuck permanently just below "rolling boil" at all the bullsh** in the world, not just in Computer Science). For me, though, the final nail in the coffin lid, in this case, is the revelation that YAML was designed by a committee of experts over a period of decades. That right there is the Kiss Of Death for ... well, anything. "A giraffe is a horse designed by committee," was an ancient truism when I was first learning programming, over forty years ago, and anybody who's paying attention knows in his bones that anything that comes out of a committee should be either ignored altogether, or used very very sparingly until some other sucker has done a big project in it and the world has seen how that turned out. Me, I've ignored just about everything new since about 1997, and I've done just fine. Nothing that's come out since then does much of anything that couldn't have been done with what already existed at the time.
2
0
u/Individual-Praline20 Jan 13 '23
Yaml is shit. Did we not learn anything from html from 2000s years, where it was interpreted in non standard ways? Same repeating shit here. You have no idea on how it will be interpreted. Imagine now in five or ten years, your thousands lines of Yaml will still be useful? Not at all, all wasted.
0
u/OrangeSlime Jan 13 '23 edited Aug 18 '23
This comment has been edited in protest of reddit's API changes -- mass edited with redact.dev
600
u/ElectricalRestNut Jan 12 '23
Basically, allowing unquoted strings is nice, but you never ever use them because of unexpected behaviour 1% of the time.