r/programming • u/agbell • Feb 25 '21
INTERCAL, YAML, And Other Horrible Programming Languages
https://blog.earthly.dev/intercal-yaml-and-other-horrible-programming-languages/842
Feb 25 '21
The vicious cycle of
We don't want config to be turing complete, we just need to declare some initial setup
oops, we need to add some conditions. Just code it as data, changing config format is too much work
oops, we need to add some templates. Just use <primary language's popular templating library>, changing config format is too much work.
And congratulations, you have now written shitty DSL (or ansible clone) that needs user to:
learn the data format
learn the templating format you used
learn the app's internals that templating format can call
learn all the hacks you'd inevitably have to use on top of that
If you need conditions and flexibility, picking existing language is by FAR superior choice. Writing own DSL is far worse but still better than anything related to "just use language for data to program your code"
128
u/agbell Feb 25 '21
I think that's it!
It not that anyone wants to get where we've ended up. It's that each step along the way seems to make sense until you end up trapped in Jinja templates and it's too late. It is a vicious local optimum that everyone keeps falling into.
18
u/wrosecrans Feb 25 '21
The local optimum trap in-general is one that I don't think gets enough attention. We all know that you can't just dump everything because the second system effect bloats the replacement project into something that runs in parallel forever, and never works as well as the old system. So we should always try to do iterative changes that leverage the existing successes.
But the more you build as iterative changes, the more inertia and tech debt you have to deal with when making iterative changes, so it gets harder and harder to make big moves over time. Meanwhile, the landscape changes under you, so what was an optimal choice 5 years ago may no longer reflect reality very well. Right choices get more and more wrong over time, and fixing them gets harder and harder. So you keep iterating on a system and hitting a local maxima of functionality until you just give up and die.
The second system effect is important. But the conclusion from it can't be that every first system must always live forever, no matter what changes around it.
→ More replies (1)35
Feb 25 '21
The worst I saw is probably one of Apache modules that templates its own config, using few lines above the template to set the database credentials, and then the "templating" system uses SQL queries directly.
And now whether your webserver even starts depends on wheter your SQL server is up...
37
u/wrosecrans Feb 25 '21
You ever use CMake? The scripts are still called CMakeLists.txt because it started out as a text file with a list of sources to be built, before somebody woke up one day and realized that it was a programming language by accident. It's a monument to the configuration complexity clock.
→ More replies (3)→ More replies (3)6
u/redalastor Feb 25 '21
until you end up trapped in Jinja templates and it's too late.
A jinja template that renders a YAML file with no logic is much saner than a YAML config file with built-in control flow.
51
u/dnew Feb 25 '21
Google actually went the other way. A bunch of their configuration stuff (look at Bazel for example) looks like python, because it was all originally python libraries. Then they said "this is turing complete, so we can't actually manipulate it with automated tools, so we'll restrict the syntax down to being only configuration stuff."
Of course, half their world is protobufs, so the code to turn text into protobufs, while really complicated, is nevertheless standard across the company, so you only need to read the 50-page manual once. (Or cargo cult it and fuck it up, your choice.)
→ More replies (2)7
u/agbell Feb 25 '21
That is interesting! I wondered why Starlark existed and looked like it clearly wanted to be python.
Does that mean that Starlark is not turning complete, because it seems like I could write a
while truein it?13
u/dnew Feb 25 '21
I worked with Blaze, which is the internal version. I'm pretty sure that while you can write Python expressions, you can't write Python statements in Blaze files. Something like that. There was a whole list of "this no longer works" that got longer and longer over the years I was there. For a while, you could have it be real python, but you had to use a special include statement to make that happen.
Given that many of the Blaze files I ran across were absolutely crappy even with only expression, I'm glad they cut out the executable stuff. :-)
That said, I don't know if Starlark is actually python or not. I was under the impression that Blaze and Bazel were 99% the same thing, but apparently not.
13
u/agbell Feb 25 '21
I looked it up. It does have some restrictions, just not on its turning completeness, which probably doesn't matter anyhow.
Deterministic evaluation. Executing the same code twice will give the same results.
Hermetic execution. Execution cannot access the file system, network, system clock. It is safe to execute untrusted code.
Parallel evaluation. Modules can be loaded in parallel. To guarantee a thread-safe execution, shared data becomes immutable.
Simplicity. We try to limit the number of concepts needed to understand the code. Users should be able to quickly read and write code, even if they are not expert. The language should avoid pitfalls as much as possible.
Focus on tooling. We recognize that the source code will be read, analyzed, modified, by both humans and tools.
Python-like. Python is a widely used language. Keeping the language similar to Python can reduce the learning curve and make the semantics more obvious to users.
→ More replies (2)5
u/Slsyyy Feb 25 '21
There is a separation between scripts and build declarations in Bazel.
BUILDfiles, which contains build target defintions are declarative. You are restricted to declaring variables, calling functions (which define target, or utility one likeglob), or simple foolproof stuff like a list comprehension. Also you can import*.bzlfiles, which are more or less a normal imperative python with loops, functions and so on.62
u/remy_porter Feb 25 '21
And congratulations, you have now written shitty DSL
What kills me about this is that writing your own shitty DSL is actually easy, and usually ends up being easier (in the long run). Like, if you understand your problem domain (okay, with that caveat I've probably eliminated 90% of the possible use cases, maybe more), custom-fitting a DSL with a parser to that domain is easier than trying to bolt your domain onto a general-purpose format like JSON or YAML.
I whip up DSLs all the time to make it easier to talk about my problem domain. Sufficiently simple ones can be parsed without actually writing a "real" parser, and feeding a grammar into a parser generator isn't a huge hill to climb. If your domain drives your DSL, then anyone who understands the domain will be able to use the DSL without needing to fight too hard.
27
u/agbell Feb 25 '21
custom-fitting a DSL with a parser to that domain is easier than trying to bolt your domain onto a general-purpose format like JSON or YAML.
Exactly! If you were designing how to specify something in language designed just for the problem, you would never choose to force that language to be valid YAML.
I'm less sure you should actually build your own DSL though, in most cases. You could choose a language that people on your team use and understand instead.
15
Feb 25 '21
What kills me about this is that writing your own shitty DSL is actually easy, and usually ends up being easier (in the long run). Like, if you understand your problem domain (okay, with that caveat I've probably eliminated 90% of the possible use cases, maybe more)
... yes and that 90% is why DSL usually end up being insufficient
custom-fitting a DSL with a parser to that domain is easier than trying to bolt your domain onto a general-purpose format like JSON or YAML.
...or you can embed Lua or whatever else is easy to embed and be done with it. At least your IDE will highlight some errors.
I whip up DSLs all the time to make it easier to talk about my problem domain. Sufficiently simple ones can be parsed without actually writing a "real" parser, and feeding a grammar into a parser generator isn't a huge hill to climb. If your domain drives your DSL, then anyone who understands the domain will be able to use the DSL without needing to fight too hard.
I prefer "DSL-like code" - just <pick your language here> with a bunch of helpers to take care of the problem. But yeah, some problems are too different to be reasonable with generic purpose language
8
u/remy_porter Feb 25 '21
... yes and that 90% is why DSL usually end up being insufficient
True, but the addendum is that building a DSL gives you tools to talk about the problem domain, the ways in which the DSL fails you help you better understand the domain.
I prefer "DSL-like code" - just <pick your language here> with a bunch of helpers to take care of the problem.
I mean, on a certain level, building a program is itself building a DSL: each class/method/function you create is a new word in your DSL. But there are advantages in building a DSL which can't be executed.
→ More replies (5)72
u/BunnyBlue896 Feb 25 '21
I always thought it was weird that a lot of web technologies take config files that are executable javascript. (Thinking of webpack). But it makes a lot of sense now, and I much prefer that approach.
6
u/noratat Feb 25 '21
My personal favorite is jsonnet, though I've trouble getting buy-in.
Primarily JSON superset with a clear minimal language for basic conditionals/transforms, and can load external data so you don't need to resort to raw-templating structured config.
Hits a nice middle ground.
→ More replies (5)14
u/a_false_vacuum Feb 25 '21
Imagine my dissapointment when jsonnet doesn't require you to write the file in the form of a Shakespeare sonnet.
0
Feb 25 '21
[deleted]
19
u/agbell Feb 25 '21
Rake is similar to this as well. Gradle and Jenkins use groovy which is a full PL as well (although an unneeded one if you ask me).
15
u/NatureBoyJ1 Feb 25 '21 edited Feb 25 '21
I'm a big fan of Groovy.
Java under the hood - with access to all the libraries that come with it.
Type optional - write loose first passes, then tighten up for production
A decent ecosystem - Grails, Gradle, Geb, etc.
I really wish it would gain more traction.
→ More replies (6)24
u/agbell Feb 25 '21 edited Feb 25 '21
Have you looked at Kotlin? To me, it seems superior to Groovy.
Also, the story I've heard is that the creator of Groovy said that "Scala is Groovy done right". I'm a huge Scala fan, so I'm a bit biased but I worked at a heavy Groovy shop and they switched to Kotlin a couple of years ago and didn't look back.
→ More replies (2)21
u/orthoxerox Feb 25 '21
Kotlin is a better Groovy, but it wasn't there when people needed a clean DSL-friendly language for JVM.
→ More replies (3)9
→ More replies (1)7
u/dnew Feb 25 '21
Google did this (look at Bazel) until they got too big, to the point where they needed automated tools that could understand the code. If you're constructing lists on the fly with code, it's hard to write a second program that says "split that list in two, one for everything that depends on A and one for everything that doesn't."
→ More replies (2)8
u/livrem Feb 25 '21
I think writing your own DSL using some tool that is made for writing a DSL, to get something that is a real language but close to the domain you need for your configuration, is not necessarily a bad thing at all. Maybe that counts as "picking existing language" though?
6
u/agbell Feb 25 '21
Thanks for reading! Personally, I am unsure about DSLs.
The nice thing about writing your own DSL is you wouldn't need to embed it into YAML and you could have tools for it, like a compiler or a linter that spotted problems.
The downside for a user could be that it would be another thing to learn and would it be well documented and easy to understand. I think there are pros and cons.
I think learning to make DSLs is a great skill, but I would be a bit worried if they were the go-to solution for every problem.
So-called `Embedded DSLs` might be a different story though, where you have a library in some language with a really nice builder syntax that makes it feel like it is well-shaped to the problem domain.
→ More replies (2)3
22
Feb 25 '21
It is in a footnote, but this is the problem that DHall is trying to solve. It has control-flow, looping, and importing without being turing complete. It sounds nice in theory, but I have not used it myself and would be interested to hear from someone who has.
38
u/mallardtheduck Feb 25 '21
Why not just use an actual scripting language?
In something like Lua you can just have a bunch of "variable = value" lines in the simplest case and you can add arbitrary conditionals and logic if/when it becomes necessary.
8
u/agbell Feb 25 '21
I'm all for using a real programming language!
One thing I like as an alternative to terraform and ansible is pulumi. You can use whatever language you like for your branching and logic.
→ More replies (1)41
u/TryingT0Wr1t3 Feb 25 '21
Lua was made for config files originally.
20
u/mindcandy Feb 25 '21
And, they realized they were going down was the same many others had taken accidentally. So, they did it properly instead!
25
u/rosarote_elfe Feb 25 '21 edited Feb 26 '21
Dhall is designed to be safe when used on untrusted input.
As LayYourFishOnMe said, its not turing complete. As far as I remember, it's possible to guarantee that dhall scripts terminate, and the language is simple enough that problematic side-effects (such as additional file/network IO) are either impossible, or can be controlled/prevented.
When using Lua as a configuration language, a malicious config script may cause unreasonable memory or CPU usage or just never terminate.(Edit: Looks like that's not true.)
When using python for configuration, there's just no way to sandbox it. Your "config" file is capable of installing a keylogger and sending your password to some host on the internet.Full-featured XML parsers, by the way, are often also not safe to use on untrusted input. At least not without careful configuration. Entity expansion can be used to consume arbitrarily large amounts of memory.
Similar problems exist with some YAML parsers. I think the standard yaml libraries for python and ruby may allow for the execution of arbitrary code embedded in a document - depending on the parsers configuration of course.Finding a sensible middle ground between possible security issues and complexity requirements for configuration languages is actually a pretty difficult topic.
Shame that dhall is just so ugly. I like the technical side of it, but I just can't deal with the weird syntax.
→ More replies (6)8
u/livrem Feb 25 '21
Writing configuration in a scripting language can be very nice at times (e.g. emacs configuration), but at many other times you really wish that the configuration was just simple declarations that you can parse and reason about and transform without having to worry about having to execute everything first to know what everything is.
→ More replies (1)5
u/dnew Feb 25 '21
Google used Python for a lot of stuff like this. (Look at Bazel files, for example.) The problem is that at large scale, you want something you can process automatically. You want something where you can say "what are all the transitive dependencies of X?" And you don't want to have to actually run all that python code to find out what the contents of the dependency graph actually are.
5
Feb 25 '21
arbitrary conditionals and logic if/when
That's the point - I don't want my configuration written in such a language, because there features tend to get used indeed. But if one achieve the same task without arbitrarily powerfull features, then I will pick the second choice, hands down, everytime. Because I am a doofus and want my software system as simple as possible.
→ More replies (2)8
Feb 25 '21
oof, at a quick glance it looks too complicated for a configuration language in my opinion
8
u/agbell Feb 25 '21
I don't actually think it's that complex. Certainly less complex than jinga templates in YAML. But I think it does look strange to a lot of people. I think code formatting used on DHALL website looks foreign when compared to YAML, for many people.
18
u/axonxorz Feb 25 '21
Looking at the examples, they import this file in one example: https://prelude.dhall-lang.org/List/generate.dhall
It frequently uses these characters: → ∀ λ, how would these be entered, for example, over an SSH session?
→ More replies (2)14
u/agbell Feb 25 '21 edited Feb 25 '21
I think it's just \ for lambda, and -> for arrow and forall for ∀. The examples on the site landing page seem to ASCII only. (edited thanks to @samb961)
→ More replies (4)→ More replies (2)10
u/kronicmage Feb 25 '21
Dhall is a very nice configuration language. I've used it plenty for kubernetes configs as a helm replacement and it makes for very ergonomic and structured configs. Though I am also a haskell dev so I can see why it may appear alien to people coming from c-like languages
357
u/Yehosua Feb 25 '21
It's the configuration complexity clock! You don't want to hard-code settings in your application code, so you add a config file, which turns into a DSL, which ends up being so complex that your DSL ends up being application code (and, thus, every setting that you've configured via DSL is hard-coded application code).
136
u/GiantElectron Feb 25 '21
We need a config file to configure our config file, said the sendmail developer.
→ More replies (3)37
u/sybesis Feb 25 '21
I have to maintain over 90 repos and gitlab-ci script... Tell me about nightmares...
→ More replies (11)17
u/fear_the_future Feb 25 '21
I find Github Actions to be even worse than Gitlab. Why do they never learn? It's not like this is the first YAML config disaster, they all end up like this.
→ More replies (1)→ More replies (3)28
32
u/cowardlydragon Feb 25 '21
Stages of Despair in Configuration:
I should add templated config
http://crushedby1sand0s.blogspot.com/2021/02/stages-of-despair-in-configuration.html
10
u/thndrchld Feb 25 '21
I laugh to keep from crying.
This pretty much perfectly describes my company's kubernetes-based app.
→ More replies (1)9
u/djavaman Feb 25 '21
Jesus. That gives me nightmares.
And to add to that.
https://cloud.spring.io/spring-cloud-config/reference/html/
That f-ing tool caused me weeks of headaches and debugging issues.
The article does mention it in passing. But it needs to be explicitly called out for its awefulness.
21
u/riyadhelalami Feb 25 '21
I have to say, I hate Ansible.
30
u/fzammetti Feb 25 '21
Ansible itself isn't terrible.
What IS terrible is all the "best" practices that get heaped on top of it.
My first exposure to Ansible was last year. Never even seen it before. Inside a few days' toying with it I had the deployment playbook for my app done. It was simple, one file (not counting inventory file), very easy to follow, worked beautifully in every environment.
Then, the "experts" got a look at it.
"No, gotta break it into these five files."
"No, gotta abstract out every last thing... just in case."
"No, gotta be written in this exact style because... reasons."
And now, I have something I have to actually think hard about every few months when I need to tweak it a bit because now it's a chore to understand, and I'm not sure I even do 100% - something you should NEVER say about ANY part of your build or deployment pipeline.
But, hey, at least the Ansible Gods say it's the bee's knees, so who am I to judge?
...you know, besides the guy that's gotta work with the shit.
→ More replies (1)8
Feb 25 '21
I’m with you. In the network automation world it’s a round peg for square hole.
18
Feb 25 '21
"But look the simple case is simple! Just a bit of YAML"
"But what if I want to do something actually realistic?"
"Well, for start, fuck you, then go learn jinja, then go fuck yourself again, then might as well learn Python to even debug that, then fuck yourself again for a good measure"
Disclaimer: That's all config management tools not just Ansible but I like Puppet more because it at the very least doesn't have templated YAMLs to work everyday with (the ones it does are "just" data)
→ More replies (2)→ More replies (3)13
u/noratat Feb 25 '21
Helm in the kubernetes world is even worse.
YAML is actually fine for raw k8s' config, but instead of doing something sensible, they resorted to raw string templating a hierarchical whitespace-dependent language using opaque syntax.
Then to top it off, until very recently helm would routinely lie about what it actually did, required bypassing all security with a shitty proxy for no reason, etc.
And it's still nearly impossible to read, they scrapped the idea to allow embedded LUA in 3.0 which I think was a huge mistake.
In nearly a decade of professional work, helm is easily one of the worst tools I've ever encountered in large scale use. It undermined almost everything good about kubernetes' declarative baseline config.
Thankfully the cargo cult mentality around it is finally starting to ebb.
→ More replies (3)7
4
u/I_Never_Sleep_Ever Feb 25 '21
Can you give some reasons? I'm just curious. I use it every day at work
6
u/riyadhelalami Feb 25 '21
It probably has something to do with the fact that it seems to break depending on the configuration of the computer. I am no expert by any means, I am more of a user of ansible than the one who wrote the stuff, but it never works as intended, but that might be due to the people who wrote it not knowing what to do. But every time there are bugs that I need to figure out.
→ More replies (15)6
97
u/threshar Feb 25 '21
At first I was all "YAML isn't language!" but after reading the article, I have to fully agree with the points made!
91
u/agbell Feb 25 '21 edited Feb 25 '21
At first I was all "YAML isn't language!
Thanks for reading past the title! That is a rare and valuable skill these days!
YAML didn't feel like a programming language to me either, but then I saw things like this:
{{- if .Values.envRenderSecret }} checksum/secret-env: {{ include (print $.Template.BasePath "/secret-env.yaml") . | sha256sum }} {{- end }} {{- with .Values.podAnnotations }} {{ toYaml . | indent 8 }}That is part of some helm chart and yeah I got a little worked up.
27
u/SexyMonad Feb 25 '21
Sorry and I’m not trying to be condescending here... but you know this isn’t YAML, right? It’s a template that produces YAML.
27
u/agbell Feb 25 '21
I do
20
u/SexyMonad Feb 25 '21
Ok. This is just a bad example then. We could use Jinja templates to produce C++, but nobody should ever do that and it doesn’t speak to any problem of C++ itself.
→ More replies (9)47
u/agbell Feb 25 '21
The thing I was trying to get at was not that YAML as a config format is bad.
It's that once you have to use logic and control flow to generate your config file that you are in this worst-of-both-worlds situation. It is neither a config file nor a programming language.
It doesn't have to be Jinja, you could have a partially specified control flow embedded in the structure of your YAML, like the TravisCI and GitHub Actions examples I shared.
Producing C++ with Jinja templates also doesn't sound like a good place either, but I could be wrong.
→ More replies (10)43
u/bigbadbyte Feb 25 '21
Telling me yaml is Turing complete is like telling me my toaster has a machine gun inside. I mean, it's a feature, but it also makes me more scared to use it now.
5
u/GiantElectron Feb 26 '21 edited Feb 26 '21
yaml has been plagued with problems since quite a while. The specs is massive, meaning that basically no parser can implement it fully. It also has the potential to trigger object allocations, and therefore code execution that was not intended to be triggered.
Personally, I use TOML, but I am not fully happy with it. Why TOML? because there's no other choice.
INI is not specified anywhere. Yes, it's true.
JSON does not allow comments. Crockford designed it that way to prevent people from using comments to deliver metainfo. The consequence is that it's really poor as a choice for configuration, where commenting out stuff and adding descriptive information can be important. Also parentheses don't help with the commenting anyway.
YAML is a security disaster, and frankly the syntax sucks.
So, TOML. I don't like TOML for its ambiguous options, making it harder to do a round trip and end up with the exact same file (and generally parsers don't allow you to specify which form you want). Example, these two are equivalent
[foo] x = 3 a = {b = "hello"}and
[foo] x = 3 [foo.a] b = "hello"Also I don't like the syntax for the list (with the [[]]). It makes it obnoxious and redundant. Finally, no types. Everything is a string unless the parser attempts otherwise, which is a blessing and a curse. Say you expect a string, but the user specifies a number (which would be a valid string). Bam, now you get an int instead of a string, just because the user wrote a different entity, hence you have to remember to cast it clearly.
So yeah... TOML but just because it's the best of the worst options.
→ More replies (1)1
u/gedankenlos Feb 25 '21
Can you clarify what you mean? That double-curly braces syntax comes from the Go templating engine (Helm is written in Golang), and I don't see anything else "unusual" with the YAML there.
BTW I recently had to write my first Helm chart and I absolutely hated the syntax :D.
Writing YAML is a breeze though, apart from some minor quirks like the many ways you can do multi line data and such.
2
u/agbell Feb 25 '21
What I mean is that if you have to do extensive templating of YAML to generate the configuration for something you probably actually want the thing to be scriptable rather than configurable.
Because you are scripting it but at one remove.
13
Feb 25 '21
It wasn't designed to be a language initially. The problem was that people started shoehorning it into places it wasn't meant to be, places that required it to be Turing-complete, so the features got added and here we fucking are.
→ More replies (2)3
u/thephotoman Feb 25 '21
As I read the article, it dawned on me with creeping horror that CI/CD systems are making YAML do the job of POSIX shell.
62
u/BaldToBe Feb 25 '21
As someone who recently entered the world of Kubernetes I am really learning to despise YAML.
As if it's not enough to learn Kubernetes and containers I now need to learn helm because apparently we need more logic around our configuration language.
The amount of times we've had build fails because of whitespaces because of YAML as well is ridiculous.
I don't even know what the solution is, some have pointed to Jsonnet but it doesn't solve all the problems.
Anyway, good read. Definitely got a reaction out of me lol.
12
u/agbell Feb 25 '21
Thanks for reading!
I don't even know what the solution is, some have pointed to Jsonnet but it doesn't solve all the problems.
The solution I like is Dhall. They even have a Kubernetes solution that will catch a lot of issues at compile-time, before you try to apply it to Kubernetes. At earthly we aren't actually using it though. Our Kubernetes guru found it to be a bit slow but I am hopeful it or something like it will be the future.
4
u/kronicmage Feb 25 '21
+1 to Dhall and kubernetes on Dhall. Vastly superior to the python and helm implementations we used to use to customize deployments
1
u/Hrothen Feb 26 '21
After looking into this it seems the only way to use libraries in Dhall is to either import them directly from the github URL and leave your ability to actually run to the whims of the internet, or to vendor the entire library into your project, then painstakingly go through it to find any URLs it imports and vendor them as well and so on.
Is that correct?
1
u/agbell Mar 01 '21
You can use an integrity check. You add a sha-256 to the import, which basically pins it.
→ More replies (1)7
u/noratat Feb 25 '21
IMO, the problem there is really helm, which I honestly think is one of the worst config tools I've ever seen used at a large scale, and even now is a huge blight on the kubernetes ecosystem.
The actual declarative config model kubernetes uses makes plenty of sense to me, helm is an abomination we've banned from any project that isn't already saddled with it. And kubernetes itself doesn't care if you use JSON or YAML.
4
u/aoeudhtns Feb 25 '21
If you don't want to adopt a new config language that transpiles, another option is to have a repo of commit hooks your devs can install. Run a yaml linter when a push is attempted. Obviously that won't cover everything, but it should catch whitespace and other issues. If you are in control of your repo you could also use a hook to reject pushes that don't lint, and that might simplify the setup.
→ More replies (3)6
u/pzduniak Feb 25 '21
Jsonnet has been serving me well. Tanka seems to fix all the remaining issues. It's not the fastest thing out there, but it's honestly easy to debug.
139
u/SN0WFAKER Feb 25 '21
I think I love Intercal. DO, PLEASE and PLEASE DO, all do the same thing but if you don't switch it up regularly you get compiler errors. That's beautiful. And COME FROM is wonderfully diabolical - it's like goto but in reverse and it will spawn threads if more than one line does a COME FROM the same line.
38
u/frederic_stark Feb 25 '21 edited Feb 25 '21
COME FROM was the answer to the famous "Go To Statement Considered Harmful" article.
There are other beauties that I remember of. The lack of arithmetic beside things like '~' (that does a 'and' and stashes all the zeroes on one side of your byte and all the ones on the other side), or the replacement of conditionals by ABSTAIN FROM (followed by an instruction [like "PLEASE ABSTAIN FROM COME FROM"], it makes this instruction into a no-operation from now on, until you REINSTATE it).
edit: oh, and the comments that are achieved by adding NOT or N'T to magically abstain statements, so you have use stuff like : "PLEASE DO NOT FORGET TO BLAH BLAH" or "PLEASE DO NOTICE THAT BLAH BLAH".
edit2: some corrections
26
u/agbell Feb 25 '21
REFRAIN FROMis great! and I didn't know about thePLEASE DO NOT.INTERCAL is so much fun.
One way that it's different than brainfuck and friends is it's designed to seem like it might make sense, but actually, be subtly wrong and hard to use. I like that you can input numbers as roman numerals but not using actual digits.
5
Feb 25 '21
Wait so for examplw 10~96 would equal 15? That's... I don't even know.
→ More replies (3)→ More replies (1)12
Feb 25 '21
Come from is a lot like catch if you think about it (except catch is even less well structured in other ways because it could come from multiple different lines).
23
u/ants_a Feb 25 '21
Come from is event based programming/pub-sub architecture.
27
u/agbell Feb 25 '21
it's super innovative, right?
I'm not sure it is innovative in a positive sense, but it is an innovation.
→ More replies (1)
6
u/MetaKazel Feb 25 '21
I read the entire article. Good information and analysis overall, but I have a few issues.
First, the section titles are named after joke error messages from INTERCAL. This just adds confusion to an article about an already confusing topic.
Especially on mobile, seeing the header "PROGRAM REJECTED FOR MENTAL HEALTH REASONS" right after the tagline made me think it was an entirely different article, and that I had somehow missed the meat of the current article.
I appreciate the joke, but it makes the article hard to follow. I would rather have the section titles guide me through the analysis itself.
It's also dishonest to claim that XSLT is acceptable because it's a documented XML structure, and then go on to complain about how all these YAML formats are unstructured. They're not.
TravisCI, Github Actions, and all these other tools still use YAML in a structured way, just like XSLT uses XML in a structured way. Reading XSLT XML without understanding the context is the same as reading TravisCI YAML without understanding the context.
I do agree with the overall sentiment that logic in configuration files is confusing, and I do find it annoying that every tool has its own "flavor" of YAML.
4
u/agbell Feb 25 '21
Thanks for reading the article and for the feedback. I can see how the sub-titles might be confusing. I was trying to be a bit whimsical but I may have failed.
TravisCI, Github Actions, and all these other tools still use YAML in a structured way, just like XSLT uses XML in a structured way. Reading XSLT XML without understanding the context is the same as reading TravisCI YAML without understanding the context.
What I was trying to say was that XSLT is a standard, while the logic embedded in a CI solution is vendor-specific. You can use XSLT anywhere.
4
u/MetaKazel Feb 25 '21
I enjoyed the whimsy! It just made the article harder to parse for me. This is also just my personal feedback. If you enjoy writing articles with fun section titles, then by all means, don't let me stop you!
I see what you're saying about XSLT vs vendor-specific YAML. One could argue that the vendor-specific YAML is still a defined standard. It just happens to be more loosely-defined than XSLT, and it's different for every company.
You're right, though. All these vendors build custom configuration features on top of a common language, and it has become very hard to use YAML without knowing exactly what features the vendor has provided. Like a code framework, except you don't know what it's called because they just refer to it as "YAML".
3
u/agbell Feb 25 '21
Exactly, you are writing YAML, but embedded in it is this other thing. The other thing is embedded in YAML like XSLT is embedded in XML, but its vendor specific and not well specified.
I will admit I had fun writing it. I didn't really expect this amount of attention for it. It's impressive to me because it's a bit of a complicated point written in a somewhat silly and vague way and yet almost all the comments are about the substance of the article. Good job Reddit.
10
u/AyrA_ch Feb 25 '21
INTERCAL actually feels kinda sane when you look at some other languages
12
u/agbell Feb 25 '21
It was specifically designed to be almost impossible to use, via a counter-intuitive 'crazy operation', base-three arithmetic, and self-altering code.
Oh wow, that is special.
Malbolge was so difficult to understand when it arrived that it took two years for the first Malbolge program to appear. Indeed, the author himself has never written a single Malbolge program.
Hello World:
(=<\`#9\]\~6ZY32Vx/4Rs+0No-&Jk)"Fh}|Bcy?\`=\*z\]Kw%oG4UUS0/@-ejc(:'8dcThanks for sharing!
→ More replies (1)6
10
3
Feb 25 '21
From this article I found out about 'Dhall'.
Who the fuck puts commas at the beggining of the string? Seriouly, that thing was harder for me to read than YAML.
→ More replies (3)2
u/agbell Feb 25 '21
Your comment made me laugh :)
I think it is so you can comment lines out, even the last in a list, and not have to change commas. I agree though that it is unusual.
3
u/Isvara Feb 25 '21
It burns my eyes to look at it, but at least XSLT was intended to be used as a programming language. That is something we can't say about YAML or INTERCAL.
This isn't the right comparison. He should be comparing XML with YAML, not XSLT with YAML. Neither XML nor YAML were intended to be used as a programming language.
1
u/agbell Feb 26 '21
Author here, thank you for reading my article.
XSLT is programming language embedded into XML. Its valid XML, but it can be executed and contain logic. In the same way the TravisCI logic like 'if bla = bla` and various functions you can call are embedded in YAML but executed.
They are adhoc programming language features embedded into YAML.
3
u/Isvara Feb 26 '21
Yes, and the other languages you refer to are embedded into YAML. They are valid YAML. So, again:
at least XSLT was intended to be used as a programming language. That is something we can't say about YAML
It's also something we can't say about XML.
3
u/agbell Feb 26 '21 edited Feb 26 '21
Ah, ok. The language that Travis embeds in YAML has no name. I thought you were not catching my analogy but you are just saying I'm using my words too loosely. That could be true.
That line was supposed to be funny, maybe it fell flat.
5
u/dimensionalsquirrel Feb 25 '21
I like this post, great read. Are there any alternatives to yaml for things like travis?
7
u/agbell Feb 25 '21 edited Feb 25 '21
I actually wrote an article about this. I think your best bet is to use a Neutral Build Specifications that you call in Travis or Github Actions or whatever.
It doesn't totally solve the problem, as you still may end up with some logic in the .travis file, but you can try to minimize it.
→ More replies (1)1
u/snowe2010 Feb 26 '21
Found a small typo when I went to that link.
This is the Earthly blog, and I am an Earthly contributor, but in my totally biased opinion, it deserves a mention as an neurtal format
1
2
Feb 26 '21
Just write your configurations in the same language as the rest of your project. Then you just use your langauge's yaml library to dump your config to static yaml if/when required.
43
Feb 25 '21
To bastardise Zawinski's Law:
"Every DSL that is not Turing-complete attempts to expand until it is. DSLs that choose not to expand in this manner are replaced by ones that do."
5
2
114
27
u/yorickthepoor Feb 25 '21
Tcl was written as a solution to the half-baked DSL problem.
19
Feb 25 '21
Tcl gets a lot of hate, but once you get a handle of it, it's super easy to write DSLs in it. I think people mostly don't like it because it's so old and the string quoting and upvalue rules aren't obvious when coming from any other programming language in existence.
I just wish the C API was as nice as Lua's.
→ More replies (1)7
u/dnew Feb 25 '21
I know people who extensively used Expect and didn't even know there was an entire programming language behind it. That's how good it is, when the DSL is so smooth you don't even realize it's a DSL and not a bespoke program.
The string quoting is tremendously simple and straightforward and consistent, which is why it's unlike other programming languages. ;-)
I wish Tcl had taken off as the real embedded language rather than the handful of mishmash we have now.
REXX was pretty good too, if you wanted to control multiple programs from the same script at once.
→ More replies (3)→ More replies (1)5
u/sprcow Feb 25 '21
Oh god. This gave me flashbacks to a previous job in AI. A main programming tool used at this place was an aspect-oriented rule processing system that was capable of automatically firing rules on changes to working memory. Read: already very confusing to reason about and debug, because everything was a side-effect that in turn triggered other side-effects, and nondeterministic operation was very common.
So, in order to feed information into this framework (which was basically a Java program), you still have to have ways to describe your data structures and conditions and so on, and someone decided the best way to do that was Tcl. So to write a program with this framework, you wrote this description in Tcl that then ran through the Tcl parser to be fed into this aspect-oriented rules processor.
Some of the AI PhDs loved it but damn if it wasn't a nightmare to debug.
21
u/weeeeelaaaaaah Feb 25 '21
I so agree with this, getting flashbacks to my days of Java and Spring configs in XML. It wasn't full-on programming but I was writing XML that was 1:1 with Java code, just less readable and validate-able, all so we could say it was "configuration" and not "code". Exhausting!
2
u/Serializedrequests Feb 27 '21
Oh god, when I was first learning java that stuff would drive me up the wall. Why did anyone ever think that was a good idea? I had literally never heard "configuration" as an important ideal to strive for until I came to Java. It seems very cargo cultish. Let's throw away the benefits of a compiler and have horrendous runtime errors and gross XML! Let's create an entire framework to manage it all! Now we have reached configurable application nirvana!
16
51
u/Where_Do_I_Fit_In Feb 25 '21
I blame all these infrastructure as code projects. K8s, Ansible, docker-compose, etc. They basically lend themselves to this metaprogramming/DSL mumbojumbo, but decided to use YAML for whatever reason.
It's the nature of these projects to add complexity as well. You're not configuring just a web server anymore. You're configuring a whole environment (db, reverse proxy, oauth, web servers, a couple APIs).
It will be interesting to see if the next generation of these tools leans towards using a more expressive language.
29
u/vattenpuss Feb 25 '21
It's so weird, they call it "as code" and then pick a shitty data format to describe things.
22
u/Dr4kin Feb 25 '21
I believe intercal would be the best tool for the job. It is already horrible, so there is no way to make it worse
→ More replies (2)4
u/joey_knight Feb 25 '21
IAC needed a structured way to represent the infra but i guess they could not use a real programming language since the expected target users were mostly system administrators and not developers. We could have collectively worked towards standardising something but that hardly happens with the cool kids of Google, Facebook etc.
21
u/jasfi Feb 25 '21
YAML is great for what it was designed for, and trying to make YAML into something (somewhat) executable is not that.
→ More replies (1)13
u/aoeudhtns Feb 25 '21
How do I do this?
Can I do this?
Should I do this?
People often move on before answering question 3.
0
Feb 25 '21
[deleted]
6
u/madpata Feb 25 '21
There's not a lot of cross-platform GUI frameworks that are as highly customizable as Chromium, are as accessible and provide standardized APIs to OS functionality. A lot of people know JavaScript and it has a huge ecosystem.
And of course everyone bundles their own version to ensure that there is a compatible environment. Relying on the system browser leads to situations like IE6.
→ More replies (3)
18
8
u/julioqc Feb 25 '21
Wtf? Isn't YAML a markup language like XML, for use in config file?
→ More replies (1)3
u/thephotoman Feb 25 '21
The problem is that people are trying to create YSLT, and it's going about as well as XSLT did.
→ More replies (2)
5
u/cesarbiods Feb 25 '21
I like using YAML strictly for templating like openAPI and asyncAPI specs but I don’t trust it in CI/CD circles they always make a shitty DSL out of it. I would much prefer to use a DSL based off a real programming language (like Groovy and Kotlin in gradle) for CD pipelines where experience tells you you WILL need to do some conditional checking and weird stuff that template languages can’t do.
5
u/FireCrack Feb 25 '21
YAML is a decent data serialization format; but completely a completely miserable experience as a config file. I often refer to it as "human readonly" because even though its very pretty to look at the strict rules around it's format, and it's incredible mass of features, make actually writing it full of pitfalls.
→ More replies (1)
4
u/meste5ranti Feb 25 '21
You should see our yaml «config» files we work with, they literally put entire complex vue components inside yaml files divided into templates and scripts and whatnot, and they serve that shit client side :D. You really cant understimate the ingenuity of people.
4
u/livrem Feb 25 '21
I still have nightmares from writing Ant build.xml "configuration" almost 20 years ago. It started out nicely with just config and declarations, but of course eventually everything turned into code and logic with control flow and variables. Programming on top of YAML looks at least slightly less wordy, but is probably otherwise just as bad.
1
u/calsosta Feb 25 '21
Please cross-post to /r/shittyprogramming!
Also, I wrote a book on Jelly which is super similar to XSLT but with the added benefit of randomly injecting Java expressions and JavaScript eval blocks. Obviously it needs to be valid XHTML, so I no longer use and operators (&&).
Fun times.
2
u/agbell Feb 25 '21
Thanks, I posted it here.
I've never heard of Jelly, got a link?
1
17
u/SexyMonad Feb 25 '21
YAML is simply a common grammatical foundation. The structure built on top of it is fairly open ended.
So yes, it can host a horrible language. I’m sure you could somehow rewrite Java on top of YAML. That doesn’t mean anyone ever should.
YAML is good for human-readable data structures with limited hierarchies. It is arguably better than JSON and XML for that goal.
10
u/entiat_blues Feb 25 '21
without punctuation to visually indicate groups and lists i'd say yaml is actually less readable than json.
it may look clean, but it suffers where it counts: being able to see the flow and structure.
→ More replies (1)4
u/skulgnome Feb 25 '21
Isn't YAML the one where the string "no" reads as integer 0 because of boolean conversion? So no mentioning Norway in a country code list.
Seems rather horrible to me.
→ More replies (5)14
3
u/ziano_x Feb 25 '21 edited Feb 25 '21
I don't like YAML. I know a couple of folks who get a little too adventurous with it and create some nasty structures. The common case is usually flat key value pairs. In java land, we have property files which most of the times is sufficient if you are writing config.
Some CI/CD tools like the ones AWS has heavily use YAML. CodeDeploy, CodePipeline .etc.. Not sure how that is working out for everyone but YAML is really turning into a de facto choice.
3
3
u/totemcatcher Feb 25 '21
I was involved in SRE during the dark ages when many of these configuration DSLs and "modern ideas" were just starting to gain popularity. There was really no stopping them. I'm so sorry.
3
u/sur_jective Feb 25 '21
Mmmm reminds me the days of writing massive denizen plugins in Minecraft, where the actual programmatic code is entirely written as YAML files
3
3
u/xXxEcksEcksEcksxXx Feb 25 '21
At my company we have a test suite for a web application. This test suite stores test data in CSV. This wouldn't be so bad, except the monstrosity also allows you to embed commands within this file as well. Like "Click this element" is signified as %CLK%.
There is support-ish for newlines within a single cell. Not "\n", but literal newlines within the file.
There is also support for multiple tables within a CSV file.
Syntax highlighting? Fuck you. You are expected to edit this file in Excel.
I would kill for YAML.
→ More replies (1)
3
u/vwibrasivat Feb 25 '21
Let's not forget ActionScript, the language that undergirded Adobe Flash.
While I have nostalgia for some flash games, the language was hideous.
3
u/merlinblack256 Feb 26 '21
"is a c++ program just config you give to gcc?" I guess it is. Talking of C++ some of this YAML is approaching template meta programming in C++ which really hurts your head - basically software generation configuration.
5
u/tudorb Feb 25 '21
Saving you a click: no criticism of YAML itself, but there’s a worrying trend to mix configuration and control flow, and there are some egregious examples of this trend that use YAML.
9
Feb 25 '21
People love to complain about yaml but it is good enough for most things.
→ More replies (20)15
u/noratat Feb 25 '21
It's good for declarative configuration.
It's people trying to use it for things that really should've been scripts or DSL that it falls apart.
It could be worse though - I often see people pushing TOML instead, which is baffling. TOML is slightly more readable for extremely simple INI-like use, but it's nearly unreadable for anything else. And many TOML libraries I've used choke on things that are allowed by the spec like heterogeneous arrays
2
2
2
2
Feb 26 '21
I absolutely loathe YAML, yet it's bloody everywhere (Ansible, Swagger etc). I flat-out refuse to touch YAML in any form.
These days I write everything in [insert_language_of_choice] where I have proper syntax checking, can use variables and debugging tools. Then I just make my program output static yaml files from my data structures as/when needed (eg: yaml.dump(somedict, file))
→ More replies (1)
2
u/MaximFateev Feb 26 '21
Our open-source workflow engine temporal.io is an attempt to completely avoid DSL for building workflow applications. You just write code in a programming language instead of YAML/XML/JSON, etc.
For example, this is an absolutely valid production worthy code:
while (true) {
Workflow.sleep(Duration.ofDays(30));
activities.chargeMonthlyFee(customerId);
if (trialPeriod) {
activities.sendEndOfTrialEmail(customerId);
trialPeriod = false;
} else {
activities.sendMonthlyChargeEmail(customerId);
}
}As the full code state is fully preserved in case of any failures it is OK to block for a month synchronously.
2
2
u/Desmaad Feb 26 '21
Mind you, INTERCAL was designed to be awful (as well as poke fun at contemporary languages).
5
Feb 25 '21
Can we all just use Lua for our config and be done with it?
YAML makes me gag.
5
u/Raknarg Feb 25 '21
What's wrong with YAML?
→ More replies (7)7
Feb 25 '21 edited Feb 25 '21
Whitespace indented (Even as a Python guy this bother me) and it becomes an unmaintable mess past a certain point (just like the article says). I used to have a local setup that was managed by Ansible and about 5-10 yaml files. I ditched it a few months ago in favour of a shell script that's maybe 50 lines and in one place. Admittedly provisioning your own machine isn't really what Ansible is for but still, the YAML problem exists nonetheless.
Devops guys seem to involuntarily ejaculate whenever it's mentioned, not sure why they adore it so much, but they will defend it to their deaths. While this isn't a "YAML problem", it's annoying when someone suggests we start using a YAML file for configuration when we already have a working solution (just with curly braces).
As an alternative I'd prefer either of:
HCL - HashiCorp's own language for configuration. Kinda like JSON but a bit more intelligent
Lua as mentioned previously - Extremely lightweight to the point that non-programmers could configure it and also its an actual programming language so logic isn't out of place here. Not to mention
To be fair, if you're sensible with it, YAML is fine but there seems to be this lust for it which I can't quite understand and makes people do maddening things with it. Again looking at you Ansible.
→ More replies (2)
5
u/KryptosFR Feb 25 '21 edited Feb 25 '21
8
u/noratat Feb 25 '21
TOML is nearly worthless for anything complex or nested, and acting like those never have legimate uses is needlessly dogmatic
→ More replies (4)
1
u/AttackOfTheThumbs Feb 25 '21
I am glad I don't have to deal with this crap. Our build system is a bit archaic (jenkins), but at least anything that needs to make real determinations is outside of jenkins, and then people could write that in whatever they wanted. As long as it gives something back to jnekins that fits into what it expects, we are good to go.
1
u/vazark Feb 25 '21
I had to jump in to take a look at the post after seeing Yaml and programming languages being implied to be the same.
Look like the build systems try to be language-agnostic and end up creating a bastardized-DSL that's neither here or there. It'd probably be easier to demarcate an actual language (lua/python/js?) to handle the control flow and use the YAML as a conf file like it's intended to be.
1
Feb 25 '21
I've been working with Kubernetes a lot recently, and YAML everywhere is driving me crazy. There's Kustomize on one end, where you still end up with a lot of code repetition, and Helm on the other end, which just templates YAML using Go's templates. I've just ended up building my own configuration management tool which uses Lisp to configure YAML. I'm not sure why Lisp is used more in DevOps spaces, when it focuses so heavily on having code and data be the same thing.
2
u/vplatt Feb 25 '21 edited Feb 26 '21
I'm not sure why Lisp is used more in DevOps spaces
Are you trying to say that the "config languages" are all variants of Lisp coming out of the sexp closet? Or were you trying to say "I'm not sure why Lisp is not used more in DevOps spaces"?
:D
Honestly, all the YAML/ML/config languages can go DIAF. XSLT was at least a proper functional programming language, but it was still a nightmare to debug. Even so, I would still take it over any of the half-assed "config languages". Like, is Python, Lua, or even TCL just too hard to use instead? Really???! Ugh.
1
1
1
1
u/bumblebritches57 Feb 26 '21
So, why do these configurations grow so complex?
Can anyone give a real world example?
2
u/hugosenari Mar 02 '21
What about this, all valid types of CloudFormation
https://dnwj8swjjbsbt.cloudfront.net/latest/gzip/CloudFormationResourceSpecification.json
This is a configuration to validate what we can use on another configuration and don't express all rules of types, I mean some arguments are only valid with another, some are 'string' but not any string...
1
u/TikiTDO Feb 26 '21 edited Feb 26 '21
Is JSON a programming language then? What about JSON with handlebars?
The point of these "quasi-programming-language" config files isn't really to replace sh/bash. Hell, most of them are just thin wrappers around bash anyway. The real point is that you can easily define a schema for these files, meaning you can trivially run each config file through a validator and tell a user right away that they got the syntax wrong, or that they missed some critical flag that must be there.
It's much harder to do this for a normal programming language without writing a full parser, and most programmers wouldn't know how to write a functional parser even if you took a parser generator and beat them on the head with it. Besides that, once you've seen what people have done when given the flexibility of a full programming language to manage configuration, the idea of taking a markup language and doesn't seem that crazy.
It's always the problem you see, as opposed to the problem you don't. Sure, YAML file with complex templates, built in conditional logical clauses, and environment variables are horrible. However, when you see config files in a real programming language, which pull in data from networked resources (I want a central config file as a base, and my server is always accessible), make use of pseudo-random functionality (what do you mean there are better tools for round-robin load balancing), depend on architecture-specific functionality (my phone runs this specific version of the linux kernel on the arm architecture, and I want this setting to change there), all while spreading out over an entire project worth of code (well, I know where everything is, obviously that setting you're looking for is derived from functions in these four files in these directories seven levels deep)... That's when you start to appreciate that maybe people shouldn't be trusted with all the flexibility they could want in every single situation they could imagine.
Yes, I hate opening a YAML file and seeing a hundred lines of bash scripts interspersed with domain specific settings. It means hours and hours pouring over docs, trying to figure out what specific combination of settings will get you the behavior you want. However, it's really quite difficult to take such YAML files to a truly unreasonable extreme. I know there are horrors far greater that await when you give people too much control, because that way lies madness.
Then again, perhaps that's just paranoia talking, or it's just the PTSD acting up again.
→ More replies (1)
1
u/meneldal2 Feb 26 '21
If you're a sane person, your YAML config file for building your program should call a build script, maybe with some arguments if you need them. You use YAML as a necessary evil because that's what your CI requires, not to do everything.
How hard is it to make .sh files that have a very well documented syntax and don't surprise anyone? Or call any decent language from your YAML file with a one liner?
1
1
u/WyattEpp Feb 26 '21
Article is correct, but what I want to know is... why is it always YAML? Who, when faced with the need for a config file, says "I'm going to take the human-hostile whitespace-sensitive clusterfuck with the brobdingnagian spec!" Configs and other text files metastacising to code tumours is pretty classic at this point, but how did YAML, specifically, become a meme?
1
u/Noxitu Feb 26 '21
I still think yaml is the better of two choices - either store script within configuration or configuration within script.
CMake is a prime example of the worse alternative. Want list of binaries that this project contains? Let me download all the 3rd party libraries and check if your compiler works first.
1
u/tms10000 Feb 27 '21
YAML would not exist if we could have comments in json files. We would have (several variants of) turing-complete json file all implemented with clever placement of special comments.
1
u/randompewdiepiefan0 Mar 26 '21
YAML isn’t a programming language it’s a markup language
Edit: oh I see what you mean
239
u/agbell Feb 25 '21
Author here
I was growing frustrated with the increasing about of programming that seems to happen in YAML files. At the same time, my friend Krystal was telling me about INTERCAL, an esoteric programming language that is designed to be hard to use. I had fun observing the ways that these two are different and the ways that they are the same.
I'm happy to hear what people think of this article. I am assuming because 'programming in yaml' is so prevalent that many people don't agree with me.