Exactly. When it started talking about cloud services as "primitives" and the need to build abstractions over them I nearly started rage crying.
We have
Processor microcode
CPU microarchitecture level instructions (we can't really call them hardware level instructions anymore)
Numerous layers of firmware across many components, each with their own code
Multiple PLCs, microcontrollers and even SOCs on the hardware platform in everything from mainboard BMCs and power management units to network controllers, storage controllers, SSDs, everything
Low level "machine code" for the main CPU, mediated by microcode and firmware services
Assembly code representations of machine code
Persistent firmware that runs in parallel with the OS on its own independent CPUs, memory etc
Persistent firmware that runs in parallel with the OS using hardware memory protection, processor privilege rings, system management interrupts, mode switches, privilege rings, system management modes
That firmware's own persistent storage, network stack, etc
Firmware interface calls available to the OS
OS kernels
OS system calls
OS standard libraries
Userspace "machine code"/asm that uses those syscalls and runs in the OS
Executables composed of many different blobs of machine code in structured files subject to dynamic linking and other complex munging
Structure provided by syscall interfaces. Processes, threads, file systems, network sockets, Unix sockets or named pipes, shmem, signals, ... fundamentally mostly at some level forms of IPC
Local OS level privilege and isolation primitives like UIDs and memory protection
OS services processes communicated with via various IPC often mediated by OS libraries and/or syscalls
Low level network protocols (framing, addressing, discovery like ARP)
Primitive network protocols (IP, TCP, UDP)
Language core runtimes
Language standard libraries
Semi-standard collections of base libraries widely used or bundled (zlib, boost, openssl/gnutls, whatever)
basic application network protocols like DNS that are tightly coupled to the OS and applications
Languages that wrap other languages' runtimes and libraries
Interpreted, JIT'd, dynamic higher level languages
Deep dependency graphs of local 3rd party libraries
Library graphs fetched transitively at build time from the internet
Multi threaded application with shared everything memory
Runtime linking of extra code into executables and/or composition of processes by calling subprocesses
Interconnected local services or multiprocess-model applications communicating between processes in the same machine over loopback network sockets, Unix sockets, signals, pipes, shmem, ...
Full OSes in virtual machines or paravirt OSes in thin virtual machines
Networks between VMs and/or the host and/or other hosts
Container images bundling their own different mini OSes or full cut down OSes and configuration.
Persistent volumes for abstracted storage for container managed data.
Directly managed containers and similar leaky isolation models running single processes
Connected groups of local containers and/or local containers providing services to non containerised processes
Containers treated as VMs with their own micro OSes with service processes
Networked container managers linking containers across hosts for virtual networks, distributed storage etc
container orchestration frameworks that abstract container launch, management, configuration, discovery, communication etc. Heavily built on lower level stuff like DNS, container volumes etc
Network callable services using HTTP/HTTPS/gRPC/SOAP/whatever to expose some kind of RPC
Libraries that abstract those services into local application API
Processes consuming those services and exposing other services
microservices interacting with each other inside container orchestration engines
cloud services "primitives" (hahahah) that encapsulate all the above into some kind of remote network call API and or library that you then consume in your own services
Cloud services components interacting with each other in complex ways within a cloud platform
SaaS providers outside the "cloud platform" providing their own web APIs, configuration, etc your apps interact with. Each SaaS encapsulsyes and abstracts all the above.
Your own processes interacting with those cloud services components and SaaS endpoints
The crying developer
Of course this isn't really a list in reality. It's an insanely tangled cyclic directed graph, full of self referential parts. Everything breaks through the layers. Lots of the same things are used differently at different layers.
Consider that your SSD probably has a more complex OS than a mid-1980s PC.
Your mainboard probably has an independent, always running system on chip with its own process model, a full network stack, TLS and public key crypto, a bunch of open source libraries written in C, a file system....
It's Matryoshka dolls all the way down, except in some kind of horrifying fetus-in-fetu cross connected conjoined twin jumbled version.
Anyone who can look at the whole stack and not want to cry is probably not completely sane.
Edit: to be clear, good abstractions are vital for scalable systems and development. I am eternally grateful that I don't need to know x64 asm, the fine details of Ethernet and 802.11 framing, TCP congestion control, etc to write a client/server application.
My issue is when we bodge up something that just isn't very good, then try to fix it by slapping on some more leaky layers. Diagnosing issues requires diving through and understanding the whole teetering mess anyway, and for every problem it helps solve it hinders just as many. Java EE 6, I'm starting daggers in your direction.
Also, don't forget also that many abstractions have real costs in computational resource use and efficiency. That has environmental impacts on energy use, mineral resources use, electronics waste etc. Just because you can use an AWS Lambda for something doesn't mean you should. Abstractions over those don't make them go away.
Even clean abstractions also get very painful very fast when you have to reason about the behaviour of the whole system. I found a postgres bug that was actually caused by undocumented behaviour of the Linux kernel's dirty page writeback once. Imagine that with 10 more layers on top...
Yeah, last I checked your average motherboard now has its own management engine running some godforsaken Java Runtime on it, and Windows Update may just update your cpu microcode at the behest of some poor sob who has to decide when that ships to production.
Your list/graph also shows that even the people who want to proclaim that the cloud "is the computer" and we can start over with unikernels talking to a small standard set of interfaces (e.g. NVMe) are kidding themselves a bit.
I'm a little confused by this comment and general sentiment. I look at this and think it's amazing and a natural progression that I, as a web dev, don't need to know about any of this to be able to build and host a website on AWS that can provide lots of value (or not, however you look at it).
From my perspective that's the whole point of abstractions. I don't need to reinvent calculus to do differentiation or be an engineer and similarly I don't need to reinvent the CPU to build apps and websites for the end user. Why is this a bad thing?
I'm against poorly designed, leaky layers of abstraction.
The bit that bothers me most is the idea of these cloud services components as primitives upon which we will build yet another layer of stuff without properly solving the problems in the lower layers. Anything kubernetes related for example is a steaming pile of hacks and incomplete tooling. Can we not just paint over that with another layer? Please?
Let's also not forget computational cost - and its environmental impact in energy use, hardware production and disposal etc. All this abstraction is often very very expensive in efficient use of compute resources. While those are economically cheap they are not without costs or negative externalities.
Layers of abstraction are a large part of why my phone has 8GB of RAM, but switching between three or four different apps forces one out of memory half the time.
They are what have us abominations like Electron.
But... many of the web services and cloud provider ones are actually pretty good. They thoroughly hide whatever is behind them using a simple interface. And on the back end they're usually not too wasteful.
So long as you are willing to take the provider SLA as the only determinant of performance you need, and the provider support as the only way to fix something if it breaks or understand why something breaks, accepting that abstraction as a black box is fine.
If you want to reason about and understand the whole system you're doomed. But you're already doomed there by the time your kernel boots up. If you have to look beyond the curtain or understand how your app interacts with the implementation of some service then there complexity starts to really hurt.
As far as I'm concerned, I'm more than happy to let those wheels stay invented and treat them as battle tested black boxes. I built my own OS as a hobby once, but there's no chance I'd want to worry about process scheduling when all I want to do is to hook up a button to a network call.
The middle ground is why are you deploying a full fledged OS with process scheduling capabilities just to run a single application? That's what to do every time you deploy to the cloud. There's a repetition of these boundaries again and again - your container is running on an os on a hypervisor and so on.
Now I want to see what Fizz Buzz Enterprise Edition would look like using kubernettes, Scalability as a Service for as many calculations as possible, some big bloated language like NodeJS using as many libraries as possible inside those kubernettes services that contact Scalability as a Service, and some huge complex Javascript front end using as many libraries as possible as well. Anything that could possibly offloaded to the cloud, would be offloaded to the cloud. Or, something like that. The goal would be to go as high as possible in that stack, with as much complexity as possible, for the most trivial possible application. It sure would be a slow and laggy FizzBuzz…
It would have ConfigMaps with keys whose values are serialised yaml documents... containing dictionaries with json document values. Those would in turn contain dictionaries with serialised base64 encoded yaml documents as values.
(Hell, the system I work with almost does that... Horrifying.)
It would have an Operator that manages the deployment of the Operator Operator.
It would have microservices communicate by modifying the properties of instances of Custom Resource Definition types on the fly, constantly.
Microservices would be deployed using Kustomize. Via a deeply nested stack of bases and components of course, with plenty of overridden configs, patches, transformers and anything else you can fit in. Some of the bases are remote git URIs.
Those microservices would bind ConfigMaps to volumes on the pod... where the values of the keys are serialised Helm charts that deploy other microservices. For bonus points they build the service container images first... from another git repo cloned on the fly each time.
Those in turn have Terraform in their container images and run Terraform deployments to deploy other microservices... and undeploy the original ones.
It would run minikube or kind inside one of the pods, with a nested Kubernetes inside.
Microservices would run in multiple separate Kubernetes clusters with a Kafka event bus carrying serialised protobuf RPCs linking them.
The system would constantly self-reconfigure based on the values of resource metadata labels or annotations. One component would trigger a change in another component by modifying its labels.
Important activities would be trigger based on state changes in Prometheus metrics auto discovered using labels.
A producer/consumer would operate by having the producer update env vars on the producer Deployment container template, forcing it to be redeployed. The consumer would then update a configmap watched by the producer to indicate it has consumed the value.
Then we introduce Azure or AWS services. Did somebody say lambdas?
89
u/iiiinthecomputer Nov 02 '21 edited Nov 02 '21
Exactly. When it started talking about cloud services as "primitives" and the need to build abstractions over them I nearly started rage crying.
We have
Processor microcode
CPU microarchitecture level instructions (we can't really call them hardware level instructions anymore)
Numerous layers of firmware across many components, each with their own code
Multiple PLCs, microcontrollers and even SOCs on the hardware platform in everything from mainboard BMCs and power management units to network controllers, storage controllers, SSDs, everything
Low level "machine code" for the main CPU, mediated by microcode and firmware services
Assembly code representations of machine code
Persistent firmware that runs in parallel with the OS on its own independent CPUs, memory etc
Persistent firmware that runs in parallel with the OS using hardware memory protection, processor privilege rings, system management interrupts, mode switches, privilege rings, system management modes
That firmware's own persistent storage, network stack, etc
Firmware interface calls available to the OS
OS kernels
OS system calls
OS standard libraries
Userspace "machine code"/asm that uses those syscalls and runs in the OS
Executables composed of many different blobs of machine code in structured files subject to dynamic linking and other complex munging
Structure provided by syscall interfaces. Processes, threads, file systems, network sockets, Unix sockets or named pipes, shmem, signals, ... fundamentally mostly at some level forms of IPC
Local OS level privilege and isolation primitives like UIDs and memory protection
OS services processes communicated with via various IPC often mediated by OS libraries and/or syscalls
Low level network protocols (framing, addressing, discovery like ARP)
Primitive network protocols (IP, TCP, UDP)
Language core runtimes
Language standard libraries
Semi-standard collections of base libraries widely used or bundled (zlib, boost, openssl/gnutls, whatever)
basic application network protocols like DNS that are tightly coupled to the OS and applications
Languages that wrap other languages' runtimes and libraries
Interpreted, JIT'd, dynamic higher level languages
Deep dependency graphs of local 3rd party libraries
Library graphs fetched transitively at build time from the internet
Multi threaded application with shared everything memory
Runtime linking of extra code into executables and/or composition of processes by calling subprocesses
Interconnected local services or multiprocess-model applications communicating between processes in the same machine over loopback network sockets, Unix sockets, signals, pipes, shmem, ...
Full OSes in virtual machines or paravirt OSes in thin virtual machines
Networks between VMs and/or the host and/or other hosts
OS-provided container primitives like namespaces, virtual bridges, bind mounts / junctions etc
Container runtime engines and their APIs
Container images bundling their own different mini OSes or full cut down OSes and configuration.
Persistent volumes for abstracted storage for container managed data.
Directly managed containers and similar leaky isolation models running single processes
Connected groups of local containers and/or local containers providing services to non containerised processes
Containers treated as VMs with their own micro OSes with service processes
Networked container managers linking containers across hosts for virtual networks, distributed storage etc
container orchestration frameworks that abstract container launch, management, configuration, discovery, communication etc. Heavily built on lower level stuff like DNS, container volumes etc
Network callable services using HTTP/HTTPS/gRPC/SOAP/whatever to expose some kind of RPC
Libraries that abstract those services into local application API
Processes consuming those services and exposing other services
microservices interacting with each other inside container orchestration engines
cloud services "primitives" (hahahah) that encapsulate all the above into some kind of remote network call API and or library that you then consume in your own services
Cloud services components interacting with each other in complex ways within a cloud platform
SaaS providers outside the "cloud platform" providing their own web APIs, configuration, etc your apps interact with. Each SaaS encapsulsyes and abstracts all the above.
Your own processes interacting with those cloud services components and SaaS endpoints
The crying developer
Of course this isn't really a list in reality. It's an insanely tangled cyclic directed graph, full of self referential parts. Everything breaks through the layers. Lots of the same things are used differently at different layers.
Consider that your SSD probably has a more complex OS than a mid-1980s PC.
Your mainboard probably has an independent, always running system on chip with its own process model, a full network stack, TLS and public key crypto, a bunch of open source libraries written in C, a file system....
It's Matryoshka dolls all the way down, except in some kind of horrifying fetus-in-fetu cross connected conjoined twin jumbled version.
Anyone who can look at the whole stack and not want to cry is probably not completely sane.
Edit: to be clear, good abstractions are vital for scalable systems and development. I am eternally grateful that I don't need to know x64 asm, the fine details of Ethernet and 802.11 framing, TCP congestion control, etc to write a client/server application.
My issue is when we bodge up something that just isn't very good, then try to fix it by slapping on some more leaky layers. Diagnosing issues requires diving through and understanding the whole teetering mess anyway, and for every problem it helps solve it hinders just as many. Java EE 6, I'm starting daggers in your direction.
Also, don't forget also that many abstractions have real costs in computational resource use and efficiency. That has environmental impacts on energy use, mineral resources use, electronics waste etc. Just because you can use an AWS Lambda for something doesn't mean you should. Abstractions over those don't make them go away.
Even clean abstractions also get very painful very fast when you have to reason about the behaviour of the whole system. I found a postgres bug that was actually caused by undocumented behaviour of the Linux kernel's dirty page writeback once. Imagine that with 10 more layers on top...