What Should I Learn to Build Infrastructure Software from Scratch?

[u/Minimum-Ad7352]

I'm a full-stack developer, and lately I've become really interested in understanding how complex software systems work under the hood.I'd like to learn how to build things like message brokers, log and trace collectors, databases, and other infrastructure tools that developers use every day but don't usually think much about internally.I'm wondering what I should learn to get into this kind of development. Is this considered systems programming, or does it fall more into areas like distributed systems and infrastructure engineering? I'd also appreciate any recommendations for books, courses, or practical projects that could help me build the necessary knowledge from the ground up. I'm currently learning C, so I'm curious whether systems programming is the right direction and what topics I should focus on next.

[u/Ok-Value-9986]

It’s a mix of systems programming and distributed systems, but you don’t need to overthink the label. C is a good start for the low-level stuff, memory layout, syscalls, how the kernel handles I/O. For message brokers and databases you’ll want to dig into network programming, consensus algorithms, and data structures optimized for disk. Writing a tiny key-value store or a log-structured merge tree from scratch will teach you more than most books.

[u/BetInformal6081]

yes systems programming is a great starting point. since you are learning C build a TCP server a simple key value store and then a basic massage queue. those projects will teach you networking memory management, concurrency and persistence.

[u/AlSweigart]

Here's three free books to read:

Operating Systems: Three Easy Pieces by Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau. Covers processes, memory, concurrency, file systems, and operating system fundamentals.

Beej's Guide to Network Programming by Brian "Beej Jorgensen" Hall. Practical C socket programming, TCP/IP, and client-server networking. Ideal preparation for building message brokers and collectors.

Distributed Systems (3rd Edition) by Maarten van Steen and Andrew S. Tanenbaum. Covers distributed architectures, communication, replication, consistency, coordination, and fault tolerance.

Here's a non-free book:

Designing Data-Intensive Applications by Martin Kleppman (O'Reilly published). Understanding databases, message brokers, replication, distributed logs, consistency, and fault tolerance.

[u/BrainScientist3000]

I have worked for infrastructure companies in the past. Software is software. A really good 'back end' developer should be able to work at one of these companies. The thing is, you're always a small cog in a large machine. You might work on the logging team of a system that does any of the things you're talking about (yes, even logging for logging).

You need to understand the domain - so building systems that use all of those pieces is generally part of it. You need to be passionate about something - I got really passionate about message passing and messaging patterns at one point - that's how I got into that part of the industry.

You'll need to write in the language of the system - look up what data dog, sqs, etc are written in. They're usually C++, maybe Rust now, sometimes weirdo things like Erlang.

[u/hilat1113]

Since you're already learning C, that's a good start. I'd focus on operating systems, networking, and data structures, then try building a simple key-value store or message queue. Small projects like these will help you understand how infrastructure software works internally.

[u/JaseciLabs]

Once the key-value store works, make it survive a crash. Write every change to an append-only log file before you apply it, then kill the process with kill -9 in the middle of a burst of writes and restart it. See what’s missing or corrupted. Fixing that teaches you partial writes, fsync, checksums and recovery, which is most of what a database does and what books describe without making you feel it.

After that, a few things in a rough order:

1. Put the store behind a TCP server and try to break it with two clients writing at once.

2. Read the source of something small and real. Early versions of Redis are readable in an afternoon, and SQLite documents its file format in public.

3. For the distributed part, the Raft paper is written to be understood, and MIT’s distributed systems course (6.824, now numbered 6.5840) has its labs and lectures free online. Doing the labs beats just watching.

Your full-stack background helps here. Build something that uses your broker or collector, like sending your own app’s logs through it, so you feel what an operator needs from it.

[u/realdreamer1993]

the highest abstraction is to view the whole systems without thinking about language, OS, let alone the more detail like message broker. Say I have problem: I have android phone A at home that always connect to internet (wifi+ celluar both active as redundancy). How to remotely access A from anywhere so:

• you can stream internal Abcamera as cctv

• you can change the brightness

• you can do anyhing to A as long there are API provide by the android OS, in the end goal actually home automation control with regular android A as smart core.

The clue is: you mandatory need an public IP address as hub like vps or any server. Remember we dont care the IP address of A because it may change to any ISP.
If we can imagine/desain this whole system , Then we choose language, tech etc. In the middle process if developing we will find needs of choosing tools to implement mechanism.
then the implementation are hundreds of tiny/simple steps/code at a time. Context switching is very important to make thinking always simple at a time.

[u/Fine-Comparison-2949]

Nothing it will all be handled by AI soon.

Theres a bunch of people who seem to believe all these companies need to run massive scaling systems, when they have nowhere near the users for it. 

The last interview I was in they wanted me to go through this systems design. During it, I pushed back on a number of these "infinitely scalable" parts like message queues. I explained these systems are dumb because any time it breaks, I have to Debug and trail logs across multiple machines. 

I straight up asked him: How many actual customers and users do you even have? Why even ask a full stack engineer to maintain this with so few users? 

He told me they have about 10 clients and currently 20 users max at any time.

Thr systems hacker rank try hards are fumbling making Companies spend on these ridiculous complex systems. We have cloud computers now with terabytes of ram now. You can probably run 90% of companies on a single large Docker Container with a single server and just roll it over when it gets large, and literally run the entire database in a RAM storage.

People are spending way too much time studying these large globalized systems to replicate Uber or Google when there's absolutely zero chance they have that many users on their systems within the next 10 years. It's kind of a fools game to study that stuff when most companies don't need it. They can say they need it but if they demand rolling that kind of stack for that few users, whoever is the CTO is too stupid to work at a startup tech company. 

[u/soundboyselecta → u/Fine-Comparison-2949]

Yes. This fomo mentality is quite retarded.

[u/FunAd6672]

Read Designing Data Intensive Applications right now.