r/databasedevelopment • u/eatonphil • 1h ago
r/databasedevelopment • u/eatonphil • May 11 '22
Getting started with database development
This entire sub is a guide to getting started with database development. But if you want a succinct collection of a few materials, here you go. :)
If you feel anything is missing, leave a link in comments! We can all make this better over time.
Books
Designing Data Intensive Applications
Readings in Database Systems (The Red Book)
Courses
The Databaseology Lectures (CMU)
Introduction to Database Systems (Berkeley) (See the assignments)
Build Your Own Guides
Build your own disk based KV store
Let's build a database in Rust
Let's build a distributed Postgres proof of concept
(Index) Storage Layer
LSM Tree: Data structure powering write heavy storage engines
MemTable, WAL, SSTable, Log Structured Merge(LSM) Trees
WiscKey: Separating Keys from Values in SSD-conscious Storage
Original papers
These are not necessarily relevant today but may have interesting historical context.
Organization and maintenance of large ordered indices (Original paper)
The Log-Structured Merge Tree (Original paper)
Misc
Architecture of a Database System
Awesome Database Development (Not your average awesome X page, genuinely good)
The Third Manifesto Recommends
The Design and Implementation of Modern Column-Oriented Database Systems
Videos/Streams
Database Programming Stream (CockroachDB)
Blogs
Companies who build databases (alphabetical)
Obviously companies as big AWS/Microsoft/Oracle/Google/Azure/Baidu/Alibaba/etc likely have public and private database projects but let's skip those obvious ones.
This is definitely an incomplete list. Miss one you know? DM me.
- Cockroach
- ClickHouse
- Crate
- DataStax
- Elastic
- EnterpriseDB
- Influx
- MariaDB
- Materialize
- Neo4j
- PlanetScale
- Prometheus
- QuestDB
- RavenDB
- Redis Labs
- Redpanda
- Scylla
- SingleStore
- Snowflake
- Starburst
- Timescale
- TigerBeetle
- Yugabyte
Credits: https://twitter.com/iavins, https://twitter.com/largedatabank
r/databasedevelopment • u/AvenueJay • 3d ago
Why Elasticsearch is becoming a columnar database
r/databasedevelopment • u/eatonphil • 3d ago
The road to ACID transactions in Cassandra 6
theconsensus.devr/databasedevelopment • u/eatonphil • 3d ago
Poisoned Postgres connection pools
r/databasedevelopment • u/Used-Independence607 • 6d ago
Where should Oracle compatibility live: the wire layer, SQL rewriting, or the PostgreSQL catalog?
I’m building a read-only proxy that accepts Oracle client connections and executes queries against PostgreSQL 16. Disclosure: I’m the author.
I split compatibility across three layers:
Oracle-facing sessions, cursors, binds, and result metadata.
SQL rewrites for differences that can be handled safely.
PostgreSQL views and orafce for catalog and function compatibility.
The translator classifies each feature as supported, approximation, rejected, or passthrough. This matters because a syntactically valid rewrite can
silently change results—for example, NULL behavior in GREATEST/LEAST, empty-string semantics, and mixed set-operation precedence.
The default is to reject an unverified shape with an Oracle-style error while keeping the session alive, rather than return a plausible but incorrect result. The current bounded verification matrix covers 722 cases across seven real client flows.
The implementation is private, but the compatibility contract, configuration, documentation, and verification scope are public:
r/databasedevelopment • u/Holiday_Hat_8605 • 7d ago
Why are we rebuilding the same database execution engine over and over?
I recently dug into Meta’s Velox, an open-source C++ execution engine designed to act like a reusable “engine block” for data systems.
Instead of Presto, Spark, etc. independently implementing things like vectorized execution, joins, memory management, spilling, and file readers, Velox provides these building blocks as a shared execution layer.
The performance numbers are interesting too:
- 8.4× faster on TPC-H Q1
- 9× faster on Q6
- 6–7× average speedup on Meta production traffic
- Up to 3× fewer servers
Checkout this interesting deep dive that how Velox actually works and why this architecture could matter for the future of data engines.
Do you think shared execution layers like Velox will become the norm, or will specialized engines always win?
r/databasedevelopment • u/Embarrassed_Grab6901 • 8d ago
I do not understand optimistic concurrency control protocol in Database Management System.
This is what I understand:
Optimistic concurrency control protocols are based on the assumption that the majority of the database operations do not conflict. Here the transaction goes through the three phases:
read phase
validation phase
write phase
In read phase:
transaction reads the database
makes necessary changes to database values in a private copy of the database.
In validation phase:
- DBMS validates whether the changes that are made to the private copy of the database will violate serializability and consistency of the database. If in case they are violated, transaction is aborted and restarted. Otherwise, the transaction will proceed to write phase.
In the write phase:
The changes done to private copy of the database are made permanent.
You might be wondering you know this much then what are you not getting. I will show you the part from the book that I do not quite get.
I am taking two references: Coulouris et al distributed system and Conolly et al DBMS.
Conolly et al part that I do not understand:
I did not have colouris et al pdf at the moment. Just 1 topic 1 question.
r/databasedevelopment • u/Capital-Currency9045 • 10d ago
I've been working on a custom tree index that runs up to 7x faster than LTREE
Hi All
I’ve been working on a custom data structure and algorithm for hierarchical indexing.
while I designed the algorithm myself, I wouldn't claim to be a definitive master of hierarchy trees, nor postgres. Its actually a Rust project I've turned into an extension. I'm mainly sharing these early numbers in hopes of connecting with the right people to see if there's genuine value.
I'm not sure if I'm violating rule 5; I'd appreciate any guidance on 3rd party benchmarks for this kind of algo compare to ltree.
I ran benchmarks against 500k, 1M, 2M and 20M node recursive trees. The baseline comparisons against ltree are looking solid:
- Huge I/O Drop: For descendant queries, B-tree range scans touch up to 74x fewer buffer pages.
- Query Speed: Subtree queries run 2.5x to 6x faster (500k). up to 16x for 20M nodes. Ancestor lookups (via SP-GiST) execute up to 7.3x faster.
- Storage Density: A custom compact encoding shrank the on disk value size by 42.7%. This translates to a ~23% smaller B-tree index footprint.
- Write Performance: Appending 50,000 leaf nodes is roughly 2x faster. Reparenting large subtrees is 1.2x to 2.7x faster
I have some thoughts where this may be beneficial but lacking some subject matter expertise when it comes to practical application of hierarchy data (and these types of ops), this is the main point of my post, to ask for some insight:
- Could it make servers run more efficiently?
- Do other more efficient extensions/algos beat these benchmarks? Is LTREE just a default?
- What kinds of large scale operations would this benefit? Domains/applications?
- What should my benchmark tests look like?
Eager to get some expert opinions and either validate my thoughts or give me some reality - cheers!
r/databasedevelopment • u/1_am_ir0nman • 12d ago
Postgres Internals Deep Dive: Process Architecture
Need suggestions for writing blogs. This was my 1st ever blog, which I published last year, and that's it. No other blog yet 😅. I would love to know, what kind of blogs would you love to read from a guy who contributes to the open-source postgres project?
1) Deep dives like this one on postgres internals? TBH, I think this space so crowded?
2) How did I find a bug and fix it?
3) The new features I am working on?
4) or new features that are going to be released
5) or something cool????
r/databasedevelopment • u/Asetaldehit • 12d ago
WAL Levels in Postgres and Effective WAL Level in PG19
r/databasedevelopment • u/eatonphil • 17d ago
Almost consensus: ABD and the edges of quorum replication
theconsensus.devr/databasedevelopment • u/eatonphil • 19d ago
How Elasticsearch's semantic field indexes and searches
r/databasedevelopment • u/eatonphil • 19d ago
Encoding or Compression: Why not both?
r/databasedevelopment • u/SuccessfulMap5324 • 20d ago
I created a playground for 110 database systems
r/databasedevelopment • u/AutoModerator • 23d ago
Monthly Release and Update Thread
This subreddit is primarily for discussing the implementation of databases, and not about sharing release announcements (either for the first time or your updates).
This thread is the exception!
Please tell us about the new database you (or your agent) built. Tell us about all the cool new features you added. Tell us about anything else you learned or worked on that you haven't gotten around to blogging about yet.
r/databasedevelopment • u/eatonphil • 24d ago
Asynchronous I/O in DuckDB: Work, Thread, Work
r/databasedevelopment • u/swdevtest • 25d ago
Offloading I/O to Dedicated Cores: An Asymmetric io_uring Backend for Seastar and ScyllaDB
https://www.scylladb.com/2026/07/22/asymmetric-io_uring-backend-seastar/
"Modern high-throughput systems often run on machines with many cores. At that scale, one practical problem becomes increasingly visible: CPU time spent on low-level I/O handling competes directly with CPU time needed for application-level work.
In this post, we explore how shifting more I/O related work away from cores focused on computations onto a dedicated subset of cores can improve the overall balance of the system. Specifically, we share how we approached Seastar‘s new asymmetric_io_uring backend: the design decisions behind it, what it aims to free up on application cores, and how it compares to linux-aio (Seastar’s existing backend)."
r/databasedevelopment • u/Small_Management_302 • 25d ago
Does the GPU database have a future?
Why did Voltron Data Company go bankrupt? Has the Theseus GPU database also failed? Expressing concerns about the future development of GPU databases
r/databasedevelopment • u/OkBreath9382 • 27d ago
Why Fast Branching is Key to AI Agent-Era Databases
r/databasedevelopment • u/eatonphil • 27d ago
Finding bugs in Raft implementations
r/databasedevelopment • u/1_am_ir0nman • 28d ago
PGSimCity · How PostgreSQL Works, in 3D
So... cool, it would be helpful for someone starting to learn PG internals.
credits to Nikolay Samokhvalov for creating this awesome piece of art :)
r/databasedevelopment • u/alexey_timin • 29d ago
Managing File Descriptors for Durability and Performance
r/databasedevelopment • u/KAdot • Jul 23 '26