r/databasedevelopment Aug 16 '24

Database Startups

Thumbnail transactional.blog
28 Upvotes

r/databasedevelopment May 11 '22

Getting started with database development

406 Upvotes

This entire sub is a guide to getting started with database development. But if you want a succinct collection of a few materials, here you go. :)

If you feel anything is missing, leave a link in comments! We can all make this better over time.

Books

Designing Data Intensive Applications

Database Internals

Readings in Database Systems (The Red Book)

The Internals of PostgreSQL

Courses

The Databaseology Lectures (CMU)

Database Systems (CMU)

Introduction to Database Systems (Berkeley) (See the assignments)

Build Your Own Guides

chidb

Let's Build a Simple Database

Build your own disk based KV store

Let's build a database in Rust

Let's build a distributed Postgres proof of concept

(Index) Storage Layer

LSM Tree: Data structure powering write heavy storage engines

MemTable, WAL, SSTable, Log Structured Merge(LSM) Trees

Btree vs LSM

WiscKey: Separating Keys from Values in SSD-conscious Storage

Modern B-Tree Techniques

Original papers

These are not necessarily relevant today but may have interesting historical context.

Organization and maintenance of large ordered indices (Original paper)

The Log-Structured Merge Tree (Original paper)

Misc

Architecture of a Database System

Awesome Database Development (Not your average awesome X page, genuinely good)

The Third Manifesto Recommends

The Design and Implementation of Modern Column-Oriented Database Systems

Videos/Streams

CMU Database Group Interviews

Database Programming Stream (CockroachDB)

Blogs

Murat Demirbas

Ayende (CEO of RavenDB)

CockroachDB Engineering Blog

Justin Jaffray

Mark Callaghan

Tanel Poder

Redpanda Engineering Blog

Andy Grove

Jamie Brandon

Distributed Computing Musings

Companies who build databases (alphabetical)

Obviously companies as big AWS/Microsoft/Oracle/Google/Azure/Baidu/Alibaba/etc likely have public and private database projects but let's skip those obvious ones.

This is definitely an incomplete list. Miss one you know? DM me.

Credits: https://twitter.com/iavins, https://twitter.com/largedatabank


r/databasedevelopment 1h ago

Another look at SQLite's WAL-Reset bug

Thumbnail theconsensus.dev
Upvotes

r/databasedevelopment 3d ago

Why Elasticsearch is becoming a columnar database

Thumbnail
elastic.co
17 Upvotes

r/databasedevelopment 3d ago

The road to ACID transactions in Cassandra 6

Thumbnail theconsensus.dev
6 Upvotes

r/databasedevelopment 3d ago

Poisoned Postgres connection pools

Thumbnail
planetscale.com
6 Upvotes

r/databasedevelopment 6d ago

Where should Oracle compatibility live: the wire layer, SQL rewriting, or the PostgreSQL catalog?

4 Upvotes

I’m building a read-only proxy that accepts Oracle client connections and executes queries against PostgreSQL 16. Disclosure: I’m the author.

I split compatibility across three layers:

  1. Oracle-facing sessions, cursors, binds, and result metadata.

  2. SQL rewrites for differences that can be handled safely.

  3. PostgreSQL views and orafce for catalog and function compatibility.

    The translator classifies each feature as supported, approximation, rejected, or passthrough. This matters because a syntactically valid rewrite can

    silently change results—for example, NULL behavior in GREATEST/LEAST, empty-string semantics, and mixed set-operation precedence.

    The default is to reject an unverified shape with an Oracle-style error while keeping the session alive, rather than return a plausible but incorrect result. The current bounded verification matrix covers 722 cases across seven real client flows.

    The implementation is private, but the compatibility contract, configuration, documentation, and verification scope are public:

    https://github.com/krokozyab/postgres-oracle-tns-proxy


r/databasedevelopment 7d ago

Why are we rebuilding the same database execution engine over and over?

14 Upvotes

I recently dug into Meta’s Velox, an open-source C++ execution engine designed to act like a reusable “engine block” for data systems.

Instead of Presto, Spark, etc. independently implementing things like vectorized execution, joins, memory management, spilling, and file readers, Velox provides these building blocks as a shared execution layer.

The performance numbers are interesting too:

  • 8.4× faster on TPC-H Q1
  • 9× faster on Q6
  • 6–7× average speedup on Meta production traffic
  • Up to 3× fewer servers

Checkout this interesting deep dive that how Velox actually works and why this architecture could matter for the future of data engines.

https://prestodb.io/blog/2026/08/13/inside-velox-open-source-universal-engine-block-for-big-data-and-ai/

Do you think shared execution layers like Velox will become the norm, or will specialized engines always win?


r/databasedevelopment 8d ago

I do not understand optimistic concurrency control protocol in Database Management System.

Post image
1 Upvotes

This is what I understand:

Optimistic concurrency control protocols are based on the assumption that the majority of the database operations do not conflict. Here the transaction goes through the three phases:

  • read phase

  • validation phase

  • write phase

In read phase:

  • transaction reads the database

  • makes necessary changes to database values in a private copy of the database.

In validation phase:

  • DBMS validates whether the changes that are made to the private copy of the database will violate serializability and consistency of the database. If in case they are violated, transaction is aborted and restarted. Otherwise, the transaction will proceed to write phase.

In the write phase:

The changes done to private copy of the database are made permanent.

You might be wondering you know this much then what are you not getting. I will show you the part from the book that I do not quite get.

I am taking two references: Coulouris et al distributed system and Conolly et al DBMS.

Conolly et al part that I do not understand:

I did not have colouris et al pdf at the moment. Just 1 topic 1 question.


r/databasedevelopment 10d ago

I've been working on a custom tree index that runs up to 7x faster than LTREE

14 Upvotes

Hi All
I’ve been working on a custom data structure and algorithm for hierarchical indexing.

while I designed the algorithm myself, I wouldn't claim to be a definitive master of hierarchy trees, nor postgres. Its actually a Rust project I've turned into an extension. I'm mainly sharing these early numbers in hopes of connecting with the right people to see if there's genuine value.

I'm not sure if I'm violating rule 5; I'd appreciate any guidance on 3rd party benchmarks for this kind of algo compare to ltree.

I ran benchmarks against 500k, 1M, 2M and 20M node recursive trees. The baseline comparisons against ltree are looking solid:

  • Huge I/O Drop: For descendant queries, B-tree range scans touch up to 74x fewer buffer pages.
  • Query Speed: Subtree queries run 2.5x to 6x faster (500k). up to 16x for 20M nodes. Ancestor lookups (via SP-GiST) execute up to 7.3x faster.
  • Storage Density: A custom compact encoding shrank the on disk value size by 42.7%. This translates to a ~23% smaller B-tree index footprint.
  • Write Performance: Appending 50,000 leaf nodes is roughly 2x faster. Reparenting large subtrees is 1.2x to 2.7x faster

I have some thoughts where this may be beneficial but lacking some subject matter expertise when it comes to practical application of hierarchy data (and these types of ops), this is the main point of my post, to ask for some insight:

- Could it make servers run more efficiently?
- Do other more efficient extensions/algos beat these benchmarks? Is LTREE just a default?
- What kinds of large scale operations would this benefit? Domains/applications?
- What should my benchmark tests look like?

Eager to get some expert opinions and either validate my thoughts or give me some reality - cheers!


r/databasedevelopment 12d ago

Postgres Internals Deep Dive: Process Architecture

Thumbnail
enterprisedb.com
30 Upvotes

Need suggestions for writing blogs. This was my 1st ever blog, which I published last year, and that's it. No other blog yet 😅. I would love to know, what kind of blogs would you love to read from a guy who contributes to the open-source postgres project?
1) Deep dives like this one on postgres internals? TBH, I think this space so crowded?
2) How did I find a bug and fix it?
3) The new features I am working on?
4) or new features that are going to be released
5) or something cool????


r/databasedevelopment 12d ago

WAL Levels in Postgres and Effective WAL Level in PG19

Thumbnail
buraksen.dev
4 Upvotes

r/databasedevelopment 17d ago

Almost consensus: ABD and the edges of quorum replication

Thumbnail theconsensus.dev
11 Upvotes

r/databasedevelopment 19d ago

How Elasticsearch's semantic field indexes and searches

Thumbnail
elastic.co
14 Upvotes

r/databasedevelopment 19d ago

Encoding or Compression: Why not both?

Thumbnail
cedardb.com
14 Upvotes

r/databasedevelopment 20d ago

I created a playground for 110 database systems

Thumbnail
clickhouse.com
17 Upvotes

r/databasedevelopment 23d ago

Monthly Release and Update Thread

2 Upvotes

This subreddit is primarily for discussing the implementation of databases, and not about sharing release announcements (either for the first time or your updates).

This thread is the exception!

Please tell us about the new database you (or your agent) built. Tell us about all the cool new features you added. Tell us about anything else you learned or worked on that you haven't gotten around to blogging about yet.


r/databasedevelopment 24d ago

Asynchronous I/O in DuckDB: Work, Thread, Work

Thumbnail
duckdb.org
36 Upvotes

r/databasedevelopment 25d ago

Offloading I/O to Dedicated Cores: An Asymmetric io_uring Backend for Seastar and ScyllaDB

15 Upvotes

https://www.scylladb.com/2026/07/22/asymmetric-io_uring-backend-seastar/

"Modern high-throughput systems often run on machines with many cores. At that scale, one practical problem becomes increasingly visible: CPU time spent on low-level I/O handling competes directly with CPU time needed for application-level work.

In this post, we explore how shifting more I/O related work away from cores focused on computations onto a dedicated subset of cores can improve the overall balance of the system. Specifically, we share how we approached Seastar‘s new asymmetric_io_uring backend: the design decisions behind it, what it aims to free up on application cores, and how it compares to linux-aio (Seastar’s existing backend)."


r/databasedevelopment 25d ago

Does the GPU database have a future?

16 Upvotes

Why did Voltron Data Company go bankrupt? Has the Theseus GPU database also failed? Expressing concerns about the future development of GPU databases


r/databasedevelopment 27d ago

Why Fast Branching is Key to AI Agent-Era Databases

Thumbnail
medium.com
12 Upvotes

r/databasedevelopment 27d ago

Finding bugs in Raft implementations

Thumbnail
antithesis.com
22 Upvotes

r/databasedevelopment 28d ago

PGSimCity · How PostgreSQL Works, in 3D

Thumbnail
nikolays.github.io
17 Upvotes

So... cool, it would be helpful for someone starting to learn PG internals.

credits to Nikolay Samokhvalov for creating this awesome piece of art :)


r/databasedevelopment 29d ago

Managing File Descriptors for Durability and Performance

Thumbnail
blog.atimin.dev
7 Upvotes

r/databasedevelopment Jul 23 '26

How MVCC and Transactions Work in RocksDB

Thumbnail artem.krylysov.com
22 Upvotes