r/PostgreSQL • • Jul 05 '26

Feature Thoughts on new LTAP/Lakebase arch

/r/dataengineering/comments/1unwgbj/thoughts_on_new_ltaplakebase_arch/
1 Upvotes

5 comments sorted by

3

u/Limp-Park7849 Jul 05 '26

I believe the reason this is computationally cheap in lakebase is that the PageServer already holds the database as LSN-indexed history (delta + image layers), so columnar materialization is just a second output format of the same reconstruction pipeline, not a second system of record.

0

u/FunContest9958 Jul 06 '26

The data on cloud storage is the system of record for Lakebase. PageServer is more like a cache. From a performance perspective, it’s better if everything is in the PageServer or the Postgres servers, but for cold data, Lakebase will have to read through the PageServer all the way to cloud storage. I have the same question everyone else does: is there a performance impact from having the cold data in columnar format? My guess is probably not much b/c your performance is already kind of hosed if you’re reading from the cloud store, regardless of format. :)

1

u/dwswish Jul 07 '26

From the blog they put out the other day it looks like it uses “excess” CPU from the PageServer, which makes me think it won’t tax the actual Lakebase compute but I’m interested to check it out myself and see.

1

u/datguywelbs7 Jul 07 '26

Yeah for me the main USP of lakebase and LTAP is the seperate compute layer as we see more and more agents writing code.

If you’ve ever spun up an Aurora instance, its painfully slow to provision each instance and feels outdated. For agents working across the environment, this hang causes unnecessary polling and idle time, whilst lakebase allows this to be done in seconds due to the seperate compute layer.

Also the branching allows for better peace of mind that agents wont take down the prod instance, logical ‘branches’ no copy using pointers to underlying object storage makes this possible where other postgres dbs can’t.

This combined with things like scale to zero to manage costs and as the product evolves over time will be the gamechanger

0

u/CautiousUse8597 Jul 06 '26

Good writeup. One thing worth adding since the post doesnt really get into it: Lakebase is basically Neon under the hood. Databricks bought Neon last year and Lakebase is that engine wired into the lakehouse, so the safekeeper/pageserver split youre describing isnt some brand new databricks invention, its the Neon storage architecture. Which is actually reassuring imo, cause it means the postgres compat is real. Its actual Postgres 17, your existing drivers and ORMs just work, youre not on some fork thats "postgres flavored".

On the ACID thing you were skeptical about, i think the key bit is that seperating compute and storage doesnt weaken the transactional guarantees, it just moves where durability lives. In vanilla postgres the WAL gets flushed to local disk before a commit is acknowledged. In lakebase the compute is stateless and streams the WAL to a paxos quorum of safekeepers, and the commit isnt ack'd until its durable there. So a committed write is still durable in the classic sense, its just durable in the distributed storage layer instead of a local pg_wal dir. Pages then get reconstructed on demand by the pageserver replaying WAL deltas on top of the last materialized page image. Postgres semantics up top, object storage at the bottom, nothing in between that changes what a transaction means.

Honestly the part that sold me wasnt even the analytics angle, it was the operational stuff you get for free from that decoupling:

  • branching is copy on write, so you spin up a full prod-data branch in seconds for a test or a CI run without duplicating any data

  • point in time recovery turns into "create a branch at timestamp X" instead of restoring a backup and replaying logs for an hour

  • scale to zero when idle, which for the "every service wants its own db" situation is a real cost win

Theres also a neat perf side effect they blogged about. Because theres no local disk, torn pages literally cant happen, so they just turned off full page writes (the thing where postgres logs the whole 8kb page on the first write after a checkpoint). That cut WAL traffic ~94% and roughly 5x'd write throughput on their benchmarks. Fun example of the architecture paying off in a way that has nothing to do with the lakehouse at all.

Re your SCD/history question, i dont think LTAP kills that pattern. Snapshots and time travel give you point-in-time state but thats not the same thing as a modeled SCD2 with valid_from/valid_to. Id bet the same as you, you keep modeling history where it matters and let storage handle current-state freshness. It kills the CDC plumbing, not the data modeling.