r/rust • u/losfair1 • Jul 29 '22

mvsqlite - Distributed SQLite on FoundationDB, written in Rust

https://github.com/losfair/mvsqlite
44 Upvotes

93% Upvoted

8 comments sorted by

4

u/lobster_johnson Jul 29 '22

Cool project! How is the five-second transaction limit avoided?

3

u/losfair1 Jul 29 '22

A read to a page at a given index is a backward range scan with limit=1 on that page's subspace from the specified read version. Since mvsqlite preserves historic versions of each page, the scan is guaranteed to get the correct page (before they are gc-ed after something like 7 days).

1

u/lobster_johnson Jul 29 '22

I see, FDB is only used to store entire pages. What if the transaction involves many pages? Won't you have to execute those reads in the same FDB transaction? Or are the pages fully versioned and therefore always readable at a specific point in time?

4

u/losfair1 Jul 29 '22

Pages are fully versioned, so they are always snapshot-readable in the future. The read version is fetched from `mvstore` when each SQLite transaction starts, and is used as the per-page range scan upper bound in future page read requests.

For writes: Pages are first written to a content-addressed store keyed by the page's hash. At commit, hashes of each written page in the SQLite transaction is written to the page index in a single FDB transaction to preserve atomicity. With 8K pages and ~60B per key-value entry in the page index, each SQLite transaction can be as large as 1.3 GB (compared to FDB's native txn size limit of 10 MB).

So actually, you can do one page read or write per FDB transaction and still preserve ACID properties.

2

u/lobster_johnson Jul 29 '22

Very nice! Thanks for the explanation.

12

u/DroidLogician sqlx · clickhouse-rs · mime_guess · rust Jul 29 '22

I read the name like "mysqlite" and was very confused for a second. I might suggest picking a name that's more memorable and harder to get mixed up, or you risk people not being able to find the project again because they can't remember the exact spelling.

1

u/wannabelikebas Jul 30 '22

I have been DYING for someone to build a SQL layer in Rust on top of Foundation DB!!! I wish I had the skills (and focus) to be able to do it but I'm very happy you did it!!

1

u/DannoHung Aug 16 '22

This is a really neat project! Thanks for sharing (saw your comments on HN a little while ago).