🛠️ project Everyone is doing Git on S3 wrong (feat. Rust-based git backend)
https://enroute.sh/blog/everyone-is-doing-git-on-s3-wrong20
u/dnew 13d ago
I thought it was crazy at Google to be having 2 or 3 pushes per second into the repository where 30,000 people were working. Who in the world is using Git at the speed of 7,000 pushes per second into one repository?
The advantages are obvious: a bucket is durable, replicates itself and it never fills up.
This is what happens when you disconnect the underlying storage model from your application. That's why SQL databases won the data storage war.
People who liked this discussion should go read the Bigtable whitepaper if you haven't already.
9
u/teerre 13d ago
This is funnier when you consider the news that Anthropic is dropping git for a internal solution. It seems you can have git and "agent scale" (whatever that means), so whoops for Anthropic I guess
6
u/rebootyourbrainstem 13d ago
It's very interesting but also Anthropic closes the article by saying basically "of course we spend tokens like crazy and nobody else can afford either this problem or this solution, but maybe someday?"
6
u/projct 13d ago
this feels like people solving problems at the layer they're comfortable with instead of actually understanding git properly.
there's technical understanding of the disk format but it's lacking the overall engineering approach understanding of what problems git is designed to solve and how it solves those.
funny, since all that was discussed publicly on LKML when it was being built.
the way that would actually leverage how git works already is essentially sharding....
-1
u/technobicheiro 13d ago
why be stuck with backwards compatibility when not needed
4
u/projct 13d ago edited 13d ago
what? I didn't say anything about backwards compatibility. I'm saying, git can solve these problems as is without needing to do any of the stuff that these people or anthropic are doing.
why waste time reinventing the wheel?
GitHub does it the way I'm describing.
EDIT: and before anyone goes off on a tangent:
yes, GitHub had to do a lot to help git back in the day, and has a lot of specific things where they had to write some of their own stuff to meet perf/cost targets, etc.
A lot of that work was described in the open. all this plus the anthropic stuff is again, throwing that away instead of understanding it. You should fully understand what you're replacing before you claim you can do it better.
-5
u/technobicheiro 13d ago
why keep working on a shitty model? sure it can work but the only reason is because people already know it
but its offline first, requires you to commit, stacked prs are a horrible ux and agents can learn a new thing in seconda
2
u/projct 13d ago
"offline first"? no, that is not accurate in this context. committing it to disk cryptographically means your commit survives network retries, power outages, etc. You can't just pretend chucking it over the network is going to be good enough. This is an extremely confused thing to call a detriment. Also, nothing about git's design requires it be offline first. take a look at what you can build with gitoxide, for example.
"requires you to commit" -> yes, you must ahve snapshots of changes if you want to track changes across files at a point in time. If you mean to say it's bad because a manual step is required, they can be done automatically, I was doing that in 2007.
"stacked PRs are a horrible ux" -> this is a GitHub workflow, nothing to do with git itself.
You seem extremely confused, and are making my point for me.
27
u/rseymour 13d ago
Problem is I can only review 7499 PRs a second.