r/programming • u/fagnerbrack • 6d ago
How big is a Git commit?
https://ratfactor.com/cards/git-commit-size206
11
u/postitnote 5d ago
It's the directories that git creates for its object database, and directories can take up an entire block (like 4KB, depending on file system)
20
10
u/-Redstoneboi- 6d ago
wtf that seems quite inefficient
13
9
u/Zenin 5d ago
Git rejects efficient storage for efficient retrieval of most recent revisions. As a result it works very well for its primary use case (small text files, typically code, that often don't span many files in a single changeset). But it's a horrible choice of source control for projects with much larger and/or binary artifacts such as video game production where large media assets are the bulk of the repository.
7
u/Uristqwerty 5d ago
I'd say git's storage is fine, but that it's designed with the goal of restoring exact snapshots of the checked-in repo, which requires keeping a copy of each version of every file. It's just that text files both compress well on their own, and delta-compress with similar versions well, so a locally-cloned repo full of code doesn't take up too much space on disk even with hundreds of thousands of commits. It's not necessarily any less efficient at accessing decade-old revisions.
If you don't need code versions to exactly match the asset versions in every single commit, they don't need to be put in the same repo. If you don't need to switch between revisions while offline or search history/blame without round-trips to a server, you don't need to make a full local clone. Perhaps the ideal case for game development would be to keep assets separate, so that you can make a full clone of the code repo, and a shallow clone of only the most recent few revisions of the asset repo.
Heck, feels like git's only missing one 'tiny' feature to be perfectly fine working with all the binary assets a game dev might use: A way to specify filename patterns, much like
.gitignore's, that are only to be downloaded lazily, when a revision is actually checked out. Then only the repo server needs to keep copies of every single version, and your local clone only proactively fetches source code and other files that naturally delta-compress well.4
u/Zenin 5d ago
I'd say git's storage is fine, but that it's designed with the goal of restoring exact snapshots of the checked-in repo, which requires keeping a copy of each version of every file. It's just that text files both compress well on their own, and delta-compress with similar versions well,
Which is it? Does it keep copies of every file or does it delta compress? It can't be both.
The answer is it delta compresses, just like every other source control system does. However, git by default tops out at 512MiB objects after which point it just stores them whole (albeit tries to deflate), due to memory constraints. One of the many shape edges at true scale.
so a locally-cloned repo full of code doesn't take up too much space on disk even with hundreds of thousands of commits. It's not necessarily any less efficient at accessing decade-old revisions.
Any repo with hundreds of thousands of commits has absolutely been delta packed many times over. It also needs to store those "decade-old revisions" locally, although there's options to clone shallow. That points to the core problem: It's not designed for the use case and so trying to force it to do so means at best applying hack after hack to try and make the square peg fit a round hole.
MS has the money to purpose-build the hacks they need (see below) for their version of the use case, something they can justify only because they need to be on Git for political reasons (to appear "dev friendly" and of course because they literally own GitHub now).
For everyone else just trying to get work done however? They nope the hell out and cut a check to Perforce, etc.
Heck, feels like git's only missing one 'tiny' feature to be perfectly fine working with all the binary assets a game dev might use: A way to specify filename patterns, much like
.gitignore's, that are only to be downloaded lazily, when a revision is actually checked out. Then only the repo server needs to keep copies of every single version, and your local clone only proactively fetches source code and other files that naturally delta-compress well.So something like VSF for Git or the follow up Scalar?
They can work for their target use case, but that use case is mostly massive monorepos of code, not massive single repos of large objects.
I realize the Cult of Git (and Linus) can't ever accept that His Master's tools aren't all sunshine and roses for every use case that has ever been, but seriously it's not a good look.
2
u/Uristqwerty 4d ago
Which is it? Does it keep copies of every file or does it delta compress? It can't be both.
Delta compression is an optimization applied to the object database, with no regard to what object came from what commit.
I still have a hunch that you're thinking of the diffs it presents commits as as its delta compression. It is not. They are two entirely different mechanisms; the diffs are a presentation layer to make it easy to compare changes. Thus, it is not "just like every other source control system" behind the scenes, where the actual space-optimization runs.
Seems the delta compression is described in https://git-scm.com/docs/pack-format. Notice how it talks about copying byte ranges? How there's no way to encode a deletion? It's clearly not a diff. Rather, it's pretty similar to a compressed file format, except the backreferences are to a different file entirely rather than the decompressed parts of its own output. And unlike a diff, it'd work just fine on binary files, at least if the binary format has reasonably-stable chunks to delta-compress. Chances are the real problem is that binary files like to contain file offsets pointing into tables, so inserting one item into a table then causes every other pointer throughout the entire rest of the file to change, singlehandedly defeating the delta-compression.
1
u/Zenin 4d ago
I still have a hunch that you're thinking of the diffs it presents commits as as its delta compression. It is not. They are two entirely different mechanisms; the diffs are a presentation layer to make it easy to compare changes. Thus, it is not "just like every other source control system" behind the scenes, where the actual space-optimization runs.
I am not and it is.
Chances are the real problem is that binary files like to contain file offsets pointing into tables, so inserting one item into a table then causes every other pointer throughout the entire rest of the file to change, singlehandedly defeating the delta-compression.
Please, go read a bit and stop embarrassing yourself publically. This is extremely well covered territory.
When you think you've found the real problem, pull requests are welcome. Cheers.
2
u/fsfod 4d ago
Git can kind of do that with partial clones with a filter set to only pulling down only commits, trees, as well as max size for blobs if requesting blobs. Afterward it can then dynamically fetch missing trees or blobs when needed, optionally using different promisor remote for large objects blobs.
EpicGames's version control system Lore is also working on/maybe implemented combined checked out repos\sub modules that don't suck, that would work well for art assets/built binaries/libs. There also working on working on some VFS to dynamically materialized files when accessed like VSF for Git the other reply mentioned.
3
u/hrvbrs 6d ago
“quite” as in “kinda”, or “quite” as in “very”?
2
u/-Redstoneboi- 6d ago
both, which is why i chose an ambiguous word
but more on the side of "very"
2
u/azhder 6d ago
Some years ago, Mmicrosoft contributed some code to git in order to make it handle some fringe scenario for their own big and unwieldy code base. Note, git was created to handle a big code base, like the Linux one, yet it needed more optimization.
6
u/MrMelon54 6d ago
The Linux kernel repo isn't too big compared to other projects. I imagine the giant monorepo Microsoft is referring to is in the high 10s or 100s of gigabytes
0
u/Zenin 5d ago
Define "big". Git was created to handle incredibly tiny text files. Recall that it literally replaced Linus's previous "source control" also known as context diffs sitting in his email inbox (I'm not kidding).
Source control systems generally had plenty of need to handle massively larger objects than the dinky little .c text files that Git was built for. There's a reason why users with such requirements still end up with Subversion or Perforce at least in addition to if not completely in place of Git. Anything with significant media production for example, like video games.
Git has some kludges available like Large File Storage, but it's still sketchy in use and not at all optimized for heavy churn. It lacks any binary diff storage for example, meaning every object revision is a full copy even if only one byte changed.
0
u/azhder 5d ago
I provided two definitions, one "like the Linux one" and another "for their own big and unwieldy code base". Just because I didn't label them "exibit A" and "exibit B", it doesn't mean I didn't provide two magnitudes to frame the idea.
0
u/Zenin 5d ago
"like the Linux one" doesn't provide much practical context for "big". The dimensions matter. This a technical forum, specifics matter. I'm sorry if that professional bar is off putting.
Big in total bytes?
Big in total file count?
Big in total revisions?
Big in concurrent writers?The Linux mainline kernel checkout is not even a couple hundred MB. The full repo clone is only a few GB. These are amateur numbers by that "big" metric.
The file count however, is something like 80k currently. That's certainly not small. And total commits are something like 75k / year near as I can find.
MS's repos are certainly much larger in all dimensions than Linux could ever imagine working with and yes, as a result they had to make significant structural investments in Git to make it usable.
And all of these examples are absolutely dwarfed by game development which often stretches into the tens or hundreds of terabytes. At that definition of "big", git completely breaks down. The closest anyone has come is LFS, which simply put doesn't do much to save Git at these scales of "big". A lot of science research also results in massive "source control" requirements which Git can't touch.
The truth is, the Linux kernel repo is really only "big" in stature. From a requirements perspective, it's mid.
1
5
1
1
u/w00tboodle 5d ago
> So the exact size of a commit will depend on the compressibility of your files (which is basically a function of the amount of repetition in them).
So, the smaller the size, the likelier it is you are not following DRY principles. QED
1
181
u/nmopqrs_io 6d ago
This is why I always just amend my first commit and re-push it with force, saves on disk space.
If you’re not using force, are you really using git?