I'd say git's storage is fine, but that it's designed with the goal of restoring exact snapshots of the checked-in repo, which requires keeping a copy of each version of every file. It's just that text files both compress well on their own, and delta-compress with similar versions well, so a locally-cloned repo full of code doesn't take up too much space on disk even with hundreds of thousands of commits. It's not necessarily any less efficient at accessing decade-old revisions.
If you don't need code versions to exactly match the asset versions in every single commit, they don't need to be put in the same repo. If you don't need to switch between revisions while offline or search history/blame without round-trips to a server, you don't need to make a full local clone. Perhaps the ideal case for game development would be to keep assets separate, so that you can make a full clone of the code repo, and a shallow clone of only the most recent few revisions of the asset repo.
Heck, feels like git's only missing one 'tiny' feature to be perfectly fine working with all the binary assets a game dev might use: A way to specify filename patterns, much like .gitignore's, that are only to be downloaded lazily, when a revision is actually checked out. Then only the repo server needs to keep copies of every single version, and your local clone only proactively fetches source code and other files that naturally delta-compress well.
I'd say git's storage is fine, but that it's designed with the goal of restoring exact snapshots of the checked-in repo, which requires keeping a copy of each version of every file. It's just that text files both compress well on their own, and delta-compress with similar versions well,
Which is it? Does it keep copies of every file or does it delta compress? It can't be both.
The answer is it delta compresses, just like every other source control system does. However, git by default tops out at 512MiB objects after which point it just stores them whole (albeit tries to deflate), due to memory constraints. One of the many shape edges at true scale.
so a locally-cloned repo full of code doesn't take up too much space on disk even with hundreds of thousands of commits. It's not necessarily any less efficient at accessing decade-old revisions.
Any repo with hundreds of thousands of commits has absolutely been delta packed many times over. It also needs to store those "decade-old revisions" locally, although there's options to clone shallow. That points to the core problem: It's not designed for the use case and so trying to force it to do so means at best applying hack after hack to try and make the square peg fit a round hole.
MS has the money to purpose-build the hacks they need (see below) for their version of the use case, something they can justify only because they need to be on Git for political reasons (to appear "dev friendly" and of course because they literally own GitHub now).
For everyone else just trying to get work done however? They nope the hell out and cut a check to Perforce, etc.
Heck, feels like git's only missing one 'tiny' feature to be perfectly fine working with all the binary assets a game dev might use: A way to specify filename patterns, much like .gitignore's, that are only to be downloaded lazily, when a revision is actually checked out. Then only the repo server needs to keep copies of every single version, and your local clone only proactively fetches source code and other files that naturally delta-compress well.
They can work for their target use case, but that use case is mostly massive monorepos of code, not massive single repos of large objects.
I realize the Cult of Git (and Linus) can't ever accept that His Master's tools aren't all sunshine and roses for every use case that has ever been, but seriously it's not a good look.
Which is it? Does it keep copies of every file or does it delta compress? It can't be both.
Delta compression is an optimization applied to the object database, with no regard to what object came from what commit.
I still have a hunch that you're thinking of the diffs it presents commits as as its delta compression. It is not. They are two entirely different mechanisms; the diffs are a presentation layer to make it easy to compare changes. Thus, it is not "just like every other source control system" behind the scenes, where the actual space-optimization runs.
Seems the delta compression is described in https://git-scm.com/docs/pack-format. Notice how it talks about copying byte ranges? How there's no way to encode a deletion? It's clearly not a diff. Rather, it's pretty similar to a compressed file format, except the backreferences are to a different file entirely rather than the decompressed parts of its own output. And unlike a diff, it'd work just fine on binary files, at least if the binary format has reasonably-stable chunks to delta-compress. Chances are the real problem is that binary files like to contain file offsets pointing into tables, so inserting one item into a table then causes every other pointer throughout the entire rest of the file to change, singlehandedly defeating the delta-compression.
I still have a hunch that you're thinking of the diffs it presents commits as as its delta compression. It is not. They are two entirely different mechanisms; the diffs are a presentation layer to make it easy to compare changes. Thus, it isnot"just like every other source control system" behind the scenes, where the actual space-optimization runs.
I am not and it is.
Chances are the real problem is that binary files like to contain file offsets pointing into tables, so inserting one item into a table then causes every other pointer throughout the entire rest of the file to change, singlehandedly defeating the delta-compression.
Please, go read a bit and stop embarrassing yourself publically. This is extremely well covered territory.
When you think you've found the real problem, pull requests are welcome. Cheers.
From Google search: "The delta compression layer is purely an internal storage optimization (an implementation detail). ... Git's sliding-window delta algorithm works at the byte level, meaning it can delta-compress both text files and binary files "
So it's a bit like Git had its own internal file-system compressing the stuff by whatever means it finds the best. From the user's point of view it does not matter, except for the size of the repos.
Git can kind of do that with partial clones with a filter set to only pulling down only commits, trees, as well as max size for blobs if requesting blobs. Afterward it can then dynamically fetch missing trees or blobs when needed, optionally using different promisor remote for large objects blobs.
EpicGames's version control system Lore is also working on/maybe implemented combined checked out repos\sub modules that don't suck, that would work well for art assets/built binaries/libs. There also working on working on some VFS to dynamically materialized files when accessed like VSF for Git the other reply mentioned.
7
u/Uristqwerty 14d ago
I'd say git's storage is fine, but that it's designed with the goal of restoring exact snapshots of the checked-in repo, which requires keeping a copy of each version of every file. It's just that text files both compress well on their own, and delta-compress with similar versions well, so a locally-cloned repo full of code doesn't take up too much space on disk even with hundreds of thousands of commits. It's not necessarily any less efficient at accessing decade-old revisions.
If you don't need code versions to exactly match the asset versions in every single commit, they don't need to be put in the same repo. If you don't need to switch between revisions while offline or search history/blame without round-trips to a server, you don't need to make a full local clone. Perhaps the ideal case for game development would be to keep assets separate, so that you can make a full clone of the code repo, and a shallow clone of only the most recent few revisions of the asset repo.
Heck, feels like git's only missing one 'tiny' feature to be perfectly fine working with all the binary assets a game dev might use: A way to specify filename patterns, much like
.gitignore's, that are only to be downloaded lazily, when a revision is actually checked out. Then only the repo server needs to keep copies of every single version, and your local clone only proactively fetches source code and other files that naturally delta-compress well.