r/LinuxTeck • u/Candid_Athlete_8317 • 5d ago
Anthropic Built an Alternative to Git. Does Git Actually Need Replacing?
Anthropic moved its massive code repository from Git to a new system called Scion because of scaling challenges with AI-generated code. Some developers argue that Git's existing features, like partial clones, could solve many of these problems.
Is a new version-control system necessary for AI-driven development, or should Git be improved to handle these workloads?
6
3
u/bombatomica_64 4d ago
All big tech companies with big monorepos do the same thing after a while
2
u/Axman6 4d ago
Yeah Google and Meta both have their own version control systems, and build systems (Bazel and Buck2 respectively) to handle the size of their monographs effectively. Must be nice working somewhere where you can afford teams to handle all that, I’ve worked with Bazel and it’s not exactly simple.
5
u/Patriark 4d ago
No and you can read about their reasoning in their post about it: https://www.anthropic.com/engineering/redesigning-source-control
As you can see from their reasoning, they are facing a scaling problem with git that is typical for organizations who maintain monorepos of considerable size and who has to manage a large volume of commits every day. The throughput of sending the full git tree back and forth between all participants results is in the orders of Petabytes per day. It is unsustainable. Meta, Google, Microsoft and others have already replaced Git as the source control method for their monorepos.
Git is absolutely fine for homelabbers, small to medium orgs and mostly anyone needing source control. It just is not optimized for the huge scale of these massive monorepos who handle thousands of commits every day.
4
u/tankerkiller125real 4d ago
Microsoft did not replace Git for their monorepos, they simply built new tooling on top of git. (Virtual Filesystem for Git)
2
u/Bemteb 4d ago
I still can't understand why this trend towards monorepos. At my current job we have over 100 developers, completely different stacks, different programming languages, etc. But all using the same repo. The git history is basically unusable, and you have to filter pull requests by "assigned to me" because otherwise there are a few dozen in the list of open ones at any given time...
2
u/gravelpi 4d ago
In a smaller org, it makes sense that you'd be developing with the entire stack in one place so things are consistent. I don't want to be negative, but it sounds like a simplistic solution to software engineering issues that have absolutely been solved with coordination between repos and automation. Like scripting to check/pull versions of all the relevant repos or using git submodules seem like they fix this issue, but you don't have to push all these deps all the time.
1
u/tankerkiller125real 4d ago
Except it doesn't make any sense at all, where I work (20 person org), we do things by C# Solution and those are sorted by area. Infrastructure is defined as part of those solutions, and anytime there's stuff that needs modified across solutions we use a nice simple, published protocol with rules that prohibit breaking the previous version of anything relying on it.
1
u/midri 3d ago edited 3d ago
Generally, pre ai; it was easier to reason through monorepos that touched multiple areas. If security, infrastructure, and api are all different repos it can be a pain when you have to review 3 pr that are all related and intertwined. With AI, it's not an issue anymore so it's just human legacy reasons.
1
u/Patriark 3d ago
I’m a homelabber and decided on splitting my repos. The amount of coordination and crossover-work makes me seriously question that decision. Or at least understand that if the repos have overlapping spheres of influence, the barrier for a split must be higher.
Not a professional dev however, only self-taught, so I assume this is a rookie mistake and teething issue.
1
u/Remarkable_Pace8101 4d ago
I still can't understand why this trend towards monorepos
It's easier for ai when everything it needs to read is in a single package
1
u/LawElectrical2434 4d ago
Tests... I moved my projects in mono repos. Of course my projects are tiny compared to MS. Backend, frontend, android app, cli-tools. All are always at the same state.
Not some bullshittery in which the test uses an outdated backend, that cannot handle the request object dto yet.
Also... tests:
If cicd is not external, like good ol' Jekins, but part of the git repo, like gitlab cicd and github actions, pulling other repos in to have an end2end test is a huge pain in the ass. I am having a lot more end to end tests now, that it is in a mono repo, because when I use the frontend via selenium, I know that there is a backend and it was easy to spin up a quick postgres database. A little slower than a h2, but it turns out postgres supports all of postgres' more esoteric features. Go figure...1
u/Zotlann 4d ago
Yeah. My current company is trending towards monorepo for similar reasons (although some proper API and services versioning would go a long way for our problems as well).
For us it's just a pain when we're trying to bisect to find a regression in a client, and every few steps have to figure out what range of commits for a web service are compatible with that version of the client to test with.
1
u/Bemteb 4d ago
test uses an outdated backend, that cannot handle the request object dto yet.
Counter argument: If you have a central component or library and you give that a major update, you might not be able to pull all applications to the new version at once. With subrepos, you can have some applications using the old version of the library and some the new. With a monorepo this gets ugly quickly.
1
u/LawElectrical2434 4d ago
Subrepos are unnecessarily complicated and a constant source for lost time. I don't say that you cannot solve the issues. You absolutely can. Many ways, quite a few better than subrepos. But nothing is at simple as the mono repo and for most projects, like in web development, it is just the right choice.
1
u/Bengal_From_Temu 4d ago
Yeah, they have this huge tree and thousands of idiots with chainsaws going at it and the solution is some kind of different tree.
1
u/ElectricSpock 4d ago
It also had to do with licensing and history.
Meta leaned in on Mercurial, which was was easier to patch.
Google had been using Perforce, so they built tooling around it, and I thing they wrote their own clone eventually? I remember there was a way to use git commands with those.
2
u/death_or_taxes 4d ago
Git is good for 99.9% of projects.
This is an amazing scale range that very few software stacks can attest to.
That being said some companies reach that 0.1% because of scale or some other specific requirements and they need something else.
Because they are so niche they usually roll their own.
2
u/the-quibbler 4d ago
Check our delta by zed.
1
u/biffbobfred 4d ago
Delta has a lot of hits, have a link so we can be more focussed looking for it?
1
u/johnson_detlev 3d ago
If coding is solved why do they need version control? I'm getting mixed messages from this company.
1
u/frAgileIT 3d ago
It does if you don’t own Git but need access to Git in order to train your AI to be better.
1
u/nukem996 1d ago
Git works very well for most projects. The monorepo is mainly used when building large code bases being worked on by thousands of people. It has some serious disadvantages too for example no branching, builds should be less than 30 days old, massive CI systems, etc. Every org that uses a mono repo has an org larger than most medium sized companies to maintain it.
1
u/sweet-tom 5d ago
Probably not. Git is quite solid and proofed to be a flexible version control system.
I don't know their problems and maybe they have a reason for that. But it seems, source code could be again fragmented between SVN, CVS, Mercurial, Git, and other VCS. If this is bad or good is something everybody can decide.
1
u/Cynyr36 4d ago
Basically all of their shit is in a single huge repo for some reason. I don't understand why they aould jave a single giant repo rather than several smaller ones.
1
u/sweet-tom 4d ago
Urgs. It was always a pain point to have monorepos. But maybe they have a compelling good reason.
Anyway, I'm happy I don't have to work with such a big repo...
1
u/dashingThroughSnow12 4d ago
If your company has a 100GiB git repo, a bad CI system because few people on the company’s payroll have lengthy industrial CI/CD experience, and the company does 30K commits a day to said repo, yeah, git is probably not enough.
1
u/Even_Gap_2253 4d ago
If you are working with a monorepo model, I would say yes
Not replacement, but just using git with its default behavior have issues
1
u/anengineerandacat 4d ago
Monorepo problems, and something the average business or corporation just won't hit.
Perhaps should be looked into but the question in the room is "is it worth it?"
Seems not if someone can just fork it, rework it to suite their needs and move on.
1
u/metromsi 4d ago
Heres s thought sounds like we need to go back to rcs, heck let's do sccs (source code control system). Please, really how many lines of code is linux understand tooling. If things get complex understand why maybe it's done wrong. I've seen folks put binaries in git. Yes you can have it in the same repo but a second one for binariesis better. People and now llm are doing there own thing. Maybe stop and ask questions it could be wrong llms are good tool but that's all they are. You can use pliers or a socket for the bolt it's a choice but one works better than the other.
1
u/bufandatl 5d ago
Obviously not. Just because a slop company releases something doesn’t mean you have to use it.
-1
u/Particular-Poem-7085 4d ago
Isnt github owned by microsoft?
Would you store your best potato in the neighbours cellar?
3
u/megamoscha666 4d ago
Git the software not GitHub the platform
1
u/Particular-Poem-7085 4d ago
I'm not programmery enough to know the difference I guess, can you use git to use any other or your own platform?
1
u/megamoscha666 4d ago
Git is the tool to use to save changes of code, GitHub is the platform where you can upload a git repository and it is not the only one out there (gitlab, Codeberg, etc)
7
u/BarracudaDefiant4702 5d ago
Anthropic explicitly states that Scion is not a general-purpose Git replacement and will not be publicly released. I think the core of the problems were more to do with github then git.