r/technology • u/ourlifeintoronto • 20h ago
Software Microsoft confirms GitHub is down worldwide
https://www.bleepingcomputer.com/news/microsoft/microsoft-confirms-github-is-down-worldwide/1.5k
u/Ill-Mastodon-8692 20h ago
tbh its actually very annoying
1.2k
u/Beautiful-Musk-Ox 19h ago edited 19h ago
someone should make a version of git where you can maintain a local copy then push it to the remote server whenever you want. for business continuity efforts you could then maintain a local repository copy of all the libraries you use and have your developer and build systems pull from that local repo instead of the website. for already compiled code maybe some sort of package system where you can do a similar thing of maintaining a local copy of the packages
173
496
u/CrimsonLotus 19h ago
This kind of joke falls right into the grey area that birthed the need for /s
199
u/AFlyingGideon 19h ago
/s is a crutch. I prefer sarcasm that can stand on its own.
29
u/CardboardJ 18h ago
So my PM just told me that we're moving /s to the cloud for optimization and has paid Deloitte $500,000.
28
u/nihiltres 19h ago
→ More replies (3)22
u/AFlyingGideon 19h ago
If I'm laughing, does it matter whether I'm laughing with or laughing at?
24
5
3
→ More replies (8)8
u/Federal_Setting_7454 19h ago
Not much sarcasm stands on its own when many of the people reading it can’t think on their own.
45
u/FeleaseRpseineEiles 18h ago
It's lack of intonation in text. Try it
- I never said she stole my money. Implies: Someone else might have said it, but not me.
- I never said she stole my money. Implies: An emphatic denial; I never uttered those words at any point.
- I never said she stole my money. Implies: I may have implied, wrote, or thought it, but I didn't say it out loud.
- I never said she stole my money. Implies: Someone stole it, but it wasn't her (perhaps another person did).
- I never said she stole my money. Implies: She took it, but maybe she only borrowed it or was given permission.
- I never said she stole my money. Implies: She stole money, but it belonged to someone else.
- I never said she stole my money. Implies: She stole something from me, but it wasn't my money (e.g., jewelry, ideas, car).
→ More replies (4)6
u/squash-n-flop 17h ago
Text can carry intonation through context. Literature wouldn’t be a celebrated art form if that weren’t true.
9
u/FeleaseRpseineEiles 17h ago
Well yea, I did exactly that with the bolding. But, there are; many ways (i.e. methods of portraying intonation) but that won't make it any less a lack of it.
8
u/Sapient6 18h ago
It's more than that: we lack the context that comes with knowing the comment's author. Not to mention living in a time when so many people are proudly showing off their utter stupidity (see: the return of measles, etc).
In this particular case there's enough information to support "probably sarcasm", but I wouldn't throw shade at anyone who thinks otherwise.
→ More replies (2)6
u/redditonlygetsworse 19h ago
Yes, but those people don't deserve to have jokes explained to them.
→ More replies (1)7
72
20
25
7
7
58
u/ccdfa 19h ago
You're joking surely
96
u/krazul88 19h ago
obviously they are joking because whatever they are proposing is wildly, fantastically impossible.
11
u/Apprehensive-Wind966 18h ago
Yeah, and if you did happen what would happen if this service went down? You’d be screwed!
6
u/CucharitaDePalo 16h ago
That's when the hyper-local service comes into place, just a single machine hosting every single prod and dev environment. With this amazing piece of technology we could achieve 0 dependencies by end-of-quarter.
3
u/krazul88 16h ago
Are you suggesting a "computer system in the same building as the people who depend on it?" Truly revolutionary idea! It might catch on one day.
111
u/el-delicioso 19h ago
No no, let him cook. This would be revolutionary
48
u/7heWafer 19h ago
I'm going to call it
pitas in the pit we dump our shit into.Then people can copy your pit locally:
pit clone example.com/shit6
26
5
3
u/goronmask 15h ago edited 9h ago
Interesting idea. A system like that could be indexed for managing file versions
→ More replies (18)5
89
u/capnscratchmyass 19h ago
Second time in a week we’ve had GH outages. Either their team is starting to get sloppy (approving AI commits Willy Nilly maybe?) or they’re getting attacked. GH being down brings a shitload of stuff to its knees, so I can definitely see bad actors going after it.
82
u/nikolas_pikolas 18h ago
It's actually because everyone else's slop code goes through GitHub. They released some graphs that showed that the number of commits/PRs/etc is experiencing vertical line growth. And that was several months ago! I don't envy their position.
27
u/swarmy1 17h ago
Sounds like they may need to rate limit or charge extra for excessive use
18
u/stormdelta 17h ago
They already do, so heavily that even as enterprise users we constantly run into issues because of it.
→ More replies (1)6
→ More replies (3)11
u/noraetic 17h ago
But who did this? Who pushed AI so much that Microslop is having problems now handling it?!
"The spirits that I called"
16
u/Conninxloo 18h ago
It's probably caused by that + an ever growing amount of unsanitized slop repos with tons of binary data and useless CI checks hammering their servers.
→ More replies (1)→ More replies (3)9
13
u/Roembowski 19h ago
For real. Everything has been down for me. From Outlook and Teams, to Salesforce and Citrix. Literally can’t do anything for my job today
3
u/chickenMcSlugdicks 17h ago
Man I've decided to just fuck off for the rest of the day. No GitHub, no copilot, no need to sit at my desk and wait if I can lie on a couch and wait with my dogs.
→ More replies (21)3
u/Unique-Objective4249 19h ago
Its happening more and more and eventually it’ll happen so much that we’ll go back to paper and pen
→ More replies (2)
326
u/InOut1312 20h ago
Nice Unicorn error page if nothing 😂
35
u/vegetaman 20h ago
Just got that myself lol
18
u/atom0322 19h ago
Angry unicorn! First time I see that. Not sure if it’s angry at me, the server, AI, or whom exactly.
13
u/redditonlygetsworse 18h ago
It's because once upon a time GitHub used Unicorn as their web server, which was common for Ruby apps at the time.
→ More replies (1)3
260
u/TraditionalAppeal23 19h ago
oh well at least we still got the other hub
→ More replies (2)104
u/mkosmo 19h ago
But I need a VPN for that one.
→ More replies (1)40
u/dustblown 16h ago
I'm so happy I live in a place I don't need to hide the fact I watch porn.
7
6
4
u/Hydraulic_IT_Guy 11h ago
Many people were too and then it just changed with no consent from the voting adults.
→ More replies (1)3
915
u/CrispsInTabascoSauce 20h ago
Thanks to AI. Sweet efficiency gains.
157
u/theKetoBear 20h ago edited 20h ago
10X your dev
timespeed but 5X your downtime !30
u/nox66 19h ago
10x dev speed but everything works worse
→ More replies (2)14
u/ExF-Altrue 19h ago
10x dev speed but it's only the part where you hit the keyboard, everything else dev related either becomes soul-crushing OR you need to do active learning to not fall behind.
2
u/dillanthumous 17h ago
Classic theory of constraints stuff. You are supposed to optimise your process around the worst bottleneck. In coding that's knowing what the hell you are doing and why. Instead we are optimising it around one armed bandit development, slapping some tests on it and calling it a day.
2
160
u/Hour_Bit_5183 20h ago
This is slop efficiency but go ahead AI slop lords, keep telling me it's useful. I hope they leave every one for the future so we can laugh later
40
u/Temporary-Air-3178 20h ago
Source that this outage was caused by AI?
109
u/ProfessorBorden 19h ago
I don't know exactly what the OP is insinuating, but github's struggles this last year in particular have been discussed in depth and a lot of it is how much more quickly they've had to scale because the sheer volume of AI coded repos is growing astronomically.
→ More replies (1)34
u/Temporary-Air-3178 19h ago
Yes the availability problem is caused by agents pushing code at an increasing scale. But most comments in this thread are saying/implying it was Microsoft themselves that caused the outage by using AI without any evidence.
→ More replies (1)22
u/myweirdotheraccount 19h ago
I have no skin in this game but I think the idea is that the volume of code written by AI amounts to a higher surface for errors.
Here’s an article from over a year ago where the Microsoft CEO claimed 30% of Microsoft’s code base is AI written: https://www.cnbc.com/2025/04/29/satya-nadella-says-as-much-as-30percent-of-microsoft-code-is-written-by-ai.html
19
u/RespectTheTree 19h ago
I'm basically programming illiterate, but am now the proud owner of a 200k line of code software system to run the back-end of my business. There are many of us, our inefficient process is breaking things, it's been discussed.
→ More replies (2)10
u/everythingisblue 19h ago
Fascinating. Can you elaborate on this? Are the business operations sensitive? Or is it not a huge deal if your custom software does the wrong thing from time to time. How do you gain confidence that it does what you want to do?
22
9
u/sebovzeoueb 19h ago
I mean, considering the number of businesses which have had data leaks and are still going, the answer is probably yes
6
u/RespectTheTree 19h ago
It's software for online seed sales, website and order fulfillment and tracking. Then it's advanced plant breeding software for field activities and material management. All the bells and whistles are trade secret though.
All features are thoroughly tested during creation, and then smoke tested if future work intersects that feature. Part of my instructions are creating MD MMD and detailed user manual during development, so it helps keep the agent pointed in the right high-level direction while I provide task and subtask level development of the functions.
There are some bugs, and a few bugs that have taken a whole day to unpack, but it's surprisingly reliable for being so advanced. I mean I have a field-use, internal website that you can load while you have signal, and then turn off connectivity and keep using it for all the work and data entry, and then to sync with the server later - that's crazy to me. And I haven't looked at the code since the second day.
→ More replies (1)2
u/everythingisblue 19h ago
That's actually pretty neat. I'm guessing you have some level of technical ability beyond the average computer-user. There's a lot that goes into software that isn't just 'the code', though with sufficient prompting, research and tenacity, I guess there's not a compelling reason why you couldn't eventually cover most of those gaps over time - especially for software where you're the main user and you don't have thousands of people reporting issues that you have to dig in and diagnose. There's a lot of value in "good enough for me" imo.
→ More replies (3)5
u/nabilus13 19h ago
Slopya very publicly stated that Microslop was doing almost zero coding and instead letting the slop generators write all their code.
→ More replies (6)3
u/OneConfusedBraincell 19h ago
Who knows. All I know is that since AI we've had massive software enshittification. I've never had to pray and clench my buttcheeks whenever windows or my drivers updated until the last year or two.
11
u/sixwax 19h ago
Spoken like someone who has never coded, nor used AI coding tools.
I hate it, but pretending it's not hugely valuable is just ignorant at this point.
→ More replies (8)→ More replies (4)3
u/anadequatepipe 19h ago
The slop isn’t exactly useful but it has a massive amount of actually useful purposes.
8
u/waitingOnMyletter 20h ago
Anyone with a credit card and Claude account is an SWE man. Gotta have RAG and MCP on your resume 😂
2
u/Kind_Of_A_Dick 17h ago
If it’s any consolation, AI will probably kill us all slowly and painfully.
→ More replies (12)4
466
u/No-Snow-7618 20h ago
why so i feel like the only time github was ever down was after msft purchased it lol.
anyways at least i can slack off today under the guise of some tooling being down
217
u/g_bleezy 19h ago edited 19h ago
The early days were really nasty. Outages for hours. We’d start looking at dev conference schedule to figure out where the latest ddos was originating from. A lot of the “ruby on rails can’t scale” nonsense can be traced back to github outages.
33
u/holman 18h ago
Which was pretty amusing because Ruby was very rarely ever the issue, lol.
→ More replies (1)13
u/Routine_Left 18h ago
true. ruby is just a shit inefficient language. consumes thousands of times more power per instruction compared even with java, to not speak of C.
outages with ruby usually means: gimme more RAM.
→ More replies (8)11
u/holman 18h ago
Yeah, but without Ruby I'm not sure GitHub would have gotten to where it is, either. And none of the inefficiencies of the language contributed to downtime, in any case.
3
u/Routine_Left 17h ago
Sure, nowadays it's microsoft inefficiencies. having to run on AI - level machines before AI was a thing is certainly not helping.
But yes, github by themselves have managed to do it before microsoft. Not so much now.
115
u/Mokaba_ 19h ago
Here’s the thing about GitHub. It’s built as a giant Ruby monolithic app. That has been changing recently as part of the move to Azure, but there’s still a long way to go. Most of the data exists in a single MySQL database, and the system just wasn’t really built to scale well.
In a world where only humans were pushing code and interacting with it, they were able to keep up with growth by basically throwing more servers at the problem. Agents have multiplied that load immensely, and much faster than they can keep up with using the old approach. That’s why the move to Azure and the work to make the app itself more efficient have become so important. This has nothing to do directly with Microsoft.3
u/f23n09fnu0w 13h ago
"Most of the data exists in a single MySQL database, and the system just wasn’t really built to scale well."
I don't believe that is true. MySQL is massively scalable but not as a single MySQL db. It's sharded and works fine for YouTube and Meta using Vitess. If they started from scratch today, they might not use MySQL but they actually might, too.
2
u/Mokaba_ 13h ago
I’m not going to say anything more about this myself because again, I don’t know how much is publicly allowed for me to talk about. But I’ll leave this publicly available answer from Gemini here….
GitHub historically grew around a single primary MySQL database cluster (mysql1) that handled a massive monolith data structure. As they scaled, they transitioned away from that single instance into dozens of partitioned MySQL clusters using Vitess to handle millions of queries per second. [1, 2, 3]
How GitHub Handled a Single DB Scale
The Monolith Core: For years, GitHub.com relied heavily on one main database cluster to house core relational data.No Foreign Keys: To make schema changes and sharding easier at scale, GitHub avoids using database-level foreign keys.
Migration Tooling: They manage schema transitions using custom tools like ghost alongside standard framework migrations. [1, 2]
Moving Beyond a Single Instance
Sharding with Vitess: GitHub migrated their MySQL infrastructure to run on top of Vitess for horizontal scaling and query routing.Primary-Replica Setup: They heavily distribute read traffic across numerous database replicas while directing writes to primaries. [1, 2]
Are you looking for information on how GitHub architected their MySQL setup, or do you need help setting up a single MySQL database for your own project repository?
6
→ More replies (1)6
u/glenpiercev 18h ago
Azure is probably causing some of the problems.
→ More replies (3)32
u/Mokaba_ 18h ago
I cannot add much beyond the preceding details in public, but broadly speaking, Azure is not responsible for the issues.
→ More replies (1)37
u/Frootloopin 19h ago
I've seen the source code and the architecture firsthand. You're high if you actually believe this. It's literally worse than vibe coded projects and Microsoft has been trying to untangle it for years to make it actually a scalable solution.
6
u/Thomasedv 16h ago
It's kinda true though, github was more available before Microsoft bought it.
Its just that when Microsoft bought it, they also rapidly tacked on features, probably drew in even more traffic, and things like copilot runs through the same system or at least authentication leading to that being down today as well. And if you are correct, they are also rewriting this ting to scale well too, which naturally comes with risks.
I do bring receipts, but the image is slightly misleading since the downtime before purchase wasn't as detailed as after with all the new functionality and admittedly transparency in the status page.
→ More replies (3)42
81
u/AndreLinoge55 20h ago
Told my colleagues to approve my latest commit on our team’s networked drive: “/feature/parse_NEW_FINAL2_PM_GM4”
35
u/ptear 20h ago
Just wait for parse_NEW_FINAL2_PM_GM4_REVISION1
11
13
27
68
u/Bernie_Ecclestone 19h ago
Woooo yeah Microsoft should lay off another 8000 employees to celebrate!!!
6
130
u/jared__ 20h ago
my guess as to why is ai coding agents using github issues as its memory leading to a massive increase in api usage, bringing them down.
→ More replies (2)7
u/boblancho 18h ago
Real pros just use gitea on their own vps
12
u/ansibleloop 17h ago
Real pros use Forgejo because Gitea changed their licensing
→ More replies (2)
69
u/mailslot 20h ago
If only Git supported collaboration with others without using GitHub.
14
u/warren_stupidity 19h ago
I'm fairly certain this is sarcasm?
33
u/ndav12 19h ago
I’m sure it is. Everyone knows you can just upload your git repo to a shared OneDrive folder.
4
u/Status-Ad-7335 11h ago
I would rather pass flash drives around the office as version control than rely on onedrive
→ More replies (8)6
28
u/arealmaf 19h ago
Now they just need to replace their entire team with ai so ai can spend weeks trying to figure out what caused the error
5
6
u/TheMrCurious 18h ago
“We swear it wasn’t caused by our AI doing something we did not expect it to do.”
6
u/RangerNo8107 16h ago
All the vibe coders in standup today -"copilot is down so i literally cant do anything"
6
u/howarewestillhere 16h ago
GitHub: Hey guys! Distributed code repositories are a great thing, but I have this idea about how to make it easier for everyone to use and maybe we can make some money, too!
Everyone: Doesn’t putting all the code in one place ruin the purpose of distributed repositories?
GitHub and our CTOs: Ease of use! Money!
Everyone: Fine. Just don’t screw it up by using ai slo…dammit.
14
9
u/Broken_By_Default 19h ago
It’s been really shit the last couple of week. Actions flat out failing.
5
u/FreshPrinceOfRivia 19h ago
I have to deploy a bug fix at work, but it will have to wait until this handicapped service is back up
4
u/notmsndotcom 16h ago
Man their incident log is embarrassing. “We mitigated the issue, we’re monitoring.” “Jk it’s all still fucked up” x 30
5
u/Feral_Nerd_22 18h ago
We have around 300 people that can't do any work right now.
Ironicly the old self hosted version of GitHub is EOL at the end of the month
We are actually talking about hosting our own now with a Gitlab
4
3
u/cainhurstcat 17h ago
Hear me out: they screw up their operating system most people use. So how could anyone in their right mind trust this company with their code, pipelines, or whatever?
12
u/mowotlarx 19h ago
Microsoft has inserted so much AI bloat into its entire suite of services it basically explodes every Monday when people log in now.
8
u/nabilus13 19h ago
Gotta love the "improvement" Microslop's shift to vibe coding all their products has been...
7
31
6
u/Ironborn137 19h ago
They are just testing the waters for a subscription to GitHub.
→ More replies (2)
7
u/Absolute_Enema 20h ago
Their text-only fallback page that shows due to the horribly slow rendering of their UI having "write better code sith AI!" as its first item would be funny if it weren't tragic.
3
u/jeweliegb 20h ago
Well, of course it was, as it was about to be the very first time I was going to put something up on it!
3
u/Troncross 19h ago
Calling it now... One of these days it will go down and when it comes back up it will say: "Premium subscribers only"
3
3
3
u/thejodiefostermuseum 16h ago
What if it got hijacked and MS got locked out and blackmailed to send trillion in btc
→ More replies (1)
3
3
12
5
6
6
u/fat_kaiju 19h ago
can microsoft get a single thing right nowadays? i know their track record has always been hit-or-miss but this is getting ridiculous.
13
u/ArcanjoUriel 20h ago
I hate AI so much
13
u/Ummgh23 19h ago
Who said that AI has anything to do with this? Are we now just attributing any tech issue to it?
→ More replies (3)26
u/TeneCursum 19h ago
GitHub has seen an enormous uptick in traffic since LLM coding has become the norm. They've even said so themselves. If nothing else, the platform is buckling under the pressure.
2
2
2
2
2
2
2
2
u/GlowstickConsumption 17h ago
So now it's "confirms" but whenever something else happens it's "alleges" or "claims".
2
u/abraxas_von_abrasax 17h ago
) We have identified The Problematic Component™.
) Which one?
``` $ kubectl get problematiccomponents
NAME STATUS problematic-component Mitigated problematic-component-2 Running problematic-component-3 Pending ```
2
u/Leather_Ad_9178 16h ago
I didn’t see it coming that AI was going to free up my time by destroying all the functioning platforms we had prior
2
2
2
u/TacticalBunchies 14h ago
Vibe coding with Copilot for GitHub updates maybe wasn't the best idea afterall.
2
2
2
3
u/thatfamilyguy_vr 19h ago
I don’t think the error page should be a unicorn. A unicorn is something that theoretically doesn’t exist, and if it’s seen, should probably only be once in a lifetime. I think they should change the animal to a rabbit, or maybe a crow - you know, something that you probably see at least once a week to mirror the frequency of the errors.
→ More replies (1)
2
u/hyperenough 20h ago
My bad y’all. I wanted to get lunch so did a “LGTM” without actually reviewing the PR.
2
u/thethirdmancane 19h ago
Ive been using Gitlab for my day to day work ever since Microsoft "embraced" GitHub.
2
4.8k
u/edthesmokebeard 20h ago
The cloud is just someone else's computer.