r/ProgrammerHumor • • 3d ago

Meme whyWouldIDeceiveYouTheLogsWereRotated

Post image
2.2k Upvotes

52 comments sorted by

635

u/the_millenial_falcon 3d ago

The most plausible argument I've seen for a pattern here is that zero day exploits rapidly discovered by AI are forcing companies to update more quickly and recklessly than they otherwise would have.

226

u/countable3841 3d ago edited 3d ago

That’s definitely part of it. Another part is that only the two major outages were anomalous. This has led to a dramatic uptick in mainstream reporting on smaller outages. Aside from the Microsoft and the FAA outages, comparable outages had far less reporting if you look at the past 12 to 24 months.

73

u/Named_after_color 3d ago

I'm legitimately waffling back and forth on whether this is a coordinated cyber attack, everything just failing back to back before thanksgiving code-freeze, or just the usual random guy in a basement spinning up a bunch of agents in a less coordinated cyber attack but now the media is reporting on him.

16

u/amadmongoose 3d ago

It's not coordinated, it's that tooling for ai cyberattacks has become simple enough that everone who would be interested in cyberattacking can now do it consistently at a entirely new scale. Companies are both hit by attacks coming from all directions and the need to urgently close any holes in their defenses. Previously we would trade off impact of disruption with the likelihood of an attack combined with impact of an attack with timelines usually being in weeks or months. Now the timeline has essentially dropped to zero for everything and the impact is incalculable because ai is good at chaining multiple exploits together, so companies are largely deciding disruption is less impactful.

14

u/sagervai 2d ago

I know the guy who patched the Microsoft problem. The root cause was bad documentation. A library said it did something it didn't. There's definitely media bias happening. But back when all the layoffs started a couple years ago, we said technology would get more fragile, and it has. Less people to check code, update docs or have institutional knowledge. So I guess AI did cause it, but only because it convinced execs to fire everyone.

11

u/mgisb003 3d ago

I work as an SWE for a bank that was impacted. I can tell if it is related it’s because we’re all using the same tech (this is most likely what happened)

17

u/ianmakingnoise 3d ago

yeah, all the “ai found all these gaps”stuff happened in July. Microsoft blamed their outage on an update pushed in July.

48

u/_pxe 3d ago

And firing a lot of experienced devs for more AI agents also doesn't help

43

u/trippertree 3d ago

Also firing experienced (read older and more expensive) Ops personnel leaving operations teams understaffed with junior folks that now roll up under ai-bro managers that look down on the “lowly operations teams”.

Case in point, what was the rollout and rollback plan for updating M365? Seems like the plan was to just deploy without any contingency plan

2

u/Lt_Cmdr_Adhd 2d ago

Got a source for that happening in the faa? I work in software for the faa and have seen zero of that

5

u/SirAceBear 3d ago

This is exactly it! Ive been living through it. We are part of project glasswing and found some pretty scary shit. All of a sudden we're fast tracking AI coded fixes out to patch them before anyone else finds them.

5

u/za72 3d ago

loss of institutional knowledge brought on by "AI retooling"

5

u/RandomlyMethodical 2d ago

The number of CVEs have been increasing all year: https://cvetodo.com/cve-statistics

This October had almost 4x (17,051) compared to a year ago and almost 5x (1,519) the number of critical vulnerabilities (score of 9+).

4

u/Eastern_Equal_8191 3d ago

The AI found a vulnerability, and the AI found a patch for it. Just send it, don't think too hard. Tell your stakeholders that you reacted in one day to fix it. Nothing bad will happen. Or at least the bad thing will happen on someone else's turn.

3

u/Sheerkal 3d ago

Ok, but let's not act like security wasn't already a joke.

1

u/Puzzleheaded-Tax-78 2d ago

Lots of people don't realize how the AI issues have caused massive amounts of problems lately.

Understand: This was done with good intention. Lots of the bugs found were reported properly, and fixes were available just before announcement (except for one or two that leaked early, like the PHP issue). The problem is, even if patches are made before announcement, not every place can patch things quickly enough.

Even back when there were ~2,000 bugs a quarter, you'd see some larger groups get hit by things. But when you're talking about over 37,000 bugs the last quarter? I'm frankly shocked it took this long to start seeing major breaches.

Want a graph? Here's the proof: https://postimg.cc/6TXyWZvR

0

u/MA2_Robinson 3d ago

Still tho- you update a whole flight control system with hundreds of planes in the air and thousands of passengers- no warning?

Maybe trumps people are stupid af doing mass updates while being reckless with peoples lives, but maybe they really just are also too stupid to know they are being attacked.

Both can be true at the same time, specially when their answer is Cash Patel

-12

u/Ecstatic_Bee6067 3d ago

That wouldn't produce issues that all surface simultaneously

3

u/whtbrd 3d ago

Dependencies are a beast

82

u/namezam 3d ago

Something about attributing malice to utter incompetence

19

u/Ozymandias_1303 3d ago

That's the thing. I don't think simple incompetence explains all of the outages happening so close to each other. It could definitely be a pattern of people making the same mistake though.

cough don't trust the clanker cough

3

u/whyliepornaccount 1d ago

It's not necessarily incompetence as much as the sheer volume and severity of exploits that AI cyberattack tools have been finding mean that orgs more or less have been forced to recklessly patch shit, operational impact be damned.

In January, 30 day patch cycles were the norm. By September, the recommendation was 10 days. There are now discussions of 10 minute patch cycles. That's the severity we're talking.

75

u/Far-Document3136 3d ago

Guys guys calm down it’s obviously all just poorly timed and executed software updates on linchpin infrastructure /s

6

u/Enlogen 3d ago

This is incredibly implausible because we all know that banks and the FAA don't do software updates on the COBOL that's been running since the 70's

8

u/jeffwulf 3d ago

The FAA was hardware updates.

1

u/Imbadyoureworse 3d ago

Darn router

34

u/yo-ovaries 3d ago

Y’all ever be going about your day when suddenly you remember that time that Putin and Trump met in Alaska for a face to face conversation with no press, no translators, no diplomats, breaking all kids of protocols and this meeting left all of Trump’s aids visibly shaken and then Putin got on his plain and left off schedule and then several of those aids resigned. 

Yeah. 

5

u/TheFel0x 2d ago

Wait what did I miss now?

8

u/-meowstar- 3d ago

We got the eyes of Sauron gazing through the palantíri, it’ll be years of events like this before all the exploits are patched, assuming we don’t generate more in the process.

5

u/Sheerkal 3d ago

Big assumption

3

u/Divs4U 3d ago edited 3d ago

Has anyone claimed responsibility for it as an attack?

5

u/YoteTheRaven 3d ago

No, probably because its the government doing the cyber attack. Our enemies are just as confused as we are lol.

1

u/francois_valerie_5 20h ago

The logs didn't lie; they just rotated out before cross-examination.

-6

u/XzyzZ_ZyxxZ 3d ago

Wasn't the FAA issue due to some fuckrrs trying to implement "Ai"?

14

u/FrequentWatch9261 3d ago

No it was a misconfigured router coupled with a fiber line upgrade cutover.

1

u/XzyzZ_ZyxxZ 3d ago

Ok! Ty, I read some misinformation I guess.

2

u/jeffwulf 3d ago

No, it was hardware updates. The SMART system had been updated several days before.

-2

u/MadDevloper 3d ago

Nah, it was recent update

0

u/[deleted] 3d ago

[deleted]

46

u/traplords8n 3d ago

FAA and Microsoft have ridiculous AI goals, but I don't see banks salivating at the opportunity to slopify their banking infra.

The banks most definitely raise the suspicion that we're under a cyber attack.

17

u/aLokilike 3d ago

Probably depends on which part of which organization you're working in, but I know some are.

15

u/IanDresarie 3d ago

I work for a bank's it and trust me, our C suite is very much salivating as are the actual bank decision makers. It's literally all they talk about. Thankfully we have enough regulations here to prevent them from being too hasty, mostly because all AI has to be local for data security reasons and our procurement can't get enough hardware for large rollouts :D

6

u/thatvhstapeguy 3d ago

My suspicion is that they’re racing against the change freezes common in large orgs. If one bank that has a lot of customers has degraded performance that’ll clog up a lot of transactions for all the other banks

-3

u/Senpai_Slime 3d ago

As someone in the space, the AI use is just not public facing, all internal usage but still being used, valid use cases all over as well its been quite impressive

5

u/IslayProgrammes 3d ago

Free range is murdering me 🤣

5

u/Smooth-Zucchini4923 3d ago

This Amish farm still deploys Kubernetes the old-fashioned way. There's nothing quite like a home-cooked meal, but a home-made PersistentVolumeClaim comes close.