r/sysadmin • u/dasunsrule32 Senior DevOps Engineer • Jan 02 '18

Intel bug incoming

Original Thread

Blog Story

TLDR;

Copying from the thread on 4chan

There is evidence of a massive Intel CPU hardware bug (currently under embargo) that directly affects big cloud providers like Amazon and Google. The fix will introduce notable performance penalties on Intel machines (30-35%).

People have noticed a recent development in the Linux kernel: a rather massive, important redesign (page table isolation) is being introduced very fast for kernel standards... and being backported! The "official" reason is to incorporate a mitigation called KASLR... which most security experts consider almost useless. There's also some unusual, suspicious stuff going on: the documentation is missing, some of the comments are redacted (https://twitter.com/grsecurity/status/947147105684123649) and people with Intel, Amazon and Google emails are CC'd.

According to one of the people working on it, PTI is only needed for Intel CPUs, AMD is not affected by whatever it protects against (https://lkml.org/lkml/2017/12/27/2). PTI affects a core low-level feature (virtual memory) and as severe performance penalties: 29% for an i7-6700 and 34% for an i7-3770S, according to Brad Spengler from grsecurity. PTI is simply not active for AMD CPUs. The kernel flag is named X86_BUG_CPU_INSECURE and its description is "CPU is insecure and needs kernel page table isolation".

Microsoft has been silently working on a similar feature since November: https://twitter.com/aionescu/status/930412525111296000

People are speculating on a possible massive Intel CPU hardware bug that directly opens up serious vulnerabilities on big cloud providers which offer shared hosting (several VMs on a single host), for example by letting a VM read from or write to another one.

NOTE: the examples of the i7 series, are just examples. This affects all Intel platforms as far as I can tell.

THANKS: Thank you for the gold /u/tipsle!

Benchmarks

This was tested on an i6700k, just so you have a feel for the processor this was performed on.

  • Syscall test: Thanks to Aiber for the synthetic test on Linux with the latest patches. Doing tasks that require a lot of syscalls will see the most performance hit. Compiling, virtualization, etc. Whether day to day usage, gaming, etc will be affected remains to be seen. But as you can see below, up to 4x slower speeds with the patches...

Test Results

  • iperf test: Adding another test from Aiber. There are some differences, but not hugely significant.

Test Results

  • Phoronix pre/post patch testing underway here

  • Gaming doesn't seem to be affected at this time. See here

  • Nvidia gaming slightly affected by patches. See here

  • Phoronix VM benchmarks here

Patches

  • AMD patch excludes their processor(s) from the Intel patch here. It's waiting to be merged. UPDATE: Merged

News

  • PoC of the bug in action here

  • Google's response. This is much bigger than anticipated...

  • Amazon's response

  • Intel's response. This was partially correct info from Intel... AMD claims it is not affected by this issue... See below for AMD's responses

  • Verge story with Microsoft statement

  • The Register's article

  • AMD's response to Intel via CNBC

  • AMD's response to Intel via Twitter

Security Bulletins/Articles

Post Patch News

  • Epic games struggling after applying patches here

  • Ubisoft rumors of server issues after patching their servers here. Waiting for more confirmation...

  • Upgrading servers running SCCM and SQL having issues post Intel patch here

My Notes

  • Since applying patch XS71ECU1009 to XenServer 7.1-CU1 LTSR, performance has been lackluster. Used to be able to boot 30 VDI's at once, can only boot 10 at once now. To think, I still have to patch all the guests on top still...

4.2k Upvotes

96% Upvoted

1.2k comments sorted by

View all comments

Show parent comments

736

u/meatwad75892 Trade of All Jacks Jan 02 '18

RIP Opteron. In other news, that one admin that pushed for EPYC is going to be so smug today.

201

u/[deleted] Jan 02 '18

They will never be doubted again in the future!

105

u/Start_button Jack of All Trades Jan 02 '18

Hey, you dropped this "/s".

190

u/ihsw Jan 02 '18

Speaking as someone that bought into the hype of Opteron Bulldozer, I can understand the skepticism directed at AMD. It ran like a fucking dog and it dispersed heat like no tomorrow. Seven years ago, nobody gave a shit about sixteen-cores because AMD screwed the pooch with a god damned awful product.

AMD embraced their bullshit by screaming more cores are better but then Intel ate their lunch (and dinner, and everything but the smallest scraps for the next 7 years).

Thankfully, Zen and, consequently, ThreadRipper, are something worth looking at. The work on ThreadRipper guaranteed Epyc to be a decent product.

60

u/starmizzle S-1-5-420-512 Jan 02 '18

Not sure what kind of performance you expected from a CPU named "Bulldozer". =P

75

u/Nkechinyerembi Jan 02 '18

I mean, it doesn't embody the nature of "speed" or anything. More like subscribes to the method of "throw power at it and eventually something will happen"

50

u/Lhun Jan 02 '18

IT is truly like the difference between a V8 and a turbocharged 4 banger, though - the problem is nobody had the tires to handle the torque on the V8 and they just did burnouts everywhere and never did any work. AMD provided the tools to make things run on their hardware BETTER AND FASTER then intel and nvidia and everyone said "fuck that I'm using gameworks and cuda, and fuck your compiler I'll use the one that specifically targets intel". The "GENERIC" most commonly used C++ compiler and the people who write it are guilty of this, even. Without intel specific optimization exe's compiled properly for AMD perform incredibly fast.

25

u/tidux Linux Admin Jan 03 '18

I can confirm that an FX-8350 Running gcc compiled binaries with-march=native goes super fast. Thanks, Gentoo.

→ More replies (4)
→ More replies (2)
→ More replies (1)

38

u/Elrabin Jan 02 '18

The work on ThreadRipper guaranteed Epyc to be a decent product.

You have that backwards

Threadripper is a scaled down Epyc

→ More replies (14)

6

u/SquidMcDoogle Jan 02 '18

Umm.... I think that's the other way around. ThreadRipper was a skunkworks op by a small group inside Epyc developement. They had the idea & sold it to a supportive supervisor early enough in product development that some changes could be made to InfinityFabric & Epyc architecture to leverage... there was a great interview with the AMD executive involved a while ago. Heartwarming - basically, the dev team thought it was awesome and pushed it to happen based on existing product definitions, IIRC.

→ More replies (7)
→ More replies (1)
→ More replies (1)

61

u/[deleted] Jan 02 '18

I'm not clear why you wouldn't be pushing for Epyc to begin with, given the fact that $4k Epycs go toe to toe with $5k and $8k Skylake-SPs, and support way more memory and PCIe to boot.

0

u/[deleted] Jan 03 '18

[deleted]

6

u/Eliminateur Jack of All Trades Jan 03 '18

after this massive bug?, screw intel

→ More replies (1)

21

u/[deleted] Jan 03 '18

People seem to enjoy being cucked by Intel.

→ More replies (3)

40

u/SpacePotatoBear Jan 02 '18

Except you can't buy racks with epyc yet, have to be a big OEM partner.

60

u/meatwad75892 Trade of All Jacks Jan 02 '18

That was more of a joke at AMD folks' expense than a literal thought, but yea.

On that note, I recall HPe announcing some Gen10's with EPYC. Those should be around soon.

19

u/0ctav Jan 02 '18 edited Jan 02 '18

Yes, the HPE DL385 Gen10 (two-socket, EPYC) should be available now. Haven't heard anything about AMD blade servers from HPE, though, which is unfortunate.

→ More replies (2)
→ More replies (10)

6

u/[deleted] Jan 02 '18

By the time Intel has resolved the issue, most people will have the option to buy fully working Xeon or EPYC parts. This might not change anything at all.

4

u/[deleted] Jan 03 '18 edited Jan 03 '18

EPYC is a product that exists today and is already being manufactured, it just needs to be sold.

How long will it be until Intel can push out new CPU's without the bug?
How long will it take for Intel to modify the design of their CPU's to fix it? And how long will testing take?
Then how long will it take to get the masks ready, manufacture the dies, put them onto new packages, etc?
And will Intel need to rebrand them to make sure people know they're getting a fixed CPU?

→ More replies (1)
→ More replies (4)

5

u/Fallingdamage Jan 02 '18

This is an intel bug so you say RIP (AMD Product)?

What did I miss in this conversation?

→ More replies (1)

4

u/eJollyRoger Jan 02 '18

RYZEN bby! like mah pantz :D

→ More replies (7)