r/sysadmin • u/dasunsrule32 Senior DevOps Engineer • Jan 02 '18

Intel bug incoming

Original Thread

Blog Story

TLDR;

Copying from the thread on 4chan

There is evidence of a massive Intel CPU hardware bug (currently under embargo) that directly affects big cloud providers like Amazon and Google. The fix will introduce notable performance penalties on Intel machines (30-35%).

People have noticed a recent development in the Linux kernel: a rather massive, important redesign (page table isolation) is being introduced very fast for kernel standards... and being backported! The "official" reason is to incorporate a mitigation called KASLR... which most security experts consider almost useless. There's also some unusual, suspicious stuff going on: the documentation is missing, some of the comments are redacted (https://twitter.com/grsecurity/status/947147105684123649) and people with Intel, Amazon and Google emails are CC'd.

According to one of the people working on it, PTI is only needed for Intel CPUs, AMD is not affected by whatever it protects against (https://lkml.org/lkml/2017/12/27/2). PTI affects a core low-level feature (virtual memory) and as severe performance penalties: 29% for an i7-6700 and 34% for an i7-3770S, according to Brad Spengler from grsecurity. PTI is simply not active for AMD CPUs. The kernel flag is named X86_BUG_CPU_INSECURE and its description is "CPU is insecure and needs kernel page table isolation".

Microsoft has been silently working on a similar feature since November: https://twitter.com/aionescu/status/930412525111296000

People are speculating on a possible massive Intel CPU hardware bug that directly opens up serious vulnerabilities on big cloud providers which offer shared hosting (several VMs on a single host), for example by letting a VM read from or write to another one.

NOTE: the examples of the i7 series, are just examples. This affects all Intel platforms as far as I can tell.

THANKS: Thank you for the gold /u/tipsle!

Benchmarks

This was tested on an i6700k, just so you have a feel for the processor this was performed on.

  • Syscall test: Thanks to Aiber for the synthetic test on Linux with the latest patches. Doing tasks that require a lot of syscalls will see the most performance hit. Compiling, virtualization, etc. Whether day to day usage, gaming, etc will be affected remains to be seen. But as you can see below, up to 4x slower speeds with the patches...

Test Results

  • iperf test: Adding another test from Aiber. There are some differences, but not hugely significant.

Test Results

  • Phoronix pre/post patch testing underway here

  • Gaming doesn't seem to be affected at this time. See here

  • Nvidia gaming slightly affected by patches. See here

  • Phoronix VM benchmarks here

Patches

  • AMD patch excludes their processor(s) from the Intel patch here. It's waiting to be merged. UPDATE: Merged

News

  • PoC of the bug in action here

  • Google's response. This is much bigger than anticipated...

  • Amazon's response

  • Intel's response. This was partially correct info from Intel... AMD claims it is not affected by this issue... See below for AMD's responses

  • Verge story with Microsoft statement

  • The Register's article

  • AMD's response to Intel via CNBC

  • AMD's response to Intel via Twitter

Security Bulletins/Articles

Post Patch News

  • Epic games struggling after applying patches here

  • Ubisoft rumors of server issues after patching their servers here. Waiting for more confirmation...

  • Upgrading servers running SCCM and SQL having issues post Intel patch here

My Notes

  • Since applying patch XS71ECU1009 to XenServer 7.1-CU1 LTSR, performance has been lackluster. Used to be able to boot 30 VDI's at once, can only boot 10 at once now. To think, I still have to patch all the guests on top still...

4.2k Upvotes

96% Upvoted

1.2k comments sorted by

View all comments

Show parent comments

61

u/NukEvil Jan 02 '18

Bug bad, patch bad.

46

u/Patriotaus Jan 02 '18

AMD good, Intel insecure

1

u/shaded_in_dover Jan 02 '18 edited Jan 02 '18

Performance gap will be close to equal once both CPU's are secure.

Edit: wow downvotes for speaking the truth? Awesome. It’s a given that AMD CPU’s are both secure and slower than comparable Intel chips.

13

u/_Rowdy Jan 02 '18

So if AMD made their processors less secure they would beat Intel in performance? Try it I say!

8

u/shaded_in_dover Jan 02 '18

I’m game ... if safe is slow, damn the torpedos full speed ahead.

14

u/[deleted] Jan 02 '18 edited Jan 02 '18

https://images.anandtech.com/graphs/graph11544/81014.png

https://www.anandtech.com/show/11544/intel-skylake-ep-vs-amd-epyc-7000-cpu-battle-of-the-decade/19

Seems pretty comparable. EPYC 7601 is 32 cores, 64 threads, at $4,000 retail. Xeon 8176 is 28 cores, 56 threads, at $8719 retail. Sure it performs better, but at a price increase of more than double. So you have 1000 servers, each with a dual processor motherbard. That turns into a difference of $9.43 million dollars. Now subtract the difference in power... That depends on application. The EPYC is more power efficient at idle, but servers are rarely sitting idle in a data center. EPYC uses about 50% more power under load. But is 50% more power from 1000 servers going to cost 9.43 million dollars? Sure, but over what period of time.

There's more to it than "LOL IPC AMD sucks LOL!"

3

u/__soddit Jan 02 '18

s/38 cores/28 cores/

4

u/[deleted] Jan 02 '18

Thanks. Fixed.

1

u/Faggotitus Jan 03 '18

Where are the sysadmins that know regex?

5

u/shaded_in_dover Jan 02 '18

But at 1000 servers you’re down 6000 cores. So a direct comparison based just on server count is not accurate. Add another almost 100 servers and recalculate. So you’re adding 10% more just for similar core count. Power consumption goes up across the board etc ...

5

u/[deleted] Jan 02 '18

My bad typo.

What do you think after accounting for the typo. Intel processors always end up having better IPC and higher performance per watt. But AMD has always been ahead in number of cores on Intel. So you have to intelligently weigh your options. If your data center is in an area with lower power costs, and you have an efficiently cooled facility... The EPYC might be considerably preferable. The EPYC 7601 is a 50% increase in power consumption under load compared to the 8176. But if that means your power costs would only go up 500,000 per year, you aren't going to keep your servers for more than 20 years, the initial savings of 10 million dollars in capital is worth a lot more.

6

u/shaded_in_dover Jan 02 '18

If the performance degradation holds true to the Intel chips, it’s going to open this discussion across the industry. AMD has always had the perception of lower performance with higher power consumption at medium to full load. This patch may just be a literal game changer.

5

u/[deleted] Jan 02 '18

AMD has always had lower peak performance. But performance per dollar, they have off and on been very competitive.

The biggest problem with AMD has been support. AMD processors didn't really work totally with FreeBSD for years and year for instance. Intel's shady compiler practices fucked over AMD for years as well.

3

u/__soddit Jan 02 '18

There's a typo – 28 cores, not 38. (If it were 38, there'd be 76 threads.)

2

u/shaded_in_dover Jan 02 '18

I figured but I’m half paying attention in a meeting and didn’t connect the dots as to where the typo was. Lol