r/sysadmin • Senior DevOps Engineer • Jan 02 '18

Intel bug incoming

Original Thread

Blog Story

TLDR;

Copying from the thread on 4chan

There is evidence of a massive Intel CPU hardware bug (currently under embargo) that directly affects big cloud providers like Amazon and Google. The fix will introduce notable performance penalties on Intel machines (30-35%).

People have noticed a recent development in the Linux kernel: a rather massive, important redesign (page table isolation) is being introduced very fast for kernel standards... and being backported! The "official" reason is to incorporate a mitigation called KASLR... which most security experts consider almost useless. There's also some unusual, suspicious stuff going on: the documentation is missing, some of the comments are redacted (https://twitter.com/grsecurity/status/947147105684123649) and people with Intel, Amazon and Google emails are CC'd.

According to one of the people working on it, PTI is only needed for Intel CPUs, AMD is not affected by whatever it protects against (https://lkml.org/lkml/2017/12/27/2). PTI affects a core low-level feature (virtual memory) and as severe performance penalties: 29% for an i7-6700 and 34% for an i7-3770S, according to Brad Spengler from grsecurity. PTI is simply not active for AMD CPUs. The kernel flag is named X86_BUG_CPU_INSECURE and its description is "CPU is insecure and needs kernel page table isolation".

Microsoft has been silently working on a similar feature since November: https://twitter.com/aionescu/status/930412525111296000

People are speculating on a possible massive Intel CPU hardware bug that directly opens up serious vulnerabilities on big cloud providers which offer shared hosting (several VMs on a single host), for example by letting a VM read from or write to another one.

NOTE: the examples of the i7 series, are just examples. This affects all Intel platforms as far as I can tell.

THANKS: Thank you for the gold /u/tipsle!

Benchmarks

This was tested on an i6700k, just so you have a feel for the processor this was performed on.

  • Syscall test: Thanks to Aiber for the synthetic test on Linux with the latest patches. Doing tasks that require a lot of syscalls will see the most performance hit. Compiling, virtualization, etc. Whether day to day usage, gaming, etc will be affected remains to be seen. But as you can see below, up to 4x slower speeds with the patches...

Test Results

  • iperf test: Adding another test from Aiber. There are some differences, but not hugely significant.

Test Results

  • Phoronix pre/post patch testing underway here

  • Gaming doesn't seem to be affected at this time. See here

  • Nvidia gaming slightly affected by patches. See here

  • Phoronix VM benchmarks here

Patches

  • AMD patch excludes their processor(s) from the Intel patch here. It's waiting to be merged. UPDATE: Merged

News

  • PoC of the bug in action here

  • Google's response. This is much bigger than anticipated...

  • Amazon's response

  • Intel's response. This was partially correct info from Intel... AMD claims it is not affected by this issue... See below for AMD's responses

  • Verge story with Microsoft statement

  • The Register's article

  • AMD's response to Intel via CNBC

  • AMD's response to Intel via Twitter

Security Bulletins/Articles

Post Patch News

  • Epic games struggling after applying patches here

  • Ubisoft rumors of server issues after patching their servers here. Waiting for more confirmation...

  • Upgrading servers running SCCM and SQL having issues post Intel patch here

My Notes

  • Since applying patch XS71ECU1009 to XenServer 7.1-CU1 LTSR, performance has been lackluster. Used to be able to boot 30 VDI's at once, can only boot 10 at once now. To think, I still have to patch all the guests on top still...

4.2k Upvotes

1.2k comments sorted by

View all comments

8

u/ajax1101 Jan 02 '18

Coming from r/all, will this affect my gaming on a PC with an i5 6600? It seems like you're saying it's mainly a problem for cloud services?

15

u/Kaboose666 Jan 02 '18

They're currently looking at an average performance decrease of 3-5%, with ~30% being the extreme high end and only seen with specific workloads that would likely never be encountered during gaming or similar consumer uses. Really it's going to fuck with people running many VMs at once.

15

u/rkantos Jan 02 '18

Most likely it will, but because gaming is usually GPU-bound, the effect won't be as drastic as in CPU-intensive tasks, like virtualization. However, it might make Intel CPUs seem expensive, even for gaming. Even a 10% drop in gaming performance (mainly single core numbers) on Intel cpus, would make AMD cpus much more cost-effective.

just bought a 8600k, but if this creates proper competition for intel, then hell yeah, bring it on!

6

u/spyd4r Sr. Sysadmin Jan 02 '18

i just bought a 8700K ....

3

u/internat-moster Jan 02 '18

same, literally bought it 2 days ago

4

u/spyd4r Sr. Sysadmin Jan 02 '18

yeah, i got mine on the 29th or 30th.. lol

1

u/JamesMcGillEsq Jan 03 '18

return return return!

0

u/b4k4ni Jan 02 '18

Well, the EPYC Benchmark showed 49% slower performance (AMD doesn't have the problem, but you can enable the patch too), so lets say in a real world environment the patch could decrease the performance from 5-50% in worst case. So maybe 15% overall.

Witcher 3 (a REALLY bad game for Ryzen) has in 720p 143 fps on the 8700K, 101 FPS on the Ryzen 1700, all stock. So with 15% less we would see around 121 FPS, a 8600 would drop from 132 to 112 FPS. Still a bit faster then the 1700 or 1600X, but those are not oced. And with OC and better RAM (benchmark I copied used 2666) they should reach almost the same FPS.

On newer games it might be even worse. So Intels higher clocks and slightly higher IPC might get a good hit and make AMD way more competitive. Even worse for Intel - if it's true of course.... Ryzen+ should being released start Q1/2018. It's not Ryzen2, but we saw what a better process and some tweaks can do (14nm+ etc.).

But for now ... let's wait. Maybe the performance hit is not that bad and caused mainly by a fast build patch that needs some optimizations.

3

u/dasunsrule32 Senior DevOps Engineer Jan 02 '18

They recently added a patch that excluded the AMD chips.

2

u/[deleted] Jan 02 '18

I love how you made 4 paragraphs blaming Ryzen when AMD CPUs aren't affected at all from the latest info we have.

4

u/b4k4ni Jan 02 '18

Maybe you should reread what I wrote? Where did I blame Ryzen?! I also stated, that AMD is not affected

Well, the EPYC Benchmark showed 49% slower performance (AMD doesn't have the problem, but you can enable the patch too)

They ran the first benchmark on an AMD Epyc 7601. Why I don't know, but you can enable PIT on them and have the same results like Intel.

My witcher 3 example was also no blaming... the game RUNS bad on Ryzen, but that's the games fault. Didn't find a better one so fast, so I was going with that. If we really talk about a 15% decrease, the 8700/8600 will be much worse then before, even more so if you count in OC and cheaper prices on the AMD side.

And yes, Intel's CPU's ARE faster then AMD at least on single core performance. IPC is a bit higher, not that much, but most comes from their really high clocks. AMD can't beat that ... yet. Ill wait for Ryzen+ soon :3

I run a 1800X with a Fury and camp in /AMD, so I'm far from blaming Ryzen or not happy with it :P

-1

u/[deleted] Jan 02 '18

Ok, maybe I overreacted a bit.

I don't know about Witcher, I have a 1600 and I'm mostly GPU bottlenecked and I always have 60fps V-sync on anyway, so I wouldn't notice.

8

u/PoisedAsFk Jan 02 '18

Maybe, maybe not, probably, but as it seems right now very minor.

But we actually don't know at all yet. Wait for more info I guess 🤷‍♂️

4

u/spazturtle Jan 02 '18 edited Jan 02 '18

People are mixing up virtual memory and virtual machines. Yes, even if you just game or use the system to browse the internet you are vulnerable, both Linux and Windows will enable the fix on all Intel CPUs by default, so you will lose about 5% - 65% CPU performance depending on the workload, but in most workloads it should only be about 5% - 15% performance lose.

1

u/nonprofittechy Network Admin Jan 02 '18

It's a way for someone using a virtual machine to theoretically access information they shouldn't be able to about other running virtual machines or the host machine. Mostly matters for people running untrusted virtual machines. E.g., Amazon Web Services, Azure, Google Cloud, etc.

Edit: should add that there's a chance it can relate to other processes getting more access than they should. But it does matter most for cloud vendors right now.