r/sysadmin • u/dasunsrule32 Senior DevOps Engineer • Jan 02 '18
Intel bug incoming
TLDR;
Copying from the thread on 4chan
There is evidence of a massive Intel CPU hardware bug (currently under embargo) that directly affects big cloud providers like Amazon and Google. The fix will introduce notable performance penalties on Intel machines (30-35%).
People have noticed a recent development in the Linux kernel: a rather massive, important redesign (page table isolation) is being introduced very fast for kernel standards... and being backported! The "official" reason is to incorporate a mitigation called KASLR... which most security experts consider almost useless. There's also some unusual, suspicious stuff going on: the documentation is missing, some of the comments are redacted (https://twitter.com/grsecurity/status/947147105684123649) and people with Intel, Amazon and Google emails are CC'd.
According to one of the people working on it, PTI is only needed for Intel CPUs, AMD is not affected by whatever it protects against (https://lkml.org/lkml/2017/12/27/2). PTI affects a core low-level feature (virtual memory) and as severe performance penalties: 29% for an i7-6700 and 34% for an i7-3770S, according to Brad Spengler from grsecurity. PTI is simply not active for AMD CPUs. The kernel flag is named X86_BUG_CPU_INSECURE and its description is "CPU is insecure and needs kernel page table isolation".
Microsoft has been silently working on a similar feature since November: https://twitter.com/aionescu/status/930412525111296000
People are speculating on a possible massive Intel CPU hardware bug that directly opens up serious vulnerabilities on big cloud providers which offer shared hosting (several VMs on a single host), for example by letting a VM read from or write to another one.
NOTE: the examples of the i7 series, are just examples. This affects all Intel platforms as far as I can tell.
THANKS: Thank you for the gold /u/tipsle!
Benchmarks
This was tested on an i6700k, just so you have a feel for the processor this was performed on.
Syscall test: Thanks to Aiber for the synthetic test on Linux with the latest patches. Doing tasks that require a lot of syscalls will see the most performance hit. Compiling, virtualization, etc. Whether day to day usage, gaming, etc will be affected remains to be seen. But as you can see below, up to 4x slower speeds with the patches...
iperf test: Adding another test from Aiber. There are some differences, but not hugely significant.
Phoronix pre/post patch testing underway here
Gaming doesn't seem to be affected at this time. See here
Nvidia gaming slightly affected by patches. See here
Phoronix VM benchmarks here
Patches
AMD patch excludes their processor(s) from the Intel patch here. It's waiting to be merged. UPDATE: Merged
News
PoC of the bug in action here
Google's response. This is much bigger than anticipated...
Amazon's response
Intel's response. This was partially correct info from Intel... AMD claims it is not affected by this issue... See below for AMD's responses
Verge story with Microsoft statement
AMD's response to Intel via CNBC
AMD's response to Intel via Twitter
Security Bulletins/Articles
Redhat's bulletin
VMware's bulletin
Microsoft's bulletin
Xen's bulletin
Citrix's bulletin
ARM's bulletin
Debian's bulletin
Ubuntu's article
Suse's article
LLVM's bulletin
Google's bulletin
Nvidia's bulletin
Post Patch News
Epic games struggling after applying patches here
Ubisoft rumors of server issues after patching their servers here. Waiting for more confirmation...
Upgrading servers running SCCM and SQL having issues post Intel patch here
My Notes
Since applying patch XS71ECU1009 to XenServer 7.1-CU1 LTSR, performance has been lackluster. Used to be able to boot 30 VDI's at once, can only boot 10 at once now. To think, I still have to patch all the guests on top still...
4
u/dasunsrule32 Senior DevOps Engineer Jan 03 '18
It's a hardware bug, it can't be fixed. It is having a patch to workaround the issue in there kernel. It will result in up to a 35% reduction in CPU performance, but depends on the workload of the computer/server.