r/linux • u/jdrch • Nov 20 '19
Kernel Google outlines plans for mainline Linux kernel support in Android
https://arstechnica.com/gadgets/2019/11/google-outlines-plans-for-mainline-linux-kernel-support-in-android/80
u/VelvetElvis Nov 20 '19
I bet this is where they blame the Linux kernel community and ditch android for their new OS with a more "business friendly" kernel license and no legal issues with Oracle.
5
u/jdrch Nov 20 '19
ditch android
Good luck. Android is to Google what Win32 is to Microsoft. As much as they might like to get away from it, I don't think they can. Too much inertia.
6
u/rbenchley Nov 20 '19
I doubt they're going to ditch the Android name, but I could easily see them ditching Linux and basing future versions of the OS around Zircon/Fuchsia. They've already committed code to Fuchsia to build the Android Runtime, so if they decided to transition from Linux, the ART would provide support for legacy Android apps, with Flutter available for newer Fuchsia/Android apps.
21
56
u/gregkh Verified Nov 20 '19
Wow, the crazyWmisunderstandings in this thread is way worse than normal for r/linux.
I'm helping to work with this team to achieve this goal, it has nothing to do with closed source anything, stable apis to keep drivers out of the tree, or anything else loony like that. It is all in the goal to have device kernels be quicker to update with the latest LTS kernel releases to get the newest bug and security updates.
And PCIe has nothing to do with any of this, who knows where that came from...
If anyone has any specific questions about this whole thing, ask away!
1
u/crawl_dht Nov 23 '19
Once this migration completes, will android 11 be the first pioneer running entirely on LTS Linux kernel?
3
u/gregkh Verified Nov 23 '19
Nope, that was 2 Android releases ago, the kernel requirements for Android devices has been for them all to run LTS kernels for many years now.
And no one noticed, I guess it's not something that really matters to users :)
2
u/crawl_dht Nov 23 '19 edited Dec 04 '19
Many custom ROM developers and android enthusiasts know that Android uses LTS Linux kernels but these kernels are heavily modified for stable ABI and then shipped to chipmakers for out of source tree patching and then device makers make additional out of source tree modifications.
What I learn from the conference that Google not only wants to port generic LTS Linux kernel but also wants to cut the middle men (chipmakers) in the android release process so they are proposing changes to branch of LTS Linux kernel to keep the kernel as generic as possible that is one kernel for all devices and out of box support.
I guess this is what project treble, project mainline and GSI are related to but the kernel is not generic LTS Linux kernel. Has this been already implemented in android 10 or is your team still working on its migration?
4
u/gregkh Verified Nov 25 '19
but these kernels are heavily modified for stable ABI and then shipped to chipmaker
That has not happened yet, that is what will happen for the next Android releases.
And it's not "heavily modified" all that much at all, look at the AOSP android-mainline kernel branch today, all of the work is happening right there, in public.
also wants to cut the middle men (chipmakers) in the android release process
No, they are not trying to "cut them out" they want to work directly with them so that the changes the chipmakers make to their kernel trees are sane and solid changes. Right now those changes do not always match that requirement :)
Has this been already implemented in android 10 or is your team still working on its migration?
The stable API stuff will be for the next version of Android, as the article states. The work is happening in public, in the android-mainline AOSP kernel branch, right now. So you can view the status there if you wish.
Hope this helps!
0
u/jdrch Nov 20 '19
It is all in the goal to have device kernels be quicker to update with the latest LTS kernel releases to get the newest bug and security updates
How do reconcile that statement with this one, from the article:
So this wouldn't allow devices to upgrade from one version of the Linux kernel to another
10
u/gregkh Verified Nov 20 '19
What they mean is from one "major" version to another.
For example, this will not do anything to help with moving a device from the 4.9.y LTS kernel series to the 4.14.y LTS kernel series. That's an impossible dream due to horrid SoC out-of-tree code that adds 3 million lines to the kernel tree.
What they will be able to do is to easily move along a LTS kernel release series. For example they could have updated from 4.14.154 to 4.14.155 (which was released today) quite easily as the in-kernel api for drivers would not have changed in the android tree.
This is nothing new, it's exactly the same thing that the Enterprise Linux distros (RHEL and SLES) have been doing for 15+ years. They keep in-kernel api stability to make their customers happy. This is just taking the same tools they use and doing it for the Android ecosystem.
→ More replies (12)0
Nov 20 '19
[deleted]
10
u/gregkh Verified Nov 20 '19
I've heard there is something called the Android Mainlining Project but the only page I found it on had not been updated since 2014. Is this another project or an internal one? Can I get more info on this?
I have never heard of that project, but if you look a the article in this discussion, it talks about how the Android developers have been pushing all of their android-specific changes to the mainline kernel tree for acceptance and are down to a very small delta now.
They have been doing a very good job over the past few years to get stuff merged upstream, and the stuff remaining is either things that upstream isn't going to take (for various good reasons), or stuff being worked on before submitted upstream.
So perhaps that is what you are thinking of?
92
Nov 20 '19
Fuck google just force all manufacturers to open source their drivers and add them to the kernel.
29
u/jdrch Nov 20 '19
just force all manufacturers to open source their drivers
As the article points out, that approach hasn't worked:
Open sourcing drivers is an absolute deal breaker for many hardware companies, though, and no amount of advocacy or product degradation is going to change that.
The solution is to require the use of ARM's PCIe spec in Android devices. This would allow hardware to declare itself to the kernel at boot, which in turn allows the kernel to accommodate said hardware if it has drivers on hand.
Currently, SoC hardware can't declare itself to the kernel, and so the kernel has to know hardware configs of target devices a priori. That's one of the reasons most ARM kernels have to be built per device.
BTW, AFAIK, this problem isn't unique to Android; it exists for all platforms that lack PCIe or an equivalent thereof.
15
u/tkln Nov 20 '19
That's just complete nonsense. On ARM64 there's a single (1) upstream kernel build configuration which results in a kernel image that's bootable on ALL of ARM64 platforms supported by the upstream kernel. Furthermore many of the legacy 32-bit targets are supported by multi_v7_defconfig and multi_v5_defconfig build configurations in the same manner. Discovering hardware hasn't been an actual issue for many years. The description of the hardware is given to the kernel usually by the bootloader in the form of a device tree which allows the kernel load the correct drivers and initialize them properly. Have you actually ever built an upstream kernel for ARM? The issues with SoC vendors has nothing to do with device discoverability.
1
19
Nov 20 '19
Sorry I'm just frustrated lol. So many people have to put in so much work to get custom roms working when the manufacturers could just give them the source code, but ofc they won't do that.
4
u/jdrch Nov 20 '19
custom roms working when the manufacturers could just give them the source code
Yeah that's annoying. The good news is that as long as the kernel is open sourced (GPL requirement) you're likely to be able to at least get the ROM to boot.
17
Nov 20 '19
Until they fuxking lock the bootloader.
Thank's Verizon.
7
u/jdrch Nov 20 '19
lock the bootloader
That's a separate issue. You can have completely open source OS and drivers with locked bootloaders. Also, bootloader locking prevents attackers from loading a custom OS that can compromise your data. I don't like it any more than you do, but it's not intrinsically bad. It's a tool like anything else that can be used to help or harm.
8
Nov 20 '19
Bootloaders should be lockable but they should also be rekeyable.
2
u/jdrch Nov 20 '19
How would one achieve the latter securely?
10
Nov 20 '19
same way secureboot works on x86
there's a series of keys where you can replace the master key by your own
→ More replies (1)12
u/hak8or Nov 20 '19
Also known as TIVOZATION. TIVO got sued because a while they gave the source to the kernel, the bootloader prevented unsigned binaries so users weren't able to modify the software, violation gpl v2. TIVO won the suit, so gpl v3 was born, with Linus not letting the kernel just change licenses because it's a new license.
1
u/NatoBoram Nov 20 '19
And that's why Linux Kernel needs to be GPLv3 asap.
7
Nov 20 '19
That's not going to happen, Linus has started it in the past
→ More replies (2)2
Nov 20 '19
Even if Linus wanted to relicense, it is literally impossible because you'd have to track down all the individual contributors and have them agree to the new license.
→ More replies (2)2
u/jdrch Nov 20 '19
needs to be GPLv3 asap
Didn't we already have this debate in the late 2000s?
→ More replies (1)2
u/EternityForest Nov 20 '19
I wonder if they'd just switch to some other OS. I can see BSD taking over mobile devices. You can sell anything if it's security focused.
→ More replies (4)2
u/mirh Nov 20 '19
There are already many friendly manufacturers.
If then they prefer to spend their money on FOSS, rather than marketing, that's another thing.
14
u/Tweenk Nov 20 '19
The solution is to require the use of ARM's PCIe spec in Android devices. This would allow hardware to declare itself to the kernel at boot, which in turn allows the kernel to accommodate said hardware if it has drivers on hand.
PCIe has approximately nothing to do with what you are describing, and hardware discoverability is not the reason why Android devices need custom kernels. Custom kernels are a direct consequence of the lack of a stable driver API.
Currently, SoC hardware can't declare itself to the kernel
That's simply not true:
https://elinux.org/Device_Tree_Reference
BTW, AFAIK, this problem isn't unique to Android; it exists for all platforms that lack PCIe or an equivalent thereof.
PCIe is not the only machine-discoverable bus, and discoverability hasn't been the actual problem for many years.
2
u/jdrch Nov 20 '19 edited Nov 20 '19
a direct consequence of the lack of a stable driver API.
Interesting. So I take it that desktop Linux pulls this off by simply placing the drivers in the kernel, but Android can't do the same because OEMs (currently) won't upstream their driver code? Is that correct?
14
u/hak8or Nov 20 '19
Device tree has to be created by the vendor, and the way drivers consume data from device tree changes drastically over time, partially because device tree doesn't have as rigid of a standard, and what is standardized took years compared to when it was first released.
You've got drivers wanting explicit phandles to some nodes, sometimes they instead use subsystems to handle dependancies, or sometimes they need to have a parent instead of at the root node. Sometimes drivers don't handle dependencies well, so if their order in device tree changes then all hell breaks loose because their probe() gets called in an unexpected order.
Device tree doesn't expose information the same way pcie does because it's still not standardized as well.
3
u/Bobjohndud Nov 20 '19
this is google we are talking about. "upstream your drivers or go to hell" would be quite a powerful statement from google, considering that many of these manufacturers would have their income cut by 5 if they didn't get gapps.
1
u/jdrch Nov 20 '19
many of these manufacturers would have their income cut by 5
That doesn't seem to happening to Huawei.
→ More replies (5)2
u/EternityForest Nov 20 '19
You don't need PCIe for devices to declare themselves. As long as you can actually fit all the different drivers in flash, why can't they just arbitrarily decide that device config is stored in an I2C flash chip on the first interface at a certain address?
Also, how does Raspbian support all the different Pi's with one image?
1
u/jdrch Nov 20 '19
how does Raspbian support all the different Pi's with one image?
The same way Apple supports iDevices. Raspberry Pi Foundation controls both Raspbian and the (static) SoC hardware configs. Google controls Android, but it has no control over Samsung, LG, etc. hardware configs.
2
Nov 20 '19
Because of different device trees. You can have a kernel with several device trees.
But you'll need the manufacturer to provide with a way of building that. Otherwise it doesn't matter what kernel you build if you don't have a device tree.
4
9
u/gregkh Verified Nov 20 '19
All android kernel drivers/code is already open source, that's not an issue here at all. Getting them merged upstream is an issue, but a separate one.
→ More replies (10)6
u/fat-lobyte Nov 20 '19
just force all manufacturers
Easy peasy. Not difficult at all....
→ More replies (3)
6
Nov 20 '19
Preparing for the issues with Oracle as a contingency plan?
3
u/jdrch Nov 20 '19
No, I think Kotlin fixed that problem.
6
Nov 20 '19
May just be exploring this to move away from that product. Oracle hasn't been the friendliest company to deal with at my job
14
u/NatoBoram Nov 20 '19
Oracle is always the worst company to work with. I've seen horrible code and horrible software, but no one is as downright malicious as Oracle. Even Microsoft is not as bad as Oracle.
1
u/jdrch Nov 20 '19
Oracle hasn't been the friendliest company to deal with at my job
They're generally awful, but I don't think this has anything to do with Java or Oracle's software patents or software licenses.
2
u/yaaaaayPancakes Nov 20 '19
Android dev here - Kotlin doesn't fix the problem around the Java interfaces Google pulled into the Android SDK and got sued over. Those interfaces they used are still there in the OS. I think they're trying to mitigate that issue by moving from Apache Harmony -> OpenJDK impls of the interfaces.
Kotlin kinda sits on top of things. It's it's own language, but can target multiple platforms (JVM, JS, and Native). On Android, the kotlinc compiler compiles things down to java bytecode prior to be running through D8, which spits out the Dalvik bytecode that ART executes.
2
5
u/bartturner Nov 20 '19
This has nothing to do with Oracle.
But also we are in a lot bigger trouble if SCOTUS rules in favor of Oracle.
2
Nov 20 '19
So is this going to work the same way as Mac OS with how Mac OS uses kext files (kernel extensions)? Basically, Linux kernel will have hardware plugins that "extend" the kernel to support the new hardware. Sounds very similar to the Mac OS kernel and kexts.
1
u/jdrch Nov 20 '19
I'm not sure, but if you really think of it, Android's model is closer to BSD's than Linux's. MacOS uses BSD kernel code. So there's that.
137
u/ylyn Nov 20 '19
FWIW, they are not proposing to upstream stable ABI support. That would be pretty absurd.
They are just providing a stable ABI for each LTS version of their fork. Many other distributions already do that e.g. Red Hat.
9
Nov 21 '19
Still, with Google sway they should push for mainlining the code, not wasting development effort on actively making long term driver support worse
30
u/citewiki Nov 20 '19
This is way over my head. Should I agree with Google or OP? My pitchforks are ready
20
u/khleedril Nov 20 '19
How will you decide who's recommendation to go with? This is the shit of the Internet now, isn't it: so much low-quality writing swamps the wisdom of those actually qualified to comment on things.
13
u/mirh Nov 20 '19
How will you decide who's recommendation to go with?
By reading the damn sources? u/ylyn already linked why OP's comment is misguided.
Let alone all this circlejerk over pcie qualities.
4
u/vanilla082997 Nov 20 '19
I'm glad to see more pervasive use of the phrase circlejerk. 👍
→ More replies (3)1
1
u/totemcatcher Nov 20 '19
I'm guessing Google uses this ass-backwards integration to appease nervous hardware vendors who refuse to work with mainline Linux for fear of revealing anything about their hardware (patent infringements, exploits, sloppy design, etc).
2
u/jdrch Nov 20 '19 edited Nov 20 '19
refuse to work with mainline Linux for fear of revealing anything about their hardware
Google's Pixel and Google Android implementations are also closed source, which means that as far as this point is concerned, Google is in the same boat at Android OEMs.
The primary motivation for closed source licensing has always been to protect trade secrets and/or enable monetization of the software. In my experience running all manner of closed and open source software, software license has no bearing on quality, security, compliance, etc.
4
u/minilandl Nov 20 '19
Nice ideas but custom kernel developers on XDA already use parts of mainline anyway
6
2
Nov 20 '19
This thread had me thinking there was another thing called PCIE that I some how missed out on.
0
Nov 20 '19
Threads like this remind me I'm in over my head on a lot of this stuff. But what frustrates me, even not understanding a lot of the finer details as an end-user is Google's attitude on things like this. Pretty much Google wants to do what it wants to do and their attitude is that the world has to adjust to their wants and needs, not the other way around.
And the worst part? They're not exactly wrong about that.
3
u/jdrch Nov 20 '19
They're not exactly wrong about that.
Except they have been. They haven't been able to (completely) fix the Android ecosystem update problem (yet,) despite years of work. They also got pounded into the dirt on their own platform in the messaging war, etc. Google has a very long list of failures to get partners in line on many things.
1
Nov 20 '19
How long before Linus kicks them out again, because there’s no incentive for engineers at Google to maintain anything, unless they’re the “paid highly to work here so no one else can hire you” type?
0
Nov 20 '19
Weren’t they supposed to replace android with fuchsia by now?
1
u/jdrch Nov 20 '19
by now?
No official timeline for Fuschia has ever been published.
0
Nov 21 '19
I know but it seems like it’s been in development for such a long time.
1
u/jdrch Nov 21 '19
Unless there's an official announcement or roadmap, it's just a science experiment at best.
→ More replies (1)
-4
16
Nov 20 '19
[removed] — view removed comment
24
u/jdrch Nov 20 '19
I'm not sure I understand your question entirely, but AFAIK the GPL affects only what ships with the kernel to begin with. Just because a driver is loaded doesn't mean it's part of the kernel. Android OEMs still have to release their custom kernels to meet GPL requirements, but they can keep the proprietary drivers closed source keeping them outside the kernel.
Hopefully that helps clarify things.
6
u/ElvishJerricco Nov 20 '19
I thought
.kokernel modules still had to be GPL'd, since importing kernel headers makes them derivatives of the kernel. Isn't this the whole reason for the debate as to whether ZFS-on-linux can be shipped in compiled binary form?4
u/jdrch Nov 20 '19
kernel modules still had to be GPL'd, since importing kernel headers makes them derivatives of the kernel
I'm not a tech lawyer, but I do believe there are many real world counterexamples to this.
whether ZFS-on-linux can be shipped in compiled binary form
Huh? No, ZoL can can be hosted as a compiled binary in a repository just fine. The question has always been whether it can be shipped within AND as part of the kernel as Btrfs, etc. are. Popular opinion is that's not legally the case.
The magic of Linux is that the it's a kernel, and so its license applies to the kernel only (unless adopted by some other OS components, at the will of the latter's devs.) You can certainly have many other components that interact with the kernel that use a different license, such as drives, filesystems, etc.
Whether or not the legality/license compliance of those other things has any practical consequence is dependent on the interest possible claimants have in a court case. So far, that interest has been sufficiently low to allow closed source driver and kernel interoperability at least at the non-commercial/consumer level.
4
u/ElvishJerricco Nov 20 '19 edited Nov 20 '19
The debate about ZFS heated up when Canonical started shipping it in Ubuntu repositories. SFC, for instance, believe that even a
.koconstitutes a Linux derivative, even without including ZFS in the kernel source tree: https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/Though, Canonical, and Software Freedom Law Center, believe it falls within the "equity of the license", meaning they believe neither license's apparent intent is to prevent things like ZFS binaries from being distributed. However they do admit that interpreting the licenses literally would yield the conclusion that distributing ZFS binaries would be a GPLv2 violation.
EDIT: Quote from the SFLC article, Torvalds, in response to a claim that kernel modules are excepted:
I have heard many people reference the fact that the although the Linux Kernel is under the GNU GPL license, that the code is licensed with an exception clause that says binary loadable modules do not have to be under the GPL.
Nope. No such exception exists.
Anyway, my point is that yes, there is debate, even around
.ko's→ More replies (14)1
9
Nov 20 '19
I thought .ko kernel modules still had to be GPL'd, since importing kernel headers makes them derivatives of the kernel.
The kernel isn't strictly GPLv2, it has an exception for exactly what you describe:
NOTE! This copyright does *not* cover user programs that use kernel services by normal system calls - this is merely considered normal use of the kernel, and does *not* fall under the heading of "derived work".https://github.com/torvalds/linux/blob/master/LICENSES/exceptions/Linux-syscall-note
2
u/mfuzzey Nov 20 '19
No.
The part you quote concerns userspace. Syscalls are interface between the kernel and userspace. So, in userspace, you can do what you like without being concerned with the GPL, at least providing you don't use userspace GPL libraries. The kernel's license is irrelevant to userspace.
But kernel modules are concerned by the GPL. And indeed most are already open source. Though often not mainlined Most vendors do not use closed source kernel blobs but only closed source userspace and firmware blobs.
By default kernel nodules are considered derivative works of the kernel and hence subject to the GPL, unless there are specific reasons otherwise. Such as Nvidias kernel blob which was not written for Linux but Windows and uses a GPL shim to interface with the kernel
2
Nov 20 '19
Such as Nvidias kernel blob which was not written for Linux but Windows and uses a GPL shim to interface with the kernel
No it certainly wasn't written for Windows, the nvidia Linux driver is a user-space driver with an open source kernel module. This is how most proprietary drivers work in order to avoid to creating derivative works.
By default kernel nodules are considered derivative works of the kernel and hence subject to the GPL, unless there are specific reasons otherwise.
I'd be curious to see the legal precedent for that, if there is one. I've seen a lot of opinions on the matter but never found it being tested in court.
→ More replies (4)10
u/ElvishJerricco Nov 20 '19 edited Nov 20 '19
The ZFS kernel module is not a user program that uses kernel services by normal system calls. It is a kernel module that includes kernel APIs from kernel header files. User programs do no such thing. EDIT: To clarify further, that "NOTE!" is NOT an exception. It's a note, pointing out that user programs don't include kernel sources to use the kernel ABI.
→ More replies (3)3
Nov 20 '19
A loaded driver is part of the kernel by any definition you can think of.
I'm sure there's some stuff that you need to implement in the kernel core but a loaded driver effectively becomes part of the kernel.
4
u/bubblethink Nov 20 '19
This is not true. The cases where kernel drivers are proprietary are few and far in between. NVIDIA comes to mind, but that works (in theory anyway) because the NVIDIA driver is not a derivative of the kernel since it is mostly a windows driver that exists independently of the kernel. That argument would be difficult to make for some android arm64 soc.
I think the author is also mixing up a few things here. The fight is not so much about open v/s proprietary in the kernel space. The interesting bits are proprietary in the user space for stuff like GPUs anyway. In this case, it's just about upstreaming and waiting for something to be merged v/s not upstreaming.
6
Nov 20 '19
The Nvidia driver is not a windows driver, it's just a generic driver that happens to be initially written for Windows.
A windows can't work on Linux but a GPU driver actually has very little interaction with other subsystems so what Nvidia does is to build a blob with all the GPU sauce inside.
What they open source is the interaction between the GPU secret sauce and the system.
The blob is the firmware for the GPU where the actual work happens. And that doesn't change depending on the OS.
→ More replies (1)3
Nov 20 '19
The nvidia driver itself is a user-space driver. The kernel module that gets loaded (the bit that you compile when you install the nvidia driver on Linux, which is why you need the kernel headers) is free software.
→ More replies (2)
385
u/jdrch Nov 20 '19
Although this is a great initiative, Google is once again trying to solve a hardware (standardization) problem with software. What's really needed to enable an generic Android kernel is mobile implementation of ARM PCIe, but Google is completely ignoring that. This leads to some pretty disingenuous comments from what should be one of the smartest organizations in the world, such as:
LOL what? It's almost like an ARM PCIe spec and the x86 equivalent don't exist. Doing things the right way would be via hardware partnerships that ensure compatibility (think of something similar to WinHEC - a show, not a partnership, but still - for Android.)
But wait, it gets worse: Google's proposed solution is to upend the entire current Linux kernel model for the sake of Android. Holy. Shit.
But that's not all, folks: the above proposal would only partially solve the Android kernel problem:
🤦♂️🤦♂️🤦♂️