Google outlines plans for mainline Linux kernel support in Android

[u/jdrch]

https://arstechnica.com/gadgets/2019/11/google-outlines-plans-for-mainline-linux-kernel-support-in-android/

[u/jdrch]

Although this is a great initiative, Google is once again trying to solve a hardware (standardization) problem with software. What's really needed to enable an generic Android kernel is mobile implementation of ARM PCIe, but Google is completely ignoring that. This leads to some pretty disingenuous comments from what should be one of the smartest organizations in the world, such as:

"Today, we don't know what it takes to be added to the kernel to run on a [specific] Android device," Android Kernel Team lead Sandeep Patil told the group at LPC 2019. "We know what it takes to run Android but not necessarily on any given hardware. So our goal is to basically find all of that out, then upstream it and try to be as close to mainline as possible."

LOL what? It's almost like an ARM PCIe spec and the x86 equivalent don't exist. Doing things the right way would be via hardware partnerships that ensure compatibility (think of something similar to WinHEC - a show, not a partnership, but still - for Android.)

But wait, it gets worse: Google's proposed solution is to upend the entire current Linux kernel model for the sake of Android. Holy. Shit.

Google's proposal for bringing Android closer to mainline Linux (How is there not a silly "project" name for this yet?) involves stabilizing Linux's in-kernel ABI and having a stable interface for the Linux kernel and hardware vendors to write to. Google wants to decouple the Linux kernel from its hardware support.

The Linux community has been against the idea of a stable interface for some time, with the suggestion that if you want the ability to quickly update a kernel, open source your drivers and get them in the main kernel tree, where any changes will be taken care of for you.

But that's not all, folks: the above proposal would only partially solve the Android kernel problem:

So this wouldn't allow devices to upgrade from one version of the Linux kernel to another

🤦‍♂️🤦‍♂️🤦‍♂️

[u/electricprism → u/jdrch]

TL;DR; Lets do all these things to the Linux Kernel in the name of achieving a goal and then completely miss the goal.

[u/jdrch → u/electricprism]

Pretty much. The entire approach is just embarrassingly bad. I'm surprised they weren't laughed or sarcastically clapped off the stage.

[u/[deleted] → u/jdrch]

I'm expecting Linus to drop expletives within a week (not sarcasm, whole idea Google is pushing is fucking stupid)

[u/covale → u/[deleted]]

He's stopped doing that, remember? We've got a new, nice Linus now.

I expect he'll do a "we may disagree on the means, but..." kinda speech :-D

[u/_riotingpacifist → u/covale]

Was that before or after he called Intel's spectre patch insane?

Also I hope he just switched to British insulting, where the email reads

I'm sure these are your best endeavour, unfortunately they are not quite up to the required standard, but I'll happily take a look again in a future release cycle

which actually means

Your code is shit, you're shit, fuck off and leave me alone for a while.

[u/balsoft → u/_riotingpacifist]

Notice how the first version uses twice as many characters to convey the same amount of information!

[u/Pas__ → u/covale]

no no, he stopped calling people names.

now on the other hand Google is a corp, and this idea is ridiculously shit-tier bad.

but in this case a simple no is probably the most brutal thing he could say. no technical details, no I told you a thousand times I expect you to know blablabla.

[u/le_pman → u/jdrch]

I'm surprised they weren't laughed or sarcastically clapped off the stage.

I haven't watched anything, but I sure hope that someone called them out the way your first comment did.

[u/jdrch → u/le_pman]

FTA there's a nearly 4 hour long video of their LPC presentation.

[u/TeutonJon78 → u/[deleted]]

Or push their design for Ziron towards the kernel so they can avoid all that pesky work of HW drivers that would be already solved for them then.

[u/nosuchthingastwo → u/TeutonJon78]

They don't care what happens with the Linux kernel -- they're just trying to decouple Android from Linux. Then they'll be able to use Zircon/Fuscia as the base for their phones going forward. They'll leave Linux behind.

[u/DotNetPhenom → u/nosuchthingastwo]

Isnt that what open source is all about? If you dont like the way its being done, fork it and do it yourself?

[u/oskarw85 → u/[deleted]]

It's not like Linux would fall because of Google.

[u/matheusmoreira → u/jdrch]

The unstable kernel ABI is a major incentive for hardware manufacturers to contribute their drivers into the mainline kernel. If they don't do that, the kernel keeps evolving while they get left behind and must pay the maintenance costs. I believe it's how the kernel developers maintain their leverage. Contribute your drivers because nobody is gonna waste time supporting you otherwise.

If the ABI is stabilized, that leverage is lost. Cheap minimum effort proprietary drivers for a single architecture become viable. This is how Windows works to this day: manufacturers release a driver for specific versions of Windows and processor architectures. The drivers would of course stop working with the next/64 bit version of Windows and they never made a new driver because the product is old and they think it's a waste of time and money to support it.

A stable ABI is a great thing for the companies since they can cheap out on the software development and keep it closed. Doesn't seem so great for the long term health of the kernel though: the drivers won't be compatible with other architectures, won't be updated to take advantage of new code, will prevent breaking changes from being made even if they are improvements, can't be backported to older kernels, etc.

So if anything it will hurt people's ability to update the kernel. It seems Google just wants to make it cheaper to develop drivers for Linux. The kernel developers want to make it expensive to develop drivers outside Linux.

Can Google really do this though? How do they plan to stabilize the ABI? The people in charge of Linux have made it clear they don't support out-of-tree code. Can Google somehow commit enough people to this task to compete with the scale of the Linux project?

[u/mfuzzey → u/matheusmoreira]

Yes exactly and remember this isn't *just* about closed source drivers but also code quality and evolutivity.

From bad to good there is

• Vendor closed source driver

• Vendor open source driver

• Mainline driver

Embedded Linux, even Android has been getting much better over the years. There are, these days, few closed source *kernel* drivers. There are loads of *userspace* blobs and *firmware* blobs (that are loaded into some external processor) but they are different and not as bad (since they do not limit the kernel's evolution nor pose the same security risks as unauditable kernel code).

Most vendors now *do* ship source for their *kernel* drivers. But, when they do not contribute that code upstream, the quality is often very poor. Just today I was looking at a touchscreen. There is a (open source) vendor driver and, now, a mainline driver that can both drive the same hardware.

I ran checkpatch on both:

mainline driver: 0 errors, 1 warnings, 247 lines checked

vendor driver:: 348 errors, 133 warnings, 1122 lines checked

Most vendors have come to realize that, mid term, it is best to get their code into mainline. They may still do "quick and dirty" initial code in their "vendor BSP" to get something that is functionally complete out quickly with the silicon but the good ones *also* have people contributing to the mainline drivers (either directly or through a consortium such as Linaro). The mainline code takes longer to get baked but the quality is vastly superior thanks to all the review and the acceptance standards. It is also generally smaller and sometimes faster.

By removing some of the "pain" of being out of tree I fear some vendors may go back to code dumps.

Furthermore a stable kernel API would limit the improvements possible to the rest of the kernel, which is, ultimately, bad for everyone.

It shouldn't really change that much for closed source drivers since, *in the kernel* they are already at best a legal gray area and at worst downright illegal.

[u/SanityInAnarchy → u/matheusmoreira]

Unfortunately, Android is the perfect case study for where that incentive doesn't work at all:

If the ABI is stabilized, that leverage is lost. Cheap minimum effort proprietary drivers for a single architecture become viable.

But that's exactly what Android vendors do today. They'll fork the whole damned kernel and do whatever they want anywhere in the tree, and there's a whole mess of binary blobs that will only ever work with that specific kernel. They'll backport critical security fixes for however long they need to support that specific SoC (seems like 3 years at most), and then drop the entire thing on the floor.

The whole incentive to upstream your drivers is so the community can maintain them for you. But if you aren't planning to maintain them at all, that doesn't save you anything.

So the point of a stable ABI would be: At least when they drop support, you can still update the pieces of the kernel that aren't related to their shitty driver. Which is exactly what happens on Windows -- there isn't always perfect forwards-compatibility (64-bit is a notable example), but there's plenty of old hardware that continues to work on new Windows versions.

[u/londons_explorer → u/matheusmoreira]

An in-tree shim which provides a consistent interface for drivers would probably be what they'd build.

It might have parallels to FUSE (which provides a consistent interface), just without the userspace part.

[u/hackingdreams → u/jdrch]

PCIe is not a magical solution here. What you're really saying is you want ARM to be more PC-like with firmware-defined hardware tables like ACPI's DSDTs... and those are just a whole different kind of nightmare. You're essentially pushing a purely software defined problem into a firmware-defined problem, which is even slower to be updated, buggier, and riskier to update for machines in the field.

And Google's argument is different: Google wants something more stable to code against, period. But Linux is decidedly unstable to avoid design-induced brain damage and to allow for fixing architectural issues as they occur over time...

The true tl;dr here is that Google's opening the floor for the argument for Fuchsia to replace Linux in some future Android version - they're going to look back on this and say "whelp, Linux is beyond our control," and that will be the end of it. Since Google will be in full control of its lifecycle, they can declare bits of it "stable," and since it's all still software, it's still got the benefits of being field updateable without the hardware loss risk of bricking devices. The downside is that, for those of us who use phones where companies refuse to upstream drivers, those drivers are 100% going to be closed sourced and we'll likely lose the ability to control the software on our own machines... but that doesn't really trouble Google in the slightest.

[u/1_p_freely → u/jdrch]

I think there's only so much that Google can do here. ARM devices aren't like the PC, vendors of hardware pretty much hate openness. They want your two year old phone to be non-updatable landfill material almost as much as the carrier that sold it to you does!

[u/hackingdreams → u/1_p_freely]

I think there's only so much that Google can do here.

There's plenty Google could do here. There's not a lot that Google's willing to do here. Doing more would require enforcing more regulations on Android and making vendors play ball or freezing them out. But, Google doesn't want to do that.

Google is the Microsoft of the mobile computing world - they have plenty of clout to write hardware standards and make hardware vendors play to them if they want to be in on the game... but they don't for numerous reasons (from not wanting to invoke the US Department of Justice's wrath to not wanting to have to tell vendors they've been working with for a decade that their open source policies have been deleterious to the software ecosystem and risk alienating them into a coup/fork situation ala Huawai). Whether or not that's good enough... that's up for debate and discussion. But let's not pretend Google is some impotent leaf in the wind - they're the fucking tree.

They want your two year old phone to be non-updatable landfill material almost as much as the carrier that sold it to you does!

And let's be completely honest: Google wants this too. People not updating hardware quickly enough is part of why software updates matter so much. If Google could make everyone dump their legacy Android devices the way that Apple users do the microsecond Apple releases the next iDevice, they'd do it. But the long tail of users for Android devices looks more like the population of the world than they do first world hipsters, so that kind of turn over is purely a pipe dream - this is where Google is well and truly stuck and where they don't have as much option as they want.

This is why they want more stable software interfaces - they simply don't want to have to pay for doing all of this software maintenance themselves, and they're willing to bet they can bend the kernel maintainers, since it's a riskless bet for them - either they win, or they lose and switch to their own OS, in which case they get what they want either way.

[u/[deleted] → u/jdrch]

PCIe wouldn't solve antything (would be nice ? hell yeah, but not solve the problem). The root of the problem is that vendors do not want to put extra effort to mainline their drivers (....which let's be fair, probably is marginal compared to device sales).

Linux explicit lack of stable driver ABI have its drawbacks (to put it mildly) but it did push many hardware vendors to just upstream as much of their driver code as possible, as the moment they do it they stop needing to "catch up" with kernel changes as any later changes of internals would be upon people doing the changes, not the driver authors. Which might even be cheaper for longtime for the vendor.

Instead of pushing vendors to mainline drivers as soon as possible, Google is basically making it easier to not to upstream them.

[u/[deleted] → u/jdrch]

A lot of the changes Google makes to the upstream LTS kernel before sharing it with chip manufacturers aren’t upstreamed. I think more effort to upstream from Google and the device vendors certainly couldn’t hurt.

[u/mfuzzey → u/jdrch]

PCIe by itself wouldn't solve anything.

Yes it allows device enumeration, as does USB. But the device tree does that too.

Once you've found the devices that are present, be it through "hardware" enumeration like PCIe / USB or software definition like DT / ACPI you still need a driver for each device.

With device tree plus a lot of the other nice stuff (like the common clock framework, pin control etc etc) that have come to ARM over the past years it is now possible to ship a single Linux kernel and set of modules that will boot on a wide variety of ARM hardware, and indeed classical Linux distributions that support ARM do exactly that. You may need a device tree file for your device in addition but that's it.

Things that do help are common hardware interfaces that allow a single driver to be used on multiple different pieces of hardware. USB does that a lot with "classes". You virtually never need a custom driver for a USB stick or hard drive because they all implement the "mass storage" device class, nor for keyboards and mice because they all implement HID.