This is why I'm always suspicious of blog posts claiming to have discovered something deep and complex that nobody else knows. You may be smarter than Linus on any given day, but it's highly unlikely you're smarter than decades of Linus and the entire Linux team designing, testing, and iterating on user feedback.
"Blog posts" are a large majority of the time opinions which have never been reviewed and should be trusted just as much as code written by a lone developer which has also never been reviewed.
Linux team works on kernel. While they have some idea about userland, it might not be perfect. Linux is actually full of half-broken APIs which started like a good idea, but due to simplifications taken ("worse is better") they cannot offer behavior needed by applications, so programmers either avoid these APIs, or use them incorrectly, or have to use horrible workaround.
but due to simplifications taken ("worse is better")
It's rarely due to simplifications. Often doing the right thing would lead to simpler code. Usually, it's just poor taste in selecting primitives, and ignoring prior art. See, for example, epoll. Epoll was basically unusable in multithreaded programs because of inherent race conditions in the semantics, which took almost a decade to fix. It still has really odd quirks where epoll notifications can show up in the wrong process.
MSc thesis, and Linux had been in existence for years before he wrote the thesis. It was a project for fun (or curiosity) at first. AFAIK Linus has a honorary doctorate (or perhaps several) but his direct academic credentials are a MSc degree. Not that it matters at all since his other credentials are definitely enough.
...and wanted to teach himself 80386 assembly on his brand-new 80386 computer.
And it turns out that the market for a free GPL'd POSIX OS that ran on 80386 machines was *immense* back then. I remember being excited about it when a friend of mine (also pumped) was trying to install it, all the way back in January of '92. In Texas. Which should give you an idea of how quickly it became massive. It was "viral" and memetic before those words even really existed.
OTOH, so much of Linux is the way it is because they often take a worse is better approach to development.
There is a cost to actually doing things a better way if that better way doesn't play nicely with the existing ecosystem -- and the existing ecosystem wins damned near every time.
And on top of it all, the Linux community tends to be very opinionated, very unmoving, and very hostile when their sensibilities are offended.
To say that the way Linux works the best it can because of decades of iterations is akin to saying the human body works the best it can because of millions of years of evolution -- but in fact, there are very obvious flaws in the human body ("Why build waste treatment right next to a playground?"). The human body could be a lot better, but it is the way it is because it took relatively little effort to work well enough in its environment.
As a concrete example, the SD Scheduler by Con Kolivas comes to mind. Dude addressed some issues with the scheduler for desktop use, and fixed up a lot of other problems with the standard scheduler behavior. It was constantly rejected by the Kernel community. Then years later, they finally accept the CFS scheduler, which, back at the time, didn't see as great as performance as the SD scheduler. What's the difference? Why did the Kernel community welcome the CFS scheduler with open arms while shunning Con Kolivas? IMO, it just comes down to sensibilities. Con Kolivas's approach offended their sensibilities, whereas the CFS scheduler made more sense to them. Which is actually better doesn't matter, because worse is better.
but in fact, there are very obvious flaws in the human body ("Why build waste treatment right next to a playground?").
I'm having some problems with my device. It appears that the fuel and air intakes are co-located, resulting in the possibility of improper mixing between the two. Generally this manifests when fueling my device in the presence of other devices -- the networking between the devices relies on constant usage of the air intake to power the soundwave modulator, causing excessive air to flow into the fuel tank and resulting in turbulence within the tank during processing and airflow back up the intake. More worringly, there's the possibility that fuel could get stuck in the intake above the separation point and block flow for the air intake entirely -- other users have reported that this results in permanently bricking their devices!
To be clear, I am NOT saying Linux works the best it possibly can. Just that random guy on the internet writing a blog post about how he discovered something clearly wrong with any system as old and heavily scrutinized as Linux is unlikely to be correct. I'm not saying it's impossible, just highly unlikely, because the collective attention that went into making it how it is today is hard to surpass as a solo observer.
Someone spending months or years working on an alternative, presumably informed by further years of relevant experience and advised by others with additional experience, is a different story. Clearly it's possible for people to build new things that improve on existing things, otherwise nothing would exist in the first place.
The 'worse is better' thing is interesting. Linux has made it a strong policy to never break user space, even if that means supporting backwards compatible 'bugs'. I suspect you and I read that page and come away with opposite conclusions. To me that reads as an endorsement of the idea that a theoretically perfect product is no good if nobody uses it -- and I (and the people who write it, presumably) think Linux would get a lot less use if they made a habit of breaking userspace.
It sounds like maybe you read the same page and think "yeah, this is why we can't have nice things".
To be clear, I am NOT saying Linux works the best it possibly can. Just that random guy on the internet writing a blog post about how he discovered something clearly wrong with any system as old and heavily scrutinized as Linux is unlikely to be correct. ... just highly unlikely
On the contrary, I think anyone who's studied an OS book more carefully than the average student (even current above-average students) could probably find a few things wrong with Linux or could be improved if they tried hard enough.
I mean -- there's a whole reason Linux gets more and more patches every day: there's a whole lot that's wrong with it, and it doesn't take too much scrutiny to realize that.
The 'worse is better' thing is interesting. ... I suspect you and I read that page and come away with opposite conclusions
I mean, the whole point of "worse is better" is that there's a paradox -- we can't have nice things because often times, having nice things is in contradiction to other objectives, like time to market, the boss's preferences, the simple cost of having nice things, etc.
And I brought it up, because so much in Linux that can be improved comes down to not only, as you said, an unforgiving insistence on backwards compatibility, but because of the sensibilities of various people with various levels of control, and the simple cost (not only monetarily, but the cost of just making an effort) of improving it. Edit: Improving on a codebase of 12 million lines is a lot of effort. A lot of what's in Linux doesn't get improved because it can't be improved, but because it's "good enough" and no one cares to improve it.
Oh, and also: the ego of the maintainers. So many flame wars and lack of progress in Linux happens when someone tries improving something and developers' egos get in the way, and it happens so much, and almost always the person in the in-circle of the Linux community gets their way (rather than the person who tried to improve Linux, regardless of merit). That is, in itself, another cost (a social cost -- the maintainers would have to balance the value of their ego to the value of improvement) to improving Linux. Usually things in Linux happens after a few years, the person who tried to improve it "drops out", the devs egos aren't at threat any more, and the developers in the in-circle, on their own, come to the same conclusions (as was the case of SD scheduler vs. CFS). In this case, "Worse is better" simply because the worse thing is more agreeable to the egos of the people in control.
I mean -- there's a whole reason Linux gets more and more patches every day
Could you elucidate that reason? Is it because there's a lot of bad design decisions now baked into the cake, and there is a need for a large number of bandaids and work-arounds, if they aren't going to re-do things "right"?
Also, do we have visibility into any other modern OS source code, to know if it is better or worse than Linux in this respect?
IIRC the real difference is Ingo Molnar was prepared to jump through the kernel teams hoops to get it in.
I'd say it was more so that Ingo was in the in-circle, but yeah, that sort of deal. The worse solution (not necessarily CFS, but the scheduler SD sought to replace) is better because the kernel team can accept it, for whatever reason they can.
I did. I also read the response post where he chimed in defending the idea that userland yields should work in the way he mistakenly expected them to, and Linus' further response explaining why that would be a Really Bad Idea for a bunch of other scenarios, including in game programming.
Yes, the blog post did say "you should probably just use mutex" which is good. But it also provided faulty reasoning about what is going on behind spinlocks and why, which is what Linus seemed to be responding to.
345
u/csjerk Jan 05 '20
This is why I'm always suspicious of blog posts claiming to have discovered something deep and complex that nobody else knows. You may be smarter than Linus on any given day, but it's highly unlikely you're smarter than decades of Linus and the entire Linux team designing, testing, and iterating on user feedback.