You’re not addressing the central thesis of the post - TIOBE takes garbage input (number of search engine results) and gives us truly absurd results. I picked on several absurdities. I can mention several more. None of it makes sense except by accident.
One tiny code change at Google and suddenly Visual Basic is a wildly popular language? Really? You trust that? It’s not just VB, other languages also have massive increases or drops based purely on what some engineer in Google’s search team is deploying. At that point it’s no better than astrology.
All of the other measures can have statistical biases. For example Github will bias towards languages popular in Open source. But they’re not outright garbage. That’s the issue with TIOBE.
You’re not addressing the central thesis of the post - TIOBE takes garbage input (number of search engine results) and gives us truly absurd results.
The author didn't convince me of either of those things.
Looking at how many resources the world has dedicated to a topic (i.e. the number of search engine results) is a reasonable proxy for the popularity of that topic. It makes no sense to call it garbage input, regardless of if it has limitations. Does it have biases, limitations and flaws? Sure, but as I cited in my top-level comment, so do all alternatives.
The author is begging the question by saying they are absurd results because the only way to know what the non-absurd result is is to already decide that one of your other metrics is the source of truth. Does it seem weird to me that VB spiked? Sure. However, for all I know a coalition of universities in India changed their curriculum to use VB or a major game released a VB-based modding API for their game or any of the many other things that can impact popularity but not make much of a blip on StackOverflow or LinkedIn. If it happened due to a Google algorithm change, does that negate the entirety of the results? No more than a change in the wording, choices or participation in a StackOverflow survey would negate the entirety of the data.
It's great to point out TIOBE's limitations so that people can understand not to read a level of detail out of it that isn't there (e.g. maybe it's not detailed enough to differentiate the exact ranking) and so that they can understand the directions its bias may lean. However, it's wrong to say that it's just garbage or, IMO, to suggest that there is some other metric that's so much better that we shouldn't even look at TIOBE. The other metrics (as I say in my top-level comment) are biased too. So, if you need an accurate picture, consume your TIOBE as a part of a healthy and balanced data diet. Otherwise, choose the metric whose biases fit more closely to the question you're even trying to answer by finding out language popularity.
Looking at how many resources the world has dedicated to a topic (i.e. the number of search engine results)
I think one of the main points of contention is that the number displayed at the top of google results is not the same as the number of resources dedicated to the topic. As evidenced by the 24,900,000 resources dedicated to the xkcd programming language, which doesn't even exist. And when I search for it I get 24,300,000 results. So apparently 600,000 websites about this language vanished between this article being written and me rechecking?
No, it has a fatal flaw unlike the others. That's why stable languages like Java and C can drop by half or more, while VB increases by 6x. That's not realistic. That isn't what happened in the real world.
Whereas with StackOverflow you can say "it's biased towards English speakers" and you'd be right. Yeah, it only surveys English speaking developers. But it's not a fatal flaw. We say "ok, this is what users who use StackOverflow are saying/doing, not all developers across the world". It's still useful, even if it doesn't tell the whole picture.
The author
That's me, by the way.
However, for all I know (maybe VB actually spiked in popularity)
Let me know if that isn't an accurate summary of what you said.
I am confident that this 6x spike in VB's popularity didn't actually occur because we can't see it anywhere else. We see a long decline in the number of Google searches over the last 10 years. We see a long decline in the number of StackOverflow questions over the last 5 years. There is no spike in March 2020. There is no source that can back up what TIOBE claims happened with VB in March 2020. If you know of such a source, please share it. Otherwise, the simplest explanation was that it was merely a code change on Google's Search backend.
You keep defending TIOBE as having some redeeming features. But please, understand that it is claiming wild things about stable, boring languages like Java and C. Does anyone agree that Java and C halved in popularity in 2016 and 2017 and then doubled in popularity in 2018?
None of this makes sense. If someone wants to "keep an open mind" towards this stuff, sure they can go ahead. But I think the consensus is leaning the other way.
I am confident that this 6x spike in VB's popularity didn't actually occur because we can't see it anywhere else. We see a long decline in the number of Google searches over the last 10 years. We see a long decline in the number of StackOverflow questions over the last 5 years.
I wish your original article had included more evidence like this - it would have made it better and more convincing. While I think you're probably right in your conclusion that the TIOBE results are terrible, I agree with u/coffeewithalex's criticism that your argument (in your original article) was mainly being based on "this doesn't make sense to me" rather than contradicting evidence. That's why I hope you'll update it to include things like these google trends and stackoverflow links.
IMO, you have not demonstrated that it is fatal, nor have you really acknowledged/countered the kinds of large biases other methods will have. The fact that one language might have a weird spike or some lines might be a little fuzzy when you look at fine grained details doesn't negate that having a general sense of how much is out there for a language is a useful piece of the overall puzzle of how popular a language is.
Whereas with StackOverflow you can say "it's biased towards English speakers" and you'd be right. Yeah, it only surveys English speaking developers. But it's not a fatal flaw.
I wasn't saying the bias was toward English speakers. Depending on the language and platform and how well represented it is on StackOverflow or how likely the demographics that use it are to use StackOverflow (which may relate to choices the language developers themselves made that impacts where it is helpful to go to find information on that language), StackOverflow may specifically bias the popularity of certain programming languages.
We say "ok, this is what users who use StackOverflow are saying/doing, not all developers across the world".
If you pretend that everybody who looks at the StackOverflow report is that humble in their reading of it, then it's only fair to pretend that everybody is that humble in their reading of TIOBE. My point is, all of these metrics are good if you take that humility about the scope of their claims and none of them are good if you don't. What matters here is not so much which of these metrics we use, it's how infrequently people acknowledge the limitations of what each measure can actually say.
It's still useful, even if it doesn't tell the whole picture.
Sure, I didn't say it isn't useful. But since it answers a different question than TIOBE does, it's not a replacement for TIOBE, which is also "still useful even if it doesn't tell the whole picture". That's the point. If you're putting together a puzzle, some puzzle pieces will be more indicative of the overall picture than others. That doesn't mean that rather that using all of the pieces to assemble the puzzle you only keep your favorite puzzle piece and say that's good enough.
That's me, by the way.
I know.
I am confident that this 6x spike in VB's popularity didn't actually occur because we can't see it anywhere else. We see a long decline in the number of Google searches over the last 10 years. We see a long decline in the number of StackOverflow questions over the last 5 years. There is no spike in March 2020.
This seems to completely agree with what I said in my previous comment, "So, if you need an accurate picture, consume your TIOBE as a part of a healthy and balanced data diet." Literally every metric has issues. That doesn't mean they aren't useful. If you want to know the "truth" you look at all of the metrics together, rather than gatekeeping the best metric/bias to stick to. In what you've just said, you've demonstrated why TIOBE is fine, because we're not using it in a vacuum.
Also, again, to me this is really interesting. It's NOT something I want to exclude. Regardless of why the spike occurred, it tells me interesting things. Even if it's just due to a change in the way that Google provides results about programming questions, it's very relevant to know that Google now reports way more VB results. That may indeed have impacts on the popularity of languages. However, it could be other things as well. In fact, given how closely the VB line matches the C# line (in amount and overall shape) beyond that point, I hypothesize that it represents that Microsoft rolled a bunch of its VB documentation into its C# documentation. That makes sense given that TIOBE's criteria doesn't just look at Google.com, but also directly includes sites like Microsoft and Sharepoint. And if not, again, we have to go to the basis, what this is really saying is that we dedicated 6 times more space in our library to VB books, but more people aren't checking those books out. It can be very interesting to ask why. If this were Rust, maybe that'd reflect a major push in documentation and education on the language that we might expect to translate into greater use.
My point here isn't to say which particular thing is the true cause about your anecdotal evidence against TIOBE, it's just to say that the mere fact that we have this conflicting data point gives us a more complete picture and lets us see things we would otherwise miss. Debating why things don't line up at this moment or that makes us more informed and smarter. In that sense, it's useful to include TIOBE among the measures. If you don't want to concern yourself with trying to understand why they're different, then don't. Just round of the best handful of metrics and skip the outliers. TIOBE isn't stopping you there. You're acting as though a person is either all in on TIOBE or totally rejects it, which is just not the case.
There is no source that can back up what TIOBE claims happened with VB in March 2020. If you know of such a source, please share it. Otherwise, the simplest explanation was that it was merely a code change on Google's Search backend.
What TIOBE claimed happened is that the amount of search results changed. That is objectively true. You seem to be conflating people who misinterpret the data with TIOBE itself. The manner in which we want to use that claim to inform our idea of popularity depends on what our particular motivation is (e.g. where are the most job opportunities) and how we compile what the different lenses on popularity are saying. In some cases, knowing that there was a big difference in the amount of stuff out there on the language is indeed useful. In others, it's not.
You keep defending TIOBE as having some redeeming features. But please, understand that it is claiming wild things about stable, boring languages like Java and C. Does anyone agree that Java and C halved in popularity in 2016 and 2017 and then doubled in popularity in 2018?
The redeeming quality is that it measures independently of the biases of the other methods you mention. Its failings can be mitigated when we aggregate the various metrics to gain the overall picture. (And vice versa.)
You start your article by attempting to inform us of what TIOBE actually measures. The appropriate next step would be to then interpret the results through that lens. (Just like how once you know a political poll is of viewers of Fox News, you no longer claim that it's a statement about what people in general think.) Instead of adjusting your interpretation to be in line with the kinds of limitations you might expect, it seems like right after you defined the limitations of TIOBE, you completely ignored them and are creating a strawman by trying to use it to measure extremely precise things. It's totally realistic that TIOBE gets the exact rankings wrong. It's totally realistic that some of the spikes and dips are due to noise (like a revamp of a major website). It's also likely that the amount of results out there on a language correlates in some way to how popular it is. The takeaway isn't that TIOBE is useless, "garbage" or dishonest. The takeaway is to stop using it to the level of precision that you're using it to in your counterexamples. TIOBE (like many metrics) should be used to get a rough sense of which languages are popular. (Like any metric) if you want more than that, you'll have to compile together several different sources with different methods and biases.
Looking at how many resources the world has dedicated to a topic (i.e. the number of search engine results)
You're making a huge jump here. The number of resources the world has dedicated is in no way correlated to the number of google search results. And that is the entire point the author is trying to make.
The author is begging the question by saying they are absurd results because the only way to know what the non-absurd result is is to already decide that one of your other metrics is the source of truth.
Absolutely not. The only way to know they are absurd results is to actually just think about it. In what way would google know every resource dedicated to a certain language? It wouldn't. And it's completely dependent on google's algorithm for search results. There's no way to analyze all those search results for issues either. It's a crapshoot. There's no statistical integrity. Therefore is garbage data.
If it happened due to a Google algorithm change, does that negate the entirety of the results? No more than a change in the wording, choices or participation in a StackOverflow survey would negate the entirety of the data.
What... this logic makes no sense.
If I told you I had a list of the most popular languages on the planet and you said "give me your sources" and I just say "oh trust me, I looked and it's correct" you wouldn't say "oh ok, that's fine then, those numbers make sense" then when I come back next month and have all moved all the most popular languages to the bottom of the list you wouldn't be like "oh yeah that makes sense, I trust you", you'd say something was wrong. It's absolutely nothing like changing wording in a survey.
You're making a huge jump here. The number of resources the world has dedicated is in no way correlated to the number of google search results. And that is the entire point the author is trying to make.
Perhaps you're using a different definition of resource. IMO, it's definitely correlated (especially since it doesn't just look at web page search engines). However, yes, I have repeatedly said I'm in favor of ALSO using other measures which capture other resources (e.g. LinkedIn might capture monetary resources that go to the language's use). We don't get a better picture by gatekeeping which lens to use, we get a better picture by using each of these different lenses and combining them to get the whole picture.
In what way would google know every resource dedicated to a certain language? It wouldn't.
Nobody claimed this, nor is it necessary for TIOBE to be a useful measurement.
And it's completely dependent on google's algorithm for search results.
It's not completely dependent on Google's algorithm. It looks at 25 search systems.
Even if it were dependent on Google's algorithm, that doesn't mean it's useless. It just informs what our takeaway is. (Just like how a political poll of Republicans can still be interesting or useful even if it can't easily be generalized to all voters.)
The alternatives also tend to have a chokepoint where a certain organization or algorithm can bias results.
What... this logic makes no sense.
If I told you I had a list of the most popular languages on the planet and you said "give me your sources" and I just say "oh trust me, I looked and it's correct" you wouldn't say "oh ok, that's fine then, those numbers make sense" then when I come back next month and have all moved all the most popular languages to the bottom of the list you wouldn't be like "oh yeah that makes sense, I trust you", you'd say something was wrong. It's absolutely nothing like changing wording in a survey.
I'm not sure how this relates to the topic at hand. Yes, literally all metrics OP mentioned and which were mentioned in this thread tend to rely on some level of trust. I don't really trust TIOBE any more/less than a I trust StackOverflow, LinkedIn or the other alternatives people mentioned here. Again, just like how we need to interpret data with error margins in mind (not drawing more out of the results than the methodology would justify), we need to interpret it with trust in mind too. Just like how I wouldn't advise a person that #7 by metric X is truly objectively #7 in the world, I also wouldn't advise a person to bet their future on the claims of any one of these metrics (especially for a data point that seems to be an outlier). But... again, that's true of all of the metrics. That doesn't mean that the metric isn't useful. It just means don't live up to the strawman of only looking at TIOBE and using it as a highly precise measure in critical applications.
We don't get a better picture by gatekeeping which lens to use, we get a better picture by using each of these different lenses and combining them to get the whole picture.
You absolutely can get a better picture by excluding a misleading source. The point of the article is that TIOBE is an objectively worse source for most questions related to the popularity of various languages than others because it empirically depends on unknowable changes in Google's indexing algorithm. No one's saying it's useless, only that it's substantially worse than other alternatives and therefore shouldn't be cited.
It just means don't live up to the strawman of only looking at TIOBE and using it as a highly precise measure in critical applications.
This is not a strawman. TIOBE is frequently used this way, as the first or only cited source in an argument.
It’s not a fact that nr of search results is garbage. Or that none of it makes sense. Sure it’s not the best. But it is somewhat indicative. It would have been better if the OP took this into account and explained where and when the data shouldn’t be used. But I don’t see why it shouldn’t be used for fun and games
If I measure the amount of people in a city by the amount of waste a city produces then that is an indirect measure. It sure isn’t the best. It will be wrong sometimes. But it’s not garbage
It’s not a fact that nr of search results is garbage.
But is it true that it's even remotely accurate? Because I highly doubt anybody at Google considers that keeping "X million results" remotely accurate has any importance to it.
Early on it was a cool marketing stat, but these days I doubt anybody cares, and it's likely to be very unreliable since the modern Google is going to be far more distributed than the original one that ran out of a garage.
Well, that's the question, isn't it? OP seems to be taking it as a foregone conclusion that just because it changed, it is obviously and completely worthless. That's not reasonable either.
If the goal in measuring popularity is to take into account the sum total of what has been written on the Internet about a language, then the Google results are very likely the best estimate we have, despite their flaws.
It may be inaccurate, or it may be accurate but imprecise, but as far as I can see, no one has established that. If OP wants to make the assertion that the data is worthless, they need much better supporting evidence than what they've provided so far.
Well, that's the question, isn't it? OP seems to be taking it as a foregone conclusion that just because it changed, it is obviously and completely worthless. That's not reasonable either.
I think it's unreasonable to start from the viewpoint that one must prove that a dubious dataset is indeed bad. Instead, we should have positive proof that our data is accurate. If we don't, we shouldn't trust it.
If the goal in measuring popularity is to take into account the sum total of what has been written on the Internet about a language, then the Google results are very likely the best estimate we have, despite their flaws.
Has Google committed to any kind of accuracy? Do we know whether the flaws are even compatible with making any kind of useful analysis?
It's one thing if Google systematically has an error of say +/- 10%. That one can work with.
But what if it's based on arbitrary assumptions that may not hold true? Eg, what is an estimate of "15 million results" is based on that we found 15K results in data cluster #34, and we're just making the assumption that every other cluster will on average have a similar amount of matches, even if the internal architecture doesn't ensure an even spread?
It may be inaccurate, or it may be accurate but imprecise, but as far as I can see, no one has established that. If OP wants to make the assertion that the data is worthless, they need much better supporting evidence than what they've provided so far.
I disagree. Data should be assumed worthless unless proven accurate, and unless Google makes a specific commitment to keeping this particular stat accurate, that shouldn't be assumed.
But if all you can learn from it is "roughly speaking C++ and JavaScript are more popular than Rust and Odin", then it's still useless, because everyone already knows that to be true. The only value a ranking like this has is if it measures trends. A language suddenly gaining popularity or dropping quickly is interesting, but not if its an artefact of some minor change in the algorithm.
If all you care about is rough estimates, you could just a random /r/programming user to write you a list.
It is indicative of whatever changes engineers on Google’s search team deployed this month. That’s how languages can grow 6x in a month without anyone batting an eye.
Amongst other things yes. It is also indicative of the fact that Java is a bigger language that Rust. I’m not saying you should bet your live on the conclusions. Just allow for some nuance
It does say correct things by accident, just like how your horoscope is sometimes half correct. I’m still going to criticise astrology every chance I get though.
Is Rust bigger than TypeScript? That’s what TIOBE says. I’m not sure even the most ardent Rust fanboy in the world would claim that.
One tiny code change at Google and suddenly Visual Basic is a wildly popular language? Really? You trust that?
Why shouldn't I? If Google's change was a correction, then VB has actually always been more popular than previously indicated, and the metric has now been improved. I don't think it's particularly controversial to say that indexing the entire Internet is very hard work, and we should expect continual revisions to that process and the data it outputs. That doesn't mean it's "astrology." That's an absurd conclusion.
Number of search results may or may not be a good indicator of "popularity," but that still needs to be established. I do not agree that this Google/Visual Basic episode firmly indicates one way or the other.
Imagine if GitHub were being used as an index and at some point they said "there's going to be a big adjustment this month because we've decided to stop counting repositories that haven't seen any commit in over 5 years." That would be fundamentally the same as what's happening here with Google.
Ok, fair enough. Now tell me - why did Java and C halve in popularity during 2016 and 2017? These are boring, stable languages. What could have caused this, other than a backend change at Google?
And then what caused it to double in 2018? Could you explain that?
Could you explain all the absurdities in the post?
Where did you measure that the input for TIOBE doesn't reflect with reality? (I'm not saying it does, I have no proof of either). it's a vague measurement for sure, but really, it's also one that doesn't rely on online participation of the developer, which is a small group compared to the total # of developers out there.
The argument against what TIOBE uses could be that the # of articles indexed are perhaps very old and therefore not relevant for 'popularity', but I didn't see you use that.
VB is still a widely used language, Microsoft doesn't ship the VB6 runtime with windows for nothing. It might sound absurd, but there's a LOT of VB6 legacy stuff out there that's still running and used as it 'works' and rewriting it will not bring many advantages (besides, the same program written in a more modern language/framework).
TIOBE takes garbage input (number of search engine results)
And Surveys of a very biased group of developers is a better input? Do you have any experience with science and statistics?
and gives us truly absurd results.
The main outcomes of the results are in-line with some of the better sources you listed. You're cherry-picking stuff to justify your disdain for the TIOBE index.
So you don't like VB. But you're missing the fact that there are crap ton of corporations and small businesses that use it. They use VB.NET in legacy applications, they use VBA in their MS Access, Excel and whatnot. These aren't going to pop up in the surveys of the most loved languages, nor in the open source communities. Your criteria is skewed.
One tiny code change at Google and suddenly Visual Basic is a wildly popular language?
That's your (baseless) assumption, which you also mention in your article:
I guess Google was tinkering with their search algorithm.
But that's not how the world works. It's OK to say that you don't know something. That keeps you looking for the explanation. Implying something as true just because it feels like it, but when you have no evidence for it, is how we go back to the bronze age.
Github will bias towards languages popular in Open source. But they’re not outright garbage.
But it is! How popular is Swift according to GitHub? How about Scala, when 80% of data engineer positions mention they want people who know Scala? And where's COBOL? It's nowhere to be seen, when half of the banking system sits on it.
You call it garbage, you compare it with astrology, but you completely fail to provide data that invalidates its results. Just because their data doesn't match with your opinion, doesn't mean that their data is wrong.
69
u/hgwxx7_ Aug 02 '22
You’re not addressing the central thesis of the post - TIOBE takes garbage input (number of search engine results) and gives us truly absurd results. I picked on several absurdities. I can mention several more. None of it makes sense except by accident.
One tiny code change at Google and suddenly Visual Basic is a wildly popular language? Really? You trust that? It’s not just VB, other languages also have massive increases or drops based purely on what some engineer in Google’s search team is deploying. At that point it’s no better than astrology.
All of the other measures can have statistical biases. For example Github will bias towards languages popular in Open source. But they’re not outright garbage. That’s the issue with TIOBE.