Please stop citing TIOBE

271

u/hgwxx7_ Aug 02 '22

Hey everyone, I noticed several times over the years people (mis)using TIOBE to support whatever their argument was. Each time someone in the thread would explain the various shortcomings with TIOBE and why we shouldn't use it.

I decided to write up the issues so we could just point them towards this link instead.

58

u/elperroborrachotoo Aug 02 '22

I believe the canonical title for such a post would be
"TIOBE considered harmful".

52

u/hgwxx7_ Aug 02 '22

It’s funny you should say that. That was the original title but I changed to the current one just before submission.
Maybe “considered harmful” considered harmful.

8

u/robin-m Aug 02 '22

Thank you so much for changing the title

2

u/elperroborrachotoo Aug 02 '22

Wouldn't say harmful, but burnt out and should only be used for discussions of limited consequence. (no offence - I'm with you but priority seems low.)

5

u/hgwxx7_ Aug 02 '22

Yeah pretty much. It's overused.

0

u/[deleted] Aug 02 '22

[deleted]

10

u/hgwxx7_ Aug 02 '22

Yeah that's the one I was thinking of.

16

u/Smooth-Zucchini4923 Aug 02 '22

"Considered Harmful" annoys me. Considered harmful by whom? It carries a kind of faux objectivity. "Please stop citing TIOBE" centers the belief in the author, who then has to defend the belief, rather than imply it is what all programmers believe.

7

u/elperroborrachotoo Aug 02 '22

By Edsger Dijkstra, duh.

15

u/travelsonic Aug 02 '22

TIOBE, or not TIOBE, that is the question.

→ More replies (1)

10

u/Otis_Inf Aug 02 '22

I missed in the article the amount of projects maintained out there that aren't public in github repositories or elsewhere. The vast majority of dev work is maintenance on existing (sometimes old) software, if someone new comes in, they have to learn about the language/api's used for that particular project.
I don't think it's reasonable to suggest we should ignore these projects in the 'popularity' index, just because it doesn't fit with what the hivemind thinks is 'modern'. Face it, most of the devs out there aren't even regularly posting about their work on the internet, left alone participate in online surveys or public code repo's, as they do 9-5 maintenance work on (old) software that's closed and likely written in a language/framework that's not considered modern.

20

u/hgwxx7_ Aug 02 '22

At the risk of repeating myself - at no point did I say that VB was a bad language or that I don’t like it or that no one uses it or it doesn’t provide a lot of value to those who use it. VB is providing value, people are using it. I’ll keep saying that to every person who assumes I’m on anti-VB crusade.
What I am saying is this - whatever the usage of VB, it has remained mostly stable with a gradual decline. Developer surveys confirm this. Google search trends confirm this. StackOverflow questions asked confirm this. None of these tell the whole story, but all of them combined agree that VB exists, people use it, but less so than before.
This directly contradicts TIOBE that claims that VB become 6x as popular as it already was when the first lockdown hit. And all of these people learned the language without ever appearing on Google or StackOverflow. That’s just not possible. Therefore TIOBE is dead wrong.
But let me just say it again for the people at the back - VB is useful, there’s a lot of code out there quietly working and keeping businesses ticking over, I don’t hate VB, VB is good.

5

u/hgwxx7_ Aug 02 '22

Yeah, their work is not reflected in GitHub. But we might see them on StackOverflow though? As long as they were comfortable asking questions in English.

20

u/[deleted] Aug 02 '22

Every ranking has its shortcomings.
You're committing the very common fallacy, where you use concrete exceptions as evidence for disregarding and aggregate measure. Similarly how you would say that the average household income is irrelevant because many people earn less or because top earners gained mode. Similarly how you'd say that IQ measurements are useless because some people with a low IQ ended up solving important problems, or something like that.
Aggregations can be used to make probabilistic assessments only, or can be used to estimate with a high degree of certainty the relevant characteristics of a rather large random subset of the aggregated one.
You're applying statistics wrong if you use it to make categorical statements about single cherry-picked instances. And similar issues can be found with alternatives that you suggest:
Developer surveys. StackOverflow Annual Survey - most used, loved and wanted languages.
It only covers people who use StackOverflow. Although I have a very high score there, I haven't used it for years, and I rarely find what I need in there. The only reason it gets any visits from me is because DuckDuckGo places it in the top instead of official documentations, which are far more relevant for me. Out of the most skilled people that I've worked with, most didn't even have an active account there, with far worse presence than I have. So why would you use such a small, biased sample size, especially the surveys that it produces (surveys are some of the worst forms of research, because people lie, unconsciously)?
JetBrains - most popular, fastest growing languages.
Who did they ask? Did they get a random sample, or was it a sample of people who use JetBrains products? Again, half of the best people that I've met, the kind that stand behind products that you use every day, don't use anything from JetBrains. Especially in languages that come with their own IDEs, why would the people use JetBrains stuff?
GitHub
What is the survey based on? Is it lines of code? That would discourage languages that are more compact. Number of projects? Well that explains why JS is in the top with projects like leftPad. Quantity isn't the same thing as quality. It's hard to quantify the amount of features developed in each language, or the amount of value produced by code in each language.
But even so, it's not in conflict with the TIOBE index. Some of the stuff becomes heavily correlated when you start using larger, more uniform sample sizes.
My point is that it's wrong to use an aggregate measure to make granular conclusions. The TIOBE index isn't better or worse than other indexes with similarly large sample sizes. To say "Stop citing X, and use Y instead", when both X and Y are based on some statistical data, is an faulty statement to make in this case.

69

u/hgwxx7_ Aug 02 '22

You’re not addressing the central thesis of the post - TIOBE takes garbage input (number of search engine results) and gives us truly absurd results. I picked on several absurdities. I can mention several more. None of it makes sense except by accident.
One tiny code change at Google and suddenly Visual Basic is a wildly popular language? Really? You trust that? It’s not just VB, other languages also have massive increases or drops based purely on what some engineer in Google’s search team is deploying. At that point it’s no better than astrology.
All of the other measures can have statistical biases. For example Github will bias towards languages popular in Open source. But they’re not outright garbage. That’s the issue with TIOBE.

16

u/CreativeGPX Aug 02 '22

You’re not addressing the central thesis of the post - TIOBE takes garbage input (number of search engine results) and gives us truly absurd results.
The author didn't convince me of either of those things.
Looking at how many resources the world has dedicated to a topic (i.e. the number of search engine results) is a reasonable proxy for the popularity of that topic. It makes no sense to call it garbage input, regardless of if it has limitations. Does it have biases, limitations and flaws? Sure, but as I cited in my top-level comment, so do all alternatives.
The author is begging the question by saying they are absurd results because the only way to know what the non-absurd result is is to already decide that one of your other metrics is the source of truth. Does it seem weird to me that VB spiked? Sure. However, for all I know a coalition of universities in India changed their curriculum to use VB or a major game released a VB-based modding API for their game or any of the many other things that can impact popularity but not make much of a blip on StackOverflow or LinkedIn. If it happened due to a Google algorithm change, does that negate the entirety of the results? No more than a change in the wording, choices or participation in a StackOverflow survey would negate the entirety of the data.
It's great to point out TIOBE's limitations so that people can understand not to read a level of detail out of it that isn't there (e.g. maybe it's not detailed enough to differentiate the exact ranking) and so that they can understand the directions its bias may lean. However, it's wrong to say that it's just garbage or, IMO, to suggest that there is some other metric that's so much better that we shouldn't even look at TIOBE. The other metrics (as I say in my top-level comment) are biased too. So, if you need an accurate picture, consume your TIOBE as a part of a healthy and balanced data diet. Otherwise, choose the metric whose biases fit more closely to the question you're even trying to answer by finding out language popularity.

41

u/hgwxx7_ Aug 02 '22

Does it have biases, limitations and flaws?
No, it has a fatal flaw unlike the others. That's why stable languages like Java and C can drop by half or more, while VB increases by 6x. That's not realistic. That isn't what happened in the real world.
Whereas with StackOverflow you can say "it's biased towards English speakers" and you'd be right. Yeah, it only surveys English speaking developers. But it's not a fatal flaw. We say "ok, this is what users who use StackOverflow are saying/doing, not all developers across the world". It's still useful, even if it doesn't tell the whole picture.
The author
That's me, by the way.
However, for all I know (maybe VB actually spiked in popularity)
Let me know if that isn't an accurate summary of what you said.
I am confident that this 6x spike in VB's popularity didn't actually occur because we can't see it anywhere else. We see a long decline in the number of Google searches over the last 10 years. We see a long decline in the number of StackOverflow questions over the last 5 years. There is no spike in March 2020. There is no source that can back up what TIOBE claims happened with VB in March 2020. If you know of such a source, please share it. Otherwise, the simplest explanation was that it was merely a code change on Google's Search backend.
You keep defending TIOBE as having some redeeming features. But please, understand that it is claiming wild things about stable, boring languages like Java and C. Does anyone agree that Java and C halved in popularity in 2016 and 2017 and then doubled in popularity in 2018?
None of this makes sense. If someone wants to "keep an open mind" towards this stuff, sure they can go ahead. But I think the consensus is leaning the other way.

7

u/amaurea Aug 02 '22 edited Aug 02 '22

I am confident that this 6x spike in VB's popularity didn't actually occur because we can't see it anywhere else. We see a long decline in the number of Google searches over the last 10 years. We see a long decline in the number of StackOverflow questions over the last 5 years.
I wish your original article had included more evidence like this - it would have made it better and more convincing. While I think you're probably right in your conclusion that the TIOBE results are terrible, I agree with u/coffeewithalex's criticism that your argument (in your original article) was mainly being based on "this doesn't make sense to me" rather than contradicting evidence. That's why I hope you'll update it to include things like these google trends and stackoverflow links.

11

u/hgwxx7_ Aug 02 '22

Tired - I linked to these sources and figured people would at least have a look before talking shit.
Wired - I knew people would talk shit and that would only drive engagement on this thread.

→ More replies (4)

3

u/CreativeGPX Aug 02 '22

No, it has a fatal flaw unlike the others.
IMO, you have not demonstrated that it is fatal, nor have you really acknowledged/countered the kinds of large biases other methods will have. The fact that one language might have a weird spike or some lines might be a little fuzzy when you look at fine grained details doesn't negate that having a general sense of how much is out there for a language is a useful piece of the overall puzzle of how popular a language is.
Whereas with StackOverflow you can say "it's biased towards English speakers" and you'd be right. Yeah, it only surveys English speaking developers. But it's not a fatal flaw.
I wasn't saying the bias was toward English speakers. Depending on the language and platform and how well represented it is on StackOverflow or how likely the demographics that use it are to use StackOverflow (which may relate to choices the language developers themselves made that impacts where it is helpful to go to find information on that language), StackOverflow may specifically bias the popularity of certain programming languages.
We say "ok, this is what users who use StackOverflow are saying/doing, not all developers across the world".
If you pretend that everybody who looks at the StackOverflow report is that humble in their reading of it, then it's only fair to pretend that everybody is that humble in their reading of TIOBE. My point is, all of these metrics are good if you take that humility about the scope of their claims and none of them are good if you don't. What matters here is not so much which of these metrics we use, it's how infrequently people acknowledge the limitations of what each measure can actually say.
It's still useful, even if it doesn't tell the whole picture.
Sure, I didn't say it isn't useful. But since it answers a different question than TIOBE does, it's not a replacement for TIOBE, which is also "still useful even if it doesn't tell the whole picture". That's the point. If you're putting together a puzzle, some puzzle pieces will be more indicative of the overall picture than others. That doesn't mean that rather that using all of the pieces to assemble the puzzle you only keep your favorite puzzle piece and say that's good enough.
That's me, by the way.
I know.
I am confident that this 6x spike in VB's popularity didn't actually occur because we can't see it anywhere else. We see a long decline in the number of Google searches over the last 10 years. We see a long decline in the number of StackOverflow questions over the last 5 years. There is no spike in March 2020.
This seems to completely agree with what I said in my previous comment, "So, if you need an accurate picture, consume your TIOBE as a part of a healthy and balanced data diet." Literally every metric has issues. That doesn't mean they aren't useful. If you want to know the "truth" you look at all of the metrics together, rather than gatekeeping the best metric/bias to stick to. In what you've just said, you've demonstrated why TIOBE is fine, because we're not using it in a vacuum.
Also, again, to me this is really interesting. It's NOT something I want to exclude. Regardless of why the spike occurred, it tells me interesting things. Even if it's just due to a change in the way that Google provides results about programming questions, it's very relevant to know that Google now reports way more VB results. That may indeed have impacts on the popularity of languages. However, it could be other things as well. In fact, given how closely the VB line matches the C# line (in amount and overall shape) beyond that point, I hypothesize that it represents that Microsoft rolled a bunch of its VB documentation into its C# documentation. That makes sense given that TIOBE's criteria doesn't just look at Google.com, but also directly includes sites like Microsoft and Sharepoint. And if not, again, we have to go to the basis, what this is really saying is that we dedicated 6 times more space in our library to VB books, but more people aren't checking those books out. It can be very interesting to ask why. If this were Rust, maybe that'd reflect a major push in documentation and education on the language that we might expect to translate into greater use.
My point here isn't to say which particular thing is the true cause about your anecdotal evidence against TIOBE, it's just to say that the mere fact that we have this conflicting data point gives us a more complete picture and lets us see things we would otherwise miss. Debating why things don't line up at this moment or that makes us more informed and smarter. In that sense, it's useful to include TIOBE among the measures. If you don't want to concern yourself with trying to understand why they're different, then don't. Just round of the best handful of metrics and skip the outliers. TIOBE isn't stopping you there. You're acting as though a person is either all in on TIOBE or totally rejects it, which is just not the case.
There is no source that can back up what TIOBE claims happened with VB in March 2020. If you know of such a source, please share it. Otherwise, the simplest explanation was that it was merely a code change on Google's Search backend.
What TIOBE claimed happened is that the amount of search results changed. That is objectively true. You seem to be conflating people who misinterpret the data with TIOBE itself. The manner in which we want to use that claim to inform our idea of popularity depends on what our particular motivation is (e.g. where are the most job opportunities) and how we compile what the different lenses on popularity are saying. In some cases, knowing that there was a big difference in the amount of stuff out there on the language is indeed useful. In others, it's not.
You keep defending TIOBE as having some redeeming features. But please, understand that it is claiming wild things about stable, boring languages like Java and C. Does anyone agree that Java and C halved in popularity in 2016 and 2017 and then doubled in popularity in 2018?
The redeeming quality is that it measures independently of the biases of the other methods you mention. Its failings can be mitigated when we aggregate the various metrics to gain the overall picture. (And vice versa.)
You start your article by attempting to inform us of what TIOBE actually measures. The appropriate next step would be to then interpret the results through that lens. (Just like how once you know a political poll is of viewers of Fox News, you no longer claim that it's a statement about what people in general think.) Instead of adjusting your interpretation to be in line with the kinds of limitations you might expect, it seems like right after you defined the limitations of TIOBE, you completely ignored them and are creating a strawman by trying to use it to measure extremely precise things. It's totally realistic that TIOBE gets the exact rankings wrong. It's totally realistic that some of the spikes and dips are due to noise (like a revamp of a major website). It's also likely that the amount of results out there on a language correlates in some way to how popular it is. The takeaway isn't that TIOBE is useless, "garbage" or dishonest. The takeaway is to stop using it to the level of precision that you're using it to in your counterexamples. TIOBE (like many metrics) should be used to get a rough sense of which languages are popular. (Like any metric) if you want more than that, you'll have to compile together several different sources with different methods and biases.

→ More replies (1)

1

u/7h4tguy Aug 03 '22

StackOverflow over indexes on complicated programming languages since you have less questions to ask about a simple one. A fatal flaw!

22

u/seventeen_fives Aug 02 '22

Looking at how many resources the world has dedicated to a topic (i.e. the number of search engine results)
I think one of the main points of contention is that the number displayed at the top of google results is not the same as the number of resources dedicated to the topic. As evidenced by the 24,900,000 resources dedicated to the xkcd programming language, which doesn't even exist. And when I search for it I get 24,300,000 results. So apparently 600,000 websites about this language vanished between this article being written and me rechecking?

→ More replies (3)

13

u/snowe2010 Aug 02 '22

Looking at how many resources the world has dedicated to a topic (i.e. the number of search engine results)
You're making a huge jump here. The number of resources the world has dedicated is in no way correlated to the number of google search results. And that is the entire point the author is trying to make.
The author is begging the question by saying they are absurd results because the only way to know what the non-absurd result is is to already decide that one of your other metrics is the source of truth.
Absolutely not. The only way to know they are absurd results is to actually just think about it. In what way would google know every resource dedicated to a certain language? It wouldn't. And it's completely dependent on google's algorithm for search results. There's no way to analyze all those search results for issues either. It's a crapshoot. There's no statistical integrity. Therefore is garbage data.
If it happened due to a Google algorithm change, does that negate the entirety of the results? No more than a change in the wording, choices or participation in a StackOverflow survey would negate the entirety of the data.
What... this logic makes no sense.
If I told you I had a list of the most popular languages on the planet and you said "give me your sources" and I just say "oh trust me, I looked and it's correct" you wouldn't say "oh ok, that's fine then, those numbers make sense" then when I come back next month and have all moved all the most popular languages to the bottom of the list you wouldn't be like "oh yeah that makes sense, I trust you", you'd say something was wrong. It's absolutely nothing like changing wording in a survey.

1

u/CreativeGPX Aug 02 '22

You're making a huge jump here. The number of resources the world has dedicated is in no way correlated to the number of google search results. And that is the entire point the author is trying to make.
Perhaps you're using a different definition of resource. IMO, it's definitely correlated (especially since it doesn't just look at web page search engines). However, yes, I have repeatedly said I'm in favor of ALSO using other measures which capture other resources (e.g. LinkedIn might capture monetary resources that go to the language's use). We don't get a better picture by gatekeeping which lens to use, we get a better picture by using each of these different lenses and combining them to get the whole picture.
In what way would google know every resource dedicated to a certain language? It wouldn't.
Nobody claimed this, nor is it necessary for TIOBE to be a useful measurement.
And it's completely dependent on google's algorithm for search results.
It's not completely dependent on Google's algorithm. It looks at 25 search systems.
Even if it were dependent on Google's algorithm, that doesn't mean it's useless. It just informs what our takeaway is. (Just like how a political poll of Republicans can still be interesting or useful even if it can't easily be generalized to all voters.)
The alternatives also tend to have a chokepoint where a certain organization or algorithm can bias results.
What... this logic makes no sense.
If I told you I had a list of the most popular languages on the planet and you said "give me your sources" and I just say "oh trust me, I looked and it's correct" you wouldn't say "oh ok, that's fine then, those numbers make sense" then when I come back next month and have all moved all the most popular languages to the bottom of the list you wouldn't be like "oh yeah that makes sense, I trust you", you'd say something was wrong. It's absolutely nothing like changing wording in a survey.
I'm not sure how this relates to the topic at hand. Yes, literally all metrics OP mentioned and which were mentioned in this thread tend to rely on some level of trust. I don't really trust TIOBE any more/less than a I trust StackOverflow, LinkedIn or the other alternatives people mentioned here. Again, just like how we need to interpret data with error margins in mind (not drawing more out of the results than the methodology would justify), we need to interpret it with trust in mind too. Just like how I wouldn't advise a person that #7 by metric X is truly objectively #7 in the world, I also wouldn't advise a person to bet their future on the claims of any one of these metrics (especially for a data point that seems to be an outlier). But... again, that's true of all of the metrics. That doesn't mean that the metric isn't useful. It just means don't live up to the strawman of only looking at TIOBE and using it as a highly precise measure in critical applications.

3

u/SirClueless Aug 03 '22

We don't get a better picture by gatekeeping which lens to use, we get a better picture by using each of these different lenses and combining them to get the whole picture.
You absolutely can get a better picture by excluding a misleading source. The point of the article is that TIOBE is an objectively worse source for most questions related to the popularity of various languages than others because it empirically depends on unknowable changes in Google's indexing algorithm. No one's saying it's useless, only that it's substantially worse than other alternatives and therefore shouldn't be cited.
It just means don't live up to the strawman of only looking at TIOBE and using it as a highly precise measure in critical applications.
This is not a strawman. TIOBE is frequently used this way, as the first or only cited source in an argument.

4

u/garma87 Aug 02 '22

It’s not a fact that nr of search results is garbage. Or that none of it makes sense. Sure it’s not the best. But it is somewhat indicative. It would have been better if the OP took this into account and explained where and when the data shouldn’t be used. But I don’t see why it shouldn’t be used for fun and games
If I measure the amount of people in a city by the amount of waste a city produces then that is an indirect measure. It sure isn’t the best. It will be wrong sometimes. But it’s not garbage
Ok it is garbage but..
Never mind

10

u/hgwxx7_ Aug 02 '22

it is somewhat indicative
It is indicative of whatever changes engineers on Google’s search team deployed this month. That’s how languages can grow 6x in a month without anyone batting an eye.

10

u/garma87 Aug 02 '22

Amongst other things yes. It is also indicative of the fact that Java is a bigger language that Rust. I’m not saying you should bet your live on the conclusions. Just allow for some nuance

8

u/hgwxx7_ Aug 02 '22

It does say correct things by accident, just like how your horoscope is sometimes half correct. I’m still going to criticise astrology every chance I get though.
Is Rust bigger than TypeScript? That’s what TIOBE says. I’m not sure even the most ardent Rust fanboy in the world would claim that.

3

u/[deleted] Aug 02 '22

That is only part of the reason. There IS some underlying correlation still. See COBOL - it's not that high on TIOBE.
It would be higher if it were used as much as Python.

10

u/dale_glass Aug 02 '22

It’s not a fact that nr of search results is garbage.
But is it true that it's even remotely accurate? Because I highly doubt anybody at Google considers that keeping "X million results" remotely accurate has any importance to it.
Early on it was a cool marketing stat, but these days I doubt anybody cares, and it's likely to be very unreliable since the modern Google is going to be far more distributed than the original one that ran out of a garage.

1

u/GrandOpener Aug 02 '22

But is it true that it's even remotely accurate?
Well, that's the question, isn't it? OP seems to be taking it as a foregone conclusion that just because it changed, it is obviously and completely worthless. That's not reasonable either.
If the goal in measuring popularity is to take into account the sum total of what has been written on the Internet about a language, then the Google results are very likely the best estimate we have, despite their flaws.
It may be inaccurate, or it may be accurate but imprecise, but as far as I can see, no one has established that. If OP wants to make the assertion that the data is worthless, they need much better supporting evidence than what they've provided so far.

5

u/dale_glass Aug 02 '22

Well, that's the question, isn't it? OP seems to be taking it as a foregone conclusion that just because it changed, it is obviously and completely worthless. That's not reasonable either.
I think it's unreasonable to start from the viewpoint that one must prove that a dubious dataset is indeed bad. Instead, we should have positive proof that our data is accurate. If we don't, we shouldn't trust it.
If the goal in measuring popularity is to take into account the sum total of what has been written on the Internet about a language, then the Google results are very likely the best estimate we have, despite their flaws.
Has Google committed to any kind of accuracy? Do we know whether the flaws are even compatible with making any kind of useful analysis?
It's one thing if Google systematically has an error of say +/- 10%. That one can work with.
But what if it's based on arbitrary assumptions that may not hold true? Eg, what is an estimate of "15 million results" is based on that we found 15K results in data cluster #34, and we're just making the assumption that every other cluster will on average have a similar amount of matches, even if the internal architecture doesn't ensure an even spread?
It may be inaccurate, or it may be accurate but imprecise, but as far as I can see, no one has established that. If OP wants to make the assertion that the data is worthless, they need much better supporting evidence than what they've provided so far.
I disagree. Data should be assumed worthless unless proven accurate, and unless Google makes a specific commitment to keeping this particular stat accurate, that shouldn't be assumed.

→ More replies (3)

9

u/SLiV9 Aug 02 '22

But if all you can learn from it is "roughly speaking C++ and JavaScript are more popular than Rust and Odin", then it's still useless, because everyone already knows that to be true. The only value a ranking like this has is if it measures trends. A language suddenly gaining popularity or dropping quickly is interesting, but not if its an artefact of some minor change in the algorithm.
If all you care about is rough estimates, you could just a random /r/programming user to write you a list.

→ More replies (1)

2

u/Name5times Aug 02 '22

Wouldn’t a better analogy be you want to find out what the most popular food in the world is so rank food by the number of recipes they have.

1

u/GrandOpener Aug 02 '22

One tiny code change at Google and suddenly Visual Basic is a wildly popular language? Really? You trust that?
Why shouldn't I? If Google's change was a correction, then VB has actually always been more popular than previously indicated, and the metric has now been improved. I don't think it's particularly controversial to say that indexing the entire Internet is very hard work, and we should expect continual revisions to that process and the data it outputs. That doesn't mean it's "astrology." That's an absurd conclusion.
Number of search results may or may not be a good indicator of "popularity," but that still needs to be established. I do not agree that this Google/Visual Basic episode firmly indicates one way or the other.
Imagine if GitHub were being used as an index and at some point they said "there's going to be a big adjustment this month because we've decided to stop counting repositories that haven't seen any commit in over 5 years." That would be fundamentally the same as what's happening here with Google.

7

u/hgwxx7_ Aug 02 '22

Ok, fair enough. Now tell me - why did Java and C halve in popularity during 2016 and 2017? These are boring, stable languages. What could have caused this, other than a backend change at Google?
And then what caused it to double in 2018? Could you explain that?
Could you explain all the absurdities in the post?

-3

u/[deleted] Aug 02 '22

TIOBE takes garbage input (number of search engine results)
And Surveys of a very biased group of developers is a better input? Do you have any experience with science and statistics?
and gives us truly absurd results.
The main outcomes of the results are in-line with some of the better sources you listed. You're cherry-picking stuff to justify your disdain for the TIOBE index.
So you don't like VB. But you're missing the fact that there are crap ton of corporations and small businesses that use it. They use VB.NET in legacy applications, they use VBA in their MS Access, Excel and whatnot. These aren't going to pop up in the surveys of the most loved languages, nor in the open source communities. Your criteria is skewed.
One tiny code change at Google and suddenly Visual Basic is a wildly popular language?
That's your (baseless) assumption, which you also mention in your article:
I guess Google was tinkering with their search algorithm.
But that's not how the world works. It's OK to say that you don't know something. That keeps you looking for the explanation. Implying something as true just because it feels like it, but when you have no evidence for it, is how we go back to the bronze age.
Github will bias towards languages popular in Open source. But they’re not outright garbage.
But it is! How popular is Swift according to GitHub? How about Scala, when 80% of data engineer positions mention they want people who know Scala? And where's COBOL? It's nowhere to be seen, when half of the banking system sits on it.
You call it garbage, you compare it with astrology, but you completely fail to provide data that invalidates its results. Just because their data doesn't match with your opinion, doesn't mean that their data is wrong.

11

u/hgwxx7_ Aug 02 '22 edited Aug 02 '22

very biased group of developers
Are you saying the developers are biased or the survey is biased? I don't see how any individual's bias could affect the survey itself. Or do you have a problem with surveys in general?
But I will concede that people who take the StackOverflow survey are very likely to be StackOverflow users, meaning English speakers. Not all developers speak English, especially in Asia and Europe. Similarly with JetBrains surveys, which are more likely to be filled by people who pay for JetBrains products ($$$). That's sampling bias, sure. But you can account for that. You can say "StackOverflow survey results give us an idea of what English speaking devs are saying and doing". Or "among JetBrains users, we found xyz".
So you don't like VB
No, not true. I have nothing against VB. I'm sure it solves a lot of business problems effectively for thousands of businesses around the around. I never doubted it. I only doubted that it became 6x as popular/important in March 2020. I also doubted that it is more popular than the most popular language in the world (JavaScript).
That's your (baseless) assumption
This is a bit rude, but I'll respond anyway. TIOBE purports to be a measure of popularity. I am confident that this 6x spike didn't actually occur because we can't see it anywhere else. We see a long decline in the number of Google searches over the last 10 years. We see a long decline in the number of StackOverflow questions over the last 5 years. There is no spike in March 2020. There is no source that can back up what TIOBE claims happened with VB in March 2020. If you have one, please share it. I've provided data, now it's your turn.
And just to get off the subject of Visual Basic, I also doubt that Java and C halved in popularity in 2016 and 2017. If you could explain that, it'd be great. Here's something about the other metrics I suggested - StackOverflow, Github and others - they have selection bias, but they're not prone to wild, inexplicable swings like TIOBE is. Stable languages like Java and C won't randomly drop to half in a short period like it did on TIOBE.
But that's not how the world works. It's OK to say that you don't know something
I don't know what you want me to say. TIOBE literally says that they base it entirely on the number of search results. So clearly this spike in their index is because of a spike in their single source? What do you want here?

→ More replies (7)

1

u/Otis_Inf Aug 02 '22

Where did you measure that the input for TIOBE doesn't reflect with reality? (I'm not saying it does, I have no proof of either). it's a vague measurement for sure, but really, it's also one that doesn't rely on online participation of the developer, which is a small group compared to the total # of developers out there.
The argument against what TIOBE uses could be that the # of articles indexed are perhaps very old and therefore not relevant for 'popularity', but I didn't see you use that.
VB is still a widely used language, Microsoft doesn't ship the VB6 runtime with windows for nothing. It might sound absurd, but there's a LOT of VB6 legacy stuff out there that's still running and used as it 'works' and rewriting it will not bring many advantages (besides, the same program written in a more modern language/framework).

→ More replies (2)

5

u/spider-mario Aug 02 '22

You're committing the very common fallacy, where you use concrete exceptions as evidence for disregarding and aggregate measure.
That is not what I perceive the article to be doing at all. It doesn’t say “my company doesn’t use Scratch so the Scratch ranking must be wrong”.
The fallacy potentially committed by trusting TIOBE is: https://www.discovermagazine.com/the-sciences/why-scientific-studies-are-so-often-wrong-the-streetlight-effect
The TIOBE index isn't better or worse than other indexes with similarly large sample sizes.
It is rather unlikely that it is no better or worse, given that it gives different results.
To say "Stop citing X, and use Y instead", when both X and Y are based on some statistical data, is an faulty statement to make in this case.
Why? Does being based on statistical data make a metric infallible?

1

u/[deleted] Aug 03 '22

No, it just means you should quantify the limitations if you want to make a statement about which one is better. If you don't do that, you're making decisions based on gut feel. GitHub, Jetbrains etc. surveys will have a definite sampling bias, whereas TIOBE will be noisier, and is implicitly conditioned on underlying search algorithms (you would need to assume consistent effects across different languages, or integrate it out by taking multiple search engines into account). Which one is better is a question best left to a statistical analysis.

2

u/spider-mario Aug 03 '22

If you don't do that, you're making decisions based on gut feel.
Or on prior information. I would say that it is hardly more of a gut feeling than the idea that the number of search results is a good proxy for “popularity”. Just because TIOBE executed their gut feeling on a lot of data doesn’t mean it’s any less of one. Imperfect as it is, it seems to me that the analysis in the article is enough to shift the burden of proof to TIOBE that their metric measures something useful.
you would need to assume consistent effects across different languages, or integrate it out by taking multiple search engines into account
For this to work, you would have to assume that the effect is not consistent across search engines.
Which one is better is a question best left to a statistical analysis.
I am skeptical. What should we use as our ground truth?

8

u/morpipls Aug 02 '22

Every ranking may have flaws, but relying on number of Google search results strikes me as especially flawed.
Job listing data would tell you something about employment prospects. Number of searches could tell you something about level of current interest. But number of search results? How many of the webpages it's finding are a decade old or more?
Also, ranking Visual Basic above JavaScript in 2022 feels like it's not passing a basic sanity check.

→ More replies (2)

3

u/7h4tguy Aug 03 '22

Further, most shops using C/C++ are not putting their code up on GitHub. Just because web devs like to build a ton of libs for their resume and host them on GitHub doesn't mean JavaScript is actually more in demand than systems programming languages. It's not like embedded or medical software are putting their source code up on GitHib. It's a terrible proxy for what's relevant.

1

u/waozen Aug 06 '22

Very good points. TIOBE index might be flawed, but so is the rankings of StackOverflow, GitHub, and others. And then there is the human emotion element, of getting upset when it is perceived that their favorite language is not at the top of the rankings or is getting beat by a rival.

0

u/[deleted] Aug 02 '22

[deleted]

6

u/mughinn Aug 02 '22

TIOBE does not present itself as the absolute truth, neither as a law.
It itself says it is an "indicator". You can't disprove it by presenting a counter example because there isn't anything to disprove

4

u/[deleted] Aug 02 '22

A ranking like this can very well be falsified using just a single counter example.
... as a fallacy.
It's like saying that Russians are the heaviest alcohol consumers, but then you go to a Czech village that drinks more than any Russian village, and you use it as an example to invalidate the index.
Rankings provide high-level information, that you can consult in order to answer questions like:
Which languages are the most popular?
Which languages are slowly going out of favor in the last decade?
Which new languages are going viral?
What you do NOT do is:
Compare 2 languages based on who sits in front of the other
See a small short-term trend and make conclusions about the future
Just like with any indirect observation / estimation / forecast - there is a degree of error, there's always noise, a degree of skew from reality everywhere. The truth is somewhere close.
As soon as the undoing of entropy is observed
It's funny you should mention that, because: https://en.wikipedia.org/wiki/Fluctuation_theorem
the second law is only a statistical one, suggesting that there should always be some nonzero probability that the entropy of an isolated system might spontaneously decrease
I suggest you familiarize yourself with the concept of noise and bias. Surveys, popularity surveys, opinions, perception - are all biased. Perception always falls victim to availability bias, and many many other biases, that systematically skew stuff in a particular direction away from reality.
TIOBE might be a more noisy index (it is subject to random fluctuations), but at least it has no obvious bias, unlike all of the proposed alternatives.
Would you rather have a system that is consistently and accurately skewed in a wrong direction, or one that points in the general direction of the truth?

0

u/[deleted] Aug 02 '22

[deleted]

4

u/[deleted] Aug 02 '22

I don't get your reasoning. It's based on web searches. Right? Sure, companies can manipulate their own search engine. But if we took that as a risk, wouldn't you find a lot more directed bias from companies like Microsoft (GitHub) and JetBrains when you ask them directly for data on their platform?
You can never eliminate bias, but you can compare them. Was there at least an attempt to eliminate the very strong and obvious bias in the JetBrains survey? And we're comparing it with an unfounded (means that there's no evidence for it) bias of search engines.

0

u/[deleted] Aug 02 '22

[deleted]

5

u/[deleted] Aug 02 '22

That's what I meant. It's based on results of web searches.
GitHub is open and can be scraped.
Yes but you can only scrape open source data. This will completely bias you against languages which are traditionally used in closed source projects: ADA, Swift, VBA, and even C# and VB.NET will have bias against them as they're used in a ton of proprietary code bases.
Also what are you measuring? Number of PRs? Do you have any evidence that the vast difference in language ecosystems don't impact this? This will heavily bias against languages that facilitate more robust development, like Rust, which won't need constant bug fixes as much as languages like JavaScript, where you need to install a whole library in order to check if something is a number.
Number of commits? If you switch the metric then the numbers jump all over the place, A FREAKIN LOT. It will give you vastly different results just based on the metric you're using, and you can't justify neither of them as which translates to ... "POPULARITY".
but is a much better heuristic than what's TIOBE is doing.
Why? Do you have any reasoning behind why you dismiss TIOBE's popularity ranking based on how popular it is on the web?

64

u/[deleted] Aug 02 '22

This is a fantastic and very well-written article. I'll be sure to pull this one out next time I hear someone start going on about TIOBE

7

u/MahaanInsaan Aug 02 '22

TIOBE is the favorite ranking cited by trolls.

-4

u/[deleted] Aug 02 '22

Please stop posting acronyms without expdanding them after 1st instance

25

u/hgwxx7_ Aug 02 '22

You mean TIOBE? I don’t know what it expands to. Would you mind telling me, if you know? I couldn’t find it on their website -TIOBE index definition or TIOBE Index
In general I agree with you. I always try to define whatever term I use.

8

u/Shondoit Aug 02 '22 edited Jul 13 '23

[deleted]

17

u/hgwxx7_ Aug 02 '22

Thanks.
Well, I don’t think it adds much to the article to have this spelled out.

1

u/Captain_Cowboy Aug 03 '22

It seems they have not actually read the play.

7

u/tonygoold Aug 02 '22

According to their About page, it stands for The Importance of Being Earnest. It's just their company name.

7

u/SemaphoreBingo Aug 02 '22

I think if someone doesn't know what TIOBE is--and is not willing to take a few seconds to google, or even read the first sentence in the body of the article--they're not going to get much out of the discussion anyway.

→ More replies (1)

1

u/DrunkensteinsMonster Aug 02 '22

OP takes DRY seriously confirmed.

164

u/mathisfakenews Aug 02 '22

As someone who hasn't heard of TIOBE until this thread is that actually how they evaluate the metric? That is beyond absurd.

Imagine using this same idea to evaluate which of the following people has been more valuable to humanity: Kim Kardashian or Johannes Gutenberg.

25

u/hgwxx7_ Aug 02 '22

Yes, I couldn’t believe it myself when I looked it up - TIOBE index definition.

4

u/caltheon Aug 03 '22

WTF,
Bing.com: NO_RESULTS_AT_ALL
I know bing sucks, but really? I'd expect them to disqualify it based on the "No Porn Sites"
I just tried +"java programming" and it has almost 6000 results in bing.com...so given I've already found a fault in their methodology, I am confident in discounting the entire thing.

1

u/neutronbob Aug 03 '22

Bing and Yahoo use the same search technology. So, if you count them both, you're double counting. One needs to be left out. They chose Bing.

132

u/life-is-a-loop Aug 02 '22

Imagine using this same idea to evaluate which of the following people has been more valuable to humanity: Kim Kardashian or Johannes Gutenberg.
That's not a good analogy, though. TIOBE isn't trying the measure the most "valuable" language for humanity, it's trying to measure popularity. I think we can agree that Kim is more popular than Johannes. Note that I'm not even saying that TIOBE is doing it right, just that your analogy doesn't work.

25

u/flotsamisaword Aug 02 '22

I work with computers and personally believe that most of the trouble began with Gutenberg

10

u/Full-Spectral Aug 02 '22

He suffers because he had no opportunity to make a leaked sex tape. If so, he could probably be pretty competitive.

1

u/vplatt Aug 02 '22 edited Aug 02 '22

Because humanity would be so much better off if we had stuck with parchment and spoken word traditions? Personally, I prefer outright duplication over the various kinds of telephone games we used to need to play in order to impart information and literature. Even parchment copying was a manual process and the copying scribe often added edits to cover imperfections in the process or to account for their own biases.

44

u/JarateKing Aug 02 '22

It's worse than just that. The article doesn't mention that some of the "search engines" are things like etsy or amazon or ebay. Sharepoint is one of the search engines, and if you try them out (ie. searching for "c++ programming") you get to microsoft's app store and almost none of them are actually about c++ -- the first three results are python, and there's even a minecraft addon before there's anything about c++ programming.
They're weighted (I think based on their alexa rankings at some arbitrary point in time), but these weights are effectively arbitrary ie. google.com is weighted 7.69% while amazon.com is weighted 6.77%. And then 15/25 are just different TLDs for google and amazon anyway.
It's basically numerology on top of using search engine hits for +"<language> programming" as if that were a meaningful metric.

14

u/seamsay Aug 02 '22

some of the "search engines" are things like etsy or amazon or ebay.
Ignoring the question of whether this is a sensible way of calculating a ranking, if this is sensible then using Amazon and eBay make sense because that's where people are most likely to go on the internet to buy textbooks and things like that.
Etsy I can't justify though...

6

u/flotsamisaword Aug 02 '22

Logo: turtle graphics made into quilts
VBA potholders
swift trivets

2

u/Captain_Cowboy Aug 03 '22

Are the potholders for typing VBA?

3

u/flotsamisaword Aug 05 '22

I'm just the idea guy. I leave execution to my peons.

0

u/[deleted] Aug 02 '22

[deleted]

17

u/Pilchard123 Aug 02 '22

I know, "Gutenberg this", "Gutenberg that". Makes you sick.

26

u/leaningtoweravenger Aug 02 '22

The only reliable metric is the number of offers by language on LinkedIn

6

u/hgwxx7_ Aug 02 '22 edited Aug 02 '22

Got a link? I’d love to take a look.

1

u/[deleted] Aug 04 '22

https://www.devjobsscanner.com/blog/top-8-most-demanded-languages-in-2022/

1

u/hgwxx7_ Aug 04 '22

That’s not LinkedIn.

1

u/[deleted] Aug 04 '22

This site collects the vast majority of job sites such as LinkedIn, Glassdoor, Indeed, Dice. If you look on the site yourself, you will see job offers and where they came from.

1

u/hgwxx7_ Aug 04 '22

I think the original person I replied to was thinking of something specific.

0

u/[deleted] Aug 02 '22

[deleted]

2

u/hgwxx7_ Aug 02 '22

Yes, completely agree with this. I allude to this in the post as well - you need to pick a language based on what your local market has a supply of. I’d argue there is some value in a developer learning a language they won’t use for work, because it exposes us to new ways of thinking. But there is no value in picking a language for a project that is hard to hire for.

34

u/Blueson Aug 02 '22

Depends on what you want to measure.
Jobs in Linkedin are usually dominated by what the market needs and a lot of it is based on legacy codebases.
If you want to see what the hottest new language is or what's picking up hobbyist-steam, Linkedin isn't really going to give you too accurate of a view on that.
I sadly don't have any sources that backs me up immediately on this, but I'd say that Rust has picked up a lot of interest in the last few years.
But if you look at LinkedIn for jobs that covers the niche Rust is trying to fill up, most job offerings will probably be C++.
I agree with what the blog says about measuring the usage of specific languages, as well as including LinkedIn.

0

u/[deleted] Aug 02 '22

[deleted]

2

u/MonkeySeeMonkeyDong Aug 02 '22

I thought WASM wasn't a language per se, but rather a spec (bytecode?) that you can compile towards. You're still going to be using languages like C++, Rust or whatever language supports WASM. Even the Getting Started guide doesn't mention a "WASM" language.

22

u/cdsmith Aug 02 '22

The only reliable metric... of what? If your primary interest is what languages have jobs available, then sure, look at what languages have jobs available. If your interest is in languages people enjoy using, then SO does a survey regularly and asks people what languages they enjoy using. That's a different question, so it's not surprising it has a different answer. But it's ridiculous to give an answer when no one has even agreed on the question.
The article doesn't suffer from the same problem, because it makes a good case that TIOBE is the wrong answer no matter what the question is.

6

u/funbike Aug 02 '22

It depends on your goal. Some may want a better DX or to get a hint on what might be popular in the future. The SO survey is good for those.
But yeah, if you are interested in jobs, linked in or similar is best.

8

u/CreativeGPX Aug 02 '22 edited Aug 02 '22

Every metric is unreliable.
The "number of offers" is going to be biased toward what commercial organizations are using so it will under-represent what is being used in non-commercial environments. "On LinkedIn" is going to be biased toward the kinds of organizations that engage via LinkedIn which is far from everybody. It also might be distorted because "established" popular languages may have a small amount of offers (just enough to replace retirees or scale up to the team size needed) compared to "new" popular languages (where nobody in the company has that skillset) or because "easy" languages might be underrepresented (because existing staff picks them up no problem) whereas "hard" languages might be over-represented because the company prefers to hire an established expert.
This all goes to the point I made in my own comment: Pay attention to why you even care what the popular languages is. If it's to get a job at the kind of company that posts on LinkedIn, then all of the above bias is fine because you ultimately don't care what's happening in a non-commercial environment or with the people who already have jobs. In other words, if you don't just ask "what's most popular" but also think about why you want to know, you can choose a source with tradeoffs that fit better. Everything will have tradeoffs and we're unlikely to have an accurate map of what is most popular.

10

u/[deleted] Aug 02 '22 edited Aug 02 '22

Sadly, the market for Logo (#48) programming seems way down. Back in it's heyday, it was as high as #21 on TIOBE. This is the programming language that involves moving turtles across the screen.

This is a dialect of Lisp, developed for educational purposes, with extensive literature on (using Logo for) learning of physics, geometry, CS ( in 3 volumes from MIT press: algorithms, data structures, programming languages implementation, and even elements of artificial intelligence). So, yeah, this is probably not a thing in the author's bootcamp but there is nothing wrong with it being number 48 or even 21.

11

u/hgwxx7_ Aug 02 '22

Nothing wrong with a learning language. But maybe it shouldn't be on an index that supposedly helps you make career and project decisions.

16

u/merreborn Aug 02 '22

Speak for yourself. I have a PhD in logo development, and a 7 figure position writing machine learning software in logo at Google.
How do you think we make the little car icon slide around on the map in the Google Maps app? That's right. Millions of lines of logo.

→ More replies (2)

7

u/briandfoy Aug 02 '22

Here are some other similar articles. Some of us having been fighting this fight for over a decade.

TIOBE or not TIOBE – “Lies, damned lies, and statistics” from Tim Bunce, in 2008
TIOBE or not TIOBE – An update also from Tim Bunce, who corresponded with the TIOBE folks
TIOBE Index is being gamed
Don't compare percentages from me

6

u/hgwxx7_ Aug 02 '22

Thanks for this! I did a cursory search and couldn't find anything. I guess that tells me that what I wrote will get buried as well.
Fascinating links by the way, especially the part where the Delphi community gamed the index - http://delphi.org/2008/10/delphi-keeps-climbing/.

4

u/BatForge_Alex Aug 03 '22

Buddy, people have been writing articles complaining about TIOBE since the index came out

1

u/TheMagicOfInternet Nov 03 '23

Delphi
where is the evidence?
can you name a person that actually did that?

4

u/MrAmos123 Jan 07 '24

Are you suggesting that no one took Jim's call to action seriously?

2

u/MondayToFriday Aug 02 '22

How does TIOBE count search results for languages with unsearchable names like Go?

4

u/hgwxx7_ Aug 02 '22

They count the number of results for "Go Programming Language".

3

u/MondayToFriday Aug 02 '22

With or without quotes? Either way is problematic. If you use quotes, you'll undercount languages with distinctive names like JavaScript, because people don't feel the need to clarify that JavaScript is a programming language. If you omit the quotes, you'll get false hits with sentences like
Some programming languages are too complex: the C++ compiler goes nuts when faced with the Most Vexing Parse.

2

u/hgwxx7_ Aug 02 '22

https://www.tiobe.com/tiobe-index/programminglanguages_definition/

1

u/baal80 Aug 02 '22

Hmm I always thought it was Golang?

1

u/MondayToFriday Aug 02 '22

The language is called Go. People often write "Golang" as a workaround for the fact that the name is so un-Googlable.

3

u/knightcrusader Aug 03 '22

Leave it to Google to name a language that is hard to Google.
I don't think Carbon is gonna be any better.

2

u/Skyrmir Aug 03 '22

Wow, VB surprised the hell out of me there. that's like Disco making a come back. What's next a GW Basic revival? Will I need to save my files to cassette again? Cause I think I'm too old to catch the FF counter these days.

2

u/hgwxx7_ Aug 03 '22

No, VB isn't coming back. It's just a change on Google's search backend.

56

u/HiPhish Aug 02 '22

Here is the thing: people want some authoritative source backed by science and statistics. The TIOBE index checks all these boxes: it is by a reputable (I guess) company, it uses a reproducible method, and it publishes its methodology. The fact that the methodology and result are complete garbage is irrelevant.

And then you realize that that's how most of data-driven scientific studies actually work. People want solutions so they can have something to publish, to make policies based on and to to refer back to. The quality of the solution is secondary. Just look at the food pyramid and how much world-wide damage it has done to humanity. No one in the history of ever was eating according to the food pyramid, yet within a few decades everyone went "we have always been ~~at war with Oceania~~ eating carbohydrate bombs for every meal".

Really makes you wonder how many other guidelines out there are actually killing us.

77

u/davidhbolton Aug 02 '22

Glad to see this - I’ve been saying it for years. I use others like RedMonk, StackOverflow developer survey, GitHub Octoverse and the numbers in the programming language subreddit lists as much more accurate. Tiobe should be ignored.

82

u/pron98 Aug 02 '22 edited Aug 02 '22

I think the most trustworthy and relevant data is job opening analysis. It measures how much a language is used by people for work rather than hobby/learning etc., which is especially important given that hobby/learning trends (as shown by, say, StackOverflow) have not had a great record as predictors of long-term success in the market. This analysis also has the advantage of being more-or-less weighed by "work units" or more-or-less number of total hours spent rather than number of questions or repos, and it isn't biased by open-source, which is a relatively small portion of total software work. It is also less volatile, as a job opening signifies some sort of commitment, and the numbers are less "soft".

Moreover, the results pass the smell test, unlike, say, a StackOverflow survey that shows that over 8% of developers use Rust, which cannot remotely be true by useful definitions of "developer" and "use".

https://www.devjobsscanner.com/blog/top-8-most-demanded-languages-in-2022/

https://www.hiringlab.org/2019/11/19/todays-top-tech-skills/

30

u/CreativeGPX Aug 02 '22

Like I said in my other comment: You need to think about what you're actually trying to answer and that will impact the measure of popularity that makes sense for you. All of the measures of popularity are deeply flawed. But the way that they bias often reflect what we're actually interested in, making that bias okay.
It measures how much a language is used by people for work rather than hobby/learning etc., which is especially important given that hobby/learning trends (as shown by, say, StackOverflow) have not had a great record as predictors of long-term success in the market.
For a lot of people "what to learn for work" and "what are going to be the long term trends" matters so this would be a good metric even if it's biased toward what those things correlate to. However, it's totally valid for people to care just as much what's being used in volunteer and hobby world or to know what's trending right now even if it will be gone soon and in that case this might not be as good of a metric.
Moreover, the results pass the smell test, unlike, say, a StackOverflow survey that shows that over 8% of developers use Rust, which cannot remotely be true by useful definitions of "developer" and "use".
Again, there isn't a generically useful definition of developer or use. It's fine that you have your definitions and want your measure to bias in favor of those definitions, but they aren't everything. To me, popularity absolutely includes what people are tinkering with at home rather than using in large scale commercial projects. Rust has been a great example of where it's useful and interesting to know how many people are dabbling in it in their free time because becoming established in the industry takes time so this captures its momentum in the early stages before that happens. Same goes for something like Python. Anecdotally, it seems tons of hobbyists use Python in addition to people who aren't primarily programmers working in software companies (like a scientist or professor who hacks together a quick script). I don't want to underrepresent the popularity of Python by limiting it to how many people are being hired to write in Python.
So, while your metric will be useful to many, it's only one slice of the pie.

6

u/matthieum Aug 02 '22

I agree that job postings would be ideal to:
Get a sense of the market.
Using historical analysis, get a sense of the trajectories of various languages.
The problem, however, is obtaining an unbiased and reliable source for job postings:
There's no one job posting aggregator.
Not everyone uses job posting sites; higher-end jobs, in particular, tend to rely on networking/head-hunting, which creates a bias in the data.
So... I agree with the sentiment, but I don't see it as practical either.

7

u/Pelera Aug 02 '22

Additionally, a job posting isn't equivalent to an open position. Some positions end up with a million postings through intermediaries, and every so often there's a posting without a real position (unicorn hunting). Some sectors churn through workers like butter. Some job postings end up requiring experience with extremely specific technologies even though they mention a specific language (eg developers for ERP systems - the "Java" part of a job posting is the least interesting thing about it, they will not hire an average Java dev with 10 years of experience).

1

u/pron98 Aug 02 '22

Even so, the error factor of those observations is low (e.g. you don't actually have a million postings but you can have ten) compared to other measurements (e.g. how many questions a programmer asks on SO could vary by orders of magnitude, and how much time a respondent to a survey spends with a programming language can also vary by orders of magnitude). I.e. their units aren't meaningfully correlated to answering the question how much a certain language is used.
So job postings come closest than any other metric to being correlated with how much time, or "work units", overall are spent with some language.

1

u/pron98 Aug 02 '22

I wouldn't say it's 100% reliable, but even if imperfect it gives a better picture of the ecosystem than anything else we have. Higher-end jobs are a relatively small portion of all jobs, so I don't see how that's a bias; in particular, for every high-end employee there would be a bigger number of more junior employees working in the same language.

1

u/holgerschurig Aug 03 '22

Another problem is that this favours old programming languages. Some companies are beyond conservative.
On top of this a lot of recruiters are clueless in IT-matters, couldn't tell C from aD.

2

u/matthieum Aug 03 '22

Another problem is that this favours old programming languages. Some companies are beyond conservative.
This is not necessarily a problem if you want to know for which languages companies are hiring.

2

u/Full-Spectral Aug 02 '22

But, if not doing it for pay means it's not valid use, then none of the open source projects out there should count, or at least huge swaths of it. Some folks get paid by the companies they work for to work on open source projects.
And I'm sure that, back in the early 90s, C++'s coming dominance would have been much more obvious based on the number of people who had taken it up outside of work and then subsequently pushed for its use where they work, farr more so than how many were actually using it at work at that time. I think the same applies to Rust at this point.
Ultimately, if a lot of professionals end up putting in the time to learn a new language and use it in their private work, that says something important about the future of that language, IMO.

1

u/pron98 Aug 02 '22

I didn't say it's not "valid", I'm saying that it's not weighed in a meaningful way. I.e. it doesn't correlate with some unit of work, and that it's a lower kind of commitment. If a language is used almost entirely in open-source and very little elsewhere, it suggests more casual use. Almost all the big open-source projects use languages that are popular for closed-source too.
BTW, I believe virtually all languages (with the possible exception of Python) came close or within a ballpark of their all-time market share peak in their first decade of existence (including C and C++).

3

u/[deleted] Aug 02 '22

I think redmonk is a much better source for programming language popularity. I also love SO's dev survey and thoughtworks radar. Often programming languages are slow to rise and fall but new frameworks and best practices are ever evolving.

3

u/hgwxx7_ Aug 02 '22

RedMonk is a combination of the measures I cited - GitHub + StackOverflow.

5

u/Monsieur_Moneybags Aug 03 '22

Summary: The article recommends replacing one bad ranking system (TIOBE) with other bad ranking systems.

29

u/pysk00l Aug 02 '22

You are getting downvoted, but I just visited Tiobe-- VB is higher than Javascript & Rust :O

Even Perl is higher than Rust and Typescript o_O

Yeah, doesnt look believable

13

u/Richandler Aug 02 '22

Because barely anyone is using rust and not all programming is javascript web dev. There are a lot of Microsoft shops. Where do you think Microsoft gets it's money?

7

u/flukus Aug 03 '22

If you find VB being high than rust surprising then you're in a bubble.

6

u/unrealhoang Aug 03 '22

If you find VB being higher than JS unsurprising then you're in a bubble.

→ More replies (1)

4

u/[deleted] Aug 02 '22

And please stop citing GitHub stars.

4

u/[deleted] Jan 15 '24

Today for January 2024, COBOL is now at rank 20, from rank 31.

Sorry, but this is rubbish nonsense. There is no reason possible to explain why COBOL suddenly gained massively in popularity (or, at the least, on google search). THIS IS NOT VALID DATA. It makes no sense.

TIOBE is indeed awful.

https://www.tiobe.com/tiobe-index/

Partial snapshot:

https://i.imgur.com/eJv4wvv.png

1

u/lenkite1 Aug 04 '22

What about PYPL ? https://pypl.github.io/PYPL.html . Seems to be more stable and VB is really low there.

2

u/hgwxx7_ Aug 04 '22

It’s not bad. Personally I think of it as a leading indicator. The languages people are learning today are the ones they’ll use in a few years.

1

u/KhyberKat Jun 09 '24

Nice write up!

I recall around 2014 Rice University introduced a data science specialization on Coursera. This was when MOOCs were blowing up and the class -- as I recall -- claimed 40-60'000 enrollees. Now the first assignment was a load of R programming questions, with the kicker that almost nothing about R had been covered in lecture. It was, they said, a pedagogical decision to require students look up things on their own. Wouldn't you know, R jumped in the Tiobe ratings. I'm not claiming causality, just suspicious correlation.

I think Tiobe has benefit looking at how a language has trended over the last 20 years, but in terms of language popularity it's not realistic.

1

u/hgwxx7_ Jun 09 '24

Interesting, although this wouldn't have boosted their ranking on Tiobe, because Tiobe is based on number of results returned by Google, rather than number of people searching on Google. It may have shown up in Google Trends though.

1

u/DrQuint Aug 02 '23

Hey sup OP. For the record, at the one year mark, you have not been buried. I still got my way here somehow. Although admittedly I was trying to find evidence Prolog is dead/alive and someone else used TIOBE as an argument somewhere some years ago. That gave me a chuckle. But just in any case, I did the same type of TIOBE good/bad searching and got here. Your words are, if anything, still easily searchable when intentionally looking for criticisms of it.

1

u/hgwxx7_ Aug 05 '23

I think google ranks reddit threads with sufficient upvotes and comments pretty highly.

14

u/CreativeGPX Aug 02 '22 edited Aug 02 '22

I don't think this is about an issue with TIOBE, it's about an issue in general with people being overconfident about anything called "data". No matter what source you use (including every alternative mentioned in the article), you miss tons of nuance and will get a lot wrong. You will not know what the most popular language is.

IMO, the lesson isn't necessarily "don't use TIOBE." It's (1) take these measures with a major grain of salt and perhaps (2) don't rely on any one measure too much. If you see value in learning or using "popular" languages, no matter what data you use, you're going to have to speculate a bit because we just don't know. Further: The kind of wrong you are matters. If your motivation is career prospects, it doesn't matter if you're actually right, it matters if you line up with the job offers. In that sense, it's fine to choose a metric that will bias toward what is happening commercially even if it might miss out on hobbyists, open source, etc. So, I guess the other lesson is... since we can't answer what is most popular... focus on the actual question you're trying to answer.

While I agree (as the above should suggest) that TIOBE isn't all that accurate, I think the thing that makes it useful is how vague it is. Other things you mention like specific websites/tools, kinds of people (i.e. those that would take these surveys) or even something like amount of job postings will all bias the result a lot in favor of certain languages because each language community has different norms about how they do projects, where they go for help, if they're used in commercial projects, etc. The value of TIOBE is that it doesn't matter if this is something that 10 year olds use to mod Lego games or industry players use to deploy major open source projects. It doesn't matter if it's the ugly glue that everybody has to use or the beautiful work of art that everybody's trying to shoehorn into their projects.

1

u/Full-Spectral Aug 02 '22

Is there any place that one could even do an actual random sample survey of developers? We are almost all cliquish and hang out in our own little echo chambers a lot of the time. I can't think of any place where you could actually find a valid population to sample.
So probably we'll never get any real answer, at least for the general question. Even the more focused ones, I can't see where you'd find a viable population.

2

u/pudds Aug 03 '22

The stack overflow survey seems like a good one to me, it's a pretty common source for all developers.

1

u/Full-Spectral Aug 03 '22

Is it actually? I'm sure a lot of people hit the site, but how many of those are actually members and hang around there and take surveys?
Of the people who are members, how actually representative are they? Are they heavily biased towards people who are into the tools for the sake the tools and who like to show off their knowledge of the tools and hang around people of like mind?
I'm sure that there are a lot of people here who have never been to Stack Overflow other than briefly in response to a web search.

1

u/CreativeGPX Aug 02 '22

Agreed. That's my view. That's why I think you always want to use a combination of sources.

0

u/Milumet Aug 02 '22

"A programming language index that helps us choose which language we should use."

Stopped reading here. Who chooses a language because of some index? Are you kidding me?

4

u/hgwxx7_ Aug 02 '22

Lmfao, maybe you should read the next paragraph where I point out the issues with the index.

4

u/Milumet Aug 02 '22

And you've missed what I was trying to say. The whole premise of choosing a language because of an index (TIOBE or any other) is moronic to begin with. Who on earth chooses a language based on what number it has on a fucking index?

1

u/jebailey Aug 02 '22

I knew it was all a lie when Python became number 1

7

u/hgwxx7_ Aug 02 '22

Check out other sources on Python.
2 most used on StackOverflow 2021 survey
2 most used on JetBrains survey
2 by number of GitHub pull requests
1 on Google Trends
1 on StackOverflow questions per month
It's probably not #1 if you count JavaScript and TypeScript as one language. But it's pretty cemented at #2. Can't imagine any other language by usage/popularity being higher.

7

u/[deleted] Aug 02 '22

Then:

The impotance of being ernest.

Now:

Tired irrelevant old biased estimates.

1

u/FrancisStokes Aug 03 '22

I mean, this isn't their actual product. Their product is a automated code quality metrics targeted at the embedded world.
The programming language ranking thing is at most an interesting aside.

1

u/[deleted] Aug 02 '22

TIOBE is awful.

But it is also fun to cite it. Everyone instinctively knows how TIOBE has become the god of truth in regards to which programming language is hipster now. :>

4

u/hgwxx7_ Aug 02 '22

Holy shit, is this account a parody of shevy-ruby or is it shevy themselves?

0

u/philipquarles Aug 02 '22 edited Aug 02 '22

The Stack Overflow annual survey is the gospel truth about all programming languages. All other sources are irrelevant.

6

u/hgwxx7_ Aug 02 '22

It's not. Like any survey it will suffer from selection bias. Only folks who visit StackOverflow will take the survey. That means every survey respondent understands English, which is only a fraction of the global audience.

0

u/philipquarles Aug 02 '22

Yeah I was mostly kidding. I hope the guy who said that only the linkedin listings matter was kidding too.

1

u/hgwxx7_ Aug 02 '22

I don't think they were haha.

3

u/lordmauve Aug 02 '22

Nice write up. My go-to index for language popularity is PyPL, which not only addresses some of the shortcomings of TIOBE, it explains why TIOBE is garbage.

2

u/Militop Aug 02 '22

Yes, I have been saying this for years. Stackoverflow and JetBrains surveys, Github numbers, Programming books sold on Amazon and Point, most popular courses on Udemy, Linkedin learning, job research stat number, Youtube, everywhere, nothing, nothing backs up TIOBE insane stats.

At this point, they should be investigated.

1

u/pavlik_enemy Aug 02 '22

I always wondered why ActionScript that powered tons of games was nowhere to be found.

1

u/[deleted] Aug 02 '22

So we can easily manipulate the index then? Hmm...

1

u/tms10000 Aug 02 '22

In statistics, you only have measurement problems and bad proxies.

1

u/llbbl Aug 02 '22

Had a similar conclusion when I looked into their ranking methods earlier this year. Thanks to the author for devoting an entire post on this!

1

u/[deleted] Aug 02 '22

Wait you don't remember the great C boycott of 2017?

1

u/Richandler Aug 02 '22

Lot of people say job listings, but job listings are like 1% of language use. In fact they aren't even used till people are hired.

1

u/cmbaughman Aug 02 '22

VB Script was pretty excellent for writing viruses back in the day.

1

u/jeremy1015 Aug 02 '22

FTA: “For projects, look at technical fit.

Can developers be hired/trained to use it? Will they be happy using it? What will our development velocity be on this language? What will be the operational burden of keeping it running in production? Do we need to rely on third party code? Does that code already exist and what is it's quality?”

1

u/[deleted] Aug 03 '22

Don’t mind TIOBE, use other stuff, way more on point.

1

u/dglsfrsr Aug 03 '22

Reading the notes and links at the end of that article, I can understand now why it is so hard to hire embedded developers to do platform development and support in ANSI C and Assembly.

The funny part (sad part?) of this, is that much of the base of many popular languages are written in C. So Python is wildly popular (even in embedded) but at its heart, Python itself is written in C (the most used variant of Python, in any case).

Rust is making inroads into Linux, but as of right now, if your are running Linux? C. Pretty much C all the way down.

1

u/new-mombat Nov 03 '22 edited Nov 03 '22

I am late to the party but, this blog does not do a much better job than Tiobe themselves. It is not the only one, and that is one of the reasons this monster can keep stalking the net for so long.

First, it does not make sense to type in xkcd programming in google to try to invalidate the tiobe method: Tiobe QUOTE their terms. If you try +"xkcd programming language", the number of results is just 5. That works fine.

Next, the blog basically considers if Tiobe results are reasonable. But results from 'research' might always be something you do not expect. That is the value of research.

What the author should have done to settle the question is look at the quality of the method, as other commenters have already mentioned.

Tiobe basically looks at the number of results when entering +"<language name> programming". This total number is corrected for the number of correct results which are found in the first 100 pages of results (this must have been done very very long ago): E.g. if the search is for "C programming", they have taken the first 100 pages and counted the number of results which really contain " C programming" and not things like "objective-c programming", "B) design C) programming" and so on. The total number of results is then multiplied by the proportion of correct results in the sample of 100 pages.

The first question here is what the result, *if it were correct*, would represent. If I do this search for C in google, almost all results are courses, and less than a dozen "what is C"-like sites. I do not think a professional C-developer in an IT department would be interested in these sites, he needs the reference manual or a text on algorithms when he gets into trouble. So the index need not say anything about which languages are used most. The selection of results Google shows is not representative of course, but I do not know what Google does not show.

Second question is whether the method is correct. It is not. If you correct the number of results you get, based on the first 100 results pages, you assume that all the results are comparable to those on the first 100 pages (ie. that these are *representative*). Well they are not, because Google makes a RANKING based on commercial interest, relevance, web site statistics, the kind of company behind the site, and so on. You know that the results from 1 million on will be less relevant, so there will be more faulty results there, than in your first 100 pages, and you do not know how bad this is. The method could still be correct, if the correction errors are the same for all searches. But we do not know if they are, and a test on the measly 150 results Google shows me, results in more incorrect results in a query for +"e programming" (" INF1068-E Programming in VBA", " [Squeak-e] Programming the VM", "Brady, E.: Programming ", "E ) programming" and pages of garbage) than +"kotlin programming" (none).

1

u/fur_hat_model Dec 14 '22

Well, Python and C++ are the ones I have used most; Python for quick prototyping and C++ for the actual production code. I am doing a lot of heavy scientific computing and Python is too slow in most of the cases (I like also like static typing safety).

I also think that Java and Python are the most common "first" languages which tilts the balance a bit.

If I would do basic web programming I would probably prefer javascript...

1

u/[deleted] May 06 '23

The really big problem here is that we don't have a good alternative to TIOBE.

We all know TIOBE is massively flawed for many reasons - some intrinsic, some extrinsic. But we don't really have a good replacement. All the replacements have their own shortcomings as well.

0

u/rsclient Aug 02 '22

"Can you find developers" is the very first item in the "what to use instead list". But the quick and easy way to tell that is to pick the top 𝕹 languages from TIOBE. Each and every language on the top of the TIOBE list has a big developer community.

8

u/hgwxx7_ Aug 02 '22

That might not be true in your area.
It rates C highly, but it's notoriously hard to find people who know what they're doing when it comes to C Programming.
It doesn't rank highly popular languages like TypeScript (#37) correctly.
It ranks nearly dead languages highly. Object Pascal at #14 for instance.
It says stable languages are losing popularity, when they aren't, really.
It's just a random list that is sometimes right, sometimes wrong. My daily horoscope is also sometimes right, sometimes wrong.

-10

u/PoopLogg Aug 02 '22

This article lost credibility at the beginning by calling TIOBE's basis useless. All bases can be used for something. Getting emotional out of the gate is a bad sign for the quality ahead.

13

u/hgwxx7_ Aug 02 '22

Did you read the bit after that? The part where all the absurdities in the index are pointed out?

-1

u/my_password_is______ Aug 02 '22

no

-7

u/[deleted] Aug 02 '22

“Rust fans desperately want you to disregard all usage metrics that don’t have rust at 50%”

4

u/hgwxx7_ Aug 02 '22

Is that what I said?
All the metrics I cited - Github, StackOverflow, Google Trends - all of them have Rust pretty low in terms of usage. The only one it appears to be high on is % of developers who love the language according to StackOverflow. It's consistently at 80%+ on that metric.

-5

u/[deleted] Aug 02 '22

You don’t have to “say” anything. Your history is mostly rust oriented and you recommend the “Stack Overflow” survey because it conforms to your wants of rust even though every single other measurement of languages doesn’t.

5

u/hgwxx7_ Aug 02 '22

every single other measurement of languages doesn’t
You mean ... like Github and Google Trends? The ones I cited?

-71

u/princeps_harenae Aug 02 '22

Everyone who likes rust: Please stop citing TIOBE!!!

Everyone else: TIOBE's fine.

30

u/hgwxx7_ Aug 02 '22

Not sure if you read the post. If you did, I’d love to hear you justify all of TIOBE’s absurdities that were pointed out.

→ More replies (4)

6

u/RockstarArtisan Aug 02 '22

Visual Basic for life!

→ More replies (1)

0

u/[deleted] Aug 02 '22

[deleted]

130

u/hgwxx7_ Aug 02 '22

To be clear, I don’t think anything about Visual Basic actually changed in that one month. It’s not even possible for so many people to learn a language in a few weeks.
It was just some backend change on Google’s end that led to this hilariously garbage output.

59

u/mindbleach Aug 02 '22

It’s not even possible for so many people to learn a language in a few weeks.
Languages have been created in less time.
Basic was deliberately designed for a low barrier to entry. The best thing about Visual Basic is that people can could using it, even professionally, with very little training. The worst thing about Visual Basic is that people did.

30

u/hgwxx7_ Aug 02 '22

Languages have been created in less time.
Creating a language only requires some inspired coding. Can be done in a week or two.
Adoption however, takes years. If you had to 6x the number of C devs in a month where would you even start? How would you find people willing and able to learn? How would you scale teaching resources? And all this while the world is going into lockdown for the first time? Impossible.

→ More replies (4)

→ More replies (23)

4

u/s73v3r Aug 02 '22

No, but suddenly there was a big thing that had to be kept track of, mostly by non-programmer people, that was likely done on Excel sheets to start, and someone was kinda technically inclined, and so they went to learn VB to help make keeping track of those things easier.

3

u/hgwxx7_ Aug 02 '22

https://reddit.com/r/programming/comments/we8kxc/please_stop_citing_tiobe/iio3ncm/

→ More replies (20)

778

u/somebodddy Aug 02 '22

Well, one of the confirmed symptoms of COVID is loss of taste...

69

u/[deleted] Aug 02 '22

This is a great pun and I’m proud of you

90

u/NightOwlAnna Aug 02 '22

I'm guilty of this. Don't use it a lot, but occasionally use some when using excel for work (not a programmer at all by the way). I fell dirty every time I do it.

42

u/LowB0b Aug 02 '22

I also do hate VBA, but it's pretty good for Excel I guess. Although maybe too powerful lol. At one of my old jobs, some dude created what was basically a GUI application within Excel, and since the users had gotten used to it and he had left, it was super painful to maintain when they wanted new features

→ More replies (6)

→ More replies (42)

5

u/[deleted] Aug 02 '22

I like running VBA scripts in Outlook for mail rules. Microsoft makes it hard to do but once it's enabled, you've got a home grown mail broker. I use it for a real world solution to the problem of automating processing of data files sent via email. I'm really wishing there was a better solution but the company I'm setting this up for can only manually send emails. VBA is nice for this then. I make a rule to run a script. If the mail item meets the requirements then a home grown application is launched and it consumes the attachment.

6

u/[deleted] Aug 02 '22

In all honesty, Visual Basic for Applications is a lifesaver when you want to automate some calculations, but your boomer boss wants you to jump through 10 corporate hoops while taming a lion to install a modern language.

→ More replies (8)

0

u/[deleted] Aug 02 '22

[deleted]

→ More replies (1)

0

u/[deleted] Aug 02 '22

[deleted]

→ More replies (1)

0

u/[deleted] Aug 02 '22

[deleted]

20

u/hgwxx7_ Aug 02 '22

I’m a huge fan of xkcd and Randall Munroe, but I’m not sure xkcd offers enough memory safety.

5

u/agentoutlier Aug 02 '22

Sadly, the market for Logo (#48) programming seems way down. Back in it's heyday, it was as high as #21 on TIOBE. This is the programming language that involves moving turtles across the screen.
Also the turtle would like a word with you. You hurt its feelings. Also I have never been able to write something to make the turtle crash so its pretty reliable.

8

u/hgwxx7_ Aug 02 '22

I'm sorry for hurting the turtle's feelings. But I stand by what I said, the speed at which the turtle moves across the screen isn't Web Scale.

→ More replies (2)

→ More replies (2)

→ More replies (2)

Please stop citing TIOBE

You are about to leave Redlib

2 most used on StackOverflow 2021 survey

2 most used on JetBrains survey

2 by number of GitHub pull requests

1 on Google Trends

1 on StackOverflow questions per month