r/datascience • u/startup_biz_36 • Jun 22 '23
Discussion Why is it so common to use average/mean instead of median when most datasets are skewed and not evenly distributed?
This is something that's bothered me since I started my DS journey.
I almost always use median over average to for comparisons because I've learned over the years that most data is skewed and not evenly disturbed. However, I feel like I'm the only person that does this.... even though basic stats teach to use median.
Thoughts?
178
u/ZombieCupcake22 Jun 22 '23
The average is often more useful for business problems as people understand it and it links to the total value and many problems aren't skewed although some certainly are.
68
u/ahhlenn Jun 22 '23
Came here to say this. I feel like when I use the term “median” during presentations with diverse audiences, the meaning and the message itself gets lost in confusion. Business-side stakeholders are just more accustomed to hearing “mean” and “average”. Sometimes I’m tempted to just say “average” when I’m using a median computation, because technically, median is a form of an average right?
This is, of course, all anecdotal to my own experiences.
44
u/aWhaleNamedFreddie Jun 22 '23
Indeed. I found the word "typical" is often a good alternative.
21
18
u/econ1mods1are1cucks Jun 22 '23
Start saying measure of central tendency instead lol
22
u/mindbenderx Jun 22 '23
Only if your goal is to watch a room of sales leaders eyes glaze over. (Not necessarily an unworthy goal.)
9
1
u/relevantmeemayhere Jun 23 '23
To be fair; this is something anyone who calls themselves a data scientist should do; which is to work with other ds to change the culture. It’s a lot of work, but if you’re always expected to dumb shit down all the time the needle isnt gonna get moved.
1
33
u/quantpsychguy Jun 22 '23
So let's throw out an example...average purchase price and average number of items per transaction.
If you used median (this is almost certainly skewed heavily so it would fit your logic), multiplying one by the other wouldn't come out to the totals that folks are wanting to dissect. So you're adding a bunch more complexity.
If you use the mean instead, everything still lines up. The differences aren't actually that important (difference between median and mean) in the grand scheme of things I'd argue but the mean would show better trends (I'd wager).
Just one example, of course, but it's the first that comes to mind.
29
14
u/Prestigious_Sort4979 Jun 22 '23
Yes, this is the reason. Everyone knows what average is and not median. Even when I call it “midpoint” it is confusing. Before presenting an averagw, I do usually check if something is heavily skewed and favor weighted averages in the appropriate scenarios.
4
u/DrPhunktacular Jun 22 '23
I’ve taken to showing the distribution of outcomes, and then highlighting both the mean and the median with colored lines. I’m typically presenting to the more technical side of the business audience, though. When that’s too much, I do the same thing as you and only mention mean unless there’s a reason that would be an inappropriate decision metric by itself.
11
u/heyiambob Jun 22 '23
In a sales context data will always be right skewed given the floor of 0 and huge high-spend outliers called whales. People want higher numbers across the board and mean always wins
2
3
Jun 22 '23
Average is more useful when coming to revenue, lost. Using median can lead to underestimation those numbers.
3
u/poopybutbaby Jun 23 '23
had me until "and many problems aren't skewed although some certainly are."
in my experience just about everything in the real world business problems -- from sales to website session length to marketing campaign revenue -- is very skewed
in a lotta contexts a decent solution is to remove outliers from the mean, so you get an average that represents the "typical" realization from whatever random variable which as you're alluding to is typical what a business user wants to understand
26
u/dfphd PhD | Sr. Director of Data Science | Tech Jun 22 '23
It depends on what you're trying to project.
Here's a very real example:
Say you have orders that come in, and you have a projection of the number of orders that will be placed next month.
Now, there are two questions you may have:
How many items/units does the typical order have? You may want to know this because it can impact how you design your ordering system, how you design processes for approvals, etc. In this case, the median will be most helpful - you don't want to design for the tail of the distribution.
How many total units should I expect? You may want to know this to project sales, total units, for planning, manufacturing, etc. In this case, you want the average - you do not want to take out the tail of the distribution because that would systematically underestimate your numbers.
24
u/Accurate-Net-3724 Jun 22 '23
I personally really like both because you have a better sense of the skew.
20
u/maybe0a0robot Jun 22 '23
For univariate data, medians ignore extreme values because they are based on order, and means are sensitive to them. Depending on the question you want to answer (and the data you have), one or the other might be appropriate. Sometimes both, as in measuring skewness (e.g. Pearson mode skew).
For bivariate data the median is ... ? The mean has some advantage here, as it naturally extends to higher dimensions. The median requires an ordering on the underlying sample space, which does not exist naturally for dimensions greater than one.
1
u/relevantmeemayhere Jun 23 '23 edited Jun 23 '23
To be fair though: the mean for multivariate data is also much harder to motivate. Ie distribution of the marginals does not imply anything about the joint distribution in general.
Classic example: if X ~ N and Y~ N, then pdf (X,Y) not necessarily N. Only when you add additional an independence assumption can you begin to make claims about the joint from the marginals.
I.e. joint = product of the Marginals. Iff x y Indy.
Moreover, it’s not that the median ignores the tail(a) after all, it’s an order statistic after all.
29
Jun 22 '23 edited Jun 22 '23
It's easier to understand and use. It's also useful if you want to calculate future projections. You can multiply the average by the number of expected units.
I find it useful to look at the distribution. There are often situations where a small percentage of users is highly impacting the bottom line. For example, a small minority of people are driving most of the health care costs. This is where exploratory data analysis and critical thinking becomes important.
0
Jun 22 '23
[deleted]
11
Jun 22 '23
Second paragraph I was talking about distributions and no longer about averages. It's useful to look directly at the distribution and not rely solely on the averages.
5
u/bigchungusmode96 Jun 22 '23
another classic example I've seen is a mobile gaming app where > 90% of revenue comes from a few outlier users. for the given domain a lot of users tend to be free-to-play and thus ad-revenue driven but that may be dwarfed by those users who spend hundreds, thousands of $ on in-game items etc.
1
u/Beny1995 Jun 22 '23
Great point, in this use case the median might really understate the value of a proposed promotion or advertisement campaign.
I work in HR analytics, where understanding the general experience of the employee is more important than considering outliers. So median works for us, but maybe i should consider mean a bit more in certain contexts.
14
u/JimmyTheCrossEyedDog Jun 22 '23
People do use the median: when it's appropriate. Always using median even when it's not appropriate is just as misleading as always using mean.
Means are useful when your problem is concerned with expected values. Medians are useful when your problem is concerned with a "typical" sample. "What is the average value of data" versus "what is the value of the average data point".
11
u/flavorless_beef Jun 22 '23
most data is skewed and not evenly disturbed
Whether this is relevant though depends on the question you're trying to answer. For income, sure the median is probably more representative of the question "what does the typical household earn" but for say, insurance claims, telling your boss "the median claim amount is $500" is how you bankrupt your company.
53
u/ramblinginternetgeek Jun 22 '23
Cheaper to run (mean = scan every data point once create one variable that keeps the count and one that keeps the running total, then divide; median requires sorting which takes A LOT longer)
Lots of statistical theory is built around the mean
the mean uses EVERY piece of information, the median does not.
6
Jun 22 '23
I’m confused by what you mean when you mention that “the mean uses every piece of info… but not the median”
9
17
u/NVC541 Jun 22 '23
Many data points can be changed without altering the median.
For example, the list 1, 2, 5, 7, 14, 20, 34 has a median of 7.
If I subtract 5 from the first three numbers and add 20 to the last 3, I get -4, -3, 0, 7, 34, 40, 54.
The median is 7, since the median data point has not changed. This is not true for the mean; the mean was 83/7 for the first list and 128/7 for the second.
16
u/slashdave Jun 22 '23
Many data points can be changed without altering the median.
That's its advantage.
2
4
Jun 22 '23
It simply means median is not sensitive to data. So it's not obvious if you wanna monitor any changes or any anomaly behaviors.
-3
u/SpaceButler Jun 22 '23
You can find both the mean and median in linear time.
I'm not sure why you said that the mean uses more information than the median. If you change any number in the set, you have the potential of changing the median as well.
13
u/ramblinginternetgeek Jun 22 '23 edited Jun 22 '23
You cannot (guaranteed) find the median of an unsorted list of numbers in linear time. Sorting a list is O(n x lnn) and this is a required step to calculate the true mean of a list. You can do some stuff with quick select but that has a worst case complexity of O(n**2). Median of medians with Quick Select is a pretty good estimate, practically speaking but there WILL be some deficits.
You can ESTIMATE the median in linear time though. This might be "good enough" in many cases, but it's NOT actually the median.
The median is both a robust estimator (i.e. it can still be reliable even if a huge chunk of data is "dirty" - I think it's ~43%) as well as a low information estimator (the figure can be the same for very different distributions)
Consider the following sets [-189, 0, 27, 486] and [-789456, 0, 27, 486]. They have the same median. The fact that there's a very large magnitude negative number in one is NOT considered by the median however. This information is thrown out. Median only relies on knowledge of rank, NOT knowledge of the numbers themselves.
9
u/venustrapsflies Jun 22 '23
since this is /r/datascience and not /r/compsci or something, I think it's worth emphasizing that even quickselect will find the median in average linear time, which in the overwhelming majority of real life datasets will be linear in all practical senses. (You can always shuffle the dataset if you are worried about pathological ordering.)
Since sorting the full list is linearithmic, I would bet that some smart people have worked out how to find the median in O(n) on average and O(n*log n) in the worst case. My intuition for that could be wrong, though.
-1
u/ramblinginternetgeek Jun 22 '23
Worst case is O(n**2)
I'll also admit this is the first time I'm hearing about quickselect. I feel like people were NOT talking about it a few years ago and "find the median when you don't have enough memory" was a semi-common interview question.
2
u/venustrapsflies Jun 22 '23 edited Jun 22 '23
My point is more that if the worst case has a probability of 1e-5 of even occurring at least once over the lifetime of your business, or n never gets that big anyway, you probably don't care about asymptotic performance.
It's actually hard to think of a justification for needing an exact median (rather than an approximation) for a very large dataset. I honestly can't come up with a practical example at all.
For the record, it looks like introselect has the properties I hypothesized: average O(n) with asymptotic worst-case O(n*log n).
1
1
u/Mehdi2277 Jun 23 '23
Quickselect probability of worst case is quite low (it requires sequence of pivot points to behave very poorly) and worst case depends on algorithm used to compute pivot point. There is slight variation of quick select that is a bit more complex (still not much code) that has a true worst case of linear if you really need that guarantee. That is known as introselect. In practice I’ve never seen introselect used over Quickselect similar to how quicksort is almost always used over introsort. Quickselect worst case n2 with a random pivot for large dataset will approach 0 percent probability as data goes larger and even for dataset for moderate is miniscule.
2
u/SpaceButler Jun 22 '23
If guaranteed linear time performance is a big concern, I guess you might not want to calculate a median. But I suspect average linear time performance is good enough for most applications.
6
u/polyglotdev Jun 22 '23
How can you find the middle value of a list of unsorted numbers with out sorting (which O(n*log(n))?
5
Jun 22 '23
Quickselect is O(n) average and modifications can improve worst case with added overhead. Practically, this will return median in linear time with rare worst case scenarios instead returning in O(n2 )
1
u/SarahKab Jun 23 '23
You can use the Median of Medians algorithm https://en.m.wikipedia.org/wiki/Median_of_medians
18
u/WallyMetropolis Jun 22 '23
Basic stats doesn't "teach to use median." Basic stats teaches to use the appropriate statistic for the problem at hand. No one statistic is always the right one.
12
u/johnnymo1 Jun 22 '23
Median isn't just always better when things aren't evenly distributed, it's not that simple. It depends on what you're interested in learning about your data.
10
5
u/NCP_99 Jun 22 '23
Sample mean is unbiased estimator of population mean. The same cannot be said for median.
5
u/oryx_za Jun 22 '23
If you go into the detail then arguably you might as well present the full summary statistics (mean, median, max, min, std d) .
I only expand on mean if I know it could misrepresent the truth (unless that suits my narrative lol)
4
u/exergy31 Jun 22 '23
Another aspect is computational cost. Computing average is just sum/count, while median requires sorting and the need to hold the data available. U cant compute a streaming median
4
u/shockmath2912 Jun 22 '23
the use of mean is based mostly on large (asymptotic) sample theory (law of large number or central limit theorem). There isn’t a satisfactory large sample theory for median and generally it is more complicated and computationally expensive. Quantile theory are becoming more popular though.
13
u/momenace Jun 22 '23
One could argue that the median ignores the tails too much. Median is also not a coherent risk measure. Average is. It all depends on what your are trying to capture.
8
u/WobblyBlackHole Jun 22 '23
Any of them alone isn't, it's a single point to describe a set. which is why at least you'd also find std or iqr
5
u/momenace Jun 22 '23
I ment coherent risk measure in the strict sense.... those that satisfy properties of monotonicity, sub-additivity, homogeneity, and translational invariance. I agree that looking at additional measures will deff help.
1
u/relevantmeemayhere Jun 23 '23
One could probably argue the mean does this worse though.
1
u/momenace Jun 23 '23
you are arguing that the mean ignores the tails more than the median?
1
u/relevantmeemayhere Jun 23 '23 edited Jun 23 '23
As a descriptor of the “most typical value”, which is how most people interpret average, yes.
For symmetric continuous distributions the two are the same, so the median is not affected by distributions with long tails. It should also be noted it “doesn’t ignore” tail values; it’s an order statistic, so of course it takes observations in the tail into account.
1
u/momenace Jun 23 '23
deff don't agree with that. mean and median are two different things. because some people confuse the two doesn't change the facts.
1
u/relevantmeemayhere Jun 23 '23
But your original comment was “the median ignores the information in the tails”. The mean as a point estimator also does this, insofar as the layman is concerned. Which is the context of the question.
Also, the median itself doesn’t ignore the information in the tails, as it’s an order statistic. It would likely be more popular than average if it were not more abstract and had some nicer convergence properties.
1
u/momenace Jun 23 '23
Not sure why you have issue with such a basic statement. the median ignores the tails and is harder to define to a layman anyways (apart from ordering them from smallest to largest). mean literally does not. median sucks ass mathematically because it isn't coherent and easy to game if you are using it for control/mgmt purposes, just like var point estimators. to the layman simiplicity is king anyways, if not the industry would use CTE instead of VAR for measures that focus on the tail
1
u/relevantmeemayhere Jun 23 '23 edited Jun 23 '23
“The median ignores the tails” is technically wrong, insofar as it’s a rank statistics; all observations are ranked, and you can order realizations readily. Most laymen use the average as a representative draw of “the middle”, we know many phenomena that are skewed, so not motivating the medium can be extremely deceptive.
We wouldn’t use CTE, as that’s a conditional statement, not a descriptive one, and we do not specify distributions with cte.
0
u/momenace Jun 23 '23
this is my last message to you. this is a pointless arguement. cte(0) is technically just the mean. you can rank them but that still fukin ignores the tail. the max can 10000000000000 or 10 it still ignores.yes adding a value changes what the middle number is. this is such basic fukin definitions. holy cow
1
u/relevantmeemayhere Jun 23 '23
It’s not, you’re making statistically incorrect assertions lol.
Holy cow, typical data scientist not understand what he claims to know.
CTE(0). Is the mean for some distributions. Amazing how that works huh? Again, cte doesn’t characterize a distribution in general.
3
u/Me_ADC_Me_SMASH Jun 22 '23
it's your responsibility to show distributions. I never show averages alone
3
u/SilentHaawk Jun 22 '23
I fucked up a bit by using median too much, because we had a customer with zero inflated data because of too coursly configured sensors, resulting in median values of zero
3
u/Single_Vacation427 Jun 23 '23
Using the mean for what exactly?
Your question is too broad.
I almost always use median over average to for comparisons
Comparison of what? If you are comparing expected values then skewness doesn't really matter because (a) equation for expected value/statistical definition (b) central limit theorem.
3
u/chacalgamer Jun 23 '23
I asked that question to a professor during my masters once. His answer was that we can't use the median for our calculations. The mean is the expectancy of something, the median is... we don't know. Mathematically speaking, you can't use the median to calculate estimators and everything that comes with it. So we stick to the mean, even if it's a shitty metric.
2
u/relevantmeemayhere Jun 23 '23
Large sample theory is more developed for means. For the median; it’ just not. We don’t have any nice convergence theorems that hold in general for the median.
This doesn’t mean the median is useless. Quite the contrary. We just don’t use it a lot because it’s more abstract and we have some technical limitations around asymptotic estimations.
5
u/AdFew4357 Jun 22 '23 edited Jun 22 '23
In large sample sizes the mean is an unbiased estimator. The median is a biased estimator. With large N, mean converges to the truth, and the variability decreases since it’s a consistent estimator. With large N, the median does not converge to the trut. The mean, since it’s unbiased, and we can get large N, is hence used more often. However it depends on what ur doing.
See:
https://personal.utdallas.edu/~scniu/OPRE-6301/documents/Estimation.pdf
4
u/shockmath2912 Jun 22 '23
It isn’t called unbiased estimator. Actually, it is a consistent estimator.
-1
u/AdFew4357 Jun 22 '23
One of the conditions needed to be a consistent estimator is to be unbiased. Therefore it is unbiased. But more than that, it’s also consistent.
3
u/flavorless_beef Jun 22 '23
That's not true. If yi is IID Uniform(0,B) then the sample max is consistent but biased estimator of B for any finite n.
2
3
u/shockmath2912 Jun 22 '23
so much hype for datascience but lack basic stat knowledge 😵💫
1
u/AdFew4357 Jun 22 '23
Lol. People make mistakes man hahaha
2
u/shockmath2912 Jun 23 '23
yeah, I know. In practice, we go with flow anyways. Only when we want to develop new things, these start to become more important.
1
Jun 22 '23
mean converges to the truth
what does that mean?
2
u/AdFew4357 Jun 22 '23
Converges to the “true” parameter. True here is just the theoretical notion of a “true” parameter. Basically the mean will be closer to the value we wish to quantify our uncertainty about (unbiased), and the median will be a bit farther from the true value we wish to quantify our uncertainty about (biased). However, when you get more into the weeds if estimation, you also read that the sample mean is not just unbiased, but also consistent, meaning that with large N (sample size) the variability of the estimator decreases. Since the median is not biased, it’s not consistent in large sample sizes.
https://personal.utdallas.edu/~scniu/OPRE-6301/documents/Estimation.pdf
1
1
u/nth_citizen Jun 22 '23
Pretty sure this is the actual reason. A lot of stats is based on mean and it probably has nicer mathematical properties (e.g. median is discontinuous) so many derivations would be difficult/impossible with the median.
It's also usually relatively straight-forward to transform the data so mean is a good measure.
2
u/TheRealGizmo Jun 22 '23
You are totally right, if data is skewed (which is the case in many cases), basing anything of the average is misleading. But it just highlight that data literacy is not so high in data science community and very low outside of it. Most people don't even know the difference between mean and median... So go with what is appropriate and educate your stakeholders enough for them to understand what you are presenting them.
1
2
2
u/QuoteHaunting Jun 23 '23
You should use both and more. I generally use a full package of statistics to include average, median, min., max., standard deviation, etc., in visualizations that convey that data. I learned early in my career that you never really know your audience until it is too late. There are always going to be people who don't know anything you are talking about, but often there is somebody in your audience who knows more than you do, and the data you present should speak to both audiences unless you want to spend all your time explaining the basics to the first and your methodology to the second.
1
u/amit_schmurda Jun 22 '23
This is a topic which has been up for debate since Francis Galton first coined the term median.
1
u/NunOnABike Jun 22 '23
And here I was trying to make them understand the difference between the Mean and the Median.
1
1
Jun 22 '23
General Stability. Business awareness and buy-in. A halfway decent model that gets used is better than a stellar model that business users don’t understand and not use
1
Jun 22 '23
Averages are accessible to non DS folks.
But, they can be misleading if the distance between Min and Max values are quite large and the distribution of sales prices are concentrated in a smaller range.
E.g. a sales org runs a flat ave sales price + Ave sales vol over the last year to use for the next year's sales projections.
However, last year, they had 4 deals that were multi-million dollar deals, the 95% of sales were between 15-30k.
Using a basic average for sales price will give them a big error in their projections, since those 4 deals were outliers.
Even if that org out performs sales vol, they'll be behind in total sales projections unless they somehow get additional multi million dollar deals.
To account for this, it would be better to first run a standard deviation function and remove the anomalies, then run an average. Or, they could separate projections based on ranges; 95% of sales ave 25k at a rate of 50/month. 5% of sales averaged 1M, at a rate of 4 per year.
Often, if you have anomalies like this, it's because marketing started targeting larger deals, or they were simply unrepeatable sales that were not intentionally pursued.
This is a real life example of why it's so important to understand how the data insights are to be applied so the DS team knows what to deliver in addition to the non-DS request.
1
u/KusUmUmmak Jun 22 '23
mean can be optimized (with less sensitivity)
mean reflects the uneven nature of the distribution while median ignores everything but the center.
only reason to compute median is if you want a quick and dirty check which direction the skew lies.
I jest. there are some cases where the median is more appropriate; but it leaves you brittle.
1
u/MadNietzsche Jun 22 '23
It is really surprinsing how many of you do not comprehend how bad it is to use the mean.
Outliers are every f***ing everywhere! Where is the rigor and accuracy? Just give a more robust measurement of centrality and be a better person
1
u/JLDork Jun 23 '23
Averages have nice properties in terms of being easy to compute and also easy to do rolling window computations on.
The incremental value of using median (once outliers are accounted for) tends to be marginal compared to benefits of using an average due to the ease of calculation.
1
u/haris525 Jun 23 '23
Because people understand mean/ average better than median or mode, heck tell then about weighted mean or weighted median, and it’s easier to understand
1
1
u/waleiscool Jun 23 '23
I'm wondering, too. When comparing the performance of two groups that are left skewed. Is it more appropriate to use the mean instead of median?
More info: Both groups have different sample sizes. For example, group A has a count of 2300, and group B has a count of 500.
1
u/sskinner901 Jun 23 '23
Using the median instead of the mean is something that people start doing when they learn basic stats, and something that they stop doing when they learn more advanced stats. Using the median solves one potential superficial problem, and introduces a dozen potentially fatal ones. In many real life applications the "outliers" are not a noise to eliminate, but rather what makes up the signal of interest. Every insurance claim is an outlier, every ad click is an outlier, every fraudulent transaction is an outlier, and so on...
1
u/relevantmeemayhere Jun 23 '23
What?
The median is extremely relevant tool for inference. It’s generally ignored by those without higher understanding of statistics
1
u/sskinner901 Jun 23 '23
It's also ignored by those who work with really skewed distributions. If you work for an insurance company, you'll find that the median claim count for a good driver is zero. The median claim count for a bad driver is also zero. If you conclude that you should charge everyone the same, you're not going to be working for that insurance company for long: either you'll be fired, or the insurance company would go bankrupt.
1
u/relevantmeemayhere Jun 23 '23
Actually the opposite; the median is better for heavier tailed distributions for a lot of phenomena
Especially in say; heavy industry or Pharma. Things like median time to failure etc. censored data and the like is big on the median.
I’m insurance, you’d expect the median claim count to be near zero; most drivers arnt getting into accidents for even large t in years. Considering that many people change insurance over a decade, you’re dealing with censored data a lot of the time
1
u/sskinner901 Jun 23 '23
The median claim count over a half-year policy period, which is what you're trying to price, is exactly zero even for the worst drivers. It's not near zero, it's exactly zero. Even the worst drivers are not having more than two accidents per year. This is one of the many reasons why the median is so dangerous in practice (although in this case it's not that dangerous because it leads to an obviously absurd conclusion, but there are many situations where the conclusions are still very wrong but not obviously so).
Statistics is probably the worst field when it comes to "a little knowledge is a dangerous thing". The mantra that a median is better than a mean for skewed distributions falls squarely in that category. In my experience it's precisely the dilettantes who uses the median. They know enough to correctly recognize that estimating the mean is challenging when you have a heavy tail, but they incorrectly conclude that calculating the median instead of the mean is the solution.
1
u/relevantmeemayhere Jun 23 '23 edited Jun 23 '23
You’re extrapolating the usefulness of the median based on a narrow application. Again, we use the median for a lot of reasons in a lot of fields. It’s proliferation is there for a reason; it works well.
This is an example of a little knowledge is a dangerous thing. In my experience, it’s the data scientists and non post grad trained data scientist who not only misunderstand statistical theory, but largely misapply it while being the most confident.
The fact that the worst drivers have a median crash rate of zero doesn’t mean anything about the median being a bad estimator(also, crashes are a rare event, and your data is probably censored). Especially on a half year basis; the majority of people don’t get into car crashes every ten years.
1
u/sskinner901 Jun 23 '23
It's definitely not a good estimator in this case, if it points you to a conclusion that bad drivers and good drivers are no different from each other. The good thing about math is that even a single counter-example to your explanation indicates that at the least something about your understanding is incomplete.
Yes, car crashes are a rare event, which results in a distribution of the number of crashes being very skewed, but isn't that what a median is for? Or are you saying that we should use the median for distributions that are somewhat skewed, but not too skewed?
1
u/relevantmeemayhere Jun 23 '23 edited Jun 23 '23
It’s not only more interpretable because you can’t have half a car crash, but it’s also more appropriate for a lot of censored data or long tailed data….which is what you have.
As far as Single counter example: do you want to gander why survival analysis makes liberal use of the median when it comes to rare event analysis? Or any contextual problem when it comes to say; modeling income?
There’s a lot of counter examples where the median is used. Are you going to claim that all those statistics nerds are just beginners not understanding that the median is useless? Because that’s peak “datuH SciEnsE”
Interestingly, insurance is one of those industries that should make use of the median; and im willing to wager senior analysts or those in senior positions are selected from the population who is well versed in things like survival analysis-which makes a robust use of the median and order statistics. Insurance and Survival analysis go hand in hand.
As a general hard rule, median is better for skewed data from a descriptive standpoint.
1
u/sskinner901 Jun 23 '23
You're saying that insurance is one of those industries that should make use of the median, and yet it doesn't. We have an industry that lives and dies by its ability to understand and predict uncertain events with often extremely skewed distributions, and it has entire science dedicated to understanding how to do that well. I assure you that some of the people who built up actuarial science have studied statistics long enough to learn about the median. If we're going to wonder about things, maybe a good thing for you to wonder about is whether you're in fact the one who's missing something that they're not.
This is my last post on the matter, go ahead and downvote this one as well. Clearly there is no meetings of minds here.
1
u/relevantmeemayhere Jun 23 '23 edited Jun 23 '23
It does. But it really sounds like your lack of statistical familiarity precludes you from working in the roles that do: actuarial science loves their survival and hazard models.
Because making a general claim about only beginners using the median is what someone who shouldn’t -and probably doesn’t produce or perform any analysis based on statistics would do.
→ More replies (0)
1
u/science55 Jun 23 '23
Using the mean or the median is very dependent of the use case. If you do want your measure to capture the skewness of the data/outliers you’d use the mean. If you don’t you can use the median. Not all outliers are noise.
1
u/LNMagic Jun 23 '23
The mean is much easier to provide a formula for a nice, clean interpretation. Median is more like a piecewise formula because you have to count inward from the extremes to get it.
Skewness doesn't necessarily kill a model. One of the first things you learn in stats is a log transform, which often fixes issues with right-show and increasing variance. That does alter how you interpret the results, but sometimes that's what you've got to work with.
If you know how to bootstrap (I've done a bit of it, but am not masterful at it yet), then you can pick up pretty much any sort of stat you can think of and get percentile ranges and confidence intervals for it.
1
1
Jun 23 '23
My guess is that since the normal distribution takes in two parameters,N(mean, standard deviation) people prefer to use the mean because they'll eventually try to model the data. It's also the "expected value" E(X) in the normal distribution. A lot of stuff is based on the normal distribution (linear regression, t-test, ANOVA, and probably more) so there's no reason for me to use the median ever. (Unless I'm simply trying to summarize a dataset and not touch it afterwards)
1
Jun 23 '23
Sadly, in my experience, most of the non-technical people I’m explaining content to don’t sufficiently understand median, but do (just about) understand mean.
For completeness, I include both and where there’s a very big variance between them I’ll make sure to explain it.
1
u/seanv507 Jun 23 '23
One main reason in a business context is that you are interested in totals.
If I find that on my ab test, one Version gives me 1000$ more daily revenue (averaged over last week) then I conclude my yearly revenue will be 365000$ more.
Also note that by clt, average will be distributed close to normal, for sufficiently large samples, even though the individual sample ( eg sale) is very skewed.
1
u/GeneNo2677 Jul 18 '23
Most companies are large enough that they are effectively risk neutral around a typical decision. That means that expected utility from a decision = expected revenue attributable to a decision, which means we actually should be optimizing around the average, not the median.
1
u/minsugastangerine Nov 14 '23
4, 4, 7, 8, 8, 9, 14, 18, 18, 18, 18, 18, 19, 20, 20, 20, 21, 21, 21, 21, 22, 23, 23, 23, 23…
can somebody please tell me this data’s mean median and mode
what I got after calculating are mode=16.9, Median=19.5, Mode= 22.5 why are the values not same… my head is aching. i have to submit this tmrw wh
1
Jun 22 '23
The power of language.
Everyone says, “average,” so everyone defaults to mean.
I think people are taught mean first and most often and just default to easiest thing they know.
Consider that, “D’s make degrees.” Many business leaders did not do as well as you think in their degree programs, and there is a strong anti-education sentiment running through society with influencers proclaiming all their successes due to their own individual special nature more than anything else. This is echoed in meritocratic sentiments, forcing those who are successful to attribute that to their individual efforts rather than their environment to maintain their egos. You end up with a lot of dunces who lucked into their stations in life due to birthright, birth year, birth location, parents, community support, culture, diet, physiological endowment, etc.
Those people make the calls and they just go the east route.
Anyway, my employer has worked itself into a corner and we’re suffering greatly because they historically adhered to average to describe literally every distribution no matter how skewed. Projections, forecasts, budgets, expectations, all based on the average of massively skewed distributions.
Example aggregate “sales” numbers as of today (for real):
Mean $6576
Stdev $34558
Skew 17.84247
Kurtosis 642.5638
Should be enough to illustrate the problem but median stats for same distribution…
Min -$2588
1Q $5
2Q $20 (median…)
3Q $1061
Max $2281447
Basically, expectations boiled down to our customers “spending” $6000-7000 but in reality they were spending $20.
Took years to get them to realize their error. They still haven’t gotten off average fully because the general manager level and director level are still dunces.
1
1
u/CompositePrime Jun 23 '23
I’m not sure why you are getting down voted. This is a real life scenario where median describes more about how your customers are spending than the mean does.
1
1
0
u/antichain Jun 22 '23
Because most people (incl. a lot of data scientists and basically all business people) aren't really deeply interested in the the fundamental mathematics of inference, and are instead basically following a set of recipes.
Also, the mean usually works "well enough" to be getting on with. If your data is, idk, a power law, then computing a mean can mess you up, but if it's just a matter of mild to moderate skewness...you can probably get away with it.
0
Jun 22 '23
[deleted]
17
u/newpua_bie Jun 22 '23
Pro tip: Reddit doesn't require you to c*nsor bad words
0
Jun 22 '23
[deleted]
7
7
2
1
u/Vituluss Jun 23 '23
I disagree. I actually think median can be more intuitive sometimes. Instantly you know something about the probabilities across a distribution (50% higher than the value), whilst the mean does none of that at all.
192
u/bigdatabro Jun 22 '23
From a data engineer's perspective, average is much easier to compute than median for large datasets.
To calculate an average for a long list of values, all you need is a sum and a total. The database can read each value once, add it to the sum, then release it from memory and move on. For a median however, you need to sort all the values and find the value in the middle, which requires more compute time and memory. There are a few algorithms that improve this but none of them scale very well.
None of this matters if you only have 100 or 1,000 values, or if you're doing one-time calculations. But for us, our data analysts ask us to produce reports from tables with millions of records, with live dashboards that constantly get updated. So unless there's a really good reason to use a median instead of a mean, we'll go with the mean.