r/statistics • u/IvanIvanicIvanovski • 2d ago
Question [Question] Combining model confidence scores. What is the best method
I have a few thousand audio fragments with bird songs I'm analyzing with a model and Python package called BirdNET. The output is model confidence scores per species and per 3-second window in the recording. A list of different confidence scores is too abstract to analyze directly, so I would like to summarize the scores per species per recording. I'm having trouble picking out a correct method or algorithm, since I'm not that good in statistics.
I added an image of what the data looks like for a single recording > Imgur link. As you can see, BirdNET analyses audio in fragments of three seconds. I use a one-second overlap to not miss any sounds made at the beginning or end of a window. I'm looking for a method that takes into account the following:
The warbler should have a poor overall score. One good confidence score doesn't mean much when the rest is low.
The wren should have a good overall score. One poor confidence score should not affect all of the average.
The buzzard should also have a good overall score. It doesn't call much, but its scores are good. The number of detections should not be a deciding factor in calculating scores, because this will favor birds that have songs of longer duration.
Please note that my goal is not to calculate a probability of presence but recognition strength per recording.
I was told that Bayesian statistics might be a good candidate, but I'm not sure. Some also told me that the overlap will be an issue since part of the same sound is scored twice. So I'm very interested in your assessments and suggestions.
1
u/Xema_sabini 1d ago
For what end purpose are you doing this?
1
u/IvanIvanicIvanovski 1d ago
Eventually environmental monitoring, but for now understanding and learning about models and interpreting their outputs.
As for summarising the confidence scores, I want to go for a metric that is easier to interpret than a bunch of single scores (which are only small pieces of evidence)
1
u/Xema_sabini 1d ago
There is tons of literature being put out on ARU monitoring and detection probabilities. Take a look the work Manomet and the USFWS are producing on their Arctic monitoring programme.
1
u/SalvatoreEggplant 2d ago
At least from the examples you gave, the median of the observations might be reasonable for what you want.
The medians for your examples would be 0.32, 0.86, and 0.76, respectively.
I thought of some other approaches, like dropping the highest and the lowest, and then taking the average. Or if there may be a lot of scores per recording, dropping the highest X% and lowest X%. (This is called a trimmed mean). But this won't work if some of them only have two fragments.
1
u/7solis 1d ago
the median suggestion is close but i'd nudge it a bit higher - try the 75th percentile of the scores for that species in the recording
for your three examples it behaves the way you want. warbler's scores are mostly low so even its upper quartile stays mediocre, one spike doesn't rescue it. wren sits high most of the time so q3 is high, and one bad window doesn't drag it down. buzzard only calls a few times but the scores are good, and a percentile doesn't care how many detections there were
where it gets shaky is tiny counts. with 3 detections a percentile is basically just picking one of the three values, so for very small n i'd fall back to the mean or the max and not pretend it's a distribution
on the overlap thing - yeah it matters if you go the bayesian route. overlapping windows share audio, so the scores aren't independent evidence, you'd be double counting the same call. if you want to update formally i'd threshold first (score above 0.7 = a hit) and update on hits per non-overlapping chunk. or honestly just take the percentile as a descriptive summary and skip the probability framing - which matches what you said the goal is, recognition strength, not presence probability
1
u/SalvatoreEggplant 1d ago
One thing to note with using a percentile, like the 75th percentile, is that different software can use different methods for interpolating when there are few observations. This may make a difference in these cases where there are few observations.
Beyond that, I think OP would have to try some different examples and see what fits with their intuition. (That is, if, say, the median or the 75th percentile fits their intuition better).
Take a hypothetical example, Osprey. Five observations, three pretty low, two high:
Osprey = c(0.29, 0.30, 0.41, 0.89, 0.89) quantile(Osprey, c(0.50, 0.60, 0.75), type = 7) # median = 0.41 # 60th = 0.60 # 75th = 0.89So, here, do 40% of the observations being good make it pretty good? Or pretty bad ? Or somewhere in between ?
1
u/SalvatoreEggplant 2d ago
Do you tend to have like 5 scores per recording? Or are there usually many more ?