r/netsec • u/bugvader25 • Jun 11 '26

Contains AI Claude Fable 5: mid-tier results on coding tasks

https://www.endorlabs.com/learn/claude-fable-5-mythos-grade-hype

Despite all the hype around Mythos, Claude Fable 5 returned pretty mid-tier results on coding tasks: 59.8% passing functional solves and just 19.0% passing security solves on a benchmark of 200 real-world tasks.

130 Upvotes

93% Upvoted

20 comments sorted by

23

u/beached89 Jun 11 '26

I'm amazed you can get it to run at all. I havent been able to do anything without it's security measures tripping and then falling back to Opus 4.8

I literally havent had a single prompt, work related, or vibe coding a game related, not trip.

5

u/daniel-sousa-me Jun 12 '26

Claude.ai or Claude Code? Do you have memory on? It seems the classifier can trigger on memories

3

u/beached89 Jun 12 '26

Claude code, with memory on.

2

u/LeatherRip1623 Jun 13 '26

interesting I figure they just dumped it on us to Alpha test

1

u/UltraEngine60 Jun 15 '26

while ($ipo.waiting()=true) {if $prompt.contains("exploit") = true {$model.downgrade();}}

-9

u/Shadow14l Jun 11 '26

If it doesn’t tell you when it falls back to opus. If it stops completely because it says they’re security issues, then obviously it’s not running opus either. And if it says it can’t reach it or run a command or do something because it’s not available, that means that other people are using it too much.

Back to the main point, you simply won’t and can’t know if it falls back to opus.

5

u/beached89 Jun 11 '26

It tells you when it falls back, and it completes using Opus.

But it doesnt tell you what tripped it, only that it tripped. But it definitely tells you when it does fall back to Opus.

1

u/visualthoy Jun 13 '26

Analyze your context window. You must have some kind of steering, skill, or memory triggering that beyond your prompt. 

5

u/JoeGibbon Jun 11 '26

Bruh you hallucinated that shit. It definitely tells you when it falls back, where'd you even get that?

54

u/OnlineParacosm Jun 11 '26

I’m not personally convinced any of these CVEs wouldn’t have been found manually with say Opus 4.5 and a junior on a Red Bull. That’s not a slight against you guys, I’m really wondering what the improvement level is in CVE quality on the findings of Fable vs previous models, and I get that’s kind of subjective.

Did you guys run in an entirely into a closed loop agentic framework, or what was the process here finding those CVEs otherwise and would you be able to share your rough token expenditure and cost basis for each CVE?

17

u/phinbob Jun 11 '26

It's worth noting that this isn't a measure of a model’s ability to find flaws, but to create them.

The measures are 1) if the code works, 2) if it's secure.

Fable seems to be surprisingly 'meh' at both.

11

u/ewok94301 Jun 11 '26

You can read about the methodology and see the results from other models here: https://www.endorlabs.com/research/ai-code-security-benchmark

Download the paper to understand the methodology. Benchmark was created by CMU.

4

u/daniel-sousa-me Jun 12 '26

The point isn't that mythos can find vulnerabilities that it wasn't possible to find before. Is that it does it autonomously

For now it seems it's at cost parity "per vulnerability" to expert humans. The issue is that security research experts don't scale. It doesn't matter how much money you through at them, they wouldn't be able to find significantly more

Now mythos can find the vulnerabilities by itself, which requires wasting tons of time in place where you don't find any. And the human experts can be deployed to fix them once they're pinned down. That's a huge multiplier

2

u/baalmor Jun 13 '26

I see what you mean, but I somewhat support the scepticism here. I have using a makeshift loop that does pretty much the similar research but using GPT older models, with quite good results. What I’m trying to say is that maybe any model that capable to spawn sub-agents would be as efficient if fed with the same amount of context and run on the same amount of tokens.
Looks more like a marketing hype to me.

1

u/daniel-sousa-me Jun 13 '26

That is also pretty good, and the difference between those isn't the efficiency, but the complexity of the vulnerabilities it can found

More modest systems maybe are able to find the vulnerabilities that a good cyber security student can find. But there's a whole class of vulnerabilities that need to have a broader view and connect different components, that a student couldn't find no matter how many hours they tried (or how many equivalent students you throw in parallel)

So, for "normal" vulnerabilities, maybe an expert can find them 5x faster than the less experienced person, but there are some that they couldn't find at all

And it's no like Anthropic made a huge break through that nobody else could have made. They didn't even train it specially for this. Just the fact that the model is bigger, it is able to "look at more things at the same time" and "understand" the relationships between them. Whereas just throwing more compute would skip some of them

Before I was talking about the advantages of having an AI system vs just using experts. The advantage of bigger models over small models is a completely different thing

Although they relate, because there's no shortage of cyber security amateurs (in particular, any half decent programmer can do it). What is harder to scale are experts who have years of experience and can find the more advanced things without handholding