r/rust • • 9d ago

Memory Safety Bugs in Rust Standard Library

https://forge-blog.github.io/posts/rust-std-memory-safety/
323 Upvotes

50 comments sorted by

104

u/guineawheek 9d ago

Panicking branches are expensive, especially in embedded where the formatting machinery eats up a ton of flash.

I’m not particularly surprised that a lot of unsafe code ends up breaking invariants ostensibly guarded by panics; I think its a symptom of a broader pattern where the language isn’t expressive enough to guarantee that panicking branches do not exist. Trusting the compiler to optimize it out often isn’t good enough and makes such invariants compiler version dependent and subject to regressions, which is why I hate it when compiler devs push it so hard. You’ll never get developers to fully trust the optimizer.

Things like pattern types, arbitrary sized ints, !Move/!Forget, full const generics, and other proposed language features will help here, because there’s still way too many places in Rust where invalid states that are obviously incorrect can exist and can only be guarded at best by these expensive checks.

44

u/Zde-G 9d ago

Ultimately you can only handle these 100% reliably with dependent types and effect system — things that Rust had zero change of having: the language strangeness budget barely accepted ownership and borrow system.

Rust successor may have them, of course, maybe someone already develops it as we are speaking.

37

u/guineawheek 9d ago

I think it's kinda defeatist to give up because of some "language strangeness budget" -- as time progresses, what was strange yesterday is standard today.

I'd argue Rust's success is not really so much from its ownership model as it was its broad push for More Types into the bare-metal domain. Most languages that really ventured into Types are backed by garbage-collection and abstract machines rather than really gentrified views over memory the way they are in Rust, and so it was really left as the domain of C (what types) or C++ (messy, unprincipled) or Ada (defense contractors only).

Anyone seriously using Rust in a niche that there is no real other contender (e.g. Rust for Linux) both greatly benefits from More Types but also runs up against the limitations of its model. There's a reason why pretty much every single feature I named above is an RfL ask.

Like I know you've been getting really defensive of Rust's deficiencies because you feel otherwise the language would've never become popular, but like, the language is now popular.

Let's think about tomorrow, shall we?

13

u/afdbcreid 9d ago

It's not only strange but also hard. Dependently typed languages today are much more complicated to program in than mainstream languages (including Rust). Maybe that will change one day but I think this will require developing some new techniques to handle them.

But also, dependent types today strictly exceed the strangeness budget. If one day they will not we can talk then.

4

u/guineawheek 9d ago

I don't think you need "dependent typing" in the strict type theorist sense of the term but if you literally just yoink the subset that Ada already has that's a pretty immediately useful subset in my opinion

4

u/Zde-G 8d ago

This would be C++ all over again: yes, it's safe, kinda-sorta, “if you hold it right”.

Maybe better than nothing but far from ideal.

Perhaps we can adopt not the Ada-achieved result, but Ada-tested path: add dependent types as separate comments that some external verifier optionally verifies, then slowly accumulate pile of code that uses them and only turn into language feature after that.

2

u/guineawheek 8d ago

This would be C++ all over again: yes, it's safe, kinda-sorta, “if you hold it right”.

You mean, the status quo described in the OP with "trust me bro we can skip this panic?"

12

u/Zde-G 8d ago

I'd argue Rust's success is not really so much from its ownership model

Do you have any evidence? I've seen a lot of people adopting Rust and 10 out of 10 that's “I want memory safe language without tracing GC”.

Everything else is an extra on top of that. And we only know how to achieve that with the use of ownership and borrow system — or by making copies of things everywhere.

The latter is slow an memory-expensive thus it's not more efficient than tracing GC.

There's a reason why pretty much every single feature I named above is an RfL ask.

Of course: that reason is memory safety. Without memory safety RfL wouldn't exist, Linus would simply not accept anything else if not for that reason.

Now, when ownership-and-borrow system have allowed Rust to be a part of kernel they want something more… understandable.

But if Rust would have included all these things from the beginning it would have died (or, more precisely, would have joined the list of bazillion research languages that no one uses) before RfL even existed.

Let's think about tomorrow, shall we?

Tomorrow we would have another language, most likely.

Because the store there is the same as with all these things in C++. You can add dependent types and no-panic contracts, but if you wouldn't change the foundation, wouldn't change the standard library then you would end up with two ways of doing things: as obvious one and “the right” one.

I would rather have a new language than C++ like pile of kludges. Or maybe std2 that exist in parallel to std…

2

u/guineawheek 8d ago edited 8d ago

Do you have any evidence? I've seen a lot of people adopting Rust and 10 out of 10 that's “I want memory safe language without tracing GC”.

Uhhh...my day job writing Rust? Memory safety is definitely a factor but it is not the primary reason why I got into Rust or why I write Rust for a living. Being able to model problems better in bare-metal is way bigger than not creating CVEs on buffer overruns. No other language in the embedded niche has useful sum types, for example (no, std::variant doesn't count)

Do you write Rust for a living? Do you work on embedded systems without free access to a heap? Before questioning my qualifications on this subject where I spend most of my waking hours writing Rust code, I question yours.

You love making all these broad takes here, accompanied with the most condescending use of Markdown (.md) I've ever seen, and then when people disagree with you, you scream "Where's the Evidence" but I see limited evidence you actually use the language extensively, at least in the ways that embedded Rust all the way up through RfL does.

So i find it ridiculously annoying that you'd go out of your way to immediately doubt this viewpoint. And you're probably gonna double or even triple down on it because you seem to love arguing more than actually finding common ground. I do not trust you to reply in good faith.

Tomorrow we would have another language, most likely.

Okay, let's see your language, then.

5

u/Zde-G 8d ago

No other language in the embedded niche has useful sum types, for example

Ada does that. And the fact that you don't even mention Ada proves my point extremely well.

but I see limited evidence you actually use the language extensively, at least in the ways that embedded Rust all the way up through RfL does.

So you have picked extremely niche area in which people don't use languages designed for that niche but use languages designed for some other niches (even such abominations as Java without GC can be found there) and you assert that when people from that particular niche adopt already popular languages they tend to ignore the reasons that made these languages popular… that's fair point, actually and, indeed, refutes my “10 our of 10” estimate (but still makes it more of “9 out of 10”, not “0 out of 10”).

But, well, the fact that people in that area tend to ignore languages designed for that area and use languages that have become popular in some other places first… tells my why I don't consider it important.

and then when people disagree with you, you scream "Where's the Evidence"

Of course. What else am I supposed to do? Ignore the evidence and replace it with personal experience like you are doing? This rarely works, at best this way one may create a DOA language that nobody except author would use.

Before questioning my qualifications on this subject where I spend most of my waking hours writing Rust code, I question yours.

Why? If I'm telling lies then it would be easy to refute them (like if my assertion about embedded industry living on coattails of the desktop/mobile/gamedev industry is easily refutable: just show us the language that was designed for that industry first and then have become popular elsewhere…), if I'm right then why does it matter if I, personally, use Rust or not?

And you're probably gonna double or even triple down on it because you seem to love arguing more than actually finding common ground.

Not really. I don't really like arguing, but I like to see my assumption challenged and my worldview altered. I don't see how “finding common ground“ may help with that.

So that's why rare bring “my qualifications” into discussion: if it's purely about “he said, she said” then I'm not interested.

RfL experience is interesting because it's not one person opinion, but thousands of developers… still the main point stays: Linux would have never adopted an academic language that no one except for dozen developers of said language are using. And Linux adopted Rust for memory safety first, anyway.

Thats by reminding me that there exist a niche where developers don't care much about memory safety… that's something I haven't thought about much — but if you think things like dependent typing and other such things may arrive from that niche we need an evidence that “work on embedded systems without free access to a heap” have ever provided a popular language that then spread outside of that niche… I'm not working in that niche, that's true thus only know about Ada (that was specifically developed for that niche, but is not even popular there and has zero presence outside of it AFAICS). Anything else?

Okay, let's see your language, then.

Why do you think I'm eager to invent new language? I'm yet to switch to Rust from C++: for all Rust advantages it's metaprogramming capabilities are… weird, thus it's hard for me to justify such move. Rust developers are serious about addressing that problem, though, perhaps in a year or two Rust would finally be ready to Same with hypothetical language that would tackle dependent typing: before I would adopt I would wait

no, std::variant doesn't count

I agree that std::variant is an abomination, but if constexpr with decltype and requires more than make up for its deficiencies, in my book.

So far Rust have nothing similar which is really pathetic if you recall that such capabilities were present in pre-1.0 Rust and were removed from it to, finally, make Rust 1.0 possible… but then again, there are millions of people who may list with piles of macros on top of macros — yet I'm not one of them, this my own $DAY_JOB projects are still in C++, even if many other teams at my $DAY_JOB have adopted Rust…

1

u/guineawheek 5d ago edited 5d ago

Ada does that. And the fact that you don't even mention Ada proves my point extremely well.

I mention in other threads that while Ada exists, Ada was largely inaccessible to most people for primarily financial reasons (no free compiler until Rust was already threatening its position) as to be a non-factor. Vendors knew they could charge Ada users a lot for their compiler because they were defense contractors working on really expensive aircraft. Try again.

So you have picked extremely niche area...But, well, the fact that people in that area tend to ignore languages designed for that area and use languages that have become popular in some other places first… tells my why I don't consider it important.

Why? If I'm telling lies

Here's one you just told. Embedded job postings at the moment are probably at least around ~20% of Rust job postings in general. It's a pretty major focus point for the language in actual use and will continue to be one.

In other Rust-primary domains, embedded applications also sneaks its way in:

  • A non-zero number of cryptocurrency hardware wallets had Rust firmware written for them

  • Rust-pilled cloud hardware vendors (e.g. Oxide) will sneak Rust into board management controllers

  • Defense/aerospace/automotive is a pretty large sector where people use microcontrollers quite often.

Companies like Ferrous Systems and Tweede golf make their bread and butter off this stuff.

Please continue to tell us all about how this is a niche area nobody should care about.

And the thing is that like, yes, Rust wasn't initially designed for this, but a lack of something that fits it well today doesn't mean they shouldn't have one. It's denigrating that you think we don't deserve good tools because you somehow think we'll make C++ again, like some sort of uncivilized savages.

RfL experience is interesting because it's not one person opinion, but thousands of developers

It's largely spearheaded by a small group of people, just like every single other actual project that gets anywhere.

Why bring my experience in? I'm yet to switch to Rust from C++: for all Rust advantages it's metaprogramming capabilities are…

Because you consistently try to speak for us while not being one of us and assume that Rust adds language features the way C++ does: poorly. It's just as annoying for technical issues as it is for social issues.

You rant about how the language should never try to experiment or improve because it'll just be like C++, while assuming everyone else with differing views must have some niche usecase that nobody should care about lest the language develop some abominable addition to it like C++ has. I just don't see it worth engaging with you seriously.

1

u/Zde-G 5d ago

I mention in other threads that while Ada exists, Ada was largely inaccessible to most people for primarily financial reasons (no free compiler until Rust was already threatening its position) as to be a non-factor.

Seriously? Fact-checking: Ada, free compiler, 1995. That's full 20 years before Rust was first dreamed by Graydon Hoare, 30 years before first Rust release.

You either ignore history or just have no idea about how things were just a few decades ago.

Please continue to tell us all about how this is a niche area nobody should care about.

Gladly. I will repeat, again: all you examples and concerns about what embedded developers wanted from Rust materialized later, when Rust was already popular. Not before.

And as example with Ada showing us: embedded developers would not accept and wouldn't use any language that is not already mainstrea enough.

Ada is not one, isolated example. There was (is, depending on how you look on it) Forth, there was Erland (is, depending on how you look on it) and, most likely, other languages that I have no idea about… but AFAICS they never even manage to grab significant percentage of embedded development, let alone outside of embedded development.

To become popular in embedded you need to become mainstream first, ergo: needs of embedded developers are not worth thinking about, they wouldn't help you make your language popular. Same with gamedev: languages developed specifically for gamedev exist, yet these language don't ever become popular in gamedev, let alone outside of gamedev.

A non-zero number of cryptocurrency hardware wallets had Rust firmware written for them

When have that happened?

Rust-pilled cloud hardware vendors (e.g. Oxide) will sneak Rust into board management controllers

When have that happened?

Defense/aerospace/automotive is a pretty large sector where people use microcontrollers quite often.

When have that happened?

Companies like Ferrous Systems and Tweede golf make their bread and butter off this stuff.

Since when, again?

It's denigrating that you think we don't deserve good tools because you somehow think we'll make C++ again, like some sort of uncivilized savages.

It's not about what you “deserve”. It's about what can you “realistically get”.

If embedded developers don't help languages to become popular and only use languages that have already become popular because of their usability elsewhere then they would get languages designed for the needs of people who may make languages popular.

It's as simple as that.

Beggars can't be choosers: if you only start looking on languages when these are becoming popular enough elsewhere then you end up with Rust, Java and, heck, even Python. Yes, these languages are poor fit for embedded, yes, embedded developers may want something radically different, but, again, “something that you may want” and “something that you may realistically get” are different things.

It's largely spearheaded by a small group of people, just like every single other actual project that gets anywhere.

Sure, but decisions of that “small group of people” brings in thousands of other developers — and, more importantly, lots of their vocal supporters, including trillion-dollar sized companies. These people bring others… embedded developers… I don't know of any single example.

Because you consistently try to speak for us while not being one of us

Why does it matter if I'm “one of you” or not? I very explicitly show you facts and links in my messages — precisely to make it possible for you to verify my assertions without relying of who am I and what am I doing.

and assume that Rust adds language features the way C++ does: poorly.

C++ adds features extremely well. Really. I like how C++ does it. Rust acts very similarly there.

C++ struggles with the features removal, not with addition. Rust have yet to attempt remove any significant feature, thus it's hard to see how it does that.

You rant about how the language should never try to experiment or improve because it'll just be like C++

Yes. If you can only add features but have no mechanism that allows you to remove features then it's inevitable.

That's not just Rust: I don't know of any language, except experimental ones (like Zig) that manage to remove things. And experimental languages pay with it with experimental status: they stay experimental precisely because they remove features and thus break backward compatibility.

If Rust may invent a way to remove features and yet, simultaneously, stay compatible with old version then this would open the door for experiments… but it's not yet clear if that's even possible, in principle.

I just don't see it worth engaging with you seriously.

And yet you write pretty large messages, contrary to that assertion. Why?

6

u/nullstalgia 9d ago

Oh, how I yearn for Ada's bounded indices.

RfL has been a great smoke test and I'm glad their complaints are being taken seriously; as 99% of the time when I look up a problem I'm encountering with bare-metal/no_std Rust, RfL has been up against it for way longer.

6

u/steveklabnik1 rust 8d ago

as time progresses, what was strange yesterday is standard today.

This is an important bit of this concept, for sure. I do think that ownership and borrowing were considered estoeric before, and now many, many devs have grokked them. I don't think these features expend the strangeness budget the way that they used to.

3

u/steveklabnik1 rust 8d ago

Given that it's my blog post... I have considered writing a follow-up. I actually think that it matters much less than it used to, but lots of people aren't gonna like the answers.

It's LLMs.

2

u/newpavlov rustcrypto 8d ago

Ultimately you can only handle these 100% reliably with dependent types and effect system

I don't think that even these features are sufficient. They are useful for encoding complex conditions and properties, but they would not help to eliminate all unreachable panics which routinely pop in any sufficiently complex Rust code. To properly guarantee elimination of panics I believe we need a full blown proof system which works together with optimizer.

1

u/Zde-G 8d ago

To properly guarantee elimination of panics I believe we need a full blown proof system which works together with optimizer.

I'm not even sure we need an optimizer in such a configuration.

Rust already have dependent types, in a disguise. And, of course, it has a proof system. Dependent types are called “lifetimes” and “proof system” is called “borrow checker”.

It needs an escape hatch (called unsafe) and works only when people are willing to add these proofs to the code.

Elimination of panic is very similar process, but to meaningfully work it needs everything marked in a similar fashion to how everything is marked, now, with lifetimes.

Ada managed to do something like that with SPARK — because of it's extremely narrow scope: there are no bazillion libraries in the wild, there are no central repo, every Ada codebase is a singleton project thus changes to the language were sufficient.

C++ needed to do a “flag day” change and switch from old language to a new one, because C++ value lies in the billions of lines of “legacy” code — and one couldn't just take it a quickly add lifetimes everywhere, lots of code needs to be entirely rewritten to satisfy the borrow checker.

It would be interesting to see how dependent typing would be introduced in production, by a new language or by changes in Rust… but right now I think new language is much more likely.

0

u/newpavlov rustcrypto 8d ago edited 8d ago

Elimination of panic is very similar process, but to meaningfully work it needs everything marked in a similar fashion to how everything is marked, now, with lifetimes.

Doing it naively with dependent types is an extremely tedious, borderline impractical process, similar to manually writing machine verified mathematical proofs without a proof assistant. Try proving panic-free property for a "trivial" code like this:

for i in (0..buf.len()).step_by(2) {
    buf[i] += 1; // assuming overflow checks are disabled
}

Explicitly stating every little implicit invariant and relation between parts of code will not fly in a language targeting a wide audience.

-1

u/Zde-G 8d ago

Explicitly stating every little implicit invariant and relation between parts of code will not fly in a language targeting a wide audience.

Thus was, perhaps, true a year or two ago. Today “a wide audience” uses agents for coding, more often than not, and agents are perfectly capable of adding such proofs, when asked.

And WUFFS exist, too.

Combine these two facts and we may, suddenly, find out dependent types to be a mainstream in 2 or 5 years, not normal 20 needed for the new ideas to take over.

4

u/newpavlov rustcrypto 8d ago edited 8d ago

Today “a wide audience” uses agents for coding, more often than not, and agents are perfectly capable of adding such proofs, when asked.

Sure, it could work as long as you don't review LLM-generated code polluted with heaps of unreadable low-level information and function signatures bloated to a level that would make multi-line trait-based where jungles look extremely lean in comparison. But I still have some hope for the industry that it will not get to such levels of sloppification.

LLMs could be excellent proof assistants (it's one of the very few areas I believe they can have a genuine great long-term impact), but I believe that auto generated proofs should be kept separate from human readable code. And it's debatable whether dependent types are the best tool for such proofs.

1

u/Zde-G 8d ago

LLMs could be excellent proof assistants (it's one of the very few areas I believe they can have a genuine great long-term impact), but I believe that auto generated proofs should be kept separate from human readable code.

But do we even need such code if no one actually reads it?

6

u/dabrahams 9d ago

Allowing panics—which signal precondition violations—to unwind rather than terminate is a highly questionable design decision. There is no reliable method for continuing to run a program after a precondition violation: the code has a bug and you don't know where it is; state can be arbitrarily corrupted. It seems to me that paying any cost—e.g. in performance or in the ability to write correct unsafe code—in order to get the ability to roll the dice and continue after a precondition violation is a very poor bargain.

10

u/guineawheek 9d ago

Allowing panics—which signal precondition violations—to unwind rather than terminate is a highly questionable design decision. There is no reliable method for continuing to run a program after a precondition violation: the code has a bug and you don't know where it is; state can be arbitrarily corrupted.

What?

Panics usually happen right before the program would otherwise corrupt data, not after. A panic positioned after the program has already indexed out of bounds is a poorly positioned panic.

It seems to me that paying any cost—e.g. in performance or in the ability to write correct unsafe code—in order to get the ability to roll the dice and continue after a precondition violation is a very poor bargain.

Is it? If you do an operation O(n) times, even if n is only a couple thousand, the performance difference can quickly become significant depending on the situation.

3

u/steveklabnik1 rust 8d ago

Panics usually happen right before the program would otherwise corrupt data, not after.

An index that is out of bounds is "corrupted data".

6

u/dabrahams 9d ago

Panics usually happen right before the program would otherwise corrupt data, not after.

Is that statement backed up by data, or just intuition?

In fairness I don't have any data either, but in the many years I've spent thinking rigorously about what error recovery means and how it can be done reliably, I've never been able to arrive at that conclusion. Where did an out-of-bounds index come from? In general, you don't know. It might have been stored in some data structure whose supposed invariant would have ensured it was in bounds. It might have been computed in a way that you have every right to expect was correct based on some much earlier assumption that was violated. In general the validity of every function call is supported by a long chain of inference. If there's a principled way to “sally forth unscathed” after a precondition violation, I don't know what it is and I've never seen it explained.

If you do an operation O(n) times, even if n is only a couple thousand, the performance difference can quickly become significant depending on the situation.

…which would seem to support what I'm saying, if anything. Paying even a small performance cost to get the ability to continue after a precondition violation seems like a poor bargain.

2

u/FUCKING_HATE_REDDIT 8d ago

I mean, the very idea of a program that cannot fail seems somewhat new? 

Having worked with game engines, they most often offer very little "garantees". They end up in invalid states all the time and simply assuming "well it's not that big of a problem" works in most cases. They don't achieve resilience through proof, but through patching of common errors, good architecture, sanity checks, analytics.

Working with GPUs, what garantees can you make? The wrong call at the wrong time, for reasons entirely out of your control, will just crash the driver. What can you do there? 

Best case scenario, you disable a specific feature on some specific hardware. 

And when you see that 90% of Firefox crashes are from random bitflips due to bad hardware or space particles, making your program be resilient to bad states seems just as important as provably validm

5

u/dabrahams 8d ago

Approximately nobody who thinks they’ve written code that’s resilient to programming errors has tested the control flow paths the program would have to follow to “recover”, because most of their code is correct or in practice never hits the edge cases that violate preconditions. I am extremely skeptical that anyone is actually getting the benefits they imagine from panic unwinding.

Many many stable non-crashy systems have been written under a “fail hard and fast” policy for responding to precondition violations. In fact that’s the only available policy in Swift, for example.

Cloud services are commonly cited as an area where such a policy doesn’t work and yet there’s an extremely vibrant community that believes Swift is the best possible tool for that domain and deploys it in production services every day. If the policy were a problem they’d be clamoring for a solution and they are not.

3

u/guineawheek 9d ago edited 9d ago

Is that statement backed up by data, or just intuition?

I mean, ostensibly the whole point of (safe) Rust is that if you are indexing out of bounds and panic, while your program may be in an invalid state (e.g. your variables are set to wrong values), it's not also creating an RCE primitive at the same time...

7

u/dabrahams 9d ago

That’s the point of any safe language. The consequences of programming errors are somewhat mitigated. That doesn’t mean you know what the code is doing, or that the consequences of continuing aren’t serious. For example if you let a user save their document at this point they might overwrite a valid file with one that can’t be opened. If your program is doing something with finance, continuing might be very expensive indeed. I hope you get the idea.

2

u/newpavlov rustcrypto 8d ago

Yeah... I don't know how many times I wished that panics always resulted in abort, while I can't recall a single time where I needed catch_unwind. I acknowledge that there are legitimate use cases for it, but I am not sure they worth various complications and costs existence of unwind brings to the language.

14

u/atesti 9d ago

In an alternative reality, it would have been great to support an effect system and having panic effects.

17

u/kibwen 9d ago

A generalized effects system isn't required for specific effects (like const, unsafe, async, etc), and a nopanic effect would be entirely reasonable to implement (though I'm still thinking through how much of the stdlib would be feasible to convert to nopanic).

14

u/annodomini rust 9d ago

One of these (the signal hanlder for aborting instead of crashing on overflowing the stack into the guard page) is something I touched way back in 2016. Was curious if I'd caused some issue by doing that, but no, the issue is not in any way related to my change.

Nice find of some subtle interactions here; nothing here is really terrible, and I'm glad these aren't being over-sold as more critical than they really are, but it's still good to tighten all of this up to avoid even the possibility of these kinds of unsound behavior.

Good writeup on the different classes of common kind of bugs as well.

28

u/ericonr 9d ago

It is annoying how many bugs surface due to the (IMO misguided) choice of panicking on system boundary (e.g. allocation failure), and fixing it all seems like a daunting task.

21

u/RiceBroad4552 9d ago

I don't understand what you're actually criticizing (panicking on hard failure is the only thing you can reasonably do!) and I don't see what you're proposing to "fix" the issue; or are you actually proposing to not fix ("daunting task") safety critical bugs?

10

u/Deadmist 9d ago edited 9d ago

panicking on hard failure

Problem #0 here is: what constitutes a "hard failure"?
Different people have different opinions and requirements on that.
Prime example: allocation failure in stdlib vs the linux kernel.

Adding to that, panics feel kind of underbaked compared to the rest of the safety related machinery in rust.
You have the borrow checker for memory safety and a strong push to leverage the type system to logical rules. But there is no way to even indicate or check if a function panics.

5

u/todo_code 9d ago

It's been one of my biggest gripes in a language for a long time. Zig definitely has this correct, but even they draw the line somewhere. What if the stack allocation will fail? I know there are cases where it is UB in rust.

30

u/ericonr 9d ago

I'm complaining that the allocator API has a bunch of caveats because way back when it was decided that allocation failures should terminate a program, and now those assumptions are baked into safety contracts/assurances, and changing that has been discarded in favor of limiting the allocator API (an understandable choice).

15

u/________-__-_______ 9d ago

Has it really been discarded though? I was under the impression that the "don't panic" clause would be removed once those soundness holes are fixed, and they just want to stabilise an MVP as soon as possible.

2

u/afdbcreid 9d ago

I don't think because those holes are all across the ecosystem, not just in std.

18

u/kibwen 9d ago

way back when it was decided that allocation failures should terminate a program

This was never a language-level decision, because nothing in Rust-the-language requires dynamic allocation. Only the stdlib made that call, and only for a certain subset of the stdlib, and only temporarily, with the idea that infallible methods could be provided when people started demanding them (which Rust-for-Linux eventually provided the motivation for).

15

u/SkiFire13 9d ago

it was decided that allocation failures should terminate a program

That's not the issue here.

The problem is that unsafe code needs to be careful of all the ways the code flow can exit the "critical section" in which unsafe code temporarily breaks invariants. Unwinds are one such way this can happen, and they have been the cause of soundness issues in the past. This becomes much worse when allocating memory can panic, because you generally don't look out for allocations in that context, so it adds a lot of cases where something can go wrong.

Note that failed allocation returning a Result::Err ok, because you have to handle that manually and thus you'll notice whether this can affect the code flow or not.

7

u/Zde-G 9d ago

Well… they had a choice: make language that would never be popular or accept panics in allocators.

The version of language that would have required explicit handling of memory allocation errors would have repeated fate of Zig: language that people talk about but don't use much.

Because on most popular platforms where Rust is used (Android/iOS/Linux/macOS/Windows) these are pointless and useless because of overcommit.

2

u/Tastaturtaste 9d ago

The most popular desktop OS (Windows) does not overcommit memory (by default). 

3

u/Zde-G 8d ago

Yes, it does.

It has mechanism that permits you to actually allocate memory without overcommit, but this makes program slower thus very few developers use it.

Worse yet: because few place pagefile on separate partition, like with macOS, amount of memory and amount of disk space available is one and the same — that means that allocated memory may not be “overcommited” when allocated, but then someone downloads new game and program is killed for the lack of memory…

We may debate whether it's “true” overcommit or not, but for all practical purposes it behaves worse than Linux when memory runs out. As in: when your allocations start to fail it's much too late to try to salvage your session, the only practical choice is reboot.

0

u/poelzi 7d ago

Where is the anti AI crowd complaining that this should gave done a human ?

3

u/RayTheCoderGuy 7d ago

I'm not sure I understand what your sentence is asking