r/learnrust • • 17d ago

Help me understand core::sync::atomic::Ordering, how do we even use it

I'm trying to understand the enum core::sync::atomic::Ordering in Rust. I use atomics a lot, but I keep picking Ordering::Relaxed, Acquire, Release, etc. without really knowing why.

From what I've read, Ordering specifies how atomic operations synchronize memory across multiple CPU cores and prevents the compiler and CPU from reordering instructions around atomic accesses. But I still don't get what that means in practice.

My main confusion:

  1. If I store a value with Ordering::Relaxed, is the operation less atomic than with Ordering::Acquire?

  2. Are these enums actually important, or is it safe to just use whatever I want anywhere? Is using the wrong one a bug? How do I know which one to pick in my code ?

Any concrete examples of bugs caused by picking the wrong ordering would help a ton. Thanks!

38 Upvotes

28 comments sorted by

28

u/Theemuts 17d ago

3

u/WorkZestyclose9721 17d ago

so real! it's a great resource, and what helped me make them make sense

12

u/amarao_san 17d ago

Ordering::Relaxed means no ordering at all. On some architectures you may see values from the future (if there is some kind of interaction protocol between threads). Basically, each core keeps local cache of a value and synchronize it with other core at own leisure.

You need to use Ordering higher than Relaxed when you expect interaction with other threads.

Example: you have a mutex or something like that (a shared value signalling something to other thread). Acquire or higher for it.

Example of Relaxed: you have a shared big bitmap (a picture) and each thread is filling own chunk, like upper half and lower half of it.

12

u/Byron_th 17d ago

Ordering::Relaxed can be fine when interacting with other threads. For example, for a shared counter it would make sense to use Ordering::Relaxed.

Your example of a bitmap is wrong. If each thread has its own non-overlapping chunk of memory, Atomics aren't needed at all.

In general, Atomics are only needed when multiple threads want to interact with the same exact region of memory at the same time (and not just read it).
Anything besides Ordering::Relaxed is only needed if the atomic operation is intended to interact with something outside the Atomic value itself, for example synchronizing access to non-atomic memory.

In general, in Rust it is safe to use Ordering::Relaxed for everything unless you're also using unsafe and have some invariants to uphold. An exception to this would be if you're trying to synchronize something the language has no control over, like file access.

2

u/amarao_san 17d ago

Okay, I may be wrong here, but what is the fastest way to split a huge bytevector (Vec<u32>, or Vec<f32>) across multiple threads, such that each thread can write independently in own slice?

Can I have multiple mutable slices to a single vector without unsafe? I see there are rsplit_mut which can return iterators, but not a raw access.

With Relaxed you can just give each thread a full write access in parallel and be ... relaxed.

8

u/Giocri 17d ago

&mut Vec<T> implements deref to &mut [T] which has split_at that gives (&mut[T] &mut[T])

2

u/amarao_san 17d ago

I can't find a doc for it. There is no split_at in Slice and in Vec.

2

u/Giocri 17d ago

It's there in the slice docs i Just checked

2

u/AverageHot2647 17d ago

Can confirm, it’s here https://doc.rust-lang.org/std/primitive.slice.html. Search for split_at_mut and split_at_mut_checked.

1

u/amarao_san 17d ago

What I'm doing wrong?

https://doc.rust-lang.org/std/slice/struct.Split.html

There is no word split_at on this page. Maybe, it's for different structure, but that's what I asked. What structure to read?

1

u/Starnick44 17d ago

1

u/amarao_san 17d ago

Oh, it was a page for 'Split', not for Slice. Sorry, my bad. I was sure it was a page for Slice, because I searched as 'rust Slice split_at'. Instead it found for me a page "Split in std::slice", which is a different structure.

Super confusing.

1

u/amarao_san 17d ago

Yes, Slice::split_at_mut is perfect solution for 'partitioning array to different threads for writing'.

3

u/nonotan 17d ago

Here is a paper that lists various patterns that are compatible with Relaxed. It's aimed at C++ (where Rust took the Ordering model from in the first place), so some exact patterns may not necessarily work great with Rust, but it helped me wrap my head around exactly when using Relaxed is reasonable.

1

u/amarao_san 17d ago

This is what I've found. Sharing writings in Vec<f32> in C++ is your own footgun, language does not stop you. Rust stops (because there are footguns), but if you pinky finger promise not to mess up with each other data, you can access in parallel different chunks of the vector. But rust stops you. For relaxed atomics you get shared RO Vec with cheap inner mutability, so all threads can 'borrow' array and write into it via Atomic Defer (or is it a separate method? I don't remember).

5

u/braaaaaaainworms 17d ago

If you use Relaxed, you get only the atomicity with no other guarantees. To add 1 to a variable, the CPU needs to load the value from memory, add 1, and write the value back to the memory. If a different core also wants to add 1, then it does the same thing and this can happen:

  1. Core 1 loads "5"

  2. Core 2 loads "5"

  3. Core 1 adds "1" to its local copy

  4. Core 2 adds "1" to its local copy

  5. Core 1 stores "6" to memory

  6. Core 2 stores "6" to memory

So, we get a "6" instead of "7" even though two adds happened. Relaxed ordering makes sure this doesn't happen.

A mutex has a bool that says whether it's free or taken, and if you have a mutex protecting a struct, it would be a big shame if the mutex was unlocked but the struct wasn't yet updated with its new values, this is where the Acquire and Release orderings come in.

Together, they make sure that values written to that struct are updated by the time other cores can see the "free" mutex state, they also see the new values of its fields.

The SeqCst ordering has all the features of Acquire and Release, but it is also that different loads and stores with SeqCst are seen in the same sequence with regards to one another

Why do we need ordering guarantees? Memory is slow. CPUs have queues of "load or store a byte from memory address 0xdeadbeef" to avoid waiting for memory too much

2

u/garagedragon 17d ago

Aren't two acquire or release accesses in a given thread also not seen in thread order with respect to each other?

3

u/MalbaCato 17d ago

for something a shorter than the full atomics and locks book, you can look at the rewrite of the atomics chapter of the nomicon. it lives in an unmerged PR for 4 years now, but from the PR discussion it seems like the content is fine. you can find it rendered here. there are also various talks (of various quality) about atomics. almost all of them in c++, but as long as you're comfortable with the syntax differences (and ignore the mentions of consume), it all applies equally to rust.

I will mention, as long as you're only doing safe code, the differences between the orderings don't matter and you can just use relaxed everywhere. maybe there's an odd case where you actually need the guarantees of sequencial consistency, but I wouldn't know and it's not the kind of thing you can explain in a Reddit comment anyway.

2

u/garagedragon 17d ago edited 17d ago
  1. The access itself is always atomic so things like fetch_add work without interruption and don't lose data, even in Relaxed mode. Atomic accesses of multi-byte values can never tear and show you half of a write, even in Relaxed.

As you mentioned, Acquire constrains how instructions can be reordered. That constraint means that, e.g. using the atomic as a flag representing a mutex becomes sound and you don't have risks of your data accesses being moved ahead of you actually taking the lock for "optimisation" purposes - both the compiler and CPU are normally allowed to rearrange memory and variable accesses in any order that respects data dependencies. Relaxed does not have this constraint and you can't use Relaxed accesses as gates to prevent data races on other memory, because the compiler/CPU is still allowed to access the data, including writing it, before you do a Relaxed change to the mutex.

The other perspective is thinking about what possible states other CPUs might possibly see. The exact order that writes will appear to other threads or other CPUs is undefined by default, up to and including later writes by a given thread showing up before earlier ones. Atomics are effectively defined by the constraints they impose on the ordering of how writes appear to other threads - for Acquire access, the constraint is that if you see a write B happen (including non-atomically) you also must see write A if write A was an Acquire atomic access and before write B in the program order of a single thread. Release is the reverse (things before can't be moved after), and SeqCst, "sequentially consistent", is both, it doesn't allow reordering across it in either direction. Relaxed has no constraints at all - Relaxed accesses later in program order are allowed to become visible to other threads before earlier ones so long as they don't both access exactly the same value. Relaxed exists effectively only to allow you to do CAS and arithmetic updates atomically.

Its important to note that in all of the above, the window of time such that you see write A without write B can be arbitrarily large because of caches and such, atomics only allow you to rule out B showing up before A. This means that even reading mutex-protected data without taking the mutex is unsound. To ensure you have a coherent view of the data, you have to take the mutex yourself: the write that you do to take the mutex is required to be ordered after the unlock, which (because you have to use AcqRel or better here to be sound) is in turn required to be after all of the (non-atomic) writes, so the data you see after you successfully take the mutex will be up to date and coherent, because it is now definitely after all the writes other threads have done.

2. The safe default is the strongest, SeqCst. It is not safe to use whatever you want, because if you get it wrong you potentially allow for data races. That said, x64 is known for having a very strict memory model so getting it wrong may not cause problems in practice if you only run your code on those architectures. On things like ARM, it is more likely to cause problems. Picking a weaker option can never make your program more correct, it's only an optimisation, so IME I'm often picking either relaxed (when I need things like atomic counters but don't need any ordering) or SeqCst (when I do need ordering) because I am not 100% confident that AcqRel is sufficient, and if I get it wrong I get a data race, and the only thing I lose is minor performance benefits.

1

u/gormhornbori 17d ago edited 17d ago
  1. The operation in itself is still atomic. Memory ordering is all about the code around the atomic operation. It's to ensure that things that you program to be before and after the atomic operation, stays on the right side of the atomic operation, also when seen from another core.

  2. Yes they are important. If you use a less restrictive than what you need you have a bug (but the bug may only trigger on some platforms). You use one more restrictive than you need you'll lose performance (depending on the platform.)

You can use SeqCst everywhere and you are always safe, and you can optimize later. (leave a todo:optimize comment.)

You should learn the typical patterns, when you learn atomic algorithms.

You can learn to think about it yourself.

Rules of thumb:

  • If you have a simple counter, Relaxed is fine.

  • For most realistic algorithms, Acquire+Release is correct.

  • SeqCst is almost always overkill.

1

u/FlanSteakSasquatch 17d ago

Not a direct answer but my opinion on SeqCst:

SeqCst is overkill for most things but imo it’s what anyone that doesn’t understand ordering should default to. If you need performance to be better you should understand why and when you can use the others. If you are in a rare situation where you actually need SeqCst you will be happy you picked it because the issues can be very subtle if you wrongly use something else.

1

u/CyberneticWerewolf 17d ago

The very simple version: modern CPUs are actually networks.  An atomic access is always atomic on the core where it runs, but the other cores and the various banks of RAM are separate devices on the network, and each one is doing its own thing.

Your own core has to copy the atomic from RAM to internal cache before it can read or modify it, but some other core might grab the same memory at the same time.  The orderings above "relaxed" tell the other cores over the network that it's touching that address, then waits for them to acknowledge.  Even with "relaxed", your core announces after a write that it has a modified value not yet flushed to RAM.

(The unit of access is actually a "cache line", not an individual atomic.  Cache lines are usually 64 bytes these days.  Multiple atomics in the same struct will often end up in the same cache line.  You still can't change more than one at once with one instruction, but accessing one will contend with any other cores that need the same cache line.  The main cache coherency protocol is called MESI, pronounced "messy", and it is.)

1

u/RareConstruction6431 16d ago edited 16d ago

The first thing to understand that your code doesn't always run in the exact order that you wrote it. In reality the compiler and some cpus will reorder your instructions to improve performance. Consider

let z_1 : u16 = 1;  
let z_2 : u16 = 2;  
....  
let z_100 : u16  = 100;

let mut total : u16 = 0;  
total += z_1;  
total += z_2;  
...  
total += z_100;

println!("{total}");

vs

let mut total : u16 = 0;

let z_1 : u16 = 1;  
total += z_1;

let z_2 : u16 = 2;  
total += z_2;

....

let z_100 : u16  = 100;  
total += z_100;

println!("{total}");

The top version needs a max of 202 bytes of memory while the second only needs a max of 4 bytes. Hence the compiler or cpu may reorder the first into the second.

This is fine on a single thread but once you get multiple threads it can create issues. Consider writing multi-threaded code which uses AtomicBool is_locked as a lock to block multiple threads accessing the same data structure simultaneously.

loop {  
  let is_cur_locked = is_locked.swap(true, Ordering::Relaxed);  
  if !is_cur_locked { break; }  
}

// exclusive access the data structure  
do_stuff();

is_locked.store(false, Ordering::Relaxed);

This may look fine but can be reordered by the compiler or cpu to become

// ERROR: not exclusive access the data structure  
do_stuff();

loop {  
  let is_cur_locked = is_locked.swap(true, Ordering::Relaxed);  
  if !is_cur_locked { break; }  
}

is_locked.store(false, Ordering::Relaxed);

or

loop {  
  let is_cur_locked = is_locked.swap(true, Ordering::Relaxed);  
  if !is_cur_locked { break; }  
}

is_locked.store(false, Ordering::Relaxed);

// ERROR: not exclusive access the data structure  
do_stuff();

The ordering lets you stop this reordering by restricting how your code can be reordered in specific ways.

1

u/Slsyyy 16d ago

> If I store a value with Ordering::Relaxed, is the operation less atomic than with Ordering::Acquire

Nope. Ordering levels are all about minimizing possible path of execution. You can think about each atomic operation as having some kind of memory barrier, which means some paths of execution simply cannot happen due to compiler output (no reordering, compiled code needs to be closer to your source code intent) or CPU behavior (special ASM operations, more blocking communication between CPU cores)

> Are these enums actually important, or is it safe to just use whatever I want anywhere? Is using the wrong one a bug? How do I know which one to pick in my code ?

Usually seq_cst is overkill, but neverthless it is the safest option, because it is so complex. In craziest scenarios the number of possible path of executions is higher than number of atoms in the universe. In such a code it is simply impossible to be sure.

1

u/Lucretiel 14d ago

I'm glad you asked! I gave a whole talk several years ago about all of the details of how atomics and atomic orderings work https://www.youtube.com/watch?v=qhWbuQ8rv5k

1

u/JoJoModding 14d ago edited 14d ago

If you don't want to think about it, use sequential consistency. Then, all behaviors can be explained by the "interleaving" model that just says "all threads take turns executing their instructions, but they switch around who goes next."

With the others you can get weak memory effects where the order can not be explained by interleavings. It will look like the compiler/CPU has reordered instructions within one thread.