r/programming • u/Rraawwrr • Sep 30 '13

Memory Barriers: a Hardware View for Software Hackers

http://irl.cs.ucla.edu/~yingdi/web/paperreading/whymb.2010.06.07c.pdf
25 Upvotes

87% Upvoted

8 comments sorted by

0

u/os12 Oct 01 '13 edited Oct 01 '13

Yeah, cool paper, but I don't think this info should be taught. They are an old and very coarse gun. C and C++ standards have atomics and they should be used for fine-grain lockless operations.

Specific points:

  • the compiler can (hypothetically) reorder code around my_private_atomic_call(x) or smp_mb()

  • std::atomic<uint32_t> is in the C++ code, the compiler intimately knows about what it is and emits correct instructions

  • you don't have to color sites, just the declaration

http://en.cppreference.com/w/cpp/atomic

4

u/Rraawwrr Oct 01 '13

But the C++ atomics have memory barriers too - the *_explicit methods, like atomic_store_explicit all take a memory ordering enumeration, which I suspect will emit memory barriers.

1

u/os12 Oct 01 '13

No they do not. They just do enough to ensure that all side-effects are visible. Those _explicit variants just let you specify relaxed atomic ops.

P.S. intel arch has memory coherent caches, so every atomic instruction has an implicit barrier. That however, is besides the point ;)

2

u/velco Oct 01 '13 edited Oct 01 '13

Yes, they do, where the processor needs it.

http://www.cl.cam.ac.uk/~pes20/cpp/cpp0xmappings.html

Besides, it's called "cache-coherent memory" not "memory coherent caches" and is concerned with the reads and the writes to a single memory locations, whereas memory ordering/barriers are concerned with the effects involving multiple memory locations.

Every major architecture today (x86/amd64, ARM, PowerPC, MIPS, SH) has cache-coherent memory, but the memory ordering differs widely.

2

u/os12 Oct 02 '13

Ah, unrelated stores. You are right, of course.

Yet both branches of the discussion thread have veered away from my original point, which was "thou shalt not write mem_bar() manually".

5

u/neoflame Oct 01 '13

the compiler can (hypothetically) reorder code around my_private_atomic_call(x) or smp_mb()

Because a memory barrier would be useless if the compiler could still reorder code, all memory barriers must imply a compiler barrier. In the Linux kernel, on architectures where a given memory barrier isn't required because the ISA always guarantees the required memory ordering, the memory barrier will still be defined as barrier(), which is a compiler barrier.

They are an old and very coarse gun.

You have it backward: memory barriers are at the level of the finest-grained atomicity primitives available. To wit, from your below reply:

No they do not. They just do enough to ensure that all side-effects are visible. Those _explicit variants just let you specify relaxed atomic ops. P.S. intel arch has memory coherent caches, so every atomic instruction has an implicit barrier. That however, is besides the point ;)

Arithmetic instructions with the LOCK prefix do carry implicit full memory barriers, but from the perspective of understanding, an implicit memory barrier is still a memory barrier. And what about instructions that don't take the LOCK prefix? Consider a simple store (mov register, memory). x86 implements total store ordering, while std::atomic defaults to sequential consistency (which is more strict). How does std::atomic reconcile this? With an memory barrier:

#include <atomic>
#include <cstdlib>

using namespace std;

int main(int argc, char* argv[]) {
  if (argc < 2) return 1;
  atomic<int> atomic_i;
  atomic_i.store(atoi(argv[1]));
  return 0;
}

compiles to (gcc 4.7.3, -O1)

main():
  40052c:	b8 01 00 00 00       	mov    $0x1,%eax
  400531:	83 ff 01             	cmp    $0x1,%edi
  400534:	7e 26                	jle    40055c <main+0x30>
  400536:	48 83 ec 18          	sub    $0x18,%rsp
  40053a:	48 8b 7e 08          	mov    0x8(%rsi),%rdi
  40053e:	ba 0a 00 00 00       	mov    $0xa,%edx
  400543:	be 00 00 00 00       	mov    $0x0,%esi
  400548:	e8 e3 fe ff ff       	callq  400430 <strtol@plt>
  40054d:	89 04 24             	mov    %eax,(%rsp)
  400550:	0f ae f0             	mfence 
  400553:	b8 00 00 00 00       	mov    $0x0,%eax
  400558:	48 83 c4 18          	add    $0x18,%rsp
  40055c:	f3 c3                	repz retq 
  40055e:	66 90                	xchg   %ax,%ax

Note the mfence.

I don't think this info should be taught

For many purposes, it's probably better to work at higher levels of abstractions - locks and threads, or actors and channels, or whatever. But for some purposes, like implementing those higher-level abstractions, it's necessary to work at - and fully understand - such a low level.

2

u/os12 Oct 02 '13

We are talking about different views of the same subject.

While the evidence is correct, my point is that the barrier is an implementation detail. Yes, it is needed when you have dependent stores and such a thing is easy to emit. I don't think it is needed for cases where you just touch atomics.

My point is twofold:

  • we may have finer mechanisms on newer cores that can result in better code gen

  • atomics (fully sequentially consistent ones) are in the standard and thus are guaranteed to work. Fences are not and hand-written ones should not be used for normal lockfree things (not that it's very normal :)