Skip to content

Commit 757d0f9

Browse files
committed
perf: relax EpochLock memory ordering from seq_cst to acquire/release
The epoch counter is used to detect if meshing occurred during a lock-free operation. Sequential consistency is stronger than needed. The correctness argument: - Writers (meshing thread) use release ordering when incrementing the epoch, ensuring all prior/subsequent meshing work is properly ordered - Readers use acquire ordering, ensuring they see all meshing work that happened before the release store they observe - Additionally, mprotect() on source pages during meshing provides a hard barrier that catches any actual data races This is safe because: 1. There's only one writer (meshing thread holds all locks) 2. If a reader doesn't see an epoch change, either meshing hasn't started, or the reader's work is on unrelated data 3. If the reader's work would conflict with meshing, mprotect causes a fault that synchronizes the threads No performance change expected on x86 (already uses acquire/release for seq_cst atomics), but may improve performance on ARM/POWER.
1 parent e0124d4 commit 757d0f9

File tree

1 file changed

+12
-6
lines changed

1 file changed

+12
-6
lines changed

src/global_heap.h

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,26 +41,32 @@ class EpochLock {
4141
}
4242

4343
inline size_t ATTRIBUTE_ALWAYS_INLINE current() const noexcept {
44-
return _epoch.load(std::memory_order::memory_order_seq_cst);
44+
// Acquire ordering: if we read a value stored with release, we see all
45+
// operations that happened-before that store. This ensures readers see
46+
// all meshing work that completed before the epoch was updated.
47+
return _epoch.load(std::memory_order_acquire);
4548
}
4649

4750
inline size_t ATTRIBUTE_ALWAYS_INLINE isSame(size_t startEpoch) const noexcept {
4851
return current() == startEpoch;
4952
}
5053

5154
inline void ATTRIBUTE_ALWAYS_INLINE lock() noexcept {
52-
// make sure that the previous epoch was even
53-
const auto old = _epoch.fetch_add(1, std::memory_order::memory_order_seq_cst);
55+
// Release ordering: all subsequent meshing operations will be ordered
56+
// after this store. Readers with acquire loads will see this update.
57+
// The old value is only used for assertion, so relaxed read is fine.
58+
const auto old = _epoch.fetch_add(1, std::memory_order_release);
5459
hard_assert(old % 2 == 0);
5560
}
5661

5762
inline void ATTRIBUTE_ALWAYS_INLINE unlock() noexcept {
63+
// Release ordering: all prior meshing operations are ordered before this
64+
// store. Readers with acquire loads will see all meshing work completed.
5865
#ifndef NDEBUG
59-
// make sure that the previous epoch was odd
60-
const auto old = _epoch.fetch_add(1, std::memory_order::memory_order_seq_cst);
66+
const auto old = _epoch.fetch_add(1, std::memory_order_release);
6167
d_assert(old % 2 == 1);
6268
#else
63-
_epoch.fetch_add(1, std::memory_order::memory_order_seq_cst);
69+
_epoch.fetch_add(1, std::memory_order_release);
6470
#endif
6571
}
6672

0 commit comments

Comments
 (0)