Debugging Race Conditions with ThreadSanitizer: A Practical Walkthrough

Some bugs appear every time you run the test. A race condition appears on the third Tuesday of the month, on the build server, in a function nobody has touched since 2021. You add a log line and it vanishes. You remove the log line and it comes back.

That is the nature of the problem: a data race depends on timing, so anything that changes timing — a log line, a debugger, an extra core under load — can hide it. ThreadSanitizer, usually shortened to TSan, sidesteps this by instrumenting every memory access and checking whether the accesses were ordered by a synchronisation operation. It does not need the race to hit at the worst possible moment. It only needs both threads to touch the same location while you are watching.

What ThreadSanitizer actually finds

TSan ships with clang and gcc and works on Linux, macOS and BSD. Give it a binary and it reports, among other things:

• Data races: two threads accessing the same memory location, at least one of them writing, with no happens-before relationship between them — no mutex, no atomic, no join.
• Mutex misuse: destroying a mutex while it is locked, unlocking one thread did not lock, and lock-order inversions that could deadlock.
• Thread leaks: threads still running when the program is about to exit.

The important limitation is that TSan is a dynamic tool. It only reports code that actually ran, so a race on an error path your tests never take will stay invisible. That is a reason to point it at realistic workloads, not just a handful of unit tests.

Building with ThreadSanitizer

Compile and link with the same flag. Using the compiler driver for both steps is the simplest approach:

clang++ -fsanitize=thread -g -O1 -pthread racy.cpp -o racy

Include -g or your reports will show raw addresses instead of variable names and line numbers. -O1 is the usual sweet spot: fast enough to run a real workload, but without the aggressive inlining that makes stack traces harder to read. In CMake, add the flag to your compile options and your link options; on a modern CMake that means target_compile_options and target_link_options on the target under test.

A few things to keep in mind:

• TSan cannot be combined with AddressSanitizer or MemorySanitizer in the same binary. Build separate targets.
• Do not run a TSan build under Valgrind. Pick one tool at a time.
• Instrument the libraries you can. A race inside an uninstrumented shared library is invisible to the tool.
• Expect a slower program and a noticeably larger memory footprint. Plan your test sizes accordingly.

A deliberately racy example

Here is a small program with two threads, each incrementing a shared integer one hundred thousand times. The counter is a plain static int, the worker loops with ++counter, and main starts two std::thread objects, joins both and prints the result.

Nothing there looks suspicious, which is the point. ++counter is not one operation. It reads the value, adds one, and writes it back. If both threads read the same value before either writes, one increment is lost. You expect 200,000. You might see 174,318 on one run and 199,004 on the next. Occasionally you get the correct answer, which is exactly why the bug survives code review.

Compile that file with the flags above and run it. Alongside the wrong total, TSan prints a warning.

Reading the race report

The report looks intimidating the first time. It has a fixed shape, and once you know the parts, it reads quickly.

• WARNING: ThreadSanitizer: data race
• Write of size 4 at 0x7b0400000000 by thread T2: #0 worker() racy.cpp:4
• Previous read of size 4 at 0x7b0400000000 by thread T1: #0 worker() racy.cpp:4
• Location is global 'counter' of size 4 at 0x7b0400000000
• Thread T2 created by main thread at: #0 std::thread::start_thread
• Summary: ThreadSanitizer: data race racy.cpp:4 in worker()

Start with the location. Naming the global variable tells you which piece of state is unprotected, and for heap memory TSan prints the allocation stack instead, so you can see who created the object.

The two stacks are the next thing to read. TSan reports a pair because the race is the relationship between them, not either access on its own. The access listed first is simply the one it observed second; do not read too much into the ordering. Then look at the thread creation stack. In a small program it is uninteresting, but in a server with a thread pool it tells you which piece of work queued the task, which is often the difference between a ten-minute fix and an afternoon of guessing.

If the report mentions a mutex — lines such as Mutex M0 acquired here — pay attention. TSan is telling you that one side held a lock and the other did not, which usually points straight at the missing lock_guard.

Fixing the race

There are three sensible repairs for the counter, and the right one depends on how hot the loop is.

1. Protect it with a mutex. Add a std::mutex and take a std::lock_guard around the increment. Correct and obvious, though you are now locking 200,000 times for a single addition.
2. Make it atomic. Change the declaration to std::atomic<int> counter{0};. The same ++counter line becomes an atomic read-modify-write and no lock is needed. The default memory_order_seq_cst is fine here; for a plain tally, fetch_add(1, std::memory_order_relaxed) is enough.
3. Give each thread its own counter and combine the totals after joining. Best when contention would otherwise dominate the loop.

Rebuild and run again. Silence from TSan is real evidence, provided the workload exercised the same paths.

One warning worth repeating: making a variable atomic fixes the data race, but it does not always fix the program. If the shared state is two fields that must change together — a pointer and a length, a map and a flag — an atomic on one of them leaves you with torn state and a bug that is harder to find than the one you started with. Use the report to ask what invariant the two accesses were supposed to maintain, then enforce that invariant.

Fitting TSan into your workflow

A few habits make the tool far more useful than running it once:

• Run the workload that fails, not just the unit tests. TSan sees executed code only.
• Set TSAN_OPTIONS=halt_on_error=0 to collect every race from one run instead of stopping at the first.
• Keep TSan runs short and concurrent. Slow suites get skipped, and skipped suites find nothing.
• Rebuild your own libraries with the flag, and suppress third-party noise with a suppressions file rather than ignoring reports wholesale.
• Put a TSan job in CI, even if it runs nightly on a subset of tests.

Start with the flaky test

Do not try to sanitise an entire codebase in one go. Pick the test that fails once a fortnight. Build just that target with -fsanitize=thread -g -O1, run it under the same conditions that trigger the failure, and read the report from the variable name down through the two stacks. Fix one race, confirm it is gone, and leave the TSan target in place so the next one is caught before it reaches production. The first report usually comes as a surprise. The tenth is usually obvious, and that is the whole point.

Photo: https://kaboompics.com/ / Pexels