Your bug fails once in a while and never when you watch it, so you need a plan, not luck. Here is how to debug intermittent bugs: make the failure repeatable, then use tools that catch races.
This guide belongs to the wider set of debugging techniques on this site, and it focuses on failures that only show up some of the time.
| Item | Value |
|---|---|
| Typical ThreadSanitizer slowdown (LLVM) | 5x-15x |
| Typical ThreadSanitizer memory overhead (LLVM) | 5x-10x |
| Optimisation level for reasonable performance | -O1 or higher |
| Accesses remembered per memory location (OCaml TSan) | 4 accesses |
| Observed slowdown range (OCaml TSan) | 2x to 7x |
These figures come from the LLVM ThreadSanitizer documentation and the OCaml manual. They can change, so check the source before you rely on them.
Why intermittent bugs hide from you
An intermittent bug depends on something you do not control: timing, ordering, input or environment. When the trigger is missing, the code passes, which is why the failure feels random. Treat that randomness as a clue.
Common causes include:
- Timing and thread interleaving between concurrent tasks
- Shared state written without any ordering between writers
- Inputs that vary, such as random seeds, clocks or locale
- Environment differences, such as machine load, files, network or configuration
Each cause leaves a different fingerprint. A failure that tracks machine load suggests timing, while a failure that follows one particular input suggests data. A failure that appears on only one machine points at the environment, so note which pattern fits before you change any code.
It also helps to separate the symptom from the trigger. The symptom is what you see, such as a crash, a hang or a wrong value. The trigger is the condition that must hold for that symptom to appear, and finding it is the real task.
Ask these questions about every failure you record:
- Did it fail on a fast machine, a slow machine or both?
- Was other work running at the same time, such as a build, a backup or another test?
- Did the failing run use different data, a different order of steps or a different configuration?
- Has anything been updated since the last run that passed, such as a dependency, a setting or the operating system?
Resist the urge to patch the symptom. Adding a retry, a longer wait or an extra null check can make the failure rarer without removing it, and a rarer bug is harder to catch later. Keep the failing behaviour visible until you understand why it happens, and write each guess down so you can rule it out on purpose rather than by accident.
Concurrency is the usual suspect. The OCaml manual defines a data race as unordered accesses to the same memory location with at least one of them being a write.
Make the failure repeatable first
Your first goal is to turn “sometimes” into “often”. A bug you can trigger on demand is already half solved.
- Write down every failure: what ran, which inputs, which machine and the exact error text.
- Run the failing case in a loop and count how often it fails.
- Pin what you can: seed random generators, fix the clock and use the same data.
- Change one variable at a time, such as thread count or load, and watch whether the failure rate moves.
- Keep the logs from failing runs and compare them with passing runs.
Once the failure rate rises under a specific change, you have found a lever. That lever usually points at the cause.
How can a race detector help you?
If you suspect threads, a dynamic race detector can find the problem without waiting for a visible failure. Firefox’s documentation describes Thread Sanitizer (TSan) as a fast data race detector for C/C++ and Rust programs. OCaml offers its own TSan support, so check whether your language has one.
For C and C++ with Clang, the LLVM documentation gives this recipe:
- Compile and link with -fsanitize=thread.
- Add -O1 or higher to get reasonable performance.
- Add -g so warnings show file names and line numbers.
Expect a cost. The same LLVM page puts the typical slowdown at 5x-15x and the typical memory overhead at 5x-10x. Run it in a test or stress job, not in your everyday build.
What a race report tells you, and what it misses
For each detected race, TSan reports the location of the conflicting accesses, their nature (read, write and so on) and the associated stack trace, according to the OCaml manual. Start with the two stack traces and ask which lock or ordering should have connected them.
A detector is only as good as the run you give it. The same manual says TSan reports only the data races encountered during execution. Its own example shows the output varying between runs: sometimes two races, sometimes one and sometimes none.
That is why a clean run proves little. The OCaml manual also notes that TSan remembers a finite number of accesses per location, and that at the time of writing this number is 4. Races involving a forgotten access can go undetected.
Two practical details from the LLVM documentation are worth knowing:
- TSan generally requires all code to be compiled with -fsanitize=thread. Precompiled libraries can cause missed races or false reports.
- Disabling ASLR may stop TSan from allocating shadow memory, and GDB disables ASLR by default, so you must configure GDB to preserve it.
Keep the evidence without changing the timing
Observation can change the bug. Pausing a thread at a stop point can alter the timing that triggers the failure, so read how to use breakpoints and consider conditional or logging-only stops for timing-sensitive code.
Logging works better, but keep it light. The OCaml manual shows that data races can sometimes be hidden by unrelated synchronizing operations, so a heavy lock or flush in a log call may make the bug vanish. Record a timestamp, a thread or task identifier and the key values at each step.
Not every sometimes-failure is a race. If the failure appears only after the process has run for a long time and ends in running out of memory, the cause is likely elsewhere, and how to debug a memory leak covers that path.
When you have a candidate cause, check it in this order:
- Write a test or loop that fails reliably because of that cause.
- Apply the fix.
- Rerun the same loop many more times than the failure rate you measured earlier.
- Rerun it under the race detector if threads are involved.
Frequently asked questions
Why does the bug disappear when I add logging or a debugger?
Show answer
Both can change timing. The OCaml manual notes that data races can sometimes be hidden by unrelated synchronizing operations, so extra work in the code path may hide the failure without fixing it.
Does a clean ThreadSanitizer run prove there are no races?
Show answer
No. TSan reports only the races encountered during that execution, according to the OCaml manual, so exercise the code with varied and repeated workloads.
Can I leave ThreadSanitizer on in production?
Show answer
The LLVM documentation says the runtime is not meant to be linked against production executables. Use it in testing instead.
How many times should I rerun a test to trust a fix?
Show answer
The sources give no fixed number. Measure how often the bug failed before the fix, then rerun well beyond that rate so a silent pass becomes unlikely.
