Can We Know Whether a Profiler Is Accurate?

Trace data are sent through a large/fast port (PCIe or 60-pin connector) and captured using fast dedicated hardware at something like 10 GB per second. The trace data are usually compressed and often only need to indicate whether a branch is taken or not taken (TNT packets from x86, Arm has ETM but similar enough trace path) with a little bit of timing, exception/interrupt, and address overhead. The bottleneck is streaming and storing trace data from a hardware debugger (since its internal buffer is usually under half a second at max throughput) although you can further filter by application on Intel processors via CR3 matching. (Regarding the last five years of Apple: I'm not sure you'll find any info on Apple's debuggers and modifications to the Arm architecture. Ever.)

If you encounter a slowdown using RTIT or IPT (the old and new names for hardware trace) it's usually a single-digit percentage. (The sources here are Intel's vague documentation claims plus anecdotes; Magic Trace, Hagen Paul Pfeifer, Andi Kleen, Prelude Research.)

Decoding happens later and is significantly slower, and this is where the article's focus, JIT compilation, might be problematic using hardware trace (as instruction data might change/disappear, plus mapping machine code output to each Java instruction can be tricky).

It's not an assumption, this is based on claims made by CPU manufactures. It's possible to get it down to within 1-2% overhead.

Intuitively this works because the hardware can just spend some extra area to stream the info off on the side of the datapath -- it doesn't need to be in the critical path.

My experience is the same.

Isn’t not that they’re “not perfectly accurate”, it’s that you can find half an order of magnitude of performance after the profiler tells you everything is fine.

That’s perfectly inaccurate.

Most of the people who seem to know how to actually tune code are in gaming, and in engine design in particular. And the fact that they don’t spend all day every day telling us how silly the rest of us are is either a testament to politeness or a shame. I can’t decide which.

They don’t count cache eviction, memory pressure, or GC time being attributed to consumers of data instead of producers. Or sometimes consistently to barely involved function calls. When a task has a certain shape and occupies more than half of the heap size in the middle, it’s often the case that the cleanup code or teardown code gets blamed for the cost. The only real performance improvement available is to go stop the mess at the source.

Last time I checked, Intel's MSRs (https://en.wikipedia.org/wiki/Model-specific_register) allow Intel PCM (https://github.com/intel/pcm) to work, are indeed used to profile, or "measure performance" (sorry if my vocabulary is not the most accurate). Last time I checked the code of Intel PCM, it still relies on hardcoded values for each CPU which are as close as possible to reality but are still an estimation.

It doesn't mean that you get wrong measurements, it means there's a level of inaccuracy that has to be accepted.

BTW, I am aware that Intel PCM is not a profiler, and more of a measurement tool, however you CAN you use it to 'profile' your program and see how it behaves in terms of computing and memory utilization (with deep analysis of cache behavior (cache hit, cache miss, etc.))

That's part of it, but it's not even the plurality.

Getting a non-sampling profiler is pretty expensive. Nobody has worked out how to create one that audits cause and effect. That's up to you to suss out on your own.

There's a fiction about how computation works that has been built into processors for more than thirty years. And when that fiction breaks down the code doesn't just get slower. It gets 10 times, 100 times, 1000 times slower.

Profilers mostly play into that fiction, and the gap between the fiction and reality has only grown bigger over time. See also the 'memory access is O(n^1/3) ' thread a few weeks ago for hints.

It’s also a big factor of “the fastest code is deleted code”. Someone sneaks in a loop or a second call to a function and disrupts the code flow and you can see some really weird shit.

One of my first big counter examples to standard “low hanging fruit” philosophy was a profiler telling me a function was called 2x as often as the sequence diagram implied and occupying 10% of the overall cpu time for the operation. So I removed half of the calls by inverting a couple of calls to expose the data, which should have been a 5% gain (0.1 / 2). Total time reduction: more than 20%. Couple jobs later I managed a 10x improvement on one page transition from a similar change with an intersection test between two lists. Part of that was algorithmic but most was memory pressure.

Remember if microbenchmarks lie, a profiler is just an inversion of the benchmark idea. Two sides of the same coin.