Performance8 min read

The widest block in a cold flame graph is the autoloader

The same endpoint profiled twice - 65.8 ms and 8.4 ms. What changed in the picture, and how to tell one slow call from ten thousand fast ones.

Alden PikeBackend engineer and technical writer

Six requests to the same endpoint, in one process, in order:

req1 200 0.065798s
req2 200 0.008755s
req3 200 0.008282s
req4 200 0.008233s
req5 200 0.008585s
req6 200 0.008538s

The flame graph of the first one and the flame graph of the rest are pictures of different programs. The code did not change between them.

How these were produced

PHP 8.5.10, NTS, arm64, Homebrew build, on an Apple M4 Pro laptop running macOS and not quiesced. The application is a stock symfony/skeleton at 8.1 with one controller added, running in prod with its container already compiled to disk, served by the built-in web server with OPcache enabled. That is not PHP-FPM, and the cold/warm distinction below is about OPcache rather than about a pool.

Sampling is the userland profiler this series built in the previous post, at a 500 microsecond interval, recording the full stack on every signal. Timings are curl’s time_total for single requests rather than medians, because the point of the first row is that it happens once.

One setting had to change to make this work at all, and it is the reason this technique is a demonstration rather than a tool:

Uncaught FFI\Exception: FFI API is restricted by "ffi.enable" configuration directive

ffi.enable defaults to allowing FFI only in the CLI, so the sampler is refused under a web SAPI until it is set to true. That is a setting nobody should turn on in production, and it is why real sampling profilers are compiled extensions or read the process from outside.

What a folded stack is

Before the picture, the data. On every signal the profiler walks the stack from the outermost frame inward and joins the names with semicolons:

Kernel::handle;HttpKernel::handle;HttpKernel::handleRaw;ReportController::index;gzcompress 22
Kernel::handle;HttpKernel::handle;HttpKernel::handleRaw;ReportController::index 18
Composer\Autoload\ClassLoader::loadClass;include 2

One line per distinct stack, with the number of times it was seen. That is the whole input format a flame graph is drawn from, and it is worth knowing because it is greppable, diffable and countable in ways the picture is not.

The picture is a bar chart of that file, one row per stack depth, each block as wide as its share of the samples, with children drawn above their parent — the format Brendan Gregg defined and named.

Producing that file from the sampler is a handler that walks the backtrace outward-in and one function that prints the tally:

pcntl_signal(SIGPROF, static function (): void {
    $stack = [];
    foreach (array_reverse(debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS)) as $frame) {
        $fn = $frame['function'] ?? '';
        if ($fn === '' || str_starts_with($fn, '{closure')) { continue; }
        $stack[] = isset($frame['class']) ? $frame['class'] . '::' . $fn : $fn;
    }
    $key = implode(';', $stack ?: ['main']);
    $GLOBALS['__stacks'][$key] = ($GLOBALS['__stacks'][$key] ?? 0) + 1;
});

array_reverse is the whole difference between a flame graph and an upside-down one: debug_backtrace() returns the innermost frame first, and the format wants the outermost first so that shared prefixes merge.

The cold request

Seventy-one samples, drawn at 72 columns:

0 [ClassLoader::l][][Kernel::handle                                  ][req]
1 [include     ][][HttpKernel::handle][][Kern][Kernel::preBoot    ][Com]
2 [Clas][cl][][HttpKernel::handle][][Cont][Kernel::ini][Ke][Ap][][][]
3 [incl][Cl][][ReportCont][Event][][][Cont][Kernel::ini][Ke][Ap]
4 [Cl][i][][][gz][Event][][][App_][Kernel::doI][Ke][Co]
5 [in][][][][][A][][][][][Cl][][][require    ][in][][][]
inclusive (function anywhere in the stack)
  Composer\Autoload\ClassLoader::loadClass          52   73.2%
  include                                           51   71.8%
  Symfony\...\Kernel::handle                        49   69.0%
  class_exists                                      22   31.0%
  Symfony\...\Kernel::preBoot                       21   29.6%
  Symfony\...\HttpKernel::handle                    20   28.2%
  App\Controller\ReportController::index             8   11.3%

Nearly three quarters of the request is the autoloader, and the controller — the only code in this application anybody wrote — is 11.3%. Those samples are inclusive, so what they are really counting is the compilation happening underneath the autoloader rather than the autoloader’s own probing. A reader who arrives at this graph looking for their bottleneck finds a framework and concludes the framework is the bottleneck.

It is not. It is a cost paid once.

The warm request, which you cannot read

The seventh row of the timings is where the real lesson starts. Profiling request four on its own:

0 [Kernel::handle                                                        ]
1 [HttpKernel::handle                                                    ]
2 [HttpKernel::handleRaw                                                 ]
3 [ReportController::index                                   ][EventDispa]
4 [ClassLoade][gzcompress                        ][EventDispa]
5 [ClassLoade][ClassLoade]
  total samples: 6

Six samples. Every percentage that can be computed from six samples moves by 17 points if one of them lands somewhere else, so this graph supports no claim at all beyond “the controller was running”.

That is not a defect of the profiler; it is arithmetic. An 8.4 millisecond request at a 500 microsecond interval can hold about seventeen samples, and signal coalescing delivered six. The response is to aggregate, which is exactly what a continuous profiler in production does — many requests, one picture.

Requests two through six added together:

0 [Kernel::handle                                                        ]
1 [HttpKernel::handle                                                  ][]
2 [HttpKernel::handleRaw                                               ][]
3 [ReportController::index                                       ][Even][]
4 [][gzcompress                       ][][Even][]
5 [][C][][][]
  total samples: 45
inclusive (function anywhere in the stack)
  Symfony\...\Kernel::handle                        45  100.0%
  Symfony\...\HttpKernel::handleRaw                 44   97.8%
  App\Controller\ReportController::index            40   88.9%
  gzcompress                                        22   48.9%
  Composer\Autoload\ClassLoader::loadClass           5   11.1%

The same application, the same endpoint, and the inverse picture: the controller is 88.9% and the autoloader is 11.1% of a request that is itself eight times shorter. The framework did not get faster. Its cost was in the first request, and the compiled container is most of what made that possible.

The third picture: a deploy

There is a colder state than the one above. The runs so far had the compiled container already on disk; the first request after a deploy does not. Profiling that state — a CLI run against an empty cache directory — puts something else at the top:

...dumpContainer;Filesystem::dumpFile;Filesystem::tempnam;Filesystem::box;tempnam            12
...dumpContainer;Filesystem::dumpFile;Filesystem::rename;Filesystem::box;rename              10
...dumpContainer;Filesystem::dumpFile;Filesystem::box;file_put_contents                       7
...dumpContainer;ClassLoader::loadClass;include                                               3
...dumpContainer;Filesystem::dumpFile;file_exists                                             3

Forty-eight of that run’s 186 samples are inside dumpContainer — 25.8% of the request spent writing the compiled container to disk, with tempnam, rename and file_put_contents as its leaves. Autoloading is 53.2% on top of that. gzcompress, the thing the endpoint exists to do, is 5 samples, and the controller as a whole is 13.

A graph like that is not about the request at all. It is a picture of a build step that happened to be triggered by a request, which is what a cache warmer exists to prevent, and quoting it as “our endpoint takes 350 ms” is the most expensive misreading on this list.

Three facts that stop the common misreadings

Width is sample count, not slowness. A block twice as wide was on the stack twice as often. Whether that is one call taking twice as long or a thousand calls each taking the same time is not in the picture.

Depth is the call stack, not a timeline. Row three is “things called by whatever is in row two”. A deep graph is a deep call chain; a tall thin spike is a recursion or a long chain of thin wrappers, not a long-running anything.

Neither axis is chronological. Blocks are ordered so that identical stacks merge, which is the entire point of the format — that is why 22 samples of gzcompress become one block instead of 22 scattered ones. If you need to know what happened when, a flame graph is the wrong artifact and a trace is the right one.

One slow call against ten thousand fast ones

This is the reading skill worth practising, and the folded file answers it where the picture does not. Count the distinct stacks a function appears in:

Distinct stacks containing it Samples
Cold: loadClass 36 of 46 52
Cold: gzcompress 1 of 46 4
Warm: gzcompress 1 of 8 22
Warm: loadClass 4 of 8 5

loadClass in the cold profile is called from thirty-six different places. In the picture that is thirty-six narrow slivers scattered across every part of the graph, none of them wide enough to read, and the total only becomes visible when something adds them up — which the inclusive table does and the drawing does not.

gzcompress is the opposite: one call site, one block, 48.9% of the warm profile in a single wide bar.

So the shape tells you which kind of problem you have. A wide contiguous block is one expensive thing and you can go and look at it. A function that is everywhere and nowhere is a per-call cost multiplied by a call count, and the fix is usually to make the count smaller rather than the call faster.

What is missing from every graph here

One more thing decides what a profile can contain, and it is the clock the sampler counts against. ITIMER_PROF counts CPU time, so a process that is waiting is not sampled at all. A request that sleeps for 200 milliseconds and then hashes for 45:

wall 244.8 ms, 8 samples
  renderRows;md5                           6
  renderRows                               1

Two hundred milliseconds — 82% of the request — produced no samples and appears nowhere. The function that did the waiting is not in the graph, not as a thin block, not at all.

Substitute a database query, an HTTP call to another service or a file read for the sleep and the shape is the same. A CPU-time flame graph of a request that is mostly waiting is a picture of the 18% that was not, drawn at full width, and it will confidently rank the hashing as the whole problem.

Wall-clock sampling sees the wait, at the cost of sampling every idle process too. Which one a given tool uses is worth knowing before trusting a graph that looks suspiciously CPU-bound.

Reading the bottom rows

The lowest rows of a PHP flame graph are the request lifecycle, and they are the same on every graph from the same application: the front controller, the kernel, the HTTP kernel, then the routing and controller resolution. A block down there being wide means everything above it is wide, which is why row zero is always 100% and never interesting.

The row worth looking at first is the deepest one that still spans most of the width. In the warm profile that is ReportController::index at 88.9% — the narrowest frame that still owns the request, and the place where the application stops being framework and starts being yours.

What to do with a graph you were handed

Ask whether it is a cold request before anything else. If the autoloader, the container, or anything with Compiler in its name is near the top of the inclusive list, you are looking at a picture of a process starting up, and it tells you about deploys rather than about traffic.

Ask how many samples it contains. A profile of one request is not a measurement; the number to look for is in the hundreds, and if the tool does not tell you, that is worth knowing about the tool.

Then read the inclusive list and the picture together. The picture shows you where a single expensive thing is; the list is what finds the cost that has been smeared across thirty-six call sites, and the difference between those two findings is the difference between a fix and an afternoon.

Frequently asked

My flame graph is mostly framework internals. Is the framework the problem?

Check whether you profiled a cold request first. On the first request through this Symfony skeleton the autoloader accounted for 73.2% of samples; on warm requests it was 11.1% of a request eight times shorter. Same code, same endpoint, opposite pictures.

Does a wide block mean a slow function?

It means the function was on the stack for many samples, which can be one slow call or a great many fast ones. The way to tell them apart is stack diversity - a function called from everywhere appears in many narrow slivers rather than one wide block.

Why does my flame graph of one request look like noise?

Because it probably is. An 8.4 ms request sampled every 500 microseconds yields about six samples, and six samples cannot rank anything. Aggregate across requests, which is what a continuous profiler does for you.

Is the left-to-right order the order things happened?

No. Frames are ordered alphabetically or by whatever the tool chose so that identical stacks merge into one block. A flame graph has no time axis; if you need one, that is a trace, not a profile.

Written by

Alden Pike

Alden Pike writes about PHP internals, performance, architecture and production behavior — the layer beneath the frameworks. Measurements over assumptions, trade-offs over universal rules.

More about the author

Arrow keys to move, Enter to open.