mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
10573afd0a
Answers "what is burning the CPU?" for a process you cannot attach a debugger to.
HINDSIGHT_API_PROFILE holds JSON -- the shape config.py already uses for structured env
config -- and unset, nothing starts:
HINDSIGHT_API_PROFILE='{"every": 60, "top": 20}'
It reports to the log stream rather than a file or an endpoint, because the case it
exists for is a process dying without explanation: a file inside the container dies with
the container unless a volume was mounted in advance, and an endpoint needs a live
process and a route to it. Container runtimes keep the previous container's stdout, so
the last report before a crash is still readable afterwards. Each report is flushed as it
is written, since a fatal signal takes buffered output with it.
Three things the implementation had to work around, all verified on a running API:
* The profiler is process-wide and single-instance. Since 3.12 it is a global
monitoring tool, so enable() covers every thread whatever thread calls it, and a
second concurrent profiler raises `tool 2 is already in use`. Per-thread profilers
are not possible; this arms one for the process.
* Snapshots must use getstats(), which reads the accumulated entries without stopping
the profiler. Snapshot-and-clear through pstats disables the global tool, and every
report after the first then silently contains nothing. Reports are deltas between
snapshots, and a test asserts a second window still has data.
* Sampling via sys._current_frames() is not an alternative on a free-threaded build: it
stops the world, so it catches threads parked at safe points, which are I/O waits. It
reported event-loop threads idle in selectors.select while /proc showed those same
threads at 50-65% of a core. Every report therefore carries per-thread CPU from
/proc, which profiler overhead cannot distort, as the arbiter.
py-spy remains the better tool on a GIL build. It cannot read a Py_GIL_DISABLED process
at all -- it locates threads through the GIL -- which is what left free-threaded
deployments with nothing, and is why this exists.
Website
This website is built using Docusaurus, a modern static website generator.
Installation
npm install
Local Development
npm start
This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
Build
npm run build
This command generates static content into the build directory and can be served using any static contents hosting service.
Deployment
Using SSH:
USE_SSH=true npm run deploy
Not using SSH:
GIT_USER=<Your GitHub username> npm run deploy
If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the gh-pages branch.