axon

One daemon owns the accelerator. Every process on the machine is a client, local processes included. That is the whole design; the useful properties are consequences of it.

The device is always remote

The daemon creates the device context at boot from a reserved arena. Applications never create a driver context of their own. Because the local path and the network path are the same path, dlopen("libcuda.so.1") resolves to axon's shim and the work can run on another machine with the program unchanged.

Round trip over the local shared-memory ring is 1.9 µs, against 29 µs over TCP. Collapsing repeated boundary crossings into a single graph replay gives 4.6–6.3×.

axon	The equivalent it is modelled on
`Local` — answered without crossing	the vDSO: calls that do not deserve a trap
`Deferred` + graph capture	io_uring: amortise the crossing, not the work
`DevPtr::Minted`	virtual addresses, handed out without consulting the device
`Mirror`	the page cache
`Frontier`	a completion queue
`axon top`	`strace`

What follows from owning the boundary

Allocation stops costing a driver call. cuMemAlloc is 63 µs for a single 4 KiB page, per call rather than per byte, and the driver does no pooling. Suballocating from one reservation measured 3.8× cheaper to acquire and 44.8× cheaper to release for the same 4 GiB. Minted addresses make the substitution invisible to the application.

The fragmentation spiral has nowhere to live. A resident GPU process is what stops the next one starting: order-9 allocations went 0 → 248 → 0 across stopping and restarting one server. Application processes never create driver contexts, so the failure mode cannot occur.

Accounting is per-client and cannot be bypassed. One process sees every allocation and which client asked for it, which makes cgroup v2 accounting a consequence rather than a project.

Client checkpoint is nearly free, because clients hold no device state.

Why this has to be the distribution

Policy is real only when bypass is impossible. A library shim is advisory: any process can dlopen the vendor library and walk around the accounting, the limits and the arena. Making it mandatory is three packaging decisions, and only whoever builds the package set can make them.

The only libcuda.so.1 in the package set is axon's.
/dev/nvidia* is permissioned so that only the daemon may open it.
The arena is reserved on the kernel command line, before anything has fragmented memory.

A container cannot do any of it. A normal distribution cannot do the first two.

Full design: ARCH-accelerator.md, ARCH-axon.md