Performance follow-up evidence

Performance index · Benchmark guide · Reproduction playbook

Verdict and scope

Verdict: ready with minor improvements.

The confirmed search-latency concern and the workbar-worker lifecycle concern from the previous audit are fixed. This report records the follow-up measurements at the current revision; the 2026-08-03 audit remains unchanged historical evidence.

Measurement context

Item	Value
Rozi base	2b85924b1ecf83958ae0d5427b053920da6cf0ef; clean before final audit worktree changes
tui-lipan path dependency	includes f951197 allocation-free text, 7f30677 app metrics, 5dc6dd0 visibility, and d410398 controlled palette selection
OS	Arch Linux, kernel 7.1.4
CPU	AMD Ryzen 7 5700X3D, 8 cores / 16 threads
Rust	rustc 1.93.0
Criterion plotting	gnuplot unavailable; Plotters used

The dependency is still the sibling ../tui-lipan/ checkout. Standalone CI remains gated on publishing the required framework version and replacing the path dependency; no registry version is active.

Phase 1 used detached temporary worktrees with the application fixed at c07b6be and changed only tui-lipan from 934d7b1 to f951197; separate CARGO_TARGET_DIRs isolated both builds. Later saved before baselines were captured after that framework fix but before application search optimization and remain the comparison point for the search phases. The final saturation, resurrection, and descendant-cleanup evidence exercised the current uncommitted audit worktree rooted at the Rozi base above.

Confirmed measured outcomes

Phase 1 snapshot rebuild

Screen	Before (934d7b1)	After (f951197)	Improvement
80x24	64.074 µs	39.631 µs	38.1%
200x60	374.72 µs	229.10 µs	38.9%
320x90	879.19 µs	539.27 µs	38.7%

This is a genuine same-application, same-machine framework-only comparison and directly attributes the roughly 38-39% reduction to tui-lipan's allocation-free cell text append. It is consistent with the 2026-08-03 audit's 377.64 µs at 200x60 and 892.11 µs at 320x90, while controlling the revision more strictly. It does not change the earlier conclusion that lazy, once-per-frame snapshot rebuilding removes most burst cost.

The later saved before run measured snapshots at 39.226 µs, 230.89 µs, and 536.07 µs for the same three dimensions, plus app render at 219.41 µs for 8 panes and 380.29 µs for 16 panes. That run is post-Phase-1 context and supplied the pre-search-optimization baseline below.

Scrollback search

The pre-search-optimization 16-pane full scans were 35.604 ms sparse, 49.320 ms dense, and 34.637 ms with no match.

Phase 3a, compared with --baseline before, produced:

16-pane full scan	Phase 3a estimate	Change
Dense	46.182 ms	-6.36%
Sparse	35.899 ms	+0.83%
No match	35.077 ms	+1.27%

The sparse and no-match movement is noise. More importantly, a complete scan still exceeded the 16 ms stop gate, so Phase 3a did not satisfy input-latency acceptance and Phase 3c was implemented.

Phase 3c measures the full production work for one 512-line cooperative slice, including update thread mapping, accumulated item-cache cloning, and description formatting:

Full production slice	Criterion estimate interval
Sparse	205.45-206.70 µs
Dense	342.42-344.80 µs
No match	196.91-197.85 µs

All three are far below 16 ms. These are Criterion estimate intervals, not p95 latency. Aggregate full-scan timings remain useful throughput evidence; input responsiveness is protected by command-only cooperative slices, cancellation, and bounded result publication.

Server fairness

With sustainable continuous deterministic PTY ingress, key-to-helper acknowledgement had a 2.3376 ms estimate and a 2.3260-2.3512 ms Criterion estimate interval.

The one-shot saturation probe reached 4,194,246 of 4,194,304 PTY ingress bytes. Production then intentionally disconnected the bounded downstream client after 243 ms, before a key acknowledgement could complete. No saturated RTT exists: the probe falsified the assumption that production's bounded overflow policy permits one to be sampled. The sustainable result remains the relevant fairness evidence, and server-loop redesign stays deferred.

Durable resurrection

The resurrection matrix uses the server-reported complete durable export/write/fsync/rename duration, excluding trigger and polling delay:

Panes	History 0	History 1,000	History 5,000
1	3.277 ms	5.656 ms	14.923 ms
8	17.041 ms	83.675 ms	169.917 ms
16	26.619 ms	57.555 ms	207.730 ms

These are mean point estimates, not p95. Some setup and storage cases are noisy, but the matrix now provides direct full-duration evidence across pane and history counts. Slow-storage behavior and the design case for async resurrection remain open.

Memory lifecycle smoke

tools/memory-matrix.sh --smoke exercised a two-pane, two-client image workload:

State	Application PSS	Active clients	Interpretation
Disconnected	38,357 KiB	1	one client remains attached
Reconnected	38,090 KiB	2	both clients attached
Killed	38,613 KiB	0	live probe clients remain; server count is 0

This confirms the bounded smoke lifecycle completes across disconnect, replay-complete reconnect, and session kill. Before shutdown, the killed case captures every server/PTY descendant PID and process start time; afterward it confirmed zero surviving descendants, so reparenting cannot hide a leaked process. This is still lifecycle smoke, not a before/after leak proof or long-duration soak.

Implementation verification

The current audit worktree rooted at the measured base contains:

• permanent scrollback-search, server-fairness, resurrection, and memory-matrix harnesses;
• runtime resource metrics and the metrics control command;
• allocation-free terminal cell text append in the framework;
• epoch-gated workbar command workers;
• 8 MiB piped buffering with a 4 MiB resume low-water mark, plus 4 MiB and 4,096-key global orphan-output caps;
• bounded, cancellable, cooperative scrollback search with a 2,000-result cap and restart on live pane output.

The search acceptance row covers the complete production slice rather than only the range scanner. The fairness benchmark uses the public server/client path and a real continuously emitting PTY. The memory smoke records application PSS separately from child-process memory and reports active attachments separately from live probe processes.

Follow-up: asynchronous resurrection at 9f66a6a

Snapshots now capture pane replay bytes on the server loop and hand the write, sync, and rename to a worker. last_blocking_us reports the part the loop is held for; the Criterion figure remains the whole attempt.

Panes	History	Whole attempt	Server-loop blocking	Moved off the loop
1	0	3.409 ms	0.12-0.31 ms	~93%
16	0	34.494 ms	2.2-3.6 ms	~93%
16	1,000	111.41 ms	33.1-37.0 ms	~69%
8	5,000	109.34 ms	83.9-98.4 ms	~20%
16	5,000	237.08 ms	160.9-178.1 ms	~28%

This did not deliver what the design predicted for the case that motivated it. The assumption was that fsync dominated a snapshot, so moving the write would remove most of the stall. The measurement says otherwise: at 5,000 retained rows the export dominates, and it cannot leave the loop because it needs the live TerminalScreen. The worst case improved from roughly 208 ms of blocking to roughly 170 ms — real, but not the order of magnitude the split implies elsewhere.

The result is strongly shaped by retained history rather than pane count. With little scrollback the loop keeps only ~7% of the work; at 5,000 rows it keeps most of it. So the change helps least exactly where snapshots hurt most.

Export cost has the same shape as the scrollback-search finding: 16 panes x 5,000 rows x 250 columns is 20M cells, and ~170 ms over them is ~8.5 ns each — slower than search's ~1.8 ns because the replay path emits SGR runs rather than plain text. It already uses the allocation-free push_cell_text path, so there is no allocation win left to collect here.

Two levers remain. The first is measured in the next section; the second is not implemented:

• Skip unchanged panes. A snapshot re-exports every pane even though a typical session has one or two active panes and a dozen idle ones.
• Bound persisted history. Persisting fewer rows per pane than the live scrollback retains is a policy decision with a user-visible trade-off, not a transparent optimization.

Follow-up: replay reuse for unchanged panes at 4d45cf4

Each pane carries a content generation, and a snapshot hard-links the replay file the last successful snapshot wrote when that generation still matches. Same fixture and method as above; one pane changes per iteration.

Panes	History	Whole attempt	Server-loop blocking
1	0	3.375 ms	0.13-0.28 ms
1	1,000	5.780 ms	2.09-3.46 ms
1	5,000	15.578 ms	10.18-14.90 ms
8	0	4.160 ms	0.12-0.30 ms
8	1,000	6.662 ms	2.29-3.90 ms
8	5,000	16.571 ms	10.75-13.45 ms
16	0	4.753 ms	0.12-0.27 ms
16	1,000	7.525 ms	2.34-4.12 ms
16	5,000	16.876 ms	10.68-12.03 ms

Three distinct numbers appear above and are easy to conflate. For 16 panes at 5,000 rows:

Criterion estimate (whole attempt):        16.876 ms
Observed server-loop blocking range:       10.68-12.03 ms
Maximum recorded server-loop blocking:     12.03 ms

The headline metric for this work is maximum recorded server-loop blocking, because that is what bounds input latency. The Criterion estimate is throughput of a whole attempt including the worker's durable write, which no longer holds the loop. Comparisons with the pre-async numbers must use the blocking figure on both sides; the original synchronous implementation had no distinction, since its whole attempt was the blocking.

Cost is now driven by retained history and barely by pane count: 1 pane at 5,000 rows costs 15.6 ms and 16 panes at 5,000 rows costs 16.9 ms, because fifteen of the sixteen are links rather than exports.

Against the original synchronous behaviour, maximum server-loop blocking for the 16-pane 5,000-row case went from 207.730 ms to 12.03 ms — about 17x, and now within the same order as the 2.34 ms key round trip rather than roughly 90x it.

What this measures, and what it does not. The benchmark changes exactly one pane per iteration, which matches a session where one pane is busy and the rest are idle. Reuse pays off in proportion to how many panes are unchanged, so a session where every pane is actively producing output still re-exports every pane and lands back at the previous section's numbers. That is the ceiling, not a regression: this optimization removes redundant work rather than making export cheaper.

The reuse path is the one that can silently lose scrollback, so it is guarded structurally rather than by convention: ServerPane's terminal is private and reached through screen_mut, which bumps the generation, or screen_without_change for operations that need &mut but leave persisted content identical.

Reused replay files are treated as immutable artifacts identified by pane and content generation. The invariants, each covered by a test in src/session/server/resurrect.rs:

1. A published replay file is never modified in place. Writes only ever create files in the snapshot's own temporary directory.
2. Every file and link is in place before the manifest is published, and publication is one atomic rename.
3. Reuse claims are recorded only after a successful write, and dropped entirely on any failure.
4. A missing reuse source degrades to a full export in the same attempt: existence is confirmed during capture, on the server loop, where a screen is still available to export from. The worker has no such fallback, so the check cannot be left to it.
5. If hard-linking is unavailable - a filesystem without it, or a snapshot root spanning a mount point - the write falls back to copying.
6. Restore reads file contents through manifest references and never depends on inode identity, so carrying bytes forward by link is invisible to it.

Because links share an inode, a reused file outlives every directory it was published under. reuse_chains_across_generations_and_outlives_the_directory_it_came_from exercises three generations with a different pane dirty each time and asserts the pane untouched throughout keeps its original bytes after both earlier directories are gone.

last_exported_panes, last_reused_panes, and last_exported_bytes are reported through rozi metrics and the DevTools Snapshot row, so a slow snapshot in the field can be attributed to its dirty-pane count without reproducing it.

Reconciling the two changes

Taken alone, the asynchronous split looked like a weak result against the case that motivated it, and the section above says so. That judgement is too narrow in hindsight. The split removed filesystem variance — a slow write, sync, or rename — from the server loop entirely, which is a different source of unpredictable latency than export cost and is not addressed by reuse at all. It also established what reuse is built on: generation-aware completion, publication only after a successful write, bounded shutdown, the capture/persist separation, and a blocking metric distinct from the total. Without it, reuse would have cut CPU-bound export while leaving storage behaviour sitting directly on the loop.

The accurate summary of the two together:

Asynchronous persistence removed fixed filesystem blocking and enabled correct background snapshot publication. Per-pane replay reuse eliminated the dominant redundant export work.

Snapshot cost went from roughly O(all panes x retained history) to O(dirty panes x retained history) + O(clean panes), where the second term is a hard link per pane.

Limitations and open work

• The server's fixed idle polling remains measurable; adaptive backoff or readiness-driven waiting needs design and comparative latency measurement, and must be judged on key round-trip latency as well as idle cost.
• Configurable image-budget work remains gated on Phase 5 measurement and design.
• Resurrection snapshots now write off the server loop and reuse unchanged panes' replay files, taking the 16-pane 5,000-row case from roughly 208 ms of blocking to roughly 11 ms. A session where every pane is busy still exports every pane; bounding persisted history is the remaining lever, and slow-storage evidence is still missing.
• The server's fixed idle polling was re-examined rather than changed. Switching the loop's fast/slow predicate from "any connected client" to "any attached client" was rejected: query and non-attached connections are closed after flush (set_close_after_flush), so they live one or two iterations, while the change would add up to 20 ms to every discovery probe and query response for no measurable idle saving.
• A long-duration/full memory soak was not run, so no leak conclusion follows from the smoke result.
• Standalone CI still cannot resolve the sibling path dependency.
• tests/sidebar_keyboard_smoke.rs's keyboard_navigation_suppresses_stale_row_hover_until_the_pointer_moves fails at this revision. It is baseline-existing and unrelated to this work: it fails identically, with the same two colors, at a612172 - the commit before any performance change - when built against the same framework tree. It belongs to in-flight sidebar file-tree work spanning both repositories. No performance commit touches src/view/sidebar/, src/view/pane.rs, or theme code.

Commands

Environment and revision:

git rev-parse HEAD
git status --short
uname -r
rustc --version
lscpu

Isolated Phase 1 framework comparison:

ROOT=$(mktemp -d)
git worktree add --detach "$ROOT/before/rozi" c07b6be
git -C ../tui-lipan worktree add --detach "$ROOT/before/tui-lipan" 934d7b1
git worktree add --detach "$ROOT/after/rozi" c07b6be
git -C ../tui-lipan worktree add --detach "$ROOT/after/tui-lipan" f951197
CARGO_TARGET_DIR="$ROOT/target-before" cargo bench \
  --manifest-path "$ROOT/before/rozi/Cargo.toml" --bench snapshot_rebuild \
  -- 'snapshot_rebuild/(80x24|200x60|320x90)'
CARGO_TARGET_DIR="$ROOT/target-after" cargo bench \
  --manifest-path "$ROOT/after/rozi/Cargo.toml" --bench snapshot_rebuild \
  -- 'snapshot_rebuild/(80x24|200x60|320x90)'

Saved search baseline and Phase 3a comparison:

cargo bench --bench scrollback_search -- --save-baseline before
cargo bench --bench scrollback_search -- --baseline before

Phase 3c, fairness, resurrection, and lifecycle acceptance:

cargo bench --bench scrollback_search -- full_slice
cargo bench --bench server_fairness -- continuous_pty_ingress
cargo bench --bench server_fairness -- --saturation-probe
cargo bench --bench server_fairness -- resurrection_snapshot
tools/memory-matrix.sh --smoke