Performance follow-up evidence

Performance index · Benchmark guide · Reproduction playbook

Historical context: this report measured a sibling ../tui-lipan/ checkout. The current tree resolves tui-lipan 0.3.1 and relswap 0.0.3 from crates.io, and standalone CI is no longer blocked by that old path dependency. Keep the measurements and conclusions tied to the recorded revisions.

Verdict and scope

Verdict: ready with minor improvements.

The measured revision fixed the search-latency and workbar-worker lifecycle concerns from the previous audit. This dated report records the follow-up measurements. The 2026-08-03 audit remains historical evidence.

Measurement context

Item	Value
Rozi base	2b85924b1ecf83958ae0d5427b053920da6cf0ef; clean before final audit worktree changes
tui-lipan path dependency	includes f951197 allocation-free text, 7f30677 app metrics, 5dc6dd0 visibility, and d410398 controlled palette selection
OS	Arch Linux, kernel 7.1.4
CPU	AMD Ryzen 7 5700X3D, 8 cores / 16 threads
Rust	rustc 1.93.0
Criterion plotting	gnuplot unavailable; Plotters used

The measured dependency was the sibling ../tui-lipan/ checkout. Standalone CI was gated on publishing the required framework version and replacing the path dependency. No registry version was active.

Phase 1 used detached temporary worktrees with the application fixed at c07b6be. It changed only tui-lipan from 934d7b1 to f951197, and separate CARGO_TARGET_DIRs isolated both builds. Later saved before baselines were captured after that framework fix and before application search optimization. They were the comparison point for the search phases. The final saturation, resurrection, and descendant-cleanup evidence exercised the uncommitted audit worktree rooted at the Rozi base above.

Confirmed measured outcomes

Phase 1 snapshot rebuild

Screen	Before (934d7b1)	After (f951197)	Improvement
80x24	64.074 µs	39.631 µs	38.1%
200x60	374.72 µs	229.10 µs	38.9%
320x90	879.19 µs	539.27 µs	38.7%

This same-application, same-machine comparison changed only the framework. It attributed the roughly 38-39% reduction to tui-lipan's allocation-free cell text append. The results agree with the 2026-08-03 audit's 377.64 µs at 200x60 and 892.11 µs at 320x90 under stricter revision control. They do not change the earlier conclusion that lazy, once-per-frame snapshot rebuilding removes most burst cost.

The later saved before run measured snapshots at 39.226 µs, 230.89 µs, and 536.07 µs for the same three dimensions, plus app render at 219.41 µs for 8 panes and 380.29 µs for 16 panes. That run was post-Phase-1 context and supplied the pre-search-optimization baseline below.

Scrollback search

The pre-search-optimization 16-pane full scans were 35.604 ms sparse, 49.320 ms dense, and 34.637 ms with no match.

Phase 3a, compared with --baseline before, produced:

16-pane full scan	Phase 3a estimate	Change
Dense	46.182 ms	-6.36%
Sparse	35.899 ms	+0.83%
No match	35.077 ms	+1.27%

The sparse and no-match movement is noise. A complete scan still exceeded the 16 ms stop gate. Phase 3a therefore did not satisfy input-latency acceptance, and Phase 3c was implemented.

Phase 3c measured the full production work for one 512-line cooperative slice, including update thread mapping, accumulated item-cache cloning, and description formatting:

Full production slice	Criterion estimate interval
Sparse	205.45-206.70 µs
Dense	342.42-344.80 µs
No match	196.91-197.85 µs

All three were below 16 ms. These are Criterion estimate intervals, not p95 latency. Aggregate full-scan timings remained useful throughput evidence. Command-only cooperative slices, cancellation, and bounded result publication protected input responsiveness.

Server fairness

With sustainable continuous deterministic PTY ingress, key-to-helper acknowledgement had a 2.3376 ms estimate and a 2.3260-2.3512 ms Criterion estimate interval.

The one-shot saturation probe reached 4,194,246 of 4,194,304 PTY ingress bytes. Production disconnected the bounded downstream client after 243 ms, before a key acknowledgement could complete. No saturated RTT existed. The probe showed that production's bounded overflow policy did not permit one to be sampled. The sustainable result remained the relevant fairness evidence, and server-loop redesign stayed deferred.

Durable resurrection

The resurrection matrix used the server-reported complete durable export/write/fsync/rename duration, excluding trigger and polling delay:

Panes	History 0	History 1,000	History 5,000
1	3.277 ms	5.656 ms	14.923 ms
8	17.041 ms	83.675 ms	169.917 ms
16	26.619 ms	57.555 ms	207.730 ms

These are mean point estimates, not p95. Some setup and storage cases were noisy. The matrix provided full-duration evidence across pane and history counts. Slow-storage behavior and the design case for async resurrection remained open.

Memory lifecycle smoke

tools/memory-matrix.sh --smoke exercised a two-pane, two-client image workload:

State	Application PSS	Active clients	Interpretation
Disconnected	38,357 KiB	1	one client remains attached
Reconnected	38,090 KiB	2	both clients attached
Killed	38,613 KiB	0	live probe clients remain; server count is 0

This confirmed that the bounded smoke lifecycle completed across disconnect, replay-complete reconnect, and session kill. Before shutdown, the killed case captured every server/PTY descendant PID and process start time. Afterward, it confirmed zero surviving descendants, so reparenting could not hide a leaked process. This remained lifecycle smoke. It was not a before/after leak proof or long-duration soak.

Implementation verification

The audit worktree rooted at the measured base contained:

• permanent scrollback-search, server-fairness, resurrection, and memory-matrix harnesses;
• runtime resource metrics and the metrics control command;
• allocation-free terminal cell text append in the framework;
• epoch-gated workbar command workers;
• 8 MiB piped buffering with a 4 MiB resume low-water mark, plus 4 MiB and 4,096-key global orphan-output caps;
• bounded, cancellable, cooperative scrollback search with a 2,000-result cap and restart on live pane output.

The search acceptance row covers the complete production slice, including work outside the range scanner. The fairness benchmark uses the public server/client path and a real continuously emitting PTY. The memory smoke records application PSS separately from child-process memory and reports active attachments separately from live probe processes.

Follow-up: asynchronous resurrection at 9f66a6a

At 9f66a6a, snapshots captured pane replay bytes on the server loop and handed the write, sync, and rename to a worker. last_blocking_us reports the time the loop was held. The Criterion figure covers the whole attempt.

Panes	History	Whole attempt	Server-loop blocking	Moved off the loop
1	0	3.409 ms	0.12-0.31 ms	~93%
16	0	34.494 ms	2.2-3.6 ms	~93%
16	1,000	111.41 ms	33.1-37.0 ms	~69%
8	5,000	109.34 ms	83.9-98.4 ms	~20%
16	5,000	237.08 ms	160.9-178.1 ms	~28%

This did not deliver the predicted result for the motivating case. The design assumed that fsync dominated a snapshot, so moving the write would remove most of the stall. The measurement showed that export dominated at 5,000 retained rows. Export could not leave the loop because it needed the live TerminalScreen. The worst case improved from roughly 208 ms of blocking to roughly 170 ms. The reduction was smaller than in the other cases.

Retained history shaped the result more than pane count. With little scrollback, the loop keeps only ~7% of the work. At 5,000 rows, it keeps most of it. The change therefore helps least where snapshots cost the most.

Export cost followed the same pattern as the scrollback-search finding. The 16 panes x 5,000 rows x 250 columns case is 20M cells, and ~170 ms over them is ~8.5 ns each. This is slower than search's ~1.8 ns because the replay path emits SGR runs instead of plain text. It uses the allocation-free push_cell_text path, so no further allocation reduction was available there.

Two options remain. The next section measures the first. The second was not implemented:

• Skip unchanged panes. A snapshot re-exports every pane even though a typical session has one or two active panes and a dozen idle ones.
• Bound persisted history. Persisting fewer rows per pane than the live scrollback retains is a policy decision with a user-visible trade-off.

Follow-up: replay reuse for unchanged panes at 4d45cf4

Each pane carried a content generation. A snapshot hard-linked the replay file from the last successful snapshot when that generation still matched. The fixture and method were the same as above, with one pane changed per iteration.

Panes	History	Whole attempt	Server-loop blocking
1	0	3.375 ms	0.13-0.28 ms
1	1,000	5.780 ms	2.09-3.46 ms
1	5,000	15.578 ms	10.18-14.90 ms
8	0	4.160 ms	0.12-0.30 ms
8	1,000	6.662 ms	2.29-3.90 ms
8	5,000	16.571 ms	10.75-13.45 ms
16	0	4.753 ms	0.12-0.27 ms
16	1,000	7.525 ms	2.34-4.12 ms
16	5,000	16.876 ms	10.68-12.03 ms

Three distinct numbers appear above and are easy to conflate. For 16 panes at 5,000 rows:

Criterion estimate (whole attempt):        16.876 ms
Observed server-loop blocking range:       10.68-12.03 ms
Maximum recorded server-loop blocking:     12.03 ms

Maximum recorded server-loop blocking is the main metric for this work because it bounds input latency. The Criterion estimate measures throughput for a whole attempt, including the worker's durable write. That write no longer holds the loop. Comparisons with the pre-async numbers must use the blocking figure on both sides. The original synchronous implementation had no distinction because its whole attempt was the blocking.

At the measured revision, retained history drove cost more than pane count. One pane at 5,000 rows cost 15.6 ms, and 16 panes at 5,000 rows cost 16.9 ms. Fifteen of the sixteen panes used links instead of exports.

Against the original synchronous behaviour, maximum server-loop blocking for the 16-pane 5,000-row case went from 207.730 ms to 12.03 ms, about 17x. It was then within the same order as the 2.34 ms key round trip. Previously, it was roughly 90x that round trip.

The benchmark changes exactly one pane per iteration. This matches a session where one pane is busy and the rest are idle. Reuse pays off in proportion to the number of unchanged panes. A session where every pane is actively producing output still re-exports every pane and returns to the previous section's numbers. Those numbers are the ceiling for this method. The optimization removes redundant work. It does not make export cheaper.

The reuse path can silently lose scrollback, so structural checks guard it. ServerPane's terminal is private. Callers reach it through screen_mut, which bumps the generation, or screen_without_change for operations that need &mut and leave persisted content identical.

The measured implementation treated reused replay files as immutable artifacts identified by pane and content generation. Tests in src/session/server/resurrect.rs covered these invariants:

1. A published replay file is never modified in place. Writes only ever create files in the snapshot's own temporary directory.
2. Every file and link is in place before the manifest is published, and publication is one atomic rename.
3. Reuse claims are recorded only after a successful write, and dropped entirely on any failure.
4. A missing reuse source degrades to a full export in the same attempt: existence is confirmed during capture, on the server loop, where a screen is still available to export from. The worker has no such fallback, so the check cannot be left to it.
5. If hard-linking is unavailable - a filesystem without it, or a snapshot root spanning a mount point - the write falls back to copying.
6. Restore reads file contents through manifest references and never depends on inode identity, so carrying bytes forward by link is invisible to it.

Because links share an inode, a reused file outlives every directory it was published under. reuse_chains_across_generations_and_outlives_the_directory_it_came_from exercises three generations with a different pane dirty each time and asserts the pane untouched throughout keeps its original bytes after both earlier directories are gone.

rozi metrics and the DevTools Snapshot row reported last_exported_panes, last_reused_panes, and last_exported_bytes. These metrics linked a slow snapshot in the field to its dirty-pane count without reproducing it.

Reconciling the two changes

Taken alone, the asynchronous split produced a small improvement in the case that motivated it. The split still moved a slow write, sync, or rename off the server loop. That source of unpredictable latency differs from export cost and reuse does not address it. The split also established generation-aware completion, publication after a successful write, bounded shutdown, capture and persistence separation, and a blocking metric distinct from the total. Without the split, reuse would have reduced CPU-bound export and left storage work on the loop.

The accurate summary of the two together:

Asynchronous persistence removed fixed filesystem blocking and enabled correct background snapshot publication. Per-pane replay reuse eliminated the dominant redundant export work.

Snapshot cost went from roughly O(all panes x retained history) to O(dirty panes x retained history) + O(clean panes), where the second term is a hard link per pane.

Limitations and open work

• The server's fixed idle polling remained measurable. Adaptive backoff or readiness-driven waiting needed design and comparative latency measurement. Evaluation had to include key round-trip latency and idle cost.
• Configurable image-budget work remained gated on Phase 5 measurement and design.
• Resurrection snapshots wrote off the server loop and reused unchanged panes' replay files. This took the 16-pane 5,000-row case from roughly 208 ms of blocking to roughly 11 ms. A session where every pane was busy still exported every pane. Bounding persisted history was the remaining option, and slow-storage evidence was still missing.
• The server's fixed idle polling was re-examined without a change. Switching the loop's fast/slow predicate from "any connected client" to "any attached client" was rejected. Query and non-attached connections closed after flush (set_close_after_flush), so they lived one or two iterations. The change would have added up to 20 ms to every discovery probe and query response with no measurable idle saving.
• A long-duration/full memory soak was not run, so no leak conclusion follows from the smoke result.
• Standalone CI could not resolve the sibling path dependency at the measured revision.
• tests/sidebar_keyboard_smoke.rs's keyboard_navigation_suppresses_stale_row_hover_until_the_pointer_moves fails at this revision. The failure existed in the baseline and was unrelated to this work. It fails identically, with the same two colors, at a612172 - the commit before any performance change - when built against the same framework tree. It belonged to in-flight sidebar file-tree work spanning both repositories. No performance commit touched src/view/sidebar/, src/view/pane.rs, or theme code.

Commands

Environment and revision:

git rev-parse HEAD
git status --short
uname -r
rustc --version
lscpu

Isolated Phase 1 framework comparison:

ROOT=$(mktemp -d)
git worktree add --detach "$ROOT/before/rozi" c07b6be
git -C ../tui-lipan worktree add --detach "$ROOT/before/tui-lipan" 934d7b1
git worktree add --detach "$ROOT/after/rozi" c07b6be
git -C ../tui-lipan worktree add --detach "$ROOT/after/tui-lipan" f951197
CARGO_TARGET_DIR="$ROOT/target-before" cargo bench \
  --manifest-path "$ROOT/before/rozi/Cargo.toml" --bench snapshot_rebuild \
  -- 'snapshot_rebuild/(80x24|200x60|320x90)'
CARGO_TARGET_DIR="$ROOT/target-after" cargo bench \
  --manifest-path "$ROOT/after/rozi/Cargo.toml" --bench snapshot_rebuild \
  -- 'snapshot_rebuild/(80x24|200x60|320x90)'

Saved search baseline and Phase 3a comparison:

cargo bench --bench scrollback_search -- --save-baseline before
cargo bench --bench scrollback_search -- --baseline before

Phase 3c, fairness, resurrection, and lifecycle acceptance:

cargo bench --bench scrollback_search -- full_slice
cargo bench --bench server_fairness -- continuous_pty_ingress
cargo bench --bench server_fairness -- --saturation-probe
cargo bench --bench server_fairness -- resurrection_snapshot
tools/memory-matrix.sh --smoke