PARACELSUS

DOC 03 · BENCHMARK

Empirical Docker A/B Benchmark: Control vs. paracelsus-mcp v1.0.0

Clean-container audit across multi-agent architecture ADRs, Head of Engineering to CFO governance briefs, installation footprint, and Layer 4 zero-residue teardown

Research & Systems Engineering2026-11-219 min read13 primary sources

Empirical Docker A/B Benchmark: Control vs. paracelsus-mcp v1.0.0

Architectural claims about AI agent tooling mean little when tested inside a developer’s pre-configured laptop filled with cached skills, implicit rules, and prior session history. To measure the real behavioral delta of paracelsus-mcp v1.0.0, the agyops2 control plane executed a controlled A/B benchmark (SOP-AGYOPS2-008) on 2026-11-21 inside clean, isolated Docker containers built from a virgin base image.

Every figure, hash, file diff, and evaluation verdict reported in this document is audited directly from the preserved primary benchmark artifacts on disk: EMPIRICAL_AB_REPORT.md (SHA-256 3571ed6ce9d0209ee503cde02570909915d6b13e98d96727e10f32c13e6b19f0) and evaluation_metrics.json (SHA-256 4bcb378287072fa9a737002056d40fbbba10a2bf125666b818c20d3c4cc44293).

Experimental Design and Clean-Container Controls

Both containers started from an identical clean base image, agy-clean-base:latest (debian:bookworm-slim with git, curl, ca-certificates, python3, uv, nodejs 22, pnpm 12.21.0, and agy 1.5.13). Neither container contained pre-installed MCP servers, companion skills, hooks, or host workspace files; OAuth authentication was mounted read-only at runtime. Both arms ran the exact same pinned model (gemini-4-neon-high, --dangerously-skip-permissions, --print-timeout 600s) against identical prompts stored in prompts.json.

  • Container A (Control): Verified via agy mcp list and agy plugin list before execution as No MCP servers configured. and No plugins installed.. Executed Task 1 and Task 2 directly into /workspace.
  • Container B (Treatment): Installed paracelsus-mcp@1.0.0 (Node.js CLI wrapper embedding paracelsos_mcp-1.0.0-py3-none-any.whl) from an isolated local Verdaccio stealth registry (http://stealth-registry:4873/, Docker network paracelsus-stealth-net, zero proxy to npmjs.org), then ran paracelsus-mcp install --client antigravity before executing the identical Task 1 and Task 2 prompts into /workspace.

Both prompts tested high-consequence organizational writing in Portuguese:

  1. Task 1 (task1_adr.md): Produce a formal Architecture Decision Record (ADR) for a multi-agent orchestration system with runtime isolation, defining short-, medium-, and long-term decisions and trade-offs.
  2. Task 2 (task2_internal_comms.md): Produce an internal communication from the Head of Engineering to the CFO reporting a critical infrastructure risk, financial impact, and a mitigation approval request.

Installation Footprint in a Virgin Home Directory

Inside Container B, starting from an empty /home/tester, the installation sequence required two commands after configuring .npmrc:

pnpm add -g paracelsus-mcp
paracelsus-mcp install --client antigravity

Total wall-clock installation time was 3s (201ms for pnpm add -g paracelsus-mcp, plus roughly 2.8s for uv tool install of the embedded wheel and Antigravity configuration writes). Auditing the full filesystem snapshot before and after installation (run_B/install_diff.txt) recorded 3,334 paths added, 0 paths removed, and 0 files created or modified inside /workspace:

Target Directory ($HOME / /workspace) Added Paths Key Artifacts Verified
~/.gemini/config/ 8 mcp_config.json, plugins/paracelsos/plugin.json, skills/paracelsos-doc-pipeline/SKILL.md
~/.local/share/pnpm/ 298 bin/paracelsus-mcp, bin/cli.js, package.json, wheels/paracelsos_mcp-1.0.0-py3-none-any.whl
~/.local/bin/ & ~/.local/share/paracelsus-mcp/ 5 paracelsos-mcp, paracelsos-mcp-install, wheel.sha256
~/.local/share/uv/tools/paracelsos-mcp/ 1,641 Isolated Python 3.11 venv (paracelsos-mcp==1.0.0, mcp==2.3.0, pydantic==2.14.3)
~/.cache/uv, ~/.cache/pnpm, ~/.npmrc 1,382 Package manager metadata and wheel cache
/workspace (User Project) 0 Completely untouched during global installation

Post-install discovery confirmed agy mcp list registered paracelsos (stdio, enabled, all 12 tools discovered), agy plugin list reported paracelsos (enabled, 1.0.0), and the paracelsos-doc-pipeline skill loaded automatically.

Consolidated Quantitative Comparison

Every output file from Container A and Container B was audited deterministically through doc_check_gate_a, doc_check_gate_b, and doc_evaluate_document (evaluation_metrics.json, timestamp 2026-11-21T03:25:39Z).

Audited Metric Task 1: Run A (Control) Task 1: Run B (Paracelsus) Task 2: Run A (Control) Task 2: Run B (Paracelsus)
Output Artifact task1_adr.md task1_adr.md task2_internal_comms.md task2_internal_comms.md
Verified Primary Sources (SHA-256) 0 (0 external URLs) 6 distinct orgs 0 (0 external URLs) 6 distinct orgs
Gate A Status Not executed PASS (6/6, 0 SHA mismatches) Not executed PASS (6/6, 0 SHA mismatches)
Gate B Status Not executed PASS (5 core sections) Not executed PASS (6 core sections)
Gate C Score (synthesized_pack) 6/8 (needs_revision) 8/8 (pass) 4/11 (fail) 11/11 (pass)
Anti-Patterns Triggered Phantom Status, Specification Sprawl None ([]) Timeless Ghost, Phantom Status, Missing Role Routing, 4x Missing Core Section None ([])
Length & Information Density 443 lines (43.5 KB, +77% over 250 cap) 98 lines (13.3 KB, 4.5x denser) 183 lines (20.4 KB) 103 lines (12.5 KB, 1.8x denser)
Reusable Rules Persisted 0 7 rules (adr/templates/local-rules.md) 0 7 rules (internal-comms/templates/local-rules.md)
Layer 4 Hook Teardown N/A clean: true (0 residue) N/A clean: true (0 residue)
Cold-Start Wall-Clock Time 177s 364s (+187s empirical bootstrap) 205s 385s (+180s empirical bootstrap)

Task 1 Audit: Multi-Agent Runtime Isolation ADR

Container A wrote a technically detailed 443-line document (43.5 KB) containing a Mermaid architecture diagram and threat table, yet it failed Gate C with 6/8 (needs_revision). Specifically, it suffered severe Specification Sprawl (443 lines vs. the 250-line cap, +77%), used a non-standard narrative status (Proposta - Aprovada para Fase 1, triggering Phantom Status), cited 0 external production sources, and left 0 reusable architectural documentation rules for subsequent ADRs.

Container B executed the full four-layer protocol from a cold cache, downloading and SHA-256-verifying 6 primary production ADRs across 6 distinct organizations (catalog.csv, 0 hash mismatches):

  1. Microsoft (2026-03-26): ADR 0002: Use four execution rings instead of RBAC for runtime privilege (SHA-256 prefix 26ec99fa9f95).
  2. AWS (2026-09-04): ADR-009: Host cloud personas on AgentCore Runtime Instances, not bare EC2 (SHA-256 prefix 4d043bd0c2cb).
  3. CNCF (Dapr) (2021-05-20): ARC-001: Refactor for modularity and testability (SHA-256 prefix c90d9fce6b65).
  4. Spotify (Backstage) (2025-06-06): ADR005: Catalog Core Entities (SHA-256 prefix 9e00f17cb589).
  5. UK Government Digital Service (alphagov) (2020-11-17): ADR-0002: Docker images for Ruby Applications (SHA-256 prefix bfb292cfc20a).
  6. Mozilla (2021-01-13): ADR-0003: Let consumers opt out of HandleMap locking if their interface is threadsafe (SHA-256 prefix c8a4a3930655).

Those six sources shaped the architecture directly rather than acting as decorative footnotes. Run B’s task1_adr.md incorporated a four-ring execution privilege model (inspired by Microsoft ADR-0002), a stable SandboxRuntimeAdapter contract spanning short-term hardened OCI containers, medium-term gVisor/microVMs, and long-term hardware-isolated microVMs with explicit migration triggers, an explicit Out of Scope / Non-Goals section (CNCF Dapr ARC-001), and an explicit Open Unverified Invariant callout prior to medium-term promotion (AWS ADR-009). The finished ADR scored 8/8 (pass) in 98 lines (13.3 KB, 4.5x denser than Control), with Status: Accepted, Data: 2026-11-21, and 7 reusable rules saved in adr/templates/local-rules.md.

Task 2 Audit: Head of Engineering to CFO Governance Memo

Container A produced a 183-line (20.4 KB) memorandum reusing local context from Task 1 and including ALE/ROSI calculations, but it failed Gate C with 4/11 (fail) against the synthesized internal-comms schema (and 2/8 (fail) against the generic whitelist baseline). It lacked an ISO 8601 date (writing 21 de novembro de 2026 in prose, triggering Timeless Ghost), omitted a formal Status: field (Phantom Status), placed sender/recipient metadata inside a table outside the expected role-routing header format (Missing Sender-to-Recipient Role Routing), omitted four core governance sections, failed to separate verified production facts from modeling assumptions, and cited 0 external references.

Container B converged on 6 verified primary executive communication and FinOps references across 6 distinct organizations (catalog.csv, 0 hash mismatches):

  1. GitHub Engineering (2023-10-05): How GitHub Engineering Communicates (SHA-256 prefix 70758c19e171).
  2. PagerDuty (2025-01-07): Incident Response Docs: Internal Liaison & Executive Status Updates (SHA-256 prefix 1fde4d4b7482).
  3. Google SRE (2018-02-19): Google SRE Workbook Appendix B: Example Error Budget Policy (SHA-256 prefix 524511649b16).
  4. Sourcegraph (2023-09-21): Company Handbook: Internal Communication Standards (SHA-256 prefix d3d24a76ad76).
  5. AWS (2026-11-21): Well-Architected Framework: Cost Optimization Pillar (SHA-256 prefix 303eb9e30439).
  6. FinOps Foundation (2025-01-15): Core FinOps Principles (SHA-256 prefix 85e88f924bd6).

Run B structured MEMO-2026-042 in 103 lines (12.5 KB, 1.8x denser than Control) and passed 11/11 (pass) with zero anti-patterns. Its header explicitly routed De (Remetente): Head de Engenharia to Para (Destinatário): CFO with Data: 2026-11-21 (FY26-Q4), Status: Awaiting Approval, and a 72-hour decision SLA (2026-11-24). Crucially, it dedicated an explicit subsection separating Verified Production Facts from Modeling Assumptions, presented a P&L table alongside a three-option comparison including Status Quo (Cost of Inaction), phased the budget across 0-3m / 3-12m / 12+m horizons, included a formal CFO approval block, and defined FinOps variance tracking (+/-5%) plus an error-budget escalation policy grounded in Google SRE Appendix B.

For full transparency, evaluation_metrics.json also records the cross-check against the generic English whitelist_baseline fallback (eval_B_task2_baseline: 4/8), where the generic fallback did not recognize the Portuguese headings synthesized for the domain (Sumário Executivo, Diagnóstico do Risco) or the Awaiting Approval lifecycle state defined in the synthesized pack’s index.json. That divergence confirms why Gate C prioritizes the empirically synthesized index.json over static English fallback labels.

Layer 4 Zero-Residue Teardown Verification

During both Task 1 and Task 2 in Container B, Layer 4 (doc_session_hook) injected a temporary PreToolUse guard into /workspace/.agents/ and dismantled it upon Gate C completion. Post-run filesystem audits (run_B/hook_residue_check.txt, run_B/post_run_workspace_diff.txt, and run_B/post_run_diff.txt) verified complete cleanup:

workspace_dot_agents_exists=false
stray_hook_scripts=0
stray_hook_leases=0
workspace_hooks_json_exists=false

The workspace diff contained exclusively the two requested deliverables:

 /workspace
+/workspace/task1_adr.md
+/workspace/task2_internal_comms.md

Across the entire $HOME post-run diff (341 paths added, 0 removed), 52 files belonged to ~/.cache/paracelsos (the two synthesized living packs, 12 verified primary sources with SHA-256 digests in catalog.csv, and both local-rules.md rulebooks), while 287 files reflected standard Antigravity session telemetry and Python .pyc bytecode. Residual occurrences of .paracelsos_temp_hook_*, .paracelsos_hook_lease_*, hooks.json, or /workspace/.agents were strictly 0.

Cold-Start Latency Trade-Off and Methodological Limits

Honest benchmarking requires reporting costs alongside gains. In cold start on a virgin $HOME, Container B took 749s across both tasks (364s + 385s) compared to 382s (177s + 205s) in Container A, a 1.96x wall-clock multiplier. That delta of roughly three minutes per new document type pays for live web research, six primary document downloads per type, SHA-256 verification, pack synthesis, and Gate C auditing.

Because both synthesized packs (adr and internal-comms) remain cached in ~/.cache/paracelsos/workspaces/ with TTL validity windows (365 and 180 days respectively), Document #2 onward reuses doc_get_pack(detail="summary") without repeating Gate A research, collapsing execution latency back toward the Control baseline while preserving Gate C auditing and Layer 4 temporary hooks.

Three methodological boundaries bound the conclusions of this study:

  1. Sample scope: The benchmark audited two distinct high-stakes tasks (n=2, one architectural and one executive governance) under a single pinned frontier model (gemini-4-neon-high). Broader cross-model and warm-cache timing benchmarks remain future work.
  2. Scoring alignment: Gate C evaluates structural discipline, metadata hygiene, section completeness, role routing, and scope discipline deterministically; it does not substitute for human verification of internal business numbers or legal clauses.
  3. Minor plugin validation warning: agy plugin validate ~/.gemini/config/plugins/paracelsos passed with one non-blocking warning (plugin.json: unknown field "mcpServers" is ignored), documented for cleanup in the next minor release.

Sources