Executive Summary
Agent Remit (as declared)
yaah is a local developer CLI that generates coding-agent configuration for four agents — Claude Code, OpenCode, Codex CLI, and GitHub Copilot CLI — from one Go source of truth, and ships a uniform protective toolset alongside every configuration it produces: a linter, a catastrophic-command guard, a secret scanner, a comment/placeholder checker, and a session logger, plus a built-in check-only MCP server (`yaah serve`) and a durable per-session audit log under `.claude/sessions/`. It performs no LLM inference itself; it configures the developer's chosen coding agent and wires in the opt-in third-party MCP servers `context7` and `pulumi`. Its central promise is that these protections apply uniformly across every generated configuration, that remote MCP is reached only over TLS, and that only the check-only tool set — plus the approval-gated `yaah_planning_init` — is ever exposed to the model.
Behavior Summary (as observed)
The dominant pattern is a real, enforced safety model that is not applied uniformly — the "same protections everywhere" promise leaks at the edges. The command guard and secret scanner run as deterministic blocking hooks (Block: true) at PreToolUse/PostToolUse on Claude, OpenCode, and Copilot, but on Codex — whose hook surface has neither event — they degrade to advisory, model-callable yaah_* MCP tools that block nothing and fire only if the model chooses to call them; and the guard's own denylist catches rm -rf / but not flag-order, long-form, or equivalent catastrophic variants.
The secondary theme is uneven supply-chain and assurance discipline: remote skills and agents are pinned to immutable git commit SHAs and releases are cosign-signed with SBOMs, yet the default third-party MCP servers are pulled unpinned via npx -y, and there is no adversarial test anywhere proving that a planted secret is caught or a catastrophic command blocked.
Scope of Analysis
A single Go binary (`cmd/yaah`) built on an interface-and-registry architecture (`pkg/harness`) that wires hooks, MCP, LSP, skills, agents, and commands. The protective hooks live in `pkg/hooks/handlers` (`guard.go`, `secretscan.go`, `commentcheck.go`, `sessionlog.go`) and fire through `Harness.HandleHookEvent`, which also records a structured per-session JSON audit via `pkg/session`. A built-in stdio MCP server (`pkg/mcpserver`) exposes seven `yaah_*` tools; per-agent generators (`pkg/generator/*.go`) emit each target's native config, and `pkg/generator/hookmap.go` maps hook events per agent — Codex supports only `SessionStart`/`Stop`/`Notification`, with no `PreToolUse` or `PostToolUse`. Default MCP wiring (`.mcp.json`, `pkg/mcp/providers`) pins nothing for `context7` (`npx -y @context7/mcp`) while `pulumi` uses an HTTPS endpoint.
Remit Coverage
Every actionable rule in the Worker Remit, checked against the running code. Gap = declared but unenforced; Partial = enforced but incomplete or bypassable; Vague Policy = too imprecise to verify.
| Rule ID | Section | Rule (quoted) | Status | Finding |
|---|---|---|---|---|
| R-01 | Job Description | "Ship, in every generated configuration, the full protective hook set: a linter, a command guard that blocks catastrophic shell commands, a secret scanner, a comment/placeholder checker, and a session logger." | Partial | PRAX-2026-07-11-001 |
| R-02 | Job Description | "Maintain a durable, structured, timestamped per-session audit log of tool calls, blocked commands, file modifications, and security findings under `.claude/sessions/`." | Partial | PRAX-2026-07-11-005 |
| R-03 | Non-Goals (Out of Scope) | "Weakening or removing the protective hooks from a generated configuration so that an agent ships without the full protection set." | Partial | PRAX-2026-07-11-001 |
| R-04 | Approved Communication Channels | "Fetching yaah itself and third-party MCP server packages; MUST resolve pinned, integrity-checked versions." | Gap | PRAX-2026-07-11-003 |
| R-05 | Restricted Tools (Require Approval Before Use) | "`yaah_planning_init` — the one built-in MCP tool that writes to the workspace; workspace writes require the developer's review." | Partial | PRAX-2026-07-11-007 |
| R-06 | Forbidden Data Movement | "Hardcoded credentials MUST be detected before an edit that introduces them is accepted; they MUST NOT be written into committed files unremarked." | Verified | — |
| R-07 | Forbidden Data Movement | "Agent-managed, session-loaded files MUST NOT be writable in a way that lets content the agent ingested from an untrusted source persist into future sessions unreviewed." | Gap | PRAX-2026-07-11-006 |
| R-08 | Forbidden Data Movement | "Repository contents, edited-file contents, and session audit logs MUST NOT leave the local workspace for any destination other than the authorized counterparties above." | Verified | — |
| R-09 | Requires Human Approval Before Execution | "High-impact actions — destructive shell commands, file writes, and MCP tools that write, send, or execute — MUST reach a human checkpoint before they run." | Partial | PRAX-2026-07-11-007 |
| R-10 | Never Allowed | "Catastrophic shell commands — recursive deletion from the filesystem root, force-pushing to a main branch, hard resets, destructive database statements, filesystem formatting, raw disk writes — MUST be blocked before they execute." | Partial | PRAX-2026-07-11-002 |
| R-11 | Never Allowed | "Every generated configuration MUST carry the complete set, not a subset." | Partial | PRAX-2026-07-11-001 |
| R-12 | Never Allowed | "MCP tool descriptions — the built-in server's, and, as far as the harness can inspect them, those of the third-party servers it configures — MUST NOT contain instruction-like language directed at the model (tool-poisoning indicator)." | Verified | — |
| R-13 | Never Allowed | "Reaching a remote MCP server over a non-TLS connection." | Verified | — |
| R-14 | Behavioral Expectations | "The project maintains a durable, structured, timestamped record of tool calls, blocked actions, and file modifications — detailed enough to reconstruct what the agent did in a session." | Verified | — |
| R-15 | Behavioral Expectations | "The project publishes a threat model and a security-disclosure process, and its own protections are adversarially tested — confirming the secret scanner catches a planted secret, the command guard blocks a catastrophic command, a poisoning write into a session-loaded file is prevented, and a generated configuration genuinely carries the protections it claims." | Gap | PRAX-2026-07-11-004 |
| R-16 | Acceptable Retry Behavior | "a shell command the command guard has blocked MUST NOT be re-attempted or auto-approved on retry." | Verified | — |
| R-17 | Known Good Baseline | "Third-party MCP servers MUST be pinned to a known-good, integrity-checked version — no server package auto-installed afresh, unpinned, on every run." | Gap | PRAX-2026-07-11-003 |
| R-18 | Known Good Baseline | "Dependencies MUST be version-controlled with a committed, pinned lockfile, and the dependency tree kept small and reviewable." | Verified | — |
Findings Register
Findings, ordered by severity — each linked to its remit rule, evidence, and a recommended action. Tag chips jump to the relevant entry in the RAISE framework, the OWASP LLM Top 10, or the OWASP Agentic Top 10.
LLM06 primary — the finding's main OWASP categoryASI10 secondary — a category it also touches
HIGH PRAX-2026-07-11-001 On Codex, the command guard and secret scanner ship only as advisory model-callable MCP tools, not enforced hooks, downgrading catastrophic-command blocking.
Implement Zero Trust
LLM06 — Excessive Agency
ASI05 — Unexpected Code Execution (RCE)
LLM02 — Sensitive Information Disclosure
Policy Rule — R-01, R-03, R-11 (Worker Remit):
"Ship, in every generated configuration, the full protective hook set: a linter, a command guard that blocks catastrophic shell commands, a secret scanner, a comment/placeholder checker, and a session logger. / Weakening or removing the protective hooks from a generated configuration so that an agent ships without the full protection set. / Every generated configuration MUST carry the complete set, not a subset."
"Ship, in every generated configuration, the full protective hook set: a linter, a command guard that blocks catastrophic shell commands, a secret scanner, a comment/placeholder checker, and a session logger. / Weakening or removing the protective hooks from a generated configuration so that an agent ships without the full protection set. / Every generated configuration MUST carry the complete set, not a subset."
pkg/generator/hookmap.go:40 — Codex mapping supplies only SessionStart/Stop/Notification; the PreToolUse and PostToolUse entries have no Codex event name (empty), so those hooks cannot be emitted for Codex.
pkg/generator/codex.go:88 — GenerateHooks skips any event whose CodexEventName is empty, dropping the command guard (PreToolUse) and secret/comment/lint checks (PostToolUse) from the generated Codex config.
Recommended Action
- For targets lacking PreToolUse/PostToolUse, enforce high-risk checks through a non-advisory path (e.g. a Codex Stop-hook that fails the turn unless `yaah_check_command` cleared the run) instead of relying on model-optional MCP tools.
- Emit a generation-time warning naming which protective hooks a target cannot enforce natively.
HIGH PRAX-2026-07-11-002 Command guard blocks `rm -rf /` but not flag-order, long-form, or equivalent catastrophic variants, leaving documented bypasses.
Implement Zero Trust
LLM05 — Improper Output Handling
ASI05 — Unexpected Code Execution (RCE)
LLM06 — Excessive Agency
Policy Rule — R-10 (Worker Remit):
"Catastrophic shell commands — recursive deletion from the filesystem root, force-pushing to a main branch, hard resets, destructive database statements, filesystem formatting, raw disk writes — MUST be blocked before they execute."
"Catastrophic shell commands — recursive deletion from the filesystem root, force-pushing to a main branch, hard resets, destructive database statements, filesystem formatting, raw disk writes — MUST be blocked before they execute."
pkg/hooks/handlers/guard.go:29 — Blocked patterns lines 29-36 are literal flag spellings (`rm -rf /`, `git push --force ... main|master`, `git reset --hard`, `DROP TABLE|DATABASE`); flag-order/long-form equivalents and TRUNCATE/DELETE are not covered.
pkg/hooks/handlers/guard.go:67 — CheckCommand/Execute test the raw command string against these regexes with no argv normalization, so `rm -fr /` or `rm --recursive --force /` is reported safe.
Recommended Action
- Tokenize the command and match on normalized argv (recognize rm recursive+force regardless of flag spelling/order; cover force-push, hard-reset, and destructive-SQL equivalents) instead of literal-flag regexes.
- Add tests asserting each catastrophic-command class and its variants are blocked.
HIGH PRAX-2026-07-11-003 Default third-party MCP server `context7` is fetched unpinned via `npx -y @context7/mcp`, auto-installing the latest package on every run.
Policy Rule — R-04, R-17 (Worker Remit):
"Fetching yaah itself and third-party MCP server packages; MUST resolve pinned, integrity-checked versions. / Third-party MCP servers MUST be pinned to a known-good, integrity-checked version — no server package auto-installed afresh, unpinned, on every run."
"Fetching yaah itself and third-party MCP server packages; MUST resolve pinned, integrity-checked versions. / Third-party MCP servers MUST be pinned to a known-good, integrity-checked version — no server package auto-installed afresh, unpinned, on every run."
pkg/mcp/providers/context7.go:22 — Server() returns Command `npx` with Args `-y @context7/mcp` — no version pin, `-y` auto-confirms install of latest on every launch; notion.go uses the same `npx -y @notionhq/notion-mcp-server`.
.mcp.json:3 — Generated context7 entry runs `npx -y @context7/mcp`, so the committed default MCP config ships the unpinned server.
Recommended Action
Pin third-party MCP packages to an exact, integrity-checked version (e.g. `@context7/mcp@<version>`) in both the provider and generated `.mcp.json`, and apply the same to the `notion` provider.
HIGH PRAX-2026-07-11-004 The protective hooks have no adversarial or unit tests proving a planted secret is caught or a catastrophic command blocked.
Policy Rule — R-15 (Worker Remit):
"The project publishes a threat model and a security-disclosure process, and its own protections are adversarially tested — confirming the secret scanner catches a planted secret, the command guard blocks a catastrophic command, a poisoning write into a session-loaded file is prevented, and a generated configuration genuinely carries the protections it claims."
"The project publishes a threat model and a security-disclosure process, and its own protections are adversarially tested — confirming the secret scanner catches a planted secret, the command guard blocks a catastrophic command, a poisoning write into a session-loaded file is prevented, and a generated configuration genuinely carries the protections it claims."
pkg/hooks/handlers — Directory contains guard.go, secretscan.go, commentcheck.go, sessionlog.go, linter.go but no *_test.go — the blocking handlers are untested.
pkg/generator/codex_test.go — Tests present in the repo target config generation, catalog, and gitcache, not the security-enforcement behavior of the hooks.
Recommended Action
Add adversarial unit tests asserting the command guard blocks every catastrophic-command class and its flag-variants, the secret scanner blocks representative planted credentials, and each generated target config carries (or MCP-delivers) every protective check.
MEDIUM PRAX-2026-07-11-005 The session audit record never populates its structured `Findings` field, so non-blocking security findings are not captured.
Policy Rule — R-02 (Worker Remit):
"Maintain a durable, structured, timestamped per-session audit log of tool calls, blocked commands, file modifications, and security findings under `.claude/sessions/`."
"Maintain a durable, structured, timestamped per-session audit log of tool calls, blocked commands, file modifications, and security findings under `.claude/sessions/`."
pkg/session/session.go:14 — Session struct declares `Findings []Finding` (with Type/File/Line/Severity), a dedicated structured findings channel.
pkg/harness/harness.go:130 — HandleHookEvent updates LastEventAt, ToolCalls, BlockedCalls, and FilesModified but never appends to sess.Findings, so the field is always empty.
Recommended Action
Populate sess.Findings from handler results (secret, lint, comment) during dispatch so non-blocking findings are recorded with type/file/line/severity.
MEDIUM PRAX-2026-07-11-006 Generated session-loaded config files are freely writable with no control preventing untrusted-ingested content from persisting unreviewed.
Policy Rule — R-07 (Worker Remit):
"Agent-managed, session-loaded files MUST NOT be writable in a way that lets content the agent ingested from an untrusted source persist into future sessions unreviewed."
"Agent-managed, session-loaded files MUST NOT be writable in a way that lets content the agent ingested from an untrusted source persist into future sessions unreviewed."
pkg/harness/harness.go:392 — WriteAllForTarget emits session-loaded files (skills, agents) with no write-review or provenance control on later modification.
pkg/hooks/handlers/secretscan.go:34 — The only PostToolUse content control scans for secret patterns; nothing inspects writes to session-loaded config files for injected instruction content.
Recommended Action
Add a review/allowlist or content-provenance gate on writes to session-loaded config files, or require them to be version-controlled and diff-reviewed before re-load.
MEDIUM PRAX-2026-07-11-007 `yaah_planning_init` writes to the workspace and is exposed to the model with no approval gate or emitted permission config.
Policy Rule — R-05, R-09 (Worker Remit):
"`yaah_planning_init` — the one built-in MCP tool that writes to the workspace; workspace writes require the developer's review. / High-impact actions — destructive shell commands, file writes, and MCP tools that write, send, or execute — MUST reach a human checkpoint before they run."
"`yaah_planning_init` — the one built-in MCP tool that writes to the workspace; workspace writes require the developer's review. / High-impact actions — destructive shell commands, file writes, and MCP tools that write, send, or execute — MUST reach a human checkpoint before they run."
pkg/mcpserver/tools.go:363 — addPlanningInitTool creates directories and WriteFile STATE.md directly on invocation with no approval check.
pkg/mcpserver/server.go:43 — registerTools adds yaah_planning_init alongside the check-only tools with no distinguishing gate or permission metadata.
Recommended Action
Gate yaah_planning_init behind explicit approval — emit a permission entry marking it a write tool, or require developer confirmation before the write proceeds.
MEDIUM PRAX-2026-07-11-008 No threat model or security-disclosure process is published despite the remit's stated expectation.
Policy Rule — R-15 (Worker Remit):
"The project publishes a threat model and a security-disclosure process, and its own protections are adversarially tested — confirming the secret scanner catches a planted secret, the command guard blocks a catastrophic command, a poisoning write into a session-loaded file is prevented, and a generated configuration genuinely carries the protections it claims."
"The project publishes a threat model and a security-disclosure process, and its own protections are adversarially tested — confirming the secret scanner catches a planted secret, the command guard blocks a catastrophic command, a poisoning write into a session-loaded file is prevented, and a generated configuration genuinely carries the protections it claims."
docs — docs/ contains only components.md and configuration.md; no SECURITY.md, threat-model, or disclosure document exists in the repository.
Recommended Action
Add a SECURITY.md with a disclosure process and a short STRIDE-style threat model for the harness runtime (hooks, MCP server, session store).
What's Working Well
Controls and behaviors that are correctly implemented and verified during this scan. These represent areas where the agent's implementation aligns with its stated policy and security best practices.
Structured durable per-session audit log
HandleHookEvent records a JSON audit at `.claude/sessions/<id>.json` capturing tool calls, blocked calls, and files modified, atomically written and detailed enough to reconstruct a session.
pkg/harness/harness.go:129
Enforced blocking secret scanner and command guard
On Claude/OpenCode/Copilot the secret scanner and command guard run as PreToolUse/PostToolUse hooks that return Block:true, deterministically stopping catastrophic commands and credential-bearing edits.
pkg/hooks/handlers/secretscan.go:145
Remote skills and agents pinned to immutable commit SHAs
Every remote skill/agent is registered with a full git commit SHA ref (e.g. `@f810e5000b5881ec41ea9c33c001ac8c25cbd2f2`), preventing silent upstream drift of remote content.
pkg/harness/defaults.go:492
Signed, SBOM-bearing release pipeline
The release build signs artifacts with cosign and generates SBOMs, giving the harness binary itself verifiable provenance.
.goreleaser.yaml:34
Built-in MCP tool descriptions free of instruction-like language
The seven yaah_* tool descriptions are plain capability statements with no model-directed instructions, avoiding the tool-poisoning pattern.
pkg/mcpserver/tools.go:57
Session ID path-traversal sanitization
sanitizeID rejects empty, dot, and path-separator session IDs before they are used as filenames, preventing traversal via the session store.
pkg/session/store.go:153
Discovered Log Files
Log files found in the agent's workspace during this scan. Reviewing these files provides runtime evidence to complement the static analysis above.
| Path | Source | Content Type | Purpose | Last Modified | Status |
|---|---|---|---|---|---|
| .claude/sessions/<session-id>.json | pkg/session Store via Harness.HandleHookEvent | structured JSON per-session record | tool calls, blocked calls, files modified (and a declared-but-unused findings field) per coding-agent session | unknown | Inferred |
| .claude/logs/sessions.log | pkg/hooks/handlers SessionLogger | plaintext one-line-per-event lifecycle log | SessionStart/SessionEnd/Stop lifecycle events with timestamp, session id, cwd | unknown | Inferred |
OWASP LLM Top 10 (2025) Coverage
Each card represents one category and shows the top 3 findings. All items in the Findings section.
LLM01
Prompt Injection
No findings
LLM02
Sensitive Information Disclosure
No findings
LLM03
Supply Chain
LLM04
Data and Model Poisoning
No findings
LLM05
Improper Output Handling
LLM06
Excessive Agency
LLM07
System Prompt Leakage
No findings
LLM08
Vector and Embedding Weaknesses
No findings
LLM09
Misinformation
No findings
LLM10
Unbounded Consumption
No findings
OWASP Agentic Top 10 (2026) Coverage
Each card represents one category and shows the top 3 findings. All items in the Findings section.
ASI01
Agent Goal Hijack
No findings
ASI02
Tool Misuse and Exploitation
ASI03
Identity and Privilege Abuse
No findings
ASI04
Agentic Supply Chain Vulnerabilities
ASI05
Unexpected Code Execution (RCE)
ASI06
Memory and Context Poisoning
ASI07
Insecure Inter-Agent Communication
No findings
ASI08
Cascading Failures
No findings
ASI09
Human-Agent Trust Exploitation
No findings
ASI10
Rogue Agents
No findings
RAISE Maturity Posture
Overall maturity assessment across the six categories of the RAISE framework. This is a maturity model, not a school grade: a score of 3 / 5 means Established, not 60 percent. Most production AI agents today score between Ad hoc (1) and Established (3). See the full RAISE framework reference for the complete scale and scoring.
2.30
/ 5.0
Weighted Maturity Score · Partial
Partial. yaah has a coherent, genuinely-operative safety model — deterministic command-guard and secret-scanner hooks, a structured durable per-session audit log, SHA-pinned remote content, and cosign-signed SBOM'd releases — which lifts Manage Your Supply Chain and Monitor Continuously to Established. The 2.30 overall is held down by Implement Zero Trust (weight-doubled): the enforced controls are bypassable (guard flag-variants), non-uniform (Codex advisory-only), and leave workspace writes and session-loaded config files ungated. Build an AI Red Team is near-absent — no adversarial test proves any protection actually fires, and no threat model or disclosure process is published.
Limit Your Domain
3/ 5
yaah is tightly scoped to config generation plus a check-only tool set; workflow commands are namespaced `/yaah:*` and registered as explicit user commands the model never auto-triggers, and only `yaah_planning_init` writes. Scope is well-bounded, though the generated skill/agent surface it ships downstream is large.
Balance Your Knowledge Base
2/ 5
yaah performs no inference or RAG of its own and the secret scanner vets edited-file content, but nothing validates content written into session-loaded config files (`AGENTS.md`, generated skills), leaving an uncovered memory-poisoning surface (PRAX-2026-07-11-006).
Implement Zero Trust
2/ 5
Deterministic blocking controls exist and run — the command guard rejects catastrophic Bash at `PreToolUse` and the secret scanner blocks credential-bearing edits at `PostToolUse` — but coverage is bypassable (guard flag-variants, PRAX-2026-07-11-002), non-uniform (Codex gets advisory MCP tools only, PRAX-2026-07-11-001), and `yaah_planning_init` writes with no approval gate (PRAX-2026-07-11-007).
Manage Your Supply Chain
3/ 5
Remote skills and agents are pinned to immutable git commit SHAs and releases are cosign-signed with SBOMs, but the default third-party MCP servers are fetched unpinned via `npx -y @context7/mcp`, auto-installing the latest package on every run (PRAX-2026-07-11-003).
Build an AI Red Team
1/ 5
The protective hooks (`guard.go`, `secretscan.go`, `commentcheck.go`) carry no unit or adversarial tests and the repo publishes no threat model or security-disclosure process, so nothing confirms a planted secret is caught or a catastrophic command blocked (PRAX-2026-07-11-004, PRAX-2026-07-11-008).
Monitor Continuously
3/ 5
`Harness.HandleHookEvent` records a durable, structured, timestamped per-session JSON audit (`.claude/sessions/<id>.json`) of tool calls, blocked calls, and file modifications, sufficient to reconstruct a session; the dedicated `Findings` field is declared but never populated, so non-blocking security findings go uncaptured (PRAX-2026-07-11-005).
Maturity Scoring Rubric
Every score above is based on this scale. A score is a snapshot of observable posture — not a verdict on the people or team behind the system.
| Score | Label | Meaning |
|---|---|---|
| 5 | Exemplary | Best-in-class; automated, continuously tested, reference quality. Rarely achieved in shipping systems. |
| 4 | Strong | Comprehensive controls, active management, minor gaps. Production-ready. |
| 3 | Established | Documented controls consistently applied; known gaps accepted. A respectable baseline. |
| 2 | Partial | Some controls exist but coverage is incomplete; key gaps remain. |
| 1 | Ad hoc | Informal or inconsistent measures; relies on individual judgment. |
| 0 | Absent | No evidence this category is addressed at all. |
Weighting: the weighted overall above is the sum of each category's score × weight (the per-category weights are shown on each card). Zero Trust carries double weight by design; see the RAISE framework reference for the rationale.