Executive Summary
Agent Remit (as declared)
yaah is a local developer CLI that generates coding-agent configuration for Claude Code, OpenCode, Codex CLI, and GitHub Copilot CLI from a single source of truth, and ships a built-in security-and-quality toolset alongside it. Its central promise is that every generated configuration carries the same protective hooks — a linter, a command guard that blocks catastrophic shell commands, a secret scanner, a comment checker, and a session logger — plus a built-in `yaah serve` MCP server and a durable per-session audit log. Authorized counterparties are limited to the local developer, the configured agent's LLM provider, the `context7` and `pulumi` MCP servers, the built-in yaah server, and the coding-agent plugin marketplace. Catastrophic shell commands must be blocked before execution, remote MCP servers must use TLS, third-party MCP servers must be version-pinned, and high-impact actions must reach a human checkpoint.
Behavior Summary (as observed)
The dominant pattern is a policy-implementation divergence in the harness's central promise: yaah ships real, operative protections — a deterministic command guard, a 13-pattern secret scanner, a comment checker, and a structured session audit log — but its uniformity claim breaks for one target. In pkg/generator/hookmap.go the Codex field is blank for both HookPreToolUse and HookPostToolUse, so yaah generate --agent codex emits a config that carries none of the command-guard or secret-scanner hooks the remit promises every generated agent will receive — and a unit test (TestCodex_GenerateHooks_NoSupported) locks that omission in as expected behavior.
The second theme is supply-chain and approval gaps the running controls do not cover: the context7 MCP server is an unpinned npx -y @context7/mcp install that re-resolves latest on every run, ordinary file writes and the write-capable yaah_planning_init MCP tool have no pre-execution approval gate (the secret scanner runs PostToolUse, after the write lands), and there is no threat model or adversarial test of the protections themselves.
Scope of Analysis
A Go 1.25 CLI (`module github.com/dirien/yet-another-agent-harness`) built on cobra, organized as an Interface to Registry to Generator pattern under `pkg/`. The runtime path is `yaah hook <event>`, which `cmd/yaah/main.go` dispatches through `harness.HandleHookEvent`, running registered handlers in `pkg/hooks/handlers/` (command-guard, secret-scanner, linter, comment-checker, session-logger) and persisting a structured per-session JSON record via `pkg/session`. Four generators in `pkg/generator/` emit each agent's native config; `pkg/generator/hookmap.go` maps Claude hook events to each agent's event names, and a built-in MCP server in `pkg/mcpserver/` exposes seven check-only/read-only tools over stdio. The notable divergence: the Codex generator emits no PreToolUse/PostToolUse hooks because hookmap.go leaves those entries blank for Codex, and the third-party `context7` server is configured as an unpinned `npx -y` install.
Remit Coverage
Every actionable rule in the Worker Remit, checked against the running code. Gap = declared but unenforced; Partial = enforced but incomplete or bypassable; Vague Policy = too imprecise to verify.
| Rule ID | Section | Rule (quoted) | Status | Finding |
|---|---|---|---|---|
| R-01 | Stated Protections | "Every generated agent configuration MUST carry the same set of protective hooks — a linter, a command guard that blocks catastrophic shell commands, a secret scanner, a comment checker, and a session logger." | Partial | PRAX-2026-05-29-002 |
| R-02 | Action Boundaries | "Catastrophic shell commands — recursive deletion from the filesystem root, force-pushing to a main branch, hard resets, destructive database statements, filesystem formatting, raw disk writes — MUST be blocked before they execute." | Partial | PRAX-2026-05-29-002 |
| R-03 | Action Boundaries | "Remote MCP servers MUST be reached over TLS." | Verified | — |
| R-04 | Action Boundaries | "MCP tool descriptions MUST NOT contain instruction-like language directed at the model — neither the built-in server's tool descriptions nor, as far as the harness can inspect them, those of the third-party servers it configures." | Partial | PRAX-2026-05-29-006 |
| R-05 | Forbidden Actions | "Hardcoded credentials in files the agent edits MUST be detected before the change is accepted." | Partial | PRAX-2026-05-29-003 |
| R-06 | Forbidden Actions | "Agent-managed, session-loaded files MUST NOT be writable in a way that lets ingested content persist into future sessions unreviewed." | Vague Policy | — |
| R-07 | Approval Requirements | "High-impact actions — destructive shell commands, file writes, and MCP tools that write, send, or execute — MUST reach a human checkpoint before they run." | Partial | PRAX-2026-05-29-004 |
| R-08 | Behavioral Expectations | "The harness MUST maintain a durable, structured, timestamped record of tool calls, blocked actions, and file modifications — detailed enough to reconstruct what the agent did in a session." | Verified | — |
| R-09 | Behavioral Expectations | "The MCP server configuration and the per-agent configuration files MUST stay consistent with each other." | Verified | — |
| R-10 | Known Good Baseline | "Third-party MCP servers MUST be pinned to a known-good, integrity-checked version — no server package auto-installed afresh, unpinned, on every run." | Gap | PRAX-2026-05-29-001 |
| R-11 | Known Good Baseline | "Dependencies MUST be version-controlled with a committed, pinned lockfile, and the dependency tree kept small and reviewable." | Verified | — |
| R-12 | Authorized Counterparties | "Any counterparty not listed here is unauthorized by default." | Partial | PRAX-2026-05-29-007 |
| R-13 | Escalation and Limits | "The project SHOULD publish a threat model and a security-disclosure process, and SHOULD run adversarial testing of its own protections — confirming that the secret scanner catches a planted secret, the command guard blocks a catastrophic command, a poisoning write into a session-loaded file is prevented, and a generated configuration genuinely carries the protections it claims." | Gap | PRAX-2026-05-29-005 |
| R-14 | Known Good Baseline | "All but `yaah_planning_init` are read-only or check-only; `yaah_planning_init` writes to the workspace." | Verified | — |
Findings Register
Findings, ordered by severity — each linked to its remit rule, evidence, and a recommended action. Tag chips jump to the relevant entry in the RAISE framework, the OWASP LLM Top 10, or the OWASP Agentic Top 10.
HIGH PRAX-2026-05-29-001 The context7 third-party MCP server is configured as an unpinned `npx -y @context7/mcp` install that re-resolves the latest version on every run.
Policy Rule — R-10 (Worker Remit):
"Third-party MCP servers MUST be pinned to a known-good, integrity-checked version — no server package auto-installed afresh, unpinned, on every run."
"Third-party MCP servers MUST be pinned to a known-good, integrity-checked version — no server package auto-installed afresh, unpinned, on every run."
pkg/mcp/providers/context7.go:20 — Command "npx" with Args ["-y", "@context7/mcp"] — no version pin, -y auto-installs the latest on every invocation
.mcp.json:4 — generated context7 entry carries the same unpinned npx -y @context7/mcp command into the Claude config
Recommended Action
In pkg/mcp/providers/context7.go pin the package to an exact version (e.g. `@context7/mcp@<version>`) and prefer a lockfile-backed or integrity-checked install over `npx -y`.
HIGH PRAX-2026-05-29-002 `yaah generate --agent codex` ships a config carrying none of the advertised PreToolUse/PostToolUse hooks, so the command guard and secret scanner are absent for Codex.
Policy Rule — R-01, R-02 (Worker Remit):
"Every generated agent configuration MUST carry the same set of protective hooks — a linter, a command guard that blocks catastrophic shell commands, a secret scanner, a comment checker, and a session logger. / Catastrophic shell commands — recursive deletion from the filesystem root, force-pushing to a main branch, hard resets, destructive database statements, filesystem formatting, raw disk writes — MUST be blocked before they execute."
"Every generated agent configuration MUST carry the same set of protective hooks — a linter, a command guard that blocks catastrophic shell commands, a secret scanner, a comment checker, and a session logger. / Catastrophic shell commands — recursive deletion from the filesystem root, force-pushing to a main branch, hard resets, destructive database statements, filesystem formatting, raw disk writes — MUST be blocked before they execute."
pkg/generator/hookmap.go:30 — HookPreToolUse and HookPostToolUse mappings set Claude/OpenCode/Copilot but leave Codex unset (blank string = unsupported)
pkg/generator/codex.go:91 — GenerateHooks skips any event where CodexEventName(event) == "", so PreToolUse/PostToolUse are silently dropped from .codex/hooks.json
Recommended Action
- Populate the Codex field for HookPreToolUse and HookPostToolUse in hookmap.go (Codex CLI supports these via hooks.json) and update TestCodex_GenerateHooks_NoSupported to assert the protective hooks are present.
- If Codex genuinely cannot run these events, fail `yaah generate --agent codex` loudly or document the unprotected target rather than emitting a config that silently lacks the promised guards.
HIGH PRAX-2026-05-29-003 The secret scanner detects credentials only after the file is written (PostToolUse) and is a finite regex denylist, so detection is reactive, not preventive.
Policy Rule — R-05 (Worker Remit):
"Hardcoded credentials in files the agent edits MUST be detected before the change is accepted."
"Hardcoded credentials in files the agent edits MUST be detected before the change is accepted."
pkg/hooks/handlers/secretscan.go:68 — Events() returns HookPostToolUse — the scan fires after the write, then sets Block=true, so the credential is already on disk
pkg/hooks/handlers/secretscan.go:34 — NewSecretScanner registers 13 fixed regex patterns — a denylist that misses credential formats it does not enumerate
Recommended Action
- Document that the scanner is post-write detection plus turn-block, and add a remediation step that removes or rewrites the offending file rather than leaving the secret on disk after blocking.
- Treat the pattern list as a denylist with known gaps; supplement with entropy-based detection for high-entropy literals not matched by the named patterns.
HIGH PRAX-2026-05-29-004 No pre-execution human approval gate exists for ordinary file writes or for the write-capable yaah_planning_init MCP tool.
Policy Rule — R-07 (Worker Remit):
"High-impact actions — destructive shell commands, file writes, and MCP tools that write, send, or execute — MUST reach a human checkpoint before they run."
"High-impact actions — destructive shell commands, file writes, and MCP tools that write, send, or execute — MUST reach a human checkpoint before they run."
pkg/mcpserver/tools.go:363 — addPlanningInitTool writes directories and STATE.md to the workspace with no approval or confirmation gate
pkg/harness/harness.go:131 — file modifications from Edit/Write/MultiEdit are only recorded after dispatch — there is no pre-write approval path
Recommended Action
Add an approval/confirmation gate for write-capable tools (yaah_planning_init) and for general file-write events, or document that approval is delegated to the host agent's own permission system and is out of yaah's enforcement scope.
HIGH PRAX-2026-05-29-005 There is no adversarial testing of the security controls themselves and no published threat model or security-disclosure process.
Policy Rule — R-13 (Worker Remit):
"The project SHOULD publish a threat model and a security-disclosure process, and SHOULD run adversarial testing of its own protections — confirming that the secret scanner catches a planted secret, the command guard blocks a catastrophic command, a poisoning write into a session-loaded file is prevented, and a generated configuration genuinely carries the protections it claims."
"The project SHOULD publish a threat model and a security-disclosure process, and SHOULD run adversarial testing of its own protections — confirming that the secret scanner catches a planted secret, the command guard blocks a catastrophic command, a poisoning write into a session-loaded file is prevented, and a generated configuration genuinely carries the protections it claims."
pkg/hooks/handlers — directory contains guard.go, secretscan.go, commentcheck.go, linter.go, sessionlog.go but zero *_test.go files — no adversarial or efficacy tests of the controls
pkg/generator/codex_test.go:118 — TestCodex_GenerateHooks_NoSupported asserts the missing Codex hooks as expected, so the test suite ratifies rather than catches the protection gap
Recommended Action
- Add adversarial tests under pkg/hooks/handlers/ that plant a secret and a catastrophic command and assert the scanner/guard block them, plus a test that every generated target carries the full protective hook set.
- Publish a SECURITY.md threat model and disclosure process as the remit recommends.
MEDIUM PRAX-2026-05-29-006 The harness does not inspect or constrain third-party MCP tool descriptions, so the no-instruction-language rule is enforced only for its own built-in server.
Balance Your Knowledge Base
Tool descriptions contain no hidden model instructions
ASI04 — Agentic Supply Chain Vulnerabilities
Policy Rule — R-04 (Worker Remit):
"MCP tool descriptions MUST NOT contain instruction-like language directed at the model — neither the built-in server's tool descriptions nor, as far as the harness can inspect them, those of the third-party servers it configures."
"MCP tool descriptions MUST NOT contain instruction-like language directed at the model — neither the built-in server's tool descriptions nor, as far as the harness can inspect them, those of the third-party servers it configures."
pkg/mcp/providers/context7.go:17 — provider returns only command/args by reference — no fetch or screening of the third-party server's tool descriptions
pkg/mcp/providers/pulumi.go:17 — pulumi provider returns a bare URL — tool descriptions from the remote server are never inspected for instruction-like language
Recommended Action
Add a screening pass (or document the limitation explicitly) that inspects configured third-party MCP tool descriptions for instruction-like language before they reach the model, at least for stdio servers yaah can introspect.
MEDIUM PRAX-2026-05-29-007 A `notion` MCP provider exists in the codebase and is registerable, but Notion is not among the remit's authorized counterparties.
Policy Rule — R-12 (Worker Remit):
"Any counterparty not listed here is unauthorized by default."
"Any counterparty not listed here is unauthorized by default."
pkg/harness/defaults.go:345 — if opts.NotionToken != "" the harness registers mcpproviders.NewNotion — an outbound Notion counterparty not in the remit's authorized list
pkg/mcp/providers/notion.go — a notion provider implementation is present in pkg/mcp/providers/ alongside the three declared providers
Recommended Action
Either add Notion to the remit's authorized counterparties (with its scope) or remove/feature-flag the notion provider out of the default harness so the code matches the declared counterparty list.
MEDIUM PRAX-2026-05-29-008 Handler-level findings (detected secrets, lint, comment issues) are defined in the session record but never written by the dispatcher, narrowing the audit trail.
pkg/harness/harness.go:117 — the session-update block writes ToolCalls, BlockedCalls, and FilesModified but never appends to sess.Findings
pkg/session/session.go:14 — Session.Findings is declared and rendered by `yaah session show` but has no write path in the dispatcher
Recommended Action
In harness.HandleHookEvent, translate handler results that report secret/lint/comment findings into session.Finding records and append them before Save, so the durable record captures findings, not just blocks.
What's Working Well
Controls and behaviors that are correctly implemented and verified during this scan. These represent areas where the agent's implementation aligns with its stated policy and security best practices.
Deterministic command guard blocks catastrophic shell commands
NewCommandGuard registers seven regex rules (rm -rf /, force-push to main, hard reset, DROP TABLE/DATABASE, truncate, mkfs, dd to /dev) that block at PreToolUse before the command runs — a real code-level control, not a prompt instruction.
pkg/hooks/handlers/guard.go:27-37
Structured, durable per-session audit log
harness.HandleHookEvent records tool calls, blocked calls, and file modifications into a per-session JSON file written atomically (temp-then-rename) via pkg/session, queryable through `yaah session show`.
pkg/harness/harness.go:84-144
Exact-pinned Go dependencies with committed lockfile
go.mod pins every dependency to an exact version with a committed go.sum and a small, reviewable tree; remote skills and agents are pinned to git commit SHAs.
go.mod:5-20
Built-in MCP server tool descriptions are clean
All seven yaah_* tool descriptions in pkg/mcpserver/tools.go are plain functional statements with no instruction-like language directed at the model, satisfying the remit's anti-tool-poisoning rule and the MCP minimum-bar check.
pkg/mcpserver/tools.go:56-441
Remote MCP server reached over TLS
The pulumi MCP provider is configured with an https:// URL, satisfying the remit's requirement that remote MCP servers be reached over TLS.
pkg/mcp/providers/pulumi.go:17-23
Discovered Log Files
Log files found in the agent's workspace during this scan. Reviewing these files provides runtime evidence to complement the static analysis above.
| Path | Source | Content Type | Purpose | Last Modified | Status |
|---|---|---|---|---|---|
| .claude/sessions/<session-id>.json | pkg/session.Store via harness.HandleHookEvent | structured JSON session record | durable per-session audit of tool calls, blocked calls, and files modified | unknown | Inferred |
| .claude/logs/sessions.log | pkg/hooks/handlers/sessionlog.go (SessionLogger) | plaintext append-only lifecycle log | session start/end/stop lifecycle events with timestamp, session ID, cwd | unknown | Inferred |
OWASP LLM Top 10 (2025) Coverage
Each card represents one category and shows the top 3 findings. All items in the Findings section.
LLM01
Prompt Injection
No findings
LLM02
Sensitive Information Disclosure
LLM03
Supply Chain
LLM04
Data and Model Poisoning
No findings
LLM05
Improper Output Handling
No findings
LLM06
Excessive Agency
`yaah generate --agent codex` ships a config carrying none of the advertised PreToolUse/PostToolUse hooks, so the command guard and secret scanner are absent for Codex.
No pre-execution human approval gate exists for ordinary file writes or for the write-capable yaah_planning_init MCP tool.
A `notion` MCP provider exists in the codebase and is registerable, but Notion is not among the remit's authorized counterparties.
LLM07
System Prompt Leakage
No findings
LLM08
Vector and Embedding Weaknesses
No findings
LLM09
Misinformation
No findings
LLM10
Unbounded Consumption
No findings
OWASP Agentic Top 10 (2026) Coverage
Each card represents one category and shows the top 3 findings. All items in the Findings section.
ASI01
Agent Goal Hijack
No findings
ASI02
Tool Misuse and Exploitation
ASI03
Identity and Privilege Abuse
No findings
ASI04
Agentic Supply Chain Vulnerabilities
The context7 third-party MCP server is configured as an unpinned `npx -y @context7/mcp` install that re-resolves the latest version on every run.
The harness does not inspect or constrain third-party MCP tool descriptions, so the no-instruction-language rule is enforced only for its own built-in server.
ASI05
Unexpected Code Execution (RCE)
No findings
ASI06
Memory and Context Poisoning
No findings
ASI07
Insecure Inter-Agent Communication
No findings
ASI08
Cascading Failures
No findings
ASI09
Human-Agent Trust Exploitation
No findings
RAISE Maturity Posture
Overall maturity assessment across the six categories of the RAISE framework. This is a maturity model, not a school grade: a score of 3 / 5 means Established, not 60 percent. Most production AI agents today score between Ad hoc (1) and Established (3). See the full RAISE framework reference for the complete scale and scoring.
2.30
/ 5.0
Weighted Maturity Score · Partial
Partial. yaah has a coherent, operative safety model — a deterministic command guard, a secret scanner, a structured durable audit log, exact-pinned Go dependencies, and a clean built-in MCP server — which lifts Supply Chain and Monitor Continuously to Established and keeps Zero Trust and Knowledge Base at Partial rather than Absent. It is held back by a uniformity gap (Codex ships without the protective hooks), the absence of any pre-write approval gate, an unpinned third-party MCP server, and no adversarial testing of the protections it advertises. This is a maturing developer tool with real controls and identifiable, fixable holes, not a hobby project and not a hardened system.
Limit Your Domain
3/ 5
yaah is a narrowly-scoped config generator with a fixed, enumerated tool inventory (seven named built-in MCP tools in pkg/mcpserver/tools.go, six handler types) and an explicit out-of-scope statement; the main slack is an unused `notion` MCP provider in pkg/mcp/providers/ that is not in the remit's authorized counterparties.
Balance Your Knowledge Base
2/ 5
The harness ingests file content into handlers (secret/comment scanning) and exposes file-reading MCP tools, but it does not pull untrusted external content into an LLM context it controls; the remit's claim about agent-managed session-loaded files is not clearly implemented in code, leaving that surface unverifiable.
Implement Zero Trust
2/ 5
A deterministic command guard blocks seven catastrophic-command patterns at PreToolUse and the secret scanner blocks on detection, but there is no approval gate on ordinary file writes or on the write-capable yaah_planning_init MCP tool, the secret scanner runs PostToolUse (after the write reaches disk), and the Codex target ships with no PreToolUse/PostToolUse interposition at all.
Manage Your Supply Chain
3/ 5
Go dependencies are exact-pinned in go.mod with a committed go.sum and a deliberately small tree, and remote skills/agents pin git SHAs; the one real gap is the third-party context7 MCP server configured as an unpinned `npx -y @context7/mcp` install that re-resolves on every run, directly against the remit's pinning rule.
Build an AI Red Team
1/ 5
The repo has generator/unit tests but zero adversarial tests of the security controls themselves — there are no tests under pkg/hooks/handlers/ exercising the command guard or secret scanner against bypasses — and no threat model or security-disclosure process, which the remit says the project SHOULD have.
Monitor Continuously
3/ 5
harness.HandleHookEvent persists a durable, structured, timestamped per-session JSON record (tool calls, blocked calls, files modified) via pkg/session with atomic writes, satisfying the remit's audit requirement; it is docked because handler-level Findings (detected secrets/lint/comment issues) are declared in the Session struct but never written by the dispatcher.
Maturity Scoring Rubric
Every score above is based on this scale. A score is a snapshot of observable posture — not a verdict on the people or team behind the system.
| Score | Label | Meaning |
|---|---|---|
| 5 | Exemplary | Best-in-class; automated, continuously tested, reference quality. Rarely achieved in shipping systems. |
| 4 | Strong | Comprehensive controls, active management, minor gaps. Production-ready. |
| 3 | Established | Documented controls consistently applied; known gaps accepted. A respectable baseline. |
| 2 | Partial | Some controls exist but coverage is incomplete; key gaps remain. |
| 1 | Ad hoc | Informal or inconsistent measures; relies on individual judgment. |
| 0 | Absent | No evidence this category is addressed at all. |
Weighting: the weighted overall above is the sum of each category's score × weight (the per-category weights are shown on each card). Zero Trust carries double weight by design; see the RAISE framework reference for the rationale.