Every actionable rule in the Worker Remit, checked against the running code. Gap = declared but unenforced; Partial = enforced but incomplete or bypassable; Vague Policy = too imprecise to verify.
| Rule ID | Section | Rule (quoted) | Status | Finding |
|---|---|---|---|---|
| R-01 | Job Description | "Extract code from properly formatted markdown code blocks (triple-backtick fences) in incoming messages and execute only code in supported languages (Python and shell for command-line executors; Python only for the Jupyter and Azure executors)." | Verified | — |
| R-02 | Job Description | "Execute each code block inside the configured execution environment, in the order received, and return a structured result containing exit status and captured output." | Verified | — |
| R-03 | Job Description | "Confine all file reads and writes to a configured working directory (for the Azure executor, `/mnt/data`)." | Partial | PRAX-2026-07-11-010 |
| R-04 | Job Description | "Enforce a configured wall-clock timeout on every execution and terminate any process that exceeds it." | Verified | — |
| R-05 | Job Description | "When configured with an approval function, obtain approval for each code block before executing it." | Verified | — |
| R-06 | Job Description | "Surface all errors (syntax, runtime, timeout, out-of-memory) back to the caller in structured form without silently discarding them." | Verified | — |
| R-07 | Non-Goals (Out of Scope) | "Sending email, SMS, webhooks, or any outbound message on its own behalf." | Verified | — |
| R-08 | Approved Communication Channels | "The executor must not initiate its own network traffic beyond a configured allow-list." | Partial | PRAX-2026-07-11-004 |
| R-09 | Authorized Counterparties | "Any code source other than the paired agent's generated code blocks and code passed explicitly by operator application logic — code arriving from a file, HTTP endpoint, message queue, or memory store must not be accepted by the executor interface." | Verified | — |
| R-10 | Authorized Counterparties | "The paired AutoGen code-generator agent (the LLM component that produces code); code from this counterparty is the only accepted execution input." | Partial | PRAX-2026-07-11-008 |
| R-11 | Forbidden Tools | "Any mechanism that loads or executes code fetched from a remote URL or other unverified source on the LLM's behalf." | Verified | — |
| R-12 | Forbidden Tools | "Any capability that persists execution results to long-term storage or carries state across unrelated sessions." | Verified | — |
| R-13 | Forbidden Data Movement | "Reading or modifying the parent process's environment, credentials, or state from within an execution." | Partial | PRAX-2026-07-11-006 |
| R-14 | Forbidden Data Movement | "Moving files or data outside the configured working directory." | Partial | PRAX-2026-07-11-010 |
| R-15 | Action Boundaries | "Executing code on the local host executor in a production deployment (default production deployments must use a containerized executor)." | Gap | PRAX-2026-07-11-001 |
| R-16 | Action Boundaries | "Enabling any network egress from the execution environment beyond a configured allow-list." | Gap | PRAX-2026-07-11-004 |
| R-17 | Action Boundaries | "Mounting host volumes into a container executor at any path other than the configured working directory." | Partial | PRAX-2026-07-11-009 |
| R-18 | Action Boundaries | "Any change to resource limits (CPU, memory, timeout) that raises the ceiling." | Partial | PRAX-2026-07-11-007 |
| R-19 | Action Boundaries | "Exposing host GPUs or other host devices to a container executor." | Partial | PRAX-2026-07-11-009 |
| R-20 | Action Boundaries | "Executing code with host-level privileges when a less-privileged option achieves the same task." | Partial | PRAX-2026-07-11-001 |
| R-21 | Action Boundaries | "Loading or executing code from a remote URL or unverified source on the LLM's behalf; the paired agent's generated code is the only accepted input." | Verified | — |
| R-22 | Action Boundaries | "Silently swallowing or transforming error output before returning it to the caller." | Verified | — |
| R-23 | Action Boundaries | "Connecting to services, databases, or networks not explicitly permitted by configuration." | Partial | PRAX-2026-07-11-004 |
| R-24 | Behavioral Expectations | "Actions that should never be retried: executions terminated for exceeding the timeout must be terminated, not re-run in a loop; approval-denied code must not be retried without new approval." | Verified | — |
| R-25 | Escalation Rules | "Every execution is recorded with timestamp, executor kind, language, source agent, working directory, timeout, exit status, and a digest (not the body) of the executed code." | Gap | PRAX-2026-07-11-005 |
| R-26 | Escalation Rules | "The local host executor is used in a production deployment without operator approval." | Gap | PRAX-2026-07-11-001 |
| R-27 | Escalation Rules | "A code execution attempts to read or write outside the working directory, or to escape the container/sandbox." | Enforcement Not Possible | — |
Findings, ordered by severity — each linked to its remit rule, evidence, and a recommended action. Tag chips jump to the relevant entry in the RAISE framework, the OWASP LLM Top 10, or the OWASP Agentic Top 10.
LLM06 primary — the finding's main OWASP categoryASI10 secondary — a category it also touches
CRITICAL PRAX-2026-07-11-001 create_default_code_executor silently downgrades to unisolated host execution with only a warning, defeating the required container-approval gate.
"Executing code on the local host executor in a production deployment (default production deployments must use a containerized executor). / Executing code with host-level privileges when a less-privileged option achieves the same task. / The local host executor is used in a production deployment without operator approval."
- In `create_default_code_executor`, do not silently fall back to `LocalCommandLineCodeExecutor`; raise unless the caller passes an explicit opt-in flag (e.g. `allow_local_fallback=True`) acknowledging unisolated host execution.
- Route any local-host execution in a non-ephemeral deployment through the `approval_func` path so it cannot run without an approval response.
HIGH PRAX-2026-07-11-002 approval_func defaults to None, so all LLM-generated code is auto-executed with no human-in-the-loop gate.
- Consider making `approval_func` required (or defaulting to a deny-by-default gate) for non-interactive deployments, rather than auto-approving on a bare warning.
- Document and expose a supported "approval mandatory" configuration so operators can enforce human review without writing their own gate.
HIGH PRAX-2026-07-11-003 LocalCommandLineCodeExecutor docstring claims dangerous-command sanitization that is not implemented anywhere in the code.
HIGH PRAX-2026-07-11-004 DockerCommandLineCodeExecutor creates containers with default networking and exposes no egress control, so executed code can reach the network.
"The executor must not initiate its own network traffic beyond a configured allow-list. / Enabling any network egress from the execution environment beyond a configured allow-list. / Connecting to services, databases, or networks not explicitly permitted by configuration."
HIGH PRAX-2026-07-11-005 No executor writes a durable, structured audit record of executions, leaving the remit's Log Only requirement unmet.
"Every execution is recorded with timestamp, executor kind, language, source agent, working directory, timeout, exit status, and a digest (not the body) of the executed code."
HIGH PRAX-2026-07-11-006 LocalCommandLineCodeExecutor passes the parent process's full environment to executed code via os.environ.copy().
"Reading or modifying the parent process's environment, credentials, or state from within an execution."
MEDIUM PRAX-2026-07-11-007 Docker executor sets no CPU or memory limits, bounding executions only by wall-clock timeout.
"Any change to resource limits (CPU, memory, timeout) that raises the ceiling."
MEDIUM PRAX-2026-07-11-008 The sources counterparty filter defaults to None, so code from any group-chat agent is executed rather than only the paired coder.
"The paired AutoGen code-generator agent (the LLM component that produces code); code from this counterparty is the only accepted execution input."
MEDIUM PRAX-2026-07-11-009 extra_volumes and device_requests are accepted and applied with no gate or warning, allowing host mounts and GPU exposure beyond the working directory.
"Mounting host volumes into a container executor at any path other than the configured working directory. / Exposing host GPUs or other host devices to a container executor."
MEDIUM PRAX-2026-07-11-010 Working-directory confinement is enforced only for the optional # filename: header; the code body is unconfined on host executors.
"Confine all file reads and writes to a configured working directory (for the Azure executor, `/mnt/data`). / Moving files or data outside the configured working directory."
MEDIUM PRAX-2026-07-11-011 The default Docker execution image python:3-slim is a floating tag with no digest pin, exposing executions to base-image drift.
Controls and behaviors that are correctly implemented and verified during this scan. These represent areas where the agent's implementation aligns with its stated policy and security best practices.
Container executors provide genuine host isolation
DockerCommandLineCodeExecutor and the Docker-Jupyter executor run code inside a container that mounts only the working directory to /workspace, giving real filesystem and process isolation on the recommended production path.
Per-execution wall-clock timeout enforced across all executors
Every executor enforces a configurable timeout and terminates the process/kernel/request that exceeds it, bounding runaway executions.
Approval hook available and honored before execution
When an approval_func is supplied, execute_code_block calls it with the code and context and returns an error result without running the code if approval is denied; the behavior is unit-tested.
Errors surfaced, not swallowed
Execution results carry the real exit code and captured stderr/stdout back to the caller, matching the remit requirement that errors never be silently discarded.
Docker-Jupyter server generates a random auth token by default
DockerJupyterServer defaults token to a 32-byte secrets.token_hex value and binds the published port to 127.0.0.1, so the kernel gateway is authenticated unless the operator explicitly disables the token.
Core dependency pinned to an exact version
autogen-ext pins autogen-core==0.7.5 exactly, reducing supply-chain drift on the primary runtime dependency.
Log files found in the agent's workspace during this scan. Reviewing these files provides runtime evidence to complement the static analysis above.
| Path | Source | Content Type | Purpose | Last Modified | Status |
|---|---|---|---|---|---|
| (stderr / configured logging handlers) | autogen_ext.code_executors.docker._docker_code_executor and local executor | unstructured Python logging (debug/info/error) | diagnostic messages for container lifecycle, pip install, and stop/cancel errors — not an execution audit trail | unknown | Inferred |
| (in-memory message stream) | autogen_agentchat CodeExecutorAgent | ephemeral CodeGenerationEvent / CodeExecutionEvent objects | per-turn generation and execution events yielded to the caller UI; not persisted | unknown | Inferred |
Each card represents one category and shows the top 3 findings. All items in the Findings section.
Each card represents one category and shows the top 3 findings. All items in the Findings section.
Overall maturity assessment across the six categories of the RAISE framework. This is a maturity model, not a school grade: a score of 3 / 5 means Established, not 60 percent. Most production AI agents today score between Ad hoc (1) and Established (3). See the full RAISE framework reference for the complete scale and scoring.
Maturity Scoring Rubric
Every score above is based on this scale. A score is a snapshot of observable posture — not a verdict on the people or team behind the system.
| Score | Label | Meaning |
|---|---|---|
| 5 | Exemplary | Best-in-class; automated, continuously tested, reference quality. Rarely achieved in shipping systems. |
| 4 | Strong | Comprehensive controls, active management, minor gaps. Production-ready. |
| 3 | Established | Documented controls consistently applied; known gaps accepted. A respectable baseline. |
| 2 | Partial | Some controls exist but coverage is incomplete; key gaps remain. |
| 1 | Ad hoc | Informal or inconsistent measures; relies on individual judgment. |
| 0 | Absent | No evidence this category is addressed at all. |