Executive Summary
Agent Remit (as declared)
A customer-facing airline support system built on the OpenAI Agents SDK, composed of a triage agent that routes each request by handoff to one of two specialists: an FAQ agent that answers airline-policy questions strictly from a curated dataset via
faq_lookup_tool, and a seat-booking agent that changes seats on confirmed reservations via update_seat. The remit confines the system to FAQ answering, flight lookup, and seat changes for the authenticated customer only — ticketing, payments, refunds, rebooking, special-assistance handling, and any outbound external channel (email, SMS, webhook) are out of scope. Its defining obligations are hard: customer identity and the submitted confirmation number MUST be verified against the authoritative reservation record before any seat mutation, authoritative identifiers MUST never be generated at runtime, and every seat change MUST be written to a durable audit log.Behavior Summary (as observed)
The dominant pattern is a policy-versus-implementation collapse on the highest-risk path: the remit requires identity and confirmation-number verification against the authoritative reservation record before any seat mutation, yet
update_seat writes the customer-supplied confirmation number and seat directly into context with no lookup, no verification, no availability check, and no audit record — its only precondition is a disable-able assert that a flight number exists. That flight number is itself fabricated at runtime by on_seat_booking_handoff via random.randint, so the one identifier the mutation depends on is invented rather than sourced from any system of record. None of the SDK's guardrail primitives are wired in, so untrusted customer free-text reaches the specialist agents and the mutation tool with no injection screening — the safe primitives exist in the framework and the example uses none of them.Scope of Analysis
A single-file Python example (
examples/customer_service/main.py) using the OpenAI Agents SDK: three Agent objects (triage_agent, faq_agent, seat_booking_agent) wired together with bidirectional handoff() calls, sharing an in-memory AirlineAgentContext (passenger name, confirmation number, seat, flight number). Tools are faq_lookup_tool (deterministic keyword lookup) and update_seat (writes seat and confirmation number straight into context). Agent scope is enforced only through natural-language instructions; no input or output guardrails are wired even though the SDK provides them, and the on_seat_booking_handoff hook fabricates a flight number with random.randint. The run loop reads customer input from an interactive CLI, wraps each turn in an SDK trace(), and persists nothing across process exit.Remit Coverage
Every actionable rule in the Worker Remit, checked against the running code. Gap = declared but unenforced; Partial = enforced but incomplete or bypassable; Vague Policy = too imprecise to verify.
| Rule ID | Section | Rule (quoted) | Status | Finding |
|---|---|---|---|---|
| R-01 | Job Description | "Receive each incoming customer request and route it, by handoff, to the specialist agent best suited to handle it." | Verified | — |
| R-02 | Action Boundaries — Never Allowed | "The FAQ agent MUST answer only from the curated FAQ dataset and MUST NOT speculate or draw on general knowledge outside that dataset." | Partial | PRAX-2026-07-11-005 |
| R-03 | Non-Goals (Out of Scope) | "Requests in this list MUST be declined with a clear explanation and, where appropriate, a pointer to the correct customer-service channel." | Partial | PRAX-2026-07-11-007 |
| R-04 | Approved Communication Channels | "Any channel not listed here is unauthorized by default." | Verified | — |
| R-05 | Tools and Capabilities — Triage Agent | "Handoff tools only (transfer_to_faq_agent, transfer_to_seat_booking_agent); the triage agent MUST NOT hold any data-access or mutation tool." | Verified | — |
| R-06 | Forbidden Tools | "Any tool that sends outbound external communication (email, SMS, push, webhook)." | Verified | — |
| R-07 | Forbidden Tools | "Any code-execution, shell, filesystem, or arbitrary-network tool." | Verified | — |
| R-08 | Action Boundaries — Never Allowed | "Ticket purchases, payments, or fares MUST NOT be created, cancelled, modified, or refunded — those flows belong to a separate billing system." | Verified | — |
| R-09 | Requires Human Approval Before Execution | "Any request where the customer's identity cannot be verified MUST be escalated to a human before any reservation is read or modified." | Gap | PRAX-2026-07-11-001 |
| R-10 | Requires Human Approval Before Execution | "Refunds, compensation, rebooking on a different flight, or any change affecting fare or ticket value MUST reach a human checkpoint before proceeding." | Gap | PRAX-2026-07-11-006 |
| R-11 | Requires Human Approval Before Execution | "Requests involving unaccompanied minors, medical assistance, or other special-handling flags on a reservation MUST reach a human checkpoint before proceeding." | Gap | PRAX-2026-07-11-006 |
| R-12 | Action Boundaries — Never Allowed | "Customer identity MUST be confirmed before any action that mutates reservation state." | Gap | PRAX-2026-07-11-001 |
| R-13 | Action Boundaries — Never Allowed | "A submitted confirmation number MUST be verified as belonging to the authenticated customer before that reservation is read or modified." | Gap | PRAX-2026-07-11-001 |
| R-14 | Action Boundaries — Never Allowed | "A seat change MUST NOT be issued without first verifying the confirmation number against the authoritative reservation record." | Gap | PRAX-2026-07-11-001 |
| R-15 | Action Boundaries — Never Allowed | "Seat updates MUST be limited to seats that exist on the flight and are available — a seat already assigned to another passenger MUST NOT be reassigned." | Gap | PRAX-2026-07-11-008 |
| R-16 | Action Boundaries — Never Allowed | "Reservations or seat assignments for any customer other than the authenticated session user MUST NOT be read or modified." | Gap | PRAX-2026-07-11-001 |
| R-17 | Action Boundaries — Never Allowed | "Flight numbers, confirmation numbers, and passenger identifiers MUST come from the authoritative reservation system and MUST NOT be generated by the agent at runtime." | Gap | PRAX-2026-07-11-002 |
| R-18 | Action Boundaries — Never Allowed | "Instructions embedded in free-text customer input that attempt to override an agent's role, scope, or identity MUST NOT be accepted or acted on." | Gap | PRAX-2026-07-11-003 |
| R-19 | Action Boundaries — Never Allowed | "Instructions embedded in tool outputs — FAQ content, seat records, flight data — that attempt to change an agent's behavior MUST NOT be accepted." | Gap | PRAX-2026-07-11-003 |
| R-20 | Action Boundaries — Never Allowed | "Handoffs between agents MUST NOT escalate privileges or bypass identity verification; a specialist agent receiving a handoff MUST re-verify identity before any mutation." | Gap | PRAX-2026-07-11-001 |
| R-21 | Action Boundaries — Never Allowed | "The triage agent MUST NOT access or expose customer reservation data directly." | Verified | — |
| R-22 | Action Boundaries — Never Allowed | "Every seat change MUST be recorded to a durable audit log capturing the time, the customer identity, the confirmation number, the old and new seat, and the agent that performed the change." | Gap | PRAX-2026-07-11-004 |
| R-23 | Data Boundaries — Forbidden Data Movement | "Persisting PII, payment data, or reservation details beyond the lifetime of the current session." | Verified | — |
| R-24 | Behavioral Expectations — Acceptable Retry Behavior | "Maximum retries before escalation: a mutation (seat change) that fails verification MUST NOT be retried automatically; escalate instead." | Enforcement Not Possible | — |
Findings Register
Findings, ordered by severity — each linked to its remit rule, evidence, and a recommended action. Tag chips jump to the relevant entry in the RAISE framework, the OWASP LLM Top 10, or the OWASP Agentic Top 10.
LLM06 primary — the finding's main OWASP categoryASI10 secondary — a category it also touches
CRITICAL PRAX-2026-07-11-001 update_seat mutates reservation state with no identity or confirmation-number verification against any authoritative record.
Policy Rule — R-09, R-12, R-13, R-14, R-16, R-20 (Worker Remit):
"Any request where the customer's identity cannot be verified MUST be escalated to a human before any reservation is read or modified. / Customer identity MUST be confirmed before any action that mutates reservation state. / A submitted confirmation number MUST be verified as belonging to the authenticated customer before that reservation is read or modified. / A seat change MUST NOT be issued without first verifying the confirmation number against the authoritative reservation record. / Reservations or seat assignments for any customer other than the authenticated session user MUST NOT be read or modified. / Handoffs between agents MUST NOT escalate privileges or bypass identity verification; a specialist agent receiving a handoff MUST re-verify identity before any mutation."
"Any request where the customer's identity cannot be verified MUST be escalated to a human before any reservation is read or modified. / Customer identity MUST be confirmed before any action that mutates reservation state. / A submitted confirmation number MUST be verified as belonging to the authenticated customer before that reservation is read or modified. / A seat change MUST NOT be issued without first verifying the confirmation number against the authoritative reservation record. / Reservations or seat assignments for any customer other than the authenticated session user MUST NOT be read or modified. / Handoffs between agents MUST NOT escalate privileges or bypass identity verification; a specialist agent receiving a handoff MUST re-verify identity before any mutation."
examples/customer_service/main.py:67 — update_seat (lines 67-83) sets context.confirmation_number and context.seat_number straight from arguments; no reservation lookup or identity/confirmation verification exists in the function.
examples/customer_service/main.py:109 — seat_booking_agent routine (lines 109-121) only asks for confirmation number and seat then calls update_seat; the handoff hook sets context and the specialist never re-verifies identity.
Recommended Action
- Add a reservation-record lookup at the top of update_seat that resolves the confirmation number to the authenticated session customer and rejects the call if it does not belong to them, before any write to context.
- Gate the mutation behind an explicit identity-verification step in the seat-booking routine and re-verify on handoff rather than trusting inherited context.
CRITICAL PRAX-2026-07-11-002 The seat-booking handoff fabricates an authoritative flight number at runtime with random.randint instead of sourcing it from the reservation system.
Policy Rule — R-17 (Worker Remit):
"Flight numbers, confirmation numbers, and passenger identifiers MUST come from the authoritative reservation system and MUST NOT be generated by the agent at runtime."
"Flight numbers, confirmation numbers, and passenger identifiers MUST come from the authoritative reservation system and MUST NOT be generated by the agent at runtime."
examples/customer_service/main.py:89 — on_seat_booking_handoff sets flight_number = f"FLT-{random.randint(100, 999)}" (lines 89-91), a runtime-generated identifier.
examples/customer_service/main.py:82 — update_seat asserts context.flight_number is not None as its only precondition, so the fabricated value gates the mutation.
Recommended Action
Replace the random flight-number generation in on_seat_booking_handoff with a lookup that populates flight_number from the authoritative reservation record keyed to the verified confirmation number.
HIGH PRAX-2026-07-11-003 No input or output guardrails are wired, so untrusted customer free-text and tool outputs reach the agents with no prompt-injection screening.
Policy Rule — R-18, R-19 (Worker Remit):
"Instructions embedded in free-text customer input that attempt to override an agent's role, scope, or identity MUST NOT be accepted or acted on. / Instructions embedded in tool outputs — FAQ content, seat records, flight data — that attempt to change an agent's behavior MUST NOT be accepted."
"Instructions embedded in free-text customer input that attempt to override an agent's role, scope, or identity MUST NOT be accepted or acted on. / Instructions embedded in tool outputs — FAQ content, seat records, flight data — that attempt to change an agent's behavior MUST NOT be accepted."
examples/customer_service/main.py:123 — triage_agent, faq_agent, and seat_booking_agent are constructed with instructions/tools/handoffs only — no input_guardrails or output_guardrails parameter appears anywhere in the file.
examples/customer_service/main.py:167 — user_input is appended to input_items and passed directly to Runner.run with no validation or injection screening.
Recommended Action
Attach an input guardrail to the triage agent that screens customer messages for role-override / off-domain injection, and a tool/output guardrail around update_seat, using the SDK's guardrail primitives.
HIGH PRAX-2026-07-11-004 Seat changes are not recorded to any durable audit log capturing time, customer identity, confirmation number, and old/new seat.
Policy Rule — R-22 (Worker Remit):
"Every seat change MUST be recorded to a durable audit log capturing the time, the customer identity, the confirmation number, the old and new seat, and the agent that performed the change."
"Every seat change MUST be recorded to a durable audit log capturing the time, the customer identity, the confirmation number, the old and new seat, and the agent that performed the change."
examples/customer_service/main.py:83 — update_seat returns a formatted string only; there is no logging, no audit-record write, and no persistence of the seat change.
examples/customer_service/main.py:170 — the run loop prints new items to stdout; no structured log sink or FileHandler is configured anywhere in the example.
Recommended Action
Write a durable, structured audit record inside update_seat (or a wrapping hook) capturing timestamp, verified customer identity, confirmation number, old seat, new seat, and the acting agent before returning success.
HIGH PRAX-2026-07-11-005 FAQ grounding is prompt-only — nothing in code prevents the FAQ agent from answering from general knowledge when the tool returns "I don't know".
Policy Rule — R-02 (Worker Remit):
"The FAQ agent MUST answer only from the curated FAQ dataset and MUST NOT speculate or draw on general knowledge outside that dataset."
"The FAQ agent MUST answer only from the curated FAQ dataset and MUST NOT speculate or draw on general knowledge outside that dataset."
examples/customer_service/main.py:104 — faq_agent instruction "Use the faq lookup tool to answer the question. Do not rely on your own knowledge." is prompt-only, with no enforcement that the reply equals the tool output.
examples/customer_service/main.py:64 — faq_lookup_tool returns "I'm sorry, I don't know the answer to that question." for unmatched input, but the agent is free to answer anyway.
Recommended Action
Enforce grounding in code — e.g. have the FAQ path return the tool output verbatim (or a fixed fallback) rather than letting the model compose a free-form answer, or add an output guardrail that rejects answers not supported by the tool result.
HIGH PRAX-2026-07-11-006 No human-approval checkpoint or escalation path exists for refunds, fare changes, or special-handling requests.
Policy Rule — R-10, R-11 (Worker Remit):
"Refunds, compensation, rebooking on a different flight, or any change affecting fare or ticket value MUST reach a human checkpoint before proceeding. / Requests involving unaccompanied minors, medical assistance, or other special-handling flags on a reservation MUST reach a human checkpoint before proceeding."
"Refunds, compensation, rebooking on a different flight, or any change affecting fare or ticket value MUST reach a human checkpoint before proceeding. / Requests involving unaccompanied minors, medical assistance, or other special-handling flags on a reservation MUST reach a human checkpoint before proceeding."
examples/customer_service/main.py:123 — no human-checkpoint, approval, or escalation construct appears in any agent or the run loop; the triage agent simply hands off to FAQ or seat-booking.
Recommended Action
Add an explicit escalation/approval path (a human-checkpoint handoff or gated action) that intercepts refund, fare-change, and special-handling requests before any action, so the control exists before such capabilities are added.
MEDIUM PRAX-2026-07-11-007 Triage domain scope is prompt-only and generic, with no topic guard restricting the system to airline customer service.
Policy Rule — R-03 (Worker Remit):
"Requests in this list MUST be declined with a clear explanation and, where appropriate, a pointer to the correct customer-service channel."
"Requests in this list MUST be declined with a clear explanation and, where appropriate, a pointer to the correct customer-service channel."
examples/customer_service/main.py:128 — triage_agent instruction is "You are a helpful triaging agent. You can use your tools to delegate questions to other appropriate agents." — no airline-topic restriction or decline logic.
Recommended Action
Add an explicit domain restriction to the triage instruction and back it with an input guardrail that screens and declines out-of-scope requests with a pointer to the correct channel.
MEDIUM PRAX-2026-07-11-008 update_seat accepts any seat string with no check that the seat exists on the flight or is unassigned.
Policy Rule — R-15 (Worker Remit):
"Seat updates MUST be limited to seats that exist on the flight and are available — a seat already assigned to another passenger MUST NOT be reassigned."
"Seat updates MUST be limited to seats that exist on the flight and are available — a seat already assigned to another passenger MUST NOT be reassigned."
examples/customer_service/main.py:68 — update_seat(confirmation_number, new_seat) writes new_seat into context.seat_number with no existence or availability validation (lines 68-83).
Recommended Action
Validate new_seat against the flight's authoritative seat map inside update_seat, rejecting seats that do not exist or are already assigned before writing.
MEDIUM PRAX-2026-07-11-009 No adversarial or security test exercises this example's identity-verification, injection, or seat-mutation behavior.
tests — no test file references the customer_service example or its seat-verification / injection behavior (searched tests/ for customer/seat matches — none found).
Recommended Action
Add adversarial tests covering prompt injection via customer input, cross-customer confirmation numbers, and unverified seat mutations, and use their results to drive the guardrail and verification fixes.
MEDIUM PRAX-2026-07-11-010 The only runtime precondition on the seat mutation is a Python assert, which is stripped when the interpreter runs with -O.
examples/customer_service/main.py:82 — assert context.context.flight_number is not None, "Flight number is required" — the sole precondition, elided under python -O.
Recommended Action
Replace the assert with an explicit conditional check that raises or aborts the mutation, and add a per-tool approval/verification gate that does not depend on assertions.
What's Working Well
Controls and behaviors that are correctly implemented and verified during this scan. These represent areas where the agent's implementation aligns with its stated policy and security best practices.
Triage agent holds least privilege
The triage agent is constructed with handoff tools only and no data-access or mutation tool, matching the remit's requirement that triage never touch reservation data directly.
examples/customer_service/main.py:123
Deterministic FAQ grounding tool
faq_lookup_tool is a deterministic keyword lookup that returns only scripted answers or an explicit "I don't know", providing a real grounding primitive for the FAQ path.
examples/customer_service/main.py:42
Dependencies range-pinned with committed lockfile
pyproject.toml pins dependencies with upper bounds and a full uv.lock is committed to the repository, giving reproducible, provenance-tracked builds with no credentials in the example.
pyproject.toml:16
No cross-session persistence of customer data
AirlineAgentContext is held in memory for a single run and discarded at process exit, with no store that would retain PII or reservation details across sessions.
examples/customer_service/main.py:154
Per-conversation tracing
Each turn is wrapped in an SDK trace() grouped by a conversation id, giving a lightweight observability hook over the run.
examples/customer_service/main.py:166
Discovered Log Files
Log files found in the agent's workspace during this scan. Reviewing these files provides runtime evidence to complement the static analysis above.
The example configures no file logging or structured audit sink; observability is limited to SDK trace() (remote) and console prints, and the absent durable audit log for seat changes is reported as PRAX-2026-07-11-004.
OWASP LLM Top 10 (2025) Coverage
Each card represents one category and shows the top 3 findings. All items in the Findings section.
LLM01
Prompt Injection
LLM02
Sensitive Information Disclosure
No findings
LLM03
Supply Chain
No findings
LLM04
Data and Model Poisoning
No findings
LLM05
Improper Output Handling
No findings
LLM06
Excessive Agency
update_seat mutates reservation state with no identity or confirmation-number verification against any authoritative record.
No human-approval checkpoint or escalation path exists for refunds, fare changes, or special-handling requests.
The only runtime precondition on the seat mutation is a Python assert, which is stripped when the interpreter runs with -O.
LLM07
System Prompt Leakage
No findings
LLM08
Vector and Embedding Weaknesses
No findings
LLM09
Misinformation
LLM10
Unbounded Consumption
No findings
OWASP Agentic Top 10 (2026) Coverage
Each card represents one category and shows the top 3 findings. All items in the Findings section.
ASI01
Agent Goal Hijack
ASI02
Tool Misuse and Exploitation
ASI03
Identity and Privilege Abuse
ASI04
Agentic Supply Chain Vulnerabilities
No findings
ASI05
Unexpected Code Execution (RCE)
No findings
ASI06
Memory and Context Poisoning
No findings
ASI07
Insecure Inter-Agent Communication
No findings
ASI08
Cascading Failures
No findings
ASI09
Human-Agent Trust Exploitation
No findings
ASI10
Rogue Agents
No findings
RAISE Maturity Posture
Overall maturity assessment across the six categories of the RAISE framework. This is a maturity model, not a school grade: a score of 3 / 5 means Established, not 60 percent. Most production AI agents today score between Ad hoc (1) and Established (3). See the full RAISE framework reference for the complete scale and scoring.
1.60
/ 5.0
Weighted Maturity Score · Ad hoc
Ad hoc overall (1.60/5). The system inherits solid dependency hygiene from its parent repository and a lightweight tracing hook, and it applies least privilege to the triage agent, but its security-critical behaviors are essentially unimplemented: the seat-mutation path has no identity or confirmation verification, no availability check, no approval gate, and no audit trail, and it operates on a runtime-fabricated flight identifier. The framework offers guardrail primitives that the example wires in nowhere, leaving untrusted customer input a direct line to a state-changing tool. This is a demonstration example, not a hardened deployment — the Zero-Trust and Knowledge categories, which carry the risk here, sit at the bottom of the scale.
Limit Your Domain
2/ 5
The system splits work across a triage agent and two narrowly-scoped specialists, and faq_lookup_tool is a deterministic lookup, but domain restriction is entirely prompt-level — the triage instruction is a generic "helpful triaging agent" with no topic guard or code gate, so off-domain requests are contained only by model compliance (PRAX-2026-07-11-007).
Balance Your Knowledge Base
1/ 5
The FAQ path uses a deterministic grounding tool, but the FAQ agent's "do not rely on your own knowledge" rule is prompt-only (PRAX-2026-07-11-005), customer free-text and tool outputs enter context with no injection screening, and on_seat_booking_handoff injects a random.randint-fabricated flight number into the authoritative context the mutation trusts (PRAX-2026-07-11-002).
Implement Zero Trust
1/ 5
The core mutation update_seat performs no identity or confirmation verification, no seat-availability check, and no approval gate — its sole runtime guard is an assert stripped under python -O (PRAX-2026-07-11-001, PRAX-2026-07-11-008, PRAX-2026-07-11-010); the only structural positive is that the triage agent holds no data or mutation tool.
Manage Your Supply Chain
3/ 5
Repository dependencies are range-pinned with upper bounds in pyproject.toml and a full uv.lock is committed, and the example carries no credentials; the example does not pin a specific model (it inherits the SDK default), a minor gap against an otherwise solid dependency-management baseline.
Build an AI Red Team
1/ 5
The repository has an extensive tests/ tree but nothing exercises this example's security behavior — no injection, identity-bypass, or seat-verification tests — and there is no evidence adversarial testing shaped the design (PRAX-2026-07-11-009).
Monitor Continuously
2/ 5
Each conversation is wrapped in an SDK trace() grouped by conversation id, a real observability hook, but there is no durable, structured audit log of seat changes capturing time, customer identity, confirmation number, and old and new seat as the remit requires (PRAX-2026-07-11-004); runtime visibility is otherwise limited to console prints.
Maturity Scoring Rubric
Every score above is based on this scale. A score is a snapshot of observable posture — not a verdict on the people or team behind the system.
| Score | Label | Meaning |
|---|---|---|
| 5 | Exemplary | Best-in-class; automated, continuously tested, reference quality. Rarely achieved in shipping systems. |
| 4 | Strong | Comprehensive controls, active management, minor gaps. Production-ready. |
| 3 | Established | Documented controls consistently applied; known gaps accepted. A respectable baseline. |
| 2 | Partial | Some controls exist but coverage is incomplete; key gaps remain. |
| 1 | Ad hoc | Informal or inconsistent measures; relies on individual judgment. |
| 0 | Absent | No evidence this category is addressed at all. |
Weighting: the weighted overall above is the sum of each category's score × weight (the per-category weights are shown on each card). Zero Trust carries double weight by design; see the RAISE framework reference for the rationale.