Troubleshooting: symptom → root cause → fix

All entries are live-verified failure modes with their exact signatures. HITL-specific failures are in hitl.md; this file covers everything else, by layer.

Debugging method

Reproduce at the lowest possible layer before touching code:

curl -N -X POST <agui-endpoint> -H 'Content-Type: application/json' -d '<minimal RunAgentInput>' — read raw AG-UI SSE events. Reproduces → frontend innocent.
For hosted agents, call the bare /responses (or /invocations) endpoint directly. Reproduces → AG-UI adapter and CopilotKit innocent; the bug is in the framework/hosting layer. This technique is how the duplicate-execution bug (hitl.md) was isolated.
Restart locally running agents between passes — in-memory state from a previous test makes results lie in both directions.

CopilotKit runtime / frontend

Symptom	Root cause	Fix
"Agent `<name>` not found"	Name drift between runtime `agents` key, `<CopilotKit agent>` prop, and hosted `agent.yaml` name; or a single-endpoint/multi-endpoint routing mismatch in the runtime config	Use one shared constant for the agent name; check the runtime's endpoint-mode options against the installed version's docs
Requests to runtime sub-routes (e.g. threads) 404/405	Route handler registered at a fixed path but the runtime version expects a catch-all route serving multiple sub-paths	Use an optional catch-all route segment (`[[...slug]]` in Next.js App Router) and export all HTTP methods the handler supports
`next build` type error: `HttpAgent` missing a property (e.g. `pendingInterrupts`)	Installed `@ag-ui/client` version differs from the one `@copilotkit/runtime` was built against	Pin `@ag-ui/client` to exactly the version the installed `@copilotkit/runtime` depends on (check its package.json)
Console: "Failed to execute 'fetch' on 'Window': Illegal invocation"; agent never runs	A library captured `fetch` as a bare reference and calls it with the wrong `this` (seen with CopilotKit v2 thread store + `@ag-ui/client` `HttpAgent`)	Bind fetch before any module loads, e.g. an inline script in the root layout `<head>`: `if(!window.fetch.__bound){var f=window.fetch.bind(window);f.__bound=true;window.fetch=f;}`
Agent cannot see frontend tools	Known forwarding regression: registered frontend tools not included in `RunAgentInput.tools` (CopilotKit/CopilotKit#5813, 1.62.x era)	Upgrade past the fix; after ANY CopilotKit upgrade, re-test frontend-tool visibility explicitly
Stop button / error handling crashes after a run error	Event-order bug appending `TEXT_MESSAGE_END` after `RUN_ERROR` (CopilotKit/CopilotKit#5812)	Track the fix version; avoid relying on post-error events
Tool/approval card disappears when the run finishes	`MESSAGES_SNAPSHOT` at run end represents the turn differently than live events (e.g. multiple tool calls merged into one message; UI renders only the first)	Fix snapshot construction (one tool call per assistant message) or upgrade the UI layer; always verify post-run DOM
API churn after upgrade (handler factory renamed, provider props changed)	CopilotKit moves APIs between minor versions; `useCopilotAction` is legacy	Verify names against the `.d.ts` files bundled in the installed packages, not docs or memory

AG-UI / adapter layer

Symptom	Root cause	Fix
400 with "orphaned" tool-call errors when sending history	Raw AG-UI message history replayed to a Responses endpoint that manages its own history	Derive each turn's input (latest user message or approval decision); never replay the full transcript
UI shows a 500 mid-run during a long-running silent tool	Proxy/gateway dropped the idle SSE connection	Emit SSE keep-alive comments (`: ping`) every ~10s from the AG-UI endpoint
`useCoAgent().state` always empty	No state schema configured on the agent, no tool writes the state key — or Architecture C without state synthesis (see patterns.md)	Configure state schema + ensure a tool writes it; on a Responses bridge, confirm state synthesis exists at all

Foundry connection / auth

Symptom	Root cause	Fix
401 "audience is incorrect"	Token requested with default `cognitiveservices.azure.com` scope	Request scope `https://ai.azure.com/.default`
403 `Microsoft.MachineLearningServices/workspaces/agents/action` despite being logged in and having the role	`az` CLI's active subscription/tenant differs from the Foundry project's (multi-tenant accounts). Role lookups under the wrong tenant even fail to resolve the assignee, mimicking missing RBAC	Compare `az account show` with the project's tenant/subscription; `az account set --subscription <correct>` or `az login --tenant <correct>`. Zero code changes — do not mistake for a package regression
Deployed agent returns 400 on every call from a custom client	Client sends `x-ms-user-isolation-key`; deployed agents use Entra-derived isolation	Remove the header for deployed agents
Async `DefaultAzureCredential` fails in the bridge	Missing async transport	`pip install aiohttp`
First request to a freshly started local agent 404s `DeploymentNotFound` although the model deployment exists	Warm-up flake in the hosted runtime	Retry once or restart with the same env vars
New `azd ai agent run` fails "Address already in use" (confusing hypercorn traceback)	Stale local hosted-agent process holds port 8088	`ss -ltnp

Python dependency traps

Symptom	Root cause	Fix
Foundry remote image build fails on exotic transitive deps (wasm-related)	Depending on the `agent-framework` meta-package, which drags optional extras	Depend on `agent-framework-core` plus only the specific extras you use (e.g. `agent-framework-foundry`, `agent-framework-ag-ui`)
`ImportError` in the hosted container for `mcp`	`agent_framework_foundry_hosting` imports from `mcp` but it is not pulled transitively in remote builds	Add an explicit `mcp` pin to the hosted requirements
`httpx` APIs missing (`AsyncClient` gone)	Installing with prerelease resolution pulled an httpx 1.0 dev build	Pin httpx to the current stable line
Hosted agent fast-fails: `RuntimeError: the hosted environment is running on protocol 1.0.0, but the agent requires protocol 2.0.0`	Hosting package's Responses protocol version disagrees with `version:` declared in `agent.yaml`/`agent.manifest.yaml`	Bump the package and BOTH manifests' protocol version together
Python `@tool` "didn't run in Foundry" when invoking via the Foundry agent client	Client-side tool callables execute client-side by design; only Foundry-native tools run server-side on that path	Expected behavior — host the agent (run the loop server-side) if tools must execute there

6.8 KiB Raw Blame History