awesome-copilot/skills/foundry-hosted-agent-copilotkit/references/troubleshooting.md

Troubleshooting and upgrades

Reproduce at the lowest failing layer first: hosted agent → gateway (curl -N) → CopilotKit route → browser. Match the exact error text below. HITL failures are covered in hitl.md.

Hosted agent and azd

Symptom	Cause	Fix
HTTP 424 session_not_ready	main() raised before serving /readiness (often missing env vars)	Check azd ai agent monitor; fail soft by serving a stub agent with an actionable message
RuntimeError: ... running on protocol 1.0.0, but the agent requires protocol 2.0.0	Hosting package and the protocols[].version in azure.yaml disagree	Bump the package and azure.yaml in one commit
azd deploy: Foundry dependencies are not ready: ... FOUNDRY_PROJECT_ENDPOINT is not set	init ran in a directory with an existing azure.yaml or env, so the project binding is incomplete (often with the wrong AZURE_LOCATION too)	Re-run azd ai agent init --no-prompt -e <new-env> -p ... -d ... in a fresh directory and check azd env get-values; don't azd provision an existing project
azd ai agent ... reports the extension "Incompatible"	azd core is older than the extension needs	azd update && azd extension upgrade --all
Remote build fails but local works	Meta-package agent-framework pulls optional extras, or unpinned pre-releases resolve differently	Depend on specific agent-framework-* packages; pin versions
Deployed agent still runs old code	Traffic pinned to an older version	azd ai agent show; move traffic to the latest version
400 "Hosted agents can only be called through the agent endpoint"	Called with agent_reference on the project Responses endpoint	Use .../agents/<name>/endpoint/protocols/openai/responses, or FoundryAgent(agent_name=...)
ImportError for agent_framework.azure / agent-framework-azure-ai	Superseded package	Migrate to agent-framework-foundry

Gateway / AG-UI

Symptom	Cause	Fix
401 "audience is incorrect"	Token requested for the wrong scope	Use https://ai.azure.com/.default
401/403 invoking the agent	Gateway identity lacks a role on the project, or the token is from the wrong tenant	Assign Azure AI User on the project; az login --tenant <id>
400 mentioning x-ms-user-isolation-key	Header sent to a deployed agent	Remove it; identity comes from Entra
Browser run fails with INCOMPLETE_STREAM: "Cannot send 'TOOL_CALL_END' event: No active tool call found with ID 'mcpr_...'"	Adapter closes a hosted approval id it never started	Use the gateway shim in wiring §2
RUN_ERROR APPROVAL_RESUME_REQUIRED: "Pending interrupts require a resume entry for every interruptId"	Approval sent as a tool message only (v1-style confirm_changes respond)	Resolve with useInterrupt (it sends resume); in curl, add a resume array
Approve → a second approval request instead of execution	Each run hits a new Foundry conversation	use_service_session=True + service_session_id_from_thread_id=True with conv_… thread IDs
After an approval, every turn returns RUN_STARTED → RUN_FINISHED with nothing	Duplicate interrupt tool message reached the adapter	Gateway shim in wiring §2
ValueError: use_service_session=True requires snapshot persistence	Service sessions without a way to map threads to conversations	Add service_session_id_from_thread_id=True and use conv_… IDs as thread IDs (wiring §2)
HTTP 500 server_error from Foundry on the first turn	Thread ID is not a Foundry conversation ID (for example, a UUID generated by CopilotKit) while service_session_id_from_thread_id=True	Create the conversation first (POST .../endpoint/protocols/openai/conversations) and pass its conv_… ID as threadId
RUN_ERROR "No pending approval interrupt found for resume interruptId"	Gateway restarted, or a different replica received the resume	Ask again (the tool didn't run); use session affinity across replicas
Tools run but the stream has no TOOL_CALL_RESULT events; tool rows render empty	Gateway runs on an older agent-framework-foundry (seen with 1.11.0; 1.13.1 works), often a global or user-site Python instead of the project venv	Run the gateway from its own venv with the latest packages; confirm with python -m pip list inside that venv
Async credential errors about aiohttp	azure.identity.aio needs aiohttp	pip install aiohttp
400 about orphaned / unanswered tool calls	Whole AG-UI transcript replayed into /responses (custom gateways)	Send only the latest user message plus the conversation / service session
useAgent().agent.state always empty	No state_schema on AgentFrameworkAgent, no tool writes state, or a custom gateway that never emits STATE_*	Declare the schema on the gateway wrapper; verify STATE_SNAPSHOT with curl

CopilotKit / frontend

Symptom	Cause	Fix
"Agent <name> not found"	Runtime agents key, agentId, and agent name disagree	One shared constant
404 on /api/copilotkit/info or /agent/.../run	v2 runtime is multi-route but the Next.js route isn't a catch-all, or basePath is wrong	Use app/api/copilotkit/[[...slug]]/route.ts with a matching basePath
TypeScript error: HttpAgent is missing a property	App's @ag-ui/* version differs from the one the runtime pins	Match @ag-ui/client/@ag-ui/core to npm view @copilotkit/runtime@<v> dependencies
"Failed to execute 'fetch' on 'Window': Illegal invocation"	A library calls a captured fetch without its this	Upgrade; or pass fetch.bind(window) where a fetch option exists
CORS errors from the browser	Browser calling the gateway directly	Browser → CopilotKit route (same origin) → gateway server-side; otherwise use an explicit CORS allowlist
Hook not exported / renamed after upgrade	v1 names (useCopilotAction, useCoAgent, CopilotKit provider) used against /v2	Use v2 imports from @copilotkit/react-core/v2; check the bundled .d.ts
Dev warning "The agent called the tool ... and no renderer is registered"; tool results invisible	No tool renderer registered	useDefaultRenderTool(...) or useRenderTool({ name, ... })
Playwright: typed text stays in the input; no /agent/.../run request	Typed before hydration, or fill() didn't trigger the input handler	Wait for networkidle, click(), pressSequentially(text), then press("Enter") and assert the box clears
Tool call renders while streaming, then disappears	Final MESSAGES_SNAPSHOT differs from live events	Upgrade; test the DOM after RUN_FINISHED

Upgrades

Always move to the latest releases, and move everything together:

1. @copilotkit/* move in lockstep. react-core, runtime, and any UI packages stay on one version, and @ag-ui/* matches the version the runtime pins.
2. agent-framework-* stay on one release line. Upgrade -core, -foundry, -ag-ui, and -foundry-hosting together.
3. Protocol version and azure.yaml change together (see the table above).
4. Workarounds have exit criteria. Keep a list mapping each local workaround to its upstream issue. Remove a workaround only when the fix has shipped and the regression test passes without it.
5. Read what shipped. CopilotKit release notes can be empty stubs, so diff the bundled .d.ts between versions.
6. Re-verify live: the chat path, frontend tools reaching RunAgentInput.tools, approve-once / reject-zero / follow-up-zero, cards after RUN_FINISHED, then redeploy and repeat against the deployed agent.