awesome-copilot/extensions/connector-namespaces/test/README.md

MCP smoke test

A standalone harness that proves the Microsoft first-party MCP servers behind a connector gateway actually work end-to-end:

connect → initialize → tools/list → a safe tools/call

It imports the connector-namespaces extension's real pipeline (install.mjs, catalog.mjs, armClient.mjs) and connects through the same native Streamable HTTP endpoint that the extension writes to the Copilot CLI config. The probe uses the configured X-API-Key, follows Mcp-Session-Id, and accepts standard JSON or SSE JSON-RPC responses.

The whole point: it runs with Node and Azure CLI. No Copilot app, no canvas, no UI. Hand it to anyone (e.g. Arjun) and they can reproduce an MCP server issue locally.

Prerequisites

1. Azure CLI signed in with az login. The harness asks Azure CLI for the same short-lived ARM token as the extension.
2. A gateway already picked once. The harness reads gateway coordinates from ~/.copilot/extensions/connector-namespaces/artifacts/gateway-config.json ({ subscriptionId, resourceGroup, gatewayName }). Pick a gateway once in the connector-namespaces canvas, or write that file by hand.
3. Node 20+ (developed on Node 24).

Run it

node extensions/connector-namespaces/test/smoke.mjs

Options:

flag	effect
--only=a,b	only test these apiNames (comma-separated)
--limit=N	stop after N connectable servers
--open-consent	open consent URLs in the browser for OAuth servers that need it
--no-cleanup	leave fresh keyless installs in place (default: uninstall them)

Examples:

# just the three already-connected WorkIQ servers
node extensions/connector-namespaces/test/smoke.mjs --only=WorkIQMail,WorkIQSharePoint,WorkIQTeams

# first 5 connectable servers, open any consent prompts
node extensions/connector-namespaces/test/smoke.mjs --limit=5 --open-consent

One-time consent, then headless forever

This is the key behavior. OAuth-backed servers (most of them) need a human to consent once in a browser. The model:

1. First run hits a server that needs consent → the harness prints a consent URL and marks it NEEDS_CONSENT. It saves a pending record to ~/.copilot/extensions/connector-namespaces/artifacts/smoke-pending-consent.json (not in the repo). No tool call is attempted.
2. You open that URL once and sign in / consent. After sign-in the browser may show "this site can't be reached" on a 127.0.0.1:7333/auth/callback/ page — that is expected and harmless. Consent completes gateway-side; the loopback page is just a redirect target and nothing is listening on it.
3. Re-run the harness. It sees the pending record, confirms the gateway connection is now Connected, finishes the install (mints the API key, writes the CLI entry), and probes it headless. From then on it's reused with zero human interaction.

So the server taxonomy is:

• Already connected (e.g. the three WorkIQ servers) → probed immediately.
• Keyless / SP / AAD (e.g. Microsoft Learn Docs) → installed + probed + cleaned up immediately, no consent.
• Consent-once OAuth → surfaced on run 1, converts to headless on run 2.

That's why the first run may probe fewer than 10 servers — the rest are waiting on their one-time consent. Consent the URLs it prints, re-run, and the count climbs. This is inherent to the consent model, not a harness bug.

Tool-call safety

The harness never blindly calls the first tool a server advertises (mutation risk). safe-tools.mjs picks a tool to call by:

1. a curated map of known-safe read tools per server (e.g. Microsoft Learn Docs → microsoft_docs_search, WorkIQ Teams → ListTeams), then
2. a read-only-name heuristic fallback — the first tool whose name starts with list/get/search/read/find/… and whose required arguments are empty or trivially fillable with benign values.

If nothing looks safe, it does tools/list only and records the call as SKIPPED (tools proven to load, no call made). Expand the curated map in safe-tools.mjs as you learn each server.

Reading the report

Each run prints a summary and writes two files to test/reports/ (gitignored — they contain live endpoint URLs):

• mcp-smoke-<timestamp>.log — human-readable table. This is the handoff artifact — attach it to a bug or send it to whoever needs to repro.
• mcp-smoke-<timestamp>.json — machine-readable, same data.

Per server you get: classification, initialize pass/fail + latency, tool count, which tool was called and why, the call result preview or error, and a direct transport error on failure. API keys are redacted; endpoint URLs are not, which is why the reports stay out of git.

Exit code is non-zero if any probed server failed a step, so it's CI-usable.

Files

file	role
smoke.mjs	orchestrator — bootstrap, classify each server, probe, report
mcp-probe.mjs	drives the native Streamable HTTP JSON-RPC handshake
safe-tools.mjs	curated safe-read-tool map + read-only heuristic + arg filler
reports/	generated .log + .json artifacts (gitignored)

Scope

Microsoft first-party servers only (category === "Microsoft" in the catalog). Partner servers (Box, Celonis, …) are filtered out — they need partner accounts and OAuth we can't automate.