chore: publish from main

2026-07-16 10:53:25 +00:00 · 2026-07-16 00:30:12 +00:00
parent 6a843b5c57
commit d171be5b52
36 changed files with 7664 additions and 0 deletions
@@ -0,0 +1,128 @@
+# MCP smoke test
+
+A standalone harness that proves the Microsoft first-party MCP servers behind a
+connector gateway actually work end-to-end:
+
+```
+connect → initialize → tools/list → a safe tools/call
+```
+
+It imports the `connector-namespaces` extension's real pipeline (`install.mjs`,
+`catalog.mjs`, `armClient.mjs`) and connects through the same native Streamable
+HTTP endpoint that the extension writes to the Copilot CLI config. The probe
+uses the configured `X-API-Key`, follows `Mcp-Session-Id`, and accepts standard
+JSON or SSE JSON-RPC responses.
+
+The whole point: it runs with **Node and Azure CLI**. No Copilot app, no
+canvas, no UI. Hand it to anyone (e.g. Arjun) and they can reproduce an MCP
+server issue locally.
+
+## Prerequisites
+
+1. **Azure CLI signed in with `az login`.** The harness asks Azure CLI for the
+   same short-lived ARM token as the extension.
+2. **A gateway already picked once.** The harness reads gateway coordinates from
+   `~/.copilot/extensions/connector-namespaces/artifacts/gateway-config.json`
+   (`{ subscriptionId, resourceGroup, gatewayName }`). Pick a gateway once in
+   the connector-namespaces canvas, or write that file by hand.
+3. **Node 20+** (developed on Node 24).
+
+## Run it
+
+```bash
+node extensions/connector-namespaces/test/smoke.mjs
+```
+
+Options:
+
+| flag | effect |
+|---|---|
+| `--only=a,b` | only test these `apiName`s (comma-separated) |
+| `--limit=N` | stop after N connectable servers |
+| `--open-consent` | open consent URLs in the browser for OAuth servers that need it |
+| `--no-cleanup` | leave fresh keyless installs in place (default: uninstall them) |
+
+Examples:
+
+```bash
+# just the three already-connected WorkIQ servers
+node extensions/connector-namespaces/test/smoke.mjs --only=WorkIQMail,WorkIQSharePoint,WorkIQTeams
+
+# first 5 connectable servers, open any consent prompts
+node extensions/connector-namespaces/test/smoke.mjs --limit=5 --open-consent
+```
+
+## One-time consent, then headless forever
+
+This is the key behavior. OAuth-backed servers (most of them) need a human to
+consent **once** in a browser. The model:
+
+1. **First run** hits a server that needs consent → the harness prints a consent
+   URL and marks it `NEEDS_CONSENT`. It saves a pending record to
+   `~/.copilot/extensions/connector-namespaces/artifacts/smoke-pending-consent.json` (not in
+   the repo). No tool call is attempted.
+2. **You open that URL once** and sign in / consent. After sign-in the browser
+   may show "this site can't be reached" on a `127.0.0.1:7333/auth/callback/`
+   page — **that is expected and harmless.** Consent completes gateway-side; the
+   loopback page is just a redirect target and nothing is listening on it.
+3. **Re-run the harness.** It sees the pending record, confirms the gateway
+   connection is now `Connected`, finishes the install (mints the API key,
+   writes the CLI entry), and probes it headless. From then on it's reused with
+   zero human interaction.
+
+So the server taxonomy is:
+
+- **Already connected** (e.g. the three WorkIQ servers) → probed immediately.
+- **Keyless / SP / AAD** (e.g. Microsoft Learn Docs) → installed + probed +
+  cleaned up immediately, no consent.
+- **Consent-once OAuth** → surfaced on run 1, converts to headless on run 2.
+
+That's why the **first** run may probe fewer than 10 servers — the rest are
+waiting on their one-time consent. Consent the URLs it prints, re-run, and the
+count climbs. This is inherent to the consent model, not a harness bug.
+
+## Tool-call safety
+
+The harness never blindly calls the first tool a server advertises (mutation
+risk). `safe-tools.mjs` picks a tool to call by:
+
+1. a **curated map** of known-safe read tools per server (e.g. Microsoft Learn
+   Docs → `microsoft_docs_search`, WorkIQ Teams → `ListTeams`), then
+2. a **read-only-name heuristic** fallback — the first tool whose name starts
+   with `list`/`get`/`search`/`read`/`find`/… **and** whose required arguments
+   are empty or trivially fillable with benign values.
+
+If nothing looks safe, it does `tools/list` only and records the call as
+`SKIPPED` (tools proven to load, no call made). Expand the curated map in
+`safe-tools.mjs` as you learn each server.
+
+## Reading the report
+
+Each run prints a summary and writes two files to `test/reports/` (gitignored —
+they contain live endpoint URLs):
+
+- `mcp-smoke-<timestamp>.log` — human-readable table. **This is the handoff
+  artifact** — attach it to a bug or send it to whoever needs to repro.
+- `mcp-smoke-<timestamp>.json` — machine-readable, same data.
+
+Per server you get: classification, `initialize` pass/fail + latency, tool
+count, which tool was called and why, the call result preview or error, and a
+direct transport error on failure. API keys are redacted; endpoint URLs are not,
+which is why the reports stay out of git.
+
+Exit code is **non-zero if any probed server failed a step**, so it's CI-usable.
+
+## Files
+
+| file | role |
+|---|---|
+| `smoke.mjs` | orchestrator — bootstrap, classify each server, probe, report |
+| `mcp-probe.mjs` | drives the native Streamable HTTP JSON-RPC handshake |
+| `safe-tools.mjs` | curated safe-read-tool map + read-only heuristic + arg filler |
+| `reports/` | generated `.log` + `.json` artifacts (gitignored) |
+
+## Scope
+
+Microsoft first-party servers only (`category === "Microsoft"` in the catalog).
+Partner servers (Box, Celonis, …) are filtered out — they need partner accounts
+and OAuth we can't automate.
@@ -0,0 +1,173 @@
+// MCP probe — drives the gateway's native Streamable HTTP endpoint.
+
+import { pickSafeTool } from "./safe-tools.mjs";
+
+const PROTOCOL_VERSION = "2025-06-18";
+
+function parseResponseBody(text, contentType, expectedId) {
+    if (!text.trim()) return null;
+    if (contentType.toLowerCase().includes("text/event-stream")) {
+        const messages = [];
+        for (const event of text.split(/\r?\n\r?\n/)) {
+            const data = event
+                .split(/\r?\n/)
+                .filter((line) => line.startsWith("data:"))
+                .map((line) => line.slice(5).trim())
+                .join("\n");
+            if (data && data !== "[DONE]") messages.push(JSON.parse(data));
+        }
+        return messages.find((message) => message?.id === expectedId) || null;
+    }
+    return JSON.parse(text);
+}
+
+class HttpClient {
+    constructor(url, key) {
+        this.url = url;
+        this.key = key;
+        this.sessionId = null;
+    }
+
+    async post(message, timeoutMs, expectedId = null) {
+        const controller = new AbortController();
+        const timer = setTimeout(() => controller.abort(), timeoutMs);
+        try {
+            const headers = {
+                Accept: "application/json, text/event-stream",
+                "Content-Type": "application/json",
+                "X-API-Key": this.key,
+            };
+            if (this.sessionId) headers["Mcp-Session-Id"] = this.sessionId;
+            const response = await fetch(this.url, {
+                method: "POST",
+                headers,
+                body: JSON.stringify(message),
+                signal: controller.signal,
+            });
+            const nextSessionId = response.headers.get("mcp-session-id");
+            if (nextSessionId) this.sessionId = nextSessionId;
+            const text = await response.text();
+            if (!response.ok) {
+                throw new Error(`HTTP ${response.status}: ${text.slice(0, 300)}`);
+            }
+            return parseResponseBody(text, response.headers.get("content-type") || "", expectedId);
+        } catch (error) {
+            if (error?.name === "AbortError") {
+                throw new Error(`timeout after ${timeoutMs}ms`);
+            }
+            throw error;
+        } finally {
+            clearTimeout(timer);
+        }
+    }
+
+    request(id, method, params = {}, timeoutMs = 30000) {
+        return this.post({ jsonrpc: "2.0", id, method, params }, timeoutMs, id);
+    }
+
+    notify(method, params = {}, timeoutMs = 30000) {
+        return this.post({ jsonrpc: "2.0", method, params }, timeoutMs);
+    }
+}
+
+function summarizeResult(result) {
+    if (!result || typeof result !== "object") return "";
+    if (Array.isArray(result.content)) {
+        const text = result.content
+            .map((content) => (typeof content?.text === "string" ? content.text : ""))
+            .join(" ")
+            .trim();
+        return text.slice(0, 200);
+    }
+    return JSON.stringify(result).slice(0, 200);
+}
+
+// Probe one server end-to-end.
+// server: { apiName, displayName, configName, url, key }
+// -> structured result with per-step pass/fail + latency.
+export async function probe(server) {
+    const out = {
+        apiName: server.apiName,
+        displayName: server.displayName,
+        steps: {
+            initialize: { ok: false },
+            toolsList: { ok: false },
+            toolsCall: { ok: false, status: "pending" },
+        },
+        toolCount: 0,
+        toolNames: [],
+        toolCalled: null,
+        toolSource: null,
+        error: null,
+    };
+
+    const client = new HttpClient(server.url, server.key);
+    try {
+        const t0 = Date.now();
+        const init = await client.request(1, "initialize", {
+            protocolVersion: PROTOCOL_VERSION,
+            capabilities: {},
+            clientInfo: { name: "mcp-smoke", version: "1.0.0" },
+        }, 45000);
+        out.steps.initialize.latencyMs = Date.now() - t0;
+        if (!init) throw new Error("initialize returned no JSON-RPC response");
+        if (init.error) throw new Error(`initialize error: ${JSON.stringify(init.error).slice(0, 300)}`);
+        const info = init.result?.serverInfo;
+        if (!init.result || !(info || init.result.protocolVersion)) {
+            throw new Error("initialize returned no serverInfo / protocolVersion");
+        }
+        out.steps.initialize.ok = true;
+        out.serverInfo = info ? `${info.name || "?"}@${info.version || "?"}` : "(no serverInfo)";
+
+        await client.notify("notifications/initialized");
+
+        const t1 = Date.now();
+        const list = await client.request(2, "tools/list", {}, 30000);
+        out.steps.toolsList.latencyMs = Date.now() - t1;
+        if (!list) throw new Error("tools/list returned no JSON-RPC response");
+        if (list.error) throw new Error(`tools/list error: ${JSON.stringify(list.error).slice(0, 300)}`);
+        const tools = list.result?.tools || [];
+        out.toolCount = tools.length;
+        out.toolNames = tools.map((tool) => tool?.name).filter(Boolean);
+        if (tools.length < 1) throw new Error("tools/list returned 0 tools");
+        out.steps.toolsList.ok = true;
+
+        const pick = pickSafeTool(server, tools);
+        if (!pick || pick.skip) {
+            out.steps.toolsCall.status = "skipped";
+            out.steps.toolsCall.ok = true;
+            out.steps.toolsCall.note = pick?.reason || "no safe read-only tool found";
+            out.toolSource = pick?.source || null;
+        } else {
+            out.toolCalled = pick.tool;
+            out.toolSource = pick.source;
+            const t2 = Date.now();
+            const call = await client.request(3, "tools/call", { name: pick.tool, arguments: pick.args }, 45000);
+            out.steps.toolsCall.latencyMs = Date.now() - t2;
+            if (!call) {
+                out.steps.toolsCall.status = "failed";
+                out.steps.toolsCall.error = "tools/call returned no JSON-RPC response";
+            } else if (call.error) {
+                out.steps.toolsCall.status = "failed";
+                out.steps.toolsCall.error = JSON.stringify(call.error).slice(0, 300);
+            } else if (call.result?.isError) {
+                out.steps.toolsCall.status = "failed";
+                out.steps.toolsCall.error = summarizeResult(call.result);
+            } else {
+                out.steps.toolsCall.status = "passed";
+                out.steps.toolsCall.ok = true;
+                out.steps.toolsCall.result = "response received";
+            }
+        }
+    } catch (error) {
+        out.error = error.message;
+    }
+
+    out.ok = out.steps.initialize.ok && out.steps.toolsList.ok && out.steps.toolsCall.ok && !out.error;
+    out.status = !out.ok
+        ? "failed"
+        : out.steps.toolsCall.status === "skipped"
+            ? "skipped"
+            : "passed";
+    return out;
+}
@@ -0,0 +1,168 @@
+// Safe read-tool selection for the MCP smoke harness.
+//
+// We never blindly call the first tool a server advertises — many servers lead
+// with mutating tools (SendMail, CreateItem, DeleteMessage). This module picks a
+// tool that is safe to call automatically: read-only by name, with an input
+// schema we can satisfy using benign placeholder values. If nothing qualifies,
+// the caller records the server as SKIPPED (tools proven to load, no call made).
+
+// Curated overrides keyed by a predicate over the server identity. Use this when
+// the heuristic would skip a server that actually has a known-safe read tool, or
+// when we want to pin a specific tool/argument set. `tool` must match a tool name
+// from tools/list; `args` is merged over the auto-filled args.
+const CURATED = [
+    {
+        name: "Microsoft Learn Docs",
+        match: (s) => /learn|docs/i.test(s.apiName) || /learn|docs/i.test(s.displayName),
+        tool: "microsoft_docs_search",
+        args: { query: "azure connectors", question: "what are azure connectors" },
+    },
+    {
+        name: "WorkIQ Teams",
+        match: (s) => /teams/i.test(s.apiName) || /teams/i.test(s.displayName),
+        tool: "ListTeams",
+        args: {},
+    },
+    {
+        // SearchMessagesQueryParameters declares queryParameters optional, but the
+        // server enforces "queryParameters OR nextLink required" — a cross-field
+        // rule the JSON schema doesn't express, so the no-arg heuristic call fails.
+        // Pin a benign read-only OData query that returns at most one message.
+        name: "WorkIQ Mail",
+        match: (s) => /outlookmail/i.test(s.apiName) || /mail/i.test(s.displayName),
+        tool: "SearchMessagesQueryParameters",
+        args: { queryParameters: "?$top=1" },
+    },
+    {
+        // copilot_chat is the only tool Work IQ Copilot exposes. Its name doesn't
+        // match the read-only heuristic, so without this pin the harness would skip
+        // the call. A benign one-sentence question is a safe read-style call.
+        name: "WorkIQ Copilot",
+        match: (s) => /copilotchat/i.test(s.apiName) || /copilot/i.test(s.displayName),
+        tool: "copilot_chat",
+        args: { message: "What is Microsoft Azure? Answer in one sentence." },
+    },
+    {
+        // Word advertises CreateDocument/AddComment/ReplyToComment (all mutating)
+        // and GetDocumentContent, which needs a real document on a drive this
+        // tenant can't resolve ("Invalid hostname for this tenancy" — a backend
+        // config issue, not an argument problem). No safe no-arg read tool exists,
+        // so prove the tools load and skip the call instead of a false failure.
+        name: "WorkIQ Word",
+        match: (s) => /wordmcp/i.test(s.apiName) || /^word\b/i.test(s.displayName || ""),
+        skip: true,
+        reason: "no safe no-arg read tool (GetDocumentContent needs a real document + a tenant drive)",
+    },
+];
+
+const READ_ONLY_NAME = /^(list|get|search|read|find|describe|show|fetch|lookup|query|count)/i;
+
+// Names that look read-only but are known to mutate or are too risky to auto-run.
+const DENY_NAME = /(send|create|update|delete|remove|add|set|post|put|patch|write|upload|move|copy|rename|share|invite|reply|forward|draft|flag|ingest|execute|run|trigger)/i;
+
+function isFillableSchema(schema) {
+    if (!schema || typeof schema !== "object") return true;
+    const required = Array.isArray(schema.required) ? schema.required : [];
+    const props = schema.properties || {};
+    for (const key of required) {
+        const prop = props[key];
+        if (!prop) return false; // required but undescribed → can't satisfy safely
+        if (!fillValue(prop, key).ok) return false;
+    }
+    return true;
+}
+
+// Produce a benign placeholder for a single property schema.
+function fillValue(prop, key) {
+    if (!prop || typeof prop !== "object") return { ok: true, value: "test" };
+    if (prop.default !== undefined) return { ok: true, value: prop.default };
+    if (Array.isArray(prop.enum) && prop.enum.length > 0) return { ok: true, value: prop.enum[0] };
+
+    const type = Array.isArray(prop.type) ? prop.type.find((t) => t !== "null") : prop.type;
+    switch (type) {
+        case "string": {
+            if (/mail|email|upn|recipient|to|address/i.test(key)) return { ok: true, value: "test@example.com" };
+            if (/url|uri|link/i.test(key)) return { ok: true, value: "https://example.com" };
+            if (/query|search|q|term|keyword|text|question|prompt/i.test(key)) return { ok: true, value: "azure" };
+            return { ok: true, value: "test" };
+        }
+        case "number":
+        case "integer":
+            return { ok: true, value: 1 };
+        case "boolean":
+            return { ok: true, value: false };
+        default:
+            // object/array/unknown required input → we can't safely fabricate it.
+            return { ok: false };
+    }
+}
+
+// Build the argument object for a tool from its required input schema.
+function buildArgs(tool) {
+    const schema = tool.inputSchema || tool.input_schema || {};
+    const required = Array.isArray(schema.required) ? schema.required : [];
+    const props = schema.properties || {};
+    const args = {};
+    for (const key of required) {
+        const filled = fillValue(props[key] || {}, key);
+        if (!filled.ok) return { ok: false };
+        args[key] = filled.value;
+    }
+    return { ok: true, args };
+}
+
+// Pick a safe tool to call for a server.
+// server: { apiName, displayName, configName }
+// tools:  the array returned by tools/list
+// → { tool, args, source } | null   (null means "no safe tool, record SKIPPED")
+export function pickSafeTool(server, tools) {
+    if (!Array.isArray(tools) || tools.length === 0) return null;
+    const byName = new Map(tools.map((t) => [t.name, t]));
+
+    // 1. Curated override: a skip directive, or a pinned tool if it's advertised.
+    for (const entry of CURATED) {
+        if (!entry.match(server)) continue;
+        if (entry.skip) {
+            return { skip: true, reason: entry.reason || "curated skip", source: "curated-skip" };
+        }
+        if (byName.has(entry.tool)) {
+            const tool = byName.get(entry.tool);
+            const built = buildArgs(tool);
+            const base = built.ok ? built.args : {};
+            return { tool: entry.tool, args: { ...base, ...entry.args }, source: "curated" };
+        }
+    }
+
+    // 2. Heuristic. Two passes, preferring tools we can call with NO fabricated
+    //    arguments — a required `id`/`resourceId` filled with a placeholder will
+    //    pass the schema check but fail at runtime (fake message id, fake
+    //    resource). A read tool with no required args is both safer and far more
+    //    likely to actually succeed, so try those first.
+    const candidate = (tool, requireEmpty) => {
+        const nm = tool.name || "";
+        const annotations = tool.annotations || {};
+        if (annotations.readOnlyHint !== true || annotations.destructiveHint === true) return null;
+        if (!READ_ONLY_NAME.test(nm)) return null;
+        if (DENY_NAME.test(nm)) return null;
+        const schema = tool.inputSchema || tool.input_schema || {};
+        const required = Array.isArray(schema.required) ? schema.required : [];
+        if (requireEmpty && required.length > 0) return null;
+        if (!isFillableSchema(schema)) return null;
+        const built = buildArgs(tool);
+        if (!built.ok) return null;
+        return { tool: nm, args: built.args, source: requireEmpty ? "heuristic-noargs" : "heuristic" };
+    };
+
+    for (const tool of tools) {
+        const pick = candidate(tool, true);
+        if (pick) return pick;
+    }
+    for (const tool of tools) {
+        const pick = candidate(tool, false);
+        if (pick) return pick;
+    }
+
+    return null;
+}
+
+export const _internals = { buildArgs, isFillableSchema, fillValue, READ_ONLY_NAME, DENY_NAME };
@@ -0,0 +1,150 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+
+import { pickSafeTool, _internals } from "./safe-tools.mjs";
+
+test("curated servers pin known-safe tools and arguments", () => {
+    const learn = pickSafeTool(
+        { apiName: "learn", displayName: "Microsoft Learn" },
+        [{
+            name: "microsoft_docs_search",
+            inputSchema: {
+                type: "object",
+                required: ["query", "question"],
+                properties: {
+                    query: { type: "string" },
+                    question: { type: "string" },
+                },
+            },
+        }],
+    );
+    assert.deepEqual(learn, {
+        tool: "microsoft_docs_search",
+        args: { query: "azure connectors", question: "what are azure connectors" },
+        source: "curated",
+    });
+
+    const mail = pickSafeTool(
+        { apiName: "outlookmail", displayName: "Mail" },
+        [{ name: "SearchMessagesQueryParameters", inputSchema: { type: "object", properties: {} } }],
+    );
+    assert.deepEqual(mail, {
+        tool: "SearchMessagesQueryParameters",
+        args: { queryParameters: "?$top=1" },
+        source: "curated",
+    });
+});
+
+test("curated unsafe servers are skipped without selecting a tool", () => {
+    const result = pickSafeTool(
+        { apiName: "wordmcp", displayName: "Word" },
+        [{ name: "GetDocumentContent", inputSchema: { type: "object", required: ["documentId"] } }],
+    );
+    assert.equal(result.skip, true);
+    assert.equal(result.source, "curated-skip");
+    assert.match(result.reason, /no safe no-arg read tool/);
+});
+
+test("deny-list terms beat read-looking prefixes", () => {
+    const result = pickSafeTool(
+        { apiName: "generic", displayName: "Generic" },
+        [
+            { name: "GetAndDeleteMessage", annotations: { readOnlyHint: true }, inputSchema: { type: "object" } },
+            { name: "ListAndSendMail", annotations: { readOnlyHint: true }, inputSchema: { type: "object" } },
+            { name: "SearchRecords", annotations: { readOnlyHint: true }, inputSchema: { type: "object" } },
+        ],
+    );
+    assert.deepEqual(result, { tool: "SearchRecords", args: {}, source: "heuristic-noargs" });
+});
+
+test("schema filling uses benign values and rejects complex required input", () => {
+    const built = _internals.buildArgs({
+        inputSchema: {
+            required: ["recipientEmail", "resourceUrl", "query", "count", "enabled", "kind", "preset"],
+            properties: {
+                recipientEmail: { type: "string" },
+                resourceUrl: { type: "string" },
+                query: { type: "string" },
+                count: { type: "integer" },
+                enabled: { type: "boolean" },
+                kind: { type: "string", enum: ["summary", "full"] },
+                preset: { type: "string", default: "safe" },
+            },
+        },
+    });
+    assert.deepEqual(built, {
+        ok: true,
+        args: {
+            recipientEmail: "test@example.com",
+            resourceUrl: "https://example.com",
+            query: "azure",
+            count: 1,
+            enabled: false,
+            kind: "summary",
+            preset: "safe",
+        },
+    });
+    assert.deepEqual(
+        _internals.buildArgs({
+            input_schema: {
+                required: ["payload"],
+                properties: { payload: { type: "object" } },
+            },
+        }),
+        { ok: false },
+    );
+});
+
+test("heuristics prefer no-argument reads and return null when none are safe", () => {
+    const preferred = pickSafeTool(
+        { apiName: "generic", displayName: "Generic" },
+        [
+            {
+                name: "GetRecord",
+                inputSchema: {
+                    required: ["query"],
+                    properties: { query: { type: "string" } },
+                },
+            },
+            { name: "ListRecords", annotations: { readOnlyHint: true }, inputSchema: { type: "object" } },
+        ],
+    );
+    assert.deepEqual(preferred, { tool: "ListRecords", args: {}, source: "heuristic-noargs" });
+    assert.equal(
+        pickSafeTool(
+            { apiName: "generic", displayName: "Generic" },
+            [
+                { name: "CreateRecord", inputSchema: { type: "object" } },
+                {
+                    name: "GetRecord",
+                    inputSchema: {
+                        required: ["payload"],
+                        properties: { payload: { type: "array" } },
+                    },
+                },
+            ],
+        ),
+        null,
+    );
+    assert.equal(pickSafeTool({ apiName: "generic", displayName: "Generic" }, []), null);
+});
+
+test("heuristics require an explicit read-only non-destructive annotation", () => {
+    assert.equal(
+        pickSafeTool(
+            { apiName: "generic", displayName: "Generic" },
+            [
+                { name: "GetAndArchiveMessage", inputSchema: { type: "object" } },
+                { name: "ListAndApproveRequests", inputSchema: { type: "object" } },
+            ],
+        ),
+        null,
+    );
+    assert.equal(
+        pickSafeTool(
+            { apiName: "generic", displayName: "Generic" },
+            [{ name: "ListRecords", annotations: { readOnlyHint: true, destructiveHint: true }, inputSchema: { type: "object" } }],
+        ),
+        null,
+    );
+});
@@ -0,0 +1,328 @@
+// MCP smoke-test orchestrator.
+//
+// Standalone harness that proves the Microsoft first-party MCP servers behind a
+// connector gateway actually work end-to-end: connect -> initialize ->
+// tools/list -> a safe tools/call. It imports the extension's real pipeline
+// (install.mjs, catalog.mjs, armClient.mjs) and connects through the same native
+// Streamable HTTP endpoint persisted for the Copilot CLI.
+//
+// Runs with `node` and a signed-in Azure CLI — no Copilot app required — so it
+// can be handed to someone else to reproduce MCP issues. See README.md.
+//
+// Usage:
+//   node extensions/connector-namespaces/test/smoke.mjs [options]
+//
+// Options:
+//   --only=a,b      only test these apiNames (comma-separated)
+//   --limit=N       stop after N connectable servers
+//   --open-consent  open consent URLs in the browser for servers that need it
+//   --no-cleanup    do not uninstall fresh keyless installs afterward
+
+import { readFileSync, writeFileSync, existsSync, mkdirSync, chmodSync } from "node:fs";
+import { join, dirname } from "node:path";
+import { homedir } from "node:os";
+import { fileURLToPath } from "node:url";
+
+import { loadSavedConfig } from "../state.mjs";
+import { getToken } from "../armClient.mjs";
+import { CATEGORY } from "../categories.mjs";
+import {
+    installConnector,
+    finishInstall,
+    getInstalledState,
+    getConnectionStatus,
+    getMcpEndpointUrl,
+    mintApiKey,
+    uninstallConnector,
+    openInBrowser,
+    assertSafeMcpTarget,
+} from "../install.mjs";
+import { fetchCatalog } from "../catalog.mjs";
+import { probe } from "./mcp-probe.mjs";
+
+const HERE = dirname(fileURLToPath(import.meta.url));
+const REPORTS_DIR = join(HERE, "reports");
+const PROFILE_MCP_PATH = join(process.env.COPILOT_HOME || join(homedir(), ".copilot"), "mcp-config.json");
+const ARTIFACTS_DIR = join(process.env.COPILOT_HOME || join(homedir(), ".copilot"), "extensions", "connector-namespaces", "artifacts");
+const PENDING_FILE = join(ARTIFACTS_DIR, "smoke-pending-consent.json");
+
+// The loopback callback never has to be listening — gateway-side consent
+// completes in the browser; this URL is only embedded in the redirect.
+const CALLBACK_BASE = "http://127.0.0.1:7333/auth/callback/";
+
+function parseArgs(argv) {
+    const opts = { only: null, limit: Infinity, openConsent: false, cleanup: true };
+    for (const a of argv) {
+        if (a.startsWith("--only=")) opts.only = new Set(a.slice(7).split(",").map((s) => s.trim()).filter(Boolean));
+        else if (a.startsWith("--limit=")) {
+            const limit = Number.parseInt(a.slice(8), 10);
+            opts.limit = Number.isNaN(limit) ? Infinity : limit;
+        }
+        else if (a === "--open-consent") opts.openConsent = true;
+        else if (a === "--no-cleanup") opts.cleanup = false;
+    }
+    return opts;
+}
+
+function readPending() {
+    try {
+        if (existsSync(PENDING_FILE)) return JSON.parse(readFileSync(PENDING_FILE, "utf-8"));
+    } catch { /* ignore corrupt */ }
+    return {};
+}
+
+function writePending(map) {
+    if (!existsSync(ARTIFACTS_DIR)) mkdirSync(ARTIFACTS_DIR, { recursive: true, mode: 0o700 });
+    chmodSync(ARTIFACTS_DIR, 0o700);
+    if (existsSync(PENDING_FILE)) chmodSync(PENDING_FILE, 0o600);
+    writeFileSync(PENDING_FILE, JSON.stringify(map, null, 2), { encoding: "utf-8", mode: 0o600 });
+    chmodSync(PENDING_FILE, 0o600);
+}
+
+// Read the native HTTP MCP credentials from the CLI config.
+function credsFromCli(configName) {
+    try {
+        const cfg = JSON.parse(readFileSync(PROFILE_MCP_PATH, "utf-8"));
+        const entry = cfg.mcpServers?.[configName];
+        if (entry?.url && entry?.headers?.["X-API-Key"]) {
+            return { url: entry.url, key: entry.headers["X-API-Key"] };
+        }
+    } catch { /* ignore */ }
+    return null;
+}
+
+// Resolve url+key for an installed+connected server, minting a key if the CLI
+// entry is missing (e.g. installed at the gateway but not added to the CLI).
+async function resolveCreds(config, state) {
+    const fromCli = credsFromCli(state.configName);
+    if (fromCli) return fromCli;
+    const url = await getMcpEndpointUrl(config, state.configName);
+    if (!url) return null;
+    const key = await mintApiKey(config, state.configName);
+    return { url, key };
+}
+
+function redact(text) {
+    if (typeof text !== "string") return text;
+    // Redact anything that looks like a gateway API key in URLs or headers.
+    return text.replace(/([?&](?:key|code|api[-_]?key)=)[^&\s"]+/gi, "$1<redacted>");
+}
+
+// Collapse CR/LF so user/network-derived error text can't forge extra log
+// lines (CodeQL js/log-injection). Kept separate from redact() so the
+// pretty-printed JSON report still retains its newlines.
+function logLine(text) {
+    return redact(String(text)).replace(/[\r\n]+/g, " ");
+}
+
+const C = {
+    reset: "\x1b[0m", dim: "\x1b[2m", bold: "\x1b[1m",
+    green: "\x1b[32m", red: "\x1b[31m", yellow: "\x1b[33m", cyan: "\x1b[36m",
+};
+const tick = (ok) => (ok ? `${C.green}PASS${C.reset}` : `${C.red}FAIL${C.reset}`);
+
+async function main() {
+    const opts = parseArgs(process.argv.slice(2));
+
+    // 1. Bootstrap: gateway coords + ARM token (fail fast).
+    const config = loadSavedConfig();
+    if (!config?.subscriptionId || !config?.resourceGroup || !config?.gatewayName) {
+        console.error(`${C.red}No gateway config found.${C.reset} Expected ${join(ARTIFACTS_DIR, "gateway-config.json")}.`);
+        console.error("Pick a gateway once in the connector-namespaces canvas, or create that file with { subscriptionId, resourceGroup, gatewayName }.");
+        process.exit(2);
+    }
+    try {
+        await getToken();
+    } catch (err) {
+        console.error(`${C.red}Could not get an ARM token.${C.reset} Sign in to Azure when the browser opens.`);
+        console.error(String(err.message || err).slice(0, 300));
+        process.exit(2);
+    }
+
+    console.log(`${C.bold}MCP smoke test${C.reset}  gateway=${C.cyan}${config.gatewayName}${C.reset}  rg=${config.resourceGroup}`);
+
+    // 2. Target set: Microsoft first-party MCP servers only.
+    const catalog = await fetchCatalog(config.subscriptionId, config.resourceGroup, config.gatewayName);
+    let servers = catalog.filter((s) => s.category === CATEGORY.microsoft);
+    if (opts.only) servers = servers.filter((s) => opts.only.has(s.apiName));
+    console.log(`${C.dim}${servers.length} Microsoft servers in catalog${C.reset}\n`);
+
+    const installedState = await getInstalledState(config);
+    const pending = readPending();
+
+    const results = [];
+    let connectable = 0;
+
+    for (const server of servers) {
+        if (connectable >= opts.limit) break;
+        const label = server.displayName || server.apiName;
+        process.stdout.write(`${C.bold}${label}${C.reset} ${C.dim}(${server.apiName})${C.reset}\n`);
+
+        const record = { apiName: server.apiName, displayName: label, classification: null, probe: null, cleanup: false };
+        let creds = null;
+
+        try {
+            const state = installedState[server.apiName];
+            const pend = pending[server.apiName];
+
+            if (pend) {
+                // We surfaced a consent URL on a previous run — check if it's done now.
+                const status = await getConnectionStatus(config, pend.connName);
+                if (status === "Connected") {
+                    console.log(`  ${C.dim}consent completed, finishing install...${C.reset}`);
+                    const fin = await finishInstall(config, server.apiName, label, pend.connName, pend.location);
+                    creds = credsFromCli(fin.configName) || { url: fin.endpointUrl, key: null };
+                    record.classification = "consented-now";
+                    delete pending[server.apiName];
+                    writePending(pending);
+                } else {
+                    record.classification = "pending-consent";
+                    console.log(`  ${C.yellow}PENDING_CONSENT${C.reset} still ${status}. Consent: ${pend.consentUrl}`);
+                    if (opts.openConsent) await openInBrowser(pend.consentUrl);
+                    results.push(record);
+                    continue;
+                }
+            } else if (state?.installed && state.connectionStatus === "Connected") {
+                record.classification = "probe-only";
+                creds = await resolveCreds(config, state);
+            } else if (state?.installed) {
+                record.classification = "installed-not-connected";
+                console.log(`  ${C.yellow}SKIP${C.reset} installed but connection is ${state.connectionStatus}`);
+                results.push(record);
+                continue;
+            } else {
+                // Not installed — try a fresh install.
+                const res = await installConnector(config, server.apiName, label, CALLBACK_BASE);
+                if (res?.ok) {
+                    record.classification = "fresh-install";
+                    record.cleanup = opts.cleanup;
+                    creds = credsFromCli(res.configName) || { url: res.endpointUrl, key: null };
+                } else if (res?.needsConsent) {
+                    record.classification = "needs-consent";
+                    pending[server.apiName] = {
+                        connName: res.connName, location: res.location,
+                        displayName: label, consentUrl: res.consentUrl, savedAt: Date.now(),
+                    };
+                    writePending(pending);
+                    console.log(`  ${C.yellow}NEEDS_CONSENT${C.reset} consent once, then re-run. URL:\n    ${res.consentUrl}`);
+                    if (opts.openConsent) await openInBrowser(res.consentUrl);
+                    results.push(record);
+                    continue;
+                } else {
+                    throw new Error("installConnector returned neither ok nor needsConsent");
+                }
+            }
+
+            if (!creds?.url || !creds?.key) {
+                record.error = "could not resolve url+key for probe";
+                console.log(`  ${C.red}FAIL${C.reset} ${record.error}`);
+                results.push(record);
+                continue;
+            }
+            assertSafeMcpTarget(creds.url);
+
+            connectable++;
+            const r = await probe({ ...server, displayName: label, url: creds.url, key: creds.key });
+            record.probe = r;
+
+            const callLine = r.steps.toolsCall.status === "skipped"
+                ? `${C.yellow}SKIPPED${C.reset}`
+                : `${tick(r.steps.toolsCall.ok)}${r.toolCalled ? ` ${C.dim}${r.toolCalled}${C.reset}` : ""}`;
+            console.log(`  init ${tick(r.steps.initialize.ok)}  tools/list ${tick(r.steps.toolsList.ok)} ${C.dim}(${r.toolCount})${C.reset}  tools/call ${callLine}`);
+            if (r.error) console.log(`  ${C.red}${logLine(r.error)}${C.reset}`);
+
+        } catch (err) {
+            record.error = String(err.message || err);
+            console.log(`  ${C.red}ERROR${C.reset} ${logLine(record.error).slice(0, 300)}`);
+        } finally {
+            if (record.cleanup) {
+                try {
+                    await uninstallConnector(config, server.apiName);
+                    console.log(`  ${C.dim}cleaned up fresh install${C.reset}`);
+                } catch (err) {
+                    record.cleanupError = String(err.message || err);
+                    console.log(`  ${C.red}CLEANUP ERROR${C.reset} ${logLine(record.cleanupError).slice(0, 300)}`);
+                }
+            }
+        }
+
+        results.push(record);
+        console.log("");
+    }
+
+    // 3. Summary + report files.
+    const probed = results.filter((r) => r.probe);
+    const passed = probed.filter((r) => r.probe.status === "passed");
+    const failed = probed.filter((r) => r.probe.status === "failed");
+    const safeCallSkipped = probed.filter((r) => r.probe.status === "skipped");
+    const orchestrationErrors = results.filter((r) => r.error || r.cleanupError);
+    const needsConsent = results.filter((r) => r.classification === "needs-consent" || r.classification === "pending-consent");
+    const skipped = [...results.filter((r) => !r.probe && !needsConsent.includes(r)), ...safeCallSkipped];
+
+    console.log(`${C.bold}Summary${C.reset}`);
+    console.log(`  probed:        ${probed.length}`);
+    console.log(`  ${C.green}passed:        ${passed.length}${C.reset}`);
+    console.log(`  ${C.red}failed:        ${failed.length}${C.reset}`);
+    console.log(`  ${C.red}errors:        ${orchestrationErrors.length}${C.reset}`);
+    console.log(`  ${C.yellow}needs consent: ${needsConsent.length}${C.reset}`);
+    console.log(`  skipped:       ${skipped.length}`);
+    if (needsConsent.length) {
+        console.log(`\n  ${C.yellow}Consent once for these, then re-run to test headless:${C.reset}`);
+        for (const r of needsConsent) console.log(`    - ${r.displayName} (${r.apiName})`);
+    }
+
+    if (!existsSync(REPORTS_DIR)) mkdirSync(REPORTS_DIR, { recursive: true, mode: 0o700 });
+    chmodSync(REPORTS_DIR, 0o700);
+    const ts = new Date().toISOString().replace(/[:.]/g, "-");
+    const jsonPath = join(REPORTS_DIR, `mcp-smoke-${ts}.json`);
+    const logPath = join(REPORTS_DIR, `mcp-smoke-${ts}.log`);
+
+    const report = {
+        timestamp: new Date().toISOString(),
+        gateway: { gatewayName: config.gatewayName, resourceGroup: config.resourceGroup },
+        totals: { probed: probed.length, passed: passed.length, failed: failed.length, errors: orchestrationErrors.length, needsConsent: needsConsent.length, skipped: skipped.length },
+        servers: results,
+    };
+    writeFileSync(jsonPath, redact(JSON.stringify(report, null, 2)), { encoding: "utf-8", mode: 0o600 });
+    writeFileSync(logPath, redact(renderLog(report)), { encoding: "utf-8", mode: 0o600 });
+    chmodSync(jsonPath, 0o600);
+    chmodSync(logPath, 0o600);
+
+    console.log(`\n  report: ${logPath}`);
+    console.log(`  json:   ${jsonPath}`);
+
+    // CI-friendly: non-zero if a probe or orchestration step failed.
+    process.exit(failed.length > 0 || orchestrationErrors.length > 0 ? 1 : 0);
+}
+
+function renderLog(report) {
+    const lines = [];
+    lines.push(`MCP smoke test  ${report.timestamp}`);
+    lines.push(`gateway: ${report.gateway.gatewayName}  rg: ${report.gateway.resourceGroup}`);
+    lines.push("");
+    for (const s of report.servers) {
+        lines.push(`## ${s.displayName} (${s.apiName})`);
+        lines.push(`   classification: ${s.classification}`);
+        if (s.error) lines.push(`   error: ${s.error}`);
+        if (s.cleanupError) lines.push(`   cleanupError: ${s.cleanupError}`);
+        if (s.probe) {
+            const p = s.probe;
+            lines.push(`   serverInfo: ${p.serverInfo || "-"}`);
+            lines.push(`   initialize: ${p.steps.initialize.ok ? "PASS" : "FAIL"} (${p.steps.initialize.latencyMs ?? "-"}ms)`);
+            lines.push(`   tools/list: ${p.steps.toolsList.ok ? "PASS" : "FAIL"} — ${p.toolCount} tools`);
+            const tc = p.steps.toolsCall;
+            lines.push(`   tools/call: ${tc.status}${p.toolCalled ? ` [${p.toolCalled}, ${p.toolSource}]` : ""}`);
+            if (tc.error) lines.push(`     callError: ${tc.error}`);
+            if (p.error) lines.push(`   probeError: ${p.error}`);
+        }
+        lines.push("");
+    }
+    const t = report.totals;
+    lines.push(`SUMMARY  probed=${t.probed} passed=${t.passed} failed=${t.failed} errors=${t.errors} needsConsent=${t.needsConsent} skipped=${t.skipped}`);
+    return lines.join("\n");
+}
+
+main().catch((err) => {
+    console.error(`${C.red}Fatal:${C.reset} ${err.stack || err}`);
+    process.exit(2);
+});