mirror of
https://github.com/github/awesome-copilot.git
synced 2026-08-13 20:59:12 +00:00
Add the WebMCPify agent skill 🤖🤖🤖 (#2400)
* feat(skills): add webmcpify skill * style(skills): quote webmcpify description
This commit is contained in:
@@ -0,0 +1,69 @@
|
||||
# Heal — failure taxonomy → fixes
|
||||
|
||||
Work one failed tool at a time. Re-verify after each fix. The triggering verify
|
||||
failure counts as attempt 0; each fix cycle increments `attempts`. At `attempts`
|
||||
= 3 → mark `skipped` with a blocker note (this is an explicit escalation to the
|
||||
human in the final report, not a silent drop) and move on. **Never** widen the
|
||||
diff, disable a check, or fake a return value to force a pass. **Mutating
|
||||
tools:** run the manifest `cleanup` between attempts — retrying a mutation
|
||||
without cleanup duplicates data.
|
||||
|
||||
**Heal fixes implementations, not contracts.** The manifest is the
|
||||
human-approved contract: if the correct fix would change a tool's `inputSchema`,
|
||||
`description`, `mutating` class, `annotations`, or `expect`, take it back to the
|
||||
gate as a mini re-approval — never silently edit the manifest to match the code.
|
||||
|
||||
## Taxonomy
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| Tool absent from enumeration | **Registration is async** — the test asserted before `registerTool()` settled; or registration never ran (bootstrap not reached, view not mounted) or wrong Chrome build/flags | FIRST make the test poll (`waitForTool`) or await `toolchange` — only if it still fails, trace the registration call; confirm `isWebMCPAvailable()` in the test env; current Chrome + `--enable-features=WebMCP,WebMCPTesting` |
|
||||
| Whole scope absent | A registration in the batch rejected (duplicate name, invalid schema, policy) — the runtime rolls back the entire scope | Check console for the `onError` report; fix the offending tool contract |
|
||||
| Tool absent after route change | Scope disposed by navigation (over-scoping) | Move to static app-level registration unless genuinely view-bound |
|
||||
| Declarative tool missing | `toolname` typo, frame without `allow="tools"`, or page sends `Origin-Agent-Cluster: ?0` | Fix attribute; check Permissions-Policy `tools` and origin-keying headers |
|
||||
| Schema mismatch (declarative) | Control lacks `name`, description not resolvable, unsupported control type in this build | Add `name`/`toolparamdescription`/`label[for]`; unsupported controls → switch that form to imperative |
|
||||
| Schema mismatch (imperative) | Manifest and code drifted | Make code match the approved manifest; if the manifest was wrong, that's a contract change — take it back to the gate for re-approval (see above), never silently update it |
|
||||
| Assertion compares object to string | Enumerated `inputSchema` is a stringified JSON Schema | `JSON.parse` before comparing (see `verify.md`) |
|
||||
| `executeTool` returns `null` unexpectedly | The execution navigated (normal for submit-navigating declarative forms) | Assert on the post-navigation page instead of the return value |
|
||||
| `executeTool` rejects | Schema violation or declarative-validation failure — rejection IS the failure signal for these | For invalid-input tests on declarative tools, assert rejection, not an `"ERROR:"` string |
|
||||
| Mutating declarative execution hangs until timeout | Chrome fills the form, then **pauses the execution awaiting a real submit interaction** — awaiting `executeTool` alone deadlocks | Use the concurrent pattern in the spec template: start `executeTool` unawaited → wait for the agent-filled value → click submit → await. **NEVER heal by adding `toolautosubmit`** (ground rule 5) |
|
||||
| Backend rejects the harness with 403/CORS despite correct auth | The endpoint **allow-lists the production `Origin`** (mailers, form gateways) — the localhost harness origin is refused before the tool logic runs, and no local fix exists | Verify the live path with the env-gated server-side replay (§Origin-allow-listed endpoints below), only with the production side-effect approval recorded in `approval.productionSideEffect` (see §Origin-allow-listed endpoints below); without it, mark the live path `skipped` with a blocker note |
|
||||
| Execution times out / canned success while UI still loading | Completion event fired before the async work finished, or listener missing/wrong event name | Fire `tool-completion-<requestId>` with `{ ok, message/error }` AFTER awaiting the real work (`runtime.md` contract) |
|
||||
| Returns success but UI unchanged | `execute()` bypassed the real UI path (parallel implementation) | Rewrite to call the same handler/store action/endpoint the UI uses |
|
||||
| Invalid input resolves successfully (imperative) | Missing in-code validation | Validate strictly in code; return `"ERROR: <what/how to fix>"` |
|
||||
| Fetch-submitted form: agent gets nothing | `preventDefault()` without `respondWith()` | Add the `e.agentInvoked → e.respondWith(promise)` bridge |
|
||||
| Works manually, fails in Playwright | Headless, missing flags, or profile without the flag | Headed + flags; persistent context; `xvfb-run` in CI |
|
||||
| 401/403 from `execute()` in test | Tool registered outside the authenticated scope, or test session lacks the role in the manifest `auth` field | Role-scope the registration; sign in with the recorded fixture |
|
||||
| Flaky: passes alone, fails in suite | Shared state between tool executions | Isolate test data per tool run (use `cleanup`); don't reorder tests to hide it |
|
||||
|
||||
## Origin-allow-listed endpoints — the replay pattern
|
||||
|
||||
Some production backends (mailers, form gateways) allow-list the production
|
||||
`Origin` header and refuse everything else — the localhost harness can never
|
||||
exercise the live path directly. When (and only when) the gate approved the real
|
||||
production side effect (`approval.productionSideEffect`), verify the live path
|
||||
with an env-gated replay: intercept the app's own request in Playwright and
|
||||
re-issue it server-side (Node context — not subject to browser CORS) with the
|
||||
production `Origin`:
|
||||
|
||||
```ts
|
||||
// Env-gated: runs only with WEBMCP_LIVE_MUTATIONS=1 — never default-on in CI.
|
||||
if (process.env.WEBMCP_LIVE_MUTATIONS === '1') {
|
||||
await page.route('**/api/contact', async (route) => {
|
||||
const response = await context.request.fetch(route.request(), {
|
||||
headers: { ...route.request().headers(), origin: 'https://example.com' }, // the prod Origin
|
||||
});
|
||||
await route.fulfill({ response });
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
This causes a REAL production side effect. Mark every payload
|
||||
`[webmcpify verification]`, run the manifest `cleanup`, list the effect in
|
||||
`report.md`, and never enable the gate by default in CI.
|
||||
|
||||
## After healing
|
||||
|
||||
Re-run verification once for **all** tools with status `integrated` or `verified`
|
||||
(not only the healed ones) — healing one tool can unregister or break another;
|
||||
scope collisions are the classic case. Only then evaluate the exit condition.
|
||||
@@ -0,0 +1,141 @@
|
||||
# Integrate — patterns per stack
|
||||
|
||||
> Prefer the live official guides when online:
|
||||
> `npx -y modern-web-guidance@latest retrieve "webmcp,agentic-forms,agentic-javascript-tools"`.
|
||||
> The patterns below follow Google's reference implementations
|
||||
> (GoogleChromeLabs/webmcp-tools) and the W3C CG draft.
|
||||
|
||||
## Declarative — standard HTML forms
|
||||
|
||||
Applies to plain HTML, SSG-emitted, server-rendered, and framework-rendered
|
||||
(uncontrolled) forms — anywhere a real `<form>` with named controls exists.
|
||||
Annotate the existing form; do not restructure it.
|
||||
|
||||
```html
|
||||
<form toolname="request_quote"
|
||||
tooldescription="Requests a project quote. A team member replies within one business day."
|
||||
action="/contact" method="post">
|
||||
<label for="email">Email</label>
|
||||
<input type="email" id="email" name="email" required
|
||||
toolparamdescription="Email address for the reply">
|
||||
<!-- …existing fields, each with label[for] + name + toolparamdescription… -->
|
||||
<button type="submit">Request quote</button>
|
||||
</form>
|
||||
```
|
||||
|
||||
Rules:
|
||||
- The browser derives the JSON Schema from the controls — every control needs
|
||||
`name`, a resolvable description (`toolparamdescription` → `label[for]` text →
|
||||
`aria-description`), and correct HTML constraints. Radio groups: description on
|
||||
the enclosing `<fieldset>`.
|
||||
- `toolautosubmit` **only** on pure read forms (search/filter/availability).
|
||||
Never on contact/checkout/settings/messaging forms.
|
||||
- Fetch-submitted forms (`preventDefault()`) MUST route the result back to the
|
||||
agent — the most common integration bug is a swallowed submit:
|
||||
|
||||
```js
|
||||
form.addEventListener('submit', (e) => {
|
||||
e.preventDefault();
|
||||
const result = doSubmit(new FormData(e.target))
|
||||
.then(() => 'Request received. Reply within one business day.');
|
||||
if (e.agentInvoked) e.respondWith(result); // pass the PROMISE, not a value
|
||||
});
|
||||
```
|
||||
|
||||
- Optional UX (verbatim from Chrome docs): style agent activity with
|
||||
`form:tool-form-active` / `:tool-submit-active` CSS pseudo-classes.
|
||||
- Forms that navigate to a thank-you page: `executeTool` returns `null` on
|
||||
navigation (expected). A JSON-LD `{"@type":"Message","text":"…"}` block on the
|
||||
target page is best-effort garnish — the mechanism is still under spec debate;
|
||||
never make behavior depend on it.
|
||||
|
||||
### Framework notes — React
|
||||
|
||||
- Vendor `templates/webmcp-jsx.d.ts` alongside the ambient types so strict TSX
|
||||
accepts `toolname`/`tooldescription`/`toolparamdescription` (it augments the
|
||||
React attribute interfaces; it is a MODULE file — keep it separate from
|
||||
`webmcp.d.ts`).
|
||||
- The typings are string-valued (boolean-attribute style): write
|
||||
`toolautosubmit=""` — and only on pure read forms (ground rule 5).
|
||||
- In `onSubmit`, the WebMCP fields live on the NATIVE event:
|
||||
|
||||
```tsx
|
||||
const native = e.nativeEvent as SubmitEvent;
|
||||
if (native.agentInvoked) native.respondWith?.(doSubmit(new FormData(e.currentTarget))
|
||||
.then(() => 'Request received. Reply within one business day.'));
|
||||
```
|
||||
|
||||
Pass the PROMISE of the result string, not an already-resolved value.
|
||||
|
||||
## Imperative — SPAs and dynamic apps
|
||||
|
||||
Use the vendored runtime (`runtime.md`). Tools live in a dedicated module per app
|
||||
(e.g. `src/webmcp/tools.ts`), decoupled from components:
|
||||
|
||||
```ts
|
||||
import { createToolScope, dispatchAndWait } from './webmcpify';
|
||||
|
||||
export const searchTicketsTool = {
|
||||
name: 'search_tickets',
|
||||
description: 'Searches tickets in the currently open project and shows results on screen.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
query: { type: 'string', description: 'Search terms, exactly as the user phrased them.' },
|
||||
},
|
||||
required: ['query'],
|
||||
},
|
||||
annotations: { readOnlyHint: true, untrustedContentHint: true },
|
||||
async execute(input: Record<string, unknown>) {
|
||||
const q = String(input.query ?? '').trim();
|
||||
if (!q) return 'ERROR: `query` must be a non-empty string.';
|
||||
return dispatchAndWait('webmcp:search_tickets', { query: q });
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
Key rules:
|
||||
- **`execute()` wraps the existing UI code path** — dispatch the same event / call
|
||||
the same store action / hit the same API the button does. Never a parallel
|
||||
implementation.
|
||||
- **Return only after the interface state is settled**: the component listener
|
||||
awaits the real work, then fires the completion event with the outcome payload
|
||||
(`{ ok, message | error }`) — full contract and component example in
|
||||
`runtime.md`. A canned success before the work finishes is a false green.
|
||||
- Return short strings; errors as `"ERROR: <what and how to fix>"` so the model can
|
||||
self-correct. Cap outputs ~1.5k chars.
|
||||
- Validate strictly in code, loosely in schema — and keep **parity with the
|
||||
form's native HTML constraints**: when a tool wraps a form, probe the real
|
||||
constraints on a detached clone instead of re-implementing them —
|
||||
`const probe = emailInput.cloneNode() as HTMLInputElement; probe.value = value;`
|
||||
then reject when `!probe.checkValidity()`. `execute()` must refuse exactly what
|
||||
the form itself would refuse.
|
||||
|
||||
### Registration & lifecycle
|
||||
|
||||
- **Static registration is the default**: register app-wide tools once at bootstrap.
|
||||
- **Per-view registration only** for tools meaningless outside their view — via
|
||||
`createToolScope` in the view's mount/unmount (React `useEffect` cleanup, Vue
|
||||
`onUnmounted`, Angular `DestroyRef`). Over-scoping makes the toolset flicker and
|
||||
strands agents mid-plan.
|
||||
- Registration failures roll back the scope and surface via `onError` — check the
|
||||
console during integration; a silently missing toolset usually means a duplicate
|
||||
name or invalid schema rejected the batch.
|
||||
|
||||
### Auth / roles (SaaS)
|
||||
|
||||
Never register a tool the current session couldn't use through the UI. On
|
||||
login/logout/role change/tenant switch: dispose the scope and re-register the
|
||||
correct set (`runtime.md` §Wiring). The server still re-checks everything (ground
|
||||
rule 3) — role-scoped registration is UX hygiene, not security.
|
||||
|
||||
## Origin trial / flags note
|
||||
|
||||
WebMCP is a Chrome origin trial (149→, stable milestone still an estimate). For
|
||||
production exposure the origin needs a token:
|
||||
`<meta http-equiv="origin-trial" content="TOKEN">` or an `Origin-Trial` response
|
||||
header — registered at the Chrome Origin Trials console. For local work,
|
||||
`chrome://flags/#enable-webmcp-testing`. Chrome **silently ignores** expired
|
||||
tokens, so nothing may depend on WebMCP being present (ground rule 4). Add a short
|
||||
note about this to the target repo's README as part of setup, and record the
|
||||
touched file path in `pipeline.setup.originTrialNoted` (e.g. `["README.md"]`).
|
||||
@@ -0,0 +1,120 @@
|
||||
# Inventory — mapping a codebase into a tool manifest
|
||||
|
||||
## Detect (Phase 0 details)
|
||||
|
||||
Establish, in this order:
|
||||
|
||||
1. **Stack**: `package.json` deps (react/vue/@angular/next/astro/eleventy…) or the
|
||||
absence of one (static HTML). Record `app.stack` and `app.typescript`.
|
||||
2. **Start command + base URL**: `dev`/`start` scripts, framework defaults
|
||||
(`vite` → 5173, `next` → 3000, static → any file server). Verification needs a
|
||||
working local run — if the app can't be started, append the blocker to
|
||||
`pipeline.blockers` and surface it at the gate; don't silently proceed to a
|
||||
verify phase that cannot run.
|
||||
3. **Auth model**: none / session / role-based — plus **how a test session signs
|
||||
in**, recorded per role under `app.authFixtures`: `obtain` (the exact steps —
|
||||
seed command, login route), `account`, and `env` (the env var **names** the
|
||||
fixture needs — never secret values in the manifest). The verify phase runs
|
||||
from this. Role-based apps need role-scoped registration (`integrate.md`
|
||||
§Auth) and a per-role verify pass.
|
||||
4. **Git baseline**: `pipeline.baselineSha` = HEAD, `pipeline.baselineDirty` =
|
||||
`git status --porcelain` paths. Dirty files are untouchable for the whole run.
|
||||
|
||||
## Building the area map
|
||||
|
||||
The area map is the unit of loop iteration. Sources, in order of preference:
|
||||
router config (React Router, Next `app/`/`pages/`, Vue Router, Angular routes) →
|
||||
navigation UI (static/SSG) → feature folders (`src/features/*`). Keep areas
|
||||
coarse: 5–30 for a big SaaS, 1–3 for a landing page. Split an area that turns out
|
||||
too big; merge trivial ones.
|
||||
|
||||
## What counts as a candidate tool
|
||||
|
||||
Walk each area's UI code and list **user actions**, not functions:
|
||||
|
||||
| UI pattern | Candidate tool | `mutating` | `readOnlyHint` |
|
||||
|---|---|---|---|
|
||||
| Search/filter form or input | `search_<noun>` | false | true |
|
||||
| Data list/detail currently rendered | `list_<noun>` / `get_<noun>` | false | true |
|
||||
| Create/edit form with submit → API call | `create_<noun>` / `update_<noun>` | "server" | — |
|
||||
| Button triggering a server state change | `<verb>_<noun>` | "server" | — |
|
||||
| Preference/theme/localStorage toggle | `<verb>_<noun>` | "client" | — |
|
||||
| Multi-step flow (wizard, checkout) | `start_<noun>_flow` (initiation) | false* | **never** |
|
||||
| Contact/booking form (static sites) | declarative form annotation | "server" | — |
|
||||
|
||||
*Initiation tools only navigate/open the flow — the human completes it. They are
|
||||
classified non-mutating (no data changes) **but must NOT carry `readOnlyHint`**:
|
||||
they change UI state, and agents skip confirmations for hinted-read-only tools.
|
||||
`readOnlyHint: true` is reserved for genuinely pure data reads.
|
||||
|
||||
`mutating` is tri-state: `false` | `"client"` (browser-local only: prefs, theme,
|
||||
localStorage — nothing leaves the browser) | `"server"` (data leaves the browser).
|
||||
`"server"` gets the full ceremony — per-tool approval, required `cleanup`,
|
||||
dev/test-data-only verification; `"client"` may be batch-approved at the gate
|
||||
(`cleanup` recommended). `toolautosubmit` is banned for **both** mutation classes
|
||||
(ground rule 5).
|
||||
|
||||
**Skip** (do not inventory): login/logout/auth flows, payment execution, account
|
||||
deletion, user management, anything irreversible, file uploads (v1), and pure
|
||||
navigation agents can do anyway.
|
||||
|
||||
## Tool budget, overlap, and priority (what keeps SaaS toolsets usable)
|
||||
|
||||
Agents degrade when many similar tools compete. Enforce while drafting:
|
||||
|
||||
- **Budget**: aim for ≤15 tools active in any app state (app-wide + current view).
|
||||
If an area yields more candidates, keep the highest-value ones as `priority: 1`
|
||||
and mark the rest `priority: 2/3` — the gate decides which waves ship.
|
||||
- **Overlap rule**: no two tools whose descriptions could plausibly match the same
|
||||
user request. Merge them (one tool, richer schema) or sharpen both descriptions
|
||||
until they are disjoint.
|
||||
- **Role/tenant coverage**: for role-scoped apps, note per tool which roles can use
|
||||
it (`auth: ["role:<name>", ...]`); the toolset a given session sees must stay
|
||||
within budget too.
|
||||
|
||||
## Naming and schema conventions (Google's, condensed)
|
||||
|
||||
- **Verb-first, execution vs initiation honest**: `create_event` acts immediately;
|
||||
`start_event_creation_process` merely opens a form. The name must never lie.
|
||||
- Name ≤30 chars, `[a-zA-Z0-9_.-]`; prefix with the app name if tools may coexist
|
||||
with other origins' tools in testing (`myapp_search_tickets`).
|
||||
- Description ≤500 chars, positive capability statement, no marketing. Param
|
||||
descriptions ≤150 chars. The description must say exactly what `execute()` does —
|
||||
agents make consent decisions from it.
|
||||
- **Raw user input rule**: schemas accept what the user would say ("11:00 to
|
||||
15:00"), never ask the agent to compute or transform. Semantic enum values
|
||||
(`"High"`, not `priority_id: 3`).
|
||||
- Tools returning user-generated or external content get
|
||||
`untrustedContentHint: true`.
|
||||
|
||||
## Choosing `kind`
|
||||
|
||||
- `declarative` — any standard `<form>` whose fields map 1:1 to the action's
|
||||
inputs: plain HTML, SSG-emitted, server-rendered, *and* framework-rendered forms
|
||||
(uncontrolled inputs), including fetch-submitted forms (they bridge results via
|
||||
`respondWith` — see `integrate.md`).
|
||||
- `imperative` — non-form actions (buttons, drag/drop, selections), actions whose
|
||||
inputs come from app state rather than form fields, and React/Vue **controlled**
|
||||
forms (agent-driven fill would bypass the framework's state).
|
||||
|
||||
## Writing manifest entries
|
||||
|
||||
Fill EVERY field of the v3 schema:
|
||||
|
||||
- `route` + `auth` (array of roles keying into `app.authFixtures`; verify runs
|
||||
once per role).
|
||||
- `annotations` — `readOnlyHint`/`untrustedContentHint` per the candidate table;
|
||||
verify asserts them on the enumerated tool.
|
||||
- `examples` — one valid + one invalid. `invalid: null` is allowed ONLY for
|
||||
readOnly tools with no/empty params (verify then asserts dual-outcome); the
|
||||
convention for a non-null invalid on zero-param tools is `{"unexpected": true}`.
|
||||
- `expect` — exactly ONE of `result` (substring of the resolved string) or
|
||||
`navigation` (destination URL/pattern when `executeTool` resolves `null`),
|
||||
plus `ui` (a UI assertion a test can check).
|
||||
- `cleanup` — required for `mutating: "server"`, recommended for `"client"`.
|
||||
|
||||
The verify phase must be able to run from the manifest alone, without re-reading
|
||||
the codebase — that is what makes runs resumable by a different agent.
|
||||
|
||||
The completeness pass at the end of Phase 1: start the app (or read the rendered
|
||||
nav), enumerate what a user can *do* per screen, and diff against the manifest.
|
||||
@@ -0,0 +1,129 @@
|
||||
# Runtime — vendoring and wiring the templates
|
||||
|
||||
Copy from this skill's `templates/` directory into the target project
|
||||
(suggested: `src/webmcp/`):
|
||||
|
||||
- **TypeScript projects**: `templates/webmcpify.ts` + `templates/webmcp.d.ts`;
|
||||
**React TSX projects additionally** `templates/webmcp-jsx.d.ts` (JSX typings for
|
||||
the declarative attributes — a MODULE file; keep it separate from
|
||||
`webmcp.d.ts`, which must stay a global script file).
|
||||
- **JavaScript projects**: `templates/webmcpify.js` — **ES module only** (`export`):
|
||||
load via a bundler or `<script type="module">`. For CommonJS/classic-script
|
||||
projects, transpile or vendor the TS variant instead.
|
||||
|
||||
**Vendor, don't depend** — the runtime is small, MIT, and a target repo must not
|
||||
gain a dependency for an origin-trial API. **Keep the full MIT notice header** in
|
||||
every copied file: the license's retention condition requires the copyright line
|
||||
and permission notice to travel with the code, and the header IS that notice —
|
||||
never trim it down to a bare link.
|
||||
Record the copied file paths in the manifest
|
||||
(`pipeline.setup.runtimeVendored: ["src/webmcp/webmcpify.ts", ...]`).
|
||||
|
||||
What it provides:
|
||||
|
||||
| Export | Purpose |
|
||||
|---|---|
|
||||
| `getModelContext()` | The ONLY place `document.modelContext` / deprecated `navigator.modelContext` is referenced — spec churn stays a one-file fix |
|
||||
| `isWebMCPAvailable()` | Feature detection — the app must work identically without WebMCP |
|
||||
| `createToolScope(key, tools, options?)` | Registers a tool set under one AbortController; returns a **callable dispose handle** carrying `ready: Promise<boolean>` (true = all registrations committed; false = no WebMCP / duplicate key / failure / disposed first — never rejects). Validates contracts BEFORE registering; **rolls back the whole scope** on any failure, including sync-throwing legacy `registerTool` (reported via `options.onError`, default `console.error` — NOT called when disposed before settling). An already-active key returns a no-op handle — safe under React StrictMode |
|
||||
| `dispatchAndWait(event, detail?, timeoutMs?)` | Bridges `execute()` to the app's own event/state flow. The dispatched detail carries `requestId` plus `signal` — an AbortSignal aborted on timeout; pass it to `fetch()` and skip state commits once aborted. Resolves only after the component confirms with an explicit **boolean** `ok`; a completion with missing/non-boolean `ok` **fails closed** to an `"ERROR: ..."` string, as do timeouts and `ok: false` (self-correction convention — never rejects). For tools whose confirmation involves a network round-trip (mailers, slow APIs), pass an explicit `timeoutMs` (e.g. `20_000`) instead of relying on the 10 s default |
|
||||
| `singleFlight(fn, busyMessage?)` | Serializes a tool's `execute`: while one call is in flight, further calls resolve immediately to a busy `"ERROR: ..."` string instead of racing shared UI state |
|
||||
|
||||
Validation note: budget checks auto-enable when the bundler substitutes
|
||||
`process.env.NODE_ENV` (Vite/webpack automatic; esbuild via `--define`) and it
|
||||
isn't `'production'`; unbundled projects default to off — pass `{ validate: true }`
|
||||
during development.
|
||||
|
||||
## The completion contract (the part integrators get wrong)
|
||||
|
||||
`dispatchAndWait` resolves when the component fires `tool-completion-<requestId>`
|
||||
with `detail: { ok: boolean, message?: string, error?: string }`. `ok` must be an
|
||||
explicit boolean — anything else fails closed to an ERROR result. Fire it **after
|
||||
the async work has truly finished** — awaited fetch, committed state, rendered
|
||||
result — never right after *starting* the action. Agents plan from what is on
|
||||
screen; a completion fired early produces false greens.
|
||||
|
||||
Hardened component bridge (React example — adapt per framework). Five clauses:
|
||||
**(1)** completion fires from an effect observing the committed state, **(2)**
|
||||
availability gate, **(3)** single-flight, **(4)** timeout coordination via
|
||||
`detail.signal`, **(5)** unmount cancellation.
|
||||
|
||||
```tsx
|
||||
const pending = useRef<{ requestId: string; count: number } | null>(null);
|
||||
const [results, setResults] = useState<Ticket[] | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
if (!isWebMCPAvailable()) return; // (2) attach only when WebMCP exists
|
||||
let inFlight = false;
|
||||
const onSearch = async (e: Event) => {
|
||||
const { query, requestId, signal } = (e as CustomEvent).detail;
|
||||
const fail = (error: string) =>
|
||||
window.dispatchEvent(new CustomEvent(`tool-completion-${requestId}`, {
|
||||
detail: { ok: false, error },
|
||||
}));
|
||||
if (inFlight) return fail('A search is already running.'); // (3) single-flight
|
||||
inFlight = true;
|
||||
try {
|
||||
const found = await runSearch(query, { signal }); // (4) the runtime aborts this signal on timeout
|
||||
if (signal?.aborted) return; // (4) timed out — runtime already answered; no late commits
|
||||
pending.current = { requestId, count: found.length };
|
||||
setResults(found); // commit → the effect below confirms
|
||||
} catch (err) {
|
||||
if (signal?.aborted) return;
|
||||
fail(err instanceof Error ? err.message : 'Search failed.');
|
||||
} finally {
|
||||
inFlight = false;
|
||||
}
|
||||
};
|
||||
window.addEventListener('webmcp:search_tickets', onSearch);
|
||||
return () => window.removeEventListener('webmcp:search_tickets', onSearch); // (5) unmount detaches
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
if (!pending.current || results === null) return; // (1) confirm AFTER the commit rendered
|
||||
const { requestId, count } = pending.current;
|
||||
pending.current = null;
|
||||
window.dispatchEvent(new CustomEvent(`tool-completion-${requestId}`, {
|
||||
detail: { ok: true, message: `Search finished — ${count} results are now visible.` },
|
||||
}));
|
||||
}, [results]);
|
||||
```
|
||||
|
||||
Why clause (1): React 18 batches renders — state set after the `await` is **not
|
||||
yet on screen** when the next line of the handler runs, so dispatching the
|
||||
completion there reports success before the user (and the agent's next snapshot)
|
||||
can see it. Dispatching from an effect keyed on the updated state guarantees the
|
||||
commit happened. Equivalents: Vue `await nextTick()`; Svelte `await tick()` —
|
||||
then dispatch inline.
|
||||
|
||||
## Wiring patterns
|
||||
|
||||
```tsx
|
||||
// bootstrap (app-wide tools, static registration — the default):
|
||||
import { createToolScope } from './webmcp/webmcpify';
|
||||
import { appTools } from './webmcp/tools';
|
||||
createToolScope('app', appTools);
|
||||
|
||||
// per-view tools (only when genuinely view-bound):
|
||||
useEffect(() => createToolScope('tickets-view', ticketViewTools), []);
|
||||
// the handle IS the dispose fn → React runs it on unmount. StrictMode's
|
||||
// double-mount is safe: the second call no-ops, an unmount before registration
|
||||
// settles rolls back silently (ready → false, no onError).
|
||||
|
||||
// when you need to know registration committed:
|
||||
const handle = createToolScope('app', appTools);
|
||||
handle.ready.then((ok) => { if (!ok) console.warn('WebMCP tools not active'); });
|
||||
```
|
||||
|
||||
Role-scoped SaaS registration — dispose and re-create on auth changes:
|
||||
|
||||
```ts
|
||||
let dispose: (() => void) | undefined;
|
||||
export function syncToolsForUser(user: User | null) {
|
||||
dispose?.();
|
||||
const tools = [...publicTools, ...(user ? memberTools : []),
|
||||
...(user?.role === 'admin' ? adminTools : [])];
|
||||
dispose = createToolScope('auth-scoped', tools);
|
||||
}
|
||||
// call on login, logout, role change, tenant switch
|
||||
```
|
||||
@@ -0,0 +1,73 @@
|
||||
# Security checklist
|
||||
|
||||
Apply at two points: **before the manifest gate** (classify + flag) and **at the
|
||||
final audit** (verify). Any unchecked box on a `mutating: "server"` tool blocks
|
||||
it. Client-only mutations (`mutating: "client"`) still must pass the
|
||||
**Trust boundary** and **Honesty & hints** boxes.
|
||||
|
||||
## Threat model in one paragraph
|
||||
|
||||
Any Chrome extension with host permissions — and any agent the user runs — can
|
||||
enumerate and execute your tools **with the user's live session**. The spec has no
|
||||
agent-identity mechanism. Page-visible strings (descriptions, labels, enum values,
|
||||
tool outputs) all enter the model's context, so they are prompt-injection surface in
|
||||
both directions. Design every tool as if it were a public, authenticated API endpoint
|
||||
— because effectively it is one.
|
||||
|
||||
## Checklist
|
||||
|
||||
**Trust boundary**
|
||||
- [ ] Every `execute()` calls only code paths the UI already uses — same endpoints,
|
||||
same validation, same authz, same rate limits. No new endpoints, no bypasses.
|
||||
- [ ] No secrets, tokens, or privileged config inside tool code or descriptions.
|
||||
- [ ] Role-based apps: tools registered per role/session and re-scoped on auth
|
||||
changes; nothing registered the current session couldn't do via the UI.
|
||||
|
||||
**Human-in-the-loop**
|
||||
- [ ] No `toolautosubmit` on any state-changing form.
|
||||
- [ ] No destructive/irreversible/payment tools at all in a first integration.
|
||||
If the human explicitly insists later: an in-page manual confirmation the
|
||||
**user** performs, PLUS a server-side two-step (short-lived confirm token).
|
||||
No client-side API exists that can force an agent to confirm — never rely on
|
||||
one.
|
||||
- [ ] Initiation tools (`start_*_flow`) genuinely only navigate/open — they must
|
||||
not pre-execute any part of the mutation, and never carry `readOnlyHint`.
|
||||
|
||||
**Production side effects (verification)**
|
||||
- [ ] Any verification that unavoidably causes a real production effect (e.g. an
|
||||
Origin-allow-listed mailer) has explicit gate approval recorded in the
|
||||
tool's `approval.productionSideEffect` — without it, the live path is
|
||||
`skipped`, never executed.
|
||||
- [ ] Every such test payload is marked `[webmcpify verification]`, and every
|
||||
caused effect is listed in `report.md`.
|
||||
- [ ] The Origin-replay pattern (`heal.md`) lives only in the env-gated harness
|
||||
(`WEBMCP_LIVE_MUTATIONS=1`) — never in shipped code, never default-on in CI.
|
||||
|
||||
**Honesty & hints**
|
||||
- [ ] Description says exactly what `execute()` does — no more, no less (agents make
|
||||
consent decisions from it).
|
||||
- [ ] `readOnlyHint: true` ONLY on genuinely pure data reads (agents skip
|
||||
confirmation based on it; mislabeling is the worst single mistake).
|
||||
- [ ] `untrustedContentHint: true` on every tool returning user-generated or
|
||||
external content.
|
||||
- [ ] Outputs capped (~1.5k chars) and free of instruction-like content where
|
||||
possible.
|
||||
|
||||
**Privacy**
|
||||
- [ ] Schemas request no more personal data than the equivalent visible form —
|
||||
agents auto-fill anything you declare (over-parameterization = silent
|
||||
profiling vector).
|
||||
|
||||
**Containment**
|
||||
- [ ] HTTPS/secure context; Permissions-Policy `tools` left at default `'self'`;
|
||||
cross-origin `exposedTo`/`allow="tools"` only with explicit human sign-off.
|
||||
- [ ] Pages that must never expose tools (un-audited checkout, admin consoles you
|
||||
didn't inventory) can send `Permissions-Policy: tools=()` — suggest it in the
|
||||
report where relevant.
|
||||
- [ ] No third-party WebMCP runtime added to the project; enumeration/execution
|
||||
surfaces (`getTools`/`executeTool`, legacy `modelContextTesting`) appear
|
||||
nowhere in shipped application code.
|
||||
- [ ] Component-side `webmcp:*` event bridges attach only when
|
||||
`isWebMCPAvailable()` and validate their event payloads — a page script can
|
||||
dispatch the same CustomEvents; the bridge must not become an unvalidated
|
||||
side door into app actions.
|
||||
@@ -0,0 +1,132 @@
|
||||
# Verify — proving every tool works in a real browser
|
||||
|
||||
## Environment
|
||||
|
||||
- **Current Chrome** (the API moved during the trial — the old
|
||||
`navigator.modelContextTesting` surface was removed 2026-07 in favor of
|
||||
production `document.modelContext.getTools()/executeTool()`).
|
||||
- Enable: `chrome://flags/#enable-webmcp-testing`, or launch with
|
||||
`--enable-features=WebMCP,WebMCPTesting` (covers both current and older builds).
|
||||
- **Headed only** — WebMCP requires a visible tab by design. In CI, run under
|
||||
`xvfb-run`. Headless will never work; don't heal toward it.
|
||||
- App running locally via `app.startCommand`, against dev/test data only.
|
||||
- Each tool's manifest entry tells you where and how: `route` (navigate there),
|
||||
`auth` (sign in with the recorded test fixture; verify under EACH role for
|
||||
role-scoped tools), `examples` (what to execute), `expect` (what to assert),
|
||||
`cleanup` (how to undo a mutating tool's effect after the test).
|
||||
|
||||
## The enumeration/execution surface (probe, don't assume)
|
||||
|
||||
In the page context, prefer the production surface and fall back for older builds:
|
||||
|
||||
```js
|
||||
const mc = document.modelContext ?? navigator.modelContext;
|
||||
const tools = mc?.getTools
|
||||
? await mc.getTools()
|
||||
: await navigator.modelContextTesting?.listTools(); // removed 2026-07; legacy only
|
||||
```
|
||||
|
||||
Contract facts that generated assertions MUST respect:
|
||||
|
||||
- Enumerated `inputSchema` is a **stringified** JSON Schema — `JSON.parse` before
|
||||
comparing against the manifest entry.
|
||||
- `executeTool(...)` resolves to a **string result, or `null` when the execution
|
||||
navigated** (normal for declarative forms that submit-navigate).
|
||||
- Execution and declarative-validation failures **reject the promise** — they do
|
||||
not resolve to `"ERROR: ..."`. Only imperative tools following the runtime's
|
||||
convention resolve with `"ERROR: ..."` strings. Assert accordingly per tool
|
||||
`kind`.
|
||||
- **Registration is asynchronous** — `registerTool()` returns a promise, so a tool
|
||||
is not enumerable the instant the page loads. Poll for it (`waitForTool` in the
|
||||
template) or await a `toolchange` event; never assert presence immediately
|
||||
after `goto`.
|
||||
- **Mutating declarative forms pause mid-execution**: Chrome fills the form, then
|
||||
waits for a real submit interaction before letting `executeTool` settle —
|
||||
awaiting it alone deadlocks into a timeout. Use the concurrent pattern: start
|
||||
`executeTool` unawaited → wait for an agent-filled value to appear → click
|
||||
submit → await the result (full example in the template).
|
||||
- These surfaces are for agents/harnesses only — they must never appear in shipped
|
||||
application code.
|
||||
|
||||
For **declarative** tools also verify the *synthesized* schema: the form-control →
|
||||
schema mapping is only partially specified, so check each annotated control appears
|
||||
as the expected property in the actual target Chrome build.
|
||||
|
||||
## Per-tool checks
|
||||
|
||||
1. Registered (poll — registration is async) with the expected name, the (parsed)
|
||||
schema, **and** the manifest `annotations` on the enumerated tool. The legacy
|
||||
`modelContextTesting` fallback cannot enumerate annotations — skip that
|
||||
assertion there and note the gap in the report.
|
||||
2. Valid example executes: assert the result per `expect` — `expect.result` as a
|
||||
substring of the resolved string, or `expect.navigation` as the destination
|
||||
when `executeTool` resolves `null` (it navigated) — **and** the `expect.ui`
|
||||
state as a **delta** (capture the relevant state *before* executing; mere
|
||||
visibility of something already on screen proves nothing). A tool that reports
|
||||
success without the UI changing is a **fail** (UI-settled rule). Because
|
||||
executions can navigate, restore the manifest `route` in `beforeEach`, not
|
||||
`beforeAll`.
|
||||
3. Invalid example: **prove the tool is present first** (a rejection from a
|
||||
never-registered tool is not a validation rejection). Then: imperative →
|
||||
resolves `"ERROR: ..."`; declarative/schema violation → rejects. Zero-param
|
||||
read tools with `examples.invalid: null` get the dual-outcome assertion
|
||||
instead: `{"unexpected": true}` may be rejected with a validation reason OR
|
||||
resolve benignly — both pass; a missing tool/surface fails.
|
||||
4. Mutating tools: run against disposable data, verify the mutation through the
|
||||
same read path the UI uses, then execute the manifest `cleanup` — a
|
||||
`mutating: "server"` tool without working cleanup blocks at the gate, and
|
||||
heal-loop retries of mutating tools must clean up between attempts.
|
||||
|
||||
## Harness
|
||||
|
||||
Instantiate `templates/webmcp.spec.ts` (bundled with this skill) — Playwright,
|
||||
headed persistent Chrome, one describe-block per tool generated from the manifest,
|
||||
with real assertions (never commented-out placeholders). Put the generated spec
|
||||
next to the repo's existing e2e tests.
|
||||
|
||||
**Repos without a test setup — the standalone-harness recipe.** The spec stays in
|
||||
`.webmcpify/webmcp.spec.ts` (single source of truth, committed per the gate's
|
||||
`commitWebmcpifyDir` choice); the Playwright installation lives in a scratch
|
||||
harness OUTSIDE the repo so the target gains no dependencies:
|
||||
|
||||
```sh
|
||||
mkdir -p /tmp/webmcpify-harness && cd /tmp/webmcpify-harness
|
||||
npm init -y && npm i -D @playwright/test typescript @types/node
|
||||
cat > playwright.config.ts <<'EOF'
|
||||
import { defineConfig } from '@playwright/test';
|
||||
export default defineConfig({
|
||||
testDir: process.env.WEBMCP_SPEC_DIR, // → <target-repo>/.webmcpify
|
||||
workers: 1, // one shared headed Chrome — never parallelize
|
||||
});
|
||||
EOF
|
||||
WEBMCP_SPEC_DIR=<target-repo>/.webmcpify WEBMCP_BASE_URL=http://localhost:5173 \
|
||||
NODE_PATH=/tmp/webmcpify-harness/node_modules npx playwright test
|
||||
```
|
||||
|
||||
`NODE_PATH` lets the out-of-repo spec resolve `@playwright/test`; if the target's
|
||||
tooling ignores `NODE_PATH`, symlink instead:
|
||||
`ln -s /tmp/webmcpify-harness/node_modules <target-repo>/.webmcpify/node_modules`
|
||||
(and make sure it isn't committed). Note in the report that verification ran from
|
||||
a standalone harness.
|
||||
|
||||
**Alternative:** Puppeteer ships a first-class experimental WebMCP API
|
||||
(https://pptr.dev/guides/webmcp) — prefer it when the target repo already uses
|
||||
Puppeteer.
|
||||
|
||||
## Tool-selection evals (recommended; mandatory for SaaS-scale toolsets)
|
||||
|
||||
Schema-level verification proves tools *work*, not that an LLM *picks* them.
|
||||
For apps exposing more than a handful of tools, run Google's **WebMCP Evals CLI**
|
||||
(GoogleChromeLabs/webmcp-tools, `evals-cli`): write one eval case per tool from the
|
||||
manifest examples ("user says X → expect tool Y with args Z") and run them —
|
||||
this catches ambiguous names/descriptions and overlapping tools that Playwright
|
||||
cannot.
|
||||
|
||||
## Manual QA (tell the human in the report)
|
||||
|
||||
- DevTools → **Application → WebMCP pane**: live tool list, invocation log,
|
||||
"Run tool" with editable params.
|
||||
- **Model Context Tool Inspector** Chrome extension (by Google's François
|
||||
Beaufort): natural-language smoke tests of tool *selection*.
|
||||
- Chrome's WebMCP audits flag missing `toolname`/`toolparamdescription`/
|
||||
`label[for]`/`name` on declarative forms.
|
||||
Reference in New Issue
Block a user