Add the WebMCPify agent skill 🤖🤖🤖 (#2400)

* feat(skills): add webmcpify skill

* style(skills): quote webmcpify description
This commit is contained in:
Jonas Tüchler
2026-07-23 19:50:25 +02:00
committed by GitHub
parent 44f3f0d06a
commit 03fb5fc96e
13 changed files with 1930 additions and 0 deletions
+69
View File
@@ -0,0 +1,69 @@
# Heal — failure taxonomy → fixes
Work one failed tool at a time. Re-verify after each fix. The triggering verify
failure counts as attempt 0; each fix cycle increments `attempts`. At `attempts`
= 3 → mark `skipped` with a blocker note (this is an explicit escalation to the
human in the final report, not a silent drop) and move on. **Never** widen the
diff, disable a check, or fake a return value to force a pass. **Mutating
tools:** run the manifest `cleanup` between attempts — retrying a mutation
without cleanup duplicates data.
**Heal fixes implementations, not contracts.** The manifest is the
human-approved contract: if the correct fix would change a tool's `inputSchema`,
`description`, `mutating` class, `annotations`, or `expect`, take it back to the
gate as a mini re-approval — never silently edit the manifest to match the code.
## Taxonomy
| Symptom | Likely cause | Fix |
|---|---|---|
| Tool absent from enumeration | **Registration is async** — the test asserted before `registerTool()` settled; or registration never ran (bootstrap not reached, view not mounted) or wrong Chrome build/flags | FIRST make the test poll (`waitForTool`) or await `toolchange` — only if it still fails, trace the registration call; confirm `isWebMCPAvailable()` in the test env; current Chrome + `--enable-features=WebMCP,WebMCPTesting` |
| Whole scope absent | A registration in the batch rejected (duplicate name, invalid schema, policy) — the runtime rolls back the entire scope | Check console for the `onError` report; fix the offending tool contract |
| Tool absent after route change | Scope disposed by navigation (over-scoping) | Move to static app-level registration unless genuinely view-bound |
| Declarative tool missing | `toolname` typo, frame without `allow="tools"`, or page sends `Origin-Agent-Cluster: ?0` | Fix attribute; check Permissions-Policy `tools` and origin-keying headers |
| Schema mismatch (declarative) | Control lacks `name`, description not resolvable, unsupported control type in this build | Add `name`/`toolparamdescription`/`label[for]`; unsupported controls → switch that form to imperative |
| Schema mismatch (imperative) | Manifest and code drifted | Make code match the approved manifest; if the manifest was wrong, that's a contract change — take it back to the gate for re-approval (see above), never silently update it |
| Assertion compares object to string | Enumerated `inputSchema` is a stringified JSON Schema | `JSON.parse` before comparing (see `verify.md`) |
| `executeTool` returns `null` unexpectedly | The execution navigated (normal for submit-navigating declarative forms) | Assert on the post-navigation page instead of the return value |
| `executeTool` rejects | Schema violation or declarative-validation failure — rejection IS the failure signal for these | For invalid-input tests on declarative tools, assert rejection, not an `"ERROR:"` string |
| Mutating declarative execution hangs until timeout | Chrome fills the form, then **pauses the execution awaiting a real submit interaction** — awaiting `executeTool` alone deadlocks | Use the concurrent pattern in the spec template: start `executeTool` unawaited → wait for the agent-filled value → click submit → await. **NEVER heal by adding `toolautosubmit`** (ground rule 5) |
| Backend rejects the harness with 403/CORS despite correct auth | The endpoint **allow-lists the production `Origin`** (mailers, form gateways) — the localhost harness origin is refused before the tool logic runs, and no local fix exists | Verify the live path with the env-gated server-side replay (§Origin-allow-listed endpoints below), only with the production side-effect approval recorded in `approval.productionSideEffect` (see §Origin-allow-listed endpoints below); without it, mark the live path `skipped` with a blocker note |
| Execution times out / canned success while UI still loading | Completion event fired before the async work finished, or listener missing/wrong event name | Fire `tool-completion-<requestId>` with `{ ok, message/error }` AFTER awaiting the real work (`runtime.md` contract) |
| Returns success but UI unchanged | `execute()` bypassed the real UI path (parallel implementation) | Rewrite to call the same handler/store action/endpoint the UI uses |
| Invalid input resolves successfully (imperative) | Missing in-code validation | Validate strictly in code; return `"ERROR: <what/how to fix>"` |
| Fetch-submitted form: agent gets nothing | `preventDefault()` without `respondWith()` | Add the `e.agentInvoked → e.respondWith(promise)` bridge |
| Works manually, fails in Playwright | Headless, missing flags, or profile without the flag | Headed + flags; persistent context; `xvfb-run` in CI |
| 401/403 from `execute()` in test | Tool registered outside the authenticated scope, or test session lacks the role in the manifest `auth` field | Role-scope the registration; sign in with the recorded fixture |
| Flaky: passes alone, fails in suite | Shared state between tool executions | Isolate test data per tool run (use `cleanup`); don't reorder tests to hide it |
## Origin-allow-listed endpoints — the replay pattern
Some production backends (mailers, form gateways) allow-list the production
`Origin` header and refuse everything else — the localhost harness can never
exercise the live path directly. When (and only when) the gate approved the real
production side effect (`approval.productionSideEffect`), verify the live path
with an env-gated replay: intercept the app's own request in Playwright and
re-issue it server-side (Node context — not subject to browser CORS) with the
production `Origin`:
```ts
// Env-gated: runs only with WEBMCP_LIVE_MUTATIONS=1 — never default-on in CI.
if (process.env.WEBMCP_LIVE_MUTATIONS === '1') {
await page.route('**/api/contact', async (route) => {
const response = await context.request.fetch(route.request(), {
headers: { ...route.request().headers(), origin: 'https://example.com' }, // the prod Origin
});
await route.fulfill({ response });
});
}
```
This causes a REAL production side effect. Mark every payload
`[webmcpify verification]`, run the manifest `cleanup`, list the effect in
`report.md`, and never enable the gate by default in CI.
## After healing
Re-run verification once for **all** tools with status `integrated` or `verified`
(not only the healed ones) — healing one tool can unregister or break another;
scope collisions are the classic case. Only then evaluate the exit condition.
+141
View File
@@ -0,0 +1,141 @@
# Integrate — patterns per stack
> Prefer the live official guides when online:
> `npx -y modern-web-guidance@latest retrieve "webmcp,agentic-forms,agentic-javascript-tools"`.
> The patterns below follow Google's reference implementations
> (GoogleChromeLabs/webmcp-tools) and the W3C CG draft.
## Declarative — standard HTML forms
Applies to plain HTML, SSG-emitted, server-rendered, and framework-rendered
(uncontrolled) forms — anywhere a real `<form>` with named controls exists.
Annotate the existing form; do not restructure it.
```html
<form toolname="request_quote"
tooldescription="Requests a project quote. A team member replies within one business day."
action="/contact" method="post">
<label for="email">Email</label>
<input type="email" id="email" name="email" required
toolparamdescription="Email address for the reply">
<!-- …existing fields, each with label[for] + name + toolparamdescription… -->
<button type="submit">Request quote</button>
</form>
```
Rules:
- The browser derives the JSON Schema from the controls — every control needs
`name`, a resolvable description (`toolparamdescription``label[for]` text →
`aria-description`), and correct HTML constraints. Radio groups: description on
the enclosing `<fieldset>`.
- `toolautosubmit` **only** on pure read forms (search/filter/availability).
Never on contact/checkout/settings/messaging forms.
- Fetch-submitted forms (`preventDefault()`) MUST route the result back to the
agent — the most common integration bug is a swallowed submit:
```js
form.addEventListener('submit', (e) => {
e.preventDefault();
const result = doSubmit(new FormData(e.target))
.then(() => 'Request received. Reply within one business day.');
if (e.agentInvoked) e.respondWith(result); // pass the PROMISE, not a value
});
```
- Optional UX (verbatim from Chrome docs): style agent activity with
`form:tool-form-active` / `:tool-submit-active` CSS pseudo-classes.
- Forms that navigate to a thank-you page: `executeTool` returns `null` on
navigation (expected). A JSON-LD `{"@type":"Message","text":"…"}` block on the
target page is best-effort garnish — the mechanism is still under spec debate;
never make behavior depend on it.
### Framework notes — React
- Vendor `templates/webmcp-jsx.d.ts` alongside the ambient types so strict TSX
accepts `toolname`/`tooldescription`/`toolparamdescription` (it augments the
React attribute interfaces; it is a MODULE file — keep it separate from
`webmcp.d.ts`).
- The typings are string-valued (boolean-attribute style): write
`toolautosubmit=""` — and only on pure read forms (ground rule 5).
- In `onSubmit`, the WebMCP fields live on the NATIVE event:
```tsx
const native = e.nativeEvent as SubmitEvent;
if (native.agentInvoked) native.respondWith?.(doSubmit(new FormData(e.currentTarget))
.then(() => 'Request received. Reply within one business day.'));
```
Pass the PROMISE of the result string, not an already-resolved value.
## Imperative — SPAs and dynamic apps
Use the vendored runtime (`runtime.md`). Tools live in a dedicated module per app
(e.g. `src/webmcp/tools.ts`), decoupled from components:
```ts
import { createToolScope, dispatchAndWait } from './webmcpify';
export const searchTicketsTool = {
name: 'search_tickets',
description: 'Searches tickets in the currently open project and shows results on screen.',
inputSchema: {
type: 'object',
properties: {
query: { type: 'string', description: 'Search terms, exactly as the user phrased them.' },
},
required: ['query'],
},
annotations: { readOnlyHint: true, untrustedContentHint: true },
async execute(input: Record<string, unknown>) {
const q = String(input.query ?? '').trim();
if (!q) return 'ERROR: `query` must be a non-empty string.';
return dispatchAndWait('webmcp:search_tickets', { query: q });
},
};
```
Key rules:
- **`execute()` wraps the existing UI code path** — dispatch the same event / call
the same store action / hit the same API the button does. Never a parallel
implementation.
- **Return only after the interface state is settled**: the component listener
awaits the real work, then fires the completion event with the outcome payload
(`{ ok, message | error }`) — full contract and component example in
`runtime.md`. A canned success before the work finishes is a false green.
- Return short strings; errors as `"ERROR: <what and how to fix>"` so the model can
self-correct. Cap outputs ~1.5k chars.
- Validate strictly in code, loosely in schema — and keep **parity with the
form's native HTML constraints**: when a tool wraps a form, probe the real
constraints on a detached clone instead of re-implementing them —
`const probe = emailInput.cloneNode() as HTMLInputElement; probe.value = value;`
then reject when `!probe.checkValidity()`. `execute()` must refuse exactly what
the form itself would refuse.
### Registration & lifecycle
- **Static registration is the default**: register app-wide tools once at bootstrap.
- **Per-view registration only** for tools meaningless outside their view — via
`createToolScope` in the view's mount/unmount (React `useEffect` cleanup, Vue
`onUnmounted`, Angular `DestroyRef`). Over-scoping makes the toolset flicker and
strands agents mid-plan.
- Registration failures roll back the scope and surface via `onError` — check the
console during integration; a silently missing toolset usually means a duplicate
name or invalid schema rejected the batch.
### Auth / roles (SaaS)
Never register a tool the current session couldn't use through the UI. On
login/logout/role change/tenant switch: dispose the scope and re-register the
correct set (`runtime.md` §Wiring). The server still re-checks everything (ground
rule 3) — role-scoped registration is UX hygiene, not security.
## Origin trial / flags note
WebMCP is a Chrome origin trial (149→, stable milestone still an estimate). For
production exposure the origin needs a token:
`<meta http-equiv="origin-trial" content="TOKEN">` or an `Origin-Trial` response
header — registered at the Chrome Origin Trials console. For local work,
`chrome://flags/#enable-webmcp-testing`. Chrome **silently ignores** expired
tokens, so nothing may depend on WebMCP being present (ground rule 4). Add a short
note about this to the target repo's README as part of setup, and record the
touched file path in `pipeline.setup.originTrialNoted` (e.g. `["README.md"]`).
+120
View File
@@ -0,0 +1,120 @@
# Inventory — mapping a codebase into a tool manifest
## Detect (Phase 0 details)
Establish, in this order:
1. **Stack**: `package.json` deps (react/vue/@angular/next/astro/eleventy…) or the
absence of one (static HTML). Record `app.stack` and `app.typescript`.
2. **Start command + base URL**: `dev`/`start` scripts, framework defaults
(`vite` → 5173, `next` → 3000, static → any file server). Verification needs a
working local run — if the app can't be started, append the blocker to
`pipeline.blockers` and surface it at the gate; don't silently proceed to a
verify phase that cannot run.
3. **Auth model**: none / session / role-based — plus **how a test session signs
in**, recorded per role under `app.authFixtures`: `obtain` (the exact steps —
seed command, login route), `account`, and `env` (the env var **names** the
fixture needs — never secret values in the manifest). The verify phase runs
from this. Role-based apps need role-scoped registration (`integrate.md`
§Auth) and a per-role verify pass.
4. **Git baseline**: `pipeline.baselineSha` = HEAD, `pipeline.baselineDirty` =
`git status --porcelain` paths. Dirty files are untouchable for the whole run.
## Building the area map
The area map is the unit of loop iteration. Sources, in order of preference:
router config (React Router, Next `app/`/`pages/`, Vue Router, Angular routes) →
navigation UI (static/SSG) → feature folders (`src/features/*`). Keep areas
coarse: 530 for a big SaaS, 13 for a landing page. Split an area that turns out
too big; merge trivial ones.
## What counts as a candidate tool
Walk each area's UI code and list **user actions**, not functions:
| UI pattern | Candidate tool | `mutating` | `readOnlyHint` |
|---|---|---|---|
| Search/filter form or input | `search_<noun>` | false | true |
| Data list/detail currently rendered | `list_<noun>` / `get_<noun>` | false | true |
| Create/edit form with submit → API call | `create_<noun>` / `update_<noun>` | "server" | — |
| Button triggering a server state change | `<verb>_<noun>` | "server" | — |
| Preference/theme/localStorage toggle | `<verb>_<noun>` | "client" | — |
| Multi-step flow (wizard, checkout) | `start_<noun>_flow` (initiation) | false* | **never** |
| Contact/booking form (static sites) | declarative form annotation | "server" | — |
*Initiation tools only navigate/open the flow — the human completes it. They are
classified non-mutating (no data changes) **but must NOT carry `readOnlyHint`**:
they change UI state, and agents skip confirmations for hinted-read-only tools.
`readOnlyHint: true` is reserved for genuinely pure data reads.
`mutating` is tri-state: `false` | `"client"` (browser-local only: prefs, theme,
localStorage — nothing leaves the browser) | `"server"` (data leaves the browser).
`"server"` gets the full ceremony — per-tool approval, required `cleanup`,
dev/test-data-only verification; `"client"` may be batch-approved at the gate
(`cleanup` recommended). `toolautosubmit` is banned for **both** mutation classes
(ground rule 5).
**Skip** (do not inventory): login/logout/auth flows, payment execution, account
deletion, user management, anything irreversible, file uploads (v1), and pure
navigation agents can do anyway.
## Tool budget, overlap, and priority (what keeps SaaS toolsets usable)
Agents degrade when many similar tools compete. Enforce while drafting:
- **Budget**: aim for ≤15 tools active in any app state (app-wide + current view).
If an area yields more candidates, keep the highest-value ones as `priority: 1`
and mark the rest `priority: 2/3` — the gate decides which waves ship.
- **Overlap rule**: no two tools whose descriptions could plausibly match the same
user request. Merge them (one tool, richer schema) or sharpen both descriptions
until they are disjoint.
- **Role/tenant coverage**: for role-scoped apps, note per tool which roles can use
it (`auth: ["role:<name>", ...]`); the toolset a given session sees must stay
within budget too.
## Naming and schema conventions (Google's, condensed)
- **Verb-first, execution vs initiation honest**: `create_event` acts immediately;
`start_event_creation_process` merely opens a form. The name must never lie.
- Name ≤30 chars, `[a-zA-Z0-9_.-]`; prefix with the app name if tools may coexist
with other origins' tools in testing (`myapp_search_tickets`).
- Description ≤500 chars, positive capability statement, no marketing. Param
descriptions ≤150 chars. The description must say exactly what `execute()` does —
agents make consent decisions from it.
- **Raw user input rule**: schemas accept what the user would say ("11:00 to
15:00"), never ask the agent to compute or transform. Semantic enum values
(`"High"`, not `priority_id: 3`).
- Tools returning user-generated or external content get
`untrustedContentHint: true`.
## Choosing `kind`
- `declarative` — any standard `<form>` whose fields map 1:1 to the action's
inputs: plain HTML, SSG-emitted, server-rendered, *and* framework-rendered forms
(uncontrolled inputs), including fetch-submitted forms (they bridge results via
`respondWith` — see `integrate.md`).
- `imperative` — non-form actions (buttons, drag/drop, selections), actions whose
inputs come from app state rather than form fields, and React/Vue **controlled**
forms (agent-driven fill would bypass the framework's state).
## Writing manifest entries
Fill EVERY field of the v3 schema:
- `route` + `auth` (array of roles keying into `app.authFixtures`; verify runs
once per role).
- `annotations``readOnlyHint`/`untrustedContentHint` per the candidate table;
verify asserts them on the enumerated tool.
- `examples` — one valid + one invalid. `invalid: null` is allowed ONLY for
readOnly tools with no/empty params (verify then asserts dual-outcome); the
convention for a non-null invalid on zero-param tools is `{"unexpected": true}`.
- `expect` — exactly ONE of `result` (substring of the resolved string) or
`navigation` (destination URL/pattern when `executeTool` resolves `null`),
plus `ui` (a UI assertion a test can check).
- `cleanup` — required for `mutating: "server"`, recommended for `"client"`.
The verify phase must be able to run from the manifest alone, without re-reading
the codebase — that is what makes runs resumable by a different agent.
The completeness pass at the end of Phase 1: start the app (or read the rendered
nav), enumerate what a user can *do* per screen, and diff against the manifest.
+129
View File
@@ -0,0 +1,129 @@
# Runtime — vendoring and wiring the templates
Copy from this skill's `templates/` directory into the target project
(suggested: `src/webmcp/`):
- **TypeScript projects**: `templates/webmcpify.ts` + `templates/webmcp.d.ts`;
**React TSX projects additionally** `templates/webmcp-jsx.d.ts` (JSX typings for
the declarative attributes — a MODULE file; keep it separate from
`webmcp.d.ts`, which must stay a global script file).
- **JavaScript projects**: `templates/webmcpify.js`**ES module only** (`export`):
load via a bundler or `<script type="module">`. For CommonJS/classic-script
projects, transpile or vendor the TS variant instead.
**Vendor, don't depend** — the runtime is small, MIT, and a target repo must not
gain a dependency for an origin-trial API. **Keep the full MIT notice header** in
every copied file: the license's retention condition requires the copyright line
and permission notice to travel with the code, and the header IS that notice —
never trim it down to a bare link.
Record the copied file paths in the manifest
(`pipeline.setup.runtimeVendored: ["src/webmcp/webmcpify.ts", ...]`).
What it provides:
| Export | Purpose |
|---|---|
| `getModelContext()` | The ONLY place `document.modelContext` / deprecated `navigator.modelContext` is referenced — spec churn stays a one-file fix |
| `isWebMCPAvailable()` | Feature detection — the app must work identically without WebMCP |
| `createToolScope(key, tools, options?)` | Registers a tool set under one AbortController; returns a **callable dispose handle** carrying `ready: Promise<boolean>` (true = all registrations committed; false = no WebMCP / duplicate key / failure / disposed first — never rejects). Validates contracts BEFORE registering; **rolls back the whole scope** on any failure, including sync-throwing legacy `registerTool` (reported via `options.onError`, default `console.error` — NOT called when disposed before settling). An already-active key returns a no-op handle — safe under React StrictMode |
| `dispatchAndWait(event, detail?, timeoutMs?)` | Bridges `execute()` to the app's own event/state flow. The dispatched detail carries `requestId` plus `signal` — an AbortSignal aborted on timeout; pass it to `fetch()` and skip state commits once aborted. Resolves only after the component confirms with an explicit **boolean** `ok`; a completion with missing/non-boolean `ok` **fails closed** to an `"ERROR: ..."` string, as do timeouts and `ok: false` (self-correction convention — never rejects). For tools whose confirmation involves a network round-trip (mailers, slow APIs), pass an explicit `timeoutMs` (e.g. `20_000`) instead of relying on the 10 s default |
| `singleFlight(fn, busyMessage?)` | Serializes a tool's `execute`: while one call is in flight, further calls resolve immediately to a busy `"ERROR: ..."` string instead of racing shared UI state |
Validation note: budget checks auto-enable when the bundler substitutes
`process.env.NODE_ENV` (Vite/webpack automatic; esbuild via `--define`) and it
isn't `'production'`; unbundled projects default to off — pass `{ validate: true }`
during development.
## The completion contract (the part integrators get wrong)
`dispatchAndWait` resolves when the component fires `tool-completion-<requestId>`
with `detail: { ok: boolean, message?: string, error?: string }`. `ok` must be an
explicit boolean — anything else fails closed to an ERROR result. Fire it **after
the async work has truly finished** — awaited fetch, committed state, rendered
result — never right after *starting* the action. Agents plan from what is on
screen; a completion fired early produces false greens.
Hardened component bridge (React example — adapt per framework). Five clauses:
**(1)** completion fires from an effect observing the committed state, **(2)**
availability gate, **(3)** single-flight, **(4)** timeout coordination via
`detail.signal`, **(5)** unmount cancellation.
```tsx
const pending = useRef<{ requestId: string; count: number } | null>(null);
const [results, setResults] = useState<Ticket[] | null>(null);
useEffect(() => {
if (!isWebMCPAvailable()) return; // (2) attach only when WebMCP exists
let inFlight = false;
const onSearch = async (e: Event) => {
const { query, requestId, signal } = (e as CustomEvent).detail;
const fail = (error: string) =>
window.dispatchEvent(new CustomEvent(`tool-completion-${requestId}`, {
detail: { ok: false, error },
}));
if (inFlight) return fail('A search is already running.'); // (3) single-flight
inFlight = true;
try {
const found = await runSearch(query, { signal }); // (4) the runtime aborts this signal on timeout
if (signal?.aborted) return; // (4) timed out — runtime already answered; no late commits
pending.current = { requestId, count: found.length };
setResults(found); // commit → the effect below confirms
} catch (err) {
if (signal?.aborted) return;
fail(err instanceof Error ? err.message : 'Search failed.');
} finally {
inFlight = false;
}
};
window.addEventListener('webmcp:search_tickets', onSearch);
return () => window.removeEventListener('webmcp:search_tickets', onSearch); // (5) unmount detaches
}, []);
useEffect(() => {
if (!pending.current || results === null) return; // (1) confirm AFTER the commit rendered
const { requestId, count } = pending.current;
pending.current = null;
window.dispatchEvent(new CustomEvent(`tool-completion-${requestId}`, {
detail: { ok: true, message: `Search finished — ${count} results are now visible.` },
}));
}, [results]);
```
Why clause (1): React 18 batches renders — state set after the `await` is **not
yet on screen** when the next line of the handler runs, so dispatching the
completion there reports success before the user (and the agent's next snapshot)
can see it. Dispatching from an effect keyed on the updated state guarantees the
commit happened. Equivalents: Vue `await nextTick()`; Svelte `await tick()`
then dispatch inline.
## Wiring patterns
```tsx
// bootstrap (app-wide tools, static registration — the default):
import { createToolScope } from './webmcp/webmcpify';
import { appTools } from './webmcp/tools';
createToolScope('app', appTools);
// per-view tools (only when genuinely view-bound):
useEffect(() => createToolScope('tickets-view', ticketViewTools), []);
// the handle IS the dispose fn → React runs it on unmount. StrictMode's
// double-mount is safe: the second call no-ops, an unmount before registration
// settles rolls back silently (ready → false, no onError).
// when you need to know registration committed:
const handle = createToolScope('app', appTools);
handle.ready.then((ok) => { if (!ok) console.warn('WebMCP tools not active'); });
```
Role-scoped SaaS registration — dispose and re-create on auth changes:
```ts
let dispose: (() => void) | undefined;
export function syncToolsForUser(user: User | null) {
dispose?.();
const tools = [...publicTools, ...(user ? memberTools : []),
...(user?.role === 'admin' ? adminTools : [])];
dispose = createToolScope('auth-scoped', tools);
}
// call on login, logout, role change, tenant switch
```
+73
View File
@@ -0,0 +1,73 @@
# Security checklist
Apply at two points: **before the manifest gate** (classify + flag) and **at the
final audit** (verify). Any unchecked box on a `mutating: "server"` tool blocks
it. Client-only mutations (`mutating: "client"`) still must pass the
**Trust boundary** and **Honesty & hints** boxes.
## Threat model in one paragraph
Any Chrome extension with host permissions — and any agent the user runs — can
enumerate and execute your tools **with the user's live session**. The spec has no
agent-identity mechanism. Page-visible strings (descriptions, labels, enum values,
tool outputs) all enter the model's context, so they are prompt-injection surface in
both directions. Design every tool as if it were a public, authenticated API endpoint
— because effectively it is one.
## Checklist
**Trust boundary**
- [ ] Every `execute()` calls only code paths the UI already uses — same endpoints,
same validation, same authz, same rate limits. No new endpoints, no bypasses.
- [ ] No secrets, tokens, or privileged config inside tool code or descriptions.
- [ ] Role-based apps: tools registered per role/session and re-scoped on auth
changes; nothing registered the current session couldn't do via the UI.
**Human-in-the-loop**
- [ ] No `toolautosubmit` on any state-changing form.
- [ ] No destructive/irreversible/payment tools at all in a first integration.
If the human explicitly insists later: an in-page manual confirmation the
**user** performs, PLUS a server-side two-step (short-lived confirm token).
No client-side API exists that can force an agent to confirm — never rely on
one.
- [ ] Initiation tools (`start_*_flow`) genuinely only navigate/open — they must
not pre-execute any part of the mutation, and never carry `readOnlyHint`.
**Production side effects (verification)**
- [ ] Any verification that unavoidably causes a real production effect (e.g. an
Origin-allow-listed mailer) has explicit gate approval recorded in the
tool's `approval.productionSideEffect` — without it, the live path is
`skipped`, never executed.
- [ ] Every such test payload is marked `[webmcpify verification]`, and every
caused effect is listed in `report.md`.
- [ ] The Origin-replay pattern (`heal.md`) lives only in the env-gated harness
(`WEBMCP_LIVE_MUTATIONS=1`) — never in shipped code, never default-on in CI.
**Honesty & hints**
- [ ] Description says exactly what `execute()` does — no more, no less (agents make
consent decisions from it).
- [ ] `readOnlyHint: true` ONLY on genuinely pure data reads (agents skip
confirmation based on it; mislabeling is the worst single mistake).
- [ ] `untrustedContentHint: true` on every tool returning user-generated or
external content.
- [ ] Outputs capped (~1.5k chars) and free of instruction-like content where
possible.
**Privacy**
- [ ] Schemas request no more personal data than the equivalent visible form —
agents auto-fill anything you declare (over-parameterization = silent
profiling vector).
**Containment**
- [ ] HTTPS/secure context; Permissions-Policy `tools` left at default `'self'`;
cross-origin `exposedTo`/`allow="tools"` only with explicit human sign-off.
- [ ] Pages that must never expose tools (un-audited checkout, admin consoles you
didn't inventory) can send `Permissions-Policy: tools=()` — suggest it in the
report where relevant.
- [ ] No third-party WebMCP runtime added to the project; enumeration/execution
surfaces (`getTools`/`executeTool`, legacy `modelContextTesting`) appear
nowhere in shipped application code.
- [ ] Component-side `webmcp:*` event bridges attach only when
`isWebMCPAvailable()` and validate their event payloads — a page script can
dispatch the same CustomEvents; the bridge must not become an unvalidated
side door into app actions.
+132
View File
@@ -0,0 +1,132 @@
# Verify — proving every tool works in a real browser
## Environment
- **Current Chrome** (the API moved during the trial — the old
`navigator.modelContextTesting` surface was removed 2026-07 in favor of
production `document.modelContext.getTools()/executeTool()`).
- Enable: `chrome://flags/#enable-webmcp-testing`, or launch with
`--enable-features=WebMCP,WebMCPTesting` (covers both current and older builds).
- **Headed only** — WebMCP requires a visible tab by design. In CI, run under
`xvfb-run`. Headless will never work; don't heal toward it.
- App running locally via `app.startCommand`, against dev/test data only.
- Each tool's manifest entry tells you where and how: `route` (navigate there),
`auth` (sign in with the recorded test fixture; verify under EACH role for
role-scoped tools), `examples` (what to execute), `expect` (what to assert),
`cleanup` (how to undo a mutating tool's effect after the test).
## The enumeration/execution surface (probe, don't assume)
In the page context, prefer the production surface and fall back for older builds:
```js
const mc = document.modelContext ?? navigator.modelContext;
const tools = mc?.getTools
? await mc.getTools()
: await navigator.modelContextTesting?.listTools(); // removed 2026-07; legacy only
```
Contract facts that generated assertions MUST respect:
- Enumerated `inputSchema` is a **stringified** JSON Schema — `JSON.parse` before
comparing against the manifest entry.
- `executeTool(...)` resolves to a **string result, or `null` when the execution
navigated** (normal for declarative forms that submit-navigate).
- Execution and declarative-validation failures **reject the promise** — they do
not resolve to `"ERROR: ..."`. Only imperative tools following the runtime's
convention resolve with `"ERROR: ..."` strings. Assert accordingly per tool
`kind`.
- **Registration is asynchronous** — `registerTool()` returns a promise, so a tool
is not enumerable the instant the page loads. Poll for it (`waitForTool` in the
template) or await a `toolchange` event; never assert presence immediately
after `goto`.
- **Mutating declarative forms pause mid-execution**: Chrome fills the form, then
waits for a real submit interaction before letting `executeTool` settle —
awaiting it alone deadlocks into a timeout. Use the concurrent pattern: start
`executeTool` unawaited → wait for an agent-filled value to appear → click
submit → await the result (full example in the template).
- These surfaces are for agents/harnesses only — they must never appear in shipped
application code.
For **declarative** tools also verify the *synthesized* schema: the form-control →
schema mapping is only partially specified, so check each annotated control appears
as the expected property in the actual target Chrome build.
## Per-tool checks
1. Registered (poll — registration is async) with the expected name, the (parsed)
schema, **and** the manifest `annotations` on the enumerated tool. The legacy
`modelContextTesting` fallback cannot enumerate annotations — skip that
assertion there and note the gap in the report.
2. Valid example executes: assert the result per `expect``expect.result` as a
substring of the resolved string, or `expect.navigation` as the destination
when `executeTool` resolves `null` (it navigated) — **and** the `expect.ui`
state as a **delta** (capture the relevant state *before* executing; mere
visibility of something already on screen proves nothing). A tool that reports
success without the UI changing is a **fail** (UI-settled rule). Because
executions can navigate, restore the manifest `route` in `beforeEach`, not
`beforeAll`.
3. Invalid example: **prove the tool is present first** (a rejection from a
never-registered tool is not a validation rejection). Then: imperative →
resolves `"ERROR: ..."`; declarative/schema violation → rejects. Zero-param
read tools with `examples.invalid: null` get the dual-outcome assertion
instead: `{"unexpected": true}` may be rejected with a validation reason OR
resolve benignly — both pass; a missing tool/surface fails.
4. Mutating tools: run against disposable data, verify the mutation through the
same read path the UI uses, then execute the manifest `cleanup` — a
`mutating: "server"` tool without working cleanup blocks at the gate, and
heal-loop retries of mutating tools must clean up between attempts.
## Harness
Instantiate `templates/webmcp.spec.ts` (bundled with this skill) — Playwright,
headed persistent Chrome, one describe-block per tool generated from the manifest,
with real assertions (never commented-out placeholders). Put the generated spec
next to the repo's existing e2e tests.
**Repos without a test setup — the standalone-harness recipe.** The spec stays in
`.webmcpify/webmcp.spec.ts` (single source of truth, committed per the gate's
`commitWebmcpifyDir` choice); the Playwright installation lives in a scratch
harness OUTSIDE the repo so the target gains no dependencies:
```sh
mkdir -p /tmp/webmcpify-harness && cd /tmp/webmcpify-harness
npm init -y && npm i -D @playwright/test typescript @types/node
cat > playwright.config.ts <<'EOF'
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: process.env.WEBMCP_SPEC_DIR, // → <target-repo>/.webmcpify
workers: 1, // one shared headed Chrome — never parallelize
});
EOF
WEBMCP_SPEC_DIR=<target-repo>/.webmcpify WEBMCP_BASE_URL=http://localhost:5173 \
NODE_PATH=/tmp/webmcpify-harness/node_modules npx playwright test
```
`NODE_PATH` lets the out-of-repo spec resolve `@playwright/test`; if the target's
tooling ignores `NODE_PATH`, symlink instead:
`ln -s /tmp/webmcpify-harness/node_modules <target-repo>/.webmcpify/node_modules`
(and make sure it isn't committed). Note in the report that verification ran from
a standalone harness.
**Alternative:** Puppeteer ships a first-class experimental WebMCP API
(https://pptr.dev/guides/webmcp) — prefer it when the target repo already uses
Puppeteer.
## Tool-selection evals (recommended; mandatory for SaaS-scale toolsets)
Schema-level verification proves tools *work*, not that an LLM *picks* them.
For apps exposing more than a handful of tools, run Google's **WebMCP Evals CLI**
(GoogleChromeLabs/webmcp-tools, `evals-cli`): write one eval case per tool from the
manifest examples ("user says X → expect tool Y with args Z") and run them —
this catches ambiguous names/descriptions and overlapping tools that Playwright
cannot.
## Manual QA (tell the human in the report)
- DevTools → **Application → WebMCP pane**: live tool list, invocation log,
"Run tool" with editable params.
- **Model Context Tool Inspector** Chrome extension (by Google's François
Beaufort): natural-language smoke tests of tool *selection*.
- Chrome's WebMCP audits flag missing `toolname`/`toolparamdescription`/
`label[for]`/`name` on declarative forms.