Phase 2: submission-gate, risk tiers, and PR status state machine (#4184) (#4190)

* Add submission-gate check, merge-risk tiers, and PR status state machine (#4184)

Phase 2 (enforcement) of github/awesome-copilot#4184:
- submission-gate aggregate required check (reader) with infra vs contribution failure classification
- merge-risk:low/medium/high classification from .github/risk-tiers.yml with tier approval policy
- Submission Gate Writer maintaining state labels and a persistent status comment
- /rerun-checks and /request-review PR commands
- validate-skills and validate-submission-gate workflows, unit tests, and maintainer docs

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix codespell findings and add spelling to submission gate checks

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Harden submission gate per review feedback

- Publish the submission-gate check from the trusted writer via the Checks API
  and flag any other submission-gate check run as tampering
- Fail closed on truncated file lists, unscannable diffs, permission lookup
  errors, and skipped required checks
- Discard evaluations when the PR head or reviews change mid-run
- Low tier requires a resource owner approval
- Commands must start the comment, matching the workflow filter

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Address #4190 review: command reader/writer, pools, paths, staleness

- Split /rerun-checks and /request-review into a read-only issue_comment
  reader and a workflow_run writer that re-reads the comment, PR, and
  permission before writing.
- Drop the workflow-security pool; agentic workflows and hooks are
  content (medium unless they add scripts or hook commands); the core
  pool is core-maintainers.
- Include each check's workflow file and eng scripts in its paths.
- Discard evaluations when the base branch or risk labels change; run
  the gate on PR edits (retargeting).
- Correct copies of the writer's check that claim success while the
  evaluation fails.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fail closed on lost contributor signal; publish gate check before labels

- A contributor check whose pr-check job succeeded but whose result artifact
  is missing or unreadable is now an infrastructure failure, so a lost HIGH
  signal can't lower the approval tier.
- Publish the submission-gate check right after stale-state revalidation;
  label and comment sync are independent and reported afterwards.
- Revert the self-path added to validate-agentic-workflows-pr.yml's trigger:
  that workflow rejects any .github change. Exempt it in the drift test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fail closed on bad routing config; advisory checks never hold the gate

- loadGateConfig: a missing review-routing.yml still falls back, but a
  file that can't be parsed (or whose pools isn't a mapping) now throws
  instead of silently relaxing approvals.
- Pending required: false checks no longer keep the PR in
  awaiting-automation or the gate check in progress, and the polling loop
  doesn't wait on them. They render as pending (advisory).
- PR commands: 👀 marks a command as claimed; 🚀 is written only after
  the reply is posted and is the only replay guard, so a run that failed
  partway can be retried.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix codespell: unparseable -> unparsable

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
James Montemagno
2026-10-01 11:39:09 +10:00
committed by GitHub
co-authored by Copilot App
parent 6c5a96403d
commit d6131471b8
18 changed files with 3550 additions and 0 deletions
@@ -5,6 +5,7 @@ on:
branches: [main]
paths:
- "plugins/**"
- ".github/workflows/check-plugin-structure.yml"
permissions:
contents: read
+98
View File
@@ -0,0 +1,98 @@
name: PR Commands Writer
# Trusted writer for /rerun-checks and /request-review. Runs default-branch code after the
# read-only PR Commands workflow completes. The artifact only names a comment and a PR;
# the comment body, the PR state, and the commenter's permission are re-read from the API
# before anything is written. See docs/maintainers/submission-gate.md.
on:
workflow_run:
workflows: ["PR Commands"]
types: [completed]
permissions:
contents: read
concurrency:
group: pr-commands-writer-${{ github.event.workflow_run.id }}
cancel-in-progress: false
jobs:
run-command:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
actions: write
contents: read
issues: write
pull-requests: write
if: >-
github.event.workflow_run.event == 'issue_comment' &&
github.event.workflow_run.conclusion == 'success'
steps:
- name: Download command request
id: download-request
continue-on-error: true
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: pr-command-request
path: ${{ runner.temp }}/pr-command-request
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}
- name: Checkout default branch
if: steps.download-request.outcome == 'success'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
persist-credentials: false
sparse-checkout: |
.github
eng
package.json
package-lock.json
- name: Setup Node.js
if: steps.download-request.outcome == 'success'
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"
cache: "npm"
- name: Install dependencies
if: steps.download-request.outcome == 'success'
run: npm ci --ignore-scripts
- name: Run command
if: steps.download-request.outcome == 'success'
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
with:
script: |
const fs = require('fs');
const path = require('path');
const { pathToFileURL } = require('url');
const root = process.env.GITHUB_WORKSPACE;
const gate = await import(pathToFileURL(path.join(root, 'eng', 'submission-gate.mjs')).href);
const config = gate.loadGateConfig(root);
const { owner, repo } = context.repo;
const workflowRun = context.payload.workflow_run;
const requestPath = path.join(process.env.RUNNER_TEMP, 'pr-command-request', 'request.json');
const raw = fs.readFileSync(requestPath, 'utf8');
if (raw.length > 4096) throw new Error('PR command request is too large');
const { prNumber, commentId } = gate.validatePrCommandRequest(JSON.parse(raw), { workflowRunId: workflowRun.id });
const result = await gate.runPrCommand(github, {
owner,
repo,
prNumber,
commentId,
config,
defaultBranch: context.payload.repository?.default_branch || 'main',
log: (message) => core.info(message),
});
core.info(result.status === 'handled' ? `Handled /${result.command} on #${prNumber}.` : `Ignored comment ${commentId}: ${result.reason}.`);
- name: Note missing artifact
if: steps.download-request.outcome != 'success'
run: echo "No pr-command-request artifact was available; nothing to do."
+53
View File
@@ -0,0 +1,53 @@
name: PR Commands
# Read-only reader for contributor and maintainer commands on pull requests:
# /rerun-checks re-run failed or incomplete checks and re-evaluate the submission gate
# /request-review ask reviewer routing to assign a reviewer (adds `needs-reviewer`)
# This workflow has no write access. It records which comment asked for a command and
# uploads that as an artifact; pr-commands-writer.yml re-reads the comment, the PR, and
# the commenter's permission from the API and performs the writes.
# The comment body is never interpolated into scripts. See docs/maintainers/submission-gate.md.
on:
issue_comment:
types: [created]
permissions:
contents: read
jobs:
record-command:
runs-on: ubuntu-latest
timeout-minutes: 5
if: >-
github.event.issue.pull_request &&
github.event.issue.state == 'open' &&
github.event.comment.user.type != 'Bot' &&
(startsWith(github.event.comment.body, '/rerun-checks') || startsWith(github.event.comment.body, '/request-review'))
steps:
- name: Record command request
env:
PR_NUMBER: ${{ github.event.issue.number }}
COMMENT_ID: ${{ github.event.comment.id }}
RUN_ID: ${{ github.run_id }}
run: |
set -euo pipefail
mkdir -p "$RUNNER_TEMP/pr-command-request"
node -e '
const fs = require("fs");
const request = {
schema_version: "pr-command-request/v1",
pr_number: Number(process.env.PR_NUMBER),
comment_id: Number(process.env.COMMENT_ID),
run_id: process.env.RUN_ID,
};
fs.writeFileSync(process.argv[1], JSON.stringify(request) + "\n");
' "$RUNNER_TEMP/pr-command-request/request.json"
- name: Upload command request
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: pr-command-request
path: ${{ runner.temp }}/pr-command-request/request.json
if-no-files-found: error
retention-days: 3
+2
View File
@@ -11,6 +11,8 @@ on:
- "plugins/**"
- "hooks/**"
- "instructions/**"
- "eng/pr-risk-scan.mjs"
- ".github/workflows/pr-risk-scan.yml"
permissions:
contents: read
+22
View File
@@ -90,6 +90,28 @@ jobs:
color: 'FBCA04',
description: 'External plugin awaiting maintainer approval'
},
// Submission gate labels (docs/maintainers/submission-gate.md).
// ready-for-review, requires-submitter-fixes, and approved are shared with intake above.
'awaiting-automation': {
color: 'FBCA04',
description: 'PR is waiting for automated checks to finish'
},
'review-in-progress': {
color: '5319E7',
description: 'PR checks passed and a maintainer review is in progress'
},
'merge-risk:low': {
color: 'C2E0C6',
description: 'Merge risk tier: low (docs, metadata, or small resource update)'
},
'merge-risk:medium': {
color: 'FEF2C0',
description: 'Merge risk tier: medium (new or substantially changed resource)'
},
'merge-risk:high': {
color: 'F9D0C4',
description: 'Merge risk tier: high (automation, scripts, MCP, hooks, or review policy)'
},
// Review routing labels (see .github/review-routing.yml)
'needs-reviewer': {
color: 'D4C5F9',
+1
View File
@@ -9,6 +9,7 @@ on:
- "agents/**"
- "plugins/**/skills/**"
- "plugins/**/agents/**"
- ".github/workflows/skill-check.yml"
permissions:
contents: read
@@ -0,0 +1,155 @@
name: Submission Gate Writer
# Trusted writer for the submission gate. Recomputes the PR evaluation from the GitHub
# API with code from the default branch (never from the PR), then publishes the
# required `submission-gate` check run on the PR head commit and applies exactly one
# merge-risk:* label, one state label, and the persistent status comment.
# See docs/maintainers/submission-gate.md.
on:
workflow_run:
workflows: ["Submission Gate"]
types: [requested, completed]
schedule:
# Safety net: re-runs started with GITHUB_TOKEN and review events do not always
# produce workflow_run events, so refresh recently updated open PRs hourly.
- cron: "23 * * * *"
workflow_dispatch:
inputs:
pr_number:
description: "PR number to refresh (leave empty to sweep recently updated open PRs)"
required: false
type: string
permissions:
actions: read
checks: write
contents: read
issues: write
pull-requests: write
concurrency:
group: submission-gate-writer-${{ github.event.workflow_run.head_repository.id || 'repo' }}-${{ github.event.workflow_run.head_branch || inputs.pr_number || github.event_name }}
cancel-in-progress: ${{ github.event_name == 'workflow_run' }}
jobs:
sync-status:
runs-on: ubuntu-latest
timeout-minutes: 20
if: >-
github.event_name != 'workflow_run' ||
github.event.workflow_run.event == 'pull_request' ||
(github.event.workflow_run.event == 'pull_request_review' && github.event.action == 'completed')
steps:
- name: Checkout default branch
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
persist-credentials: false
sparse-checkout: |
.github
eng
package.json
package-lock.json
- name: Setup Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"
cache: "npm"
- name: Install dependencies
run: npm ci --ignore-scripts
- name: Sync PR status
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ inputs.pr_number }}
SWEEP_LIMIT: "60"
SWEEP_MAX_AGE_DAYS: "30"
with:
script: |
const path = require('path');
const { pathToFileURL } = require('url');
const root = process.env.GITHUB_WORKSPACE;
const gate = await import(pathToFileURL(path.join(root, 'eng', 'submission-gate.mjs')).href);
const config = gate.loadGateConfig(root);
const { owner, repo } = context.repo;
const targets = [];
if (context.eventName === 'workflow_run') {
const workflowRun = context.payload.workflow_run;
const action = context.payload.action;
if (action === 'completed' && ['cancelled', 'skipped'].includes(workflowRun.conclusion)) {
core.info(`Gate run ${workflowRun.id} was ${workflowRun.conclusion}; a newer run supersedes it.`);
return;
}
const pullNumber = await gate.resolvePullRequestForWorkflowRun(github, { owner, repo, workflowRun });
if (!pullNumber) {
core.info(`No open PR matches gate run ${workflowRun.id} at ${workflowRun.head_sha}; it is stale or closed.`);
return;
}
targets.push({
pullNumber,
finalized: action === 'completed',
gateRunUrl: workflowRun.html_url,
expectedHeadSha: workflowRun.head_sha,
});
} else if (process.env.PR_NUMBER) {
const pullNumber = Number(process.env.PR_NUMBER);
if (!Number.isInteger(pullNumber) || pullNumber < 1) {
core.setFailed(`Invalid pr_number input: ${process.env.PR_NUMBER}`);
return;
}
targets.push({ pullNumber, finalized: 'auto' });
} else {
const limit = Number(process.env.SWEEP_LIMIT);
const cutoff = Date.now() - Number(process.env.SWEEP_MAX_AGE_DAYS) * 24 * 60 * 60 * 1000;
const open = await github.paginate(github.rest.pulls.list, {
owner,
repo,
state: 'open',
base: 'main',
sort: 'updated',
direction: 'desc',
per_page: 100,
});
for (const pr of open) {
if (targets.length >= limit || new Date(pr.updated_at).getTime() < cutoff) break;
targets.push({ pullNumber: pr.number, finalized: 'auto' });
}
}
let failures = 0;
for (const target of targets) {
try {
const evaluation = await gate.evaluateSubmission(github, {
owner,
repo,
pullNumber: target.pullNumber,
config,
finalized: target.finalized,
expectedHeadSha: target.expectedHeadSha || null,
token: process.env.GH_TOKEN,
log: (message) => core.info(message),
});
// A stale evaluation describes an older commit; the run for the new head reports.
if (evaluation.stale || evaluation.pr.state !== 'open') continue;
const gateRunUrl = target.gateRunUrl || evaluation.gateRun?.html_url || null;
await gate.syncPullRequestStatus(github, {
owner,
repo,
evaluation,
gateRunUrl,
publishCheck: true,
log: (message) => core.info(message),
});
} catch (error) {
failures += 1;
core.warning(`Could not sync PR #${target.pullNumber}: ${error.message}`);
}
}
if (failures > 0 && failures === targets.length) {
core.setFailed(`Failed to sync ${failures} PR(s).`);
}
+112
View File
@@ -0,0 +1,112 @@
name: Submission Gate
# Read-only preview for every PR. Waits for the applicable checks listed in
# .github/submission-gate.yml, classifies the PR into a merge-risk tier
# (.github/risk-tiers.yml), and previews the approvals that tier requires.
# This workflow never publishes the required `submission-gate` check: a PR can edit
# pull_request workflows, so the trusted submission-gate-writer.yml recomputes the
# result with default-branch code and publishes that check, the labels, and the
# status comment. See docs/maintainers/submission-gate.md.
on:
pull_request:
# `edited` covers retargeting to another base branch, which changes the applicable checks.
types: [opened, synchronize, reopened, ready_for_review, edited]
pull_request_review:
types: [submitted, edited, dismissed]
permissions:
actions: read
checks: read
contents: read
pull-requests: read
concurrency:
group: submission-gate-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
# Must not be named submission-gate: the writer treats any other check with that name as tampering.
evaluate:
name: evaluate
runs-on: ubuntu-latest
timeout-minutes: 50
steps:
# Gate logic and policy come from the base commit so a PR cannot change how it is judged.
- name: Checkout trusted base
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
ref: ${{ github.event.pull_request.base.sha }}
persist-credentials: false
sparse-checkout: |
.github
eng
package.json
package-lock.json
- name: Check for gate logic on base
id: base
run: |
if [ -f eng/submission-gate.mjs ] && [ -f .github/submission-gate.yml ] && [ -f .github/risk-tiers.yml ]; then
echo "has-gate=true" >> "$GITHUB_OUTPUT"
else
echo "has-gate=false" >> "$GITHUB_OUTPUT"
fi
# Bootstrap only: the PR that introduces the gate has no base copy of it. This job
# has a read-only token, and such PRs are merge-risk:high by path.
- name: Checkout PR (bootstrap)
if: steps.base.outputs.has-gate != 'true'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
persist-credentials: false
clean: true
- name: Setup Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"
cache: "npm"
- name: Install dependencies
run: npm ci --ignore-scripts
- name: Evaluate submission
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
env:
GH_TOKEN: ${{ github.token }}
BOOTSTRAP: ${{ steps.base.outputs.has-gate != 'true' }}
with:
script: |
const path = require('path');
const { pathToFileURL } = require('url');
if (process.env.BOOTSTRAP === 'true') {
core.warning('The base branch has no submission gate yet; evaluating with the gate from this PR.');
}
const root = process.env.GITHUB_WORKSPACE;
const gate = await import(pathToFileURL(path.join(root, 'eng', 'submission-gate.mjs')).href);
const config = gate.loadGateConfig(root);
const pullNumber = context.payload.pull_request.number;
const evaluation = await gate.evaluateSubmission(github, {
owner: context.repo.owner,
repo: context.repo.repo,
pullNumber,
config,
expectedHeadSha: context.payload.pull_request.head.sha,
wait: true,
token: process.env.GH_TOKEN,
log: (message) => core.info(message),
});
if (evaluation.stale) {
core.info('The PR head changed during evaluation; the run for the new head will report instead.');
return;
}
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
await core.summary.addRaw(gate.renderStatusComment(evaluation, { gateRunUrl: runUrl })).write();
// Preview only; the writer publishes the authoritative submission-gate check.
core.info(`PR #${pullNumber}: state=${evaluation.state}, risk=${evaluation.risk.tier}`);
for (const line of gate.gateFailureSummary(evaluation)) core.notice(line);
@@ -6,6 +6,8 @@ on:
types: [opened, synchronize, reopened]
paths:
- "extensions/**"
- "eng/validate-plugins.mjs"
- ".github/workflows/validate-canvas-extensions.yml"
permissions:
contents: read
+4
View File
@@ -14,6 +14,10 @@ on:
- "README.md"
- "docs/**"
- "skills/**"
- "eng/update-readme.mjs"
- "eng/generate-marketplace.mjs"
- "eng/validate-plugins.mjs"
- ".github/workflows/validate-readme.yml"
jobs:
validate-readme:
+35
View File
@@ -0,0 +1,35 @@
name: Validate Skills
on:
pull_request:
branches: [main]
paths:
- "skills/**"
- "eng/validate-skills.mjs"
- "eng/yaml-parser.mjs"
- ".github/workflows/validate-skills.yml"
permissions:
contents: read
jobs:
validate-skills:
name: validate-skills
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"
cache: "npm"
- name: Install dependencies
run: npm ci --ignore-scripts
- name: Validate skills
run: npm run skill:validate
@@ -0,0 +1,36 @@
name: Validate Submission Gate
on:
pull_request:
branches: [main]
paths:
- ".github/submission-gate.yml"
- ".github/risk-tiers.yml"
- ".github/workflows/**"
- "eng/submission-gate.mjs"
- "eng/submission-gate.test.mjs"
permissions:
contents: read
jobs:
validate-submission-gate:
name: validate-submission-gate
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"
cache: "npm"
- name: Install dependencies
run: npm ci --ignore-scripts
- name: Test submission gate
run: node --test eng/submission-gate.test.mjs