Files
awesome-copilot/.github/workflows/canvas-smoke-test.yml
T
d82df27473 Phase 3: canvas smoke-test evidence and weekly review metrics (#4184) (#4191)
* Phase 3: canvas smoke-test evidence, safe auto-merge, and weekly review metrics (#4184)

Adds the canvas-smoke-test check with review evidence, a config-gated safe auto-merge workflow (disabled by default), and a weekly review operating metrics workflow, plus maintainer docs.

Refs github/awesome-copilot#4184

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Trigger safe auto-merge sweep after Submission Gate Writer completes

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Rename thirdParty identifier to satisfy codespell

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Address Copilot review on canvas smoke test, auto-merge, and metrics

- Bind canvas report writer to the triggering workflow_run PR (base/head repo, ref, SHA)
- Bound PNG inflation by IHDR-derived size and a decoded-bytes budget
- Classify file: URLs as unsafe manifest paths
- Follow CommonJS require() and literal dynamic imports; error on missing
  reachable targets and devDependency-only runtime imports
- Validate removed extension paths and orphaned plugins instead of skipping
- Warn on .py/.rb/.pl scripts
- Auto-merge: support CODEOWNERS teams (fail closed), front matter authors,
  list PRs by head SHA, only fall back to direct merge on clean status,
  drop ineffective check_run trigger
- Metrics: ignore reviews before the PR was ready for review

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Docs: note microsoft/azure-dev-tools as a portable marketplace for the review model

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Auto-merge: only trust submission-gate runs from the Submission Gate Writer

Adds trusted_checks (default submission-gate -> external_id
submission-gate-writer). Any other check run or status with that name on the
head SHA blocks arming, so a PR-defined job cannot satisfy the gate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Harden canvas smoke-test checker

- String-, template-, and regex-aware comment stripping so literals containing // cannot hide imports
- Never follow symlinks or read non-regular files; bounded, contained reads
- Validate plugins whose plugin.json was deleted
- Reject unsafe extension IDs in plugin manifests before materialization

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Phase 3: remove auto-merge, isolate canvas smoke test, harden checker

- Remove safe auto-merge (config, workflow, script, tests, labels) per
  maintainer review; documented as deferred pending a security review.
- Restore validate-canvas-extensions.yml to main and move the smoke test
  to its own canvas-smoke-test.yml workflow scoped to extensions/**.
- Reject duplicate IHDR chunks in preview PNGs.
- Mask string/template/regex literal contents before extracting dynamic
  import() and require() specifiers.
- Resolve package.json imports aliases through the same classifier and
  module graph; fail closed on alias shapes that cannot be analyzed.
- Rename docs to canvas-evidence-and-metrics.md and update references.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix canvas smoke and review metrics findings

- Trigger canvas smoke tests for plugin changes while skipping expensive setup when target detection finds no canvas coverage.
- Fail reachable data: imports while preserving warnings for unreachable module data: imports.
- Use base plugin manifests for deleted bundle plugin detection and fail closed when base manifests cannot be read.
- Paginate GraphQL PR review pages before computing weekly review metrics.
- Report workflow run API collection errors as incomplete automation health instead of not found.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Address follow-up canvas review findings

Fail closed on non-literal runtime imports, require canonical PNG termination, restrict comment updates to github-actions[bot], and paginate external-plugin label events.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Aaron Powell <me@aaron-powell.com>
2026-10-07 16:36:13 +11:00

112 lines
4.6 KiB
YAML

name: Canvas Smoke Test
# Reader half of the canvas-smoke-test reader/writer split. Runs untrusted PR
# content with a read-only token and no secrets; extension code is parsed and
# compiled but never executed. canvas-smoke-test-comment.yml posts the report.
on:
pull_request:
branches: [main]
types: [opened, synchronize, reopened]
paths:
- "extensions/**"
- "plugins/**"
- ".github/workflows/canvas-smoke-test.yml"
- "eng/canvas-smoke-test.mjs"
permissions:
contents: read
concurrency:
group: canvas-smoke-test-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
canvas-smoke-test:
name: canvas-smoke-test
runs-on: ubuntu-latest
env:
PR_NUMBER: ${{ github.event.pull_request.number }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
CANVAS_PREVIEW_MIN_WIDTH: ${{ vars.CANVAS_PREVIEW_MIN_WIDTH }}
CANVAS_PREVIEW_MIN_HEIGHT: ${{ vars.CANVAS_PREVIEW_MIN_HEIGHT }}
CANVAS_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
fetch-depth: 0
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"
cache: "npm"
- name: Detect canvas targets
id: detect
run: |
set -euo pipefail
git diff --name-only "${BASE_SHA}...HEAD" > "${RUNNER_TEMP}/changed-files.txt"
node ./eng/canvas-smoke-test.mjs \
--changed-files "${RUNNER_TEMP}/changed-files.txt" \
--base-ref "${BASE_SHA}" \
--detect-only
- name: Write skipped report
if: steps.detect.outputs.canvas != 'true'
run: |
set -euo pipefail
mkdir -p canvas-smoke-results
printf '<!-- canvas-smoke-test -->\n## 🧩 Canvas smoke test\n\n⏭️ **Skipped** — no canvas extension or extension-bearing plugin paths changed.\n' > canvas-smoke-results/report.md
printf '{"schema_version":"canvas-smoke-test/v1","status":"skipped"}\n' > canvas-smoke-results/results.json
echo "${PR_NUMBER}" > canvas-smoke-results/pr-number.txt
echo "${HEAD_SHA}" > canvas-smoke-results/head-sha.txt
cat canvas-smoke-results/report.md >> "$GITHUB_STEP_SUMMARY"
- name: Install dependencies
if: steps.detect.outputs.canvas == 'true'
run: npm ci --ignore-scripts
- name: Install GitHub Copilot CLI
if: steps.detect.outputs.canvas == 'true'
continue-on-error: true
run: npm install -g @github/copilot
- name: Run canvas smoke test
if: steps.detect.outputs.canvas == 'true'
run: |
set -uo pipefail
export CANVAS_PREVIEW_BASE_URL="https://raw.githubusercontent.com/${HEAD_REPO}/${HEAD_SHA}/"
node ./eng/canvas-smoke-test.mjs \
--changed-files "${RUNNER_TEMP}/changed-files.txt" \
--base-ref "${BASE_SHA}" \
--install require \
--output-dir canvas-smoke-results > /dev/null
exit_code=$?
if [ ! -f canvas-smoke-results/report.md ]; then
mkdir -p canvas-smoke-results
printf '<!-- canvas-smoke-test -->\n## 🧩 Canvas smoke test\n\n⚠️ **Infrastructure error** — the checker crashed. See the workflow logs.\n' > canvas-smoke-results/report.md
echo '{"schema_version":"canvas-smoke-test/v1","status":"infra_error"}' > canvas-smoke-results/results.json
fi
echo "${PR_NUMBER}" > canvas-smoke-results/pr-number.txt
echo "${HEAD_SHA}" > canvas-smoke-results/head-sha.txt
cat canvas-smoke-results/report.md >> "$GITHUB_STEP_SUMMARY"
case "$exit_code" in
0) ;;
1) echo "::error::Canvas smoke test found contribution issues. See the job summary." ;;
*) echo "::error::Canvas smoke test hit an infrastructure error (not a contribution failure)." ;;
esac
exit "$exit_code"
- name: Upload review artifact
if: always() && hashFiles('canvas-smoke-results/report.md') != ''
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7
with:
name: canvas-smoke-test-results
path: canvas-smoke-results/
retention-days: 7