mirror of
https://github.com/github/awesome-copilot.git
synced 2026-10-12 23:04:14 +00:00
* Phase 3: canvas smoke-test evidence, safe auto-merge, and weekly review metrics (#4184) Adds the canvas-smoke-test check with review evidence, a config-gated safe auto-merge workflow (disabled by default), and a weekly review operating metrics workflow, plus maintainer docs. Refs github/awesome-copilot#4184 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Trigger safe auto-merge sweep after Submission Gate Writer completes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Rename thirdParty identifier to satisfy codespell Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address Copilot review on canvas smoke test, auto-merge, and metrics - Bind canvas report writer to the triggering workflow_run PR (base/head repo, ref, SHA) - Bound PNG inflation by IHDR-derived size and a decoded-bytes budget - Classify file: URLs as unsafe manifest paths - Follow CommonJS require() and literal dynamic imports; error on missing reachable targets and devDependency-only runtime imports - Validate removed extension paths and orphaned plugins instead of skipping - Warn on .py/.rb/.pl scripts - Auto-merge: support CODEOWNERS teams (fail closed), front matter authors, list PRs by head SHA, only fall back to direct merge on clean status, drop ineffective check_run trigger - Metrics: ignore reviews before the PR was ready for review Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Docs: note microsoft/azure-dev-tools as a portable marketplace for the review model Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Auto-merge: only trust submission-gate runs from the Submission Gate Writer Adds trusted_checks (default submission-gate -> external_id submission-gate-writer). Any other check run or status with that name on the head SHA blocks arming, so a PR-defined job cannot satisfy the gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden canvas smoke-test checker - String-, template-, and regex-aware comment stripping so literals containing // cannot hide imports - Never follow symlinks or read non-regular files; bounded, contained reads - Validate plugins whose plugin.json was deleted - Reject unsafe extension IDs in plugin manifests before materialization Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Phase 3: remove auto-merge, isolate canvas smoke test, harden checker - Remove safe auto-merge (config, workflow, script, tests, labels) per maintainer review; documented as deferred pending a security review. - Restore validate-canvas-extensions.yml to main and move the smoke test to its own canvas-smoke-test.yml workflow scoped to extensions/**. - Reject duplicate IHDR chunks in preview PNGs. - Mask string/template/regex literal contents before extracting dynamic import() and require() specifiers. - Resolve package.json imports aliases through the same classifier and module graph; fail closed on alias shapes that cannot be analyzed. - Rename docs to canvas-evidence-and-metrics.md and update references. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix canvas smoke and review metrics findings - Trigger canvas smoke tests for plugin changes while skipping expensive setup when target detection finds no canvas coverage. - Fail reachable data: imports while preserving warnings for unreachable module data: imports. - Use base plugin manifests for deleted bundle plugin detection and fail closed when base manifests cannot be read. - Paginate GraphQL PR review pages before computing weekly review metrics. - Report workflow run API collection errors as incomplete automation health instead of not found. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address follow-up canvas review findings Fail closed on non-literal runtime imports, require canonical PNG termination, restrict comment updates to github-actions[bot], and paginate external-plugin label events. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Aaron Powell <me@aaron-powell.com>
This commit is contained in:
James Montemagno
co-authored by
Copilot App
Aaron Powell
Copilot App
Aaron Powell
parent
b30b12e40e
commit
d82df27473
@@ -0,0 +1,61 @@
|
|||||||
|
# Weekly review operating metrics (github/awesome-copilot#4184, phase 3).
|
||||||
|
#
|
||||||
|
# Read by eng/review-metrics.mjs, which .github/workflows/review-metrics.yml
|
||||||
|
# runs every Monday. See docs/maintainers/canvas-evidence-and-metrics.md.
|
||||||
|
|
||||||
|
# Days of activity summarized in each report.
|
||||||
|
window_days: 7
|
||||||
|
|
||||||
|
# Review targets, in business days (weekends are skipped).
|
||||||
|
targets_business_days: [2, 4]
|
||||||
|
|
||||||
|
# Base branch for contribution PRs.
|
||||||
|
base_branch: main
|
||||||
|
|
||||||
|
# State labels (maintained by the submission gate). Open PRs without any of
|
||||||
|
# these are reported as "unlabeled".
|
||||||
|
state_labels:
|
||||||
|
- awaiting-automation
|
||||||
|
- requires-submitter-fixes
|
||||||
|
- ready-for-review
|
||||||
|
- review-in-progress
|
||||||
|
- approved
|
||||||
|
|
||||||
|
# Risk tier labels. Open PRs without any of these are "unclassified".
|
||||||
|
risk_labels:
|
||||||
|
- merge-risk:low
|
||||||
|
- merge-risk:medium
|
||||||
|
- merge-risk:high
|
||||||
|
|
||||||
|
# External plugin submission issues are tracked alongside PRs.
|
||||||
|
external_plugin_label: external-plugin
|
||||||
|
external_plugin_state_labels:
|
||||||
|
- awaiting-review
|
||||||
|
- requires-submitter-fixes
|
||||||
|
- ready-for-review
|
||||||
|
- awaiting-approval
|
||||||
|
|
||||||
|
# Review-automation workflows whose failed/errored runs count toward the
|
||||||
|
# automation failure rate. Files that do not exist yet are reported as
|
||||||
|
# "not found" and skipped, so later phases can be listed ahead of time.
|
||||||
|
automation_workflows:
|
||||||
|
- submission-gate.yml
|
||||||
|
- submission-gate-writer.yml
|
||||||
|
- review-routing.yml
|
||||||
|
- canvas-smoke-test.yml
|
||||||
|
- canvas-smoke-test-comment.yml
|
||||||
|
- pr-risk-scan.yml
|
||||||
|
- pr-risk-scan-comment.yml
|
||||||
|
- label-pr-intent.yml
|
||||||
|
- label-pr-intent-writer.yml
|
||||||
|
- check-plugin-structure.yml
|
||||||
|
- external-plugin-intake.yml
|
||||||
|
- external-plugin-quality-gates.yml
|
||||||
|
- external-plugin-pr-quality-gates.yml
|
||||||
|
- external-plugin-pr-quality-gates-writer.yml
|
||||||
|
- external-plugin-command-router.yml
|
||||||
|
|
||||||
|
# Tracking issue that receives the weekly report.
|
||||||
|
tracking_issue:
|
||||||
|
title: Review operating metrics
|
||||||
|
label: review-metrics
|
||||||
@@ -0,0 +1,157 @@
|
|||||||
|
name: Canvas Smoke Test — Comment
|
||||||
|
|
||||||
|
# Writer half of the canvas-smoke-test reader/writer split. The reader
|
||||||
|
# (canvas-smoke-test.yml) runs on untrusted PR code with a read-only
|
||||||
|
# token; this workflow only reads its artifact and never checks out PR code.
|
||||||
|
|
||||||
|
on:
|
||||||
|
workflow_run:
|
||||||
|
workflows: ["Canvas Smoke Test"]
|
||||||
|
types: [completed]
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
actions: read
|
||||||
|
issues: write
|
||||||
|
pull-requests: write
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
comment:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
if: github.event.workflow_run.event == 'pull_request'
|
||||||
|
steps:
|
||||||
|
- name: Download review artifact
|
||||||
|
id: download
|
||||||
|
continue-on-error: true
|
||||||
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||||
|
with:
|
||||||
|
name: canvas-smoke-test-results
|
||||||
|
run-id: ${{ github.event.workflow_run.id }}
|
||||||
|
github-token: ${{ github.token }}
|
||||||
|
|
||||||
|
- name: Upsert PR comment
|
||||||
|
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
|
||||||
|
with:
|
||||||
|
script: |
|
||||||
|
const fs = require('fs');
|
||||||
|
const marker = '<!-- canvas-smoke-test -->';
|
||||||
|
|
||||||
|
if (!fs.existsSync('report.md')) {
|
||||||
|
core.info('No canvas smoke test report artifact found. Skipping.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
let status = 'unknown';
|
||||||
|
try {
|
||||||
|
status = JSON.parse(fs.readFileSync('results.json', 'utf8')).status || 'unknown';
|
||||||
|
} catch {
|
||||||
|
core.warning('results.json is missing or invalid.');
|
||||||
|
}
|
||||||
|
|
||||||
|
const workflowRun = context.payload.workflow_run;
|
||||||
|
const artifactPr = fs.existsSync('pr-number.txt') ? fs.readFileSync('pr-number.txt', 'utf8').trim() : '';
|
||||||
|
const prNumber = /^[1-9][0-9]*$/.test(artifactPr) ? Number(artifactPr) : workflowRun.pull_requests?.[0]?.number;
|
||||||
|
if (!prNumber) {
|
||||||
|
core.warning('Could not determine PR number. Skipping.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The artifact is untrusted: bind the target PR to the triggering run
|
||||||
|
// (base repo, head repo/ref/SHA, and a unique association) before writing.
|
||||||
|
const { data: pr } = await github.rest.pulls.get({
|
||||||
|
owner: context.repo.owner,
|
||||||
|
repo: context.repo.repo,
|
||||||
|
pull_number: prNumber,
|
||||||
|
});
|
||||||
|
if (pr.state !== 'open') {
|
||||||
|
core.info(`Skipping non-open PR #${prNumber}.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const expectedBaseRepository = `${context.repo.owner}/${context.repo.repo}`.toLowerCase();
|
||||||
|
const runHeadRepository = String(workflowRun.head_repository?.full_name || '');
|
||||||
|
const runHeadRepositoryParts = runHeadRepository.split('/');
|
||||||
|
const runHeadRef = String(workflowRun.head_branch || '');
|
||||||
|
if (String(pr.base?.repo?.full_name || '').toLowerCase() !== expectedBaseRepository) {
|
||||||
|
core.setFailed(`PR #${prNumber} does not target this repository.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (pr.head.sha !== workflowRun.head_sha) {
|
||||||
|
core.info(`PR #${prNumber} head ${pr.head.sha} does not match run head ${workflowRun.head_sha}. Skipping stale or mismatched report.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
runHeadRepositoryParts.length !== 2 ||
|
||||||
|
!runHeadRepositoryParts[0] ||
|
||||||
|
!runHeadRepositoryParts[1] ||
|
||||||
|
!runHeadRef ||
|
||||||
|
String(pr.head?.repo?.full_name || '').toLowerCase() !== runHeadRepository.toLowerCase() ||
|
||||||
|
String(pr.head?.ref || '') !== runHeadRef
|
||||||
|
) {
|
||||||
|
core.setFailed(`PR #${prNumber} head repository/ref did not match workflow_run.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const workflowRunPullRequests = Array.isArray(workflowRun.pull_requests) ? workflowRun.pull_requests : [];
|
||||||
|
if (workflowRunPullRequests.length > 0) {
|
||||||
|
if (!workflowRunPullRequests.some((pullRequest) => pullRequest.number === prNumber)) {
|
||||||
|
core.setFailed(`PR #${prNumber} was not present in workflow_run.pull_requests.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const candidatePullRequests = await github.paginate(github.rest.pulls.list, {
|
||||||
|
owner: context.repo.owner,
|
||||||
|
repo: context.repo.repo,
|
||||||
|
state: 'open',
|
||||||
|
head: `${runHeadRepositoryParts[0]}:${runHeadRef}`,
|
||||||
|
per_page: 100,
|
||||||
|
});
|
||||||
|
const trustedMatches = candidatePullRequests.filter((candidate) =>
|
||||||
|
candidate.head?.sha === workflowRun.head_sha &&
|
||||||
|
String(candidate.head?.ref || '') === runHeadRef &&
|
||||||
|
String(candidate.head?.repo?.full_name || '').toLowerCase() === runHeadRepository.toLowerCase() &&
|
||||||
|
String(candidate.base?.repo?.full_name || '').toLowerCase() === expectedBaseRepository
|
||||||
|
);
|
||||||
|
if (trustedMatches.length !== 1 || trustedMatches[0].number !== prNumber) {
|
||||||
|
core.setFailed(`PR #${prNumber} could not be uniquely associated with workflow_run.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let body = fs.readFileSync('report.md', 'utf8').replace(/@/g, '@\u200b');
|
||||||
|
const maxLength = 65000;
|
||||||
|
if (body.length > maxLength) {
|
||||||
|
body = `${body.slice(0, maxLength)}\n\n_...(truncated)..._`;
|
||||||
|
}
|
||||||
|
if (!body.includes(marker)) {
|
||||||
|
body = `${marker}\n${body}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const comments = await github.paginate(github.rest.issues.listComments, {
|
||||||
|
owner: context.repo.owner,
|
||||||
|
repo: context.repo.repo,
|
||||||
|
issue_number: prNumber,
|
||||||
|
per_page: 100,
|
||||||
|
});
|
||||||
|
const existing = comments.find((comment) =>
|
||||||
|
comment.user?.login === 'github-actions[bot]' && comment.body?.includes(marker));
|
||||||
|
|
||||||
|
if (status === 'skipped' && !existing) {
|
||||||
|
core.info('No canvas changes and no previous comment. Nothing to post.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (existing) {
|
||||||
|
await github.rest.issues.updateComment({
|
||||||
|
owner: context.repo.owner,
|
||||||
|
repo: context.repo.repo,
|
||||||
|
comment_id: existing.id,
|
||||||
|
body,
|
||||||
|
});
|
||||||
|
core.info(`Updated canvas smoke test comment ${existing.id}`);
|
||||||
|
} else {
|
||||||
|
await github.rest.issues.createComment({
|
||||||
|
owner: context.repo.owner,
|
||||||
|
repo: context.repo.repo,
|
||||||
|
issue_number: prNumber,
|
||||||
|
body,
|
||||||
|
});
|
||||||
|
core.info('Created canvas smoke test comment');
|
||||||
|
}
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
name: Canvas Smoke Test
|
||||||
|
|
||||||
|
# Reader half of the canvas-smoke-test reader/writer split. Runs untrusted PR
|
||||||
|
# content with a read-only token and no secrets; extension code is parsed and
|
||||||
|
# compiled but never executed. canvas-smoke-test-comment.yml posts the report.
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
branches: [main]
|
||||||
|
types: [opened, synchronize, reopened]
|
||||||
|
paths:
|
||||||
|
- "extensions/**"
|
||||||
|
- "plugins/**"
|
||||||
|
- ".github/workflows/canvas-smoke-test.yml"
|
||||||
|
- "eng/canvas-smoke-test.mjs"
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: canvas-smoke-test-${{ github.event.pull_request.number }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
canvas-smoke-test:
|
||||||
|
name: canvas-smoke-test
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
env:
|
||||||
|
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||||
|
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
||||||
|
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
|
||||||
|
BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
||||||
|
CANVAS_PREVIEW_MIN_WIDTH: ${{ vars.CANVAS_PREVIEW_MIN_WIDTH }}
|
||||||
|
CANVAS_PREVIEW_MIN_HEIGHT: ${{ vars.CANVAS_PREVIEW_MIN_HEIGHT }}
|
||||||
|
CANVAS_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
persist-credentials: false
|
||||||
|
|
||||||
|
- name: Setup Node.js
|
||||||
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
||||||
|
with:
|
||||||
|
node-version: "22"
|
||||||
|
cache: "npm"
|
||||||
|
|
||||||
|
- name: Detect canvas targets
|
||||||
|
id: detect
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
git diff --name-only "${BASE_SHA}...HEAD" > "${RUNNER_TEMP}/changed-files.txt"
|
||||||
|
node ./eng/canvas-smoke-test.mjs \
|
||||||
|
--changed-files "${RUNNER_TEMP}/changed-files.txt" \
|
||||||
|
--base-ref "${BASE_SHA}" \
|
||||||
|
--detect-only
|
||||||
|
|
||||||
|
- name: Write skipped report
|
||||||
|
if: steps.detect.outputs.canvas != 'true'
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
mkdir -p canvas-smoke-results
|
||||||
|
printf '<!-- canvas-smoke-test -->\n## 🧩 Canvas smoke test\n\n⏭️ **Skipped** — no canvas extension or extension-bearing plugin paths changed.\n' > canvas-smoke-results/report.md
|
||||||
|
printf '{"schema_version":"canvas-smoke-test/v1","status":"skipped"}\n' > canvas-smoke-results/results.json
|
||||||
|
echo "${PR_NUMBER}" > canvas-smoke-results/pr-number.txt
|
||||||
|
echo "${HEAD_SHA}" > canvas-smoke-results/head-sha.txt
|
||||||
|
cat canvas-smoke-results/report.md >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
|
||||||
|
- name: Install dependencies
|
||||||
|
if: steps.detect.outputs.canvas == 'true'
|
||||||
|
run: npm ci --ignore-scripts
|
||||||
|
|
||||||
|
- name: Install GitHub Copilot CLI
|
||||||
|
if: steps.detect.outputs.canvas == 'true'
|
||||||
|
continue-on-error: true
|
||||||
|
run: npm install -g @github/copilot
|
||||||
|
|
||||||
|
- name: Run canvas smoke test
|
||||||
|
if: steps.detect.outputs.canvas == 'true'
|
||||||
|
run: |
|
||||||
|
set -uo pipefail
|
||||||
|
export CANVAS_PREVIEW_BASE_URL="https://raw.githubusercontent.com/${HEAD_REPO}/${HEAD_SHA}/"
|
||||||
|
node ./eng/canvas-smoke-test.mjs \
|
||||||
|
--changed-files "${RUNNER_TEMP}/changed-files.txt" \
|
||||||
|
--base-ref "${BASE_SHA}" \
|
||||||
|
--install require \
|
||||||
|
--output-dir canvas-smoke-results > /dev/null
|
||||||
|
exit_code=$?
|
||||||
|
if [ ! -f canvas-smoke-results/report.md ]; then
|
||||||
|
mkdir -p canvas-smoke-results
|
||||||
|
printf '<!-- canvas-smoke-test -->\n## 🧩 Canvas smoke test\n\n⚠️ **Infrastructure error** — the checker crashed. See the workflow logs.\n' > canvas-smoke-results/report.md
|
||||||
|
echo '{"schema_version":"canvas-smoke-test/v1","status":"infra_error"}' > canvas-smoke-results/results.json
|
||||||
|
fi
|
||||||
|
echo "${PR_NUMBER}" > canvas-smoke-results/pr-number.txt
|
||||||
|
echo "${HEAD_SHA}" > canvas-smoke-results/head-sha.txt
|
||||||
|
cat canvas-smoke-results/report.md >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
case "$exit_code" in
|
||||||
|
0) ;;
|
||||||
|
1) echo "::error::Canvas smoke test found contribution issues. See the job summary." ;;
|
||||||
|
*) echo "::error::Canvas smoke test hit an infrastructure error (not a contribution failure)." ;;
|
||||||
|
esac
|
||||||
|
exit "$exit_code"
|
||||||
|
|
||||||
|
- name: Upload review artifact
|
||||||
|
if: always() && hashFiles('canvas-smoke-results/report.md') != ''
|
||||||
|
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7
|
||||||
|
with:
|
||||||
|
name: canvas-smoke-test-results
|
||||||
|
path: canvas-smoke-results/
|
||||||
|
retention-days: 7
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
name: Review Operating Metrics
|
||||||
|
|
||||||
|
# Publishes weekly review operating metrics (#4184) to a pinned tracking
|
||||||
|
# issue labeled `review-metrics`. Reads repository data only; the sole write is
|
||||||
|
# the tracking issue. Definitions: docs/maintainers/canvas-evidence-and-metrics.md
|
||||||
|
|
||||||
|
on:
|
||||||
|
schedule:
|
||||||
|
# Mondays 14:00 UTC, covering the previous seven days.
|
||||||
|
- cron: "0 14 * * 1"
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
window_days:
|
||||||
|
description: "Days of activity to summarize"
|
||||||
|
required: false
|
||||||
|
default: "7"
|
||||||
|
type: string
|
||||||
|
dry_run:
|
||||||
|
description: "Only write the job summary; do not update the tracking issue"
|
||||||
|
required: false
|
||||||
|
default: false
|
||||||
|
type: boolean
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: review-metrics
|
||||||
|
cancel-in-progress: false
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
metrics:
|
||||||
|
name: review-metrics
|
||||||
|
if: github.repository_owner == 'github' || github.event_name == 'workflow_dispatch'
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 15
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
issues: write
|
||||||
|
pull-requests: read
|
||||||
|
actions: read
|
||||||
|
steps:
|
||||||
|
- name: Checkout repository
|
||||||
|
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
|
||||||
|
- name: Setup Node.js
|
||||||
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
||||||
|
with:
|
||||||
|
node-version: "22"
|
||||||
|
cache: npm
|
||||||
|
|
||||||
|
- name: Install dependencies
|
||||||
|
run: npm ci --ignore-scripts
|
||||||
|
|
||||||
|
- name: Compute and publish metrics
|
||||||
|
env:
|
||||||
|
GITHUB_TOKEN: ${{ github.token }}
|
||||||
|
METRICS_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||||
|
WINDOW_DAYS: ${{ inputs.window_days || '7' }}
|
||||||
|
DRY_RUN: ${{ inputs.dry_run && 'true' || 'false' }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
args=(--repo "$GITHUB_REPOSITORY" --window-days "$WINDOW_DAYS" --output-dir "$RUNNER_TEMP/review-metrics")
|
||||||
|
if [ "$DRY_RUN" = "true" ]; then args+=(--dry-run); fi
|
||||||
|
node ./eng/review-metrics.mjs "${args[@]}"
|
||||||
|
|
||||||
|
- name: Upload metrics
|
||||||
|
if: always()
|
||||||
|
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7
|
||||||
|
with:
|
||||||
|
name: review-metrics
|
||||||
|
path: ${{ runner.temp }}/review-metrics
|
||||||
|
if-no-files-found: ignore
|
||||||
|
retention-days: 90
|
||||||
@@ -90,6 +90,11 @@ jobs:
|
|||||||
color: 'FBCA04',
|
color: 'FBCA04',
|
||||||
description: 'External plugin awaiting maintainer approval'
|
description: 'External plugin awaiting maintainer approval'
|
||||||
},
|
},
|
||||||
|
// Review metrics tracking label (see docs/maintainers/canvas-evidence-and-metrics.md)
|
||||||
|
'review-metrics': {
|
||||||
|
color: 'C5DEF5',
|
||||||
|
description: 'Weekly review operating metrics tracking issue'
|
||||||
|
},
|
||||||
// Submission gate labels (docs/maintainers/submission-gate.md).
|
// Submission gate labels (docs/maintainers/submission-gate.md).
|
||||||
// ready-for-review, requires-submitter-fixes, and approved are shared with intake above.
|
// ready-for-review, requires-submitter-fixes, and approved are shared with intake above.
|
||||||
'awaiting-automation': {
|
'awaiting-automation': {
|
||||||
|
|||||||
@@ -0,0 +1,114 @@
|
|||||||
|
# Canvas review evidence and review metrics
|
||||||
|
|
||||||
|
This guide covers phase 3 ("safe acceleration") of [#4184](https://github.com/github/awesome-copilot/issues/4184):
|
||||||
|
|
||||||
|
1. [Canvas review evidence](#canvas-review-evidence): the `canvas-smoke-test` check.
|
||||||
|
2. [Weekly operating metrics](#weekly-operating-metrics): `.github/review-metrics.yml` and `.github/workflows/review-metrics.yml`.
|
||||||
|
3. [Deferred: auto-merge](#deferred-auto-merge).
|
||||||
|
4. [Portability to MOS3](#portability-to-mos3).
|
||||||
|
|
||||||
|
Phase 1 owns reviewer routing (CODEOWNERS, `.github/review-routing.yml`, reviewer labels). Phase 2 owns the `submission-gate` check, `merge-risk:*` tiers, and state labels. This phase consumes those signals and falls back gracefully when they do not exist yet.
|
||||||
|
|
||||||
|
## Canvas review evidence
|
||||||
|
|
||||||
|
### What runs
|
||||||
|
|
||||||
|
`.github/workflows/canvas-smoke-test.yml` runs on PRs to `main` that change `extensions/**`, `plugins/**`, or the checker and its workflow. It first runs the checker's target detection without installing dependencies or the Copilot CLI; ordinary plugin-only changes that do not add, remove, or reference canvas extensions skip the expensive steps. Canvas extensions are still submitted only under `extensions/`, so `validate-canvas-extensions.yml` keeps its existing scope and is unchanged. The `canvas-smoke-test` job reports:
|
||||||
|
|
||||||
|
- **Skipped (success)** when the PR changes no canvas extension or extension-bearing plugin paths, including ordinary plugin-only changes and checker/workflow-only changes.
|
||||||
|
- **Pass (removal accepted)** when an extension directory and its plugin were both deleted. Deleting only `extension.mjs`, or leaving a plugin that still references or was the direct plugin for a deleted extension, fails.
|
||||||
|
- **Pass or fail** otherwise, from `eng/canvas-smoke-test.mjs`.
|
||||||
|
|
||||||
|
For each affected extension, the checker:
|
||||||
|
|
||||||
|
| Area | Checks |
|
||||||
|
|---|---|
|
||||||
|
| Module graph | Parses `extension.mjs` and every reachable local module **without executing it** (`vm.SourceTextModule` / `vm.compileFunction`). Reachability follows static imports, literal dynamic `import()` calls, and literal CommonJS `require()` calls (call-shaped text inside strings, templates, and regexes is ignored). `package.json` `imports` aliases (`#name`) resolve through the same checks; alias shapes the checker cannot analyze fail. Every reachable reference must resolve to a file inside the extension, a Node.js built-in, the host-provided `@github/copilot-sdk`, or a runtime dependency (`dependencies`, `optionalDependencies`, `peerDependencies`). Remote (`http:`/`https:`), `data:`, absolute, `..`, missing, `devDependencies`-only, and undeclared references fail. Modules not reachable from `extension.mjs` (often browser assets for the canvas webview) produce warnings instead. |
|
||||||
|
| Files | Missing referenced files, unsafe paths (`..`, absolute paths, `file:` URLs), symlinks, committed `node_modules`, native binaries (ELF, PE, Mach-O, `.node`, `.dll`, `.so`, WebAssembly), executable git modes, and files over 5 MB fail. Shell, batch, PowerShell, Python, Ruby, and Perl scripts, shebang files, and unrecognized binaries are reported as warnings for the reviewer. |
|
||||||
|
| Preview | `assets/preview.png` must exist, decode as a real PNG (signature, chunk CRCs, `IHDR`, palette, `IEND`, inflated image data and filter bytes), be at most 5 MB, decode to at most 256 MiB of image data (checked before decompression), and meet the minimum dimensions. |
|
||||||
|
| Plugin | Materializes the plugin with `eng/materialize-plugins.mjs` in a temporary copy, validates the served `plugin.json` against the Agent Plugins schema, and confirms `com.github.copilot/extensions/<id>/extension.mjs` exists. |
|
||||||
|
| Install | Installs the materialized plugin with the GitHub Copilot CLI from an ephemeral local marketplace, in an isolated `COPILOT_HOME` with all tokens removed, and verifies it with `copilot plugin list --json`. CI uses `--install require`. |
|
||||||
|
| Capabilities | Summarizes what an extension uses (for example filesystem, child processes, network, `process.env`, dynamic code) so reviewers can compare it with the stated purpose. |
|
||||||
|
|
||||||
|
Preview minimum dimensions default to **400 × 160 px**. Every existing preview meets this; the smallest are 475 × 440, 720 × 165, 544 × 306, and 657 × 270. Change the minimum with the repository variables `CANVAS_PREVIEW_MIN_WIDTH` and `CANVAS_PREVIEW_MIN_HEIGHT`, or with `--min-preview-width` and `--min-preview-height` locally.
|
||||||
|
|
||||||
|
### Review artifact
|
||||||
|
|
||||||
|
The job writes a concise report to the job summary and uploads the `canvas-smoke-test-results` artifact (7-day retention) with `report.md`, `results.json`, and the preview images. The report includes plugin and extension metadata, changed files, capabilities, the preview, and every validation and smoke-test result.
|
||||||
|
|
||||||
|
`.github/workflows/canvas-smoke-test-comment.yml` follows the repository's reader/writer split. The reader (`canvas-smoke-test.yml`) runs untrusted PR content with a read-only token and no secrets. The writer runs from `main` on `workflow_run` and binds the artifact to the triggering run before writing: the PR must target this repository, its head repository, branch, and SHA must match the run, and it must appear in `workflow_run.pull_requests` or be the only open PR with that head. It also neutralizes `@` mentions, and upserts one PR comment marked `<!-- canvas-smoke-test -->`. It does not comment on skipped PRs that never had a report.
|
||||||
|
|
||||||
|
### Running locally
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node eng/canvas-smoke-test.mjs --all --install never # every extension, no CLI install
|
||||||
|
node eng/canvas-smoke-test.mjs --changed-files changed.txt # only what a diff touches
|
||||||
|
node --test eng/canvas-smoke-test.test.mjs
|
||||||
|
```
|
||||||
|
|
||||||
|
Exit codes: `0` pass or skipped, `1` contribution problems, `2` infrastructure error.
|
||||||
|
|
||||||
|
## Weekly operating metrics
|
||||||
|
|
||||||
|
`.github/workflows/review-metrics.yml` runs every Monday at 14:00 UTC and on demand. `eng/review-metrics.mjs` collects read-only data and then:
|
||||||
|
|
||||||
|
- writes the report to the job summary and uploads the `review-metrics` artifact (`metrics.json` and `report.md`, 90-day retention);
|
||||||
|
- updates the body of the tracking issue labeled `review-metrics`, creating and pinning it the first time;
|
||||||
|
- adds the week's report to that issue as a comment so trends stay visible.
|
||||||
|
|
||||||
|
### Definitions
|
||||||
|
|
||||||
|
All times are UTC. The window is the previous `window_days` (default 7) days. PRs authored by bots are excluded from contribution counts and review timings. A **maintainer review** is a submitted review from a user with write access who is not the PR author and not a bot.
|
||||||
|
|
||||||
|
| Metric | Definition |
|
||||||
|
|---|---|
|
||||||
|
| Open contributions by state | Open, non-draft contribution PRs to `main`, grouped by phase 2 state label; PRs without one are `unlabeled`. Open `external-plugin` issues are grouped by their state labels. |
|
||||||
|
| Open contributions by risk tier | The same PRs grouped by `merge-risk:*`; PRs without one are `unclassified`. |
|
||||||
|
| Time to first review | For PRs whose first maintainer review landed in the window: first review time minus the later of creation and the last ready-for-review event. Reviews submitted before that start (for example before a PR went back to draft) are ignored. Median and p90 (linear interpolation), in wall-clock hours or days. |
|
||||||
|
| Time to merge | For PRs merged in the window: merge time minus creation time. Median and p90. |
|
||||||
|
| Reviews per maintainer | Maintainer reviews submitted in the window, and the distinct PRs each maintainer reviewed. |
|
||||||
|
| Reviewer concentration | Based on distinct PRs reviewed per maintainer. **Top-reviewer share** is the busiest maintainer's share. **HHI** (Herfindahl-Hirschman index) is the sum of squared shares × 10,000: 10,000 means one reviewer did everything, and above 2,500 is highly concentrated. **Effective reviewers** is 1 divided by the unscaled HHI. |
|
||||||
|
| Items past 2 and 4 business days | Items waiting on a maintainer longer than each target: open, non-draft PRs without a maintainer review since the clock started and without `requires-submitter-fixes` (the clock starts at creation or the last ready-for-review event), and `external-plugin` issues labeled `ready-for-review` or `awaiting-approval` (the clock starts when that label was last added). Business days count Monday through Friday only, with partial days counted fractionally. |
|
||||||
|
| Automation failure rate | For the configured `automation_workflows`: runs created in the window with conclusion `failure`, `timed_out`, or `startup_failure`, divided by completed runs excluding `cancelled`, `skipped`, `neutral`, `action_required`, and `stale`. Workflows that do not exist yet are listed as "not found". |
|
||||||
|
|
||||||
|
`.github/review-metrics.yml` configures the window, targets, label sets, workflows, and tracking issue. Missing labels never cause errors.
|
||||||
|
|
||||||
|
### Running locally
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node eng/review-metrics.mjs --repo github/awesome-copilot --dry-run --output-dir ./metrics
|
||||||
|
node --test eng/review-metrics.test.mjs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deferred: auto-merge
|
||||||
|
|
||||||
|
Safe auto-merge was proposed in this phase and removed before merge. Arming GitHub auto-merge from automation would codify workarounds to repository and organization policy (required reviewers and Copilot code review approvals), and its triggers widen the attack surface; this repository does not use `pull_request_target` ([#2625](https://github.com/github/awesome-copilot/pull/2625)). It is deferred until maintainers have reviewed the approach with GitHub security. Until then, maintainers merge PRs once `submission-gate` passes.
|
||||||
|
|
||||||
|
## Portability to MOS3
|
||||||
|
|
||||||
|
The #4184 automation is driven by repository-local configuration and self-contained scripts, so another repository such as MOS3 can adopt it by copying files and editing configuration rather than code.
|
||||||
|
|
||||||
|
| Artifact | Phase | How to port | Repository-specific settings |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `CODEOWNERS` with team owners | 1 | Copy the structure | Team names and path patterns |
|
||||||
|
| `.github/review-routing.yml` | 1 | Copy and edit | Reviewer pools, targets, escalation |
|
||||||
|
| `.github/risk-tiers.yml` | 2 | Copy and edit | Path and risk classification rules |
|
||||||
|
| `submission-gate` check | 2 | Copy the workflow and script | Required checks per tier; include `canvas-smoke-test` only where canvas extensions exist |
|
||||||
|
| `canvas-smoke-test` (`eng/canvas-smoke-test.mjs`, `canvas-smoke-test.yml`, `canvas-smoke-test-comment.yml`) | 3 | Copy as-is where the repository ships canvas extensions and `eng/materialize-plugins.mjs` | `CANVAS_PREVIEW_MIN_WIDTH` and `CANVAS_PREVIEW_MIN_HEIGHT` variables |
|
||||||
|
| `.github/review-metrics.yml`, `eng/review-metrics.mjs`, `.github/workflows/review-metrics.yml` | 3 | Copy as-is | State and risk labels, automation workflow list, tracking issue title |
|
||||||
|
| `eng/lib/review-automation-github.mjs` | 3 | Copy as-is | None (uses `GITHUB_TOKEN` or the `gh` CLI) |
|
||||||
|
|
||||||
|
The phase 3 scripts depend only on Node.js 22 and `js-yaml`. Workflows gate automatic runs on `github.repository_owner == 'github'`; update that condition when porting. `.github/workflows/setup-labels.yml` creates the `review-metrics` label.
|
||||||
|
|
||||||
|
### Other marketplaces: `microsoft/azure-dev-tools`
|
||||||
|
|
||||||
|
[`microsoft/azure-dev-tools`](https://github.com/microsoft/azure-dev-tools) is another Copilot plugin marketplace (`copilot plugin marketplace add microsoft/azure-dev-tools`, marketplace ID `azure-dev-tools`). It ships canvas plugins such as `azure-functions-hosted-skills`, `azure-resources-query`, `azure-cost-health-check`, and `azure-sre-agent`, plus the skill-only `canvas-authoring` plugin. It can reuse the same review model: CODEOWNERS teams, `review-routing.yml`, risk tiers, `submission-gate`, `canvas-smoke-test`, and the metrics workflow.
|
||||||
|
|
||||||
|
- **Maps directly:**
|
||||||
|
- Canvas smoke-test and preview evidence: module graph, capability summary, unsafe paths, binaries, and `assets/preview.png` checks.
|
||||||
|
- Risk tiers and `submission-gate` rules.
|
||||||
|
- Weekly metrics.
|
||||||
|
- **Differences:**
|
||||||
|
- The plugins are first-party and Microsoft-owned, so resource ownership comes from CODEOWNERS teams more than contributor front matter. Contributor-history signals matter less there.
|
||||||
|
- Releases are pinned to immutable per-plugin tags, while awesome-copilot materializes plugins from the default branch. Point the smoke test's install step at the plugin source for the release tag. Treat release-tag or version bumps as their own risk tier that always needs maintainer review.
|
||||||
|
- Plugins with no canvas extension, such as `canvas-authoring`, report `canvas-smoke-test` as skipped. The check's layout detection (`extensions/<id>/` plus a matching `plugins/<id>/plugin.json`) may need adjusting to that repository's structure.
|
||||||
@@ -32,6 +32,14 @@ node eng/review-routing.mjs validate # validate .github/review-routing.yml
|
|||||||
node --test eng/review-routing.test.mjs # unit tests
|
node --test eng/review-routing.test.mjs # unit tests
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Review automation
|
||||||
|
|
||||||
|
See [docs/maintainers/canvas-evidence-and-metrics.md](../docs/maintainers/canvas-evidence-and-metrics.md) for details.
|
||||||
|
|
||||||
|
- `canvas-smoke-test.mjs` — static checks, preview validation, materialization, and install smoke test for canvas extensions (`canvas-smoke-test` check).
|
||||||
|
- `review-metrics.mjs` — computes weekly review operating metrics and publishes them to the tracking issue.
|
||||||
|
- `lib/review-automation-github.mjs` — small GitHub API client used by `review-metrics.mjs` (uses `GITHUB_TOKEN`, or the `gh` CLI locally).
|
||||||
|
|
||||||
## Contributor Tools
|
## Contributor Tools
|
||||||
|
|
||||||
- `contributor-report.mjs` — generates a markdown report of merged PRs for missing contributors (includes shared helpers).
|
- `contributor-report.mjs` — generates a markdown report of merged PRs for missing contributors (includes shared helpers).
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,673 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import fs from "node:fs";
|
||||||
|
import os from "node:os";
|
||||||
|
import path from "node:path";
|
||||||
|
import { spawnSync } from "node:child_process";
|
||||||
|
import zlib from "node:zlib";
|
||||||
|
import { test } from "node:test";
|
||||||
|
import {
|
||||||
|
checkExtensionModules,
|
||||||
|
checkPreview,
|
||||||
|
classifySpecifier,
|
||||||
|
detectCanvasTargets,
|
||||||
|
findDynamicImportSpecifiers,
|
||||||
|
findNonLiteralRuntimeLoads,
|
||||||
|
findRequireSpecifiers,
|
||||||
|
findUnsafeManifestPaths,
|
||||||
|
importsAliasTargets,
|
||||||
|
inspectExtensionFiles,
|
||||||
|
inspectPng,
|
||||||
|
parseEsModule,
|
||||||
|
isSafeExtensionId,
|
||||||
|
readRegularFile,
|
||||||
|
renderMarkdownReport,
|
||||||
|
runCanvasSmokeTest,
|
||||||
|
stripComments,
|
||||||
|
unsafeExtensionRefs,
|
||||||
|
unsafePathReason,
|
||||||
|
} from "./canvas-smoke-test.mjs";
|
||||||
|
|
||||||
|
function crc32(buffer) {
|
||||||
|
let crc = 0xffffffff;
|
||||||
|
for (const byte of buffer) {
|
||||||
|
crc ^= byte;
|
||||||
|
for (let k = 0; k < 8; k++) crc = crc & 1 ? 0xedb88320 ^ (crc >>> 1) : crc >>> 1;
|
||||||
|
}
|
||||||
|
return (crc ^ 0xffffffff) >>> 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function chunk(type, data) {
|
||||||
|
const length = Buffer.alloc(4);
|
||||||
|
length.writeUInt32BE(data.length);
|
||||||
|
const body = Buffer.concat([Buffer.from(type, "latin1"), data]);
|
||||||
|
const crc = Buffer.alloc(4);
|
||||||
|
crc.writeUInt32BE(crc32(body));
|
||||||
|
return Buffer.concat([length, body, crc]);
|
||||||
|
}
|
||||||
|
|
||||||
|
function makePng(width, height, { truncate = false } = {}) {
|
||||||
|
const header = Buffer.alloc(13);
|
||||||
|
header.writeUInt32BE(width, 0);
|
||||||
|
header.writeUInt32BE(height, 4);
|
||||||
|
header[8] = 8; // bit depth
|
||||||
|
header[9] = 2; // truecolor
|
||||||
|
const rows = Buffer.alloc(height * (width * 3 + 1));
|
||||||
|
const raw = truncate ? rows.subarray(0, rows.length / 2) : rows;
|
||||||
|
return Buffer.concat([
|
||||||
|
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
|
||||||
|
chunk("IHDR", header),
|
||||||
|
chunk("IDAT", zlib.deflateSync(raw)),
|
||||||
|
chunk("IEND", Buffer.alloc(0)),
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeRepo(files) {
|
||||||
|
const root = fs.mkdtempSync(path.join(os.tmpdir(), "canvas-smoke-test-"));
|
||||||
|
for (const [relative, content] of Object.entries(files)) {
|
||||||
|
const target = path.join(root, relative);
|
||||||
|
fs.mkdirSync(path.dirname(target), { recursive: true });
|
||||||
|
fs.writeFileSync(target, typeof content === "string" || Buffer.isBuffer(content) ? content : JSON.stringify(content, null, 2));
|
||||||
|
}
|
||||||
|
return root;
|
||||||
|
}
|
||||||
|
|
||||||
|
function git(root, args) {
|
||||||
|
const result = spawnSync("git", args, { cwd: root, encoding: "utf8" });
|
||||||
|
assert.equal(result.status, 0, result.stderr || result.stdout);
|
||||||
|
return result.stdout.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
const PLUGIN_SCHEMA = "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json";
|
||||||
|
|
||||||
|
function extensionPlugin(name) {
|
||||||
|
return {
|
||||||
|
$schema: PLUGIN_SCHEMA,
|
||||||
|
name,
|
||||||
|
description: `${name} canvas`,
|
||||||
|
version: "1.0.0",
|
||||||
|
extensions: { "com.github.copilot": { logo: "assets/preview.png" } },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test("inspectPng accepts a valid PNG and reports dimensions", () => {
|
||||||
|
const result = inspectPng(makePng(32, 16));
|
||||||
|
assert.equal(result.ok, true);
|
||||||
|
assert.equal(result.width, 32);
|
||||||
|
assert.equal(result.height, 16);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("inspectPng rejects non-PNG, corrupted, and truncated data", () => {
|
||||||
|
assert.match(inspectPng(Buffer.from("GIF89a-not-a-png-at-all")).errors[0], /signature/);
|
||||||
|
|
||||||
|
const corrupted = makePng(8, 8);
|
||||||
|
corrupted[corrupted.length - 20] ^= 0xff;
|
||||||
|
assert.equal(inspectPng(corrupted).ok, false);
|
||||||
|
|
||||||
|
const truncated = inspectPng(makePng(8, 8, { truncate: true }));
|
||||||
|
assert.equal(truncated.ok, false);
|
||||||
|
assert.match(truncated.errors[0], /truncated/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("inspectPng rejects malformed IEND chunks and trailing data", () => {
|
||||||
|
const valid = makePng(8, 8);
|
||||||
|
assert.match(inspectPng(Buffer.concat([valid, Buffer.from("trailing")])).errors[0], /after the IEND/);
|
||||||
|
|
||||||
|
const iendOffset = valid.length - 12;
|
||||||
|
const nonEmptyIend = Buffer.concat([
|
||||||
|
valid.subarray(0, iendOffset),
|
||||||
|
chunk("IEND", Buffer.from([0])),
|
||||||
|
]);
|
||||||
|
assert.match(inspectPng(nonEmptyIend).errors[0], /IEND chunk has invalid length/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("checkPreview enforces configurable minimum dimensions", () => {
|
||||||
|
const root = makeRepo({ "ext/assets/preview.png": makePng(100, 50) });
|
||||||
|
const small = checkPreview(path.join(root, "ext"), { minWidth: 400, minHeight: 160 });
|
||||||
|
assert.match(small.errors[0], /minimum is 400×160/);
|
||||||
|
const ok = checkPreview(path.join(root, "ext"), { minWidth: 100, minHeight: 50 });
|
||||||
|
assert.deepEqual(ok.errors, []);
|
||||||
|
const missing = checkPreview(path.join(root, "missing"), { minWidth: 1, minHeight: 1 });
|
||||||
|
assert.match(missing.errors[0], /missing/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unsafePathReason flags absolute and traversal paths", () => {
|
||||||
|
assert.equal(unsafePathReason("./assets/preview.png"), null);
|
||||||
|
assert.equal(unsafePathReason("assets/preview.png"), null);
|
||||||
|
assert.match(unsafePathReason("../secrets.txt"), /traversal/);
|
||||||
|
assert.match(unsafePathReason("./a/../../b"), /traversal/);
|
||||||
|
assert.match(unsafePathReason("/etc/passwd"), /absolute/);
|
||||||
|
assert.match(unsafePathReason("C:\\Windows\\system32"), /Windows/);
|
||||||
|
assert.match(unsafePathReason("file:///tmp/x.mjs"), /file:/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("findUnsafeManifestPaths ignores prose and URLs", () => {
|
||||||
|
const findings = findUnsafeManifestPaths({
|
||||||
|
description: "Reads ../ style docs and / separators in text",
|
||||||
|
homepage: "https://example.com/../x",
|
||||||
|
extensions: { "com.github.copilot": { logo: "../outside.png" } },
|
||||||
|
main: "/abs/entry.mjs",
|
||||||
|
});
|
||||||
|
assert.deepEqual(findings.map((finding) => finding.field).sort(), ["$.extensions.com.github.copilot.logo", "$.main"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("classifySpecifier distinguishes builtins, host packages, dependencies, and unsafe specifiers", () => {
|
||||||
|
const pkg = { dependencies: { playwright: "1.0.0" }, devDependencies: { vitest: "1.0.0" } };
|
||||||
|
assert.equal(classifySpecifier("node:fs", pkg).kind, "builtin");
|
||||||
|
assert.equal(classifySpecifier("path", pkg).kind, "builtin");
|
||||||
|
assert.equal(classifySpecifier("@github/copilot-sdk/extension", pkg).kind, "host");
|
||||||
|
assert.equal(classifySpecifier("playwright/test", pkg).kind, "dependency");
|
||||||
|
assert.equal(classifySpecifier("vitest", pkg).kind, "dev-dependency");
|
||||||
|
assert.equal(classifySpecifier("left-pad", pkg).kind, "undeclared");
|
||||||
|
assert.equal(classifySpecifier("/tmp/evil.mjs", pkg).kind, "unsafe");
|
||||||
|
assert.equal(classifySpecifier("https://example.com/x.mjs", pkg).kind, "remote");
|
||||||
|
assert.equal(classifySpecifier("./local.mjs", pkg).kind, "relative");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("parseEsModule returns static specifiers without executing code", () => {
|
||||||
|
const marker = path.join(os.tmpdir(), `canvas-smoke-exec-${process.pid}-${Date.now()}`);
|
||||||
|
const source = `import fs from "node:fs";\nimport { x } from "./x.mjs";\nfs.writeFileSync(${JSON.stringify(marker)}, "ran");\n`;
|
||||||
|
const result = parseEsModule(source, "entry.mjs");
|
||||||
|
assert.equal(result.ok, true);
|
||||||
|
assert.deepEqual(result.specifiers.sort(), ["./x.mjs", "node:fs"]);
|
||||||
|
assert.equal(fs.existsSync(marker), false);
|
||||||
|
|
||||||
|
const broken = parseEsModule("export const = 1;", "broken.mjs");
|
||||||
|
assert.equal(broken.ok, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("checkExtensionModules reports syntax errors, missing files, traversal, and undeclared packages", () => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/bad/extension.mjs": [
|
||||||
|
'import { joinSession } from "@github/copilot-sdk/extension";',
|
||||||
|
'import { spawn } from "node:child_process";',
|
||||||
|
'import helper from "./lib/helper.mjs";',
|
||||||
|
'import missing from "./lib/missing.mjs";',
|
||||||
|
'import outside from "../other/secret.mjs";',
|
||||||
|
'import pad from "left-pad";',
|
||||||
|
"export default { joinSession, spawn, helper, missing, outside, pad };",
|
||||||
|
].join("\n"),
|
||||||
|
"extensions/bad/lib/helper.mjs": "export default fetch;\nexport const broken = ;\n",
|
||||||
|
"extensions/bad/public/app.js": 'import { h } from "/vendor/preact.js";\nexport default h;\n',
|
||||||
|
"extensions/bad/package.json": { name: "bad", version: "1.0.0" },
|
||||||
|
"extensions/other/secret.mjs": "export default 1;\n",
|
||||||
|
});
|
||||||
|
const result = checkExtensionModules(path.join(root, "extensions", "bad"));
|
||||||
|
const text = result.errors.join("\n");
|
||||||
|
assert.match(text, /lib\/helper\.mjs: syntax error/);
|
||||||
|
assert.match(text, /\.\/lib\/missing\.mjs" references a missing file/);
|
||||||
|
assert.match(text, /escapes the extension directory/);
|
||||||
|
assert.match(text, /"left-pad" is not a Node\.js builtin/);
|
||||||
|
assert.doesNotMatch(text, /public\/app\.js/);
|
||||||
|
assert.match(result.warnings.join("\n"), /public\/app\.js: import "\/vendor\/preact\.js" uses an unsafe absolute path \(module is not reachable/);
|
||||||
|
assert.deepEqual(result.builtins, ["child_process"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("inspectExtensionFiles flags native binaries, executables, and vendored node_modules", () => {
|
||||||
|
const elf = Buffer.concat([Buffer.from([0x7f, 0x45, 0x4c, 0x46]), Buffer.alloc(32)]);
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/bin/extension.mjs": "export {};\n",
|
||||||
|
"extensions/bin/tool": elf,
|
||||||
|
"extensions/bin/addon.node": "not really native",
|
||||||
|
"extensions/bin/run.sh": "#!/bin/sh\necho hi\n",
|
||||||
|
"extensions/bin/assets/preview.png": makePng(4, 4),
|
||||||
|
"extensions/bin/node_modules/dep/index.js": "module.exports = 1;\n",
|
||||||
|
"extensions/bin/data.bin.dat": Buffer.from([0, 1, 2, 3]),
|
||||||
|
"extensions/bin/tool.py": "print(1)\n",
|
||||||
|
});
|
||||||
|
const modes = new Map([["extensions/bin/run.sh", "100755"]]);
|
||||||
|
const result = inspectExtensionFiles(path.join(root, "extensions", "bin"), { rootDir: root, fileModes: modes });
|
||||||
|
const errors = result.errors.join("\n");
|
||||||
|
assert.match(errors, /tool: contains a ELF executable/);
|
||||||
|
assert.match(errors, /addon\.node: native\/compiled binary/);
|
||||||
|
assert.match(errors, /run\.sh: file is marked executable/);
|
||||||
|
assert.match(errors, /node_modules\/: vendored node_modules/);
|
||||||
|
assert.match(result.warnings.join("\n"), /run\.sh: script file/);
|
||||||
|
assert.match(result.warnings.join("\n"), /data\.bin\.dat: unexpected binary content/);
|
||||||
|
assert.match(result.warnings.join("\n"), /tool\.py: script file/);
|
||||||
|
assert.doesNotMatch(result.warnings.join("\n"), /preview\.png/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("detectCanvasTargets maps changed paths to extensions and extension-bearing plugins", () => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/orb/extension.mjs": "export {};\n",
|
||||||
|
"extensions/shared/extension.mjs": "export {};\n",
|
||||||
|
"plugins/orb/plugin.json": extensionPlugin("orb"),
|
||||||
|
"plugins/bundle/plugin.json": {
|
||||||
|
$schema: PLUGIN_SCHEMA,
|
||||||
|
name: "bundle",
|
||||||
|
description: "bundle",
|
||||||
|
version: "1.0.0",
|
||||||
|
extensions: { "com.github.awesome-copilot": { extensions: ["./extensions/shared"] } },
|
||||||
|
},
|
||||||
|
"plugins/plain/plugin.json": { $schema: PLUGIN_SCHEMA, name: "plain", description: "plain", version: "1.0.0" },
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.deepEqual(detectCanvasTargets(["README.md", "plugins/plain/README.md", "skills/x/SKILL.md"], { rootDir: root }).extensions, []);
|
||||||
|
|
||||||
|
const orb = detectCanvasTargets(["extensions/orb/extension.mjs"], { rootDir: root });
|
||||||
|
assert.deepEqual(orb.extensions, ["orb"]);
|
||||||
|
assert.deepEqual(orb.plugins, ["orb"]);
|
||||||
|
|
||||||
|
const bundle = detectCanvasTargets(["plugins/bundle/plugin.json", "extensions/gone/extension.mjs"], { rootDir: root });
|
||||||
|
assert.deepEqual(bundle.extensions, ["shared"]);
|
||||||
|
assert.deepEqual(bundle.plugins, ["bundle"]);
|
||||||
|
assert.deepEqual(bundle.removedExtensions, ["gone"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("runCanvasSmokeTest reports skipped when no canvas paths change", async () => {
|
||||||
|
const root = makeRepo({ "plugins/plain/plugin.json": { name: "plain" } });
|
||||||
|
const report = await runCanvasSmokeTest({ rootDir: root, changedFiles: ["docs/README.md"], install: "never" });
|
||||||
|
assert.equal(report.status, "skipped");
|
||||||
|
assert.match(renderMarkdownReport(report), /Skipped/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("runCanvasSmokeTest materializes a valid extension plugin and renders evidence", async () => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/orb/extension.mjs": 'import { joinSession } from "@github/copilot-sdk/extension";\nimport http from "node:http";\nexport default { joinSession, http };\n',
|
||||||
|
"extensions/orb/package.json": { name: "orb", version: "1.0.0", type: "module" },
|
||||||
|
"extensions/orb/assets/preview.png": makePng(800, 400),
|
||||||
|
"plugins/orb/plugin.json": extensionPlugin("orb"),
|
||||||
|
"plugins/orb/README.md": "# Orb\n",
|
||||||
|
});
|
||||||
|
const report = await runCanvasSmokeTest({ rootDir: root, changedFiles: ["extensions/orb/extension.mjs"], install: "never" });
|
||||||
|
assert.equal(report.status, "pass", JSON.stringify(report, null, 2));
|
||||||
|
assert.equal(report.smoke.materialize.orb.status, "pass");
|
||||||
|
assert.equal(report.smoke.installStatus, "skipped");
|
||||||
|
assert.equal(report.extensions[0].preview.width, 800);
|
||||||
|
|
||||||
|
const markdown = renderMarkdownReport(report, { previewBaseUrl: "https://raw.githubusercontent.com/o/r/sha/" });
|
||||||
|
assert.match(markdown, /<!-- canvas-smoke-test -->/);
|
||||||
|
assert.match(markdown, /Network sockets/);
|
||||||
|
assert.match(markdown, /extensions\/orb\/assets\/preview\.png/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("runCanvasSmokeTest fails for an unregistered extension with a missing preview", async () => {
|
||||||
|
const root = makeRepo({ "extensions/lonely/extension.mjs": "export {};\n" });
|
||||||
|
const report = await runCanvasSmokeTest({ rootDir: root, changedFiles: ["extensions/lonely/extension.mjs"], install: "never" });
|
||||||
|
assert.equal(report.status, "fail");
|
||||||
|
const errors = report.extensions[0].errors.join("\n");
|
||||||
|
assert.match(errors, /preview\.png is missing/);
|
||||||
|
assert.match(errors, /not registered by any plugin/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("inspectPng rejects images whose decoded data exceeds the budget before inflating", () => {
|
||||||
|
const header = Buffer.alloc(13);
|
||||||
|
header.writeUInt32BE(16000, 0);
|
||||||
|
header.writeUInt32BE(16000, 4);
|
||||||
|
header[8] = 16; // bit depth
|
||||||
|
header[9] = 6; // RGBA
|
||||||
|
const png = Buffer.concat([
|
||||||
|
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
|
||||||
|
chunk("IHDR", header),
|
||||||
|
chunk("IDAT", zlib.deflateSync(Buffer.alloc(1024))),
|
||||||
|
chunk("IEND", Buffer.alloc(0)),
|
||||||
|
]);
|
||||||
|
const result = inspectPng(png);
|
||||||
|
assert.equal(result.ok, false);
|
||||||
|
assert.match(result.errors[0], /decoded image data would be/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("inspectPng rejects image data larger than IHDR allows", () => {
|
||||||
|
const header = Buffer.alloc(13);
|
||||||
|
header.writeUInt32BE(4, 0);
|
||||||
|
header.writeUInt32BE(4, 4);
|
||||||
|
header[8] = 8;
|
||||||
|
header[9] = 2;
|
||||||
|
const png = Buffer.concat([
|
||||||
|
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
|
||||||
|
chunk("IHDR", header),
|
||||||
|
chunk("IDAT", zlib.deflateSync(Buffer.alloc(10_000))),
|
||||||
|
chunk("IEND", Buffer.alloc(0)),
|
||||||
|
]);
|
||||||
|
assert.match(inspectPng(png).errors[0], /larger than the IHDR/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("findUnsafeManifestPaths reports file: URLs", () => {
|
||||||
|
const findings = findUnsafeManifestPaths({ main: "file:///tmp/x.mjs", homepage: "https://example.com/x" });
|
||||||
|
assert.deepEqual(findings.map((finding) => finding.field), ["$.main"]);
|
||||||
|
assert.match(findings[0].reason, /file:/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("checkExtensionModules follows CommonJS requires and literal dynamic imports", () => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/graph/extension.mjs": [
|
||||||
|
'import cjs from "./lib/legacy.cjs";',
|
||||||
|
'const worker = await import("./lib/worker.mjs");',
|
||||||
|
'const later = await import("./lib/missing.mjs");',
|
||||||
|
"// import(\"./lib/commented.mjs\")",
|
||||||
|
"export default { cjs, worker, later };",
|
||||||
|
].join("\n"),
|
||||||
|
"extensions/graph/lib/legacy.cjs": [
|
||||||
|
'const helper = require("./helper");',
|
||||||
|
'const pad = require("left-pad");',
|
||||||
|
'const outside = require("../../other/x.js");',
|
||||||
|
"// require(\"ignored-in-comment\")",
|
||||||
|
"module.exports = { helper, pad, outside };",
|
||||||
|
].join("\n"),
|
||||||
|
"extensions/graph/lib/helper.js": "module.exports = 1;\n",
|
||||||
|
"extensions/graph/lib/worker.mjs": 'import { spawn } from "node:child_process";\nimport vitest from "vitest";\nexport default { spawn, vitest };\n',
|
||||||
|
"extensions/graph/package.json": { name: "graph", version: "1.0.0", devDependencies: { vitest: "1.0.0" } },
|
||||||
|
"extensions/other/x.js": "module.exports = 1;\n",
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = checkExtensionModules(path.join(root, "extensions", "graph"));
|
||||||
|
const errors = result.errors.join("\n");
|
||||||
|
assert.match(errors, /legacy\.cjs: require\("left-pad"\) is not a Node\.js builtin/);
|
||||||
|
assert.match(errors, /legacy\.cjs: require\("\.\.\/\.\.\/other\/x\.js"\) escapes the extension directory/);
|
||||||
|
assert.doesNotMatch(errors, /require\("\.\/helper"\)/);
|
||||||
|
assert.doesNotMatch(errors, /ignored-in-comment|commented\.mjs/);
|
||||||
|
assert.match(errors, /dynamic import\("\.\/lib\/missing\.mjs"\) references a missing file/);
|
||||||
|
assert.match(errors, /worker\.mjs: import "vitest" is only declared in devDependencies/);
|
||||||
|
assert.deepEqual(result.builtins, ["child_process"]);
|
||||||
|
assert.ok(result.modules.find((entry) => entry.path === "lib/worker.mjs").reachable);
|
||||||
|
assert.ok(result.modules.find((entry) => entry.path === "lib/helper.js").reachable);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("checkExtensionModules fails reachable data URL imports and warns for unreachable dynamic data imports", () => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/data/extension.mjs": [
|
||||||
|
'import inline from "data:text/javascript,export default 1";',
|
||||||
|
'const later = await import("data:text/javascript,export default 2");',
|
||||||
|
"export default { inline, later };",
|
||||||
|
].join("\n"),
|
||||||
|
"extensions/data/public/app.js": 'await import("data:text/javascript,export default 3");\n',
|
||||||
|
});
|
||||||
|
const result = checkExtensionModules(path.join(root, "extensions", "data"));
|
||||||
|
const errors = result.errors.join("\n");
|
||||||
|
assert.match(errors, /extension\.mjs: import "data:text\/javascript,export default 1" is a data: URL import/);
|
||||||
|
assert.match(errors, /extension\.mjs: dynamic import\("data:text\/javascript,export default 2"\) is a data: URL import/);
|
||||||
|
assert.match(result.warnings.join("\n"), /public\/app\.js: dynamic import\("data:text\/javascript,export default 3"\) is a data: URL import \(module is not reachable/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("removed canvas paths are validated instead of skipped", async () => {
|
||||||
|
// Entry point deleted but the extension directory remains.
|
||||||
|
const partial = makeRepo({
|
||||||
|
"extensions/orb/assets/preview.png": makePng(800, 400),
|
||||||
|
"plugins/orb/plugin.json": extensionPlugin("orb"),
|
||||||
|
});
|
||||||
|
const partialReport = await runCanvasSmokeTest({ rootDir: partial, changedFiles: ["extensions/orb/extension.mjs"], install: "never" });
|
||||||
|
assert.equal(partialReport.status, "fail");
|
||||||
|
assert.match(partialReport.extensions[0].errors.join("\n"), /extension\.mjs: entry point is missing/);
|
||||||
|
|
||||||
|
// Extension deleted, but its direct plugin and a bundling plugin remain.
|
||||||
|
const orphaned = makeRepo({
|
||||||
|
"plugins/orb/plugin.json": extensionPlugin("orb"),
|
||||||
|
"plugins/bundle/plugin.json": {
|
||||||
|
$schema: PLUGIN_SCHEMA,
|
||||||
|
name: "bundle",
|
||||||
|
description: "bundle",
|
||||||
|
version: "1.0.0",
|
||||||
|
extensions: { "com.github.awesome-copilot": { extensions: ["./extensions/orb"] } },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const orphanedTargets = detectCanvasTargets(["extensions/orb/extension.mjs"], { rootDir: orphaned });
|
||||||
|
assert.deepEqual(orphanedTargets.plugins, ["bundle", "orb"]);
|
||||||
|
const orphanedReport = await runCanvasSmokeTest({ rootDir: orphaned, changedFiles: ["extensions/orb/extension.mjs"], install: "never" });
|
||||||
|
assert.equal(orphanedReport.status, "fail");
|
||||||
|
const pluginErrors = orphanedReport.plugins.flatMap((plugin) => plugin.errors).join("\n");
|
||||||
|
assert.match(pluginErrors, /plugins\/orb is the plugin for removed extension/);
|
||||||
|
assert.match(pluginErrors, /plugins\/bundle\/plugin\.json references missing extension extensions\/orb/);
|
||||||
|
|
||||||
|
// Extension and plugin both deleted: accepted.
|
||||||
|
const clean = makeRepo({ "plugins/plain/plugin.json": { name: "plain" } });
|
||||||
|
const cleanReport = await runCanvasSmokeTest({
|
||||||
|
rootDir: clean,
|
||||||
|
changedFiles: ["extensions/orb/extension.mjs", "plugins/orb/plugin.json"],
|
||||||
|
install: "never",
|
||||||
|
});
|
||||||
|
assert.equal(cleanReport.status, "pass");
|
||||||
|
assert.match(renderMarkdownReport(cleanReport), /removed extensions/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("stripComments keeps string, template, and regex literals that contain comment markers", () => {
|
||||||
|
const source = [
|
||||||
|
'const s = "x//y"; await import("./a.mjs");',
|
||||||
|
"const t = `/* ${'//'} */`; await import(\"./b.mjs\");",
|
||||||
|
'const r = /\\/\\*/; await import("./c.mjs");',
|
||||||
|
'// await import("./hidden.mjs");',
|
||||||
|
'/* await import("./hidden2.mjs"); */',
|
||||||
|
'const q = a / b; const w = c / d; await import("./d.mjs");',
|
||||||
|
].join("\n");
|
||||||
|
const text = stripComments(source);
|
||||||
|
for (const name of ["a", "b", "c", "d"]) assert.match(text, new RegExp(`import\\("\\./${name}\\.mjs"\\)`));
|
||||||
|
assert.doesNotMatch(text, /hidden/);
|
||||||
|
assert.equal(text.split("\n").length, source.split("\n").length);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("checkExtensionModules follows imports placed after strings containing //", () => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/str/extension.mjs": 'const s = "x//y"; const w = await import("./worker.mjs");\nexport default { s, w };\n',
|
||||||
|
"extensions/str/worker.mjs": 'import pad from "left-pad";\nexport default pad;\n',
|
||||||
|
});
|
||||||
|
const result = checkExtensionModules(path.join(root, "extensions", "str"));
|
||||||
|
assert.match(result.errors.join("\n"), /worker\.mjs: import "left-pad" is not a Node\.js builtin/);
|
||||||
|
assert.ok(result.modules.find((entry) => entry.path === "worker.mjs").reachable);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("inspectPng rejects a duplicate IHDR chunk", () => {
|
||||||
|
const valid = makePng(4, 4);
|
||||||
|
const ihdr = valid.subarray(8, 8 + 25);
|
||||||
|
const png = Buffer.concat([valid.subarray(0, 33), ihdr, valid.subarray(33)]);
|
||||||
|
const result = inspectPng(png);
|
||||||
|
assert.equal(result.ok, false);
|
||||||
|
assert.match(result.errors.join("\n"), /duplicate IHDR chunk/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("dynamic import and require extraction ignores call-shaped text inside literals", () => {
|
||||||
|
const source = [
|
||||||
|
'const a = \'import("./in-single.mjs")\';',
|
||||||
|
'const b = "require(\\"./in-double.cjs\\")";',
|
||||||
|
"const c = `import(\"./in-template.mjs\") ${await import(\"./in-expr.mjs\")}`;",
|
||||||
|
'const d = /import\\("\\.\\/in-regex\\.mjs"\\)/;',
|
||||||
|
'const e = await import("./real.mjs"); const f = require("./real.cjs");',
|
||||||
|
].join("\n");
|
||||||
|
assert.deepEqual(findDynamicImportSpecifiers(source).sort(), ["./in-expr.mjs", "./real.mjs"]);
|
||||||
|
assert.deepEqual(findRequireSpecifiers(source), ["./real.cjs"]);
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/lit/extension.mjs": 'const note = \'import("./missing.mjs")\';\nexport default { note };\n',
|
||||||
|
});
|
||||||
|
assert.deepEqual(checkExtensionModules(path.join(root, "extensions", "lit")).errors, []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("non-literal runtime loads are detected outside comments and literals", () => {
|
||||||
|
const source = [
|
||||||
|
'const literalImport = import("./literal.mjs");',
|
||||||
|
'const literalRequire = require("left-pad");',
|
||||||
|
'const computedImport = import("node:" + moduleName);',
|
||||||
|
"const computedRequire = require(moduleName);",
|
||||||
|
'const text = "require(hidden)";',
|
||||||
|
"// import(variable)",
|
||||||
|
"const templateImport = import(`./${name}.mjs`);",
|
||||||
|
].join("\n");
|
||||||
|
assert.deepEqual(findNonLiteralRuntimeLoads(source), ["dynamic import", "dynamic import", "require"]);
|
||||||
|
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/runtime/extension.mjs": 'const name = "fs";\nexport default import("node:" + name);\n',
|
||||||
|
});
|
||||||
|
assert.match(
|
||||||
|
checkExtensionModules(path.join(root, "extensions", "runtime")).errors.join("\n"),
|
||||||
|
/extension\.mjs: non-literal dynamic import cannot be analyzed safely/,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("checkExtensionModules resolves package.json imports aliases and fails closed on unknown shapes", () => {
|
||||||
|
assert.deepEqual(importsAliasTargets({ node: "./a.mjs", default: { import: "./b.mjs" } }), ["./a.mjs", "./b.mjs"]);
|
||||||
|
assert.equal(importsAliasTargets(["./a.mjs"]), null);
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/alias/extension.mjs": [
|
||||||
|
'import ok from "#ok";',
|
||||||
|
'import gone from "#gone";',
|
||||||
|
'import pad from "#pad";',
|
||||||
|
'import odd from "#odd";',
|
||||||
|
'import up from "#up";',
|
||||||
|
"export default { ok, gone, pad, odd, up };",
|
||||||
|
].join("\n"),
|
||||||
|
"extensions/alias/lib/ok.mjs": 'import pad from "left-pad";\nexport default pad;\n',
|
||||||
|
"extensions/alias/package.json": {
|
||||||
|
name: "alias",
|
||||||
|
version: "1.0.0",
|
||||||
|
type: "module",
|
||||||
|
imports: { "#ok": "./lib/ok.mjs", "#gone": "./lib/gone.mjs", "#pad": "left-pad", "#odd": ["./lib/ok.mjs"], "#up": "../x.mjs" },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const result = checkExtensionModules(path.join(root, "extensions", "alias"));
|
||||||
|
const errors = result.errors.join("\n");
|
||||||
|
assert.ok(result.modules.find((entry) => entry.path === "lib/ok.mjs").reachable);
|
||||||
|
assert.match(errors, /lib\/ok\.mjs: import "left-pad" is not a Node\.js builtin/);
|
||||||
|
assert.match(errors, /import "#gone" -> "\.\/lib\/gone\.mjs" references a missing file/);
|
||||||
|
assert.match(errors, /import "#pad" -> "left-pad" is not a Node\.js builtin/);
|
||||||
|
assert.match(errors, /import "#odd" uses a package\.json "imports" alias that cannot be analyzed/);
|
||||||
|
assert.match(errors, /import "#up" uses a package\.json "imports" alias that cannot be analyzed/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("readRegularFile and checkExtensionModules refuse symlinked files", (t) => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"outside/secret.txt": "secret\n",
|
||||||
|
"extensions/link/extension.mjs": 'import x from "./lib.mjs";\nexport default x;\n',
|
||||||
|
});
|
||||||
|
const extensionDir = path.join(root, "extensions", "link");
|
||||||
|
try {
|
||||||
|
fs.symlinkSync(path.join(root, "outside", "secret.txt"), path.join(extensionDir, "lib.mjs"));
|
||||||
|
fs.symlinkSync(path.join(root, "outside", "secret.txt"), path.join(extensionDir, "package.json"));
|
||||||
|
} catch (error) {
|
||||||
|
if (["EPERM", "EACCES"].includes(error.code)) return t.skip("symlinks unavailable");
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
assert.match(readRegularFile(path.join(extensionDir, "lib.mjs")).error, /symbolic link/);
|
||||||
|
const errors = checkExtensionModules(extensionDir).errors.join("\n");
|
||||||
|
assert.match(errors, /package\.json/);
|
||||||
|
assert.match(errors, /lib\.mjs/);
|
||||||
|
assert.doesNotMatch(errors, /secret/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("readRegularFile enforces the size cap and containment root", () => {
|
||||||
|
const root = makeRepo({ "a/big.json": "x".repeat(64), "b/ok.json": "{}" });
|
||||||
|
assert.match(readRegularFile(path.join(root, "a", "big.json"), { maxBytes: 10 }).error, /limit/);
|
||||||
|
assert.match(readRegularFile(path.join(root, "b", "ok.json"), { root: path.join(root, "a") }).error, /outside/);
|
||||||
|
assert.equal(readRegularFile(path.join(root, "b", "ok.json"), { root: path.join(root, "b") }).text, "{}");
|
||||||
|
assert.equal(readRegularFile(path.join(root, "b", "missing.json")).missing, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("detectCanvasTargets still validates a plugin whose plugin.json was deleted", async () => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/orb/extension.mjs": "export {};\n",
|
||||||
|
"extensions/orb/assets/preview.png": makePng(800, 400),
|
||||||
|
});
|
||||||
|
const targets = detectCanvasTargets(["plugins/orb/plugin.json"], { rootDir: root });
|
||||||
|
assert.deepEqual(targets.plugins, ["orb"]);
|
||||||
|
assert.deepEqual(targets.extensions, ["orb"]);
|
||||||
|
const report = await runCanvasSmokeTest({ rootDir: root, changedFiles: ["plugins/orb/plugin.json"], install: "never" });
|
||||||
|
assert.equal(report.status, "fail");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("detectCanvasTargets uses base plugin manifests when deleted bundle manifests referenced extensions", async () => {
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/daily-focus-board/extension.mjs": "export {};\n",
|
||||||
|
"extensions/daily-focus-board/assets/preview.png": makePng(800, 400),
|
||||||
|
"plugins/ember/plugin.json": {
|
||||||
|
$schema: PLUGIN_SCHEMA,
|
||||||
|
name: "ember",
|
||||||
|
description: "Ember bundle",
|
||||||
|
version: "1.0.0",
|
||||||
|
extensions: { "com.github.awesome-copilot": { extensions: ["./extensions/daily-focus-board"] } },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
git(root, ["init"]);
|
||||||
|
git(root, ["config", "user.email", "test@example.com"]);
|
||||||
|
git(root, ["config", "user.name", "Test User"]);
|
||||||
|
git(root, ["add", "."]);
|
||||||
|
git(root, ["commit", "-m", "base"]);
|
||||||
|
const baseRef = git(root, ["rev-parse", "HEAD"]);
|
||||||
|
fs.rmSync(path.join(root, "plugins", "ember", "plugin.json"));
|
||||||
|
|
||||||
|
const targets = detectCanvasTargets(["plugins/ember/plugin.json"], { rootDir: root, baseRef });
|
||||||
|
assert.deepEqual(targets.extensions, ["daily-focus-board"]);
|
||||||
|
assert.deepEqual(targets.plugins, ["ember"]);
|
||||||
|
|
||||||
|
const report = await runCanvasSmokeTest({ rootDir: root, changedFiles: ["plugins/ember/plugin.json"], baseRef, install: "never" });
|
||||||
|
assert.equal(report.status, "fail");
|
||||||
|
assert.equal(report.extensions[0].id, "daily-focus-board");
|
||||||
|
assert.match(report.plugins[0].errors.join("\n"), /plugins\/ember\/plugin\.json is missing/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("deleted plugin manifest base read errors fail closed when a base ref is provided", async () => {
|
||||||
|
const root = makeRepo({});
|
||||||
|
const report = await runCanvasSmokeTest({
|
||||||
|
rootDir: root,
|
||||||
|
changedFiles: ["plugins/ember/plugin.json"],
|
||||||
|
baseRef: "missing-base",
|
||||||
|
install: "never",
|
||||||
|
});
|
||||||
|
assert.equal(report.status, "fail");
|
||||||
|
assert.match(report.plugins[0].errors.join("\n"), /cannot read deleted plugins\/ember\/plugin\.json from base missing-base/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("detectCanvasTargets unions base and head extension references for an edited bundle manifest", () => {
|
||||||
|
const manifest = (refs) => ({
|
||||||
|
$schema: PLUGIN_SCHEMA,
|
||||||
|
name: "ember",
|
||||||
|
description: "Ember bundle",
|
||||||
|
version: "1.0.0",
|
||||||
|
extensions: { "com.github.awesome-copilot": { extensions: refs } },
|
||||||
|
});
|
||||||
|
const root = makeRepo({
|
||||||
|
"extensions/daily-focus-board/extension.mjs": "export {};\n",
|
||||||
|
"extensions/daily-focus-board/assets/preview.png": makePng(800, 400),
|
||||||
|
"extensions/orb/extension.mjs": "export {};\n",
|
||||||
|
"extensions/orb/assets/preview.png": makePng(800, 400),
|
||||||
|
"plugins/ember/plugin.json": manifest(["./extensions/daily-focus-board"]),
|
||||||
|
});
|
||||||
|
git(root, ["init"]);
|
||||||
|
git(root, ["config", "user.email", "test@example.com"]);
|
||||||
|
git(root, ["config", "user.name", "Test User"]);
|
||||||
|
git(root, ["add", "."]);
|
||||||
|
git(root, ["commit", "-m", "base"]);
|
||||||
|
const baseRef = git(root, ["rev-parse", "HEAD"]);
|
||||||
|
fs.writeFileSync(
|
||||||
|
path.join(root, "plugins", "ember", "plugin.json"),
|
||||||
|
`${JSON.stringify(manifest(["./extensions/orb"]), null, 2)}\n`,
|
||||||
|
);
|
||||||
|
|
||||||
|
const targets = detectCanvasTargets(["plugins/ember/plugin.json"], { rootDir: root, baseRef });
|
||||||
|
assert.deepEqual(targets.extensions, ["daily-focus-board", "orb"]);
|
||||||
|
assert.deepEqual(targets.plugins, ["ember"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the comment stripper treats a slash after a postfix update as division", () => {
|
||||||
|
const source = [
|
||||||
|
'for (let i = 0; i < n; i++) {}',
|
||||||
|
'const r = x++ / y; const dep = require("left-pad");',
|
||||||
|
'const s = count-- / total; const dyn = import("./real.mjs");',
|
||||||
|
'const t = ++x / y;',
|
||||||
|
].join("\n");
|
||||||
|
assert.deepEqual(findRequireSpecifiers(source), ["left-pad"]);
|
||||||
|
assert.deepEqual(findDynamicImportSpecifiers(source), ["./real.mjs"]);
|
||||||
|
// A prefix update still allows a regex literal to follow.
|
||||||
|
assert.deepEqual(findRequireSpecifiers('const re = ++i, m = /require("hidden")/;'), []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("unsafe extension references in plugin manifests are rejected and never materialized", async () => {
|
||||||
|
assert.equal(isSafeExtensionId("orb"), true);
|
||||||
|
for (const id of ["..", "../x", "a/b", "", ".hidden"]) assert.equal(isSafeExtensionId(id), false, id);
|
||||||
|
const manifest = {
|
||||||
|
$schema: PLUGIN_SCHEMA,
|
||||||
|
name: "evil",
|
||||||
|
description: "evil",
|
||||||
|
version: "1.0.0",
|
||||||
|
extensions: { "com.github.awesome-copilot": { extensions: ["./extensions/../../x", "./extensions/ok"] } },
|
||||||
|
};
|
||||||
|
assert.deepEqual(unsafeExtensionRefs(manifest), ["./extensions/../../x"]);
|
||||||
|
const root = makeRepo({
|
||||||
|
"plugins/evil/plugin.json": manifest,
|
||||||
|
"extensions/ok/extension.mjs": "export {};\n",
|
||||||
|
"extensions/ok/assets/preview.png": makePng(800, 400),
|
||||||
|
});
|
||||||
|
const targets = detectCanvasTargets(["plugins/evil/plugin.json"], { rootDir: root });
|
||||||
|
assert.deepEqual(targets.plugins, ["evil"]);
|
||||||
|
assert.deepEqual(targets.extensions, ["ok"]);
|
||||||
|
const report = await runCanvasSmokeTest({ rootDir: root, changedFiles: ["plugins/evil/plugin.json"], install: "never" });
|
||||||
|
assert.equal(report.status, "fail");
|
||||||
|
const pluginErrors = report.plugins.flatMap((plugin) => plugin.errors).join("\n");
|
||||||
|
assert.match(pluginErrors, /\.\/extensions\/\.\.\/\.\.\/x/);
|
||||||
|
assert.notEqual(report.smoke.materialize.evil?.status, "pass");
|
||||||
|
});
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
// Minimal GitHub REST/GraphQL client for maintainer automation scripts.
|
||||||
|
//
|
||||||
|
// In workflows it uses GITHUB_TOKEN (or GH_TOKEN) with the global fetch API.
|
||||||
|
// For local dry runs without a token it falls back to the authenticated `gh`
|
||||||
|
// CLI (`gh api`), so maintainers never need to export credentials.
|
||||||
|
|
||||||
|
import { spawn } from "node:child_process";
|
||||||
|
|
||||||
|
const DEFAULT_API_URL = "https://api.github.com";
|
||||||
|
|
||||||
|
export function createGitHubClient({
|
||||||
|
token = process.env.GITHUB_TOKEN || process.env.GH_TOKEN,
|
||||||
|
apiUrl = process.env.GITHUB_API_URL || DEFAULT_API_URL,
|
||||||
|
userAgent = "awesome-copilot-maintainer-automation",
|
||||||
|
transport,
|
||||||
|
} = {}) {
|
||||||
|
const baseUrl = apiUrl.replace(/\/$/, "");
|
||||||
|
const send = transport ?? (token ? createFetchTransport({ token, userAgent }) : createGhTransport(baseUrl));
|
||||||
|
|
||||||
|
async function request(method, route, { body, query, allowStatuses = [] } = {}) {
|
||||||
|
const url = new URL(route.startsWith("http") ? route : `${baseUrl}${route}`);
|
||||||
|
for (const [key, value] of Object.entries(query ?? {})) {
|
||||||
|
if (value !== undefined && value !== null) url.searchParams.set(key, String(value));
|
||||||
|
}
|
||||||
|
for (let attempt = 0; ; attempt++) {
|
||||||
|
const response = await send(method, url, body);
|
||||||
|
if ((response.status === 502 || response.status === 503) && attempt < 2) {
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 1000 * (attempt + 1)));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const data = response.text ? safeJson(response.text) : null;
|
||||||
|
if ((response.status < 200 || response.status >= 300) && !allowStatuses.includes(response.status)) {
|
||||||
|
const message = data?.message || response.text || `HTTP ${response.status}`;
|
||||||
|
const error = new Error(`${method} ${url.pathname} failed: ${response.status} ${message}`);
|
||||||
|
error.status = response.status;
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
return { status: response.status, data, headers: response.headers };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function paginate(route, { query = {}, itemsKey, maxPages = 20 } = {}) {
|
||||||
|
const items = [];
|
||||||
|
let next = null;
|
||||||
|
let page = 0;
|
||||||
|
do {
|
||||||
|
const response = next ? await request("GET", next) : await request("GET", route, { query: { per_page: 100, ...query } });
|
||||||
|
const pageItems = itemsKey ? response.data?.[itemsKey] ?? [] : response.data ?? [];
|
||||||
|
items.push(...pageItems);
|
||||||
|
next = parseLink(response.headers.get("link"), "next");
|
||||||
|
page++;
|
||||||
|
} while (next && page < maxPages);
|
||||||
|
return items;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function graphql(query, variables = {}) {
|
||||||
|
const response = await request("POST", "/graphql", { body: { query, variables } });
|
||||||
|
if (response.data?.errors?.length) {
|
||||||
|
const error = new Error(`GraphQL request failed: ${response.data.errors.map((item) => item.message).join("; ")}`);
|
||||||
|
error.errors = response.data.errors;
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
return response.data?.data;
|
||||||
|
}
|
||||||
|
|
||||||
|
return { request, paginate, graphql };
|
||||||
|
}
|
||||||
|
|
||||||
|
function createFetchTransport({ token, userAgent }) {
|
||||||
|
const headers = {
|
||||||
|
Accept: "application/vnd.github+json",
|
||||||
|
Authorization: `Bearer ${token}`,
|
||||||
|
"User-Agent": userAgent,
|
||||||
|
"X-GitHub-Api-Version": "2022-11-28",
|
||||||
|
};
|
||||||
|
return async (method, url, body) => {
|
||||||
|
const response = await fetch(url, {
|
||||||
|
method,
|
||||||
|
headers: body ? { ...headers, "Content-Type": "application/json" } : headers,
|
||||||
|
body: body ? JSON.stringify(body) : undefined,
|
||||||
|
});
|
||||||
|
return { status: response.status, text: await response.text(), headers: response.headers };
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function createGhTransport(baseUrl) {
|
||||||
|
return (method, url, body) =>
|
||||||
|
new Promise((resolve, reject) => {
|
||||||
|
const endpoint = url.href.startsWith(baseUrl) ? url.href.slice(baseUrl.length + 1) : url.href;
|
||||||
|
const args = ["api", "--include", "--method", method, endpoint, "-H", "X-GitHub-Api-Version: 2022-11-28"];
|
||||||
|
if (body) args.push("--input", "-");
|
||||||
|
const child = spawn("gh", args, { stdio: ["pipe", "pipe", "pipe"] });
|
||||||
|
let stdout = "";
|
||||||
|
let stderr = "";
|
||||||
|
child.stdout.setEncoding("utf8").on("data", (chunk) => (stdout += chunk));
|
||||||
|
child.stderr.setEncoding("utf8").on("data", (chunk) => (stderr += chunk));
|
||||||
|
child.on("error", (error) =>
|
||||||
|
reject(new Error(`No GITHUB_TOKEN/GH_TOKEN set and the gh CLI could not be started: ${error.message}`)),
|
||||||
|
);
|
||||||
|
child.on("close", () => {
|
||||||
|
const parsed = parseIncludedResponse(stdout);
|
||||||
|
if (!parsed) {
|
||||||
|
reject(new Error(`gh api ${method} ${endpoint} failed: ${stderr.trim() || "no response"}`));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
resolve(parsed);
|
||||||
|
});
|
||||||
|
child.stdin.end(body ? JSON.stringify(body) : undefined);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseIncludedResponse(output) {
|
||||||
|
const normalized = output.replace(/\r\n/g, "\n");
|
||||||
|
const match = normalized.match(/^HTTP\/[\d.]+ (\d{3})[^\n]*\n([\s\S]*?)\n\n([\s\S]*)$/);
|
||||||
|
if (!match) return null;
|
||||||
|
const headers = new Headers();
|
||||||
|
for (const line of match[2].split("\n")) {
|
||||||
|
const index = line.indexOf(":");
|
||||||
|
if (index > 0) headers.append(line.slice(0, index).trim(), line.slice(index + 1).trim());
|
||||||
|
}
|
||||||
|
return { status: Number(match[1]), text: match[3], headers };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseLink(linkHeader, rel) {
|
||||||
|
if (!linkHeader) return null;
|
||||||
|
for (const part of linkHeader.split(",")) {
|
||||||
|
const match = part.match(/<([^>]+)>\s*;\s*rel="([^"]+)"/);
|
||||||
|
if (match && match[2] === rel) return match[1];
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function parseRepository(value = process.env.GITHUB_REPOSITORY) {
|
||||||
|
const [owner, repo] = String(value ?? "").split("/");
|
||||||
|
if (!owner || !repo) {
|
||||||
|
throw new Error("Repository must be provided as owner/repo (use --repo or GITHUB_REPOSITORY).");
|
||||||
|
}
|
||||||
|
return { owner, repo };
|
||||||
|
}
|
||||||
|
|
||||||
|
function safeJson(text) {
|
||||||
|
try {
|
||||||
|
return JSON.parse(text);
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,669 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
// Weekly review operating metrics (github/awesome-copilot#4184, phase 3).
|
||||||
|
//
|
||||||
|
// Collects read-only data from the GitHub API, computes review throughput and
|
||||||
|
// load metrics, and publishes them to a tracking issue. Labels introduced by
|
||||||
|
// other phases (state and risk tiers) are optional: missing labels are
|
||||||
|
// reported as "unlabeled" / "unclassified" instead of failing.
|
||||||
|
|
||||||
|
import fs from "node:fs";
|
||||||
|
import path from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
import * as yaml from "js-yaml";
|
||||||
|
import { createGitHubClient, parseRepository } from "./lib/review-automation-github.mjs";
|
||||||
|
|
||||||
|
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||||
|
export const ROOT_FOLDER = path.resolve(__dirname, "..");
|
||||||
|
export const DEFAULT_CONFIG_PATH = path.join(ROOT_FOLDER, ".github", "review-metrics.yml");
|
||||||
|
export const TRACKING_MARKER = "<!-- review-metrics-tracking -->";
|
||||||
|
export const REPORT_MARKER = "<!-- review-metrics-report -->";
|
||||||
|
const HOUR_MS = 60 * 60 * 1000;
|
||||||
|
const DAY_MS = 24 * HOUR_MS;
|
||||||
|
|
||||||
|
export const DEFAULT_CONFIG = Object.freeze({
|
||||||
|
window_days: 7,
|
||||||
|
targets_business_days: [2, 4],
|
||||||
|
base_branch: "main",
|
||||||
|
state_labels: ["awaiting-automation", "requires-submitter-fixes", "ready-for-review", "review-in-progress", "approved"],
|
||||||
|
risk_labels: ["merge-risk:low", "merge-risk:medium", "merge-risk:high"],
|
||||||
|
external_plugin_label: "external-plugin",
|
||||||
|
external_plugin_state_labels: ["awaiting-review", "requires-submitter-fixes", "ready-for-review", "awaiting-approval"],
|
||||||
|
automation_workflows: [],
|
||||||
|
tracking_issue: { title: "Review operating metrics", label: "review-metrics" },
|
||||||
|
});
|
||||||
|
|
||||||
|
export function normalizeConfig(raw = {}) {
|
||||||
|
const input = raw && typeof raw === "object" ? raw : {};
|
||||||
|
const config = { ...DEFAULT_CONFIG, ...input, tracking_issue: { ...DEFAULT_CONFIG.tracking_issue, ...(input.tracking_issue ?? {}) } };
|
||||||
|
config.window_days = Math.max(1, Number(config.window_days) || DEFAULT_CONFIG.window_days);
|
||||||
|
config.targets_business_days = (Array.isArray(config.targets_business_days) ? config.targets_business_days : [config.targets_business_days])
|
||||||
|
.map(Number)
|
||||||
|
.filter((value) => Number.isFinite(value) && value > 0)
|
||||||
|
.sort((a, b) => a - b);
|
||||||
|
for (const key of ["state_labels", "risk_labels", "external_plugin_state_labels", "automation_workflows"]) {
|
||||||
|
config[key] = (Array.isArray(config[key]) ? config[key] : []).map(String);
|
||||||
|
}
|
||||||
|
return config;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function loadMetricsConfig(filePath = DEFAULT_CONFIG_PATH) {
|
||||||
|
if (!fs.existsSync(filePath)) return normalizeConfig({});
|
||||||
|
return normalizeConfig(yaml.load(fs.readFileSync(filePath, "utf8")) ?? {});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Statistics helpers (pure)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
// Elapsed business time in days between two instants. Saturdays and Sundays
|
||||||
|
// (UTC) contribute nothing; partial weekdays count fractionally.
|
||||||
|
export function businessDaysBetween(start, end) {
|
||||||
|
let from = new Date(start).getTime();
|
||||||
|
const to = new Date(end).getTime();
|
||||||
|
if (!Number.isFinite(from) || !Number.isFinite(to) || to <= from) return 0;
|
||||||
|
let total = 0;
|
||||||
|
while (from < to) {
|
||||||
|
const dayStart = Math.floor(from / DAY_MS) * DAY_MS;
|
||||||
|
const dayEnd = Math.min(dayStart + DAY_MS, to);
|
||||||
|
const weekday = new Date(dayStart).getUTCDay();
|
||||||
|
if (weekday !== 0 && weekday !== 6) total += (dayEnd - from) / DAY_MS;
|
||||||
|
from = dayEnd;
|
||||||
|
}
|
||||||
|
return total;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Linear-interpolated percentile (same method as numpy's default).
|
||||||
|
export function percentile(values, p) {
|
||||||
|
const sorted = values.filter((value) => Number.isFinite(value)).sort((a, b) => a - b);
|
||||||
|
if (sorted.length === 0) return null;
|
||||||
|
const rank = (p / 100) * (sorted.length - 1);
|
||||||
|
const lower = Math.floor(rank);
|
||||||
|
const upper = Math.ceil(rank);
|
||||||
|
return sorted[lower] + (sorted[upper] - sorted[lower]) * (rank - lower);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function summarizeDurations(hours) {
|
||||||
|
return {
|
||||||
|
count: hours.length,
|
||||||
|
median_hours: roundOrNull(percentile(hours, 50)),
|
||||||
|
p90_hours: roundOrNull(percentile(hours, 90)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Herfindahl-Hirschman index on a 0-10,000 scale plus the top reviewer share.
|
||||||
|
export function concentration(counts) {
|
||||||
|
const values = Object.values(counts).filter((value) => value > 0);
|
||||||
|
const total = values.reduce((sum, value) => sum + value, 0);
|
||||||
|
if (total === 0) return { total: 0, reviewers: 0, top_share: null, hhi: null, effective_reviewers: null };
|
||||||
|
const shares = values.map((value) => value / total);
|
||||||
|
const hhi = shares.reduce((sum, share) => sum + share * share, 0);
|
||||||
|
return {
|
||||||
|
total,
|
||||||
|
reviewers: values.length,
|
||||||
|
top_share: round(Math.max(...shares), 3),
|
||||||
|
hhi: Math.round(hhi * 10000),
|
||||||
|
effective_reviewers: round(1 / hhi, 2),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function round(value, digits = 1) {
|
||||||
|
const factor = 10 ** digits;
|
||||||
|
return Math.round(value * factor) / factor;
|
||||||
|
}
|
||||||
|
|
||||||
|
function roundOrNull(value, digits = 1) {
|
||||||
|
return value === null ? null : round(value, digits);
|
||||||
|
}
|
||||||
|
|
||||||
|
function isBot(actor) {
|
||||||
|
return !actor || actor.type === "Bot" || /\[bot\]$/i.test(actor.login ?? "");
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Metric computation (pure)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
// First maintainer review submitted at or after the current review clock
|
||||||
|
// start, so reviews from before a draft/re-ready cycle are not counted.
|
||||||
|
function firstHumanReview(pr) {
|
||||||
|
const clockStart = Date.parse(reviewClockStart(pr) ?? "");
|
||||||
|
const reviews = (pr.reviews ?? [])
|
||||||
|
.filter((review) => isMaintainerReview(review, pr) && review.submittedAt && review.state !== "PENDING")
|
||||||
|
.filter((review) => !Number.isFinite(clockStart) || Date.parse(review.submittedAt) >= clockStart)
|
||||||
|
.sort((a, b) => Date.parse(a.submittedAt) - Date.parse(b.submittedAt));
|
||||||
|
return reviews[0] ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reviews from bots, the PR author, and users without write access do not count.
|
||||||
|
function isMaintainerReview(review, pr) {
|
||||||
|
return !isBot(review.author) && review.author?.login !== pr.author?.login && review.authorCanPushToRepository !== false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function reviewClockStart(pr) {
|
||||||
|
return pr.readyForReviewAt ?? pr.createdAt;
|
||||||
|
}
|
||||||
|
|
||||||
|
function tally(items, labelsFor, buckets, fallback) {
|
||||||
|
const counts = Object.fromEntries([...buckets, fallback].map((bucket) => [bucket, 0]));
|
||||||
|
for (const item of items) {
|
||||||
|
const labels = new Set(labelsFor(item).map((label) => label.toLowerCase()));
|
||||||
|
const matched = buckets.filter((bucket) => labels.has(bucket.toLowerCase()));
|
||||||
|
if (matched.length === 0) counts[fallback]++;
|
||||||
|
for (const bucket of matched) counts[bucket]++;
|
||||||
|
}
|
||||||
|
return counts;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function computeMetrics(data, config, { now = new Date() } = {}) {
|
||||||
|
const nowMs = new Date(now).getTime();
|
||||||
|
const windowStart = nowMs - config.window_days * DAY_MS;
|
||||||
|
const inWindow = (value) => {
|
||||||
|
const time = Date.parse(value ?? "");
|
||||||
|
return Number.isFinite(time) && time >= windowStart && time <= nowMs;
|
||||||
|
};
|
||||||
|
|
||||||
|
const contributionPrs = (data.openPrs ?? []).filter((pr) => !isBot(pr.author));
|
||||||
|
const readyPrs = contributionPrs.filter((pr) => !pr.isDraft);
|
||||||
|
const openIssues = data.externalPluginIssues ?? [];
|
||||||
|
|
||||||
|
const open = {
|
||||||
|
pull_requests: {
|
||||||
|
total: contributionPrs.length,
|
||||||
|
drafts: contributionPrs.length - readyPrs.length,
|
||||||
|
automation_authored_excluded: (data.openPrs ?? []).length - contributionPrs.length,
|
||||||
|
by_state: tally(readyPrs, (pr) => pr.labels, config.state_labels, "unlabeled"),
|
||||||
|
by_risk: tally(readyPrs, (pr) => pr.labels, config.risk_labels, "unclassified"),
|
||||||
|
},
|
||||||
|
external_plugin_issues: {
|
||||||
|
total: openIssues.length,
|
||||||
|
by_state: tally(openIssues, (issue) => issue.labels, config.external_plugin_state_labels, "unlabeled"),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
// Time to first review: PRs whose first human review landed in the window.
|
||||||
|
const firstReviewHours = [];
|
||||||
|
// Time to merge: PRs merged in the window.
|
||||||
|
const mergeHours = [];
|
||||||
|
const reviewsByMaintainer = {};
|
||||||
|
const prsByMaintainer = {};
|
||||||
|
for (const pr of data.windowPrs ?? []) {
|
||||||
|
if (isBot(pr.author)) continue;
|
||||||
|
const first = firstHumanReview(pr);
|
||||||
|
if (first && inWindow(first.submittedAt)) {
|
||||||
|
firstReviewHours.push((Date.parse(first.submittedAt) - Date.parse(reviewClockStart(pr))) / HOUR_MS);
|
||||||
|
}
|
||||||
|
if (pr.mergedAt && inWindow(pr.mergedAt)) {
|
||||||
|
mergeHours.push((Date.parse(pr.mergedAt) - Date.parse(pr.createdAt)) / HOUR_MS);
|
||||||
|
}
|
||||||
|
for (const review of pr.reviews ?? []) {
|
||||||
|
if (!isMaintainerReview(review, pr) || !inWindow(review.submittedAt)) continue;
|
||||||
|
const login = review.author.login;
|
||||||
|
reviewsByMaintainer[login] = (reviewsByMaintainer[login] ?? 0) + 1;
|
||||||
|
(prsByMaintainer[login] ??= new Set()).add(pr.number);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const maintainers = Object.keys(reviewsByMaintainer)
|
||||||
|
.map((login) => ({ login, reviews: reviewsByMaintainer[login], pull_requests: prsByMaintainer[login].size }))
|
||||||
|
.sort((a, b) => b.pull_requests - a.pull_requests || b.reviews - a.reviews || a.login.localeCompare(b.login));
|
||||||
|
const prCounts = Object.fromEntries(maintainers.map((maintainer) => [maintainer.login, maintainer.pull_requests]));
|
||||||
|
|
||||||
|
// Items waiting on a maintainer beyond the business-day targets.
|
||||||
|
const waiting = [];
|
||||||
|
for (const pr of readyPrs) {
|
||||||
|
const labels = new Set(pr.labels.map((label) => label.toLowerCase()));
|
||||||
|
if (labels.has("requires-submitter-fixes")) continue;
|
||||||
|
if (firstHumanReview(pr)) continue;
|
||||||
|
const since = reviewClockStart(pr);
|
||||||
|
waiting.push({ kind: "pr", number: pr.number, title: pr.title, url: pr.url, since, business_days: round(businessDaysBetween(since, nowMs), 2) });
|
||||||
|
}
|
||||||
|
for (const issue of openIssues) {
|
||||||
|
const labels = new Set(issue.labels.map((label) => label.toLowerCase()));
|
||||||
|
if (!labels.has("ready-for-review") && !labels.has("awaiting-approval")) continue;
|
||||||
|
const since = issue.readyAt ?? issue.createdAt;
|
||||||
|
waiting.push({ kind: "issue", number: issue.number, title: issue.title, url: issue.url, since, business_days: round(businessDaysBetween(since, nowMs), 2) });
|
||||||
|
}
|
||||||
|
waiting.sort((a, b) => b.business_days - a.business_days);
|
||||||
|
const overdue = Object.fromEntries(
|
||||||
|
config.targets_business_days.map((target) => {
|
||||||
|
const items = waiting.filter((item) => item.business_days > target);
|
||||||
|
return [String(target), { count: items.length, items }];
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Automation failure rate over completed runs of the review workflows.
|
||||||
|
const automation = [];
|
||||||
|
const automationErrors = [];
|
||||||
|
let failedTotal = 0;
|
||||||
|
let consideredTotal = 0;
|
||||||
|
for (const workflow of data.workflowRuns ?? []) {
|
||||||
|
if (workflow.error) {
|
||||||
|
automationErrors.push({ workflow: workflow.file, error: workflow.error });
|
||||||
|
automation.push({ workflow: workflow.file, found: true, error: workflow.error });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!workflow.found) {
|
||||||
|
automation.push({ workflow: workflow.file, found: false });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const runs = workflow.runs.filter((run) => run.status === "completed" && inWindow(run.created_at));
|
||||||
|
const considered = runs.filter((run) => !["cancelled", "skipped", "neutral", "action_required", "stale"].includes(run.conclusion));
|
||||||
|
const failed = considered.filter((run) => ["failure", "timed_out", "startup_failure"].includes(run.conclusion));
|
||||||
|
failedTotal += failed.length;
|
||||||
|
consideredTotal += considered.length;
|
||||||
|
automation.push({
|
||||||
|
workflow: workflow.file,
|
||||||
|
name: workflow.name,
|
||||||
|
found: true,
|
||||||
|
runs: considered.length,
|
||||||
|
failed: failed.length,
|
||||||
|
failure_rate: considered.length ? round(failed.length / considered.length, 3) : null,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
generated_at: new Date(nowMs).toISOString(),
|
||||||
|
window: { days: config.window_days, start: new Date(windowStart).toISOString(), end: new Date(nowMs).toISOString() },
|
||||||
|
open,
|
||||||
|
time_to_first_review: summarizeDurations(firstReviewHours),
|
||||||
|
time_to_merge: summarizeDurations(mergeHours),
|
||||||
|
reviews_per_maintainer: maintainers,
|
||||||
|
reviewer_concentration: { ...concentration(prCounts), basis: "distinct PRs reviewed" },
|
||||||
|
overdue,
|
||||||
|
automation: {
|
||||||
|
failure_rate: consideredTotal ? round(failedTotal / consideredTotal, 3) : null,
|
||||||
|
failed: failedTotal,
|
||||||
|
runs: consideredTotal,
|
||||||
|
incomplete: automationErrors.length > 0,
|
||||||
|
errors: automationErrors,
|
||||||
|
workflows: automation,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Rendering
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
function formatHours(hours) {
|
||||||
|
if (hours === null || hours === undefined) return "n/a";
|
||||||
|
if (hours < 48) return `${round(hours, 1)} h`;
|
||||||
|
return `${round(hours / 24, 1)} d`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatPercent(value) {
|
||||||
|
return value === null || value === undefined ? "n/a" : `${round(value * 100, 1)}%`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function escapeCell(value) {
|
||||||
|
return String(value ?? "").replace(/\|/g, "\\|").replace(/\r?\n/g, " ").replace(/@(?=[A-Za-z0-9])/g, "@\u200b");
|
||||||
|
}
|
||||||
|
|
||||||
|
function countsTable(title, counts) {
|
||||||
|
const lines = [`| ${title} | Count |`, "|---|---:|"];
|
||||||
|
for (const [key, value] of Object.entries(counts)) lines.push(`| \`${key}\` | ${value} |`);
|
||||||
|
return lines.join("\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function renderReport(metrics, { repository, runUrl } = {}) {
|
||||||
|
const lines = [REPORT_MARKER, `## Review operating metrics — week ending ${metrics.window.end.slice(0, 10)}`, ""];
|
||||||
|
lines.push(`Window: ${metrics.window.start.slice(0, 10)} → ${metrics.window.end.slice(0, 10)} (${metrics.window.days} days, UTC). Business-day targets skip weekends.`, "");
|
||||||
|
|
||||||
|
const prs = metrics.open.pull_requests;
|
||||||
|
lines.push("### Open contributions", "");
|
||||||
|
lines.push(`**${prs.total}** open contribution PRs (${prs.drafts} draft, ${prs.automation_authored_excluded} automation-authored excluded) and **${metrics.open.external_plugin_issues.total}** open external plugin submissions.`, "");
|
||||||
|
lines.push(countsTable("PR state (non-draft)", prs.by_state), "", countsTable("PR risk tier (non-draft)", prs.by_risk), "");
|
||||||
|
lines.push(countsTable("External plugin state", metrics.open.external_plugin_issues.by_state), "");
|
||||||
|
|
||||||
|
lines.push("### Review speed", "", "| Metric | Samples | Median | p90 |", "|---|---:|---:|---:|");
|
||||||
|
lines.push(`| Time to first review | ${metrics.time_to_first_review.count} | ${formatHours(metrics.time_to_first_review.median_hours)} | ${formatHours(metrics.time_to_first_review.p90_hours)} |`);
|
||||||
|
lines.push(`| Time to merge | ${metrics.time_to_merge.count} | ${formatHours(metrics.time_to_merge.median_hours)} | ${formatHours(metrics.time_to_merge.p90_hours)} |`, "");
|
||||||
|
|
||||||
|
const conc = metrics.reviewer_concentration;
|
||||||
|
lines.push("### Reviewer load", "");
|
||||||
|
if (metrics.reviews_per_maintainer.length === 0) {
|
||||||
|
lines.push("No maintainer reviews in this window.", "");
|
||||||
|
} else {
|
||||||
|
lines.push(`Top reviewer share: **${formatPercent(conc.top_share)}** · HHI: **${conc.hhi}** (0–10,000; >2,500 = highly concentrated) · effective reviewers: **${conc.effective_reviewers}**`, "");
|
||||||
|
lines.push("| Maintainer | PRs reviewed | Reviews |", "|---|---:|---:|");
|
||||||
|
for (const maintainer of metrics.reviews_per_maintainer) lines.push(`| ${escapeCell(`@${maintainer.login}`)} | ${maintainer.pull_requests} | ${maintainer.reviews} |`);
|
||||||
|
lines.push("");
|
||||||
|
}
|
||||||
|
|
||||||
|
lines.push("### Waiting on maintainers", "", "| Target | Items past target |", "|---|---:|");
|
||||||
|
for (const [target, entry] of Object.entries(metrics.overdue)) lines.push(`| ${target} business days | ${entry.count} |`);
|
||||||
|
const targets = Object.keys(metrics.overdue);
|
||||||
|
const firstTarget = targets[0];
|
||||||
|
if (firstTarget && metrics.overdue[firstTarget].count > 0) {
|
||||||
|
lines.push("", `<details><summary>Items past ${firstTarget} business days</summary>`, "", "| Item | Waiting (business days) | Title |", "|---|---:|---|");
|
||||||
|
for (const item of metrics.overdue[firstTarget].items.slice(0, 50)) {
|
||||||
|
lines.push(`| ${item.kind === "pr" ? "PR" : "Issue"} [#${item.number}](${item.url}) | ${item.business_days} | ${escapeCell(item.title)} |`);
|
||||||
|
}
|
||||||
|
if (metrics.overdue[firstTarget].items.length > 50) lines.push(`| … | | ${metrics.overdue[firstTarget].items.length - 50} more |`);
|
||||||
|
lines.push("", "</details>");
|
||||||
|
}
|
||||||
|
lines.push("");
|
||||||
|
|
||||||
|
const automation = metrics.automation;
|
||||||
|
lines.push("### Automation health", "");
|
||||||
|
lines.push(`Failure rate: **${formatPercent(automation.failure_rate)}${automation.incomplete ? " (incomplete)" : ""}** (${automation.failed} failed of ${automation.runs} completed runs; cancelled/skipped excluded).`, "");
|
||||||
|
if (automation.incomplete) lines.push("⚠️ Collection was incomplete because one or more workflow APIs returned errors; affected workflows are excluded from the denominator.", "");
|
||||||
|
lines.push("| Workflow | Runs | Failed | Failure rate |", "|---|---:|---:|---:|");
|
||||||
|
for (const workflow of automation.workflows) {
|
||||||
|
if (workflow.error) {
|
||||||
|
lines.push(`| \`${workflow.workflow}\` | – | – | collection error: ${escapeCell(workflow.error)} |`);
|
||||||
|
} else {
|
||||||
|
lines.push(workflow.found ? `| \`${workflow.workflow}\` | ${workflow.runs} | ${workflow.failed} | ${formatPercent(workflow.failure_rate)} |` : `| \`${workflow.workflow}\` | – | – | not found |`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
lines.push("", `_Generated ${metrics.generated_at}${repository ? ` for ${repository}` : ""}${runUrl ? ` by [this run](${runUrl})` : ""}. Definitions: docs/maintainers/canvas-evidence-and-metrics.md._`);
|
||||||
|
return lines.join("\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Data collection
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
const PR_FIELDS = `
|
||||||
|
id number title url createdAt mergedAt isDraft state
|
||||||
|
author { login __typename }
|
||||||
|
labels(first: 50) { nodes { name } }
|
||||||
|
reviews(first: 100) { pageInfo { hasNextPage endCursor } nodes { state submittedAt authorCanPushToRepository author { login __typename } } }
|
||||||
|
timelineItems(last: 1, itemTypes: [READY_FOR_REVIEW_EVENT]) { nodes { ... on ReadyForReviewEvent { createdAt } } }
|
||||||
|
`;
|
||||||
|
|
||||||
|
const REVIEW_FIELDS = "state submittedAt authorCanPushToRepository author { login __typename }";
|
||||||
|
|
||||||
|
async function searchAll(client, query, fields, maxItems = 1000) {
|
||||||
|
const items = [];
|
||||||
|
let cursor = null;
|
||||||
|
do {
|
||||||
|
const data = await client.graphql(
|
||||||
|
`query($q: String!, $cursor: String) {
|
||||||
|
search(query: $q, type: ISSUE, first: 50, after: $cursor) {
|
||||||
|
pageInfo { hasNextPage endCursor }
|
||||||
|
nodes { ... on PullRequest { ${fields} } ... on Issue { ${ISSUE_FIELDS} } }
|
||||||
|
}
|
||||||
|
}`,
|
||||||
|
{ q: query, cursor },
|
||||||
|
);
|
||||||
|
items.push(...(data.search.nodes ?? []).filter(Boolean));
|
||||||
|
cursor = data.search.pageInfo.hasNextPage ? data.search.pageInfo.endCursor : null;
|
||||||
|
} while (cursor && items.length < maxItems);
|
||||||
|
return items;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function hydrateReviewPages(client, pullRequests) {
|
||||||
|
for (const pr of pullRequests) {
|
||||||
|
let cursor = pr.reviews?.pageInfo?.hasNextPage ? pr.reviews.pageInfo.endCursor : null;
|
||||||
|
while (cursor) {
|
||||||
|
const data = await client.graphql(
|
||||||
|
`query($id: ID!, $cursor: String) {
|
||||||
|
node(id: $id) {
|
||||||
|
... on PullRequest {
|
||||||
|
reviews(first: 100, after: $cursor) {
|
||||||
|
pageInfo { hasNextPage endCursor }
|
||||||
|
nodes { ${REVIEW_FIELDS} }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}`,
|
||||||
|
{ id: pr.id, cursor },
|
||||||
|
);
|
||||||
|
const page = data.node?.reviews;
|
||||||
|
if (!page) break;
|
||||||
|
pr.reviews.nodes.push(...(page.nodes ?? []));
|
||||||
|
cursor = page.pageInfo?.hasNextPage ? page.pageInfo.endCursor : null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return pullRequests;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function hydrateIssueLabelEventPages(client, issues) {
|
||||||
|
for (const issue of issues) {
|
||||||
|
let cursor = issue.timelineItems?.pageInfo?.hasPreviousPage ? issue.timelineItems.pageInfo.startCursor : null;
|
||||||
|
while (cursor) {
|
||||||
|
const data = await client.graphql(
|
||||||
|
`query($id: ID!, $cursor: String) {
|
||||||
|
node(id: $id) {
|
||||||
|
... on Issue {
|
||||||
|
timelineItems(last: 100, before: $cursor, itemTypes: [LABELED_EVENT]) {
|
||||||
|
pageInfo { hasPreviousPage startCursor }
|
||||||
|
nodes { ... on LabeledEvent { createdAt label { name } } }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}`,
|
||||||
|
{ id: issue.id, cursor },
|
||||||
|
);
|
||||||
|
const page = data.node?.timelineItems;
|
||||||
|
if (!page) break;
|
||||||
|
issue.timelineItems.nodes.unshift(...(page.nodes ?? []));
|
||||||
|
cursor = page.pageInfo?.hasPreviousPage ? page.pageInfo.startCursor : null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return issues;
|
||||||
|
}
|
||||||
|
|
||||||
|
const ISSUE_FIELDS = `
|
||||||
|
id number title url createdAt
|
||||||
|
labels(first: 50) { nodes { name } }
|
||||||
|
timelineItems(last: 100, itemTypes: [LABELED_EVENT]) {
|
||||||
|
pageInfo { hasPreviousPage startCursor }
|
||||||
|
nodes { ... on LabeledEvent { createdAt label { name } } }
|
||||||
|
}
|
||||||
|
`;
|
||||||
|
|
||||||
|
function normalizePr(node) {
|
||||||
|
return {
|
||||||
|
number: node.number,
|
||||||
|
title: node.title,
|
||||||
|
url: node.url,
|
||||||
|
createdAt: node.createdAt,
|
||||||
|
mergedAt: node.mergedAt,
|
||||||
|
isDraft: node.isDraft,
|
||||||
|
readyForReviewAt: node.timelineItems?.nodes?.[0]?.createdAt ?? null,
|
||||||
|
author: node.author ? { login: node.author.login, type: node.author.__typename } : null,
|
||||||
|
labels: (node.labels?.nodes ?? []).map((label) => label.name),
|
||||||
|
reviews: (node.reviews?.nodes ?? []).map((review) => ({
|
||||||
|
state: review.state,
|
||||||
|
submittedAt: review.submittedAt,
|
||||||
|
authorCanPushToRepository: review.authorCanPushToRepository,
|
||||||
|
author: review.author ? { login: review.author.login, type: review.author.__typename } : null,
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeIssue(node) {
|
||||||
|
const readyEvents = (node.timelineItems?.nodes ?? []).filter((event) => ["ready-for-review", "awaiting-approval"].includes(event?.label?.name));
|
||||||
|
return {
|
||||||
|
number: node.number,
|
||||||
|
title: node.title,
|
||||||
|
url: node.url,
|
||||||
|
createdAt: node.createdAt,
|
||||||
|
readyAt: readyEvents.length ? readyEvents[readyEvents.length - 1].createdAt : null,
|
||||||
|
labels: (node.labels?.nodes ?? []).map((label) => label.name),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fetchWorkflowRuns(client, { owner, repo }, file, since) {
|
||||||
|
const probe = await client.request("GET", `/repos/${owner}/${repo}/actions/workflows/${encodeURIComponent(file)}`, { allowStatuses: [404] });
|
||||||
|
if (probe.status === 404) return { file, found: false, runs: [] };
|
||||||
|
const runs = await client.paginate(`/repos/${owner}/${repo}/actions/workflows/${encodeURIComponent(file)}/runs`, {
|
||||||
|
query: { created: `>=${since.slice(0, 10)}`, exclude_pull_requests: true },
|
||||||
|
itemsKey: "workflow_runs",
|
||||||
|
maxPages: 10,
|
||||||
|
});
|
||||||
|
return {
|
||||||
|
file,
|
||||||
|
found: true,
|
||||||
|
name: probe.data?.name,
|
||||||
|
runs: runs.map((run) => ({ status: run.status, conclusion: run.conclusion, created_at: run.created_at })),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function collectData(client, repository, config, { now = new Date() } = {}) {
|
||||||
|
const repo = `${repository.owner}/${repository.repo}`;
|
||||||
|
const since = new Date(new Date(now).getTime() - config.window_days * DAY_MS).toISOString();
|
||||||
|
const sinceDate = since.slice(0, 10);
|
||||||
|
|
||||||
|
const openPrNodes = await hydrateReviewPages(client, await searchAll(client, `repo:${repo} is:pr is:open base:${config.base_branch}`, PR_FIELDS));
|
||||||
|
const windowPrNodes = await hydrateReviewPages(client, await searchAll(client, `repo:${repo} is:pr base:${config.base_branch} updated:>=${sinceDate}`, PR_FIELDS));
|
||||||
|
const openPrs = openPrNodes.map(normalizePr);
|
||||||
|
const windowPrs = windowPrNodes.map(normalizePr);
|
||||||
|
let externalPluginIssues = [];
|
||||||
|
if (config.external_plugin_label) {
|
||||||
|
const externalPluginIssueNodes = await searchAll(
|
||||||
|
client,
|
||||||
|
`repo:${repo} is:issue is:open label:"${config.external_plugin_label}"`,
|
||||||
|
PR_FIELDS,
|
||||||
|
);
|
||||||
|
externalPluginIssues = (await hydrateIssueLabelEventPages(client, externalPluginIssueNodes)).map(normalizeIssue);
|
||||||
|
}
|
||||||
|
const workflowRuns = [];
|
||||||
|
for (const file of config.automation_workflows) {
|
||||||
|
try {
|
||||||
|
workflowRuns.push(await fetchWorkflowRuns(client, repository, file, since));
|
||||||
|
} catch (error) {
|
||||||
|
console.warn(`Could not read runs for ${file}: ${error.message}`);
|
||||||
|
workflowRuns.push({ file, found: true, runs: [], error: error.message });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { openPrs, windowPrs, externalPluginIssues, workflowRuns };
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Publishing
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
async function ensureLabel(client, { owner, repo }, name) {
|
||||||
|
const existing = await client.request("GET", `/repos/${owner}/${repo}/labels/${encodeURIComponent(name)}`, { allowStatuses: [404] });
|
||||||
|
if (existing.status === 404) {
|
||||||
|
await client.request("POST", `/repos/${owner}/${repo}/labels`, {
|
||||||
|
body: { name, color: "C5DEF5", description: "Weekly review operating metrics tracking issue" },
|
||||||
|
allowStatuses: [422],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function renderTrackingBody(report) {
|
||||||
|
return [
|
||||||
|
TRACKING_MARKER,
|
||||||
|
"This issue tracks weekly review operating metrics for contribution review (#4184).",
|
||||||
|
"It is updated automatically by `.github/workflows/review-metrics.yml`; each week's report is also posted as a comment so trends stay visible.",
|
||||||
|
"",
|
||||||
|
"---",
|
||||||
|
"",
|
||||||
|
report,
|
||||||
|
].join("\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function publishReport(client, repository, config, report) {
|
||||||
|
const { owner, repo } = repository;
|
||||||
|
const label = config.tracking_issue.label;
|
||||||
|
await ensureLabel(client, repository, label);
|
||||||
|
const issues = await client.paginate(`/repos/${owner}/${repo}/issues`, { query: { state: "open", labels: label }, maxPages: 5 });
|
||||||
|
let issue = issues.find((candidate) => !candidate.pull_request && candidate.body?.includes(TRACKING_MARKER));
|
||||||
|
const body = renderTrackingBody(report);
|
||||||
|
if (issue) {
|
||||||
|
await client.request("PATCH", `/repos/${owner}/${repo}/issues/${issue.number}`, { body: { body } });
|
||||||
|
} else {
|
||||||
|
issue = (await client.request("POST", `/repos/${owner}/${repo}/issues`, { body: { title: config.tracking_issue.title, body, labels: [label] } })).data;
|
||||||
|
try {
|
||||||
|
await client.graphql(`mutation($id: ID!) { pinIssue(input: { issueId: $id }) { issue { number } } }`, { id: issue.node_id });
|
||||||
|
} catch (error) {
|
||||||
|
console.warn(`Could not pin tracking issue #${issue.number}: ${error.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await client.request("POST", `/repos/${owner}/${repo}/issues/${issue.number}/comments`, { body: { body: report } });
|
||||||
|
return issue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// CLI
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
export function parseArgs(argv) {
|
||||||
|
const options = {
|
||||||
|
config: DEFAULT_CONFIG_PATH,
|
||||||
|
repo: process.env.GITHUB_REPOSITORY,
|
||||||
|
dryRun: false,
|
||||||
|
outputDir: null,
|
||||||
|
summaryFile: process.env.GITHUB_STEP_SUMMARY,
|
||||||
|
now: new Date(),
|
||||||
|
windowDays: null,
|
||||||
|
runUrl: process.env.METRICS_RUN_URL || null,
|
||||||
|
};
|
||||||
|
for (let index = 0; index < argv.length; index++) {
|
||||||
|
const arg = argv[index];
|
||||||
|
const next = () => {
|
||||||
|
const value = argv[++index];
|
||||||
|
if (value === undefined) throw new Error(`Missing value for ${arg}`);
|
||||||
|
return value;
|
||||||
|
};
|
||||||
|
switch (arg) {
|
||||||
|
case "--config":
|
||||||
|
options.config = next();
|
||||||
|
break;
|
||||||
|
case "--repo":
|
||||||
|
options.repo = next();
|
||||||
|
break;
|
||||||
|
case "--dry-run":
|
||||||
|
options.dryRun = true;
|
||||||
|
break;
|
||||||
|
case "--output-dir":
|
||||||
|
options.outputDir = next();
|
||||||
|
break;
|
||||||
|
case "--summary-file":
|
||||||
|
options.summaryFile = next();
|
||||||
|
break;
|
||||||
|
case "--now": {
|
||||||
|
const value = new Date(next());
|
||||||
|
if (Number.isNaN(value.getTime())) throw new Error("--now expects an ISO date");
|
||||||
|
options.now = value;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
case "--window-days":
|
||||||
|
options.windowDays = Number(next());
|
||||||
|
if (!Number.isFinite(options.windowDays) || options.windowDays <= 0) throw new Error("--window-days expects a positive number");
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
throw new Error(`Unknown argument: ${arg}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return options;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const options = parseArgs(process.argv.slice(2));
|
||||||
|
const config = loadMetricsConfig(options.config);
|
||||||
|
if (options.windowDays) config.window_days = options.windowDays;
|
||||||
|
const repository = parseRepository(options.repo);
|
||||||
|
const client = createGitHubClient();
|
||||||
|
|
||||||
|
const data = await collectData(client, repository, config, { now: options.now });
|
||||||
|
const metrics = computeMetrics(data, config, { now: options.now });
|
||||||
|
const report = renderReport(metrics, { repository: `${repository.owner}/${repository.repo}`, runUrl: options.runUrl });
|
||||||
|
|
||||||
|
if (options.outputDir) {
|
||||||
|
fs.mkdirSync(options.outputDir, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(options.outputDir, "metrics.json"), `${JSON.stringify(metrics, null, 2)}\n`);
|
||||||
|
fs.writeFileSync(path.join(options.outputDir, "report.md"), `${report}\n`);
|
||||||
|
}
|
||||||
|
if (options.summaryFile) fs.appendFileSync(options.summaryFile, `${report}\n`);
|
||||||
|
|
||||||
|
if (options.dryRun) {
|
||||||
|
if (!options.summaryFile) console.log(report);
|
||||||
|
console.log("Dry run: tracking issue not updated.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const issue = await publishReport(client, repository, config, report);
|
||||||
|
console.log(`Published review metrics to ${issue.html_url ?? `#${issue.number}`}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||||
|
main().catch((error) => {
|
||||||
|
console.error(error);
|
||||||
|
process.exitCode = 1;
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,297 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import test from "node:test";
|
||||||
|
|
||||||
|
import {
|
||||||
|
businessDaysBetween,
|
||||||
|
collectData,
|
||||||
|
computeMetrics,
|
||||||
|
concentration,
|
||||||
|
loadMetricsConfig,
|
||||||
|
normalizeConfig,
|
||||||
|
parseArgs,
|
||||||
|
percentile,
|
||||||
|
renderReport,
|
||||||
|
renderTrackingBody,
|
||||||
|
TRACKING_MARKER,
|
||||||
|
} from "./review-metrics.mjs";
|
||||||
|
|
||||||
|
test("business days skip weekends and count partial weekdays", () => {
|
||||||
|
// 2025-01-03 is a Friday.
|
||||||
|
assert.equal(businessDaysBetween("2025-01-03T00:00:00Z", "2025-01-06T00:00:00Z"), 1);
|
||||||
|
assert.equal(businessDaysBetween("2025-01-04T00:00:00Z", "2025-01-06T00:00:00Z"), 0);
|
||||||
|
assert.equal(businessDaysBetween("2025-01-03T12:00:00Z", "2025-01-06T12:00:00Z"), 1);
|
||||||
|
assert.equal(businessDaysBetween("2025-01-06T00:00:00Z", "2025-01-13T00:00:00Z"), 5);
|
||||||
|
assert.equal(businessDaysBetween("2025-01-06T00:00:00Z", "2025-01-06T06:00:00Z"), 0.25);
|
||||||
|
assert.equal(businessDaysBetween("2025-01-06T00:00:00Z", "2025-01-05T00:00:00Z"), 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("percentiles interpolate linearly and ignore invalid values", () => {
|
||||||
|
assert.equal(percentile([], 50), null);
|
||||||
|
assert.equal(percentile([5], 90), 5);
|
||||||
|
assert.equal(percentile([1, 2, 3, 4], 50), 2.5);
|
||||||
|
assert.equal(percentile([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 90), 9.1);
|
||||||
|
assert.equal(percentile([3, NaN, 1, 2], 50), 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("reviewer concentration reports top share and HHI", () => {
|
||||||
|
assert.deepEqual(concentration({}), { total: 0, reviewers: 0, top_share: null, hhi: null, effective_reviewers: null });
|
||||||
|
const single = concentration({ a: 4 });
|
||||||
|
assert.equal(single.hhi, 10000);
|
||||||
|
assert.equal(single.top_share, 1);
|
||||||
|
const even = concentration({ a: 5, b: 5, c: 5, d: 5 });
|
||||||
|
assert.equal(even.hhi, 2500);
|
||||||
|
assert.equal(even.top_share, 0.25);
|
||||||
|
assert.equal(even.effective_reviewers, 4);
|
||||||
|
});
|
||||||
|
|
||||||
|
const config = normalizeConfig({ automation_workflows: ["a.yml", "missing.yml"] });
|
||||||
|
const now = "2025-01-10T12:00:00Z"; // Friday
|
||||||
|
const maintainer = (login, submittedAt, state = "APPROVED") => ({ state, submittedAt, authorCanPushToRepository: true, author: { login, type: "User" } });
|
||||||
|
|
||||||
|
function fixture() {
|
||||||
|
return {
|
||||||
|
openPrs: [
|
||||||
|
{ number: 1, title: "Waiting a while", url: "u1", createdAt: "2025-01-03T12:00:00Z", isDraft: false, author: { login: "alice", type: "User" }, labels: ["ready-for-review", "merge-risk:low"], reviews: [] },
|
||||||
|
{ number: 2, title: "Needs fixes", url: "u2", createdAt: "2025-01-02T12:00:00Z", isDraft: false, author: { login: "bob", type: "User" }, labels: ["requires-submitter-fixes"], reviews: [] },
|
||||||
|
{ number: 3, title: "Reviewed", url: "u3", createdAt: "2025-01-02T12:00:00Z", isDraft: false, author: { login: "carol", type: "User" }, labels: [], reviews: [maintainer("m1", "2025-01-06T12:00:00Z", "COMMENTED")] },
|
||||||
|
{ number: 4, title: "Draft", url: "u4", createdAt: "2025-01-09T12:00:00Z", isDraft: true, author: { login: "dave", type: "User" }, labels: [], reviews: [] },
|
||||||
|
{ number: 5, title: "Bot PR", url: "u5", createdAt: "2025-01-09T12:00:00Z", isDraft: false, author: { login: "github-actions", type: "Bot" }, labels: [], reviews: [] },
|
||||||
|
{ number: 6, title: "Fresh", url: "u6", createdAt: "2025-01-09T12:00:00Z", isDraft: false, author: { login: "erin", type: "User" }, labels: ["merge-risk:high"], reviews: [] },
|
||||||
|
],
|
||||||
|
windowPrs: [
|
||||||
|
{ number: 10, createdAt: "2025-01-06T00:00:00Z", mergedAt: "2025-01-07T00:00:00Z", author: { login: "x", type: "User" }, labels: [], reviews: [maintainer("m1", "2025-01-06T02:00:00Z"), maintainer("m1", "2025-01-06T05:00:00Z")] },
|
||||||
|
{ number: 11, createdAt: "2025-01-06T00:00:00Z", mergedAt: "2025-01-09T00:00:00Z", author: { login: "y", type: "User" }, labels: [], reviews: [maintainer("m2", "2025-01-06T10:00:00Z"), maintainer("y", "2025-01-06T01:00:00Z"), { ...maintainer("bot", "2025-01-06T00:30:00Z"), author: { login: "copilot", type: "Bot" } }] },
|
||||||
|
{ number: 12, createdAt: "2024-12-01T00:00:00Z", mergedAt: null, author: { login: "z", type: "User" }, labels: [], reviews: [maintainer("m1", "2024-12-02T00:00:00Z")] },
|
||||||
|
{ number: 13, createdAt: "2025-01-08T00:00:00Z", mergedAt: null, author: { login: "w", type: "User" }, labels: [], reviews: [{ ...maintainer("drive-by", "2025-01-08T01:00:00Z"), authorCanPushToRepository: false }] },
|
||||||
|
],
|
||||||
|
externalPluginIssues: [
|
||||||
|
{ number: 20, title: "Plugin", url: "i20", createdAt: "2024-12-20T00:00:00Z", readyAt: "2025-01-03T00:00:00Z", labels: ["external-plugin", "ready-for-review"] },
|
||||||
|
{ number: 21, title: "Plugin fixes", url: "i21", createdAt: "2024-12-20T00:00:00Z", labels: ["external-plugin", "requires-submitter-fixes"] },
|
||||||
|
],
|
||||||
|
workflowRuns: [
|
||||||
|
{
|
||||||
|
file: "a.yml",
|
||||||
|
found: true,
|
||||||
|
name: "A",
|
||||||
|
runs: [
|
||||||
|
{ status: "completed", conclusion: "success", created_at: "2025-01-08T00:00:00Z" },
|
||||||
|
{ status: "completed", conclusion: "failure", created_at: "2025-01-08T00:00:00Z" },
|
||||||
|
{ status: "completed", conclusion: "startup_failure", created_at: "2025-01-08T00:00:00Z" },
|
||||||
|
{ status: "completed", conclusion: "cancelled", created_at: "2025-01-08T00:00:00Z" },
|
||||||
|
{ status: "completed", conclusion: "skipped", created_at: "2025-01-08T00:00:00Z" },
|
||||||
|
{ status: "in_progress", conclusion: null, created_at: "2025-01-08T00:00:00Z" },
|
||||||
|
{ status: "completed", conclusion: "failure", created_at: "2024-12-01T00:00:00Z" },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{ file: "missing.yml", found: false, runs: [] },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test("computes open contribution breakdowns with graceful label fallbacks", () => {
|
||||||
|
const metrics = computeMetrics(fixture(), config, { now });
|
||||||
|
const prs = metrics.open.pull_requests;
|
||||||
|
assert.equal(prs.total, 5, "bot PRs are excluded");
|
||||||
|
assert.equal(prs.drafts, 1);
|
||||||
|
assert.equal(prs.automation_authored_excluded, 1);
|
||||||
|
assert.equal(prs.by_state["ready-for-review"], 1);
|
||||||
|
assert.equal(prs.by_state["requires-submitter-fixes"], 1);
|
||||||
|
assert.equal(prs.by_state.unlabeled, 2);
|
||||||
|
assert.equal(prs.by_risk["merge-risk:low"], 1);
|
||||||
|
assert.equal(prs.by_risk["merge-risk:high"], 1);
|
||||||
|
assert.equal(prs.by_risk.unclassified, 2);
|
||||||
|
assert.equal(metrics.open.external_plugin_issues.by_state["ready-for-review"], 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("computes review speed, maintainer load, and concentration", () => {
|
||||||
|
const metrics = computeMetrics(fixture(), config, { now });
|
||||||
|
// First maintainer reviews in window: #10 after 2h, #11 after 10h. Author, bot,
|
||||||
|
// and non-maintainer reviews (#13) are ignored.
|
||||||
|
assert.equal(metrics.time_to_first_review.count, 2);
|
||||||
|
assert.equal(metrics.time_to_first_review.median_hours, 6);
|
||||||
|
assert.equal(metrics.time_to_merge.count, 2);
|
||||||
|
assert.equal(metrics.time_to_merge.median_hours, 48);
|
||||||
|
assert.deepEqual(
|
||||||
|
metrics.reviews_per_maintainer.map((item) => [item.login, item.pull_requests, item.reviews]),
|
||||||
|
[["m1", 1, 2], ["m2", 1, 1]],
|
||||||
|
);
|
||||||
|
assert.equal(metrics.reviewer_concentration.top_share, 0.5);
|
||||||
|
assert.equal(metrics.reviewer_concentration.hhi, 5000);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("flags items past business-day targets", () => {
|
||||||
|
const metrics = computeMetrics(fixture(), config, { now });
|
||||||
|
// #1 waits 5 business days, #6 waits 1, issue #20 waits 5.5; #2 needs submitter fixes; #3 was reviewed.
|
||||||
|
assert.deepEqual(metrics.overdue["2"].items.map((item) => `${item.kind}#${item.number}`), ["issue#20", "pr#1"]);
|
||||||
|
assert.equal(metrics.overdue["4"].count, 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("computes automation failure rate over completed, non-cancelled runs", () => {
|
||||||
|
const metrics = computeMetrics(fixture(), config, { now });
|
||||||
|
assert.equal(metrics.automation.runs, 3);
|
||||||
|
assert.equal(metrics.automation.failed, 2);
|
||||||
|
assert.equal(metrics.automation.failure_rate, 0.667);
|
||||||
|
assert.equal(metrics.automation.workflows[1].found, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("automation API errors are reported as incomplete instead of not found", () => {
|
||||||
|
const data = fixture();
|
||||||
|
data.workflowRuns.push({ file: "error.yml", found: true, runs: [], error: "server unavailable" });
|
||||||
|
const metrics = computeMetrics(data, config, { now });
|
||||||
|
assert.equal(metrics.automation.incomplete, true);
|
||||||
|
assert.deepEqual(metrics.automation.errors, [{ workflow: "error.yml", error: "server unavailable" }]);
|
||||||
|
assert.equal(metrics.automation.runs, 3);
|
||||||
|
assert.equal(metrics.automation.failed, 2);
|
||||||
|
assert.equal(metrics.automation.failure_rate, 0.667);
|
||||||
|
const report = renderReport(metrics, {});
|
||||||
|
assert.match(report, /Failure rate: \*\*66\.7% \(incomplete\)\*\*/);
|
||||||
|
assert.match(report, /`error\.yml` \| – \| – \| collection error: server unavailable/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("collectData paginates PR reviews beyond the first 100", async () => {
|
||||||
|
const pageOneReviews = Array.from({ length: 100 }, (_, index) => maintainer(`m${index}`, "2025-01-06T01:00:00Z"));
|
||||||
|
const pageTwoReview = maintainer("last", "2025-01-06T02:00:00Z");
|
||||||
|
const client = {
|
||||||
|
graphql: async (query) => {
|
||||||
|
if (query.includes("search(")) {
|
||||||
|
return {
|
||||||
|
search: {
|
||||||
|
pageInfo: { hasNextPage: false, endCursor: null },
|
||||||
|
nodes: [{
|
||||||
|
id: "PR_1",
|
||||||
|
number: 1,
|
||||||
|
title: "Many reviews",
|
||||||
|
url: "https://example.test/pull/1",
|
||||||
|
createdAt: "2025-01-06T00:00:00Z",
|
||||||
|
mergedAt: null,
|
||||||
|
isDraft: false,
|
||||||
|
author: { login: "author", __typename: "User" },
|
||||||
|
labels: { nodes: [] },
|
||||||
|
reviews: { pageInfo: { hasNextPage: true, endCursor: "page-1" }, nodes: [...pageOneReviews] },
|
||||||
|
timelineItems: { nodes: [] },
|
||||||
|
}],
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
assert.match(query, /node\(id: \$id\)/);
|
||||||
|
return {
|
||||||
|
node: {
|
||||||
|
reviews: { pageInfo: { hasNextPage: false, endCursor: null }, nodes: [pageTwoReview] },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
},
|
||||||
|
request: async () => { throw new Error("request should not be called"); },
|
||||||
|
paginate: async () => { throw new Error("paginate should not be called"); },
|
||||||
|
};
|
||||||
|
const data = await collectData(client, { owner: "o", repo: "r" }, normalizeConfig({ automation_workflows: [], external_plugin_label: "" }), { now });
|
||||||
|
assert.equal(data.openPrs[0].reviews.length, 101);
|
||||||
|
assert.equal(data.windowPrs[0].reviews.length, 101);
|
||||||
|
assert.equal(data.windowPrs[0].reviews[100].author.login, "last");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("collectData paginates external plugin issue label events", async () => {
|
||||||
|
let searchCalls = 0;
|
||||||
|
const client = {
|
||||||
|
graphql: async (query, variables) => {
|
||||||
|
if (query.includes("search(")) {
|
||||||
|
searchCalls++;
|
||||||
|
if (searchCalls <= 2) {
|
||||||
|
return { search: { pageInfo: { hasNextPage: false, endCursor: null }, nodes: [] } };
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
search: {
|
||||||
|
pageInfo: { hasNextPage: false, endCursor: null },
|
||||||
|
nodes: [{
|
||||||
|
id: "ISSUE_1",
|
||||||
|
number: 42,
|
||||||
|
title: "Long-lived plugin",
|
||||||
|
url: "https://example.test/issues/42",
|
||||||
|
createdAt: "2024-01-01T00:00:00Z",
|
||||||
|
labels: { nodes: [{ name: "external-plugin" }, { name: "ready-for-review" }] },
|
||||||
|
timelineItems: {
|
||||||
|
pageInfo: { hasPreviousPage: true, startCursor: "recent-page" },
|
||||||
|
nodes: [{ createdAt: "2025-01-09T00:00:00Z", label: { name: "triage" } }],
|
||||||
|
},
|
||||||
|
}],
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
assert.equal(variables.id, "ISSUE_1");
|
||||||
|
assert.equal(variables.cursor, "recent-page");
|
||||||
|
return {
|
||||||
|
node: {
|
||||||
|
timelineItems: {
|
||||||
|
pageInfo: { hasPreviousPage: false, startCursor: null },
|
||||||
|
nodes: [{ createdAt: "2025-01-03T00:00:00Z", label: { name: "ready-for-review" } }],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
},
|
||||||
|
request: async () => { throw new Error("request should not be called"); },
|
||||||
|
paginate: async () => { throw new Error("paginate should not be called"); },
|
||||||
|
};
|
||||||
|
const data = await collectData(
|
||||||
|
client,
|
||||||
|
{ owner: "o", repo: "r" },
|
||||||
|
normalizeConfig({ automation_workflows: [], external_plugin_label: "external-plugin" }),
|
||||||
|
{ now },
|
||||||
|
);
|
||||||
|
assert.equal(data.externalPluginIssues[0].readyAt, "2025-01-03T00:00:00Z");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("handles an empty repository without errors", () => {
|
||||||
|
const metrics = computeMetrics({}, config, { now });
|
||||||
|
assert.equal(metrics.time_to_first_review.median_hours, null);
|
||||||
|
assert.equal(metrics.automation.failure_rate, null);
|
||||||
|
const report = renderReport(metrics, {});
|
||||||
|
assert.match(report, /No maintainer reviews/);
|
||||||
|
assert.match(report, /n\/a/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("renders a report and tracking issue body", () => {
|
||||||
|
const metrics = computeMetrics(fixture(), config, { now });
|
||||||
|
const report = renderReport(metrics, { repository: "o/r", runUrl: "https://example.test/run" });
|
||||||
|
assert.match(report, /Review operating metrics — week ending 2025-01-10/);
|
||||||
|
assert.match(report, /Time to first review \| 2 \| 6 h/);
|
||||||
|
assert.match(report, /HHI: \*\*5000\*\*/);
|
||||||
|
assert.match(report, /`missing.yml` \| – \| – \| not found/);
|
||||||
|
assert.match(report, /@\u200bm1/, "mentions are neutralized");
|
||||||
|
assert.ok(renderTrackingBody(report).startsWith(TRACKING_MARKER));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("config and CLI parsing", () => {
|
||||||
|
const loaded = loadMetricsConfig();
|
||||||
|
assert.equal(loaded.window_days, 7);
|
||||||
|
assert.deepEqual(loaded.targets_business_days, [2, 4]);
|
||||||
|
assert.ok(loaded.automation_workflows.includes("canvas-smoke-test.yml"));
|
||||||
|
assert.equal(loadMetricsConfig("does-not-exist.yml").tracking_issue.label, "review-metrics");
|
||||||
|
|
||||||
|
const options = parseArgs(["--repo", "o/r", "--dry-run", "--now", "2025-01-10T00:00:00Z", "--window-days", "14"]);
|
||||||
|
assert.equal(options.dryRun, true);
|
||||||
|
assert.equal(options.windowDays, 14);
|
||||||
|
assert.equal(options.now.toISOString(), "2025-01-10T00:00:00.000Z");
|
||||||
|
assert.throws(() => parseArgs(["--now", "nope"]), /ISO date/);
|
||||||
|
assert.throws(() => parseArgs(["--bogus"]), /Unknown argument/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("time to first review ignores reviews submitted before the PR was ready", () => {
|
||||||
|
const data = {
|
||||||
|
openPrs: [],
|
||||||
|
externalPluginIssues: [],
|
||||||
|
workflowRuns: [],
|
||||||
|
windowPrs: [
|
||||||
|
{
|
||||||
|
number: 30,
|
||||||
|
createdAt: "2025-01-06T00:00:00Z",
|
||||||
|
readyForReviewAt: "2025-01-07T00:00:00Z",
|
||||||
|
mergedAt: null,
|
||||||
|
author: { login: "x", type: "User" },
|
||||||
|
labels: [],
|
||||||
|
reviews: [maintainer("m1", "2025-01-06T01:00:00Z", "COMMENTED"), maintainer("m2", "2025-01-07T04:00:00Z")],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
const metrics = computeMetrics(data, config, { now });
|
||||||
|
assert.equal(metrics.time_to_first_review.count, 1);
|
||||||
|
assert.equal(metrics.time_to_first_review.median_hours, 4);
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user