Files
awesome-copilot/skills/copilot-pr-autopilot/references/04-triage.md
T
Gordon Lam d47a6c93b7 Add copilot-pr-autopilot skill (#1944)
* Add copilot-pr-autopilot skill

Skill that drives any GitHub pull request through repeated rounds of
Copilot Code Review until the agent has either resolved every thread
or explicitly escalated it to the human. Triggered via GraphQL (no
@copilot mention needed), triages every open thread with a fix /
decline / escalate rubric, replies and resolves each thread citing
the pushed SHA, then re-triggers until HEAD is reviewed with zero
threads awaiting the agent's reply.

Includes step scripts (01 request-review, 02 check-review-status,
03 list-open-threads, 08 reply-and-resolve, 10 cleanup-outdated),
shared library (_lib.ps1) with gh-CLI wrappers (Invoke-Gh,
Invoke-GhGraphQL, ConvertFrom-GhJson, Assert-GhReady), reply
templates, and reference docs for each step.

Repo-agnostic. Requires gh CLI on PATH and repo Triage/Write for
full autopilot; external PR authors get single-iteration mode with
manual re-trigger via the UI re-request button or a substantive
push.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review: split per-step references + add recap-gate circuit breaker

- Fix #1: give steps 1/7/10 their own reference files
  (01-request-review.md, 07-commit-push.md, 10-cleanup.md); trim the
  inline bodies out of orchestration.md so it stays cross-cutting only.
- Fix #3: add a recurring round-cap & recap gate to 09-convergence.md —
  default STOP every 10th round, recap all prior rounds, detect drift
  (out-of-scope / over-engineering / wrong-direction / belongs-in-separate-PR)
  with CONTINUE / REVERT-AND-SHIP / HAND-OFF verdicts. Agent reasoning,
  no new script.
- Surface the gate from SKILL.md and orchestration.md; regenerate
  docs/README.skills.md. Markdown-only change; scripts unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(copilot-pr-autopilot): surface recap gate in decision pseudo-code; clarify Copilot+human convergence and round definition

- Inject the round-cap recap gate into the '## Decision: loop back or exit'
  pseudo-code else-branch so an agent following the code block (not just the
  prose) runs the STOP-every-10th-round check before looping.
- Broaden the 'never terminal' paragraph: non-convergence is driven by a
  Copilot finding OR a human review comment (this skill handles both); the
  loop ends only when there are no new comments from either source AND every
  open thread (Copilot or human) has an agent reply/escalation.
- Define a 'round' explicitly as one execution of step 1 (01-request-review),
  i.e. one Copilot-review trigger — the cap counts review rounds, not tool
  calls or fix edits.

Markdown-only; no script changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* copilot-pr-autopilot: make recap-gate round count deterministic

Add 09-review-round.ps1: counts Copilot Code Review submissions straight
from the PR's API history (full GraphQL pagination), so the recap-gate
trigger is a derived number, not a fallible agent mental tally. This
removes the exact failure mode the skill exists to survive — a count
drifting across a long run (the real 156-round case).

The script reports Round + RecapDue (Round % RecapInterval == 0) only;
it never stops the loop or picks the verdict. CONTINUE / REVERT-AND-SHIP
/ HAND-OFF stays agent reasoning. 09-convergence.md updated to reference
the deterministic count while preserving 'no script stops the loop' and
'non-convergence = Copilot finding OR human comment'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Regenerate docs/README.skills.md for copilot-pr-autopilot (add 09-review-round.ps1)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-01 11:09:58 +10:00

191 lines
8.0 KiB
Markdown

# Step 4: Triage
Sub-agent type: `general-purpose`; budget: **5 min per ≤5 threads**
(parent batches in waves of ≤5 if step 3 returned more).
## Inputs
From step 3:
- Classified thread table — `{ thread_id, file, line, author,
author_class, severity, summary }` per open thread.
- Original PR context (description, recent commits to the affected
files) for re-deriving from principles in oscillation cases.
## Return contract
Table of rows, one per thread:
```
{ thread_id, action, rationale }
```
Where `action` ∈ `fix` | `decline` | `escalate-to-user` and
`rationale` is a single line citing the rule from the rubric below
that fired.
## Procedure
For each thread, apply the rubric in this file in order:
1. **Reviewer-type policy** — `human-or-bot` threads default to
`escalate-to-user` unless the user explicitly scoped them in.
2. **ROI vs Risk** — score and decide for `copilot` threads.
3. **Fix / Decline rule lists** — match the finding pattern.
4. **Project-specific policy hooks** — research repo conventions
before applying a generic "fix".
5. **Conflicting-comments resolution** — if the finding flips a
prior round's edit, apply the oscillation rules; if the agent is
about to re-do an edit it already reverted, hard-stop and escalate.
6. **Escalation rules** — escalate on design-level tradeoffs,
large/cross-cutting changes, or high-risk/irreversible actions.
Return `{ thread_id, action, rationale }` per thread to the parent;
step 5 consumes only the `fix` rows, step 8 consumes the full table.
## Gotchas
- **Batch in waves of ≤5 per sub-agent invocation.** Larger batches
blur the rationale and overrun the 5-min budget.
- **Human / advanced-security default is `escalate-to-user`.**
Auto-replying or auto-resolving a human review thread hides
unaddressed concerns and is socially wrong.
- **Cite the rule that fired in `rationale`** — one line, concrete.
"Style nit per rubric §Decline" is fine; "looks small" is not.
- **The oscillation hard-stop is non-negotiable.** If the agent is
about to make an edit it already reverted in a prior round, that's
the unambiguous signature of an oscillation loop — escalate.
---
# Triage Rubric
Decision rubric for whether to fix or decline each Copilot finding.
The goal is correctness, not appeasement — decline confidently when
warranted.
## ROI vs Risk
Score every proposed change on two axes, then decide:
- **ROI** = value of the fix MINUS the cost to implement MINUS the
user's cost to review.
- **Risk** = blast radius, reversibility, effect on unrelated code.
| ROI | Risk | Action |
|---|---|---|
| Clear positive | Low | Fix unilaterally. |
| Marginal | Low | Fix if cheap; otherwise decline with rationale. |
| Clear positive | High / Irreversible | Propose to the user first. |
| Marginal | High | Decline. |
| Negative (over-engineering for hypothetical) | Any | Decline. |
## Reviewer-type policy (foundation, not a nit)
Each thread's `author` (returned by `03-list-open-threads.ps1`) drives
who can decide it:
| Reviewer | Default action |
|----------|----------------|
| `copilot-pull-request-reviewer` / `copilot-pull-request-reviewer[bot]` | Loop-owned — triage with the rubric below. (`03-list-open-threads.ps1` reports the raw `author.login`, which may carry the `[bot]` suffix on some surfaces; match both forms.) |
| Human reviewer | **Default `escalate-to-user`** unless the user explicitly scoped them into the loop. Auto-replying or auto-resolving a human thread can hide unaddressed concerns and is socially wrong. |
| `github-advanced-security` / other automated bots | **Default `escalate-to-user`** unless the project has a documented suppression / fix convention you can follow. |
## Fix when the finding is...
- A **real correctness bug**: use-after-free / lifetime violation,
race that drops user intent, gating logic that skips legitimate
transitions, missing link dependency, off-by-one, null deref,
unhandled error path.
- A **cross-cutting concern with a clean local fix**: moving a mutex
one scope up, pulling a duplicated check into a helper.
- **Documentation / test-plan drift**: PR description claims behavior
the code no longer matches; comment block describes the wrong
thing; test-plan checkbox is inverted vs. the code.
## Decline when the finding is...
- A **purely hypothetical race** requiring cross-class plumbing,
where actual exposure is negligible. Cite the interleaving you
ruled out.
- **Style, naming, or formatting**. Out of scope for the review loop.
- **Suggestions to add abstractions** ("introduce a strategy pattern",
"extract an interface") that don't pay for themselves at the
current scale of the codebase.
- **Suggestions that contradict an established project convention** —
consistency with surrounding code is usually more valuable than the
suggestion in isolation.
- **Micro-optimizations** in code that is not on a hot path.
## Project-specific policy hooks
Some findings are decided by **project policy**, not by general
correctness reasoning. Before applying a generic "fix", **research
the repo's own conventions first** — `.github/instructions/*.md`
files (often have an `applyTo` glob that pins them to the changed
file's path), `.github/skills/`, `AGENTS.md`, `CONTRIBUTING.md`, CI
config, and recent commits to similar files. Fan out multiple
`explore` sub-agents when several axes need checking (lint, format,
spell-check, license header, etc.) — don't invent answers. What
looks like an obvious fix may violate a project rule:
- **Spell-check / dictionary findings.** If the project uses a
spell-checker (`check-spelling`, `cspell`, `typos`, or similar),
inspect its config and recent commits to learn the local
convention (reword the document, add a pattern/regex, extend a
dictionary/allowlist, use an inline ignore). Follow that
convention; don't invent a new mechanism.
- **Lint suppressions / inline ignore directives.** Most projects
require an inline rationale comment. Bare suppressions get pushed
back.
- **License headers / file boilerplate.** Project-specific format,
often CI-enforced — copy the existing header from a neighbor file.
- **Test framework, mock library, formatter choices.** Follow the
convention of the surrounding test files; never introduce a new
framework because Copilot suggested one.
Cite the project's config file or convention in your reply.
## Conflicting comments — break oscillation early
Failure mode: round N "fixes" what comment A asked for; round N+1
comment B objects and asks to revert it. Blindly flip-flopping
ships oscillation and burns rounds.
Resolution rules — apply in order, stop at the first that fires:
1. **Re-derive from principles.** Pick the side that wins on the
function's contract + surrounding patterns + user's stated
preferences — not the latest comment.
2. **Prefer the position with the explicit rationale.** Concrete
failure modes (security, correctness, data loss, perf) win over
stylistic positions.
3. **Prefer human comments over bot comments** when they directly
conflict.
4. **If still ambiguous, escalate to the user.** Summarize both
positions side-by-side with a recommendation; do not silently
pick.
**Hard stop**: if you are about to make the same edit you
*reverted* in an earlier round, stop and escalate. That is the
unambiguous signature of an oscillation loop.
## Reply hygiene
Every reply — fix or decline — states the reasoning. This makes the
PR self-documenting and gives the next review visible context.
Declined findings cite the prior round and explain why the existing
form is correct, so the next reviewer doesn't raise it again.
## Escalation rules
Stay in autopilot by default; escalate to the user when:
- The finding identifies a **design-level tradeoff** with multiple
reasonable resolutions and no clear winner.
- The fix would be **large or cross-cutting** (hundreds of lines,
new architecture, refactor across modules).
- The action is **high-risk or irreversible** (force-push, deletion,
credentials, production).
Otherwise, decide and proceed.