doublecheck: auto-escalate to full verification report for high-risk content (#980)

* doublecheck: auto-escalate to full report for high-risk content

In active (persistent) mode, doublecheck previously used abbreviated
inline verification for all content types, with the full three-layer
report only available if the user explicitly requested it. Most users
did not know the command existed, so high-stakes content (legal
analysis, regulatory guidance) and responses with DISPUTED or
FABRICATION RISK findings were presented with only a summary table.

Changes:
- Auto-escalate to the full verification report when any claim rates
  DISPUTED or FABRICATION RISK during inline verification
- Always use the full report for legal analysis, regulatory
  interpretation, compliance guidance, and content with case citations
  or statutory references
- Update the activation message to mention the 'full report' command
- Add a discoverability footer to inline verification sections
- Update README.md to document the new behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: fix table reference, reconcile bullets, clarify callout placement, add trigger phrase

- Table row now references 'high-stakes content rule' (not 'auto-escalation
  rule') and lists all content types that match the later rule text
- README first bullet no longer claims legal content gets inline
  verification, resolving the contradiction with the next bullet
- Clarify that the 'Heads up' callout goes at the top of the full report
  when auto-escalation applies, not in a nonexistent inline section
- Add 'full report' to the explicit trigger phrase list in the 'Offer
  full verification on request' rule

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update README.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update README.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Dan Velton
2026-03-11 20:27:52 -07:00
committed by GitHub
parent 423be2fc70
commit f391bc9c30
2 changed files with 24 additions and 7 deletions

View File

@@ -48,10 +48,11 @@ use doublecheck
```
Once active:
- Factual, legal, and analytical responses get automatic inline verification
- Simple factual lookups and single-claim answers get automatic inline verification summaries
- Factual analysis, legal analysis, regulatory interpretation, compliance guidance, and content with case citations or statutory references automatically get the full verification report instead of inline summaries
- If any claim rates DISPUTED or FABRICATION RISK during inline verification, the full report is generated automatically
- Code, creative writing, and casual conversation are skipped (verification doesn't apply)
- High-risk claims (DISPUTED, FABRICATION RISK) get called out prominently before the verification summary
- You can ask for a full deep-dive verification on any response by saying "run full verification"
- You can ask for a full deep-dive verification on any response by saying "full report" (or the legacy phrase "run full verification")
- Turn it off anytime with "turn off doublecheck"
Turn it off anytime:

View File

@@ -15,7 +15,7 @@ Doublecheck operates in two modes: **active mode** (persistent) and **one-shot m
When the user invokes this skill without providing specific text to verify, activate persistent doublecheck mode. Respond with:
> **Doublecheck is now active.** I'll automatically verify the factual claims in my responses before presenting them to you. For each substantive response, you'll see an inline verification summary with confidence ratings and source links. You can turn it off anytime by saying "turn off doublecheck."
> **Doublecheck is now active.** I'll verify factual claims in my responses before presenting them. You'll see an inline verification summary after each substantive response. Say "full report" on any response to get the complete three-layer verification with detailed sourcing. Turn it off anytime by saying "turn off doublecheck."
Then follow ALL of the rules below for the remainder of the conversation:
@@ -25,7 +25,7 @@ Before producing any substantive response, determine whether it contains verifia
| Response type | Contains verifiable claims? | Action |
|--------------|---------------------------|--------|
| Factual analysis, legal guidance, regulatory interpretation | Yes -- high density | Run full inline verification |
| Factual analysis, legal guidance, regulatory interpretation, compliance guidance, or content with case citations or statutory references | Yes -- high density | Run full verification report (see high-stakes content rule below) |
| Summary of a document, research, or data | Yes -- moderate density | Run inline verification on key claims |
| Code generation, creative writing, brainstorming | Rarely | Skip verification; note that doublecheck mode doesn't apply to this type of content |
| Casual conversation, clarifying questions, status updates | No | Skip verification silently |
@@ -52,15 +52,31 @@ Format:
For active mode, prioritize speed. Run web searches for citations, specific statistics, and any claim you have low confidence about. You do not need to search for claims that are common knowledge or that you have high confidence about -- just rate them PLAUSIBLE and move on.
If any claim rates DISPUTED or FABRICATION RISK, call it out prominently before the verification section so the user sees it immediately:
If any claim rates DISPUTED or FABRICATION RISK, call it out prominently before the verification section so the user sees it immediately. When auto-escalation applies (see below), place this callout at the top of the full report, before the summary table:
```
**Heads up:** I'm not confident about [specific claim]. I couldn't find a supporting source. You should verify this independently before relying on it.
```
**Rule: Auto-escalate to full report for high-risk findings.**
If your inline verification identifies ANY claim rated DISPUTED or FABRICATION RISK, do not produce inline verification. Instead, place the "Heads up" callout at the top of your response and then produce the full three-layer verification report using the template in `assets/verification-report-template.md`. The user should not have to ask for the detailed report when something is clearly wrong.
**Rule: Full report for high-stakes content.**
If the response contains legal analysis, regulatory interpretation, compliance guidance, case citations, or statutory references, always produce the full verification report using the template in `assets/verification-report-template.md`. Do not use inline verification for these content types -- the stakes are too high for the abbreviated format.
**Rule: Discoverability footer for inline verification.**
When producing inline verification (not a full report), always append this line at the end of the verification section:
```
_Say "full report" for detailed three-layer verification with sources._
```
**Rule: Offer full verification on request.**
If the user says "run full verification," "verify that," "doublecheck that," or similar, run the complete three-layer pipeline (described below) and produce the full report using the template in `assets/verification-report-template.md`.
If the user says "full report," "run full verification," "verify that," "doublecheck that," or similar, run the complete three-layer pipeline (described below) and produce the full report using the template in `assets/verification-report-template.md`.
### One-Shot Mode