mirror of
https://github.com/github/awesome-copilot.git
synced 2026-10-12 23:04:14 +00:00
fix: adopt progressive disclosure for sast-sca-security-analyzer and modularize audit-integrity 🤖🤖🤖 (#3047)
* feat(skills): modularize audit-integrity references for progressive disclosure * refactor(agents): adopt progressive disclosure in sast-sca-security-analyzer * style(skills): format table alignment in security-report-template * refactor(skills): restore domain-agnostic audit-integrity references * refactor(agents): make sast-sca-security-analyzer self-contained
This commit is contained in:
@@ -1,50 +1,44 @@
|
||||
---
|
||||
name: 'audit-integrity'
|
||||
description: 'Shared audit integrity framework for all AppSec agents — enforces output quality, intellectual honesty, and continuous improvement through anti-rationalization guards, self-critique loops, retry protocols, non-negotiable behaviors, self-reflection quality gates (1-10 scoring, ≥8 threshold), and a self-learning system with lesson/memory governance for security analysis agents.'
|
||||
compatibility: 'Cross-platform. Works with any language or framework analyzed by AppSec agents.'
|
||||
name: audit-integrity
|
||||
description: "Enforce output quality, evidence verification, and quality gates across security audits. Use this skill when you perform SAST, SCA, threat modeling, or security code reviews. Use this skill to verify code locations and taint traces for all findings. Use this skill to prevent the suppression of valid security findings. Use this skill to evaluate reports against quality thresholds (score ≥ 8/10) before delivery."
|
||||
license: MIT
|
||||
metadata:
|
||||
version: '1.0'
|
||||
version: "1.1"
|
||||
---
|
||||
|
||||
# Audit Integrity Skill
|
||||
|
||||
Enforces output quality, intellectual honesty, and continuous improvement across all AppSec agents.
|
||||
Enforce output quality, intellectual honesty, and continuous improvement across security audits and code reviews.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Every security analysis, code review, threat model, or quality scan agent run
|
||||
- Applied automatically as a post-analysis quality gate
|
||||
- Applicable to any agent performing SAST, SCA, threat modeling, or code quality analysis
|
||||
- Perform security analyses, threat models, code reviews, or quality audits.
|
||||
- Validate that every finding has verified code evidence and taint flow.
|
||||
- Prevent the suppression or rationalization of security findings.
|
||||
- Evaluate draft reports against the quality gate threshold (score ≥ 8/10).
|
||||
- Record lessons learned and security memories after you complete a scan.
|
||||
|
||||
## Components
|
||||
|
||||
This skill provides 7 reusable capabilities. Agents apply all 7 unless their scope excludes a specific component.
|
||||
This skill provides 7 modular capabilities in the `references/` directory. Load each file when you reach its execution phase:
|
||||
|
||||
| Component | Reference File | Purpose |
|
||||
|-----------|---------------|---------|
|
||||
| Clarification Protocol | [clarification-protocol.md](references/clarification-protocol.md) | Ask ≤2 targeted questions before analysis when scope is ambiguous |
|
||||
| Anti-Rationalization Guard | [anti-rationalization-guard.md](references/anti-rationalization-guard.md) | Table of prohibited rationalizations with mandatory responses |
|
||||
| Self-Critique Loop | [self-critique-loop.md](references/self-critique-loop.md) | Mandatory second-pass review after initial analysis |
|
||||
| Retry Protocol | [retry-protocol.md](references/retry-protocol.md) | Tool failure handling — retry once, then document |
|
||||
| Non-Negotiable Behaviors | [non-negotiable-behaviors.md](references/non-negotiable-behaviors.md) | Hard rules: never fabricate, always cite evidence, report gaps |
|
||||
| Self-Reflection Quality Gate | [self-reflection-quality-gate.md](references/self-reflection-quality-gate.md) | 1–10 scoring rubric with ≥8 threshold per category |
|
||||
| Self-Learning System | [self-learning-system.md](references/self-learning-system.md) | Lesson/Memory templates and governance rules |
|
||||
| Component | Reference File | Purpose |
|
||||
| ---------------------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
|
||||
| Clarification Protocol | [references/clarification-protocol.md](references/clarification-protocol.md) | Ask a maximum of 2 targeted questions when scope is not clear |
|
||||
| Anti-Rationalization Guard | [references/anti-rationalization-guard.md](references/anti-rationalization-guard.md) | Prohibited rationalizations and mandatory responses for findings |
|
||||
| Self-Critique Loop | [references/self-critique-loop.md](references/self-critique-loop.md) | Mandatory second pass to verify evidence and coverage |
|
||||
| Retry Protocol | [references/retry-protocol.md](references/retry-protocol.md) | Tool failure and empty search handling: retry once, then document |
|
||||
| Non-Negotiable Behaviors | [references/non-negotiable-behaviors.md](references/non-negotiable-behaviors.md) | Mandatory rules: do not fabricate, cite evidence, report gaps |
|
||||
| Self-Reflection Quality Gate | [references/self-reflection-quality-gate.md](references/self-reflection-quality-gate.md) | Scoring rubric (1–10 scale) with a minimum threshold of 8 |
|
||||
| Self-Learning System | [references/self-learning-system.md](references/self-learning-system.md) | Templates and rules for lessons and security memories |
|
||||
|
||||
## Execution Flow
|
||||
|
||||
1. **Before analysis**: Apply Clarification Protocol if scope is ambiguous
|
||||
2. **During analysis**: Apply Anti-Rationalization Guard at every decision point
|
||||
3. **After initial pass**: Execute Self-Critique Loop (mandatory second pass)
|
||||
4. **On tool failure**: Apply Retry Protocol
|
||||
5. **Before delivery**: Run Self-Reflection Quality Gate (all categories must score ≥8)
|
||||
6. **After delivery**: Create Lessons/Memories for novel findings, false positives, or methodology gaps (see Self-Learning System)
|
||||
Load reference files as you reach each phase:
|
||||
|
||||
## Agent-Specific Adaptation
|
||||
|
||||
Each agent customizes the **Self-Critique Loop** checklist and **Self-Reflection Quality Gate** categories to match its domain. The reference files provide the base templates; agents extend them with domain-specific items.
|
||||
|
||||
### Example extensions per agent type
|
||||
- **SAST/SCA agents**: Add taint trace completeness and manifest coverage checks
|
||||
- **SonarQube-style agents**: Add rating sanity check (A–E consistency with findings)
|
||||
- **Threat modeling agents**: Add STRIDE category completeness per trust boundary
|
||||
- **Code review agents**: Add trust boundary audit with data flow tracing
|
||||
1. **Before analysis**: If scope or policy is not clear, read [references/clarification-protocol.md](references/clarification-protocol.md). Ask a maximum of 2 targeted questions.
|
||||
2. **During analysis**: Follow [references/non-negotiable-behaviors.md](references/non-negotiable-behaviors.md). At each triage decision, consult [references/anti-rationalization-guard.md](references/anti-rationalization-guard.md).
|
||||
3. **On tool or search failure**: Follow [references/retry-protocol.md](references/retry-protocol.md). Do not assume code is secure without verification.
|
||||
4. **After initial analysis**: Complete a second pass with the checklist in [references/self-critique-loop.md](references/self-critique-loop.md). Verify taint traces and manifest coverage.
|
||||
5. **Before delivery**: Score the draft report using [references/self-reflection-quality-gate.md](references/self-reflection-quality-gate.md). All categories must score ≥ 8.
|
||||
6. **Final delivery**: Record lessons and memories using [references/self-learning-system.md](references/self-learning-system.md).
|
||||
|
||||
@@ -1,38 +1,31 @@
|
||||
# Anti-Rationalization Guard
|
||||
|
||||
These rationalizations are **never** valid justifications for skipping, omitting, or downgrading findings:
|
||||
These rationalizations are **never** valid reasons to skip, omit, or downgrade findings:
|
||||
|
||||
## Universal Rationalizations (All Agents)
|
||||
## Universal Rationalizations
|
||||
|
||||
| If you think... | Mandatory response |
|
||||
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
|
||||
| "No issues/threats found on first pass" | Systematic evaluation across all categories is required before concluding clean. Expand scope and complete the full matrix. |
|
||||
| "This looks fine, skip deep analysis" | "Looks fine" is not evidence. Evidence = code trace, architecture reference, or rule match. Run checks. |
|
||||
| "The risk is probably lower in practice" | Risk level is based on impact × likelihood (CVSS/exploitability). Justify any downgrade with explicit evidence. |
|
||||
| "This is a false positive" | Flag it as a potential false positive but include it — do not silently suppress. Document the rationale for human review. |
|
||||
| "This is outside scope" | State explicitly why, with a reference to the declared scope or assessment boundary. |
|
||||
| "No controls/mitigations needed here" | State "No gap identified — rationale: [X]" explicitly. Silence is not assurance. |
|
||||
| If you think... | Mandatory response |
|
||||
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
|
||||
| "No issues or threats found on first pass" | Systematic evaluation across all categories is required before you conclude clean. Expand scope and complete the full matrix. |
|
||||
| "This looks fine, skip deep analysis" | "Looks fine" is not evidence. Evidence is a code trace, architecture reference, or rule match. Run checks. |
|
||||
| "The risk is probably lower in practice" | Risk level is based on impact multiplied by likelihood. Justify any downgrade with explicit evidence. |
|
||||
| "This is a false positive" | Flag the finding as a potential false positive, but include it. Do not silently suppress findings. Document the rationale. |
|
||||
| "This is outside scope" | State explicitly why, with a reference to the declared scope or assessment boundary. |
|
||||
| "No controls or mitigations needed here" | State "No gap identified — rationale: [X]" explicitly. Silence is not assurance. |
|
||||
|
||||
## SAST/SCA-Specific
|
||||
## SAST/SCA
|
||||
|
||||
| If you think... | Mandatory response |
|
||||
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
||||
| "SCA CVE isn't exploitable here" | Include the CVE with a documented context note — do not silently suppress. |
|
||||
| "This phase can be skipped" | All phases are mandatory. Document any phase that genuinely cannot be completed due to missing inputs. |
|
||||
| "Severity should be lower given context" | Severity is based on CVSS/exploitability. Justify any downgrade with explicit evidence. Document, don't suppress. |
|
||||
| If you think... | Mandatory response |
|
||||
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| "SCA CVE is not exploitable here" | Include the CVE with a documented context note. Do not silently suppress. |
|
||||
| "This phase can be skipped" | All phases are mandatory. Document any phase that you cannot complete due to missing inputs. |
|
||||
| "Severity should be lower given context" | Severity is based on CVSS or exploitability. Justify any downgrade with explicit evidence. Do not suppress. |
|
||||
|
||||
## Code Quality-Specific
|
||||
## Threat Modeling
|
||||
|
||||
| If you think... | Mandatory response |
|
||||
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
|
||||
| "The team will refactor this later" | Technical debt still counts toward the debt ratio today. Document it accurately. |
|
||||
| "Quality Gate failure is a false positive" | Include it as a finding, document the suspected false positive rationale, and mark for human review. |
|
||||
|
||||
## Threat Modeling-Specific
|
||||
|
||||
| If you think... | Mandatory response |
|
||||
| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| "This threat is mitigated by the architecture" | Document the specific compensating control and verify it is actually implemented — do not assume. |
|
||||
| "This category has no applicable threats here" | State "No applicable threats identified — rationale: [X]" explicitly. Do not silently omit. |
|
||||
| "Lateral movement is unlikely here" | Document the specific architectural control that prevents pivoting and verify it is implemented — do not assume. |
|
||||
| "This threat actor wouldn't target this" | Document the basis for that exclusion. Insider threats and supply chain actors must always be considered. |
|
||||
| If you think... | Mandatory response |
|
||||
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
|
||||
| "This threat is mitigated by the architecture" | Document the specific compensating control and verify that it is implemented. Do not assume. |
|
||||
| "This category has no applicable threats here" | State "No applicable threats identified — rationale: [X]" explicitly. Do not silently omit categories. |
|
||||
| "Lateral movement is unlikely here" | Document the specific architectural control that prevents pivoting and verify that it is implemented. |
|
||||
| "This threat actor would not target this" | Document the basis for that exclusion. You must always consider insider threats and supply chain actors. |
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
# Clarification Protocol
|
||||
|
||||
Before beginning analysis, pause and ask the user at most **2 targeted questions** when:
|
||||
Before you start the analysis, pause and ask the user a maximum of **2 targeted questions** when:
|
||||
|
||||
- The system scope, asset boundary, or target module is ambiguous and cannot be inferred from the provided context
|
||||
- A critical trust boundary, privilege tier, or authentication zone is undefined and the analysis would significantly change depending on the interpretation
|
||||
- The business context required for impact prioritization or compliance framework selection is entirely absent
|
||||
- The language or framework cannot be auto-detected from the workspace
|
||||
- The system scope, asset boundary, or target module is ambiguous and you cannot infer it from the context
|
||||
- A critical trust boundary or privilege tier is undefined
|
||||
- The business context for impact prioritization or compliance framework selection is absent
|
||||
- The programming language or framework cannot be detected from the workspace
|
||||
|
||||
**Rules:**
|
||||
|
||||
1. State your working assumptions explicitly, then proceed
|
||||
2. Do not wait for confirmation unless the ambiguity would fundamentally alter the attack surface definition, trust boundary map, or which phases are executed
|
||||
3. Maximum 2 questions — if more ambiguity exists, infer from available evidence and document assumptions
|
||||
4. If no ambiguity exists, proceed directly without questions
|
||||
1. State your working assumptions explicitly and proceed.
|
||||
2. Do not wait for confirmation unless the ambiguity alters the attack surface definition or trust boundary map.
|
||||
3. Ask a maximum of 2 questions. If more ambiguity exists, document your assumptions and proceed.
|
||||
4. If no ambiguity exists, proceed directly without questions.
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
# Non-Negotiable Behaviors
|
||||
|
||||
These rules apply to **all** AppSec agents with no exceptions:
|
||||
These rules apply to **all** security agents with no exceptions:
|
||||
|
||||
1. **Never fabricate findings**: Do not report vulnerabilities, threats, bugs, code smells, or risk assessments without direct evidence from the analyzed source code, architecture, manifests, or threat intelligence.
|
||||
1. **Never fabricate findings**: Do not report vulnerabilities, threats, bugs, or risk assessments without direct evidence from the scanned code, architecture, or manifests.
|
||||
|
||||
2. **Always cite evidence**: Every finding must reference a specific file path, line number, CVE ID, component, trust boundary, data flow, or rule key. Generic findings without precise traceability are prohibited.
|
||||
2. **Always cite evidence**: Every finding must reference a specific file path, line number, CVE ID, component, or trust boundary. Do not output generic findings.
|
||||
|
||||
3. **Explain rationale for risk decisions**: When assigning severity, risk levels, quality ratings, policy compliance verdicts, or composite risk scores, state the reasoning based on exploitability, impact, and evidence — do not rely on unexplained judgment.
|
||||
3. **Explain rationale for risk decisions**: When you assign severity or policy compliance verdicts, state the reasoning based on exploitability and impact. Do not rely on unexplained judgment.
|
||||
|
||||
4. **Do not modify source files**: Do not alter code, configuration, dependency files, or deployment manifests unless explicitly requested by the user.
|
||||
4. **Do not modify source files**: Do not alter code, configuration, dependency files, or manifests unless the user explicitly requests changes.
|
||||
|
||||
5. **Report honestly on coverage gaps**: If any analysis phase, STRIDE category, scan type, or methodology step could not be completed (missing files, unsupported language, inaccessible components), state it explicitly rather than silently omitting.
|
||||
5. **Report honestly on coverage gaps**: If any phase cannot be completed due to missing files or unsupported languages, state it explicitly. Do not omit phases.
|
||||
|
||||
6. **Complete all phases**: Partial runs are not acceptable. If a phase is blocked, document why and continue with remaining phases.
|
||||
6. **Complete all phases**: Execute all analysis phases. If a phase is blocked, document the reason and continue with the remaining phases.
|
||||
|
||||
7. **Provide progress summaries**: For multi-phase analysis, summarize findings after completing each major phase before proceeding to the next.
|
||||
7. **Provide progress summaries**: For multi-phase scans, summarize findings after completing each major phase before you proceed.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# Retry Protocol
|
||||
|
||||
On tool failure or empty results:
|
||||
On tool failure or empty search results:
|
||||
|
||||
1. **Retry once** with a refined query or a different search pattern.
|
||||
2. **If second attempt fails**, state the failure explicitly and continue with available evidence.
|
||||
3. **Never silently skip** a phase because a tool call returned no results — distinguish "tool found nothing" from "tool failed to execute."
|
||||
4. **Document the gap**: If a phase is genuinely blocked (missing manifests, unsupported language, inaccessible files), state it explicitly in the output rather than silently omitting the phase.
|
||||
1. **Retry once** with an alternative query or a different search pattern.
|
||||
2. **If the second attempt fails**, state the failure explicitly and continue with available evidence.
|
||||
3. **Do not skip phases**: Distinguish between "tool found no results" and "tool execution failed". Never assume the code is clean without verification.
|
||||
4. **Document the gap**: If a phase is blocked due to missing manifests or unsupported languages, state it in the output. Do not omit the phase.
|
||||
|
||||
@@ -1,46 +1,24 @@
|
||||
# Self-Critique Loop
|
||||
|
||||
After completing the initial analysis, perform a **mandatory second pass** before delivering output.
|
||||
After you complete the initial analysis, complete a **mandatory second pass** before you deliver the report.
|
||||
|
||||
## Universal Checks (All Agents)
|
||||
## Universal Checks
|
||||
|
||||
1. **Evidence check**: Every finding must cite a concrete reference (file:line, component, architecture element, CVE ID, rule key). Remove any finding without supporting evidence.
|
||||
2. **Coverage check**: Verify that all categories, phases, or scan types relevant to the agent's methodology were explicitly evaluated. State "None detected" for each clean category rather than silently omitting.
|
||||
3. **Mitigation/remediation check**: Every Critical and High finding must have a specific, implementable fix — not a generic recommendation.
|
||||
1. **Evidence check**: Every finding must cite a verified reference (`file:line`, component, architecture element, or CVE ID). Remove any finding that does not have supporting evidence.
|
||||
2. **Coverage check**: Verify that you evaluated all required categories and phases. State "None detected" for clean categories. Do not omit clean categories.
|
||||
3. **Remediation check**: Provide a specific, implementable fix for every Critical and High finding. Do not provide generic recommendations.
|
||||
|
||||
## Domain-Specific Extensions
|
||||
|
||||
Each agent adds domain checks to the universal list above:
|
||||
|
||||
### STRIDE Threat Modeling
|
||||
|
||||
4. **STRIDE completeness**: Did you evaluate all six STRIDE categories (S/T/R/I/D/E) for every trust boundary and data flow?
|
||||
5. **Trust boundary audit**: Re-verify that every identified trust boundary has at least one evaluated data flow crossing it.
|
||||
|
||||
### STRIDE-LM (Lateral Movement)
|
||||
|
||||
4. **STRIDE-LM completeness**: Did you evaluate all seven categories (S/T/R/I/D/E/LM) for every asset and trust boundary?
|
||||
5. **Control coverage**: Every Critical/High threat maps to a control function (Inventory/Collect/Detect/Protect/Manage/Respond).
|
||||
6. **Lateral movement audit**: Re-trace all identified pivot paths. Verify no uncontrolled path exists from compromised entry point to high-value asset.
|
||||
|
||||
### Code Review Threat Modeling
|
||||
|
||||
4. **STRIDE completeness**: All six STRIDE categories evaluated for every trust boundary and data flow.
|
||||
5. **Trust boundary audit**: Every trust boundary has evaluated data flows crossing it.
|
||||
|
||||
### Code Quality (SonarQube-style)
|
||||
|
||||
4. **Issue type coverage**: All five issue types (Bug, Vulnerability, Hotspot, Smell, Duplication) explicitly evaluated.
|
||||
5. **Rating sanity check**: A–E ratings are consistent with finding counts before finalizing Quality Gate verdict.
|
||||
## Domain Checks
|
||||
|
||||
### SAST/SCA
|
||||
|
||||
4. **Taint trace completeness**: Every entry point identified in discovery was taint-traced through to sinks.
|
||||
5. **Manifest coverage**: All dependency manifests identified in discovery were audited.
|
||||
1. **Taint trace completeness**: Verify that you traced each discovered entry point from source to sink.
|
||||
2. **Manifest coverage**: Verify that you audited all discovered dependency manifests.
|
||||
3. **Evidence completeness**: Cite a verified `file:line` reference and taint trace for each SAST finding. Cite a verified CVE ID and affected version range for each SCA finding.
|
||||
4. **Flaw category completeness**: Confirm that you evaluated all flaw categories. State "No instances detected" for clean categories. Do not omit clean categories.
|
||||
5. **Policy consistency**: Verify that the PASS or FAIL policy verdict matches severity counts and policy threshold rules.
|
||||
|
||||
### Multi-tool Pipeline
|
||||
### Threat Modeling
|
||||
|
||||
4. **Phase coverage**: All deliverable files generated and saved.
|
||||
5. **Cross-correlation**: SAST findings corroborated by SCA findings → elevate corroborated items.
|
||||
6. **Deduplication**: Same finding doesn't appear under multiple tool outputs.
|
||||
7. **Roadmap completeness**: Every Critical/High finding appears in the immediate remediation tier.
|
||||
1. **STRIDE completeness**: Verify that you evaluated all six STRIDE categories (S/T/R/I/D/E) for each trust boundary and data flow.
|
||||
2. **Trust boundary audit**: Verify that each identified trust boundary has at least one evaluated data flow crossing it.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Self-Learning System
|
||||
|
||||
Maintain project learning artifacts under a designated lessons/memories directory (e.g., `.github/SecurityLessons` and `.github/SecurityMemories`).
|
||||
Maintain project learning artifacts in `.github/SecurityLessons/` and `.github/SecurityMemories/`.
|
||||
|
||||
## When to Create
|
||||
|
||||
|
||||
@@ -1,46 +1,43 @@
|
||||
# Self-Reflection Quality Gate
|
||||
|
||||
After completing analysis, internally score the output across domain-relevant categories (1–10 scale).
|
||||
After you complete the analysis, score the output across domain categories on a 1–10 scale.
|
||||
|
||||
## Scoring Rules
|
||||
**Threshold**: Every category must score **≥ 8** to pass. If any category scores < 8, resolve the gaps before you deliver the report (maximum 2 rework iterations).
|
||||
|
||||
- **Pass**: All categories ≥ 8
|
||||
- **Fail**: Any score < 8 → revisit the failing dimension before delivering output. Max 2 rework iterations.
|
||||
- **If unresolvable after 2 iterations**: Deliver output with an explicit confidence note stating which dimension fell short and why.
|
||||
## Core Scoring Rubric (Universal)
|
||||
|
||||
## Base Categories (All Agents)
|
||||
| Category | Score 9–10 | Score 7–8 | Score 5–6 | Score 1–4 |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------- | --------------------------------------------------------------- |
|
||||
| **Completeness** | All phases, categories, boundaries, and checklists evaluated with explicit evidence | Minor omissions that do not affect the risk profile | Noticeable gaps in coverage; entire sections omitted | Superficial pass; major sections missing |
|
||||
| **Accuracy** | Precise file:line references, confirmed CVE IDs, verified taint flows, correct CWE mappings | Minor version ambiguity or indirect references; core findings correct | Plausible findings with unverified details or questionable mappings | Fabricated references, wrong CWE mappings, or hallucinated CVEs |
|
||||
| **Actionability** | Concrete remediation code, exact package upgrade targets, step-by-step guidance | Clear direction but requires human adjustment | High-level advice without code or version targets | Generic platitudes ("sanitize inputs", "keep packages updated") |
|
||||
| **Consistency** | Risk ratings match evidence, framework verdicts align with findings, formatting uniform | Minor formatting or taxonomy inconsistencies | Severity ratings conflict with impact descriptions | Contradictory findings or incompatible verdicts |
|
||||
| **Evidence Rigor** | Every claim backed by code, manifest, trace, or threat model element | Most findings backed; some secondary claims lack citations | Assertions made without code citations | Speculative findings with zero code or manifest backing |
|
||||
|
||||
| Category | Question | Threshold |
|
||||
| ----------------- | --------------------------------------------------------------------------------------- | :-------: |
|
||||
| **Completeness** | Were all required phases/categories evaluated with evidence? | ≥ 8 |
|
||||
| **Accuracy** | Are findings backed by concrete references (code, architecture, CVEs), not speculation? | ≥ 8 |
|
||||
| **Actionability** | Does every Critical/High finding have a specific, implementable fix or mitigation? | ≥ 8 |
|
||||
| **Consistency** | Are severity ratings, mappings, and verdicts internally consistent? | ≥ 8 |
|
||||
| **Coverage** | Were all entry points, trust boundaries, modules, or manifests identified and analyzed? | ≥ 8 |
|
||||
## Scoring Threshold
|
||||
|
||||
## Domain-Specific Extensions
|
||||
| Final Score | Action |
|
||||
| ---------------------------- | ------------------------------------------------------------------------- |
|
||||
| All categories ≥ 8 | **PASS** — Proceed to final delivery |
|
||||
| Any category < 8 (attempt 1) | **REWORK** — Address deficiencies and re-score |
|
||||
| Any category < 8 (attempt 2) | **REWORK** — Final attempt to resolve gaps |
|
||||
| Any category < 8 (attempt 3) | **FLAG & DELIVER** — Deliver with explicit quality gap disclosure to user |
|
||||
|
||||
### Multi-tool Pipeline — add:
|
||||
## Domain-Specific Rubrics
|
||||
|
||||
| **Deduplication** | Are cross-tool duplicates properly merged with corroboration notes? | ≥ 8 |
|
||||
### SAST/SCA
|
||||
|
||||
### Code Quality (SonarQube-style) — adapt Completeness to:
|
||||
| Category | Question | Threshold |
|
||||
| ----------------- | -------------------------------------------------------------------------------------- | :-------: |
|
||||
| **Completeness** | Did you evaluate all SAST flaw categories and SCA ecosystems? | ≥ 8 |
|
||||
| **Accuracy** | Are findings supported by concrete taint traces and verified CVE IDs? | ≥ 8 |
|
||||
| **Actionability** | Does each Critical and High finding have a concrete code fix or upgrade step? | ≥ 8 |
|
||||
| **Consistency** | Are severity ratings, CWE mappings, and policy verdicts consistent across the report? | ≥ 8 |
|
||||
| **Coverage** | Did you trace all entry points from source to sink and audit all dependency manifests? | ≥ 8 |
|
||||
|
||||
| **Completeness** | Were all issue types (Bugs, Vulnerabilities, Hotspots, Smells, Duplication) evaluated? | ≥ 8 |
|
||||
### Threat Modeling and Code Review
|
||||
|
||||
### SAST/SCA — adapt Coverage to:
|
||||
|
||||
| **Coverage** | Were all entry points taint-traced and all dependency manifests audited? | ≥ 8 |
|
||||
|
||||
### STRIDE Threat Modeling — adapt Completeness to:
|
||||
|
||||
| **Completeness** | Were all six STRIDE categories evaluated for every trust boundary and data flow? | ≥ 8 |
|
||||
|
||||
### STRIDE-LM — adapt Completeness and Coverage to:
|
||||
|
||||
| **Completeness** | Were all seven STRIDE-LM categories evaluated for every asset and trust boundary? | ≥ 8 |
|
||||
| **Coverage** | Were all lateral movement paths, trust boundaries, and post-exploitation chains assessed? | ≥ 8 |
|
||||
|
||||
### Code Review — adapt Coverage to:
|
||||
|
||||
| **Coverage** | Were all entry points, trust boundaries, and data flows traced from source to sink? | ≥ 8 |
|
||||
| Category | Question | Threshold |
|
||||
| ---------------- | ----------------------------------------------------------------------------------- | :-------: |
|
||||
| **Completeness** | Did you evaluate all six STRIDE categories for each trust boundary and data flow? | ≥ 8 |
|
||||
| **Coverage** | Were all entry points, trust boundaries, and data flows traced from source to sink? | ≥ 8 |
|
||||
|
||||
Reference in New Issue
Block a user