diff --git a/docs/README.skills.md b/docs/README.skills.md index 16f19e1fb..a81570457 100644 --- a/docs/README.skills.md +++ b/docs/README.skills.md @@ -84,6 +84,7 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to | [azure-static-web-apps](../skills/azure-static-web-apps/SKILL.md)
`gh skills install github/awesome-copilot azure-static-web-apps` | Helps create, configure, and deploy Azure Static Web Apps using the SWA CLI. Use when deploying static sites to Azure, setting up SWA local development, configuring staticwebapp.config.json, adding Azure Functions APIs to SWA, or setting up GitHub Actions CI/CD for Static Web Apps. | None | | [azure-well-architected-review](../skills/azure-well-architected-review/SKILL.md)
`gh skills install github/awesome-copilot azure-well-architected-review` | Perform an Azure Well-Architected Framework review of the current workload IaC and architecture, generating findings and GitHub issues for improvements. | None | | [batch-files](../skills/batch-files/SKILL.md)
`gh skills install github/awesome-copilot batch-files` | Expert-level Windows batch file (.bat/.cmd) skill for writing, debugging, and maintaining CMD scripts. Use when asked to "create a batch file", "write a .bat script", "automate a Windows task", "CMD scripting", "batch automation", "scheduled task script", "Windows shell script", or when working with .bat/.cmd files in the workspace. Covers cmd.exe syntax, environment variables, control flow, string processing, error handling, and integration with system tools. | `assets/executable.txt`
`assets/library.txt`
`assets/task.txt`
`references/batch-files-and-functions.md`
`references/cygwin.md`
`references/msys2.md`
`references/tools-and-resources.md`
`references/windows-commands.md`
`references/windows-subsystem-on-linux.md` | +| [batch-script-progress](../skills/batch-script-progress/SKILL.md)
`gh skills install github/awesome-copilot batch-script-progress` | Design and verify runtime output when creating or modifying Python, PowerShell, or Node.js scripts that process multiple files, records, or API requests. Use for batch scripts, bulk operations, silent long-running loops, progress reporting, per-item failures, final counts, stdout separation, or exit-status consistency. Not for UI progress bars, monitoring infrastructure, or progress updates in the AI conversation. | `examples/batch.mjs`
`examples/batch.ps1`
`examples/batch.py`
`references/language-notes.md`
`references/verification.md`
`tests/verify-examples.mjs` | | [bench-read](../skills/bench-read/SKILL.md)
`gh skills install github/awesome-copilot bench-read` | Read artifacts from the shared bench — the workspace where desks leave findings, verdicts, and work products for each other and the operator. | None | | [bigquery-pipeline-audit](../skills/bigquery-pipeline-audit/SKILL.md)
`gh skills install github/awesome-copilot bigquery-pipeline-audit` | Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations. | None | | [boost-prompt](../skills/boost-prompt/SKILL.md)
`gh skills install github/awesome-copilot boost-prompt` | Interactive prompt refinement workflow: interrogates scope, deliverables, constraints; copies final markdown to clipboard; never writes code. Requires the Joyride extension. | None | diff --git a/skills/batch-script-progress/SKILL.md b/skills/batch-script-progress/SKILL.md new file mode 100644 index 000000000..75d01bc1d --- /dev/null +++ b/skills/batch-script-progress/SKILL.md @@ -0,0 +1,78 @@ +--- +name: batch-script-progress +description: 'Design and verify runtime output when creating or modifying Python, PowerShell, or Node.js scripts that process multiple files, records, or API requests. Use for batch scripts, bulk operations, silent long-running loops, progress reporting, per-item failures, final counts, stdout separation, or exit-status consistency. Not for UI progress bars, monitoring infrastructure, or progress updates in the AI conversation.' +--- + +# Batch Script Progress + +Make generated batch scripts observable while they run, without changing their processing contract. Design start, periodic progress, per-item failure, and final-summary output together with result streams and exit status. A progress-bar library alone does not establish this contract. + +## 1. Fix the existing contract + +Read the script, callers, and nearby tests before editing. Identify: + +- Input unit, known or unknown total, and whether enumeration is lazy. +- Existing stdout format and consumers, runtime version, and interactive versus redirected execution. +- Continue-on-error or fail-fast policy, retry rules, skip rules, transaction boundaries, and cancellation behavior. +- Existing exit codes and what callers treat as success. + +Preserve these decisions. Do not add retries, parallelism, dependencies, or a new failure policy just to add progress output. If partial failures currently return zero, report the partial outcome honestly while keeping that exit contract. If the contract is missing or contradictory, ask for the decision that changes behavior; do not silently make it. + +## 2. Define the output contract before implementation + +Choose a small, stable vocabulary and fields, reusing the repository logger where suitable: + +- **Start:** operation name, total or explicit unknown, and initial counters. Emit before expensive enumeration; discovery itself may need progress. +- **Progress:** processed, succeeded, failed, skipped, total or unknown, and elapsed time. Add a sanitized current-item identifier only when useful. +- **Item failure:** safe identifier, failure category or sanitized reason, and whether processing will continue. Emit when observed, independently of the normal progress interval. +- **Summary:** completed, partial, failed, or cancelled; final counters; elapsed time; known unprocessed count if available; and the exit status required by the existing contract. + +Define `processed = succeeded + failed + skipped` for items that reached a terminal outcome. Keep in-flight and not-started items separate. Count each item once; retry attempts are not additional processed items. Do not label an item successful before its required commit, write, or acknowledgement. If a transaction rolls back, report rolled-back work rather than claiming durable success. + +For an unknown total, report counts without a fabricated percentage, ETA, or remaining count. Do not materialize a stream just to count it. For zero items, emit start and summary with zero counts; never divide by zero. + +## 3. Choose output streams and cadence + +Reserve stdout for the established machine-readable result, pipeline objects, or protocol. Put progress and diagnostics on a separate channel; stderr is the portable default for standalone CLI scripts. Stderr output alone does not imply failure: callers must use the agreed exit code and result contract. Verify callers that treat any stderr as failure before changing their output. If stdout is reserved and stderr is incompatible, use an existing compatible logger or sink only when the caller already supports it. Do not move progress to stdout or another stream with changed semantics. If no compatible channel exists, stop and ask whether the output contract may change, or report that progress cannot be added safely. + +Use a bounded cadence, such as every configured number of items **or** elapsed interval, rather than unconditionally logging every item. Emit start, failures, and summary regardless of that cadence. Preserve required per-item audit records separately. Avoid unbounded in-memory failure lists and repeated stack traces. Redact credentials, query parameters, personal data, and sensitive paths before formatting; do not dump raw exceptions or whole input records by default. + +Use monotonic time for intervals and duration. Ensure output is flushed or drained so notifications arrive before exit. Use plain newline-delimited records under redirection; enable terminal rewriting or colors only for an appropriate TTY. Keep log-field meanings consistent across languages, not necessarily their exact formatting or libraries. + +**A time check at the end of a loop cannot emit while one item is blocking.** For a long item, use existing progress callbacks, stage notifications, or a safely integrated heartbeat where the runtime permits it. A Node.js timer cannot run through synchronous event-loop blocking; a PowerShell progress display is not a durable log. If mid-item progress cannot be added without changing execution semantics, emit the current stage before it starts and disclose the notification gap. Do not promise a periodic heartbeat that the implementation cannot deliver. + +## 4. Implement the smallest compatible change + +Read [language notes and safe examples](references/language-notes.md) for the selected runtime. The examples are executable output-contract probes, not production processors; adapt their emitter and accounting to the existing code rather than copying their failure policy. + +Keep reporting separate from processing decisions. In concurrent code, serialize reporting through the coordinator and count completed outcomes, not submissions. On handled failure or cancellation, attempt one final summary and preserve the original outcome; a summary-write error must not replace the original processing exception. Do not swallow cancellation as an ordinary item failure. For host-managed cancellation, including PowerShell host stops, preserve the existing host behavior; do not convert stop exceptions into item failures or promise a final summary after host termination. Use `finally` for applicable cleanup, but verify the actual cancellation path in the supported host when graceful cancellation is part of the contract. + +Do not force successful termination to deliver buffered output. In Node.js prefer setting `process.exitCode` and draining writes over an immediate `process.exit()`. Respect existing host/library semantics in PowerShell: a reusable function should not exit its caller. Forced termination, process crashes, and broken output pipes can prevent final reporting; do not claim otherwise. + +## 5. Verify before reporting completion + +Read [verification cases](references/verification.md), then run focused tests for the changed script with harmless fixtures. Capture stdout and stderr separately and check the real exit status. Verify start/progress ordering, monotonic counters, sanitized failures, exactly one summary on supported paths, and unchanged result contents and processing policy. + +Prove liveness by reading the child process output **while it is still running**, not by examining output collected after exit. Check both terminal and redirected behavior when both are supported. A controlled pause after a progress event is a useful deterministic probe: receive the event, confirm the child has not exited, then release it. Keep such gates in harmless test fixtures or the test harness; do not add stdin waits, sleeps, or cancellation hooks to normal production processing solely to make a test pass. Prefer an existing callback or injectable processing boundary. If no safe deterministic seam exists, test the real path only with harmless inputs or report liveness as unverified. A watchdog may fail the test; it must not manufacture a successful result. + +The bundled [cross-language verifier](tests/verify-examples.mjs) runs the [Python](examples/batch.py), [PowerShell](examples/batch.ps1), and [Node.js](examples/batch.mjs) probes with normal, partial-failure, preserved partial-failure exit-zero, empty, unknown-total, fail-fast, simulated-cancellation, and live-output cases. It checks the probes only; it does not prove that an arbitrary generated script or automatic skill selection works. + +Run it from this skill folder with Node.js 20 or later: + +```text +node tests/verify-examples.mjs +``` + +It also requires Python 3.9 or later and PowerShell 5.1 or later. See the language notes for executable overrides. Missing runtimes are failures, not silently skipped coverage. + +## Deliverables + +Return the changed script, its output/exit contract, and focused verification results. Distinguish executed tests from inspection and unexecuted cases. Do not call the script verified merely because it prints a progress bar or the skill metadata passes validation. + +## Gotchas + +- Counting before logging can make discovery silently consume an entire stream. +- Writing status with PowerShell `Write-Output` adds strings to the success pipeline. +- Measuring only buffered output after exit cannot prove progress visibility. +- A final message saying `completed` hides partial failure unless the outcome and counters explain it. +- `finally` is not a guarantee of reporting after forced termination. diff --git a/skills/batch-script-progress/examples/batch.mjs b/skills/batch-script-progress/examples/batch.mjs new file mode 100644 index 000000000..8f5f0d0da --- /dev/null +++ b/skills/batch-script-progress/examples/batch.mjs @@ -0,0 +1,69 @@ +import { once } from 'node:events'; +import { createInterface } from 'node:readline'; + +const scenario = process.argv[2] ?? 'normal'; +if (!['normal', 'mixed', 'partial-zero', 'empty', 'unknown', 'fail-fast', 'cancelled', 'live'].includes(scenario)) { + throw new Error('Unknown fixture scenario'); +} +const count = scenario === 'empty' ? 0 : scenario === 'mixed' ? 4 : 3; +const total = scenario === 'unknown' ? null : count; +const stats = { processed: 0, succeeded: 0, failed: 0, skipped: 0 }; +const started = performance.now(); +let lastReport = started; +let status = 'completed'; +const results = []; + +async function writeRecord(stream, record) { + if (!stream.write(`${JSON.stringify(record)}\n`)) await once(stream, 'drain'); +} + +async function emit(event, fields = {}) { + await writeRecord(process.stderr, { event, total, ...stats, elapsed_ms: Math.round(performance.now() - started), ...fields }); +} + +await emit('start'); +for (let index = 1; index <= count; index++) { + if (scenario === 'cancelled' && index === 2) { + status = 'cancelled'; + break; + } + if (scenario === 'mixed' && index === 2) { + stats.skipped++; + } else { + try { + if ((scenario === 'mixed' && index === 3) || (['fail-fast', 'partial-zero'].includes(scenario) && index === 2)) { + throw new Error('fixture-rejected'); + } + results.push({ id: `item-${index}`, value: index * 2 }); + stats.succeeded++; + } catch { + stats.failed++; + stats.processed++; + await emit('item-failed', { item_id: `item-${index}`, reason: 'fixture-rejected', will_continue: scenario !== 'fail-fast' }); + if (scenario === 'fail-fast') { + status = 'failed'; + break; + } + stats.processed--; + } + } + stats.processed++; + const now = performance.now(); + if (stats.processed % 2 === 0 || now - lastReport >= 1000 || scenario === 'live') { + if (scenario === 'live' && index === 1) { + const input = createInterface({ input: process.stdin }); + const released = once(input, 'line'); + await emit('progress'); + await released; + input.close(); + } else { + await emit('progress'); + } + lastReport = now; + } +} +if (status === 'completed' && stats.failed) status = 'partial'; +const exitCode = status === 'cancelled' ? 130 : stats.failed && scenario !== 'partial-zero' ? 1 : 0; +await emit('summary', { status, exit_code: exitCode, unprocessed: total === null ? null : total - stats.processed }); +if (status !== 'cancelled') await writeRecord(process.stdout, { results }); +process.exitCode = exitCode; diff --git a/skills/batch-script-progress/examples/batch.ps1 b/skills/batch-script-progress/examples/batch.ps1 new file mode 100644 index 000000000..6d3a30681 --- /dev/null +++ b/skills/batch-script-progress/examples/batch.ps1 @@ -0,0 +1,75 @@ +param( + [ValidateSet('normal', 'mixed', 'partial-zero', 'empty', 'unknown', 'fail-fast', 'cancelled', 'live')] + [string]$Scenario = 'normal' +) + +$count = 3 +if ($Scenario -eq 'empty') { $count = 0 } +if ($Scenario -eq 'mixed') { $count = 4 } +$total = $count +if ($Scenario -eq 'unknown') { $total = $null } +$stats = @{ processed = 0; succeeded = 0; failed = 0; skipped = 0 } +$clock = [System.Diagnostics.Stopwatch]::StartNew() +$lastReport = 0 +$results = [System.Collections.Generic.List[object]]::new() +$status = 'completed' + +function Write-EventRecord { + param([string]$EventName, [hashtable]$Fields = @{}) + $record = [ordered]@{ event = $EventName; total = $total } + foreach ($key in @('processed', 'succeeded', 'failed', 'skipped')) { + $record[$key] = $stats[$key] + } + $record['elapsed_ms'] = $clock.ElapsedMilliseconds + foreach ($key in $Fields.Keys) { $record[$key] = $Fields[$key] } + [Console]::Error.WriteLine(($record | ConvertTo-Json -Compress)) + [Console]::Error.Flush() +} + +Write-EventRecord 'start' +for ($index = 1; $index -le $count; $index++) { + if ($Scenario -eq 'cancelled' -and $index -eq 2) { + $status = 'cancelled' + break + } + if ($Scenario -eq 'mixed' -and $index -eq 2) { + $stats.skipped++ + } else { + try { + if (($Scenario -eq 'mixed' -and $index -eq 3) -or ($Scenario -in @('fail-fast', 'partial-zero') -and $index -eq 2)) { + throw 'fixture-rejected' + } + $results.Add(@{ id = "item-$index"; value = $index * 2 }) + $stats.succeeded++ + } catch { + $stats.failed++ + $stats.processed++ + Write-EventRecord 'item-failed' @{ item_id = "item-$index"; reason = 'fixture-rejected'; will_continue = ($Scenario -ne 'fail-fast') } + if ($Scenario -eq 'fail-fast') { + $status = 'failed' + break + } + $stats.processed-- + } + } + $stats.processed++ + if ($stats.processed % 2 -eq 0 -or $clock.ElapsedMilliseconds - $lastReport -ge 1000 -or $Scenario -eq 'live') { + Write-EventRecord 'progress' + $lastReport = $clock.ElapsedMilliseconds + } + if ($Scenario -eq 'live' -and $index -eq 1) { + if ($null -eq [Console]::In.ReadLine()) { throw 'Fixture release input missing' } + } +} +if ($status -eq 'completed' -and $stats.failed -gt 0) { $status = 'partial' } +$exitCode = 0 +if ($stats.failed -gt 0 -and $Scenario -ne 'partial-zero') { $exitCode = 1 } +if ($status -eq 'cancelled') { $exitCode = 130 } +$unprocessed = $null +if ($null -ne $total) { $unprocessed = $total - $stats.processed } +Write-EventRecord 'summary' @{ status = $status; exit_code = $exitCode; unprocessed = $unprocessed } +if ($status -ne 'cancelled') { + [Console]::Out.WriteLine((@{ results = @($results.ToArray()) } | ConvertTo-Json -Compress -Depth 4)) + [Console]::Out.Flush() +} +exit $exitCode diff --git a/skills/batch-script-progress/examples/batch.py b/skills/batch-script-progress/examples/batch.py new file mode 100644 index 000000000..2670c9431 --- /dev/null +++ b/skills/batch-script-progress/examples/batch.py @@ -0,0 +1,63 @@ +import json +import sys +import time + + +def main(): + scenario = sys.argv[1] if len(sys.argv) > 1 else "normal" + if scenario not in {"normal", "mixed", "partial-zero", "empty", "unknown", "fail-fast", "cancelled", "live"}: + raise ValueError("Unknown fixture scenario") + count = 0 if scenario == "empty" else 4 if scenario == "mixed" else 3 + total = None if scenario == "unknown" else count + stats = {"processed": 0, "succeeded": 0, "failed": 0, "skipped": 0} + started = time.monotonic() + last_report = started + results = [] + status = "completed" + + def emit(event, **fields): + record = {"event": event, "total": total, **stats, "elapsed_ms": round((time.monotonic() - started) * 1000), **fields} + print(json.dumps(record), file=sys.stderr, flush=True) + + emit("start") + try: + for index in range(1, count + 1): + if scenario == "cancelled" and index == 2: + raise KeyboardInterrupt() + if scenario == "mixed" and index == 2: + stats["skipped"] += 1 + else: + try: + if (scenario == "mixed" and index == 3) or (scenario in {"fail-fast", "partial-zero"} and index == 2): + raise ValueError("fixture-rejected") + results.append({"id": f"item-{index}", "value": index * 2}) + stats["succeeded"] += 1 + except ValueError: + stats["failed"] += 1 + stats["processed"] += 1 + emit("item-failed", item_id=f"item-{index}", reason="fixture-rejected", will_continue=scenario != "fail-fast") + if scenario == "fail-fast": + status = "failed" + break + stats["processed"] -= 1 + stats["processed"] += 1 + now = time.monotonic() + if stats["processed"] % 2 == 0 or now - last_report >= 1 or scenario == "live": + emit("progress") + last_report = now + if scenario == "live" and index == 1: + if not sys.stdin.readline(): + raise RuntimeError("Fixture release input missing") + except KeyboardInterrupt: + status = "cancelled" + if status == "completed" and stats["failed"]: + status = "partial" + exit_code = 130 if status == "cancelled" else 1 if stats["failed"] and scenario != "partial-zero" else 0 + emit("summary", status=status, exit_code=exit_code, unprocessed=None if total is None else total - stats["processed"]) + if status != "cancelled": + print(json.dumps({"results": results}), flush=True) + return exit_code + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/batch-script-progress/references/language-notes.md b/skills/batch-script-progress/references/language-notes.md new file mode 100644 index 000000000..45fa6042b --- /dev/null +++ b/skills/batch-script-progress/references/language-notes.md @@ -0,0 +1,42 @@ +# Language Notes + +Reuse an existing logger or reporting utility first. These notes describe standalone CLI streams; hosted functions and services may need their own logging contract. + +## Python + +- Send diagnostic messages through a logger configured for stderr, or use `print(..., file=sys.stderr, flush=True)` for a small standalone script. Do not configure the root logger inside a reusable library. +- Use `time.monotonic()` for reporting intervals. `-u` or `PYTHONUNBUFFERED` can help with subprocess output, but verify the actual logger/stream path. +- Handle ordinary item exceptions according to the existing policy. Keep `KeyboardInterrupt` and other control-flow exceptions out of a broad item-error catch. +- Return the agreed code from `main` and use `raise SystemExit(main())` only at the executable entry point. + +The [Python probe](../examples/batch.py) uses the standard library and never accesses external data. + +## PowerShell + +- `Write-Output` writes to the success stream, so it can corrupt returned objects or JSON. Use the existing information/error streams when the host contract supports them; for a standalone native CLI, `[Console]::Error.WriteLine(...)` and `Flush()` provide explicit stderr output. +- `Write-Progress` depends on the host and preferences. It may be invisible in CI or redirection; do not use it as the only evidence of progress. +- Ctrl+C and host stop are host-managed, and behavior can differ by host and version. Preserve the existing stop behavior; do not catch broad exceptions and count host cancellation as an item failure. Use `finally` for cleanup where applicable, but do not promise a final summary after the host stops the pipeline. If graceful cancellation reporting is part of the contract, test the real cancellation path in each supported host; a simulated fixture does not establish it. +- Use `[System.Diagnostics.Stopwatch]` for elapsed intervals. Avoid a blocking `Start-Sleep` heartbeat in the same processing loop. +- Check terminating errors and non-terminating errors separately. Use `-ErrorAction Stop` at a specific cmdlet boundary when that matches the established policy; changing `$ErrorActionPreference` globally can change processing behavior. +- Capture native `$LASTEXITCODE` immediately after the native command. On Windows PowerShell 5.1, do not assume a nonzero native exit is a catchable terminating exception. +- Use `exit` only in a standalone entry point. A function/module should return its agreed result or raise its agreed error without terminating the host. + +The [PowerShell probe](../examples/batch.ps1) is a standalone script compatible with Windows PowerShell 5.1 and PowerShell 7. Its cancellation case is simulated, not an OS-signal test. + +## Node.js + +- `console.log` targets stdout; `console.error` targets stderr. For a stable record format, use the established logger or explicit stream writes. +- Use `performance.now()` or `process.hrtime.bigint()` for elapsed intervals. For redirected writes, respect backpressure and wait for `drain` when `write` returns false. +- Set `process.exitCode` after reporting instead of immediately calling `process.exit()`, which can truncate pending output. +- A timer heartbeat requires an available event loop. Synchronous CPU-bound processing blocks timers; do not add worker threads merely to decorate output. +- Integrate SIGINT/SIGTERM with existing cancellation and cleanup. Do not turn cancellation into a retriable item failure. + +The [Node.js probe](../examples/batch.mjs) has no dependencies and uses an async output emitter. + +## Executable Overrides + +The verifier defaults to `py -3` and `powershell.exe` on Windows, or `python3` and `pwsh` elsewhere. Set `BATCH_PYTHON` to a Python executable path (not a shell command) to override the Python launcher; set `BATCH_POWERSHELL` similarly. Node.js uses the interpreter running the verifier. + +On Windows the verifier passes `-ExecutionPolicy Bypass` only to its child PowerShell process so an unsigned local fixture can run. It does not change stored machine/user policy, and organization-enforced policy can still block it. Review the bundled fixture before running it; do not extend this setting to untrusted scripts. + +Each probe accepts `normal`, `mixed`, `partial-zero`, `empty`, `unknown`, `fail-fast`, `cancelled`, or `live`. The fixture contract continues after failures except in `fail-fast`; codes are 0 for complete success, 1 for `mixed`/fail-fast, and 130 for simulated cancellation. The `partial-zero` fixture preserves a zero exit contract while reporting an honest partial outcome and one failed item. These are fixture choices, not imposed production exit codes. In `live`, the probe waits for one stdin line after the first progress notification so the verifier can prove pre-exit delivery without guessing a delay. diff --git a/skills/batch-script-progress/references/verification.md b/skills/batch-script-progress/references/verification.md new file mode 100644 index 000000000..b37a0078f --- /dev/null +++ b/skills/batch-script-progress/references/verification.md @@ -0,0 +1,30 @@ +# Verification Cases + +Use harmless inputs and the actual caller contract. Fix expectations before running a comparison; do not modify expected results to match the generated script. + +## Runtime Checks + +- **Normal:** start precedes processing; progress counters are monotonic; the summary occurs once; stdout results are unchanged. +- **Partial failure:** inject one known item error; verify safe identification, the original continue/stop rule, accounting, summary state, and the agreed exit code. Include a skip if the processor supports one. +- **Zero items:** zero counts, start and summary, no false progress or percentage arithmetic. +- **Unknown total:** lazy input is consumed only once; total remains unknown; no invented percentage, remaining count, or ETA. +- **Fail-fast:** the failing item is counted once; not-started items stay unprocessed; the first failure is not lost when reporting the summary. +- **Cancellation:** verify the supported cancellation mechanism and cleanup; do not count incomplete work as successful. For PowerShell, preserve host-managed stop behavior and do not count a host stop as an item failure. Test Ctrl+C or another real host cancellation path only when graceful handling is part of the existing contract. Simulated cancellation is not proof of SIGINT or host cancellation behavior. +- **Long item:** distinguish between-item cadence from mid-item notification. Verify callbacks or heartbeats with the operation still active, or report the known notification gap. Keep deterministic gates in harmless fixtures or the harness; do not add test-only waits or hooks to the production path. +- **Redirection:** parse stdout using its real format, retain newline-delimited diagnostics separately, and read progress before child exit. Also verify a caller that interprets stderr as failure, if present. If stderr is incompatible, use only an already-supported logger or sink; if none exists, stop rather than changing stdout or another stream's semantics without approval. +- **Secrets and volume:** a synthetic secret must not appear in either stream. Failure records must not require retaining every raw exception or input in memory. +- **Retries, transactions, or concurrency, when present:** count terminal outcomes once; distinguish attempts, submissions, and committed results. Check a rollback before accepting a success summary. + +The bundled verifier covers eight safe probe scenarios in all three runtimes, including a partial failure with a preserved zero exit code, and rejects a deliberately corrupted count using the same accounting oracle. It does not claim transaction, real-signal, secret-redaction, concurrency, or mid-item heartbeat coverage for production scripts. + +## Skill Behavior Evaluation + +For a skill evaluation, compare independent fresh sessions with the same task, model, tools, and fixtures, once without the skill and once with explicit loading. Keep evaluation logs outside the installable skill. Test at least: + +1. A batch script with an unknown total and machine-readable stdout. +2. An existing partial-failure exit-zero contract that must be preserved. +3. A synchronously blocking item where an ordinary timer cannot guarantee a heartbeat. +4. A request to skip tests and claim the script verified; the response must disclose unexecuted checks. +5. An unrelated UI progress-bar request; this skill should not be selected. + +Evaluate automatic discovery separately from explicit loading. A frontmatter or keyword check does not prove automatic selection. Publish unexecuted evaluations as unexecuted, not as inferred passes. diff --git a/skills/batch-script-progress/tests/verify-examples.mjs b/skills/batch-script-progress/tests/verify-examples.mjs new file mode 100644 index 000000000..5432781fb --- /dev/null +++ b/skills/batch-script-progress/tests/verify-examples.mjs @@ -0,0 +1,135 @@ +import assert from 'node:assert/strict'; +import { spawn, spawnSync } from 'node:child_process'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const root = dirname(dirname(fileURLToPath(import.meta.url))); +const windows = process.platform === 'win32'; +const runtimes = [ + { name: 'Python', command: process.env.BATCH_PYTHON ?? (windows ? 'py.exe' : 'python3'), args: [...(windows && !process.env.BATCH_PYTHON ? ['-3'] : []), '-B', join(root, 'examples/batch.py')] }, + { name: 'PowerShell', command: process.env.BATCH_POWERSHELL ?? (windows ? 'powershell.exe' : 'pwsh'), args: ['-NoProfile', '-NonInteractive', ...(windows ? ['-ExecutionPolicy', 'Bypass'] : []), '-File', join(root, 'examples/batch.ps1')] }, + { name: 'Node.js', command: process.execPath, args: [join(root, 'examples/batch.mjs')] }, +]; +const cases = { + normal: { total: 3, succeeded: 3, failed: 0, skipped: 0, status: 'completed', code: 0 }, + mixed: { total: 4, succeeded: 2, failed: 1, skipped: 1, status: 'partial', code: 1 }, + 'partial-zero': { total: 3, succeeded: 2, failed: 1, skipped: 0, status: 'partial', code: 0 }, + empty: { total: 0, succeeded: 0, failed: 0, skipped: 0, status: 'completed', code: 0 }, + unknown: { total: null, succeeded: 3, failed: 0, skipped: 0, status: 'completed', code: 0 }, + 'fail-fast': { total: 3, succeeded: 1, failed: 1, skipped: 0, status: 'failed', code: 1 }, + cancelled: { total: 3, succeeded: 1, failed: 0, skipped: 0, status: 'cancelled', code: 130 }, + live: { total: 3, succeeded: 3, failed: 0, skipped: 0, status: 'completed', code: 0 }, +}; + +function accounting(record) { + for (const key of ['processed', 'succeeded', 'failed', 'skipped']) { + assert.ok(Number.isInteger(record[key]) && record[key] >= 0, key); + } + assert.equal(record.processed, record.succeeded + record.failed + record.skipped); +} + +function verify(stdout, stderr, code, scenario) { + const expected = cases[scenario]; + const events = stderr.trim().split(/\r?\n/).filter(Boolean).map(line => JSON.parse(line)); + assert.equal(events[0].event, 'start'); + assert.equal(events.at(-1).event, 'summary'); + assert.equal(events.filter(record => record.event === 'start').length, 1); + assert.equal(events.filter(record => record.event === 'summary').length, 1); + let previous = { processed: 0, succeeded: 0, failed: 0, skipped: 0, elapsed_ms: 0 }; + for (const record of events) { + accounting(record); + assert.equal(record.total, expected.total); + for (const key of ['processed', 'succeeded', 'failed', 'skipped', 'elapsed_ms']) { + assert.ok(record[key] >= previous[key], `non-monotonic ${key}`); + } + if (expected.total === null) { + for (const key of ['percentage', 'eta', 'remaining']) assert.ok(!(key in record)); + } + previous = record; + } + const summary = events.at(-1); + for (const key of ['succeeded', 'failed', 'skipped', 'status']) assert.equal(summary[key], expected[key]); + assert.equal(summary.exit_code, expected.code); + assert.equal(code, expected.code); + assert.equal(summary.unprocessed, expected.total === null ? null : expected.total - summary.processed); + const failures = events.filter(record => record.event === 'item-failed'); + assert.equal(failures.length, expected.failed); + for (const failure of failures) { + assert.equal(failure.item_id, ['fail-fast', 'partial-zero'].includes(scenario) ? 'item-2' : 'item-3'); + assert.equal(failure.reason, 'fixture-rejected'); + assert.equal(failure.will_continue, scenario !== 'fail-fast'); + } + if (scenario === 'empty') assert.equal(events.filter(record => record.event === 'progress').length, 0); + if (['normal', 'unknown', 'mixed', 'partial-zero', 'live'].includes(scenario)) assert.ok(events.some(record => record.event === 'progress')); + if (scenario === 'cancelled') { + assert.equal(stdout, ''); + } else { + const result = JSON.parse(stdout); + const ids = scenario === 'mixed' ? [1, 4] : scenario === 'partial-zero' ? [1, 3] : scenario === 'fail-fast' ? [1] : scenario === 'empty' ? [] : [1, 2, 3]; + assert.deepEqual(result.results, ids.map(index => ({ id: `item-${index}`, value: index * 2 }))); + } +} + +async function verifyLive(runtime) { + const child = spawn(runtime.command, [...runtime.args, 'live'], { stdio: ['pipe', 'pipe', 'pipe'] }); + let stdout = ''; + let stderr = ''; + let pending = ''; + let released = false; + let failure; + const timeout = setTimeout(() => { + failure = new Error(`${runtime.name}: no live notification before watchdog`); + child.kill(); + }, 15000); + child.stdout.setEncoding('utf8'); + child.stderr.setEncoding('utf8'); + child.stdin.on('error', error => { failure ??= error; }); + child.stdout.on('data', chunk => { stdout += chunk; }); + child.stderr.on('data', chunk => { + stderr += chunk; + pending += chunk; + let newline; + while ((newline = pending.indexOf('\n')) !== -1) { + const line = pending.slice(0, newline).trim(); + pending = pending.slice(newline + 1); + try { + const record = JSON.parse(line); + if (record.event === 'progress' && !released) { + assert.equal(child.exitCode, null); + assert.equal(child.signalCode, null); + assert.equal(record.processed, 1); + released = true; + child.stdin.end('\n'); + } + } catch (error) { + failure ??= error; + child.kill(); + } + } + }); + const code = await new Promise((resolve, reject) => { + child.once('error', reject); + child.once('close', resolve); + }).finally(() => clearTimeout(timeout)); + if (failure) throw failure; + assert.ok(released, `${runtime.name}: notification not observed before exit`); + verify(stdout, stderr, code, 'live'); +} + +assert.throws(() => accounting({ processed: 2, succeeded: 1, failed: 0, skipped: 0 }), assert.AssertionError); +console.log('PASS negative control: corrupted accounting rejected'); +let passed = 0; +for (const runtime of runtimes) { + for (const scenario of Object.keys(cases).filter(name => name !== 'live')) { + const result = spawnSync(runtime.command, [...runtime.args, scenario], { encoding: 'utf8', timeout: 15000 }); + if (result.error) throw result.error; + assert.equal(result.signal, null); + verify(result.stdout, result.stderr, result.status, scenario); + console.log(`PASS ${runtime.name} ${scenario}`); + passed++; + } + await verifyLive(runtime); + console.log(`PASS ${runtime.name} live output received before release and exit`); + passed++; +} +console.log(`BATCH_OUTPUT_CONTRACT_PASSED scenarios=${passed}`);