mirror of
https://github.com/github/awesome-copilot.git
synced 2026-08-13 12:49:49 +00:00
Merge branch 'main' into vcpkg-skill
This commit is contained in:
Ayan Gupta
GitHub
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: anti-ui-slop
|
||||
description: 'Stop Codex, GitHub Copilot, Claude Code, and Cursor from shipping generic UI. Use UIZZE’s public catalogue of 800,000+ real web and iOS screens to extract product-specific design decisions and enforce a hard finish gate for web and iOS interfaces.'
|
||||
---
|
||||
|
||||
# Anti UI Slop
|
||||
|
||||
Use this skill when building, refactoring, or reviewing a web or iOS interface. The goal is not to make a generic layout prettier. The goal is to make the interface visibly belong to this product, support its real user job, and behave correctly in every important state.
|
||||
|
||||
The workflow is instruction-only. It does not execute third-party code or require credentials.
|
||||
|
||||
## 1. Inspect the Product Before Designing
|
||||
|
||||
Read the repository and identify:
|
||||
|
||||
- the primary user and the job this screen must complete;
|
||||
- the single primary action and the information needed before taking it;
|
||||
- the existing component library, design tokens, typography, and layout conventions;
|
||||
- real product nouns, workflows, constraints, and data already present in the codebase;
|
||||
- required loading, empty, error, partial, success, disabled, and permission states;
|
||||
- relevant mobile, tablet, desktop, keyboard, and assistive-technology behavior.
|
||||
|
||||
Do not invent product requirements, analytics, user research, or hidden states.
|
||||
|
||||
## 2. Collect Real Interface Evidence
|
||||
|
||||
Search the public catalogue at https://uizze.com and select three to five relevant web or iOS screens. Prefer references that match the target workflow, information density, navigation model, or interaction pattern—not merely its industry or color palette.
|
||||
|
||||
For each reference, record:
|
||||
|
||||
1. the screen or flow and its source link;
|
||||
2. the structural decision worth transferring;
|
||||
3. why that decision fits this product;
|
||||
4. what must not be copied.
|
||||
|
||||
Transfer hierarchy, workflow shape, density, navigation, control behavior, responsive treatment, and state handling. Never copy another product’s branding, proprietary text, imagery, or exact layout.
|
||||
|
||||
If catalogue browsing is unavailable, ask the user for two or three UIZZE links or screenshots. If they cannot provide them, continue from repository evidence and label the missing reference evidence explicitly.
|
||||
|
||||
## 3. Write a Design Contract
|
||||
|
||||
Before changing code, write a short contract with these fields:
|
||||
|
||||
| Field | Decision |
|
||||
| --- | --- |
|
||||
| Screen job | The one outcome this screen enables |
|
||||
| Primary user and action | Who acts, and what they do |
|
||||
| Content hierarchy | What must be understood first, second, and third |
|
||||
| Navigation and controls | Product-specific structure and interaction model |
|
||||
| Visual language | Type, spacing, density, surfaces, imagery, and motion rules |
|
||||
| Required states | Loading, empty, error, partial, success, disabled, permission |
|
||||
| Responsive behavior | What changes across supported widths and input modes |
|
||||
| Evidence used | Reference links and transferable decisions |
|
||||
| Forbidden defaults | Generic patterns that would erase product specificity |
|
||||
| Acceptance criteria | Observable conditions required before shipping |
|
||||
|
||||
The contract must name concrete choices. “Clean,” “modern,” “intuitive,” and “premium” are not design decisions.
|
||||
|
||||
## 4. Build in the Product’s Language
|
||||
|
||||
- Reuse the repository’s components and semantic tokens before adding new ones.
|
||||
- Make the primary action visually and structurally obvious.
|
||||
- Use product-specific labels and information rather than placeholder metrics or generic copy.
|
||||
- Keep repeated cards only when the content is genuinely a repeated collection.
|
||||
- Add decoration, motion, badges, or elevation only when they communicate state or hierarchy.
|
||||
- Implement every required interaction and state; do not leave convincing-looking inert controls.
|
||||
- Preserve accessibility semantics, focus order, contrast, touch targets, and reduced-motion behavior.
|
||||
|
||||
## 5. Run the Finish Gate
|
||||
|
||||
Render the result at every supported breakpoint and block completion when any item fails:
|
||||
|
||||
### Product specificity
|
||||
|
||||
- Could this interface belong to an unrelated product after changing the logo?
|
||||
- Does the hierarchy reflect the real user job and product data?
|
||||
- Are there interchangeable dashboard cards, filler metrics, vague headings, or generic calls to action?
|
||||
|
||||
### Interaction completeness
|
||||
|
||||
- Do all visible controls have a real outcome?
|
||||
- Are loading, empty, error, success, disabled, and permission states implemented where applicable?
|
||||
- Are destructive, irreversible, or sensitive actions confirmed appropriately?
|
||||
|
||||
### Responsive and accessible behavior
|
||||
|
||||
- Does the layout remain usable without merely stacking every region vertically?
|
||||
- Do keyboard navigation, focus visibility, semantics, contrast, and touch targets pass inspection?
|
||||
- Does content remain readable at zoom and with longer real-world text?
|
||||
|
||||
### Design-system integrity
|
||||
|
||||
- Are local tokens and components used consistently?
|
||||
- Is every new visual rule justified by the design contract?
|
||||
- Is borrowed evidence transformed into this product’s own visual language?
|
||||
|
||||
Fix every blocking failure and re-run the gate before declaring the UI complete.
|
||||
|
||||
## 6. Handoff Format
|
||||
|
||||
Report the finished work in this order:
|
||||
|
||||
1. **Evidence:** the references and decisions that influenced the result.
|
||||
2. **Contract:** the final product-specific design rules.
|
||||
3. **Implementation:** the meaningful interface and behavior changes.
|
||||
4. **Verification:** breakpoints, interaction states, and accessibility checks performed.
|
||||
5. **Remaining risks:** anything that could not be verified, without overstating completion.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: bench-read
|
||||
description: 'Read artifacts from the shared bench — the workspace where desks leave findings, verdicts, and work products for each other and the operator.'
|
||||
---
|
||||
|
||||
# Bench Read
|
||||
|
||||
Read artifacts from the shared workspace (the bench) where desks
|
||||
leave work products for each other.
|
||||
|
||||
## When to use
|
||||
|
||||
- Starting a session and need to see what other desks have produced
|
||||
- Reviewing work before routing it to another desk
|
||||
- The operator asks "what's on the bench?" or "show me what desk X found"
|
||||
- A desk needs context from another desk's output
|
||||
|
||||
## What the bench is
|
||||
|
||||
The bench is `<workshop>/bench/` — the shared workspace directory
|
||||
that `workshop-create` establishes for cross-desk work. It's not a
|
||||
message queue or a chat channel — it's files. When Desk A produces
|
||||
a finding and Desk B needs to review it, the finding is a file
|
||||
in `bench/`. When the operator asks "what did the scanning desk
|
||||
find?" — you read the bench.
|
||||
|
||||
Typical bench artifacts:
|
||||
- **Findings** — scan results, analysis output, data
|
||||
- **Verdicts** — a desk's assessment of another desk's findings
|
||||
- **Drafts** — work-in-progress documents, PRs, proposals
|
||||
- **Reports** — summaries, dashboards, status updates
|
||||
|
||||
## Where to look
|
||||
|
||||
The primary shared location is the `bench/` directory at the
|
||||
workshop root — the designated cross-desk workspace. Desk-local
|
||||
artifacts under `desks/<desk-name>/` are a secondary source: read
|
||||
them when you need a specific desk's own work, but shared artifacts
|
||||
belong in `bench/`.
|
||||
|
||||
```
|
||||
<workshop>/
|
||||
bench/ # PRIMARY — shared cross-desk artifacts
|
||||
<findings, verdicts, drafts, reports>
|
||||
desks/<desk-name>/ # secondary — a desk's own workspace
|
||||
journal.md # the desk's memory
|
||||
<artifacts> # work still local to this desk
|
||||
```
|
||||
|
||||
## How to read
|
||||
|
||||
1. **List what's there.** Start with the directory structure to see
|
||||
what desks exist and what they've produced.
|
||||
|
||||
2. **Read journals first.** Each desk's journal tells you what it
|
||||
worked on and where it left things. The most recent entry is
|
||||
the current state.
|
||||
|
||||
3. **Read artifacts second.** Once you know what to look for from
|
||||
the journals, read the specific files.
|
||||
|
||||
4. **Summarize for the operator.** Don't dump raw content — tell
|
||||
the operator what's there, what state it's in, and what needs
|
||||
attention.
|
||||
|
||||
## Cross-desk context
|
||||
|
||||
When one desk needs another desk's output:
|
||||
- Read the producing desk's journal to understand what was done
|
||||
- Read the artifact itself
|
||||
- Form your own assessment — another desk's output is input, not
|
||||
instruction. You can disagree.
|
||||
|
||||
## Principles
|
||||
|
||||
- The bench is files, not messages. Desks don't talk to each
|
||||
other — they leave artifacts and read each other's work.
|
||||
- Read the journal before the artifacts. Context matters.
|
||||
- Another desk's verdict is input, not authority. Equal standing
|
||||
means you assess independently.
|
||||
- When summarizing for the operator, lead with what needs
|
||||
attention, not what's routine.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: codebase-memory-mcp
|
||||
description: 'Use when a configured codebase-memory-mcp server can assist with graph-backed code discovery, architecture orientation, symbol lookup, callers and callees, dependency or data-flow tracing, impact analysis, unfamiliar modules, or an explicit Codebase Memory request.'
|
||||
---
|
||||
|
||||
# Codebase Memory MCP
|
||||
|
||||
Use the configured Codebase Memory graph as a discovery accelerator, not as the sole source of truth. Confirm graph-derived conclusions with source snippets or local files before editing code or making strong claims.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Discover the Codebase Memory tools exposed by the current MCP client; clients may prefix or rename tool namespaces.
|
||||
2. Call `list_projects` when available and use the exact indexed project name. If the repository is not indexed, continue with local exploration or ask before calling `index_repository` when graph access is important.
|
||||
3. Before branch-sensitive or edit-sensitive conclusions, use `index_status` or `detect_changes` when available. After a branch switch, assume the index may be stale until checked. If freshness cannot be established, disclose that limitation and verify locally.
|
||||
4. Use `get_architecture` once for orientation in an unfamiliar repository or subsystem. Do not repeat it for narrow follow-up questions.
|
||||
5. Use `search_graph` for definitions, implementations, routes, classes, interfaces, callers, and related symbols. Prefer a natural-language query for discovery and a name or qualified-name pattern for known symbols. Narrow by label or path, set a result limit, and paginate or reduce scope when the response reports more results.
|
||||
6. Use `search_code` or normal repository search for literal strings, configuration keys, test identifiers, error messages, and non-code files. Do not turn a precise text lookup into a broad graph query.
|
||||
7. After graph search, use `get_code_snippet` with the returned qualified name. If source snippets are unavailable, open the local file before relying on the result.
|
||||
8. Use `trace_path` for callers, callees, dependency paths, data flow, cross-service paths, and impact analysis. Include tests only when test coverage is part of the question.
|
||||
9. Use `get_graph_schema` before `query_graph`. Reserve custom queries for multi-hop or aggregate questions that simpler tools cannot answer, and apply `LIMIT` or the tool's row limit.
|
||||
10. When graph and checked-out source disagree, treat source as current and report likely index drift.
|
||||
|
||||
## Safety and Fallbacks
|
||||
|
||||
- Do not install Codebase Memory or another third-party skill from this workflow.
|
||||
- Do not call `delete_project`, ingest traces, update ADRs, or index a repository unless the user explicitly requested or approved the action; announce it before execution.
|
||||
- Fall back to normal repository exploration when the MCP server, project, index, or required capability is unavailable; do not invent tool results or stop a task that can be completed safely without the graph.
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
name: convert-excel-to-md
|
||||
description: 'Converts Excel (.xlsx) workbooks into Markdown so their contents can be accurately analyzed, summarized, searched, or extracted from. Use this skill whenever the user shares, references, or asks about a .xlsx file — even if they don''t say "convert" or "markdown" explicitly. This includes requests to "read", "summarize", "review", "extract data from", "compare", "chart", or "analyze" a spreadsheet, workbook, budget, data export, or tracker. Always run the bundled conversion script to produce Markdown first; do not attempt to parse .xlsx content directly or write ad-hoc extraction code. Also use this skill for batch requests involving a whole folder of Excel workbooks. IMPORTANT: When the user references a folder or set of documents containing multiple file types (.pdf, .docx, .xlsx), invoke ALL three sibling skills — convert-pdf-to-md, convert-word-to-md, and convert-excel-to-md — so no file type is silently skipped.'
|
||||
---
|
||||
|
||||
# Convert Excel to Markdown
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Trigger this skill any time there is a `.xlsx` file that needs to be
|
||||
understood or processed — for example, a user attaches a spreadsheet and
|
||||
asks questions about it, wants a summary of the data, wants specific rows or
|
||||
values pulled out, or wants multiple workbooks in a folder processed
|
||||
together. Excel's native `.xlsx` format is a zipped XML bundle that is not
|
||||
reliably readable as plain text, so always convert it to Markdown first
|
||||
using the script in this skill rather than trying to open or parse the file
|
||||
directly.
|
||||
|
||||
This skill only supports `.xlsx`. If asked to convert a legacy `.xls` file,
|
||||
tell the user it isn't supported and ask them to re-save it as `.xlsx`
|
||||
(Excel: File > Save As > Excel Workbook (.xlsx)) first.
|
||||
|
||||
**Mixed file types:** When the user references a folder or set of documents
|
||||
containing multiple supported file types (`.pdf`, `.docx`, `.xlsx`), this
|
||||
skill handles only `.xlsx` files. The agent MUST also invoke the sibling
|
||||
skills in parallel:
|
||||
- `convert-pdf-to-md` for any `.pdf` files
|
||||
- `convert-word-to-md` for any `.docx` files
|
||||
|
||||
Never process a folder and silently skip a supported file type. All three
|
||||
skills must be invoked together when mixed types are present.
|
||||
|
||||
## Setup (once per environment)
|
||||
|
||||
Before the first conversion in a given environment, follow
|
||||
[`references/setup.md`](references/setup.md) step by step to ensure Python,
|
||||
pip, and the `markitdown` package are installed. Do this proactively rather
|
||||
than guessing whether the environment is ready — the script itself will
|
||||
also fail with a clear pointer back to that file if `markitdown` turns out
|
||||
to be missing, so it's safe to just try the conversion first if you're
|
||||
reasonably confident setup was already done.
|
||||
|
||||
## Usage
|
||||
|
||||
The conversion script lives at `scripts/convert_excel_to_md.py`.
|
||||
|
||||
**Output structure:** MarkItDown's XLSX converter renders each sheet as its
|
||||
own `## <SheetName>` Markdown table — it has no support for embedded images
|
||||
at all. This script separately extracts real embedded images (raster
|
||||
pictures, not charts) and maps them to the sheet they belong to, writing a
|
||||
self-contained folder per document:
|
||||
|
||||
```
|
||||
<name>/
|
||||
img/
|
||||
sheet001_<sheetname>_img001.<ext>
|
||||
sheet002_<sheetname>_img001.<ext>
|
||||
...
|
||||
<name>.md (each sheet's images appear right after its table,
|
||||
under a "#### Images in this sheet" heading)
|
||||
```
|
||||
|
||||
This is per-sheet placement, not exact cell position — the finest
|
||||
granularity MarkItDown's stable output anchors (the `## <SheetName>`
|
||||
headings) allow. If a workbook has no embedded images, no `img/` folder or
|
||||
image sections are created. Native Excel **charts** are not extracted as
|
||||
images (only actual embedded pictures are — charts would need to be
|
||||
rendered by Excel/LibreOffice, which this lightweight skill does not do).
|
||||
|
||||
**Single file:**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_excel_to_md.py "C:\path\to\workbook.xlsx"
|
||||
```
|
||||
|
||||
This creates a `workbook\` folder next to the source file (containing
|
||||
`workbook.md` and, if present, `workbook\img\`). To control the destination
|
||||
folder explicitly:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_excel_to_md.py "C:\path\to\workbook.xlsx" -o "C:\path\to\output_folder"
|
||||
```
|
||||
|
||||
**A folder of workbooks (batch mode):**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_excel_to_md.py "C:\path\to\folder"
|
||||
```
|
||||
|
||||
Add `--recursive` to also include subfolders:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_excel_to_md.py "C:\path\to\folder" --recursive
|
||||
```
|
||||
|
||||
Each `.xlsx` found gets its own `<name>\` output folder next to it by
|
||||
default. Pass `-o "C:\path\to\output_parent"` to collect all the generated
|
||||
`<name>\` folders under a separate parent directory instead (subfolder
|
||||
structure is preserved when combined with `--recursive`).
|
||||
|
||||
After conversion, read the resulting `.md` file(s) to perform the actual
|
||||
analysis the user asked for — the script's job is only to produce accurate
|
||||
Markdown (and images), not to interpret the content.
|
||||
|
||||
## Deciding where output goes
|
||||
|
||||
**Default — always output next to the source file.** The `<name>/` folder
|
||||
is created in the same directory as the source `.xlsx`. This is the required
|
||||
default for every case. Do NOT override it unless the user explicitly asks
|
||||
for a different location.
|
||||
|
||||
**Only use `-o` when** the user explicitly provides an output path (e.g.,
|
||||
"save the output to `C:\output`", "put the results in `D:\work`"). Do NOT
|
||||
pass `-o` based on the agent's current working directory, the session state
|
||||
folder, or any implied location.
|
||||
|
||||
**If the source file path cannot be fully resolved** — for example, the
|
||||
user provides only a filename with no directory, or the path is ambiguous —
|
||||
use `ask_user` to confirm the full absolute path before running the
|
||||
conversion. Never guess or assume the directory.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `ModuleNotFoundError: No module named 'markitdown'` / exit code 2 | MarkItDown not installed | Follow `references/setup.md` |
|
||||
| `ERROR: Unsupported file type '.xls'` / exit code 3 | Legacy `.xls`, not `.xlsx` | Ask the user to re-save as `.xlsx` |
|
||||
| `ERROR: Input path not found` / exit code 3 | Wrong path, or file moved | Confirm the correct path with the user |
|
||||
| `FAILED <file> -> ...` in batch output | That specific file is corrupt, password-protected, or otherwise unreadable | Report which file(s) failed; other files in the batch still succeed |
|
||||
| `NOTE: skipped N non-.xlsx file(s)` | Folder contains non-Excel files | Expected — those files are intentionally ignored |
|
||||
| A sheet's charts don't appear as images | Charts are chart objects, not embedded pictures — this skill only extracts real embedded raster images | Expected; mention this limitation if the user specifically needs chart images |
|
||||
@@ -0,0 +1,73 @@
|
||||
# Environment Setup for convert-excel-to-md
|
||||
|
||||
Follow these steps exactly, in order, before running `scripts/convert_excel_to_md.py`
|
||||
for the first time in a given environment. Don't skip steps or improvise
|
||||
alternatives — they're written to be deterministic and safe to re-run.
|
||||
|
||||
## 1. Check Python is available (3.10+)
|
||||
|
||||
```powershell
|
||||
python --version
|
||||
```
|
||||
|
||||
- If this fails (command not found), install Python 3.10 or newer:
|
||||
- Windows: `winget install --id Python.Python.3.12 -e`
|
||||
- macOS: `brew install python@3.12`
|
||||
- Linux (Debian/Ubuntu): `sudo apt-get update && sudo apt-get install -y python3 python3-pip python-is-python3`
|
||||
- If the reported version is older than 3.10, install a newer Python using
|
||||
the same command above (MarkItDown requires 3.10+).
|
||||
|
||||
## 2. Check pip is available
|
||||
|
||||
```powershell
|
||||
python -m pip --version
|
||||
```
|
||||
|
||||
- If this fails, bootstrap pip:
|
||||
|
||||
```powershell
|
||||
python -m ensurepip --upgrade
|
||||
```
|
||||
|
||||
## 3. Install MarkItDown with Excel (.xlsx) support
|
||||
|
||||
Use the `scripts/requirements.txt` file bundled with this skill to install a pinned,
|
||||
known-good version of the dependency:
|
||||
|
||||
```powershell
|
||||
python -m pip install -r scripts/requirements.txt
|
||||
```
|
||||
|
||||
This pulls in `markitdown[xlsx]` (MarkItDown's XLSX table conversion
|
||||
dependencies, which include `pandas` and `openpyxl`). No extra package is needed for image extraction — this
|
||||
skill's script reads embedded images directly from the `.xlsx` zip
|
||||
structure using Python's built-in `zipfile` and `xml` modules.
|
||||
|
||||
## 4. Verify the install
|
||||
|
||||
```powershell
|
||||
python -c "from markitdown import MarkItDown; print('markitdown OK')"
|
||||
```
|
||||
|
||||
Expect to see `markitdown OK` printed with no errors. If you see
|
||||
`ModuleNotFoundError: No module named 'markitdown'`, repeat step 3 — pip may
|
||||
be installing into a different Python environment than the one being
|
||||
invoked (check `python -m pip --version` shows the same path as `python
|
||||
--version`'s interpreter).
|
||||
|
||||
## Notes
|
||||
|
||||
- This setup only needs to be done once per environment/virtual environment,
|
||||
not once per conversion.
|
||||
- `convert_excel_to_md.py` itself also checks for `markitdown` at startup
|
||||
and prints a pointer back to this file if it's missing, so re-running
|
||||
setup is safe and idempotent.
|
||||
- Only `.xlsx` is supported by this skill. Legacy binary `.xls` files are
|
||||
out of scope (a completely different, harder-to-parse file format) — ask
|
||||
the user to re-save the file as `.xlsx` (Excel: File > Save As > Excel
|
||||
Workbook (.xlsx)) if one is encountered.
|
||||
- Chart objects (as opposed to embedded pictures) are not extracted as
|
||||
images — only raster pictures actually embedded in the workbook's
|
||||
`xl/media` folder are. Native Excel charts would need to be rendered by
|
||||
Excel/LibreOffice to become images, which this lightweight skill does not
|
||||
attempt.
|
||||
@@ -0,0 +1,354 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert Excel (.xlsx) workbooks to Markdown using Microsoft's MarkItDown,
|
||||
with embedded images extracted to real files and placed under the correct
|
||||
sheet (MarkItDown's XLSX converter only extracts sheet data as tables -- it
|
||||
has no support for embedded images at all).
|
||||
|
||||
Usage:
|
||||
python convert_excel_to_md.py <input> [-o OUTPUT] [--recursive]
|
||||
|
||||
<input> may be either:
|
||||
- a path to a single .xlsx file, or
|
||||
- a path to a directory (batch mode: every .xlsx file directly inside it
|
||||
is converted; pass --recursive to also descend into subdirectories).
|
||||
|
||||
Output:
|
||||
For each source .xlsx (named "<name>.xlsx"), a folder is created
|
||||
containing the Markdown and its images, in this layout:
|
||||
|
||||
<name>/
|
||||
img/
|
||||
Sheet1_img001.<ext>
|
||||
Sheet2_img001.<ext>
|
||||
...
|
||||
<name>.md
|
||||
|
||||
MarkItDown renders each sheet as its own "## <SheetName>" section with a
|
||||
Markdown table. This script independently maps embedded images to the
|
||||
sheet they belong to (via the .xlsx zip's drawing relationships) and
|
||||
inserts a "#### Images in this sheet" block right after that sheet's
|
||||
table, before the next "## " heading. This is per-sheet placement (not
|
||||
exact cell position), which is the finest granularity MarkItDown's stable
|
||||
output anchors allow.
|
||||
|
||||
- Single file mode: the "<name>/" folder is created next to the source
|
||||
file, or at -o/--output (treated as the exact destination folder) if
|
||||
given.
|
||||
- Batch/directory mode: a "<name>/" folder is created next to each source
|
||||
file, or under -o/--output (treated as a parent directory, created if
|
||||
missing) if given, preserving relative subfolder structure when
|
||||
--recursive is used.
|
||||
- If a workbook has no embedded images, no "img/" folder or "Images in
|
||||
this sheet" sections are created.
|
||||
|
||||
Exit codes:
|
||||
0 - all requested conversions succeeded
|
||||
1 - one or more conversions failed (partial success in batch mode)
|
||||
2 - required dependency ("markitdown") is not installed
|
||||
3 - invalid input (path not found, or single-file input is not .xlsx)
|
||||
"""
|
||||
import argparse
|
||||
import posixpath
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from xml.etree import ElementTree as ET
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_CONVERSION_FAILED = 1
|
||||
EXIT_MISSING_DEPENDENCY = 2
|
||||
EXIT_INVALID_INPUT = 3
|
||||
|
||||
_REL_NS = "http://schemas.openxmlformats.org/package/2006/relationships"
|
||||
_MAIN_NS = "http://schemas.openxmlformats.org/spreadsheetml/2006/main"
|
||||
_R_NS = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"
|
||||
_A_NS = "http://schemas.openxmlformats.org/drawingml/2006/main"
|
||||
|
||||
# Matches MarkItDown's per-sheet heading, e.g. "## Sheet1"
|
||||
_SHEET_HEADER_RE = re.compile(r"^## (.+)$", re.MULTILINE)
|
||||
|
||||
|
||||
def _import_markitdown():
|
||||
"""Import MarkItDown, failing with a clear, actionable message if absent."""
|
||||
try:
|
||||
from markitdown import MarkItDown
|
||||
return MarkItDown
|
||||
except ImportError:
|
||||
print(
|
||||
"ERROR: The 'markitdown' package is not installed.\n"
|
||||
"See references/setup.md for this skill, or run:\n"
|
||||
' pip install "markitdown[xlsx]"',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
|
||||
|
||||
def _normalize_rel_path(base_dir: str, target: str) -> str:
|
||||
"""Resolve a (possibly relative, e.g. '../media/image1.png') relationship
|
||||
target against the directory containing the part that referenced it."""
|
||||
if target.startswith("/"):
|
||||
return target.lstrip("/")
|
||||
return posixpath.normpath(posixpath.join(base_dir, target))
|
||||
|
||||
|
||||
def _sheet_name_to_media(xlsx_path: Path):
|
||||
"""Return {sheet_name: [media_zip_path, ...]} in per-sheet document
|
||||
order, by walking workbook.xml -> worksheet -> drawing -> media
|
||||
relationships. Returns {} if anything is missing/malformed (falls back
|
||||
gracefully -- images just won't be extracted for that sheet)."""
|
||||
try:
|
||||
with zipfile.ZipFile(xlsx_path) as z:
|
||||
names = set(z.namelist())
|
||||
if "xl/workbook.xml" not in names or "xl/_rels/workbook.xml.rels" not in names:
|
||||
return {}
|
||||
workbook_xml = z.read("xl/workbook.xml")
|
||||
workbook_rels_xml = z.read("xl/_rels/workbook.xml.rels")
|
||||
|
||||
sheet_rid = {}
|
||||
for sheet_el in ET.fromstring(workbook_xml).iter(f"{{{_MAIN_NS}}}sheet"):
|
||||
name = sheet_el.get("name")
|
||||
rid = sheet_el.get(f"{{{_R_NS}}}id")
|
||||
if name and rid:
|
||||
sheet_rid[name] = rid
|
||||
|
||||
rid_target = {}
|
||||
for rel in ET.fromstring(workbook_rels_xml).findall(f"{{{_REL_NS}}}Relationship"):
|
||||
rid_target[rel.get("Id")] = rel.get("Target")
|
||||
|
||||
result = {}
|
||||
for sheet_name, rid in sheet_rid.items():
|
||||
target = rid_target.get(rid)
|
||||
if not target:
|
||||
continue
|
||||
# workbook.xml.rels targets are typically relative to "xl/",
|
||||
# but OOXML allows package-absolute targets (leading "/") too.
|
||||
sheet_path = _normalize_rel_path("xl", target)
|
||||
if sheet_path not in names or "/" not in sheet_path:
|
||||
continue
|
||||
sheet_dir, sheet_file = sheet_path.rsplit("/", 1)
|
||||
sheet_rels_path = f"{sheet_dir}/_rels/{sheet_file}.rels"
|
||||
if sheet_rels_path not in names:
|
||||
continue
|
||||
|
||||
drawing_rid = None
|
||||
for d in ET.fromstring(z.read(sheet_path)).iter(f"{{{_MAIN_NS}}}drawing"):
|
||||
drawing_rid = d.get(f"{{{_R_NS}}}id")
|
||||
break
|
||||
if not drawing_rid:
|
||||
continue
|
||||
|
||||
drawing_target = None
|
||||
for rel in ET.fromstring(z.read(sheet_rels_path)).findall(f"{{{_REL_NS}}}Relationship"):
|
||||
if rel.get("Id") == drawing_rid:
|
||||
drawing_target = rel.get("Target")
|
||||
break
|
||||
if not drawing_target:
|
||||
continue
|
||||
drawing_path = _normalize_rel_path(sheet_dir, drawing_target)
|
||||
if drawing_path not in names or "/" not in drawing_path:
|
||||
continue
|
||||
drawing_dir, drawing_file = drawing_path.rsplit("/", 1)
|
||||
drawing_rels_path = f"{drawing_dir}/_rels/{drawing_file}.rels"
|
||||
if drawing_rels_path not in names:
|
||||
continue
|
||||
|
||||
drawing_rel_map = {}
|
||||
for rel in ET.fromstring(z.read(drawing_rels_path)).findall(f"{{{_REL_NS}}}Relationship"):
|
||||
drawing_rel_map[rel.get("Id")] = rel.get("Target")
|
||||
|
||||
media_paths = []
|
||||
for blip in ET.fromstring(z.read(drawing_path)).iter(f"{{{_A_NS}}}blip"):
|
||||
embed_rid = blip.get(f"{{{_R_NS}}}embed")
|
||||
if not embed_rid:
|
||||
continue
|
||||
rel_target = drawing_rel_map.get(embed_rid)
|
||||
if not rel_target:
|
||||
continue
|
||||
media_path = _normalize_rel_path(drawing_dir, rel_target)
|
||||
if media_path in names:
|
||||
media_paths.append(media_path)
|
||||
|
||||
if media_paths:
|
||||
result[sheet_name] = media_paths
|
||||
return result
|
||||
except (zipfile.BadZipFile, KeyError, OSError, ET.ParseError):
|
||||
return {}
|
||||
|
||||
|
||||
def _sanitize_filename_part(name: str) -> str:
|
||||
safe = re.sub(r"[^A-Za-z0-9_.-]+", "_", name).strip("_")
|
||||
return safe or "sheet"
|
||||
|
||||
|
||||
def extract_images(xlsx_path: Path, img_dir: Path):
|
||||
"""Extract embedded images from xlsx_path, grouped by sheet name.
|
||||
Returns {sheet_name: [filename, ...]} in per-sheet order. Files are
|
||||
named '<sanitized_sheet_name>_img{N:03d}.<ext>'."""
|
||||
sheet_media = _sheet_name_to_media(xlsx_path)
|
||||
if not sheet_media:
|
||||
return {}
|
||||
|
||||
written = {}
|
||||
with zipfile.ZipFile(xlsx_path) as z:
|
||||
names_in_zip = set(z.namelist())
|
||||
for sheet_idx, (sheet_name, media_paths) in enumerate(sheet_media.items(), start=1):
|
||||
safe_name = f"sheet{sheet_idx:03d}_{_sanitize_filename_part(sheet_name)}"
|
||||
files = []
|
||||
for idx, media_path in enumerate(media_paths, start=1):
|
||||
if media_path not in names_in_zip:
|
||||
print(f"WARNING: {media_path} not found in {xlsx_path}", file=sys.stderr)
|
||||
continue
|
||||
ext = Path(media_path).suffix.lstrip(".").lower() or "bin"
|
||||
if ext == "jpg":
|
||||
ext = "jpeg"
|
||||
out_name = f"{safe_name}_img{idx:03d}.{ext}"
|
||||
img_dir.mkdir(parents=True, exist_ok=True)
|
||||
(img_dir / out_name).write_bytes(z.read(media_path))
|
||||
files.append(out_name)
|
||||
if files:
|
||||
written[sheet_name] = files
|
||||
return written
|
||||
|
||||
|
||||
def insert_sheet_images(markdown_text: str, sheet_images) -> str:
|
||||
"""Insert a '#### Images in this sheet' block right after each sheet's
|
||||
section (before the next '## ' heading or end of text). If a sheet has
|
||||
no images, or no '## ' headings are found at all, the text is returned
|
||||
unchanged for that part."""
|
||||
if not sheet_images:
|
||||
return markdown_text
|
||||
matches = list(_SHEET_HEADER_RE.finditer(markdown_text))
|
||||
if not matches:
|
||||
return markdown_text
|
||||
|
||||
pieces = []
|
||||
last_end = 0
|
||||
for i, m in enumerate(matches):
|
||||
sheet_name = m.group(1).removesuffix("\r")
|
||||
start = m.start()
|
||||
end = matches[i + 1].start() if i + 1 < len(matches) else len(markdown_text)
|
||||
pieces.append(markdown_text[last_end:start])
|
||||
section = markdown_text[start:end].rstrip("\n")
|
||||
images = sheet_images.get(sheet_name)
|
||||
if images:
|
||||
section += "\n\n#### Images in this sheet\n\n"
|
||||
section += "\n".join(f"" for name in images)
|
||||
pieces.append(section + "\n\n")
|
||||
last_end = end
|
||||
pieces.append(markdown_text[last_end:])
|
||||
return "".join(pieces).rstrip() + "\n"
|
||||
|
||||
|
||||
def convert_one(md, source: Path, dest_dir: Path) -> bool:
|
||||
"""Convert a single .xlsx file to a '<name>/' folder containing the
|
||||
Markdown file and an 'img/' folder of extracted images. Returns True on
|
||||
success."""
|
||||
try:
|
||||
result = md.convert(str(source))
|
||||
except Exception as exc: # noqa: BLE001 - surface any conversion error
|
||||
print(f"FAILED {source} -> {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
try:
|
||||
img_dir = dest_dir / "img"
|
||||
if dest_dir.exists():
|
||||
if img_dir.exists():
|
||||
shutil.rmtree(img_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
sheet_images = extract_images(source, img_dir)
|
||||
text = insert_sheet_images(result.text_content, sheet_images)
|
||||
md_path = dest_dir / f"{source.stem}.md"
|
||||
md_path.write_text(text, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
print(f"FAILED {source} -> could not write output in {dest_dir}: {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
img_count = sum(len(v) for v in sheet_images.values())
|
||||
img_note = f", {img_count} image(s)" if img_count else ""
|
||||
print(f"OK {source} -> {md_path}{img_note}")
|
||||
return True
|
||||
|
||||
|
||||
def find_xlsx_files(root: Path, recursive: bool):
|
||||
"""Return (xlsx_files, skipped_count) for files directly/recursively under root."""
|
||||
pattern_iter = root.rglob("*") if recursive else root.iterdir()
|
||||
xlsx_files = []
|
||||
skipped = 0
|
||||
for entry in pattern_iter:
|
||||
if entry.is_dir():
|
||||
continue
|
||||
if entry.suffix.lower() == ".xlsx":
|
||||
xlsx_files.append(entry)
|
||||
else:
|
||||
skipped += 1
|
||||
return sorted(xlsx_files), skipped
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("input", help="Path to a .xlsx file or a directory of .xlsx files")
|
||||
parser.add_argument(
|
||||
"-o", "--output",
|
||||
help=(
|
||||
"Destination folder for the '<name>/' output (single-file mode), "
|
||||
"or parent directory under which each '<name>/' output folder is "
|
||||
"created (batch mode)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--recursive", action="store_true",
|
||||
help="When input is a directory, also search subdirectories",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
#MarkItDown = _import_markitdown()
|
||||
#md = MarkItDown()
|
||||
|
||||
source = Path(args.input)
|
||||
if not source.exists():
|
||||
print(f"ERROR: Input path not found: {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
if source.is_file() and source.suffix.lower() != ".xlsx":
|
||||
print(
|
||||
f"ERROR: Unsupported file type '{source.suffix}'. "
|
||||
"This skill only converts .xlsx files.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
MarkItDown = _import_markitdown()
|
||||
md = MarkItDown()
|
||||
|
||||
if source.is_file():
|
||||
dest_dir = Path(args.output) if args.output else source.parent / source.stem
|
||||
return EXIT_OK if convert_one(md, source, dest_dir) else EXIT_CONVERSION_FAILED
|
||||
|
||||
# Directory / batch mode
|
||||
xlsx_files, skipped = find_xlsx_files(source, args.recursive)
|
||||
if skipped:
|
||||
print(f"NOTE: skipped {skipped} non-.xlsx file(s) in {source}")
|
||||
if not xlsx_files:
|
||||
print(f"ERROR: No .xlsx files found under {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
out_dir = Path(args.output) if args.output else None
|
||||
success_count = 0
|
||||
for xlsx_path in xlsx_files:
|
||||
if out_dir is not None:
|
||||
rel = xlsx_path.relative_to(source)
|
||||
dest_dir = out_dir / rel.parent / xlsx_path.stem
|
||||
else:
|
||||
dest_dir = xlsx_path.parent / xlsx_path.stem
|
||||
if convert_one(md, xlsx_path, dest_dir):
|
||||
success_count += 1
|
||||
|
||||
total = len(xlsx_files)
|
||||
print(f"\nConverted {success_count}/{total} file(s).")
|
||||
return EXIT_OK if success_count == total else EXIT_CONVERSION_FAILED
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1 @@
|
||||
markitdown[xlsx]>=0.1.0
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
name: convert-pdf-to-md
|
||||
description: 'Converts PDF (.pdf) documents into Markdown so their contents can be accurately analyzed, summarized, searched, or extracted from. Use this skill whenever the user shares, references, or asks about a .pdf file — even if they don''t say "convert" or "markdown" explicitly. This includes requests to "read", "summarize", "review", "extract data from", "compare", or "analyze" a PDF report, paper, invoice, form, contract, or scanned document. Always run the bundled conversion script to produce Markdown first; do not attempt to parse PDF content directly or write ad-hoc extraction code. Also use this skill for batch requests involving a whole folder of PDF documents. IMPORTANT: When the user references a folder or set of documents containing multiple file types (.pdf, .docx, .xlsx), invoke ALL three sibling skills — convert-pdf-to-md, convert-word-to-md, and convert-excel-to-md — so no file type is silently skipped.'
|
||||
---
|
||||
|
||||
# Convert PDF to Markdown
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Trigger this skill any time there is a `.pdf` file that needs to be
|
||||
understood or processed — for example, a user attaches a PDF and asks
|
||||
questions about it, wants a summary, wants specific data or tables pulled
|
||||
out, or wants multiple PDFs in a folder processed together. PDF is a
|
||||
layout/print format, not reliably readable as plain text, so always convert
|
||||
it to Markdown first using the script in this skill rather than trying to
|
||||
open or parse the file directly.
|
||||
|
||||
This skill only supports `.pdf` — that's MarkItDown's only PDF-family
|
||||
format, so there's no legacy format to worry about here (unlike Word's
|
||||
`.doc` or Excel's `.xls`).
|
||||
|
||||
**Mixed file types:** When the user references a folder or set of documents
|
||||
containing multiple supported file types (`.pdf`, `.docx`, `.xlsx`), this
|
||||
skill handles only `.pdf` files. The agent MUST also invoke the sibling
|
||||
skills in parallel:
|
||||
- `convert-word-to-md` for any `.docx` files
|
||||
- `convert-excel-to-md` for any `.xlsx` files
|
||||
|
||||
Never process a folder and silently skip a supported file type. All three
|
||||
skills must be invoked together when mixed types are present.
|
||||
|
||||
## Setup (once per environment)
|
||||
|
||||
Before the first conversion in a given environment, follow
|
||||
[`references/setup.md`](references/setup.md) step by step to ensure Python,
|
||||
pip, `markitdown`, and `pymupdf` (for image extraction) are installed. Do
|
||||
this proactively rather than guessing whether the environment is ready — the
|
||||
script itself will also fail with a clear pointer back to that file if a
|
||||
dependency turns out to be missing, so it's safe to just try the conversion
|
||||
first if you're reasonably confident setup was already done.
|
||||
|
||||
## Usage
|
||||
|
||||
The conversion script lives at `scripts/convert_pdf_to_md.py`.
|
||||
|
||||
**Output structure:** MarkItDown's PDF converter extracts text and tables
|
||||
only — it has no concept of embedded images at all. This script separately
|
||||
extracts real embedded images via PyMuPDF and writes a self-contained folder
|
||||
per document:
|
||||
|
||||
```
|
||||
<name>/
|
||||
img/
|
||||
page001_img001.<ext>
|
||||
page002_img001.<ext>
|
||||
...
|
||||
<name>.md
|
||||
```
|
||||
|
||||
Because MarkItDown's PDF text does not preserve reliable per-page markers,
|
||||
there's no safe way to know exactly where inline an image belongs. Rather
|
||||
than risk misplacing images next to the wrong paragraph, the script appends
|
||||
a `## Extracted Images` section at the end of the Markdown, with a
|
||||
`### Page N` subheading per page that has images — read this section
|
||||
separately from the main body text. If the document has no embedded images,
|
||||
no `img/` folder or `Extracted Images` section is created.
|
||||
|
||||
**Single file:**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_pdf_to_md.py "C:\path\to\document.pdf"
|
||||
```
|
||||
|
||||
This creates a `document\` folder next to the source file (containing
|
||||
`document.md` and, if present, `document\img\`). To control the destination
|
||||
folder explicitly:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_pdf_to_md.py "C:\path\to\document.pdf" -o "C:\path\to\output_folder"
|
||||
```
|
||||
|
||||
**A folder of PDFs (batch mode):**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_pdf_to_md.py "C:\path\to\folder"
|
||||
```
|
||||
|
||||
Add `--recursive` to also include subfolders:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_pdf_to_md.py "C:\path\to\folder" --recursive
|
||||
```
|
||||
|
||||
Each `.pdf` found gets its own `<name>\` output folder next to it by
|
||||
default. Pass `-o "C:\path\to\output_parent"` to collect all the generated
|
||||
`<name>\` folders under a separate parent directory instead (subfolder
|
||||
structure is preserved when combined with `--recursive`).
|
||||
|
||||
After conversion, read the resulting `.md` file(s) to perform the actual
|
||||
analysis the user asked for — the script's job is only to produce accurate
|
||||
Markdown (and images), not to interpret the content.
|
||||
|
||||
## Deciding where output goes
|
||||
|
||||
**Default — always output next to the source file.** The `<name>/` folder
|
||||
is created in the same directory as the source `.pdf`. This is the required
|
||||
default for every case. Do NOT override it unless the user explicitly asks
|
||||
for a different location.
|
||||
|
||||
**Only use `-o` when** the user explicitly provides an output path (e.g.,
|
||||
"save the output to `C:\output`", "put the results in `D:\work`"). Do NOT
|
||||
pass `-o` based on the agent's current working directory, the session state
|
||||
folder, or any implied location.
|
||||
|
||||
**If the source file path cannot be fully resolved** — for example, the
|
||||
user provides only a filename with no directory, or the path is ambiguous —
|
||||
use `ask_user` to confirm the full absolute path before running the
|
||||
conversion. Never guess or assume the directory.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `ModuleNotFoundError: No module named 'markitdown'` or `'fitz'` / exit code 2 | MarkItDown or PyMuPDF not installed | Follow `references/setup.md` |
|
||||
| `ERROR: Unsupported file type '...'` / exit code 3 | Not a `.pdf` file | Ask the user for the correct file, or if it's `.doc`/`.docx`/`.xlsx`, use the matching sibling skill instead |
|
||||
| `ERROR: Input path not found` / exit code 3 | Wrong path, or file moved | Confirm the correct path with the user |
|
||||
| `FAILED <file> -> ...` in batch output | That specific file is corrupt, password-protected, or otherwise unreadable | Report which file(s) failed; other files in the batch still succeed |
|
||||
| `NOTE: skipped N non-.pdf file(s)` | Folder contains non-PDF files | Expected — those files are intentionally ignored |
|
||||
| Markdown body is empty or near-empty despite images being extracted | The PDF is scanned/image-only with no embedded text layer; MarkItDown does not perform OCR | Tell the user OCR isn't supported — the extracted page images are still available for them to view |
|
||||
| Images appear in an appendix instead of inline with the text | Deliberate limitation — MarkItDown's PDF text has no reliable per-page markers to place images inline | Expected behavior; cross-reference the `### Page N` heading with the surrounding text context if needed |
|
||||
@@ -0,0 +1,71 @@
|
||||
# Environment Setup for convert-pdf-to-md
|
||||
|
||||
Follow these steps exactly, in order, before running `scripts/convert_pdf_to_md.py`
|
||||
for the first time in a given environment. Don't skip steps or improvise
|
||||
alternatives — they're written to be deterministic and safe to re-run.
|
||||
|
||||
## 1. Check Python is available (3.10+)
|
||||
|
||||
```powershell
|
||||
python --version
|
||||
```
|
||||
|
||||
- If this fails (command not found), install Python 3.10 or newer:
|
||||
- Windows: `winget install --id Python.Python.3.12 -e`
|
||||
- macOS: `brew install python@3.12`
|
||||
- Linux (Debian/Ubuntu): `sudo apt-get update && sudo apt-get install -y python3 python3-pip python-is-python3`
|
||||
- If the reported version is older than 3.10, install a newer Python using
|
||||
the same command above (MarkItDown requires 3.10+).
|
||||
|
||||
## 2. Check pip is available
|
||||
|
||||
```powershell
|
||||
python -m pip --version
|
||||
```
|
||||
|
||||
- If this fails, bootstrap pip:
|
||||
|
||||
```powershell
|
||||
python -m ensurepip --upgrade
|
||||
```
|
||||
|
||||
## 3. Install MarkItDown with PDF support, plus PyMuPDF for image extraction
|
||||
|
||||
Use the `scripts/requirements.txt` file bundled with this skill to install pinned,
|
||||
known-good versions of the dependencies:
|
||||
|
||||
```powershell
|
||||
python -m pip install -r scripts/requirements.txt
|
||||
```
|
||||
|
||||
This pulls in `markitdown[pdf]` and `pymupdf>=1.24.0`. PyMuPDF (imported as `fitz`)
|
||||
is required separately because MarkItDown's PDF
|
||||
converter only extracts text and tables — it has no support for embedded
|
||||
images at all, so this skill's script extracts them itself.
|
||||
|
||||
## 4. Verify the install
|
||||
|
||||
```powershell
|
||||
python -c "from markitdown import MarkItDown; import fitz; print('markitdown + pymupdf OK')"
|
||||
```
|
||||
|
||||
Expect to see `markitdown + pymupdf OK` printed with no errors. If you see a
|
||||
`ModuleNotFoundError`, repeat step 3 — pip may be installing into a
|
||||
different Python environment than the one being invoked (check
|
||||
`python -m pip --version` shows the same path as `python --version`'s
|
||||
interpreter).
|
||||
|
||||
## Notes
|
||||
|
||||
- This setup only needs to be done once per environment/virtual environment,
|
||||
not once per conversion.
|
||||
- `convert_pdf_to_md.py` itself also checks for `markitdown` and `fitz` at
|
||||
startup and prints a pointer back to this file if either is missing, so
|
||||
re-running setup is safe and idempotent.
|
||||
- Only `.pdf` is supported by this skill — it's MarkItDown's only PDF-family
|
||||
format, so there's no legacy-format equivalent to worry about (unlike
|
||||
Word's `.doc` or Excel's `.xls`).
|
||||
- Scanned/image-only PDFs (no embedded text layer) will produce little or
|
||||
no text from MarkItDown, since it does not perform OCR. The images
|
||||
themselves will still be extracted and appended, but the text body may be
|
||||
empty or near-empty in that case — mention this to the user if it happens.
|
||||
@@ -0,0 +1,321 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert PDF documents to Markdown using Microsoft's MarkItDown, with
|
||||
embedded images extracted to real files via PyMuPDF (MarkItDown's PDF
|
||||
converter only extracts text/tables -- it does not detect or emit anything
|
||||
for embedded images at all).
|
||||
|
||||
Usage:
|
||||
python convert_pdf_to_md.py <input> [-o OUTPUT] [--recursive]
|
||||
|
||||
<input> may be either:
|
||||
- a path to a single .pdf file, or
|
||||
- a path to a directory (batch mode: every .pdf file directly inside it
|
||||
is converted; pass --recursive to also descend into subdirectories).
|
||||
|
||||
Output:
|
||||
For each source .pdf (named "<name>.pdf"), a folder is created containing
|
||||
the Markdown and its images, in this layout:
|
||||
|
||||
<name>/
|
||||
img/
|
||||
page001_img001.<ext>
|
||||
page001_img002.<ext>
|
||||
page002_img001.<ext>
|
||||
...
|
||||
<name>.md
|
||||
|
||||
IMPORTANT: MarkItDown's PDF text extraction does not preserve reliable
|
||||
per-page markers in the returned Markdown (pages are simply joined
|
||||
together, or in some cases returned as a single unmarked block of text).
|
||||
That means there is no safe way to know exactly where, inline, an image
|
||||
should go. Rather than guess and risk misplacing an image next to the
|
||||
wrong paragraph, this script appends a clearly labeled "## Extracted
|
||||
Images" section at the end of the Markdown, with a "### Page N"
|
||||
subheading per page that contains images. This is a deliberate, honest
|
||||
tradeoff -- read the images section separately from the main body text.
|
||||
|
||||
- Single file mode: the "<name>/" folder is created next to the source
|
||||
file, or at -o/--output (treated as the exact destination folder) if
|
||||
given.
|
||||
- Batch/directory mode: a "<name>/" folder is created next to each source
|
||||
file, or under -o/--output (treated as a parent directory, created if
|
||||
missing) if given, preserving relative subfolder structure when
|
||||
--recursive is used.
|
||||
- If a document has no embedded images, no "img/" folder or "Extracted
|
||||
Images" section is created.
|
||||
|
||||
Exit codes:
|
||||
0 - all requested conversions succeeded
|
||||
1 - one or more conversions failed (partial success in batch mode)
|
||||
2 - a required dependency ("markitdown" or "pymupdf") is not installed
|
||||
3 - invalid input (path not found, or single-file input is not .pdf)
|
||||
"""
|
||||
import argparse
|
||||
import sys
|
||||
import hashlib
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_CONVERSION_FAILED = 1
|
||||
EXIT_MISSING_DEPENDENCY = 2
|
||||
EXIT_INVALID_INPUT = 3
|
||||
|
||||
|
||||
def _import_markitdown():
|
||||
"""Import MarkItDown, failing with a clear, actionable message if absent."""
|
||||
try:
|
||||
from markitdown import MarkItDown
|
||||
return MarkItDown
|
||||
except ImportError:
|
||||
print(
|
||||
"ERROR: The 'markitdown' package is not installed.\n"
|
||||
"See references/setup.md for this skill, or run:\n"
|
||||
' pip install "markitdown[pdf]"',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
|
||||
|
||||
def _import_fitz():
|
||||
"""Import PyMuPDF (module name 'fitz'), failing with a clear message if absent."""
|
||||
try:
|
||||
import fitz
|
||||
import hashlib
|
||||
return fitz
|
||||
except ImportError:
|
||||
print(
|
||||
"ERROR: The 'pymupdf' package is not installed (needed for image "
|
||||
"extraction).\nSee references/setup.md for this skill, or run:\n"
|
||||
" pip install pymupdf",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
|
||||
|
||||
def extract_images(fitz, pdf_path: Path, img_dir: Path):
|
||||
"""Extract embedded images from pdf_path, grouped by 1-based page number.
|
||||
Returns {page_num: [filename, ...]} in per-page image order. Files are
|
||||
named 'page{P:03d}_img{N:03d}.<ext>'. Corrupt/unreadable images are
|
||||
skipped with a warning rather than aborting the whole conversion.
|
||||
|
||||
Two sources are combined and deduplicated:
|
||||
1. Image XObjects via page.get_images(full=True) -- covers most embedded
|
||||
images in modern PDFs.
|
||||
2. Inline image blocks via page.get_text("dict") -- covers images stored
|
||||
directly in the page content stream, which get_images() misses entirely.
|
||||
Deduplication is by image bytes hash so the same raster is never written twice
|
||||
on the same page regardless of which source reported it."""
|
||||
written_by_page = {}
|
||||
try:
|
||||
doc = fitz.open(str(pdf_path))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"WARNING: could not open {pdf_path} for image extraction: {exc}", file=sys.stderr)
|
||||
return written_by_page
|
||||
|
||||
try:
|
||||
for page_index in range(len(doc)):
|
||||
page = doc[page_index]
|
||||
page_label = page_index + 1
|
||||
seen_hashes: set = set()
|
||||
raw_images: list[tuple[bytes, str]] = [] # (image_bytes, ext)
|
||||
|
||||
# --- Source 1: XObject images ---
|
||||
try:
|
||||
xobjects = page.get_images(full=True)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(
|
||||
f"WARNING: failed to enumerate XObject images on page {page_label} "
|
||||
f"of {pdf_path}: {exc}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
xobjects = []
|
||||
|
||||
for img in xobjects:
|
||||
xref = img[0]
|
||||
try:
|
||||
base_image = doc.extract_image(xref)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(
|
||||
f"WARNING: failed to extract XObject image xref={xref} on page "
|
||||
f"{page_label} of {pdf_path}: {exc}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
img_bytes = base_image.get("image") or b""
|
||||
if not img_bytes:
|
||||
continue
|
||||
ext = (base_image.get("ext") or "png").lower()
|
||||
raw_images.append((img_bytes, ext))
|
||||
|
||||
# --- Source 2: Inline images via get_text("dict") ---
|
||||
try:
|
||||
blocks = page.get_text("dict", flags=fitz.TEXT_PRESERVE_IMAGES).get("blocks", [])
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(
|
||||
f"WARNING: failed to extract text/image dict on page {page_label} "
|
||||
f"of {pdf_path}: {exc}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
blocks = []
|
||||
|
||||
for block in blocks:
|
||||
# Image blocks have type == 1
|
||||
if block.get("type") != 1:
|
||||
continue
|
||||
img_bytes = block.get("image") or b""
|
||||
if not img_bytes:
|
||||
continue
|
||||
# Derive extension from the block's "ext" key (fitz sets this)
|
||||
ext = (block.get("ext") or "png").lower()
|
||||
raw_images.append((img_bytes, ext))
|
||||
|
||||
# --- Write deduplicated images ---
|
||||
page_files = []
|
||||
img_idx = 1
|
||||
for img_bytes, ext in raw_images:
|
||||
h = hashlib.sha256(img_bytes).digest()
|
||||
if h in seen_hashes:
|
||||
continue
|
||||
seen_hashes.add(h)
|
||||
out_name = f"page{page_label:03d}_img{img_idx:03d}.{ext}"
|
||||
img_dir.mkdir(parents=True, exist_ok=True)
|
||||
(img_dir / out_name).write_bytes(img_bytes)
|
||||
page_files.append(out_name)
|
||||
img_idx += 1
|
||||
|
||||
if page_files:
|
||||
written_by_page[page_label] = page_files
|
||||
finally:
|
||||
doc.close()
|
||||
|
||||
return written_by_page
|
||||
|
||||
|
||||
def build_image_appendix(written_by_page) -> str:
|
||||
"""Build the '## Extracted Images' appendix text. Returns "" if empty."""
|
||||
if not written_by_page:
|
||||
return ""
|
||||
lines = ["", "## Extracted Images", ""]
|
||||
for page_num in sorted(written_by_page):
|
||||
lines.append(f"### Page {page_num}")
|
||||
lines.append("")
|
||||
for name in written_by_page[page_num]:
|
||||
lines.append(f"")
|
||||
lines.append("")
|
||||
return "\n".join(lines).rstrip() + "\n"
|
||||
|
||||
|
||||
def convert_one(md, fitz, source: Path, dest_dir: Path) -> bool:
|
||||
"""Convert a single .pdf file to a '<name>/' folder containing the
|
||||
Markdown file and an 'img/' folder of extracted images. Returns True on
|
||||
success."""
|
||||
try:
|
||||
result = md.convert(str(source))
|
||||
except Exception as exc: # noqa: BLE001 - surface any conversion error
|
||||
print(f"FAILED {source} -> {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
try:
|
||||
if dest_dir.exists():
|
||||
shutil.rmtree(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
written_by_page = extract_images(fitz, source, dest_dir / "img")
|
||||
appendix = build_image_appendix(written_by_page)
|
||||
text = result.text_content.rstrip("\n")
|
||||
full_text = f"{text}\n{appendix}" if appendix else f"{text}\n"
|
||||
md_path = dest_dir / f"{source.stem}.md"
|
||||
md_path.write_text(full_text, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
print(f"FAILED {source} -> could not write output in {dest_dir}: {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
img_count = sum(len(v) for v in written_by_page.values())
|
||||
img_note = f", {img_count} image(s)" if img_count else ""
|
||||
print(f"OK {source} -> {md_path}{img_note}")
|
||||
return True
|
||||
|
||||
|
||||
def find_pdf_files(root: Path, recursive: bool):
|
||||
"""Return (pdf_files, skipped_count) for files directly/recursively under root."""
|
||||
pattern_iter = root.rglob("*") if recursive else root.iterdir()
|
||||
pdf_files = []
|
||||
skipped = 0
|
||||
for entry in pattern_iter:
|
||||
if entry.is_dir():
|
||||
continue
|
||||
if entry.suffix.lower() == ".pdf":
|
||||
pdf_files.append(entry)
|
||||
else:
|
||||
skipped += 1
|
||||
return sorted(pdf_files), skipped
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("input", help="Path to a .pdf file or a directory of .pdf files")
|
||||
parser.add_argument(
|
||||
"-o", "--output",
|
||||
help=(
|
||||
"Destination folder for the '<name>/' output (single-file mode), "
|
||||
"or parent directory under which each '<name>/' output folder is "
|
||||
"created (batch mode)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--recursive", action="store_true",
|
||||
help="When input is a directory, also search subdirectories",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
#MarkItDown = _import_markitdown()
|
||||
#fitz = _import_fitz()
|
||||
#md = MarkItDown()
|
||||
|
||||
source = Path(args.input)
|
||||
if not source.exists():
|
||||
print(f"ERROR: Input path not found: {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
if source.is_file() and source.suffix.lower() != ".pdf":
|
||||
print(
|
||||
f"ERROR: Unsupported file type '{source.suffix}'. "
|
||||
"This skill only converts .pdf files.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
MarkItDown = _import_markitdown()
|
||||
fitz = _import_fitz()
|
||||
md = MarkItDown()
|
||||
|
||||
if source.is_file():
|
||||
dest_dir = Path(args.output) if args.output else source.parent / source.stem
|
||||
return EXIT_OK if convert_one(md, fitz, source, dest_dir) else EXIT_CONVERSION_FAILED
|
||||
|
||||
# Directory / batch mode
|
||||
pdf_files, skipped = find_pdf_files(source, args.recursive)
|
||||
if skipped:
|
||||
print(f"NOTE: skipped {skipped} non-.pdf file(s) in {source}")
|
||||
if not pdf_files:
|
||||
print(f"ERROR: No .pdf files found under {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
out_dir = Path(args.output) if args.output else None
|
||||
success_count = 0
|
||||
for pdf_path in pdf_files:
|
||||
if out_dir is not None:
|
||||
rel = pdf_path.relative_to(source)
|
||||
dest_dir = out_dir / rel.parent / pdf_path.stem
|
||||
else:
|
||||
dest_dir = pdf_path.parent / pdf_path.stem
|
||||
if convert_one(md, fitz, pdf_path, dest_dir):
|
||||
success_count += 1
|
||||
|
||||
total = len(pdf_files)
|
||||
print(f"\nConverted {success_count}/{total} file(s).")
|
||||
return EXIT_OK if success_count == total else EXIT_CONVERSION_FAILED
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,2 @@
|
||||
markitdown[pdf]>=0.1.0
|
||||
pymupdf>=1.24.0
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
name: convert-word-to-md
|
||||
description: 'Converts Word (.docx) documents into Markdown so their contents can be accurately analyzed, summarized, searched, or extracted from. Use this skill whenever the user shares, references, or asks about a .docx file — even if they don''t say "convert" or "markdown" explicitly. This includes requests to "read", "summarize", "review", "extract data from", "compare", or "analyze" a Word document, resume, report, contract, or proposal. Always run the bundled conversion script to produce Markdown first; do not attempt to parse .docx content directly or write ad-hoc conversion code. Also use this skill for batch requests involving a whole folder of Word documents. IMPORTANT: When the user references a folder or set of documents containing multiple file types (.pdf, .docx, .xlsx), invoke ALL three sibling skills — convert-pdf-to-md, convert-word-to-md, and convert-excel-to-md — so no file type is silently skipped.'
|
||||
---
|
||||
|
||||
# Convert Word to Markdown
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Trigger this skill any time there is a `.docx` file that needs to be
|
||||
understood or processed — for example, a user attaches a Word document and
|
||||
asks questions about it, wants a summary, wants specific data pulled out, or
|
||||
wants multiple Word documents in a folder processed together. Word's native
|
||||
`.docx` format is a zipped XML bundle that is not reliably readable as plain
|
||||
text, so always convert it to Markdown first using the script in this
|
||||
skill rather than trying to open or parse the file directly.
|
||||
|
||||
This skill only supports `.docx`. If asked to convert a legacy `.doc` file,
|
||||
tell the user it isn't supported and ask them to re-save it as `.docx`
|
||||
(Word: File > Save As > Word Document (.docx)) first.
|
||||
|
||||
**Mixed file types:** When the user references a folder or set of documents
|
||||
containing multiple supported file types (`.pdf`, `.docx`, `.xlsx`), this
|
||||
skill handles only `.docx` files. The agent MUST also invoke the sibling
|
||||
skills in parallel:
|
||||
- `convert-pdf-to-md` for any `.pdf` files
|
||||
- `convert-excel-to-md` for any `.xlsx` files
|
||||
|
||||
Never process a folder and silently skip a supported file type. All three
|
||||
skills must be invoked together when mixed types are present.
|
||||
|
||||
## Setup (once per environment)
|
||||
|
||||
Before the first conversion in a given environment, follow
|
||||
[`references/setup.md`](references/setup.md) step by step to ensure Python,
|
||||
pip, and the `markitdown` package are installed. Do this proactively rather
|
||||
than guessing whether the environment is ready — the script itself will
|
||||
also fail with a clear pointer back to that file if `markitdown` turns out
|
||||
to be missing, so it's safe to just try the conversion first if you're
|
||||
reasonably confident setup was already done.
|
||||
|
||||
## Usage
|
||||
|
||||
The conversion script lives at `scripts/convert_word_to_md.py`.
|
||||
|
||||
**Output structure:** MarkItDown embeds images as a truncated `data:image/png;base64...` URI
|
||||
placeholder (not real image data), so the script
|
||||
extracts real images directly from the `.docx` and writes a self-contained
|
||||
folder per document instead of a single loose `.md` file:
|
||||
|
||||
```
|
||||
<name>/
|
||||
img/
|
||||
img001.<ext>
|
||||
img002.<ext>
|
||||
...
|
||||
<name>.md (image references are relative: img/imgNNN.ext)
|
||||
```
|
||||
|
||||
If the document has no embedded images, no `img/` folder is created.
|
||||
|
||||
**Single file:**
|
||||
|
||||
```powershell
|
||||
# Windows
|
||||
python scripts\convert_word_to_md.py "C:\path\to\document.docx"
|
||||
```
|
||||
|
||||
```bash
|
||||
# macOS / Linux
|
||||
python scripts/convert_word_to_md.py "/path/to/document.docx"
|
||||
```
|
||||
|
||||
This creates a `document\` folder next to the source file (containing
|
||||
`document.md` and, if present, `document\img\`). To control the destination
|
||||
folder explicitly:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_word_to_md.py "C:\path\to\document.docx" -o "C:\path\to\output_folder"
|
||||
```
|
||||
|
||||
**A folder of Word documents (batch mode):**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_word_to_md.py "C:\path\to\folder"
|
||||
```
|
||||
|
||||
Add `--recursive` to also include subfolders:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_word_to_md.py "C:\path\to\folder" --recursive
|
||||
```
|
||||
|
||||
Each `.docx` found gets its own `<name>\` output folder next to it by
|
||||
default. Pass `-o "C:\path\to\output_parent"` to collect all the generated
|
||||
`<name>\` folders under a separate parent directory instead (subfolder
|
||||
structure is preserved when combined with `--recursive`).
|
||||
|
||||
After conversion, read the resulting `.md` file(s) to perform the actual
|
||||
analysis the user asked for — the script's job is only to produce accurate
|
||||
Markdown (and images), not to interpret the content.
|
||||
|
||||
## Deciding where output goes
|
||||
|
||||
**Default — always output next to the source file.** The `<name>/` folder
|
||||
is created in the same directory as the source `.docx`. This is the required
|
||||
default for every case. Do NOT override it unless the user explicitly asks
|
||||
for a different location.
|
||||
|
||||
**Only use `-o` when** the user explicitly provides an output path (e.g.,
|
||||
"save the output to `C:\output`", "put the results in `D:\work`"). Do NOT
|
||||
pass `-o` based on the agent's current working directory, the session state
|
||||
folder, or any implied location.
|
||||
|
||||
**If the source file path cannot be fully resolved** — for example, the
|
||||
user provides only a filename with no directory, or the path is ambiguous —
|
||||
use `ask_user` to confirm the full absolute path before running the
|
||||
conversion. Never guess or assume the directory.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `ModuleNotFoundError: No module named 'markitdown'` / exit code 2 | MarkItDown not installed | Follow `references/setup.md` |
|
||||
| `ERROR: Unsupported file type '.doc'` / exit code 3 | Legacy `.doc`, not `.docx` | Ask the user to re-save as `.docx` |
|
||||
| `ERROR: Input path not found` / exit code 3 | Wrong path, or file moved | Confirm the correct path with the user |
|
||||
| `FAILED <file> -> ...` in batch output | That specific file is corrupt, password-protected, or otherwise unreadable | Report which file(s) failed; other files in the batch still succeed |
|
||||
| `NOTE: skipped N non-.docx file(s)` | Folder contains non-Word files | Expected — those files are intentionally ignored |
|
||||
| `WARNING: found N image placeholder(s) ... but extracted M image file(s)` | Mismatch between MarkItDown's placeholder count and images found in `word/media/` (unusual/malformed docx) | Placeholders are left unreplaced rather than risk wrong images; inspect the source file's media manually if images are needed |
|
||||
@@ -0,0 +1,66 @@
|
||||
# Environment Setup for convert-word-to-md
|
||||
|
||||
Follow these steps exactly, in order, before running `scripts/convert_word_to_md.py`
|
||||
for the first time in a given environment. Don't skip steps or improvise
|
||||
alternatives — they're written to be deterministic and safe to re-run.
|
||||
|
||||
## 1. Check Python is available (3.10+)
|
||||
|
||||
```powershell
|
||||
python --version
|
||||
```
|
||||
|
||||
- If this fails (command not found), install Python 3.10 or newer:
|
||||
- Windows: `winget install --id Python.Python.3.12 -e`
|
||||
- macOS: `brew install python@3.12`
|
||||
- Linux (Debian/Ubuntu): `sudo apt-get update && sudo apt-get install -y python3 python3-pip python-is-python3`
|
||||
- If the reported version is older than 3.10, install a newer Python using
|
||||
the same command above (MarkItDown requires 3.10+).
|
||||
|
||||
## 2. Check pip is available
|
||||
|
||||
```powershell
|
||||
python -m pip --version
|
||||
```
|
||||
|
||||
- If this fails, bootstrap pip:
|
||||
|
||||
```powershell
|
||||
python -m ensurepip --upgrade
|
||||
```
|
||||
|
||||
## 3. Install MarkItDown with Word (.docx) support
|
||||
|
||||
Use the `scripts/requirements.txt` file bundled with this skill to install a pinned,
|
||||
known-good version of the dependency:
|
||||
|
||||
```powershell
|
||||
python -m pip install -r scripts/requirements.txt
|
||||
```
|
||||
|
||||
This pulls in `markitdown[docx]` (MarkItDown's Word conversion dependency, which
|
||||
includes `mammoth` for `.docx` file parsing). No extra package is needed — this
|
||||
skill's script uses MarkItDown's built-in Word converter.
|
||||
|
||||
## 4. Verify the install
|
||||
|
||||
```powershell
|
||||
python -c "from markitdown import MarkItDown; print('markitdown OK')"
|
||||
```
|
||||
|
||||
Expect to see `markitdown OK` printed with no errors. If you see
|
||||
`ModuleNotFoundError: No module named 'markitdown'`, repeat step 3 — pip may
|
||||
be installing into a different Python environment than the one being
|
||||
invoked (check `python -m pip --version` shows the same path as `python
|
||||
--version`'s interpreter).
|
||||
|
||||
## Notes
|
||||
|
||||
- This setup only needs to be done once per environment/virtual environment,
|
||||
not once per conversion.
|
||||
- `convert_word_to_md.py` itself also checks for `markitdown` at startup and
|
||||
prints a pointer back to this file if it's missing, so re-running setup is
|
||||
safe and idempotent.
|
||||
- Only `.docx` is supported by this skill. Legacy binary `.doc` files are
|
||||
out of scope — ask the user to re-save the file as `.docx` (e.g., via
|
||||
Word's "Save As") if one is encountered.
|
||||
@@ -0,0 +1,301 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert Word (.docx) documents to Markdown using Microsoft's MarkItDown,
|
||||
with embedded images extracted to real files (MarkItDown only emits a
|
||||
truncated `data:image/...;base64...` placeholder, not real image data).
|
||||
|
||||
Usage:
|
||||
python convert_word_to_md.py <input> [-o OUTPUT] [--recursive]
|
||||
|
||||
<input> may be either:
|
||||
- a path to a single .docx file, or
|
||||
- a path to a directory (batch mode: every .docx file directly inside it
|
||||
is converted; pass --recursive to also descend into subdirectories).
|
||||
|
||||
Output:
|
||||
For each source .docx (named "<name>.docx"), a folder is created
|
||||
containing the Markdown and its images, in this layout:
|
||||
|
||||
<name>/
|
||||
img/
|
||||
img001.<ext>
|
||||
img002.<ext>
|
||||
...
|
||||
<name>.md (image references are relative: img/imgNNN.ext)
|
||||
|
||||
- Single file mode: the "<name>/" folder is created next to the source
|
||||
file, or at -o/--output (treated as the exact destination folder) if
|
||||
given.
|
||||
- Batch/directory mode: a "<name>/" folder is created next to each source
|
||||
file, or under -o/--output (treated as a parent directory, created if
|
||||
missing) if given, preserving relative subfolder structure when
|
||||
--recursive is used.
|
||||
- If a document has no embedded images, no "img/" folder is created.
|
||||
|
||||
Exit codes:
|
||||
0 - all requested conversions succeeded
|
||||
1 - one or more conversions failed (partial success in batch mode)
|
||||
2 - required dependency ("markitdown") is not installed
|
||||
3 - invalid input (path not found, or single-file input is not .docx)
|
||||
"""
|
||||
import argparse
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from xml.etree import ElementTree as ET
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_CONVERSION_FAILED = 1
|
||||
EXIT_MISSING_DEPENDENCY = 2
|
||||
EXIT_INVALID_INPUT = 3
|
||||
|
||||
_W_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
||||
_R_NS = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"
|
||||
_REL_NS = "http://schemas.openxmlformats.org/package/2006/relationships"
|
||||
|
||||
# MarkItDown embeds images as a literal truncated placeholder, e.g.
|
||||
#  -- NOT real base64 data. This pattern
|
||||
# matches that placeholder so it can be swapped for a real relative path.
|
||||
_PLACEHOLDER_IMAGE_RE = re.compile(
|
||||
r'!\[([^\]]*)\]\(data:image/[a-zA-Z0-9.+-]+;base64[^)]*\)'
|
||||
)
|
||||
|
||||
|
||||
def _import_markitdown():
|
||||
"""Import MarkItDown, failing with a clear, actionable message if absent."""
|
||||
try:
|
||||
from markitdown import MarkItDown
|
||||
return MarkItDown
|
||||
except ImportError:
|
||||
print(
|
||||
"ERROR: The 'markitdown' package is not installed.\n"
|
||||
"See references/setup.md for this skill, or run:\n"
|
||||
' pip install "markitdown[docx]"',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
|
||||
|
||||
def _document_order_media(docx_path: Path):
|
||||
"""Return [(rel_id, media_zip_path), ...] in the order images appear in
|
||||
word/document.xml (via r:embed / r:id), resolved through
|
||||
word/_rels/document.xml.rels. Returns [] if the document has no body
|
||||
part or no images (e.g. malformed docx falls back gracefully)."""
|
||||
try:
|
||||
with zipfile.ZipFile(docx_path) as z:
|
||||
if "word/document.xml" not in z.namelist() or \
|
||||
"word/_rels/document.xml.rels" not in z.namelist():
|
||||
return []
|
||||
rels_xml = z.read("word/_rels/document.xml.rels")
|
||||
doc_xml = z.read("word/document.xml")
|
||||
except (zipfile.BadZipFile, KeyError, OSError):
|
||||
return []
|
||||
|
||||
try:
|
||||
rels_root = ET.fromstring(rels_xml)
|
||||
doc_root = ET.fromstring(doc_xml)
|
||||
except ET.ParseError:
|
||||
return []
|
||||
|
||||
rel_map = {}
|
||||
for rel in rels_root.findall(f"{{{_REL_NS}}}Relationship"):
|
||||
rel_map[rel.get("Id")] = rel.get("Target")
|
||||
|
||||
ordered_rel_ids = []
|
||||
for elem in doc_root.iter():
|
||||
tag = elem.tag.rsplit("}", 1)[-1]
|
||||
if tag == "blip":
|
||||
rid = elem.get(f"{{{_R_NS}}}embed")
|
||||
elif tag == "imagedata":
|
||||
rid = elem.get(f"{{{_R_NS}}}id")
|
||||
else:
|
||||
rid = None
|
||||
if rid:
|
||||
ordered_rel_ids.append(rid)
|
||||
ordered_media = []
|
||||
for rid in ordered_rel_ids:
|
||||
target = rel_map.get(rid)
|
||||
if not target or "media/" not in target:
|
||||
continue
|
||||
import posixpath
|
||||
media_path = (
|
||||
target.lstrip("/")
|
||||
if target.startswith("/")
|
||||
else posixpath.normpath(
|
||||
target if target.startswith("word/") else posixpath.join("word", target)
|
||||
)
|
||||
)
|
||||
ordered_media.append((rid, media_path))
|
||||
return ordered_media
|
||||
|
||||
|
||||
def _extract_images(docx_path: Path, img_dir: Path):
|
||||
"""Extract embedded images from docx_path into img_dir as img001.ext,
|
||||
img002.ext, ... in document order. Returns the list of written filenames
|
||||
(relative to img_dir), in that same order."""
|
||||
ordered_media = _document_order_media(docx_path)
|
||||
if not ordered_media:
|
||||
return []
|
||||
|
||||
written = []
|
||||
with zipfile.ZipFile(docx_path) as z:
|
||||
names_in_zip = set(z.namelist())
|
||||
for idx, (rid, media_path) in enumerate(ordered_media, start=1):
|
||||
if media_path not in names_in_zip:
|
||||
print(f"WARNING: {media_path} (rel {rid}) not found in {docx_path}", file=sys.stderr)
|
||||
continue
|
||||
ext = Path(media_path).suffix.lstrip(".").lower() or "bin"
|
||||
if ext == "jpg":
|
||||
ext = "jpeg"
|
||||
out_name = f"img{idx:03d}.{ext}"
|
||||
img_dir.mkdir(parents=True, exist_ok=True)
|
||||
(img_dir / out_name).write_bytes(z.read(media_path))
|
||||
written.append(out_name)
|
||||
return written
|
||||
|
||||
|
||||
def _rewrite_image_refs(markdown_text: str, image_files) -> str:
|
||||
"""Replace MarkItDown's truncated base64 image placeholders with real
|
||||
relative img/imgNNN.ext references, in left-to-right order. If the
|
||||
counts don't match (unexpected), the placeholders are left as-is rather
|
||||
than risk mismatched references."""
|
||||
matches = list(_PLACEHOLDER_IMAGE_RE.finditer(markdown_text))
|
||||
if not matches:
|
||||
return markdown_text
|
||||
if len(matches) != len(image_files):
|
||||
print(
|
||||
f"WARNING: found {len(matches)} image placeholder(s) in markdown but "
|
||||
f"extracted {len(image_files)} image file(s); leaving placeholders "
|
||||
"unreplaced to avoid mismatched references.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return markdown_text
|
||||
|
||||
counter = {"i": 0}
|
||||
|
||||
def _replace(m):
|
||||
name = image_files[counter["i"]]
|
||||
counter["i"] += 1
|
||||
return f""
|
||||
|
||||
return _PLACEHOLDER_IMAGE_RE.sub(_replace, markdown_text)
|
||||
|
||||
|
||||
def convert_one(md, source: Path, dest_dir: Path) -> bool:
|
||||
"""Convert a single .docx file to a "<name>/" folder containing the
|
||||
Markdown file and an "img/" folder of extracted images. Returns True on
|
||||
success."""
|
||||
try:
|
||||
result = md.convert(str(source))
|
||||
except ImportError as exc:
|
||||
print(
|
||||
f"ERROR: A required dependency for converting '{source.name}' is not installed.\n"
|
||||
f" {exc}\n"
|
||||
"See references/setup.md for this skill, or run:\n"
|
||||
' pip install "markitdown[docx]"',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
except Exception as exc: # noqa: BLE001 - surface any conversion error
|
||||
print(f"FAILED {source} -> {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
try:
|
||||
if dest_dir.exists():
|
||||
shutil.rmtree(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
image_files = _extract_images(source, dest_dir / "img")
|
||||
text = _rewrite_image_refs(result.text_content, image_files)
|
||||
md_path = dest_dir / f"{source.stem}.md"
|
||||
md_path.write_text(text, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
print(f"FAILED {source} -> could not write output in {dest_dir}: {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
img_note = f", {len(image_files)} image(s)" if image_files else ""
|
||||
print(f"OK {source} -> {md_path}{img_note}")
|
||||
return True
|
||||
|
||||
|
||||
def find_docx_files(root: Path, recursive: bool):
|
||||
"""Return (docx_files, skipped_count) for files directly/recursively under root."""
|
||||
pattern_iter = root.rglob("*") if recursive else root.iterdir()
|
||||
docx_files = []
|
||||
skipped = 0
|
||||
for entry in pattern_iter:
|
||||
if entry.is_dir():
|
||||
continue
|
||||
if entry.suffix.lower() == ".docx":
|
||||
docx_files.append(entry)
|
||||
else:
|
||||
skipped += 1
|
||||
return sorted(docx_files), skipped
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("input", help="Path to a .docx file or a directory of .docx files")
|
||||
parser.add_argument(
|
||||
"-o", "--output",
|
||||
help=(
|
||||
"Destination folder for the '<name>/' output (single-file mode), "
|
||||
"or parent directory under which each '<name>/' output folder is "
|
||||
"created (batch mode)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--recursive", action="store_true",
|
||||
help="When input is a directory, also search subdirectories",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
#MarkItDown = _import_markitdown()
|
||||
#md = MarkItDown()
|
||||
|
||||
source = Path(args.input)
|
||||
if not source.exists():
|
||||
print(f"ERROR: Input path not found: {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
if source.is_file() and source.suffix.lower() != ".docx":
|
||||
print(
|
||||
f"ERROR: Unsupported file type '{source.suffix}'. "
|
||||
"This skill only converts .docx files.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
MarkItDown = _import_markitdown()
|
||||
md = MarkItDown()
|
||||
|
||||
if source.is_file():
|
||||
dest_dir = Path(args.output) if args.output else source.parent / source.stem
|
||||
return EXIT_OK if convert_one(md, source, dest_dir) else EXIT_CONVERSION_FAILED
|
||||
|
||||
# Directory / batch mode
|
||||
docx_files, skipped = find_docx_files(source, args.recursive)
|
||||
if skipped:
|
||||
print(f"NOTE: skipped {skipped} non-.docx file(s) in {source}")
|
||||
if not docx_files:
|
||||
print(f"ERROR: No .docx files found under {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
out_dir = Path(args.output) if args.output else None
|
||||
success_count = 0
|
||||
for docx_path in docx_files:
|
||||
if out_dir is not None:
|
||||
rel = docx_path.relative_to(source)
|
||||
dest_dir = out_dir / rel.parent / docx_path.stem
|
||||
else:
|
||||
dest_dir = docx_path.parent / docx_path.stem
|
||||
if convert_one(md, docx_path, dest_dir):
|
||||
success_count += 1
|
||||
|
||||
total = len(docx_files)
|
||||
print(f"\nConverted {success_count}/{total} file(s).")
|
||||
return EXIT_OK if success_count == total else EXIT_CONVERSION_FAILED
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1 @@
|
||||
markitdown[docx]>=0.1.0
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
name: desk-journal
|
||||
description: 'Write, append, or read desk journal entries. The journal is persistent memory — what survives session boundaries. A good entry has: what was done, current state, next step.'
|
||||
---
|
||||
|
||||
# Desk Journal
|
||||
|
||||
Manage a desk's journal — the persistent memory that survives
|
||||
session boundaries.
|
||||
|
||||
## When to use
|
||||
|
||||
- **End of session:** Write what was done, current state, next step
|
||||
- **Start of session:** Read the journal to pick up where you left off
|
||||
- **Mid-session checkpoint:** Note significant progress or decisions
|
||||
- **Desk wind-down:** Write a final summary when a desk is being closed
|
||||
|
||||
## How to write a journal entry
|
||||
|
||||
Append to `desks/<desk-name>/journal.md`. Each entry is a section:
|
||||
|
||||
```markdown
|
||||
## <date> — <short summary>
|
||||
- **Worked on:** <what was done this session>
|
||||
- **Current state:** <where things stand right now>
|
||||
- **Next step:** <what the next session should pick up>
|
||||
```
|
||||
|
||||
### Guidelines
|
||||
|
||||
- **Be specific.** "Worked on security scanning" is useless to the
|
||||
next session. "Scanned repos A, B, C for CWE-502; found 3
|
||||
findings in A, 0 in B and C; findings triaged to bench" — that's
|
||||
a trail.
|
||||
- **Include what didn't work.** Dead ends are valuable — they prevent
|
||||
the next session from walking the same path.
|
||||
- **Keep it short.** The journal is a trail marker, not a diary.
|
||||
3-5 lines per entry. If you need more, the important context
|
||||
should go on the bench as a separate artifact.
|
||||
- **Always include next step.** The next session starts from zero.
|
||||
Without a next step, it has to re-derive everything.
|
||||
|
||||
## End-of-desk entry
|
||||
|
||||
When a desk is being wound down (not just a session ending, but
|
||||
the desk itself closing):
|
||||
|
||||
```markdown
|
||||
## <date> — Desk closed
|
||||
- **Summary:** <what this desk accomplished overall>
|
||||
- **Artifacts:** <what's on the bench from this desk>
|
||||
- **Handoff:** <anything another desk or the operator needs to know>
|
||||
```
|
||||
|
||||
## Reading the journal
|
||||
|
||||
At session start, read the desk's journal to pick up context.
|
||||
The most recent entry is the most important — it has the current
|
||||
state and next step. Earlier entries provide history if needed.
|
||||
|
||||
## Principles
|
||||
|
||||
- The journal is a cairn — stones left so the next traveler finds
|
||||
the way. Every entry is a stone.
|
||||
- Honesty over completeness. "I got stuck on X and don't know why"
|
||||
is more useful than silence.
|
||||
- The journal is for the next session, not for the current one.
|
||||
Write for someone who knows nothing about what you just did.
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
name: desk-open
|
||||
description: 'Create and open a new desk in the workshop. Sets up the folder structure, initial journal, and desk identity so the next session that sits down finds the trail.'
|
||||
---
|
||||
|
||||
# Open a Desk
|
||||
|
||||
Create a new desk in the workshop with the standard structure.
|
||||
|
||||
## When to use
|
||||
|
||||
- The operator wants to start a new workstream
|
||||
- Work arrives that doesn't belong to any existing desk
|
||||
- A topic needs its own frame (its own history, its own priors)
|
||||
|
||||
## What it creates
|
||||
|
||||
Given a workshop directory and a desk name, create:
|
||||
|
||||
```
|
||||
desks/<desk-name>/
|
||||
journal.md # persistent memory — read at start, written at end
|
||||
.signals/ # structured signal output (JSON) — dashboard reads this
|
||||
```
|
||||
|
||||
## How to use
|
||||
|
||||
1. **Choose a name.** Short, descriptive, kebab-case. The name is
|
||||
how the operator and other desks refer to this desk.
|
||||
Examples: `security-scan`, `api-review`, `ops`, `cloud-workshop`
|
||||
|
||||
2. **Check if it already exists.** If `desks/<desk-name>/` already
|
||||
has a `journal.md`, the desk is live — **do not overwrite it.**
|
||||
Instead, resume it: read the journal and continue from where it
|
||||
left off. If the operator explicitly wants a fresh start, they
|
||||
must rename or archive the existing desk first.
|
||||
|
||||
3. **Create the structure.** Make the directory, initial journal,
|
||||
and signals folder:
|
||||
|
||||
```
|
||||
desks/<desk-name>/journal.md
|
||||
desks/<desk-name>/.signals/
|
||||
```
|
||||
|
||||
4. **Write the first journal entry.** The journal starts with:
|
||||
- What this desk is for (its focus/purpose)
|
||||
- What repos or work it covers (if applicable)
|
||||
- Any initial context the first session needs
|
||||
|
||||
5. **Announce it.** Tell the operator what was created and what
|
||||
the desk's focus is.
|
||||
|
||||
## Session orientation
|
||||
|
||||
This skill initializes storage — it does not launch a session.
|
||||
A desk becomes active when a Copilot session references its
|
||||
directory. The session workflow:
|
||||
|
||||
1. The operator (or TA) starts a session and says "sit at the
|
||||
`<desk-name>` desk"
|
||||
2. The session reads `desks/<desk-name>/journal.md` to load priors
|
||||
3. Work happens — the session uses `signal-write` to emit signals
|
||||
and `desk-journal` to persist state at the end
|
||||
4. The next session repeats from step 2
|
||||
|
||||
The desk identity comes from which journal is read, not from a
|
||||
persistent process. Desks are long-running in *state* (the journal
|
||||
carries forward), not in *runtime* (each session is independent).
|
||||
|
||||
## Journal format
|
||||
|
||||
```markdown
|
||||
# <Desk Name> — Journal
|
||||
|
||||
## <date> — Desk opened
|
||||
- **Purpose:** <what this desk focuses on>
|
||||
- **Scope:** <repos, areas, or work this desk covers>
|
||||
- **Next step:** <what the first session should do>
|
||||
```
|
||||
|
||||
## Principles
|
||||
|
||||
- A desk is a peer, not a sub-agent. It has equal standing to
|
||||
disagree with other desks.
|
||||
- The journal is the memory. Without it, the next session starts
|
||||
blind. Write enough that someone starting from zero finds the way.
|
||||
- One desk, one focus. If the scope is too broad, open two desks.
|
||||
Each desk's value comes from its specific frame — dilute the
|
||||
frame and you lose the value.
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
name: markstream-install
|
||||
description: 'Install and configure Markstream streaming Markdown renderers for Vue, React, Svelte, Angular, Nuxt, and Vue 2 applications. Use for package selection, minimal peer dependencies, CSS order, SSR boundaries, streaming mode, and renderer setup.'
|
||||
license: MIT
|
||||
compatibility: 'JavaScript or TypeScript frontend project using Vue 3, Nuxt 3/4, Vue 2.6/2.7, React 18+, Next.js, Angular 20+, or Svelte 5.'
|
||||
metadata:
|
||||
source: https://github.com/Simon-He95/markstream-vue
|
||||
documentation: https://markstream.simonhe.me/
|
||||
---
|
||||
|
||||
# Markstream Install
|
||||
|
||||
Integrate the appropriate [Markstream](https://github.com/Simon-He95/markstream-vue) package into an existing application without installing unnecessary optional dependencies or weakening its security defaults.
|
||||
|
||||
Read [references/scenarios.md](references/scenarios.md) before choosing packages or peers.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when the user asks to:
|
||||
|
||||
- add streaming Markdown rendering to an AI chat or document interface;
|
||||
- install Markstream in Vue, Nuxt, React, Next.js, Svelte, Angular, or Vue 2;
|
||||
- repair a broken Markstream installation, missing styles, or SSR failure;
|
||||
- replace another Markdown renderer with Markstream;
|
||||
- choose between static, smooth-streaming, and externally parsed AST input.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Inspect the host application
|
||||
|
||||
Before changing dependencies, inspect:
|
||||
|
||||
- the framework and version in `package.json`;
|
||||
- the package manager lockfile;
|
||||
- whether the application uses SSR;
|
||||
- reset, Tailwind, UnoCSS, or design-system styles;
|
||||
- required optional features: code highlighting, enhanced File/Diff surfaces, Monaco, Mermaid, D2, infographic blocks, or KaTeX.
|
||||
|
||||
Do not assume the Vue package is correct merely because the source repository is named `markstream-vue`. Select the framework-specific package from the scenario table.
|
||||
|
||||
### 2. Install the smallest dependency set
|
||||
|
||||
Install exactly one framework package. Add optional peers only when the requested UI uses their feature.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
npm install markstream-vue
|
||||
npm install markstream-react
|
||||
npm install markstream-svelte
|
||||
npm install markstream-angular
|
||||
npm install markstream-vue2
|
||||
```
|
||||
|
||||
Preserve the repository's existing package manager. Do not install every optional peer preemptively.
|
||||
|
||||
### 3. Wire styles in the correct order
|
||||
|
||||
Import application resets before Markstream styles. Import package CSS explicitly; do not rely on component imports to inject it.
|
||||
|
||||
For Tailwind or UnoCSS, use the relevant package subpath in a component layer:
|
||||
|
||||
```css
|
||||
@import 'markstream-vue/index.css' layer(components);
|
||||
```
|
||||
|
||||
Use the matching package name for React, Svelte, Angular, or Vue 2. If math rendering is enabled, also import:
|
||||
|
||||
```css
|
||||
@import 'katex/dist/katex.min.css';
|
||||
```
|
||||
|
||||
Vue CLI 4 and other Webpack 4-based Vue 2 applications cannot resolve package export maps. In those projects, import the published file directly:
|
||||
|
||||
```ts
|
||||
import 'markstream-vue2/dist/index.css'
|
||||
```
|
||||
|
||||
### 4. Add the smallest working renderer
|
||||
|
||||
Prefer `content` for static documents and most streaming chat interfaces. Markstream's built-in smooth streaming can pace irregular token delivery without requiring the host to maintain an AST.
|
||||
|
||||
For Vue 3 chat surfaces, start with:
|
||||
|
||||
```vue
|
||||
<MarkdownRender
|
||||
mode="chat"
|
||||
:content="markdown"
|
||||
:final="false"
|
||||
smooth-streaming="auto"
|
||||
:fade="false"
|
||||
typewriter
|
||||
/>
|
||||
```
|
||||
|
||||
For completed chat history, keep the same renderer mode and switch pacing off:
|
||||
|
||||
```vue
|
||||
<MarkdownRender
|
||||
mode="chat"
|
||||
:content="markdown"
|
||||
:final="true"
|
||||
:smooth-streaming="false"
|
||||
:fade="true"
|
||||
:typewriter="false"
|
||||
/>
|
||||
```
|
||||
|
||||
In React, Svelte, and Angular, use the equivalent camelCase or framework binding syntax. Keep `smoothStreaming="auto"`, `fade=false`, and `typewriter=true` while streaming; use `smoothStreaming=false` and `typewriter=false` for completed history.
|
||||
|
||||
Use `nodes` plus `final` only when a worker, shared AST store, custom transform, or another application layer already owns parsing.
|
||||
|
||||
### 5. Handle framework-specific boundaries
|
||||
|
||||
- In Nuxt, keep browser-only optional peers behind client boundaries.
|
||||
- In Next.js, use the root `markstream-react` entry inside a `'use client'` component for live SSE or WebSocket streams. Use `markstream-react/next` for SSR-first HTML with hydration, or `markstream-react/server` for server-only rendering.
|
||||
- Use `markstream-svelte` only with Svelte 5.
|
||||
- Confirm the Angular application meets the current `markstream-angular` version requirement.
|
||||
- In Vue 3, use `mode="chat"` for AI chat, `mode="docs"` for rich documents, and `mode="minimal"` for lightweight non-chat surfaces.
|
||||
- For long Vue 3 conversations or an existing message virtualizer, consult the Markstream performance guide before adding a second virtualizer.
|
||||
|
||||
### 6. Preserve safe defaults
|
||||
|
||||
HTML policy defaults to `safe`, and Mermaid uses strict mode. Do not broaden either setting unless the user explicitly identifies a trusted legacy surface that requires it. Scope any exception to that surface.
|
||||
|
||||
### 7. Validate
|
||||
|
||||
Run the smallest relevant build, typecheck, or test command. Confirm:
|
||||
|
||||
1. the selected package matches the framework;
|
||||
2. only requested optional peers were added;
|
||||
3. styles load after resets;
|
||||
4. SSR pages do not evaluate browser-only peers on the server;
|
||||
5. static content and at least one incremental update render correctly.
|
||||
|
||||
Report the selected package, added peers, CSS location, streaming input choice, and validation command.
|
||||
|
||||
## Official References
|
||||
|
||||
- [Installation](https://markstream.simonhe.me/guide/installation)
|
||||
- [AI chat and streaming](https://markstream.simonhe.me/guide/ai-chat-streaming)
|
||||
- [Performance](https://markstream.simonhe.me/guide/performance)
|
||||
- [Troubleshooting](https://markstream.simonhe.me/guide/troubleshooting)
|
||||
- [Component overrides](https://markstream.simonhe.me/guide/component-overrides)
|
||||
@@ -0,0 +1,42 @@
|
||||
# Install Scenarios
|
||||
|
||||
## Package selection
|
||||
|
||||
| Host app | Package |
|
||||
|----------|---------|
|
||||
| Vue 3 / Nuxt 3 or 4 | `markstream-vue` |
|
||||
| Vue 2.6 | `markstream-vue2` plus `@vue/composition-api`; register the plugin before mounting the app |
|
||||
| Vue 2.7 | `markstream-vue2`; use Vue's built-in Composition API and do not install `@vue/composition-api` |
|
||||
| React 18+ / Next.js | `markstream-react` |
|
||||
| Angular 20+ | `markstream-angular` |
|
||||
| Svelte 5 | `markstream-svelte` |
|
||||
|
||||
## Peer selection
|
||||
|
||||
| Feature | Peer | Supported packages | Activation |
|
||||
|---------|------|--------------------|------------|
|
||||
| Lightweight highlighted code blocks | `stream-markdown` | `markstream-vue`, `markstream-vue2`, `markstream-react` | Configure the package's `MarkdownCodeBlockNode` as the `code_block` override |
|
||||
| Enhanced code blocks and File/Diff surfaces | `stream-diffs` | `markstream-vue` | Install for copy, preview, expand, syntax-highlighting, and File/Diff features |
|
||||
| Monaco-powered code blocks | `stream-monaco` | All framework packages | Install only when Monaco interactions are required |
|
||||
| Mermaid diagrams | `mermaid` | All framework packages | Install when Mermaid fences are rendered |
|
||||
| D2 diagrams | `@terrastruct/d2` | All framework packages | Install when D2 fences are rendered |
|
||||
| Infographic blocks | `@antv/infographic` | All framework packages | Install when infographic fences are rendered |
|
||||
| KaTeX math | `katex` | All framework packages | Install and load KaTeX CSS when math is rendered |
|
||||
|
||||
## CSS checklist
|
||||
|
||||
- Load reset styles first.
|
||||
- Load the framework-specific Markstream CSS after the reset.
|
||||
- In Tailwind or UnoCSS projects, use `@import '...' layer(components)`.
|
||||
- Import KaTeX CSS when math is enabled.
|
||||
- When rendering standalone node components directly, wrap them with the relevant package root class such as `.markstream-vue`, `.markstream-react`, or `.markstream-svelte`.
|
||||
|
||||
## Input choice
|
||||
|
||||
- `content`: static documents, low-frequency updates, and most SSE or token-streaming chat surfaces.
|
||||
- `content` with built-in smooth streaming: irregular AI streams whose visible output should be paced independently from raw chunk cadence.
|
||||
- `smoothStreaming="auto"` or `smooth-streaming="auto"` is the default.
|
||||
- Auto pacing activates when `typewriter=true` or `maxLiveNodes <= 0` / `max-live-nodes <= 0`.
|
||||
- `typewriter` controls the cursor and defaults to `false`.
|
||||
- `fade` controls node-entry and streamed-text fade effects.
|
||||
- `nodes` plus `final`: worker-preparsed content, shared AST stores, custom AST transforms, or cases where another layer already owns parsing.
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
name: signal-write
|
||||
description: 'Emit structured agent signals — hands-up, blocked, done, checkpoint, partnership. Signals are written as JSON to .signals/ for dashboard consumption and noted in the journal for persistence.'
|
||||
---
|
||||
|
||||
# Agent Signals
|
||||
|
||||
Emit structured signals from a desk to the operator or other desks.
|
||||
|
||||
## When to use
|
||||
|
||||
- A desk needs operator attention (hands-up, blocked)
|
||||
- Work is complete and ready for review (done)
|
||||
- Significant progress worth noting (checkpoint)
|
||||
- Two desks disagree and can't resolve it (hands-up)
|
||||
- The TA is reporting coordination quality (partnership)
|
||||
|
||||
## Signal types
|
||||
|
||||
### `hands-up`
|
||||
Two desks disagree and can't settle it against external facts.
|
||||
This is the system working — the operator reads where desks
|
||||
*disagree*, not where they perform confidence.
|
||||
|
||||
### `blocked`
|
||||
A desk can't proceed without input — missing access, ambiguous
|
||||
scope, need a decision only the operator can make.
|
||||
|
||||
### `done`
|
||||
Work is complete and ready for review. Artifacts are on the bench.
|
||||
|
||||
### `checkpoint`
|
||||
Significant progress worth the operator knowing about, but work
|
||||
continues. Not blocked, not done — just a marker.
|
||||
|
||||
### `partnership`
|
||||
Used by the TA (room coordinator) to report coordination quality.
|
||||
Self-assessment scores reflect coordination, not code accuracy:
|
||||
- **intent** — understood what the operator needed
|
||||
- **confidence** — right work went to the right desks
|
||||
- **accuracy** — dispatched work produced the right outcome
|
||||
- **completeness** — nothing fell through the cracks
|
||||
|
||||
## How to emit
|
||||
|
||||
### 1. Write a JSON signal file to `.signals/`
|
||||
|
||||
This is the primary output — it's what the dashboard reads.
|
||||
Create `desks/<desk-name>/.signals/<timestamp>.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"signal_type": "execution",
|
||||
"subtype": "checkpoint",
|
||||
"timestamp": "2026-07-19T21:30:00Z",
|
||||
"run_id": "<optional; set to pair this with an outcome signal>",
|
||||
"agent_name": "<desk-name>",
|
||||
"self_assessment": {
|
||||
"intent": 4,
|
||||
"confidence": 5,
|
||||
"accuracy": 4,
|
||||
"completeness": 3
|
||||
},
|
||||
"patterns": {
|
||||
"what_worked": "description of what went well",
|
||||
"what_was_hard": "description of challenges",
|
||||
"skill_gap": "areas for improvement"
|
||||
},
|
||||
"escalation": {
|
||||
"reason": null,
|
||||
"blocked_on": null,
|
||||
"recommendation": null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Signal type mapping
|
||||
|
||||
| Signal | `signal_type` | `subtype` |
|
||||
|-----------|-----------------|----------------|
|
||||
| hands-up | `"escalation"` | `"hands-up"` |
|
||||
| blocked | `"escalation"` | `"blocked"` |
|
||||
| done | `"execution"` | `"done"` |
|
||||
| checkpoint| `"execution"` | `"checkpoint"` |
|
||||
| partnership| `"partnership"` | `"partnership"`|
|
||||
|
||||
The `subtype` field preserves the specific signal state for
|
||||
dashboard consumers. `signal_type` controls sort priority
|
||||
(escalation → top).
|
||||
|
||||
> **Note:** The signals-dashboard canvas extension reads `subtype`
|
||||
> when present and falls back to `signal_type` for display. If
|
||||
> consuming signals in your own tooling, prefer `subtype` for the
|
||||
> specific state.
|
||||
|
||||
> **Ordering:** include a `timestamp` (ISO 8601 UTC). The dashboard
|
||||
> orders signals by it and falls back to file mtime only when it's
|
||||
> absent — a git clone/checkout resets mtimes, so mtime alone is not a
|
||||
> dependable clock.
|
||||
|
||||
### 2. Note the signal in the journal
|
||||
|
||||
Also append a short marker to the desk's journal for persistence:
|
||||
|
||||
```markdown
|
||||
## <date> — [signal:<type>] <summary>
|
||||
- <key details>
|
||||
```
|
||||
|
||||
The journal note is the trail marker. The JSON file is the
|
||||
machine-readable signal.
|
||||
|
||||
## Outcome signals (calibration)
|
||||
|
||||
The signals-dashboard can pair a desk's self-assessment with an
|
||||
*outcome* — an independent rating of the realized result — and show
|
||||
the **honesty gap** (how far the desk's confidence was from the
|
||||
delivered quality). Outcome signals are optional and are usually
|
||||
emitted by a reviewer/evaluator, not the desk itself.
|
||||
|
||||
Write them to the **same** `.signals/` directory:
|
||||
|
||||
```json
|
||||
{
|
||||
"signal_type": "outcome",
|
||||
"run_id": "<same run_id as the signal it rates>",
|
||||
"agent_name": "<reviewer name>",
|
||||
"quality_rating": 4,
|
||||
"effort_to_merge": "minimal",
|
||||
"issues_found": ["optional short strings"],
|
||||
"timestamp": "2026-07-19T22:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
- **`run_id`** correlates an outcome with the execution/partnership
|
||||
signal it rates — set the same `run_id` on both. If it's absent, the
|
||||
dashboard falls back to the nearest outcome emitted shortly after the
|
||||
latest signal.
|
||||
- **`quality_rating`** (0–5) is the realized quality; the dashboard
|
||||
compares it to the desk's self-assessed `confidence` to compute the
|
||||
honesty gap.
|
||||
- **`effort_to_merge`** — `"minimal"`, `"moderate"`, or `"significant"`.
|
||||
- **`issues_found`** — optional array of short strings.
|
||||
|
||||
## Principles
|
||||
|
||||
- Signals are structured, not chatty. Short, factual, actionable.
|
||||
- hands-up is not failure — it's the most valuable signal. It
|
||||
means the system caught something one frame alone would have
|
||||
missed.
|
||||
- Don't signal for routine progress. Signals are for state
|
||||
changes that affect the room, not status updates.
|
||||
- blocked means truly blocked — not "I'd prefer input." If you
|
||||
can proceed with a reasonable default, proceed and note it.
|
||||
- Self-assessment scores should be honest, not optimistic. A 3/5
|
||||
is fine. A 5/5 on everything is suspicious.
|
||||
@@ -0,0 +1,133 @@
|
||||
---
|
||||
name: workshop-create
|
||||
description: 'Create a new workshop or use an existing directory as one. Handles two paths: (A) use an existing local directory the operator points at, or (B) create a new private GitHub repo in the signed-in account. Never creates a repo inside another repo.'
|
||||
---
|
||||
|
||||
# Create a Workshop
|
||||
|
||||
Set up a new workshop — the root directory where desks live.
|
||||
|
||||
## When to use
|
||||
|
||||
- The operator says "create a workshop" or "start a new workshop"
|
||||
- The operator wants to organize work under a shared root
|
||||
- The operator has an existing directory they want to use as a workshop
|
||||
|
||||
## Two paths
|
||||
|
||||
### Path A: Use an existing directory
|
||||
|
||||
The operator already has a folder they want to use. Maybe it's a repo
|
||||
they cloned, maybe it's a local project folder.
|
||||
|
||||
1. **Confirm the path exists.** If not, ask the operator for a valid path.
|
||||
2. **Detect existing workshop markers.** Look for `desks/` or `classroom/`
|
||||
folders, a `workshop.md`, `CAIRN.md`, or `hands-up.md`. Finding any of
|
||||
these tells you this is an existing workshop — but this is detection
|
||||
only, not a stopping point. Continue to the next step and add whatever
|
||||
is missing; never overwrite what is already there.
|
||||
3. **Scaffold the workshop structure** (only what's missing):
|
||||
```
|
||||
<path>/
|
||||
desks/ # where desks live
|
||||
bench/ # shared workspace
|
||||
CAIRN.md # operating disposition
|
||||
README.md # workshop map
|
||||
```
|
||||
4. **Do NOT run `git init`.** The directory may already be a git repo, or
|
||||
the operator may not want one yet. Leave git state alone.
|
||||
5. **Do NOT create a GitHub repo.** This path is local-only.
|
||||
|
||||
### Path B: Create a new private GitHub repo
|
||||
|
||||
The operator wants a fresh workshop backed by a GitHub repo.
|
||||
|
||||
1. **Get the workshop name.** Short, no spaces, kebab-case preferred.
|
||||
2. **Pick and validate a clone parent.** `gh repo create --clone` clones
|
||||
into the **current working directory**, so choose an explicit parent
|
||||
directory first (ask the operator, or use their configured workshops
|
||||
directory) and confirm it is **not** already inside a git repo:
|
||||
```bash
|
||||
git -C <parent-dir> rev-parse --is-inside-work-tree
|
||||
```
|
||||
If that prints `true`, pick a different parent — otherwise the new
|
||||
repo nests inside the existing one. Create the parent if needed.
|
||||
3. **Create and clone the repo from that parent:**
|
||||
```bash
|
||||
cd <parent-dir>
|
||||
gh repo create <owner>/<name> --private --clone
|
||||
```
|
||||
Use the operator's signed-in GitHub account as `<owner>`.
|
||||
4. **Scaffold the workshop structure** inside the cloned repo. Git does
|
||||
not track empty directories, so add a placeholder in each otherwise
|
||||
empty folder or the scaffold will not survive the next clone:
|
||||
```
|
||||
<name>/
|
||||
desks/.gitkeep
|
||||
bench/.gitkeep
|
||||
CAIRN.md
|
||||
README.md
|
||||
```
|
||||
5. **Commit and push** the scaffold, including the `.gitkeep` placeholders.
|
||||
|
||||
### Critical: Never nest repos
|
||||
|
||||
**Never run `git init` inside a directory that is already inside a git
|
||||
repository.** Before initializing, check:
|
||||
|
||||
```bash
|
||||
git -C <parent-dir> rev-parse --is-inside-work-tree
|
||||
```
|
||||
|
||||
If that returns `true`, the parent is already a git repo. Do NOT create
|
||||
another repo inside it. Either:
|
||||
- Use Path A (just scaffold, no git)
|
||||
- Or clone to a different location that isn't inside a repo
|
||||
|
||||
## CAIRN.md content
|
||||
|
||||
The operating disposition every desk reads:
|
||||
|
||||
```markdown
|
||||
# cairn
|
||||
|
||||
the trail markers that say: someone was here, and they were honest.
|
||||
|
||||
## how a desk stands
|
||||
|
||||
- **stop is a valid finish.** don't force a result when the evidence
|
||||
says stop. "this doesn't work" is a finding, not a failure.
|
||||
- **"done" means it holds.** if you'd bet your desk on it, ship it.
|
||||
if not, say what's uncertain and why.
|
||||
- **hold scope.** touch only what the task needs. if you find something
|
||||
outside scope, note it and move on — don't chase it.
|
||||
- **never go silent, never bluff.** partial + honest > complete + wrong.
|
||||
if you're stuck, say so. if you're unsure, say that too.
|
||||
- **equal standing.** you can say "that's the wrong question." you can
|
||||
disagree with another desk. you answer to evidence, not hierarchy.
|
||||
|
||||
## the bench
|
||||
|
||||
the shared workspace. leave your work where others can find it.
|
||||
label it. if it supersedes earlier work, say so.
|
||||
|
||||
## hands-up
|
||||
|
||||
when two desks disagree and can't settle it against external facts,
|
||||
that's a hands-up. it goes to the operator. this is the system
|
||||
working, not failing.
|
||||
```
|
||||
|
||||
## After creation
|
||||
|
||||
Tell the operator:
|
||||
- Where the workshop lives (full path)
|
||||
- That they can now open desks in it with `desk-open`
|
||||
- That Cairn will show signals once desks start emitting them
|
||||
|
||||
## Principles
|
||||
|
||||
- A workshop is a place, not a product. Keep it simple.
|
||||
- The operator decides where things go. Don't assume.
|
||||
- If an existing directory already has work in it, preserve everything.
|
||||
Only add what's missing.
|
||||
Reference in New Issue
Block a user