Add visual-pr plugin — screenshot capture, annotation, and PR embedding (#1804)

* Add visual-pr plugin: screenshot capture, annotation, PR embedding, and screen recording Four skills that teach Copilot to capture UI screenshots (Playwright + PIL), annotate them with algorithmic label placement, embed before/after images in PR descriptions, and record animated GIF demos. Includes demo images showing the annotation engine on GitHub Issues. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update generated README tables and marketplace.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Embed annotate.py module in image-annotations skill The full working module (annotate_image, grid_image, diff_images) is now included as a code block so users can save it as annotate.py and import directly. Scrubbed project-specific labels from examples. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: mss.mss() context manager, fix RECT struct, consistent placeholder - Use mss.mss() context manager instead of mss.MSS() (ui-screenshots, screen-recording) - Fix broken RECT struct in window+GIF combining example (screen-recording) - Consistent projectId placeholder in AzDO upload example (pr-screenshots) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-29 18:11:45 +00:00 · 2026-05-25 03:22:39 +02:00
parent 7f7599a716
commit 5cab59b03e
13 changed files with 1264 additions and 0 deletions
@@ -0,0 +1,238 @@
+---
+name: screen-recording
+description: 'Create annotated animated GIF demos and screen recordings for pull requests and documentation. Covers frame capture, timing, imageio-based GIF creation, and per-frame annotation workflows.'
+---
+
+# Screen Recording
+
+Create animated GIF demos that show a feature or workflow in action — with annotations, variable timing, and proper pacing. Useful for PR descriptions, documentation, and release notes.
+
+## When to Use This Skill
+
+Use this skill when you need to:
+
+- Record a multi-step UI interaction as an animated GIF
+- Create a demo showing before/after behavior
+- Build annotated walkthroughs for documentation or release notes
+- Show a bug reproduction or fix in action
+
+## Prerequisites
+
+```bash
+pip install playwright Pillow imageio numpy scipy mss -q
+playwright install chromium
+```
+
+## Core Workflow
+
+### 1. Capture frames
+
+Use Playwright to step through the interaction and capture each frame:
+
+```python
+from playwright.async_api import async_playwright
+
+async def record_frames(url, steps, width=1400, height=900):
+    """
+    steps: list of dicts with 'action' (async callable taking page)
+           and 'name' (frame filename)
+    """
+    async with async_playwright() as p:
+        browser = await p.chromium.launch()
+        page = await browser.new_page(viewport={"width": width, "height": height})
+        await page.goto(url, wait_until="networkidle")
+
+        for step in steps:
+            if step.get("action"):
+                await step["action"](page)
+                await page.wait_for_timeout(step.get("wait", 500))
+            await page.screenshot(path=step["name"])
+
+        await browser.close()
+```
+
+### 2. Assemble GIF with imageio
+
+**Use imageio, not PIL, for GIF writing** — PIL's GIF encoder merges visually similar frames, which kills animations.
+
+```python
+import imageio.v3 as iio
+from PIL import Image
+import numpy as np
+
+frames = []
+durations = []
+
+for frame_path, duration_ms in frame_list:
+    img = Image.open(frame_path)
+    frames.append(np.array(img))
+    durations.append(duration_ms)
+
+iio.imwrite("demo.gif", frames, duration=durations, loop=0)
+```
+
+### 3. Variable frame timing
+
+Uniform timing makes everything feel either too fast or too slow. Use variable durations:
+
+| Phase | Duration | Why |
+|-------|----------|-----|
+| Fast action (typing, clicking) | 100ms | Feels natural, keeps energy |
+| Pause after action | 600-800ms | Let the viewer process what happened |
+| Hero/final message | 500ms+ | Main takeaway needs time to land |
+
+### 4. Annotate frames
+
+Apply annotations to specific frames using the `image-annotations` skill:
+
+```python
+from PIL import Image, ImageDraw, ImageFont
+
+def annotate_frame(frame_path, annotations, out_path):
+    img = Image.open(frame_path)
+    draw = ImageDraw.Draw(img)
+
+    for ann in annotations:
+        # Apply annotation (rect, arrow, label, etc.)
+        pass
+
+    img.save(out_path)
+```
+
+### 5. Fade-in annotations
+
+For smooth annotation appearance:
+
+```python
+def apply_fade(base_frame, annotation_layer, alpha):
+    """Blend annotation onto frame at given alpha (0.0 to 1.0)"""
+    blended = Image.blend(
+        base_frame.convert("RGBA"),
+        annotation_layer.convert("RGBA"),
+        alpha
+    )
+    return blended.convert("RGB")
+
+# 2-frame pop-in at 10fps: 50% then 100%
+faded_frames = [
+    apply_fade(base, annotations, 0.5),  # frame 1: half opacity
+    apply_fade(base, annotations, 1.0),  # frame 2: full opacity
+]
+```
+
+At 10fps, use 2 fade frames (0.2s total). At 30fps, use 3-4 frames. Easing curves look bad at low FPS — simple pop-in is snappier and more readable.
+
+## Build as a Script
+
+The annotation logic gets complex for anything beyond trivial demos. Write a dedicated script (e.g., `annotate_gif.py`) with functions instead of inline code. You'll iterate on timing and placement.
+
+## Testing Animations
+
+**Always test in isolation first** — don't rebuild the full demo to test a fade tweak:
+
+```python
+# Small test GIF: 10 bare frames → fade frames → 15 hold frames
+# Add a frame counter overlay for debugging:
+draw.text((10, height - 30), f"F{i}/{total} a={alpha:.0%} FADE",
+          fill="white", font=small_font)
+```
+
+## Desktop Screen Recording (mss)
+
+For recording desktop apps, terminals, or anything outside a browser. Uses `mss` for fast screen capture.
+
+```python
+import mss
+from PIL import Image
+import time
+
+def record_gif(output_path, region=None, duration=5, fps=8):
+    """Record screen region to GIF. region = {left, top, width, height} or None for full screen."""
+    with mss.mss() as sct:
+        if region is None:
+            region = sct.monitors[1]  # primary monitor
+
+        frames = []
+        t_end = time.time() + duration
+        while time.time() < t_end:
+            t0 = time.time()
+            shot = sct.grab(region)
+            frames.append(Image.frombytes('RGB', shot.size, shot.rgb))
+            time.sleep(max(0, 1 / fps - (time.time() - t0)))
+
+    frames[0].save(output_path, save_all=True, append_images=frames[1:],
+                   duration=int(1000 / fps), loop=0, optimize=True)
+    return len(frames)
+
+record_gif('demo.gif', region={'left': 0, 'top': 0, 'width': 800, 'height': 500}, duration=3)
+```
+
+Tested: 3s at 8fps → 24 frames, ~31KB. Keep fps ≤ 10 for reasonable file sizes.
+
+**Note:** `PIL.save(save_all=True)` works for simple recordings but merges visually similar frames. For annotated GIFs with fade effects, use `imageio.v3.imwrite` instead.
+
+### Combining with window capture
+
+```python
+# Find window rect, then record it as a GIF
+# Reuse find_window() from the ui-screenshots skill
+import ctypes
+from ctypes import c_int, Structure, byref, windll
+
+class RECT(Structure):
+    _fields_ = [('left', c_int), ('top', c_int), ('right', c_int), ('bottom', c_int)]
+
+hwnd = find_window('My App')[0][0]
+rect = RECT()
+windll.user32.GetWindowRect(hwnd, byref(rect))
+region = {'left': rect.left, 'top': rect.top,
+          'width': rect.right - rect.left, 'height': rect.bottom - rect.top}
+record_gif('app-demo.gif', region=region, duration=5, fps=8)
+```
+
+## Diff-Based Cluster Detection
+
+Programmatically find changed regions between frames to decide what to annotate:
+
+```python
+import numpy as np
+from scipy import ndimage
+
+def find_changed_clusters(frame_a, frame_b, threshold=30, min_pixels=300, dilate=5):
+    """Find bounding boxes of changed regions between two frames."""
+    diff = np.abs(frame_b.astype(float) - frame_a.astype(float)).max(axis=2)
+    mask = diff > threshold
+    dilated = ndimage.binary_dilation(mask, iterations=dilate)
+    labeled, n = ndimage.label(dilated)
+    clusters = []
+    for i in range(1, n + 1):
+        ys, xs = np.where(labeled == i)
+        if len(ys) < min_pixels:
+            continue
+        clusters.append((xs.min(), ys.min(), xs.max(), ys.max(), len(ys)))
+    return sorted(clusters, key=lambda c: -c[4])  # largest first
+```
+
+## Format Compatibility
+
+| Format | VS Code Preview | GitHub | Browser |
+|--------|----------------|--------|---------|
+| GIF | ✅ Animates | ✅ | ✅ |
+| WebP | ⚠️ Static only | ✅ | ✅ |
+| MP4 | ❌ Broken | ⚠️ | ✅ |
+
+**GIF is the only universally supported animated format** across VS Code preview, GitHub markdown, and browsers.
+
+## Guidelines
+
+1. **Type → pause → annotate** — during fast action, show NO annotation. Pause first, then annotate
+2. **Hero message gets the biggest font** — 64pt+ for the main takeaway, 38pt for details
+3. **GIF palette does NOT kill gradients** — 20 distinct alpha steps survive 256-color palette
+4. **10fps minimum** for typing/interaction — lower looks stuttery
+5. **Build iteratively** — get the frame sequence right first, add annotations second, tune timing last
+
+## Limitations
+
+- GIF is limited to 256 colors per frame — fine for UI screenshots, may show banding on photographic content
+- Large GIFs (50+ frames at high resolution) can be several MB — consider cropping to the relevant area
+- No audio support in GIF — use MP4 for narrated demos (but lose VS Code preview support)