Add APNG Studio canvas extension (#2272)

* Add APNG Studio canvas extension

APNG Studio is an interactive GitHub Copilot canvas extension for building
Animated PNG (APNG) files from frames: draw or upload frames, tune per-frame
timing and compositing, preview live, send the result to a phone by QR, and
export an animated .png.

Adds extensions/apng-studio/ with the required .github/plugin/plugin.json and
assets/preview.png, and regenerates .github/plugin/marketplace.json.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback for APNG Studio

Security and robustness (extension.mjs):
- Reject "." and ".." project ids so canvas/action input cannot resolve
  outside artifacts/.
- Bound request bodies (1 MiB JSON, 40 MiB frame upload); return 413 when
  exceeded.
- Reject malformed JSON with 400 instead of coercing to {}, so a truncated
  body cannot trigger destructive routes such as /frames/clear.
- Scope shares per project so one canvas cannot rotate or stop another's
  share; the LAN server is shared and torn down once no shares remain.
- get_state treats hiddenFirst as active only with >=2 frames, matching the
  encoder, so a single-frame project reports the correct duration.

Accessibility (web/):
- Expose Pen/Eraser state with aria-pressed and keep it in sync in setTool.
- Mark the toast as an aria-live status region.

Packaging:
- Add extensions/apng-studio/.gitignore with artifacts/ so the documented
  runtime-data exclusion holds for this extension.
- Update the README install section to reference the committed awesome-copilot
  extension path.

Regenerated .github/plugin/marketplace.json.

* Use "GitHub Copilot app" wording

Update the plugin description (and regenerated marketplace entry) to refer to
the host as the GitHub Copilot app.

* Address deeper review for APNG Studio

Security:
- Require a per-server access token on every loopback data/mutation request
  (minted per canvas server, carried in the iframe URL, attached to every
  renderer request). Static assets stay public. Blocks a local process or
  cross-origin page from reading state or driving mutations.

Concurrency and durability:
- Serialize the load-mutate-save cycle per project; dedupe concurrent first
  loads; write project.json via temp file + rename.

Lifecycle:
- Track SSE clients per instance and end them before server.close().
- Stop a project's LAN share only when no other panel references it.

Correctness:
- Report exact numerator/denominator total duration; handle pointercancel.

QR and accessibility:
- Place QR version bits least-significant-bit first (versions 7-10).
- Associate delay/fps/dispose/blend labels with their inputs via for=.

* Address deeper concurrency and validation review for APNG Studio

- Run assemble() under the project lock; add_color_frame renders and appends
  within one lock.
- Validate uploaded frames (PNG chunk scan, size + dimension checks) before
  writing; first frame stores real dimensions; CanvasError maps to 400.
- Validate move delta; clear frames in memory before deleting files.
- Route the open-handler rename through applySettings.
- Deduplicate LAN share startup; clean up if the canvas closes mid-start.
- End SSE streams before server.close(); skip dead streams in broadcast.
- Exact fractional duration; drawing surface stays >= 1px; refresh state before
  re-seeding "start from last frame".

* Address review: PNG format validation, share bind, limits, a11y

- Validate uploaded frames are 8-bit RGBA non-interlaced PNGs (standard
  compression/filter); reject formats the codec can't encode.
- Cap projects at 600 frames (add + duplicate).
- Bind the LAN share server to the private address, not 0.0.0.0; require a
  private address and rebind when it changes while idle; build the URL from the
  bound address.
- Align width/height inputs and clampSize to the 2048 maximum.
- Size the drawing surface by width + aspect ratio for narrow panels.
- set_frame with an unknown frameId returns a validation error.

* Address review: encoded-byte budget, timing modes, id keys, load errors

- 256 MiB aggregate encoded-byte budget (add + duplicate); per-frame size
  tracked and backfilled on load, so a frame count alone no longer bounds
  assembly memory.
- Frame timing is one exclusive mode (delayMs | fps | delayNum/delayDen);
  combinations are rejected instead of producing hybrid delays.
- Collision-resistant project storage keys: safe ids are used verbatim (existing
  projects still load) and only unsafe ids get a hash suffix, so "foo/bar" and
  "foo?bar" can't share a directory.
- Load only treats a missing file as a new project; other read/parse errors
  surface instead of silently overwriting real data.
- Share server binds to a real LAN interface (skips virtual/VPN/container,
  prefers physical NICs and common ranges).
- Generated export names get millisecond + random suffix to avoid same-second
  overwrite.
- PNG validator accepts the encoder's Uint8Array so agent solid-color frames
  aren't rejected.

* Address review: crash-safe frame writes, server startup, APNG first frame

- Write the PNG before mutating project metadata on add/duplicate, and delete
  after persisting, so an interrupted disk op can't dangle a reference or
  advance unsaved state.
- Evict a project from the in-memory cache if its save fails, so a transient
  write error can't desync the live session from disk.
- Reject on the loopback server's listen error and drop a half-initialized
  instance so startup failures clean up and can retry.
- Normalize APNG dispose_op PREVIOUS to BACKGROUND on the first animated frame
  (spec requirement).
- Release decoded ImageBitmaps after use so a large upload batch doesn't retain
  native image memory.
- Surface otherwise-silent async UI failures via a toast; check the state
  response is ok before parsing it.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Andrea Liliana Griffiths
2026-07-13 01:53:55 +02:00
committed by GitHub
parent f8d98b41d1
commit 0916fe444d
14 changed files with 3618 additions and 0 deletions
+97
View File
@@ -0,0 +1,97 @@
# APNG Studio
An interactive [GitHub Copilot app](https://github.com/features/ai/github-app) **canvas extension** for building [Animated PNG (APNG)](https://wiki.mozilla.org/APNG_Specification) files from frames — draw or upload frames, tune the full practical APNG spec surface, preview live, and export an animated `.png`.
The canvas renders in a side panel; the agent can also drive it through callable actions.
<p align="center">
<img src="assets/demo.png" alt="An animated walkthrough of the APNG Studio canvas building an animation from frames" width="640" />
</p>
<p align="center"><em>APNG Studio in action — an animated PNG built with the canvas itself.</em></p>
## Background
APNG Studio started as a hallway conversation. During a demo shift at the WeAreDevelopers Congress I got talking with [Jeff](https://github.com/GekkeBoyJeff) about APNG (animated PNG) versus GIF: APNG keeps real alpha and full color where GIF can't. We wanted an easy way to actually build one, so we made this small canvas wrapper for creating APNGs.
## Features
- **Frames** — upload images or draw them on a builtin canvas (pen/eraser, fill, onionskin, "start from last frame"). Reorder, duplicate, and delete frames.
- **Perframe timing** — set the delay as an exact `numerator / denominator` fraction with a live `= N ms · N fps` readout.
- **Perframe compositing** — `dispose_op` (None / Background / Previous) and `blend_op` (Source / Over) dropdowns, straight from the APNG spec.
- **Apply to all** — set every frame's delay (ms), snap to an exact frame rate (fps), or apply dispose + blend in one click.
- **Loop count** — `0` = infinite, or a fixed number of plays.
- **Hidden first frame** — mark frame 1 as a static fallback: shown by nonAPNG viewers, excluded from the animation loop (encoded as a default image with no leading `fcTL`, `num_frames = N1`).
- **Live preview** — a real animated PNG is assembled on every change and served from `/preview.png`; a **Reload** button resyncs state and rebuilds the preview.
- **Send to phone** — a **Send to phone** button opens a QR code; scan it with your phone camera (same WiFi) to open the live animation in your phone's browser and save it. Served readonly from a shortlived, tokengated LAN endpoint that shuts itself down after 10 minutes.
- **Export** — writes a valid animated `.png` (APNG) to disk and returns its path. The `.png` extension keeps the file bytecompatible with every PNG viewer: APNGaware ones (browsers, macOS Quick Look) animate it, others show the first frame as a static fallback.
## Install
### From GitHub Copilot (recommended)
Ask Copilot to install the committed extension URL:
```text
Install this extension: https://github.com/github/awesome-copilot/tree/main/extensions/apng-studio
```
You can also copy the folder into one of these locations:
- **User** — `~/.copilot/extensions/apng-studio/`, available in every project.
- **Project** — `.github/extensions/apng-studio/` inside a repo, committed and shared with your team.
Reload extensions in the app, then open the `apng-studio` canvas.
### Manual
Copy the source files into one of the extension directories above, keeping the layout:
```
apng-studio/
├── extension.mjs # entry point (required name)
├── apng.mjs # APNG codec + RGBA→PNG encoder
├── qr.mjs # dependency-free QR encoder (Send to phone)
└── web/ # canvas iframe renderer
├── index.html
├── app.js
└── styles.css
```
Then reload extensions. The `@github/copilot-sdk` import is resolved by the host — **do not** add a `package.json` or `node_modules` for it.
## Open the canvas
Once installed, open the **APNG Studio** canvas from Copilot. Optional open input:
| field | type | description |
| ----------- | ------ | ---------------------------------------------- |
| `projectId` | string | Animation project id (defaults to `default`). |
| `name` | string | Optional display name for the animation. |
Each project's frames persist on disk under `artifacts/<projectId>/`, so they survive reloads and are shared between every open panel and the agent actions. That folder is local user data and is **gitignored**.
## Agent actions
The extension exposes these callable actions on the `apng-studio` canvas:
| action | what it does |
| ----------------- | ------------------------------------------------------------------------------------------------------- |
| `get_state` | Return project settings + perframe timing/compositing and total duration. |
| `set_settings` | Update `width`/`height` (only with 0 frames), `loops`, and `hiddenFirst`. |
| `add_color_frame` | Append a solidcolor frame; accepts `delayNum`/`delayDen`/`disposeOp`/`blendOp`. |
| `set_frame` | Update one frame (`frameId`) or all (`all: true`): timing via `delayMs`/`fps`/`delayNum`+`delayDen`, plus `disposeOp`/`blendOp`. |
| `clear_frames` | Remove every frame. |
| `export` | Assemble and write the animated `.png` (APNG) to disk; returns the absolute path. |
All actions accept an optional `projectId` to target a specific animation.
## How it works
- **`extension.mjs`** — one loopback HTTP server per open canvas instance serves the renderer, JSON state, perframe PNGs, the live `/preview.png`, and mutation endpoints. ServerSent Events (`/events`) push a `changed` signal so every open panel and the preview stay in sync. **Send to phone** spins up a separate, readonly LAN server that serves only a landing page and the preview image, gated by a shortlived random token and torn down on expiry.
- **`apng.mjs`** — assembles the APNG chunk stream (`IHDR` / `acTL` / `fcTL` / `IDAT` / `fdAT` / `IEND`) with contiguous sequence numbers, plus a minimal RGBA→PNG encoder.
- **`qr.mjs`** — a small, dependencyfree QR encoder (byte mode, errorcorrection level M) used to render the **Send to phone** code. The QR is drawn into a PNG with the `apng.mjs` encoder.
- **`web/`** — the iframe UI. It talks to its server over plain HTTP; there is no privileged host bridge.
## License
[MIT](./LICENSE)