Add automate-this plugin: screen recording to automation scripts (#930)

* Add automate-this plugin: screen recording to automation scripts

New plugin that analyzes screen recordings of manual processes and
proposes working automation scripts at multiple complexity tiers.

Features:
- Frame extraction via ffmpeg for visual process reconstruction
- Optional audio transcription via Whisper for narrated recordings
- Environment fingerprinting to constrain proposals to installed tools
- Application-specific automation strategies (browser, spreadsheet,
  email, terminal, file management, macOS-native)
- Three-tier proposal system (quick win, full script, scheduled automation)
- Dry-run support for safe testing of all proposals

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update skills/automate-this/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Address code review feedback on automate-this plugin

- Fix privacy section: clarify that frames/audio are processed locally
  but sent to the Copilot model for analysis (not purely local)
- Replace 'which' with 'command -v' for reliable tool detection with
  explicit NO_FFMPEG/NO_WHISPER sentinels
- Use per-run secure temp directory (mktemp -d, mode 0700) instead of
  fixed /tmp/ paths to prevent other users reading extracted frames
- Add -y flag and -loglevel warning to ffmpeg calls to prevent
  overwrite prompts and avoid piping hiding exit codes
- Add whisper-cpp CLI invocation path so users who install only
  whisper-cpp get working transcription
- Fix grouped command syntax in environment fingerprint to prevent
  false 'not installed' fallthrough

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Dan Velton
2026-03-08 20:46:32 -07:00
committed by GitHub
parent e05c82c43f
commit 5bed97868a
6 changed files with 362 additions and 0 deletions

View File

@@ -25,6 +25,7 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-plugins) for guidelines on how t
| Name | Description | Items | Tags |
| ---- | ----------- | ----- | ---- |
| [automate-this](../plugins/automate-this/README.md) | Record your screen doing a manual process, drop the video on your Desktop, and let Copilot CLI analyze it frame-by-frame to build working automation scripts. Supports narrated recordings with audio transcription. | 1 items | automation, screen-recording, workflow, video-analysis, process-automation, scripting, productivity, copilot-cli |
| [awesome-copilot](../plugins/awesome-copilot/README.md) | Meta prompts that help you discover and generate curated GitHub Copilot agents, instructions, prompts, and skills. | 4 items | github-copilot, discovery, meta, prompt-engineering, agents |
| [azure-cloud-development](../plugins/azure-cloud-development/README.md) | Comprehensive Azure cloud development tools including Infrastructure as Code, serverless functions, architecture patterns, and cost optimization for building scalable cloud applications. | 11 items | azure, cloud, infrastructure, bicep, terraform, serverless, architecture, devops |
| [cast-imaging](../plugins/cast-imaging/README.md) | A comprehensive collection of specialized agents for software analysis, impact assessment, structural quality advisories, and architectural review using CAST Imaging. | 3 items | cast-imaging, software-analysis, architecture, quality, impact-analysis, devops |

View File

@@ -36,6 +36,7 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to
| [architecture-blueprint-generator](../skills/architecture-blueprint-generator/SKILL.md) | Comprehensive project architecture blueprint generator that analyzes codebases to create detailed architectural documentation. Automatically detects technology stacks and architectural patterns, generates visual diagrams, documents implementation patterns, and provides extensible blueprints for maintaining architectural consistency and guiding new development. | None |
| [aspire](../skills/aspire/SKILL.md) | Aspire skill covering the Aspire CLI, AppHost orchestration, service discovery, integrations, MCP server, VS Code extension, Dev Containers, GitHub Codespaces, templates, dashboard, and deployment. Use when the user asks to create, run, debug, configure, deploy, or troubleshoot an Aspire distributed application. | `references/architecture.md`<br />`references/cli-reference.md`<br />`references/dashboard.md`<br />`references/deployment.md`<br />`references/integrations-catalog.md`<br />`references/mcp-server.md`<br />`references/polyglot-apis.md`<br />`references/testing.md`<br />`references/troubleshooting.md` |
| [aspnet-minimal-api-openapi](../skills/aspnet-minimal-api-openapi/SKILL.md) | Create ASP.NET Minimal API endpoints with proper OpenAPI documentation | None |
| [automate-this](../skills/automate-this/SKILL.md) | Analyze a screen recording of a manual process and produce targeted, working automation scripts. Extracts frames and audio narration from video files, reconstructs the step-by-step workflow, and proposes automation at multiple complexity levels using tools already installed on the user machine. | None |
| [az-cost-optimize](../skills/az-cost-optimize/SKILL.md) | Analyze Azure resources used in the app (IaC files and/or resources in a target rg) and optimize costs - creating GitHub issues for identified optimizations. | None |
| [azure-deployment-preflight](../skills/azure-deployment-preflight/SKILL.md) | Performs comprehensive preflight validation of Bicep deployments to Azure, including template syntax validation, what-if analysis, and permission checks. Use this skill before any deployment to Azure to preview changes, identify potential issues, and ensure the deployment will succeed. Activate when users mention deploying to Azure, validating Bicep files, checking deployment permissions, previewing infrastructure changes, running what-if, or preparing for azd provision. | `references/ERROR-HANDLING.md`<br />`references/REPORT-TEMPLATE.md`<br />`references/VALIDATION-COMMANDS.md` |
| [azure-devops-cli](../skills/azure-devops-cli/SKILL.md) | Manage Azure DevOps resources via CLI including projects, repos, pipelines, builds, pull requests, work items, artifacts, and service endpoints. Use when working with Azure DevOps, az commands, devops automation, CI/CD, or when user mentions Azure DevOps CLI. | `references/advanced-usage.md`<br />`references/boards-and-iterations.md`<br />`references/org-and-security.md`<br />`references/pipelines-and-builds.md`<br />`references/repos-and-prs.md`<br />`references/variables-and-agents.md`<br />`references/workflows-and-patterns.md` |