mirror of
https://github.com/github/awesome-copilot.git
synced 2026-04-12 11:15:56 +00:00
d79183139a
* Add 9 Arize LLM observability skills Add skills for Arize AI platform covering trace export, instrumentation, datasets, experiments, evaluators, AI provider integrations, annotations, prompt optimization, and deep linking to the Arize UI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add 3 Phoenix AI observability skills Add skills for Phoenix (Arize open-source) covering CLI debugging, LLM evaluation workflows, and OpenInference tracing/instrumentation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Ignoring intentional bad spelling * Fix CI: remove .DS_Store from generated skills README and add codespell ignore Remove .DS_Store artifact from winmd-api-search asset listing in generated README.skills.md so it matches the CI Linux build output. Add queston to codespell ignore list (intentional misspelling example in arize-dataset skill). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add arize-ax and phoenix plugins Bundle the 9 Arize skills into an arize-ax plugin and the 3 Phoenix skills into a phoenix plugin for easier installation as single packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix skill folder structures to match source repos Move arize supporting files from references/ to root level and rename phoenix references/ to rules/ to exactly match the original source repository folder structures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fixing file locations * Fixing readme --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
73 lines
4.4 KiB
Markdown
73 lines
4.4 KiB
Markdown
---
|
|
name: phoenix-evals
|
|
description: Build and run evaluators for AI/LLM applications using Phoenix.
|
|
license: Apache-2.0
|
|
compatibility: Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.
|
|
metadata:
|
|
author: oss@arize.com
|
|
version: "1.0.0"
|
|
languages: "Python, TypeScript"
|
|
---
|
|
|
|
# Phoenix Evals
|
|
|
|
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
|
|
|
|
## Quick Reference
|
|
|
|
| Task | Files |
|
|
| ---- | ----- |
|
|
| Setup | [setup-python](references/setup-python.md), [setup-typescript](references/setup-typescript.md) |
|
|
| Decide what to evaluate | [evaluators-overview](references/evaluators-overview.md) |
|
|
| Choose a judge model | [fundamentals-model-selection](references/fundamentals-model-selection.md) |
|
|
| Use pre-built evaluators | [evaluators-pre-built](references/evaluators-pre-built.md) |
|
|
| Build code evaluator | [evaluators-code-python](references/evaluators-code-python.md), [evaluators-code-typescript](references/evaluators-code-typescript.md) |
|
|
| Build LLM evaluator | [evaluators-llm-python](references/evaluators-llm-python.md), [evaluators-llm-typescript](references/evaluators-llm-typescript.md), [evaluators-custom-templates](references/evaluators-custom-templates.md) |
|
|
| Batch evaluate DataFrame | [evaluate-dataframe-python](references/evaluate-dataframe-python.md) |
|
|
| Run experiment | [experiments-running-python](references/experiments-running-python.md), [experiments-running-typescript](references/experiments-running-typescript.md) |
|
|
| Create dataset | [experiments-datasets-python](references/experiments-datasets-python.md), [experiments-datasets-typescript](references/experiments-datasets-typescript.md) |
|
|
| Generate synthetic data | [experiments-synthetic-python](references/experiments-synthetic-python.md), [experiments-synthetic-typescript](references/experiments-synthetic-typescript.md) |
|
|
| Validate evaluator accuracy | [validation](references/validation.md), [validation-evaluators-python](references/validation-evaluators-python.md), [validation-evaluators-typescript](references/validation-evaluators-typescript.md) |
|
|
| Sample traces for review | [observe-sampling-python](references/observe-sampling-python.md), [observe-sampling-typescript](references/observe-sampling-typescript.md) |
|
|
| Analyze errors | [error-analysis](references/error-analysis.md), [error-analysis-multi-turn](references/error-analysis-multi-turn.md), [axial-coding](references/axial-coding.md) |
|
|
| RAG evals | [evaluators-rag](references/evaluators-rag.md) |
|
|
| Avoid common mistakes | [common-mistakes-python](references/common-mistakes-python.md), [fundamentals-anti-patterns](references/fundamentals-anti-patterns.md) |
|
|
| Production | [production-overview](references/production-overview.md), [production-guardrails](references/production-guardrails.md), [production-continuous](references/production-continuous.md) |
|
|
|
|
## Workflows
|
|
|
|
**Starting Fresh:**
|
|
[observe-tracing-setup](references/observe-tracing-setup.md) → [error-analysis](references/error-analysis.md) → [axial-coding](references/axial-coding.md) → [evaluators-overview](references/evaluators-overview.md)
|
|
|
|
**Building Evaluator:**
|
|
[fundamentals](references/fundamentals.md) → [common-mistakes-python](references/common-mistakes-python.md) → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
|
|
|
|
**RAG Systems:**
|
|
[evaluators-rag](references/evaluators-rag.md) → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
|
|
|
|
**Production:**
|
|
[production-overview](references/production-overview.md) → [production-guardrails](references/production-guardrails.md) → [production-continuous](references/production-continuous.md)
|
|
|
|
## Reference Categories
|
|
|
|
| Prefix | Description |
|
|
| ------ | ----------- |
|
|
| `fundamentals-*` | Types, scores, anti-patterns |
|
|
| `observe-*` | Tracing, sampling |
|
|
| `error-analysis-*` | Finding failures |
|
|
| `axial-coding-*` | Categorizing failures |
|
|
| `evaluators-*` | Code, LLM, RAG evaluators |
|
|
| `experiments-*` | Datasets, running experiments |
|
|
| `validation-*` | Validating evaluator accuracy against human labels |
|
|
| `production-*` | CI/CD, monitoring |
|
|
|
|
## Key Principles
|
|
|
|
| Principle | Action |
|
|
| --------- | ------ |
|
|
| Error analysis first | Can't automate what you haven't observed |
|
|
| Custom > generic | Build from your failures |
|
|
| Code first | Deterministic before LLM |
|
|
| Validate judges | >80% TPR/TNR |
|
|
| Binary > Likert | Pass/fail, not 1-5 |
|