mirror of https://github.com/github/awesome-copilot.git synced 2026-04-12 11:15:56 +00:00

Files

Jim Bennett d79183139a Add Arize and Phoenix LLM observability skills (#1204 )

* Add 9 Arize LLM observability skills

Add skills for Arize AI platform covering trace export, instrumentation,
datasets, experiments, evaluators, AI provider integrations, annotations,
prompt optimization, and deep linking to the Arize UI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add 3 Phoenix AI observability skills

Add skills for Phoenix (Arize open-source) covering CLI debugging,
LLM evaluation workflows, and OpenInference tracing/instrumentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Ignoring intentional bad spelling

* Fix CI: remove .DS_Store from generated skills README and add codespell ignore

Remove .DS_Store artifact from winmd-api-search asset listing in generated
README.skills.md so it matches the CI Linux build output. Add queston to
codespell ignore list (intentional misspelling example in arize-dataset skill).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add arize-ax and phoenix plugins

Bundle the 9 Arize skills into an arize-ax plugin and the 3 Phoenix
skills into a phoenix plugin for easier installation as single packages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix skill folder structures to match source repos

Move arize supporting files from references/ to root level and rename
phoenix references/ to rules/ to exactly match the original source
repository folder structures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fixing file locations

* Fixing readme

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-02 09:58:55 +11:00

3.0 KiB

Raw Blame History

LLM Spans

Represent calls to language models (OpenAI, Anthropic, local models, etc.).

Required Attributes

Attribute	Type	Description
`openinference.span.kind`	String	Must be "LLM"
`llm.model_name`	String	Model identifier (e.g., "gpt-4", "claude-3-5-sonnet-20241022")

Key Attributes

Category	Attributes	Example
Model	`llm.model_name`, `llm.provider`	"gpt-4-turbo", "openai"
Tokens	`llm.token_count.prompt`, `llm.token_count.completion`, `llm.token_count.total`	25, 8, 33
Cost	`llm.cost.prompt`, `llm.cost.completion`, `llm.cost.total`	0.0021, 0.0045, 0.0066
Parameters	`llm.invocation_parameters` (JSON)	`{"temperature": 0.7, "max_tokens": 1024}`
Messages	`llm.input_messages.{i}.`, `llm.output_messages.{i}.`	See examples below
Tools	`llm.tools.{i}.tool.json_schema`	Function definitions

Cost Tracking

Core attributes:

llm.cost.prompt - Total input cost (USD)
llm.cost.completion - Total output cost (USD)
llm.cost.total - Total cost (USD)

Detailed cost breakdown:

llm.cost.prompt_details.{input,cache_read,cache_write,audio} - Input cost components
llm.cost.completion_details.{output,reasoning,audio} - Output cost components

Messages

Input messages:

llm.input_messages.{i}.message.role - "user", "assistant", "system", "tool"
llm.input_messages.{i}.message.content - Text content
llm.input_messages.{i}.message.contents.{j} - Multimodal (text + images)
llm.input_messages.{i}.message.tool_calls - Tool invocations

Output messages: Same structure as input messages.

Example: Basic LLM Call

{
  "openinference.span.kind": "LLM",
  "llm.model_name": "claude-3-5-sonnet-20241022",
  "llm.invocation_parameters": "{\"temperature\": 0.7, \"max_tokens\": 1024}",
  "llm.input_messages.0.message.role": "system",
  "llm.input_messages.0.message.content": "You are a helpful assistant.",
  "llm.input_messages.1.message.role": "user",
  "llm.input_messages.1.message.content": "What is the capital of France?",
  "llm.output_messages.0.message.role": "assistant",
  "llm.output_messages.0.message.content": "The capital of France is Paris.",
  "llm.token_count.prompt": 25,
  "llm.token_count.completion": 8,
  "llm.token_count.total": 33
}

Example: LLM with Tool Calls

{
  "openinference.span.kind": "LLM",
  "llm.model_name": "gpt-4-turbo",
  "llm.input_messages.0.message.content": "What's the weather in SF?",
  "llm.output_messages.0.message.tool_calls.0.tool_call.function.name": "get_weather",
  "llm.output_messages.0.message.tool_calls.0.tool_call.function.arguments": "{\"location\": \"San Francisco\"}",
  "llm.tools.0.tool.json_schema": "{\"type\": \"function\", \"function\": {\"name\": \"get_weather\"}}"
}

3.0 KiB Raw Blame History