Start here
Ask your coding agent something like this:
Run npx skills add HaliosAI/halios --skill halios, then use the Halios skill to set up evals for this agent.
Your coding agent will inspect the project, configure Halios, create the initial eval suite, and run a baseline.
Common workflows
Create an eval suite
Ask your coding agent to add scenarios and checks for a behavior you care about.
Run and debug evals
Run the suite, inspect failures, and trace them back to agent behavior.
Verify a change
Make a prompt, tool, or code change and rerun the affected evals.
Add CI
Run evals on pull requests and block regressions.
Evaluate production behavior
Send OpenTelemetry traces to Halios and turn production failures into regression scenarios.
Developer tools
- Skill + CLI — primary coding-agent workflow
- Python SDK — runtime / guardrail-style evaluation
Engineering & research
- How Coding Agents Build Better AI Agent Evaluations ↗
How coding agents use repository context to create scenarios, define checks, and maintain eval suites.
- AI Agent Evals from Local Development to CI and Production ↗
Using one unified evaluation model across local iteration, pull-request gates, and production behavior.
- OpenTelemetry for Portable AI Agent Evaluation Traces ↗
Standardizing multi-turn agent instrumentation with OpenTelemetry GenAI conventions instead of proprietary SDKs.
- Why AI Agent Evals Belong in Your Git Repository ↗
Keeping evaluation scenarios and checks directly alongside agent code for PR reviews and coding agent context.
- Why Halios Uses Scenarios Instead of Eval Datasets ↗
Why static datasets fail multi-turn agents and how scenario simulations uncover emergent failures.
Reference
Prompting guide ↗·CLI ↗·OpenTelemetry ↗·Python SDK ↗·GitHub ↗