Open Source Alternatives
  • Alternatives
  • Categories
    Categories by size1,270 projects
    • Developer Tools652
    • AI & Machine Learning538
    • Business & Productivity306
    • Web Development217
    • Security & Monitoring197
    • Communication & Collaboration169
    • IT Management161
    • Design & Creative139
    • DevOps & CI/CD137
    • LLMOps & AI Tooling137
    Browse all 21 categoriesCloud & Hosting, Product & Project Management and 9 more
  • Blog
  • Advertise
SearchNotion, Slack, password manager.../
Submit
Open Source Alternatives
  • Alternatives
    • Developer Tools652
    • AI & Machine Learning538
    • Business & Productivity306
    • Web Development217
    • Security & Monitoring197
    • Communication & Collaboration169
    • Browse all 21 categories
  • Blog
  • Advertise
Submit your project

New open source projects by email

The projects we add and every new guide. Free, unsubscribe any time.

Open Source Alternatives LogoOpen Source Alternatives

Handpicked Open Source Alternatives to Paid Softwares

Add Open Source Alternatives as a preferred source on Google

Product

  • Categories
  • Tag
  • Advertise

Resources

  • Blog
  • Collection
  • Submit

Company

  • Privacy Policy
  • Terms of Service
  • Refund Policy
  • Sitemap

Alternatives

  • Claude Code
  • Jira
  • Notion
  • Slack
  • Linear
  • Wispr Flow
  • All alternatives
© 2026 Open Source Alternatives. All rights reserved.
Open SourceAlternatives
Home/Categories/AI & Machine Learning/SkillEvaluator
icon of SkillEvaluator

SkillEvaluator

  • 1.7K stars
  • Apache-2.0
  • Active this week
  • Last commit today

Last verified Nov 16, 2026

Run quality checks, semantic deduplication, and live sandbox evaluations on AI agent skills through a three-tier CLI and CI pipeline.

Visit websiteGitHub1.7K stars
image of SkillEvaluator
TL;DR

SkillEvaluator is an open source, three-tier framework from NVIDIA for validating AI agent skills before they ship in a catalog or agent. It fills a gap that commercial AI evaluation platforms leave open: purpose-built checks for the Agent Skills specification, covering deterministic quality gates, semantic overlap detection, and live sandboxed behavioral measurement. Apache 2.0 licensed and installable via pip, Tier 1 runs fully offline with no API key. Best for platform teams building skill catalogs for Claude Code, Codex, or OpenCode, and for open source catalog maintainers who need automated CI gates on skill contributions.

Read the full breakdownHide the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ
who it's for

Who SkillEvaluator is for#

AI platform teams publishing internal skill catalogs

Teams maintaining skills for Claude Code, Codex, or similar agents can wire SkillEvaluator into their CI pipeline to gate every pull request. Tier 1 catches security and quality issues before they reach catalog users; Tier 2 flags skill overlap that would put conflicting guidance in front of agents.

Skip if:

Your team publishes a small number of one-off skills and reviews each manually. The CI setup overhead is not worth it for catalogs with fewer than a handful of contributed skills.

Skill authors building shareable AI coding agent extensions

Individual developers and teams creating skills for Claude Code, Codex, or other skill-aware agents can run SkillEvaluator locally before submitting. The validate command runs all three tiers and surfaces quality, security, and behavioral issues before catalog review.

Skip if:

You are building a one-off skill for personal use with no plans to contribute it to a shared catalog. Local quality checks add value regardless, but the CI gate setup is not worth the overhead for private, single-skill use.

Security teams auditing AI coding agent extensions

Tier 1's static security suite scans skill files for secret leaks (Gitleaks) and skill-specific vulnerabilities (SkillSpector) with no API key or internet connection. Semgrep is optional for broader code scanning. Offline operation lets security teams review skills in isolated environments before approving deployment.

Skip if:

Your concern is evaluating LLM response quality in applications, not vetting instruction files. Braintrust or LangSmith are designed for that evaluation target.

the problem

The problem it solves#

When teams build agent skills for AI coding tools, they face no standard quality bar before publishing. A skill might carry security vulnerabilities, undocumented PII risks, duplicate guidance that conflicts with existing skills, or instructions that don't measurably improve the agent's output on real tasks. Catching these problems after the skill is live means users encounter broken or risky behavior with no warning.

The problem deepens at scale. A catalog of dozens of community-contributed skills has no automated deduplication step, so overlapping guidance accumulates. Measuring whether a skill actually helps the agent requires a live sandbox run comparing agent performance with and without it, and no general-purpose eval tool offers this for skill folders.

how SkillEvaluator solves it

How it solves it#

Tier 1 deterministic quality gates

Validates schema, documentation quality, security (SkillSpector and Gitleaks), PII, license, and script integrity offline with no API key. Scores documentation quality out of 100 with a default passing threshold of 70. The bundled text-analyzer skill lets you run a first check before preparing your own skill.

Tier 2 semantic overlap detection

Uses embedding similarity to find repeated guidance inside a single skill and to compare skills across a catalog. Catches conflicts and redundancies before they reach production. Requires an embeddings provider; intra-skill analysis also supports local OpenAI-compatible endpoints.

Tier 3 live sandboxed agent evaluation

Runs a real agent inside Docker with and without the skill on generated tasks, then scores the difference across five dimensions: Security, Correctness, Discoverability, Effectiveness, and Efficiency. The Skill Lift metric isolates whether the skill earns its place in the catalog.

CI integration with GitHub Actions and GitLab

Running the CI init command opens a setup pull request on GitHub. A GitLab CI template is included for copy-paste setup. Exit codes map cleanly to pass/fail gates so a failing skill blocks the merge without custom configuration.

LLM-backed rubric evaluation

Tier 1 includes an optional rubric-eval command that sends skill documentation to a model and returns structured feedback. Works with NVIDIA Build endpoints for free prototyping and with any OpenAI-compatible provider. Useful for catching documentation quality issues that static checks miss.

strengths · trade-offs

Strengths and trade-offs#

Strengths

  • Offline Tier 1 with no API keyQuality checks, schema validation, security scanning, and documentation scoring all run offline. Teams in security-conscious or air-gapped environments can run the full static suite without any cloud provider setup. The bundled text-analyzer example scores 86.2/100 (grade B) using only the installed package.
  • Agent Skills CI integration out of the boxA single CI init command creates a GitHub pull request configuring a workflow to check every changed skill on each pull request. Catching quality and security issues at review time prevents regressions from reaching catalog users. GitLab CI support ships as a ready-to-copy template.
  • Apache 2.0 licenseApache 2.0 permits commercial use, modification, and redistribution with no network-use restrictions. Teams building internal skill catalogs or commercial products can ship SkillEvaluator-validated skills without licensing constraints.
  • Five-dimension behavioral measurementTier 3 scores Security, Correctness, Discoverability, Effectiveness, and Efficiency in live agent runs. The Skill Lift metric gives a single number showing whether the skill improved agent task performance, making it easier to compare skill versions or justify publishing a new skill.

Trade-offs

  • -Experimental support level, no SLANVIDIA classifies SkillEvaluator's support as Experimental: community-supported through GitHub Issues on a best-effort basis, with no SLA or NVIDIA enterprise support entitlement. Teams that need production-grade support agreements should factor this in before adopting it as a CI gate.
  • -Tier 3 adds agent credential and sandbox overheadLive evaluation requires a supported agent CLI (Claude Code, Codex, OpenClaw), an API key for LLM generation and grading, and a Docker sandbox. The setup is documented but non-trivial, and sandbox usage for Tier 3 can incur model and managed-sandbox charges.
  • -Full security scan requires external executablesComplete Tier 1 security coverage depends on SkillSpector and Gitleaks binaries (auto-fetched when missing) and optional Semgrep. When a required scanner cannot run, the check exits nonzero as INCOMPLETE rather than skipping it, which blocks CI until the scanner is installed.
versus alternatives

SkillEvaluator vs alternatives#

SkillEvaluator vs Braintrust

SkillEvaluator and Braintrust both measure AI behavior, but serve different evaluation targets. Braintrust is a commercial evaluation platform for scoring LLM application outputs against test datasets; SkillEvaluator is an open source CLI purpose-built for evaluating agent skill folders, the instruction packages that extend AI coding agents.

FeatureSkillEvaluatorBraintrust
LicenseApache-2.0Proprietary
Self-hostingYes (pip install)No (hosted SaaS)
Evaluation targetAgent skill folders (agentskills.io)LLM app responses and chains
Offline Tier 1Yes, no API key neededNo
CI integrationGitHub Actions, GitLab (built-in)GitHub Actions (custom)
PricingFreePaid plans

SkillEvaluator is the right choice when your team builds, reviews, or publishes skills for AI coding agents (Claude Code, Codex, OpenCode) and needs a CI gate that runs offline security and quality checks before merge. Its three-tier pipeline provides checks that general LLM evaluation platforms do not offer for skill folders: schema validation, security scanning via SkillSpector and Gitleaks, PII detection, semantic deduplication across a catalog, and behavioral measurement via live sandboxed agent runs.

Braintrust is better suited for teams evaluating LLM application outputs: scoring text generation, testing prompt variations, comparing model responses against golden datasets, or tracking evaluation scores over time in a hosted dashboard. If your evaluation target is not an agent skill folder but rather an LLM response or a chain's output, Braintrust's evaluation runner and dataset management tools serve that use case well.

install · quick start

Quick start#

bash
Installing SkillEvaluator as a Python package gives you the CLI for static quality checks, security scans, and live skill evaluations.

```bash
pip install skillevaluator
```
tech stack · detected from GitHub

What it's built on#

Languages
Python
Infrastructure
AWS
frequently asked

FAQ#

What is an AI agent skill, and why does it need evaluation?

An agent skill is a folder of instruction files defined by the Agent Skills specification (agentskills.io) that extends an AI agent like Claude Code or Codex with new capabilities. Without evaluation, a skill may carry security vulnerabilities, low-quality documentation, or instructions that duplicate existing guidance. SkillEvaluator's three tiers check quality and safety statically, then measure whether the skill actually improves agent task performance in live runs.

Can I run SkillEvaluator without a GPU or cloud API key?

Yes, for Tier 1 deterministic checks. The base package installed via pip requires only Python 3.12 or later. No GPU, no API key, and no Docker is needed for static quality, schema, PII, and license checks. Full security scanner coverage adds the SkillSpector and Gitleaks binaries, which SkillEvaluator fetches automatically when missing.

How does SkillEvaluator fit into a GitHub Actions CI pipeline?

The CI init command opens a pull request from your repository root that sets up a GitHub Actions workflow to check every changed skill on each pull request. SkillEvaluator's exit codes serve as the pass/fail gate: a non-zero exit blocks the merge. A GitLab CI YAML template is included for teams not on GitHub.

What agents does Tier 3 live evaluation support?

Tier 3 supports Claude Code, Codex, OpenCode, and OpenClaw, running inside Docker, local OS, or cloud sandboxes. Each agent requires its own credential. Sandbox usage for live Tier 3 evaluation may incur model API and managed-sandbox charges depending on the provider configuration.

Is SkillEvaluator suitable for evaluating general LLM prompts or chains?

No. SkillEvaluator is built specifically for the Agent Skills specification: instruction folders that extend AI coding agents. It evaluates skill quality, security, and behavioral impact on agents, not arbitrary LLM responses or prompt performance. For evaluating LLM application output quality, tools like Braintrust or LangSmith are designed for that purpose.

also worth a look

Similar open-source tools#

Maestro

Alternative toConductor

Desktop orchestrator for parallel AI coding agent fleets

  • 3,574 GitHub stars
  • AGPL-3.0Licence AGPL-3.0

Bifrost

Alternative toPortkey

Open-source AI gateway with 23+ providers, failover, and MCP support.

  • 9,555 GitHub stars
  • Apache-2.0Licence Apache-2.0

Nuphos

Featured

Alternative toConductor

Agent workspace for teams running Claude Code or Codex

  • 529 GitHub stars
  • Apache-2.0Licence Apache-2.0

Hermes Agent

Featured

Alternative toGrok Bot

Self-hosted AI agent: persistent memory, multi-channel chat, model choice.

  • 263,254 GitHub stars
  • MITLicence MIT

Llama.cpp

Alternative toOpenAI API

C/C++ LLM inference engine with OpenAI-compatible API, no cloud.

  • 135,212 GitHub stars
  • MITLicence MIT

HA-MCP

Alternative toJosh.ai

Natural language Home Assistant control via MCP, MIT licensed.

  • 7,757 GitHub stars
  • MITLicence MIT
by category

Open source tools in these categories#

538

AI & Machine Learning

freeCodeCamp, REA (Reverse Engineer Anything), OpenClaw
137

LLMOps & AI Tooling

Hermes Agent, OpenCode, MarkItDown
652

Developer Tools

REA (Reverse Engineer Anything), Skills for Real Engineers, superpowers
137

DevOps & CI/CD

Nvm, Uptime Kuma, Netdata

Sponsors

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

  • 13,025 GitHub stars
  • AGPL-3.0Licence AGPL-3.0
  • #Email

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

  • 8,291 GitHub stars
  • Apache-2.0Licence Apache-2.0
  • #AI Agents

SnitchFeed

Live monitoring on LinkedIn, X, Reddit & more. Agentic social listening for startups.

Marketing & Growth

Advertise your product

New open source projects by email

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

/
  • Claude Code to OpenCode
  • Jira to Windshift
  • Notion to AppFlowy
  • Slack to Mattermost
  • Linear to GitLab
  • Wispr Flow to Handy
  • Arc to Zen Browser
  • CapCut to Concat
  • Grok Bot to rakazo
  • Superhuman to Inbox Zero
  • Discord to Stoat
  • Airtable to AITable
  • Datadog to SigNoz
  • Muse to OpenClaw
  • Railway to Coolify
  • GitHub Copilot to Cline
  • Confluence to Wiki.js
  • Microsoft Teams to Rocket.Chat
  • 1Password to KeePassXC
  • Firebase to Supabase
  • Pinecone to Qdrant
  • Zapier to n8n
  • Devin AI to OpenHands
  • Asana to Leantime
  • Claude Code to OpenCode
  • Jira to Windshift
  • Notion to AppFlowy
  • Slack to Mattermost
  • Linear to GitLab
  • Wispr Flow to Handy
  • Arc to Zen Browser
  • CapCut to Concat
  • Grok Bot to rakazo
  • Superhuman to Inbox Zero
  • Discord to Stoat
  • Airtable to AITable
  • Datadog to SigNoz
  • Muse to OpenClaw
  • Railway to Coolify
  • GitHub Copilot to Cline
  • Confluence to Wiki.js
  • Microsoft Teams to Rocket.Chat
  • 1Password to KeePassXC
  • Firebase to Supabase
  • Pinecone to Qdrant
  • Zapier to n8n
  • Devin AI to OpenHands
  • Asana to Leantime