CLI Overview

ArtemisKit CLI

One CLI. Three capabilities. The fastest way to test your LLM applications.

Installation

npm install -g @artemiskit/cli

Commands

Command	Description
`artemiskit run`	Run scenario-based evaluations
`artemiskit redteam`	Security red team testing
`artemiskit stress`	Load and stress testing
`artemiskit validate`	Validate scenario files without running
`artemiskit compare`	Compare two evaluation runs
`artemiskit baseline`	Manage baselines for regression detection
`artemiskit report`	Generate reports from saved runs
`artemiskit history`	View run history
`artemiskit init`	Initialize configuration

You can use akit as a shorter alias for artemiskit.

Quick Start

# Set your API key
export OPENAI_API_KEY="sk-..."

# Run a scenario (results are saved by default)
akit run scenarios/my-test.yaml

# Run without saving results
akit run scenarios/my-test.yaml --no-save

Configuration

Create artemis.config.yaml for default settings:

provider: openai
model: gpt-5

providers:
  openai:
    apiKey: ${OPENAI_API_KEY}
    timeout: 60000

output:
  format: json
  dir: ./artemis-output

What’s Next

Getting Started — First test in 5 minutes
Core Concepts — Scenarios, expectations, providers, evaluators
Scenario Format — YAML schema reference
Commands — Full command reference
Storage — Local and Supabase backends
Providers — LLM provider configuration
Examples — CI/CD, security testing, regression detection