Ship AI Products That Actually Work

Name: ArtemisKit CLI
Author: ArtemisKit

Catch broken prompts before your users do. Run quality checks, security scans, and stress tests — in one command.

5-minute setup

50+ attack patterns

GitHub Actions ready

Get Started Free Star on GitHub

npm install -g @artemiskit/cli

Terminal

Open source and growing

stars forks

Apache-2.0

The Challenge

LLM Applications Are Unpredictable

Inconsistent Outputs

Same prompt, different answer. Users notice before you do.

Security Blind Spots

Prompt injection attacks ship to production undetected.

Unknown Limits

First real load test happens when users flood your app.

Manual Testing

Copy-pasting prompts into a chat window is not a test suite.

The Solution

ArtemisKit Brings Order to Chaos

One command to run quality checks, security scans, and stress tests. Get results in your terminal, HTML reports, or CI pipeline.

Reproducible Results

Deterministic seeding. Same test, same result. Every time.
50+ Attack Patterns

Injection, jailbreak, extraction, hallucination, PII disclosure. Five categories, one command.
Ship with Confidence

Fail builds on regressions. Catch issues in CI, not production.

Features

Test Quality. Harden Security. Ship Faster.

One CLI to catch prompt failures, block injection attacks, and stress-test your LLM before production.

Scenario-Based Testing

Define test cases in YAML, run them across providers, and get consistent, reproducible results.

Multi-turn conversation support
Variable injection & templating
Custom evaluation criteria

scenarios:
  - name: greeting
    turns:
      - role: user
        content: "Hello!"
      - role: assistant
        expect:
          contains: ["Hi", "Hello"]

Security Red Teaming

Automatically test for prompt injection, jailbreaks, and data extraction vulnerabilities.

50+ built-in attack patterns
Custom attack definitions
Vulnerability reports

$ akit redteam \
  --target "system prompt" \
  --categories injection,jailbreak

Stress Testing

Understand your LLM's behavior under pressure with concurrent request testing.

Configurable concurrency
Latency percentiles (p50, p95, p99)
Rate limiting detection

$ akit stress \
  --concurrency 50 \
  --iterations 500

Multi-Provider Support

Native support for major LLM providers with unified configuration and results.

OpenAI, Azure, Anthropic, Google
Local models via Ollama
Consistent API across providers

provider: openrouter
model: anthropic/claude-3-opus

Rich Reports

HTML and JSON reports with detailed metrics, pass/fail breakdowns, and configuration traceability.

Interactive HTML dashboards
Machine-readable JSON export
CI/CD integration ready

$ akit run test.yaml --save
Report: artemis-output/report.html

Developer Experience

CLI-first design, YAML configuration, and seamless integration with your workflow.

Zero-config defaults
Extensive customization
TypeScript/Node.js native

# Install globally
npm install -g @artemiskit/cli

# Run your first test
akit run scenarios/test.yaml

Quick Start

Get Started in 3 Steps

From installation to your first test results in under 5 minutes.

Install

Install globally with npm, yarn, pnpm, or bun

    
 npm install -g @artemiskit/cli
# or
bunx artemiskit --help 

Define

Define your test scenarios in simple YAML

    
 # scenarios/my-test.yaml
name: chatbot-test
config:
  provider: openai
  model: gpt-4

scenarios:
  - name: basic-qa
    turns:
      - role: user
        content: "What is 2+2?"
      - role: assistant
        expect:
          contains: ["4"] 

Run

Run tests and get beautiful reports

    
 $ akit run scenarios/my-test.yaml --save

Running chatbot-test...
✓ Scenario: basic-qa .............. PASS

Results: 1/1 passed (100%)
Report: artemis-output/report.html 

Use Cases

Trusted Across the LLM Lifecycle

From development to production, ArtemisKit supports every stage of your LLM application journey.

Quality Assurance

Catch regressions before they reach production. Run comprehensive test suites on every deployment.

Key Capabilities

Pre-deploy validation
Regression detection
Automated test suites
Pass/fail CI gates

Coming Soon

ArtemisKit Cloud

The CLI you love, with managed infrastructure, team collaboration, and advanced analytics. Zero setup required.

What's coming

Team Workspaces

Collaborate on evaluation suites

Scheduled Runs

Automated periodic testing

Historical Analytics

Track trends over time

Dashboard

Visualize results at a glance

REST API & SDKs

Programmatic control

Integrations

Slack, PagerDuty, Jira

Free tier available

No credit card required

Get Early Access

Be first in line when we launch.

Open Source

Open Source at Heart

ArtemisKit is Apache-2.0 licensed and open source. Contribute, customize, and build on a foundation you control.

Star on GitHub Contribute View Roadmap

Licensed under Apache-2.0