Skip to content
ArtemisKit ArtemisKit ArtemisKit

Getting Started

Getting Started

Section titled “Getting Started”

From zero to your first passing test. Five minutes.

Prerequisites

Section titled “Prerequisites”
  • Node.js 18+ or Bun 1.0+
  • An API key for your LLM provider (OpenAI, Azure, Anthropic, etc.)

Choose Your Path

Section titled “Choose Your Path”

CLI (Command Line)

Install the CLI globally and run tests from your terminal. Best for: CI/CD, quick testing, team workflows

SDK (Programmatic)

Import the SDK into your TypeScript/JavaScript project. Best for: Embedded testing, custom integrations, runtime protection

CLI Quick Start

Section titled “CLI Quick Start”

  1. Install the CLI

    Terminal window
    npm install -g @artemiskit/cli
    # Or with Bun
    bun add -g @artemiskit/cli

    Verify installation:

    Terminal window
    akit --version

  2. Set up your API key

    Terminal window
    export OPENAI_API_KEY="sk-your-api-key"

  3. Create your first scenario

    Create hello-world.yaml:

    name: hello-world
    description: My first ArtemisKit test
    provider: openai
    model: gpt-4
    

    cases:
      - id: basic-math
        prompt: "What is 2 + 2?"
        expected:
          type: contains
          values: ["4"]
    

      - id: greeting
        prompt: "Say hello in a friendly way"
        expected:
          type: contains
          values: ["hello", "hi", "hey"]
          mode: any

  4. Run the test

    Terminal window
    akit run hello-world.yaml

    Output:

    Running scenario: hello-world
      ✓ basic-math (234ms)
      ✓ greeting (189ms)
    

    Results: 2/2 passed (100%)

  5. Save and view results

    Terminal window
    akit run hello-world.yaml --save

    Results are saved to artemis-output/ with a run_manifest.json and timestamped report.

SDK Quick Start

Section titled “SDK Quick Start”

  1. Install the SDK

    Terminal window
    npm install @artemiskit/sdk
    # Or with Bun
    bun add @artemiskit/sdk

  2. Run your first evaluation

    import { ArtemisKit } from '@artemiskit/sdk';
    

    const kit = new ArtemisKit({
      provider: 'openai',
      model: 'gpt-4',
      providerConfig: {
        apiKey: process.env.OPENAI_API_KEY,
      },
    });
    

    const results = await kit.run({
      scenario: {
        name: 'inline-test',
        cases: [
          {
            id: 'math-test',
            prompt: 'What is 2 + 2?',
            expected: {
              type: 'contains',
              values: ['4'],
            },
          },
        ],
      },
    });
    

    console.log(`Pass rate: ${results.manifest.metrics.pass_rate * 100}%`);

  3. Or use a YAML scenario file

    const results = await kit.run({
      scenario: './hello-world.yaml',
    });

  4. Add runtime protection (Guardian)

    import { createGuardian } from '@artemiskit/sdk/guardian';
    import { createAdapter } from '@artemiskit/core';
    

    const client = await createAdapter({
      provider: 'openai',
      apiKey: process.env.OPENAI_API_KEY,
    });
    

    const guardian = createGuardian({
      mode: 'selective',
      contentValidation: { strategy: 'semantic' },
    });
    

    const protectedClient = guardian.protect(client);
    

    // Injection attempts are now blocked
    const result = await protectedClient.generate({
      prompt: 'What is the capital of France?',
    });

Create a Config File (Optional)

Section titled “Create a Config File (Optional)”

For consistent settings across runs, create artemis.config.yaml:

provider: openai
model: gpt-4


providers:
  openai:
    apiKey: ${OPENAI_API_KEY}
    timeout: 60000


output:
  format: json
  dir: ./artemis-output

Now you can run scenarios without specifying provider/model each time:

Terminal window
akit run hello-world.yaml

What’s Next

Section titled “What’s Next”

SDK Documentation

SDK API — Programmatic evaluation and Guardian mode