Azure OpenAI

Azure OpenAI Provider

Use ArtemisKit with Azure OpenAI Service.

Important: Model vs Deployment

The model field in your scenario is for display purposes only when using Azure OpenAI.

Unlike standard OpenAI, Azure OpenAI uses deployments. The actual model is determined by your Azure deployment configuration, not by what you specify in ArtemisKit.

Setup

1. Create Azure OpenAI Resource

Create an Azure OpenAI resource in the Azure Portal and deploy a model.

2. Set Environment Variables

export AZURE_OPENAI_API_KEY="your-key"
export AZURE_OPENAI_RESOURCE_NAME="your-resource-name"
export AZURE_OPENAI_DEPLOYMENT_NAME="your-deployment-name"
export AZURE_OPENAI_API_VERSION="2024-02-15-preview"  # Optional

3. Configure Scenario

name: my-test
provider: azure-openai
model: gpt-5  # Display only - actual model is your deployment

cases:
  - id: example
    prompt: "Hello"
    expected:
      type: contains
      values: ["hello"]
      mode: any

Config File

Configure Azure OpenAI in your artemis.config.yaml:

provider: azure-openai
model: gpt-5  # Display only

providers:
  azure-openai:
    apiKey: ${AZURE_OPENAI_API_KEY}
    resourceName: ${AZURE_OPENAI_RESOURCE_NAME}
    deploymentName: ${AZURE_OPENAI_DEPLOYMENT_NAME}
    apiVersion: 2024-02-15-preview
    timeout: 60000
    maxRetries: 2

Environment Variables

Variable	Description
`AZURE_OPENAI_API_KEY`	Your Azure OpenAI API key
`AZURE_OPENAI_RESOURCE_NAME`	Your resource name
`AZURE_OPENAI_DEPLOYMENT_NAME`	Your deployment name
`AZURE_OPENAI_API_VERSION`	API version (optional)

API Versions

Common API versions:

2024-02-15-preview (recommended)
2023-12-01-preview
2023-05-15

Troubleshooting

Authentication Error

Error: 401 Unauthorized

Check that:

AZURE_OPENAI_API_KEY is correct
AZURE_OPENAI_RESOURCE_NAME matches your Azure resource
The API key has access to the resource

Deployment Not Found

Error: 404 DeploymentNotFound

Ensure AZURE_OPENAI_DEPLOYMENT_NAME matches an existing deployment in your Azure resource.