Getting Started

This guide will help you install Prompt Spec and run your first benchmark test to optimize your AI agent prompts.

Installation

Prerequisites

Before installing Prompt Spec, make sure you have the following:

Node.js 18 or higher
npm or pnpm (recommended)
An API key for your preferred LLM provider (OpenAI, Anthropic, etc.)

Install from npm


npm install prompt-spec

Or with pnpm:


pnpm add prompt-spec

Install from source


git clone https://github.com/Slyracoon23/prompt-spec.git
cd prompt-spec-optimizer
pnpm install
pnpm build

Configuration

After installation, you’ll need to configure your API keys. Create a .env file in your project root:


# .env file
OPENAI_API_KEY=your_openai_api_key
ANTHROPIC_API_KEY=your_anthropic_api_key
# Add other provider keys as needed

Or set them as environment variables:


export OPENAI_API_KEY=your_openai_api_key

Quick Start

After installation, you can run a simple benchmark test using the CLI:


# Run tests from a YAML specification
prompt-spec test path/to/spec.yaml

Basic Example

Create a file named simple-agent.yaml with the following content:


metadata:
  name: "Simple Question Answering Agent"
  version: "1.0"
  description: "A basic agent for testing question answering capabilities"
 
agent:
  model: gpt-4o-mini
  systemPrompt: |
    You are a helpful assistant that provides factual information.
  maxSteps: 2
 
benchmarks:
  - name: "Basic Questions"
    messages:
      - role: "user"
        content: "What is the capital of France?"
    evaluationCriteria:
      - key: "accuracy"
        description: "Is the answer correct?"
        type: "boolean"
      - key: "conciseness"
        description: "Is the answer concise?"
        type: "scale"
        min: 1
        max: 5

Then run the test:


prompt-spec test simple-agent.yaml

Optimizing Your Agent

Prompt Spec can automatically optimize your agent’s prompts based on benchmark results:


# Run optimization on your agent specification
prompt-spec optimize simple-agent.yaml --iterations 5 --output optimized-agent.yaml

This will:

Run your agent against the benchmarks
Analyze performance metrics
Generate improved system prompts
Test the new prompts
Repeat for the specified number of iterations

Viewing Results

After running benchmarks or optimizations, you can view detailed results:


# Generate an HTML report
prompt-spec report --input results.json --output report.html
 
# Or view in the terminal
prompt-spec report --input results.json --format terminal

Next Steps

Learn about Core Concepts
Explore Agent Specifications
Check out more detailed Examples
Understand Optimization Strategies