Skip to Content
Prompt-Spec - Next Generation Agent Prompt Optimization
DocumentationAPI Reference

API Reference

This page provides a comprehensive reference for Prompt Spec’s programmatic API.

Installation

To use the Prompt Spec API in your project:

npm install prompt-spec
# or
pnpm add prompt-spec

Core API Functions

testSpec

Run benchmarks against an agent specification.

import { testSpec } from "prompt-spec";
 
const results = await testSpec(specPath, options);

Parameters:

  • specPath (string): Path to the YAML specification file
  • options (object, optional):
    • watch (boolean): Watch for changes and rerun tests
    • reporter (string): Test reporter to use
    • benchmark (string): Run a specific benchmark by name
    • timeout (number): Timeout for each benchmark in seconds
    • outputPath (string): Path to save results

Returns:

  • Promise resolving to a results object

transpileSpec

Convert a YAML specification to a test file.

import { transpileSpec } from "prompt-spec";
 
const testFilePath = await transpileSpec(specPath, options);

Parameters:

  • specPath (string): Path to the YAML specification file
  • options (object, optional):
    • output (string): Output path for the generated test file
    • format (string): Output format (‘ts’ or ‘js’)

Returns:

  • Promise resolving to the path of the generated test file

optimizeAgent

Optimize an agent based on benchmark results.

import { optimizeAgent } from "prompt-spec";
 
const optimizedSpec = await optimizeAgent(specPath, options);

Parameters:

  • specPath (string): Path to the YAML specification file
  • options (object, optional):
    • iterations (number): Number of optimization iterations
    • outputPath (string): Path to save the optimized specification
    • model (string): Model to use for optimization
    • strategy (string): Optimization strategy (‘prompt’, ‘tools’, ‘full’)
    • verbose (boolean): Show detailed optimization process

Returns:

  • Promise resolving to the optimized specification object

Agent API

Agent

Create and interact with an agent.

import { Agent } from "prompt-spec";
 
// Create from a specification file
const agent = await Agent.fromSpecFile("path/to/spec.yaml");
 
// Create from a specification object
const agent = new Agent(specObject);
 
// Run a conversation
const result = await agent.runConversation([
  { role: "user", content: "Hello, can you help me?" },
]);

Methods:

  • runConversation(messages): Run a conversation with the agent
  • runBenchmark(benchmark): Run a specific benchmark
  • runAllBenchmarks(): Run all benchmarks defined in the specification

Benchmark API

Benchmark

Create and run benchmarks.

import { Benchmark } from "prompt-spec";
 
// Create a benchmark
const benchmark = new Benchmark({
  name: "Simple Test",
  messages: [{ role: "user", content: "What is the capital of France?" }],
  evaluationCriteria: [
    { key: "accuracy", description: "Is the answer correct?", type: "boolean" },
  ],
});
 
// Run against an agent
const result = await benchmark.run(agent);

Tool API

ToolRegistry

Register and manage tools.

import { ToolRegistry } from "prompt-spec";
 
// Register a tool
ToolRegistry.register("myTool", {
  description: "My custom tool",
  handler: async (input) => {
    // Tool implementation
    return { result: "Tool output" };
  },
  inputSchema: { type: "object", properties: { query: { type: "string" } } },
});
 
// Get a registered tool
const tool = ToolRegistry.get("myTool");

Evaluation API

Evaluator

Create custom evaluators for benchmarks.

import { Evaluator, registerEvaluator } from "prompt-spec";
 
// Create a custom evaluator
class MyEvaluator extends Evaluator {
  evaluate(response, expected) {
    // Evaluation logic
    return { score: 0.8, reason: "Explanation of score" };
  }
}
 
// Register the evaluator
registerEvaluator("myEvaluator", MyEvaluator);

Utility Functions

loadSpec

Load a specification from a file.

import { loadSpec } from "prompt-spec";
 
const spec = await loadSpec("path/to/spec.yaml");

validateSpec

Validate a specification.

import { validateSpec } from "prompt-spec";
 
const validationResult = validateSpec(spec);
if (validationResult.valid) {
  // Specification is valid
} else {
  console.error(validationResult.errors);
}

generateReport

Generate a report from benchmark results.

import { generateReport } from "prompt-spec";
 
const report = generateReport(results, {
  format: "markdown",
  includeDetails: true,
});

Configuration

configure

Configure global settings for Prompt Spec.

import { configure } from "prompt-spec";
 
configure({
  defaultModel: "gpt-4o",
  apiKey: process.env.OPENAI_API_KEY,
  timeout: 60,
  logLevel: "info",
});

Events

Prompt Spec provides an event system for monitoring the testing process:

import { events } from "prompt-spec";
 
events.on("benchmark:start", (benchmark) => {
  console.log(`Starting benchmark: ${benchmark.name}`);
});
 
events.on("benchmark:complete", (result) => {
  console.log(`Benchmark complete: ${result.score}`);
});

Next Steps

Last updated on
Agent SpecificationsBenchmarking