# Runhuman

> Human QA testing via API. Send a URL and test instructions, get structured results from real human testers. Integrates with AI coding agents, CI/CD pipelines, and REST APIs.

Runhuman provides human-in-the-loop QA testing. When you need to verify UI/UX, visual issues, or complex user flows that are hard to automate, call Runhuman to get a real human to test it.

---

# Quick Start

Get your first human QA test running in under 5 minutes.

## Step 1: Run Your First Test

Choose your integration method: REST API, MCP for AI Agents, or GitHub Actions.

### REST API

Make an HTTP request to create a test and wait for results:

```javascript
const response = await fetch('https://runhuman.com/api/run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.RUNHUMAN_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://example.com',
    description: 'Check if the page loads and the main heading is visible',
    outputSchema: {
      pageLoads: { type: 'boolean', description: 'Does the page load?' },
      headingVisible: { type: 'boolean', description: 'Is the heading visible?' }
    }
  })
});

const result = await response.json();
console.log(result.result.data);
// { pageLoads: true, headingVisible: true }
```

The request blocks until a human tester completes the test (typically 2-5 minutes).

### MCP for AI Agents

Add Runhuman to Claude Code, Cursor, or VS Code with one command:

**Claude Code:**
```bash
claude mcp add --transport http runhuman https://runhuman.com/mcp --header "Authorization: Bearer YOUR_API_KEY"
```

**Cursor / VS Code:** Use the MCP configuration with type "http" and URL "https://runhuman.com/mcp"

Then ask your agent:
> "Use Runhuman to test the login flow on staging.myapp.com"

### GitHub Actions

Add human QA testing to your CI/CD pipeline:

```yaml
name: QA Test
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: volter-ai/runhuman-qa-test-action@v0.0.1
        with:
          api-key: ${{ secrets.RUNHUMAN_API_KEY }}
          url: https://staging.myapp.com
          description: Verify the homepage loads correctly
          output-schema: '{"loads":{"type":"boolean","description":"Page loads?"}}'
```

## Step 2: Understand the Results

Every test returns:

| Field | Description |
|-------|-------------|
| result.data | Structured data matching your outputSchema |
| result.explanation | GPT-4o's interpretation of the test |
| testerResponse | Raw feedback from the human tester |
| testerData | Screenshots, video, console logs, network requests |
| costUsd | What the test cost |

Example response:

```json
{
  "status": "completed",
  "result": {
    "success": true,
    "explanation": "Page loaded correctly with visible heading",
    "data": {
      "pageLoads": true,
      "headingVisible": true
    }
  },
  "costUsd": 0.18,
  "testDurationSeconds": 100
}
```

---

# REST API

Integrate Runhuman directly from your backend, scripts, or any HTTP client.

## Authentication

Include your API key in the Authorization header:

```
Authorization: Bearer YOUR_API_KEY
```

Get your key from your organization's API Keys page at https://runhuman.com/dashboard

## Synchronous Endpoint

`POST /api/run` creates a test and waits for completion. The request blocks until a human tester finishes (up to 10 minutes).

```bash
curl -X POST https://runhuman.com/api/run \
  -H "Authorization: Bearer $RUNHUMAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://myapp.com/login",
    "description": "Test login with email test@example.com and password demo123",
    "outputSchema": {
      "loginWorks": { "type": "boolean", "description": "Does login succeed?" },
      "redirectsToHome": { "type": "boolean", "description": "Redirects to dashboard after login?" }
    }
  }'
```

If the test does not complete within 10 minutes, the request returns 408 Timeout.

## Asynchronous Endpoints

For longer tests or parallel testing, use the async pattern:

### Step 1: Create a job

`POST /api/jobs` creates a test and returns immediately with a job ID.

```javascript
const response = await fetch('https://runhuman.com/api/jobs', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://myapp.com/checkout',
    description: 'Complete the full checkout flow',
    targetDurationMinutes: 10,
    outputSchema: {
      checkoutWorks: { type: 'boolean', description: 'Order placed successfully?' }
    }
  })
});

const { jobId } = await response.json();
```

### Step 2: Poll for results

`GET /api/job/:jobId` retrieves the job status and results.

```javascript
async function pollJob(jobId) {
  while (true) {
    const response = await fetch(`https://runhuman.com/api/job/${jobId}`, {
      headers: { 'Authorization': `Bearer ${API_KEY}` }
    });
    const job = await response.json();

    if (job.status === 'completed') {
      return job;
    }

    if (['incomplete', 'abandoned', 'error'].includes(job.status)) {
      throw new Error(`Job failed: ${job.status}`);
    }

    await new Promise(resolve => setTimeout(resolve, 30000));
  }
}
```

## Request Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| url | string | Yes | - | URL for the tester to visit |
| description | string | Yes | - | Instructions for the tester |
| outputSchema | object | No | - | Schema defining data to extract. If omitted, only success/explanation returned |
| targetDurationMinutes | number | No | 30 | Time limit (1-60 minutes) |
| allowDurationExtension | boolean | No | true | Allow tester to request more time |
| maxExtensionMinutes | number/false | No | false | Maximum extension allowed |
| additionalValidationInstructions | string | No | - | Custom instructions for AI validation |
| deviceClass | string | No | desktop | "desktop" or "mobile" |
| repoName | string | No | - | GitHub repo ("owner/repo") for better AI context |


## Handling Responses

A completed test returns:

```json
{
  "status": "completed",
  "result": {
    "success": true,
    "explanation": "Login worked correctly. User was redirected to dashboard.",
    "data": {
      "loginWorks": true,
      "redirectsToHome": true
    }
  },
  "costUsd": 0.18,
  "testDurationSeconds": 100,
  "testerResponse": "I entered the credentials and clicked login...",
  "testerAlias": "Alex",
  "testerAvatarUrl": "https://images.subscribe.dev/uploads/.../phoenix.png",
  "testerData": {
    "screenshots": ["https://..."],
    "videoUrl": "https://..."
  }
}
```

## Error Handling

| HTTP Status | Meaning |
|-------------|---------|
| 400 | Invalid request parameters |
| 401 | Invalid or missing API key |
| 404 | Job not found |
| 408 | Synchronous request timed out (10 minutes) |
| 500 | Server error |

---

# MCP for AI Agents

Add Runhuman to your AI coding agent. Ask it to run human QA tests in natural language.

## Installation

**Claude Code:**
```bash
claude mcp add --transport http runhuman https://runhuman.com/mcp --header "Authorization: Bearer YOUR_API_KEY"
```

## Available Tools

Runhuman exposes 5 MCP tools for orchestrating human QA testing:

### create_job

Creates a custom QA test and returns immediately with a job ID.

| Parameter | Required | Description |
|-----------|----------|-------------|
| url | No* | URL to test |
| description | Yes* | Instructions for the tester |
| template | No | Template name to use as base configuration |
| outputSchema | No | JSON Schema for result extraction. If omitted, only success/explanation returned |
| targetDurationMinutes | No | Time limit (default: 30) |
| allowDurationExtension | No | Allow tester to request more time (default: true) |
| maxExtensionMinutes | No | Max extension allowed (default: unlimited) |
| additionalValidationInstructions | No | Custom instructions for AI validation |
| deviceClass | No | "desktop" or "mobile" |
| repoName | No | GitHub repo ("owner/repo") for AI context |


*Either url+description OR template is required.

### run_template

Create a job from a pre-configured template. Templates let you reuse test configurations without writing full descriptions every time.

| Parameter | Required | Description |
|-----------|----------|-------------|
| template | Yes | Template name (get from list_templates) |
| url | No | Override template's default URL |
| description | No | Additional instructions (appended to template) |
| *any create_job param* | No | Override any template default |

### wait

Idiomatic polling - Waits for a job to complete and returns results automatically. Polls every 10 seconds until completion, timeout, or failure.

| Parameter | Required | Description |
|-----------|----------|-------------|
| jobId | Yes | Job ID from create_job or run_template |
| timeoutSeconds | No | Maximum wait time (default: 600, max: 3600) |

**No manual polling needed!** Just call `wait` once and it automatically polls until the job finishes.

**Returns:** When complete, includes:
- `result`: Structured test results matching your schema
- `testerResponse`: Raw feedback from the human tester
- `testerName`/`testerAlias`: Tester identification
- `testerAvatarUrl`: Avatar image URL for UI display
- `testerData`: Testing artifacts (screenshots, video, console logs, network requests, clicks)
- `costUsd`: Exact cost in USD
- `testDurationSeconds`: Time spent by tester

### get_job

Quick status check without waiting. Get current job status instantly without polling.

| Parameter | Required | Description |
|-----------|----------|-------------|
| jobId | Yes | Job ID to check |

### list_templates

List available templates for your project.

| Parameter | Required | Description |
|-----------|----------|-------------|
| limit | No | Max templates to return (default: 50) |

## Example Prompts

These prompts work well with any AI agent that has Runhuman installed:

**Simple page check:**
> Use Runhuman to verify that example.com loads correctly and shows the main heading.

**Login testing:**
> Use Runhuman to test the login flow on staging.myapp.com. Try email test@example.com with password demo123, then try a wrong password and verify the error message.

**Checkout flow:**
> Use Runhuman to test the checkout on myapp.com. Add a product to cart, fill shipping info, and verify the order total is correct. Give the tester 10 minutes.

**Visual issues:**
> Use Runhuman to check the product page at myapp.com/products/123 for visual issues. Look for broken images, layout problems, or unreadable text.

**Mobile testing:**
> Use Runhuman to test the navigation menu on myapp.com on mobile. Check if it opens and closes correctly and all links work.

## What Happens Behind the Scenes

When you ask your agent to use Runhuman:

1. Agent checks available templates with `list_templates` (if applicable)
2. Agent calls `create_job` or `run_template` with your URL/instructions and an output schema it generates
3. Agent receives a job ID and status message
4. Agent calls `wait` with that job ID - **this automatically polls until complete!**
5. When complete, agent receives structured results and the tester's raw response
6. Agent summarizes the findings for you

---

# CLI Tool

Run human QA tests from your terminal with the Runhuman CLI.

## Installation

```bash
# Install globally
npm install -g runhuman

# Or use with npx (no installation required)
npx runhuman --help
```

**Requirements:** Node.js 18+

## Quick Start

```bash
# 1. Login with your API key
runhuman login

# 2. Create your first test
runhuman create https://myapp.com -d "Test the checkout flow"

# 3. Check the status
runhuman status <jobId>

# 4. Get results
runhuman results <jobId>
```

## Core Commands

### Job Management

```bash
# Create a new test
runhuman create https://myapp.com -d "Test checkout flow"

# With template
runhuman create https://myapp.com --template tmpl_abc123

# Synchronous (wait for result)
runhuman create https://myapp.com -d "Test login" --sync

# Check status
runhuman status job_abc123

# Wait for completion
runhuman wait job_abc123

# Get results
runhuman results job_abc123

# List all jobs
runhuman list

# Delete a job
runhuman delete job_abc123
```

### Authentication

```bash
runhuman login      # Authenticate with API key
runhuman logout     # Clear credentials
runhuman whoami     # Show current user
runhuman tokens balance  # Check balance
```

### Projects

```bash
runhuman projects list
runhuman projects create "My App"
runhuman projects show proj_abc123
runhuman projects update proj_abc123 --name "New Name"
runhuman projects delete proj_abc123
```

### Templates

```bash
runhuman templates list --project proj_abc123
runhuman templates create "Smoke Test" --project proj_abc123
runhuman templates show tmpl_abc123
runhuman templates delete tmpl_abc123
```

## JSON Output Mode

All commands support `--json` flag for machine-readable output:

```bash
runhuman create https://example.com -d "Test" --sync --json
```

---

# GitHub Actions

Add human QA testing to your CI/CD pipeline with two specialized actions.

## Issue Tester Action

Automatically test GitHub issues with human QA and video recordings.

### Quick Setup

```yaml
# .github/workflows/test-issues.yml
name: Test Linked Issues

on:
  workflow_run:
    workflows: [CI]  # Change to your deploy workflow name
    types: [completed]
    branches: [main]

jobs:
  test-issues:
    if: github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.workflow_run.head_sha }}
      - uses: volter-ai/runhuman-issue-tester-action@0.0.6
        with:
          api-key: ${{ secrets.RUNHUMAN_API_KEY }}
          test-url: ${{ vars.RUNHUMAN_TESTING_URL }}
```

### Configuration

| Input | Required | Default | Description |
|-------|----------|---------|-------------|
| api-key | Yes | - | Runhuman API key (starts with `rh_`) |
| github-token | No | `github.token` | GitHub token for API access |
| issue-number | No | - | Test a specific issue (bypasses PR detection) |
| test-url | No | - | Base URL for testing (AI appends paths from issues) |
| qa-label | No | qa-test | Label that marks issues for testing |
| auto-detect | No | true | Let AI decide which issues are testable |
| target-duration-minutes | No | 5 | Target test duration (1-60 minutes) |
| reopen-on-failure | No | true | Reopen issue if test fails |

## QA Test Action

Test any URL with human testers.

```yaml
- uses: volter-ai/runhuman-qa-test-action@v0.0.1
  with:
    api-key: ${{ secrets.RUNHUMAN_API_KEY }}
    url: https://staging.example.com
    description: Verify the homepage loads and navigation works
    output-schema: |
      {
        "pageLoads": { "type": "boolean", "description": "Page loads correctly?" },
        "navWorks": { "type": "boolean", "description": "Navigation works?" }
      }
```

---

# Reference

Complete technical specification for Runhuman.

## Job Lifecycle

| Status | Description | Terminal |
|--------|-------------|----------|
| pending | Job created, queued for posting to testers | No |
| waiting | Posted to Slack, awaiting tester claim | No |
| working | Tester claimed and is actively testing | No |
| completed | Test finished, results extracted | Yes |
| incomplete | Test finished but missing required data | Yes |
| abandoned | Tester abandoned before completing | Yes |
| rejected | Tester determined instructions were invalid | Yes |
| error | System error occurred | Yes |

## Output Schema Format

```typescript
{
  [fieldName: string]: {
    type: "boolean" | "string" | "number" | "array" | "object";
    description: string;
    example?: any;
  }
}
```

Example:

```json
{
  "loginWorks": {
    "type": "boolean",
    "description": "Does login work with valid credentials?"
  },
  "errorMessage": {
    "type": "string",
    "description": "What error appears for invalid password?"
  },
  "issuesFound": {
    "type": "array",
    "description": "List of any UI/UX issues discovered"
  }
}
```

## Cost

Tests are billed per second at $0.0085/second.

| Duration | Cost |
|----------|------|
| 60 seconds | $0.51 |
| 120 seconds | $1.02 |
| 300 seconds (5 min) | $2.55 |
| 600 seconds (10 min) | $5.10 |

Duration is rounded up using `Math.ceil()`.

## Device Class Presets

| Device Class | Dimensions |
|-------------|------------|
| desktop | 1600x900 |
| mobile | 375x812 (portrait) |

## Tester Data

The `testerData` object contains artifacts captured during the test session:

```typescript
{
  testDurationSeconds: number;
  consoleMessages: Array<{
    type: string;       // "log", "error", "warn", etc.
    message: string;
    timestamp: string;
  }>;
  networkRequests: Array<{
    url: string;
    method: string;     // "GET", "POST", etc.
    status?: number;    // HTTP status code
    timestamp: string;
  }>;
  clicks: Array<{
    x: number;
    y: number;
    timestamp: string;
    element?: string;   // Element selector if available
  }>;
  screenshots: string[];  // URLs to captured screenshots
  videoUrl?: string;      // URL to session recording
}
```

## Error Codes

| HTTP Status | Meaning |
|-------------|---------|
| 400 | Bad request. Invalid parameters. |
| 401 | Unauthorized. Invalid or missing API key. |
| 404 | Not found. Job does not exist. |
| 408 | Timeout. Synchronous request exceeded 10 minutes. |
| 429 | Too Many Requests. Rate limit exceeded. |
| 500 | Server error. |

---

# Cookbook

Step-by-step guides for common use cases.

## Issue Testing Automation

Automatically verify that issues are fixed when PRs are merged.

```yaml
# .github/workflows/test-issues.yml
name: Test Linked Issues

on:
  workflow_run:
    workflows: [CI]
    types: [completed]
    branches: [main]

jobs:
  test-issues:
    if: github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: volter-ai/runhuman-issue-tester-action@0.0.6
        with:
          api-key: ${{ secrets.RUNHUMAN_API_KEY }}
          test-url: ${{ vars.RUNHUMAN_TESTING_URL }}
```

## Preview Deployment Testing

### Vercel

```yaml
name: Test Vercel Preview
on:
  deployment_status

jobs:
  test:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: volter-ai/runhuman-qa-test-action@v0.0.1
        with:
          api-key: ${{ secrets.RUNHUMAN_API_KEY }}
          url: ${{ github.event.deployment_status.target_url }}
          description: Test the preview deployment
          output-schema: |
            {
              "pageLoads": { "type": "boolean", "description": "Page loads correctly?" },
              "noErrors": { "type": "boolean", "description": "No console errors?" }
            }
```

## Multi-Step Flow Testing

### Checkout Flow

```javascript
const result = await fetch('https://runhuman.com/api/run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://staging.myapp.com/products',
    description: `
      1. Browse products and add one to cart
      2. Go to cart and verify the item is there
      3. Proceed to checkout
      4. Fill shipping information
      5. Select payment method
      6. Verify order summary shows correct total
      7. Do not submit the final order
    `,
    targetDurationMinutes: 10,
    outputSchema: {
      addToCartWorks: { type: 'boolean', description: 'Product added to cart successfully?' },
      cartShowsItem: { type: 'boolean', description: 'Cart displays the added item?' },
      checkoutLoads: { type: 'boolean', description: 'Checkout page loads?' },
      shippingFormWorks: { type: 'boolean', description: 'Shipping form accepts input?' },
      totalCorrect: { type: 'boolean', description: 'Order total looks correct?' },
      issues: { type: 'array', description: 'Any issues encountered' }
    }
  })
});
```

### Authentication Flows

```javascript
const result = await fetch('https://runhuman.com/api/run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://staging.myapp.com/login',
    description: `
      Test authentication:
      1. Try login with valid credentials (test@example.com / demo123)
      2. Verify redirect to dashboard
      3. Log out
      4. Try login with wrong password
      5. Verify error message is shown
    `,
    targetDurationMinutes: 8,
    outputSchema: {
      loginWorks: { type: 'boolean', description: 'Valid login succeeds?' },
      logoutWorks: { type: 'boolean', description: 'Logout works?' },
      errorShown: { type: 'boolean', description: 'Error shown for wrong password?' },
      errorMessage: { type: 'string', description: 'What error message is displayed?' }
    }
  })
});
```

---

# Support & Resources

- **Main Docs**: https://runhuman.com/docs
- **Pricing**: https://runhuman.com/pricing
- **Dashboard**: https://runhuman.com/dashboard
- **API Keys**: https://runhuman.com/dashboard (navigate to your organization's API Keys page)
- **Contact**: hey@runhuman.com

---

# Quick Reference

**Endpoint**: `POST https://runhuman.com/api/run`
**Auth**: `Authorization: Bearer YOUR_API_KEY`
**Timeout**: 10 minutes
**Cost**: $0.0085/second (exact, not rounded)
**Response**: `{ status, result, costUsd, testDurationSeconds }`

**MCP Server**: `https://runhuman.com/mcp`
**CLI**: `npm install -g runhuman`
**GitHub Actions**: `volter-ai/runhuman-qa-test-action@v0.0.1`