Cookbook

Step-by-step guides for common Runhuman use cases.

Issue Testing Automation

Automatically verify that issues are fixed when PRs are merged or commits that fix issues are pushed.

How It Works

A PR with “Fixes #123” in the description is merged or a commit that closes an issue is pushed to main
The action analyzes the linked issue to generate test instructions
A human tester verifies the fix on your deployment URL
Results are posted as a comment on the issue
If the test fails, the issue is reopened

Setup

Add these secrets and variables to your repository:

Name	Type	Description
RUNHUMAN_API_KEY	Secret	Your API key
RUNHUMAN_TESTING_URL	Variable	Your staging/preview URL

Create the workflow file:

# .github/workflows/test-issues.yml
name: Test Linked Issues

on:
  workflow_run:
    workflows: [CI]  # Replace with your deploy workflow name
    types: [completed]
    branches: [main]

concurrency:
  group: test-issues-${{ github.event.workflow_run.head_sha }}
  cancel-in-progress: true

jobs:
  test-issues:
    if: github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: volter-ai/runhuman-issue-tester-action@0.0.6
        with:
          api-key: ${{ secrets.RUNHUMAN_API_KEY }}
          test-url: ${{ vars.RUNHUMAN_TESTING_URL }}

Configuration

Option	Default	Description
api-key	Required	Your Runhuman API key
test-url	-	Base URL for testing
qa-label	qa-test	Only test issues with this label
auto-detect	true	Let AI decide which issues are testable
reopen-on-failure	true	Reopen issue if test fails
failure-label	qa-failed	Label added on failure

Writing Testable Issues

Include a test URL and clear reproduction steps:

## Bug Description
The checkout button is unresponsive on Safari.

## Test URL
https://staging.myapp.com/checkout

## Steps to Reproduce
1. Add items to cart
2. Go to checkout
3. Click "Place Order"
4. Nothing happens

## Expected Behavior
Order should be submitted and confirmation shown.

Bulk Issue Testing

Test all open issues in your repository on a schedule or on-demand.

How It Works

Workflow fetches all open issues with the qa-test label
Each issue is tested in parallel using matrix strategy
Results are posted as comments on each issue
Failed issues get reopened and labeled

Setup

# .github/workflows/test-all-issues.yml
name: Test All Open Issues

on:
  # Option 1: Run daily at 9 AM UTC
  schedule:
    - cron: '0 9 * * *'
  # Option 2: Manual trigger only
  # workflow_dispatch: {}
  # Option 3: Both scheduled and manual (recommended)
  workflow_dispatch:

jobs:
  find-issues:
    runs-on: ubuntu-latest
    outputs:
      issues: ${{ steps.get-issues.outputs.issues }}
      count: ${{ steps.get-issues.outputs.count }}
    steps:
      - name: Get open issues with qa-test label
        id: get-issues
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          issues=$(gh issue list \
            --repo ${{ github.repository }} \
            --label "qa-test" \
            --state open \
            --json number \
            --jq '[.[].number]')
          echo "issues=$issues" >> $GITHUB_OUTPUT
          echo "count=$(echo $issues | jq length)" >> $GITHUB_OUTPUT

  test-issue:
    needs: find-issues
    if: needs.find-issues.outputs.count != '0'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      max-parallel: 3  # Limit concurrent tests to control costs
      matrix:
        issue: ${{ fromJson(needs.find-issues.outputs.issues) }}
    steps:
      - uses: actions/checkout@v4
      - uses: volter-ai/runhuman-issue-tester-action@0.0.6
        with:
          api-key: ${{ secrets.RUNHUMAN_API_KEY }}
          issue-number: ${{ matrix.issue }}
          test-url: ${{ vars.RUNHUMAN_TESTING_URL }}

Cost Considerations

Each test costs ~$0.32-0.54 (3-5 minutes)
10 issues = ~$3-5 per run
Use max-parallel to control concurrent spending
Consider running less frequently for large issue counts

Preview Deployment Testing

Test Vercel, Netlify, or other preview deployments automatically.

Vercel

name: Test Vercel Preview
on:
  deployment_status

jobs:
  test:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: volter-ai/runhuman-qa-test-action@v0.0.1
        with:
          api-key: ${{ secrets.RUNHUMAN_API_KEY }}
          url: ${{ github.event.deployment_status.target_url }}
          description: Test the preview deployment
          output-schema: |
            {
              "pageLoads": { "type": "boolean", "description": "Page loads correctly?" },
              "noErrors": { "type": "boolean", "description": "No console errors?" }
            }

Netlify

name: Test Netlify Preview
on:
  pull_request

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Wait for Netlify
        uses: jakepartusch/wait-for-netlify-action@v1.4
        id: netlify
        with:
          site_name: your-site-name
          max_timeout: 300

      - uses: volter-ai/runhuman-qa-test-action@v0.0.1
        with:
          api-key: ${{ secrets.RUNHUMAN_API_KEY }}
          url: ${{ steps.netlify.outputs.url }}
          description: Test the Netlify preview

Custom Preview URLs

If your preview URLs follow a pattern:

url: https://pr-${{ github.event.pull_request.number }}.preview.myapp.com

Visual Regression Testing

Catch UI bugs before they reach production.

Basic Visual Check

const result = await fetch('https://runhuman.com/api/run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://staging.myapp.com/product/123',
    description: 'Check for visual issues: broken images, layout problems, text overflow, color contrast issues',
    outputSchema: {
      imagesLoad: { type: 'boolean', description: 'All images load correctly?' },
      layoutCorrect: { type: 'boolean', description: 'Layout looks correct, no overflow?' },
      textReadable: { type: 'boolean', description: 'All text is readable?' },
      visualIssues: { type: 'string', description: 'Describe any visual problems found' }
    }
  })
});

Mobile Responsiveness

const result = await fetch('https://runhuman.com/api/run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://staging.myapp.com',
    description: 'Test on mobile: check navigation menu, forms, buttons. Look for overflow, tiny text, unreachable elements.',
    outputSchema: {
      navigationWorks: { type: 'boolean', description: 'Mobile nav opens and closes correctly?' },
      formsUsable: { type: 'boolean', description: 'Forms are usable on mobile?' },
      mobileIssues: { type: 'array', description: 'List of mobile-specific issues' }
    }
  })
});

Multi-Step Flow Testing

Test complex user journeys that span multiple pages.

Checkout Flow

const result = await fetch('https://runhuman.com/api/run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://staging.myapp.com/products',
    description: `
      1. Browse products and add one to cart
      2. Go to cart and verify the item is there
      3. Proceed to checkout
      4. Fill shipping information
      5. Select payment method
      6. Verify order summary shows correct total
      7. Do not submit the final order
    `,
    targetDurationMinutes: 10,
    outputSchema: {
      addToCartWorks: { type: 'boolean', description: 'Product added to cart successfully?' },
      cartShowsItem: { type: 'boolean', description: 'Cart displays the added item?' },
      checkoutLoads: { type: 'boolean', description: 'Checkout page loads?' },
      shippingFormWorks: { type: 'boolean', description: 'Shipping form accepts input?' },
      totalCorrect: { type: 'boolean', description: 'Order total looks correct?' },
      issues: { type: 'array', description: 'Any issues encountered' }
    }
  })
});

User Onboarding

const result = await fetch('https://runhuman.com/api/run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://staging.myapp.com/signup',
    description: `
      1. Create account with email test-${Date.now()}@example.com
      2. Complete the onboarding wizard
      3. Set up profile with sample data
      4. Verify you reach the dashboard
    `,
    targetDurationMinutes: 8,
    outputSchema: {
      signupWorks: { type: 'boolean', description: 'Account created successfully?' },
      onboardingCompletes: { type: 'boolean', description: 'Onboarding wizard completes?' },
      profileSaves: { type: 'boolean', description: 'Profile changes save?' },
      dashboardReached: { type: 'boolean', description: 'User reaches dashboard?' },
      confusingSteps: { type: 'array', description: 'Any confusing or unclear steps' }
    }
  })
});

Authentication Flows

const result = await fetch('https://runhuman.com/api/run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: 'https://staging.myapp.com/login',
    description: `
      Test authentication:
      1. Try login with valid credentials (test@example.com / demo123)
      2. Verify redirect to dashboard
      3. Log out
      4. Try login with wrong password
      5. Verify error message is shown
      6. Try forgot password link
    `,
    targetDurationMinutes: 8,
    outputSchema: {
      loginWorks: { type: 'boolean', description: 'Valid login succeeds?' },
      logoutWorks: { type: 'boolean', description: 'Logout works?' },
      errorShown: { type: 'boolean', description: 'Error shown for wrong password?' },
      errorMessage: { type: 'string', description: 'What error message is displayed?' },
      forgotPasswordWorks: { type: 'boolean', description: 'Forgot password link works?' }
    }
  })
});

Next Steps

Topic	Link
Full technical specification	Reference
REST API integration	REST API
CI/CD integration	GitHub Actions