QAbyAI logo
Book a Demo
Extras

CI/CD Integration

Integrate qaby.ai into your continuous integration and deployment pipelines

Overview

Execute your qaby.ai test suites as part of your CI/CD pipeline to ensure quality before deployment. This guide covers two integration methods: polling-based and webhook-based approaches.

API Endpoints

Execute Suite

Triggers a test suite run with optional webhook notification.

POST https://qaby.ai/trpc/browser.execute.executeSuite

Request:

{
  "json": {
    "testPlanId": 123,
    "webhookUrl": "https://your-ci.com/webhook/abc123", // optional
    "webhookSecret": "shared-secret" // optional
  }
}

Response:

{
  "suiteRunId": 456,
  "status": "pending"
}

Get Suite Run Status

Check the status of a running or completed test suite.

GET https://qaby.ai/trpc/browser.execute.getSuiteRunStatus?input={"json":{"suiteRunId":456}}

Response:

{
  "suiteRunId": 456,
  "status": "running",
  "summary": {
    "totalTests": 10,
    "passed": 5,
    "failed": 0,
    "skipped": 0,
    "running": 5
  },
  "startedAt": "2024-12-23T10:00:00Z",
  "completedAt": null
}

Authentication

You'll need an API key to authenticate your requests. Create one at https://qaby.ai/settings/api-keys and include it in the x-api-key header.

Getting Your Test Plan ID

  1. Visit https://qaby.ai/plans
  2. Navigate to the test plan you want to execute
  3. Copy the test plan ID from the URL (e.g., https://qaby.ai/plans/YOUR_TEST_PLAN_ID)

Integration Methods

Method 1: Polling

Execute the test suite and poll for status until completion.

# Execute suite
RESPONSE=$(curl -X POST https://qaby.ai/trpc/browser.execute.executeSuite \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  --data-raw '{"json":{"testPlanId":123}}')

SUITE_ID=$(echo $RESPONSE | jq -r .suiteRunId)

# Poll for status
while true; do
  STATUS=$(curl "https://qaby.ai/trpc/browser.execute.getSuiteRunStatus?input=%7B%22json%22%3A%7B%22suiteRunId%22%3A${SUITE_ID}%7D%7D" \
    -H "x-api-key: YOUR_API_KEY" | jq -r .status)

  if [ "$STATUS" = "passed" ]; then
    echo "Tests passed!"
    exit 0
  elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "cancelled" ]; then
    echo "Tests failed: $STATUS"
    exit 1
  fi

  sleep 5
done

Method 2: Webhook

Execute the test suite with a webhook URL to receive results when complete.

curl -X POST https://qaby.ai/trpc/browser.execute.executeSuite \
  -H "x-api-key: YOUR_API_KEY" \
  -H "content-type: application/json" \
  --data-raw '{
    "json": {
      "testPlanId": 123,
      "webhookUrl": "https://your-ci.com/webhook/abc123",
      "webhookSecret": "your-secret"
    }
  }'

Webhook Payload:

{
  "suiteRunId": 456,
  "testPlanId": 123,
  "status": "passed",
  "summary": {
    "totalTests": 10,
    "passed": 8,
    "failed": 2,
    "skipped": 0,
    "duration": 45000
  },
  "completedAt": "2024-12-23T10:05:00Z"
}

CI/CD Examples

GitHub Actions

Polling Method

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Execute Test Suite
        id: suite
        run: |
          RESPONSE=$(curl -X POST https://qaby.ai/trpc/browser.execute.executeSuite \
            -H "x-api-key: ${{ secrets.QABY_API_KEY }}" \
            -H "content-type: application/json" \
            --data-raw '{"json":{"testPlanId":${{ vars.TEST_PLAN_ID }}}}')

          SUITE_ID=$(echo $RESPONSE | jq -r .suiteRunId)
          echo "suite_run_id=$SUITE_ID" >> $GITHUB_OUTPUT

      - name: Wait for Test Completion
        run: |
          MAX_ATTEMPTS=60  # 5 minutes
          ATTEMPT=0

          while [ $ATTEMPT -lt $MAX_ATTEMPTS ]; do
            STATUS_RESPONSE=$(curl -s "https://qaby.ai/trpc/browser.execute.getSuiteRunStatus?input=%7B%22json%22%3A%7B%22suiteRunId%22%3A${{ steps.suite.outputs.suite_run_id }}%7D%7D" \
              -H "x-api-key: ${{ secrets.QABY_API_KEY }}")
            
            STATUS=$(echo $STATUS_RESPONSE | jq -r .status)
            
            if [ "$STATUS" = "passed" ]; then
              echo "✅ Tests passed!"
              echo $STATUS_RESPONSE | jq .summary
              exit 0
            elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "cancelled" ]; then
              echo "❌ Tests failed with status: $STATUS"
              echo $STATUS_RESPONSE | jq .summary
              exit 1
            fi
            
            echo "⏳ Status: $STATUS - waiting..."
            sleep 5
            ATTEMPT=$((ATTEMPT + 1))
          done

          echo "⏱️ Tests timed out"
          exit 1

Webhook Method

name: E2E Tests with Webhook

on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Setup Webhook Endpoint
        uses: distributhor/workflow-webhook@v3
        id: webhook
        with:
          webhook_url: ${{ secrets.WEBHOOK_URL }}
          webhook_secret: ${{ secrets.WEBHOOK_SECRET }}

      - name: Execute Test Suite
        run: |
          curl -X POST https://qaby.ai/trpc/browser.execute.executeSuite \
            -H "x-api-key: ${{ secrets.QABY_API_KEY }}" \
            -H "content-type: application/json" \
            --data-raw '{
              "json": {
                "testPlanId": ${{ vars.TEST_PLAN_ID }},
                "webhookUrl": "${{ steps.webhook.outputs.url }}",
                "webhookSecret": "${{ secrets.WEBHOOK_SECRET }}"
              }
            }'

      - name: Wait for Webhook Response
        uses: distributhor/workflow-webhook@v3
        with:
          webhook_url: ${{ steps.webhook.outputs.url }}
          timeout: 600

      - name: Check Test Results
        run: |
          STATUS=$(echo '${{ steps.webhook.outputs.data }}' | jq -r .status)
          if [ "$STATUS" != "passed" ]; then
            exit 1
          fi

Jenkins

pipeline {
    agent any

    environment {
        QABY_API_KEY = credentials('qaby-api-key')
        TEST_PLAN_ID = '123'
    }

    stages {
        stage('Execute E2E Tests') {
            steps {
                script {
                    def response = sh(
                        script: """
                            curl -s -X POST https://qaby.ai/trpc/browser.execute.executeSuite \
                              -H "x-api-key: ${QABY_API_KEY}" \
                              -H "content-type: application/json" \
                              --data-raw '{"json":{"testPlanId":${TEST_PLAN_ID}}}'
                        """,
                        returnStdout: true
                    )

                    def suiteRunId = readJSON(text: response).suiteRunId

                    def maxAttempts = 60
                    def attempt = 0
                    def testStatus = 'pending'

                    while (attempt < maxAttempts && (testStatus == 'pending' || testStatus == 'running')) {
                        sleep(time: 5, unit: 'SECONDS')

                        def statusResponse = sh(
                            script: """
                                curl -s "https://qaby.ai/trpc/browser.execute.getSuiteRunStatus?input=%7B%22json%22%3A%7B%22suiteRunId%22%3A${suiteRunId}%7D%7D" \
                                  -H "x-api-key: ${QABY_API_KEY}"
                            """,
                            returnStdout: true
                        )

                        def statusData = readJSON(text: statusResponse)
                        testStatus = statusData.status

                        echo "Test status: ${testStatus}"
                        attempt++
                    }

                    if (testStatus != 'passed') {
                        error("Tests failed: ${testStatus}")
                    }
                }
            }
        }
    }
}

GitLab CI

stages:
  - test
  - deploy

variables:
  TEST_PLAN_ID: 123

e2e-tests:
  stage: test
  script:
    - |
      RESPONSE=$(curl -s -X POST https://qaby.ai/trpc/browser.execute.executeSuite \
        -H "x-api-key: ${QABY_API_KEY}" \
        -H "content-type: application/json" \
        --data-raw "{\"json\":{\"testPlanId\":${TEST_PLAN_ID}}}")

      SUITE_RUN_ID=$(echo $RESPONSE | jq -r .suiteRunId)

      for i in {1..60}; do
        STATUS_RESPONSE=$(curl -s "https://qaby.ai/trpc/browser.execute.getSuiteRunStatus?input=%7B%22json%22%3A%7B%22suiteRunId%22%3A${SUITE_RUN_ID}%7D%7D" \
          -H "x-api-key: ${QABY_API_KEY}")
        
        STATUS=$(echo $STATUS_RESPONSE | jq -r .status)
        
        if [ "$STATUS" = "passed" ]; then
          echo "Tests passed!"
          exit 0
        elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "cancelled" ]; then
          echo "Tests failed: $STATUS"
          exit 1
        fi
        
        sleep 5
      done

      echo "Tests timed out"
      exit 1

Webhook Security

If you provide a webhookSecret, the webhook request will include an HMAC-SHA256 signature in the X-Signature header.

Verify webhook signature (Node.js):

const crypto = require('crypto');

function verifyWebhook(payload, signature, secret) {
  const expectedSignature =
    'sha256=' +
    crypto
      .createHmac('sha256', secret)
      .update(JSON.stringify(payload))
      .digest('hex');

  return signature === expectedSignature;
}

Test Status Values

  • pending: Test suite is queued for execution
  • running: Test suite is currently executing
  • passed: All tests passed successfully
  • failed: One or more tests failed
  • cancelled: Test suite was manually cancelled
  • timeout: Test suite exceeded the maximum execution time

Best Practices

  1. Set appropriate timeouts: Test suites can take time. Set timeouts of 5-30 minutes.
  2. Use webhooks for long-running tests: More efficient than polling.
  3. Store API keys securely: Use CI/CD secret management.
  4. Handle failures gracefully: Implement retry logic for transient failures.
  5. Monitor test trends: Track test suite performance over time.

Rate Limits

  • Polling intervals should be at least 5 seconds
  • Webhook notifications retry automatically with exponential backoff
  • API rate limits apply per team account