Pipeline Examples

Starter Pack

Download the starter pack to get the data directory structure ready to use:

Download umai-cicd-starter.zip

The zip contains:

starter-pack/
└── umai_data/
    └── pipeline_inputs/
        ├── api_key.txt        # Replace with your API key
        └── test_suite.json    # Export this from the UMAI web app

After downloading:

1. Copy the umai_data/ folder into your repository
2. Replace test_suite.json with the config exported from the UMAI web app (see Configuration)
3. Add umai_data/pipeline_inputs/api_key.txt to your .gitignore — the API key will be injected at runtime by your CI system

---

Pipeline Configuration

The examples below show how to integrate UMAI into your pipeline. Both assume you have already set up the starter pack in your repository and stored your API key as a CI secret.

GitHub Actions

Store your API key as a repository secret named UMAI_API_KEY under Settings > Secrets and variables > Actions, then add the following workflow to your repository:

.github/workflows/umai.yml

name: UMAI Tests

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read
  checks: write

jobs:
  umai-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Prepare data directories
        run: |
          mkdir -p umai_data/outputs umai_data/pipeline_inputs
          chmod 777 umai_data/outputs umai_data/pipeline_inputs

      - name: Write API key
        run: |
          echo "${{ secrets.UMAI_API_KEY }}" > umai_data/pipeline_inputs/api_key.txt
          mkdir -p umai_data/outputs

      - name: Pull UMAI image
        run: docker pull --platform linux/amd64 usemangoai/test-runner:latest

      - name: Run UMAI tests
        run: |
          docker run --rm \
            --platform linux/amd64 \
            -v ${{ github.workspace }}/umai_data:/umai_data \
            -e GENERATE_JUNIT_REPORT=true \
            usemangoai/test-runner:latest

      - name: Publish test results
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: UMAI Test Results
          path: umai_data/outputs/junit.xml
          reporter: java-junit

Note

The if: always() condition ensures results are published even when tests fail (exit code 1). The checks: write permission is required for the test reporter to post results to the GitHub Checks panel.

Jenkins

Add your API key as a Secret text credential in Jenkins with the ID umai-api-key (Manage Jenkins > Credentials), then add the following Jenkinsfile to your repository:

Jenkinsfile

pipeline {
    agent any

    environment {
        UMAI_API_KEY = credentials('umai-api-key')
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Run UMAI Tests') {
            steps {
                sh '''
                    docker pull --platform linux/amd64 usemangoai/test-runner:latest

                    echo "$UMAI_API_KEY" > umai_data/pipeline_inputs/api_key.txt
                    mkdir -p umai_data/outputs

                    docker run --rm \
                      --platform linux/amd64 \
                      -v ${WORKSPACE}/umai_data:/umai_data \
                      -e GENERATE_JUNIT_REPORT=true \
                      usemangoai/test-runner:latest
                '''
            }
        }
    }

    post {
        always {
            junit 'umai_data/outputs/junit.xml'
        }
    }
}

Note

The post { always { ... } } block publishes JUnit results regardless of whether the tests pass or fail, keeping your build history accurate.