Decide whether you need workers

A single Checkmate instance goes a long way. One deployment monitors 1,600+ servers in production, and a single process runs up to 500 HTTP checks in flight at once. If you run a few hundred monitors on one-minute intervals, the default setup is enough and you can stop reading here.

Workers start to pay off when:

You run more than a few thousand checks, or thousands of monitors on short intervals (10 to 30 seconds).
Checks are slow by nature (distant endpoints, PageSpeed runs, large response bodies) and tie up capacity for long stretches.
You want the dashboard and API to stay responsive no matter how busy the check load gets.
You want check capacity to survive the loss of a single machine.

Since version 3.10.0, the Checkmate image can run as either a full server or a dedicated worker. You run the same image everywhere and change one environment variable.

Note

Workers scale check capacity. They are not the same feature as geo checks, which run a check from multiple geographic locations to catch regional outages.

Understand how distribution works

Checkmate coordinates everything through MongoDB. There is no Redis or message broker to operate.

The primary node runs the API, the web client and the job scheduler. The scheduler seeds and reconciles the jobs collection in MongoDB.
Worker nodes poll that collection and claim due jobs in batches, sized to their free capacity. A claimed job carries a lock that the worker renews while the check runs.
If a worker dies mid-check, its locks expire and another worker reclaims the jobs. Every node registers in a worker registry with a heartbeat, so the fleet knows which workers are alive.

Because claiming is atomic, you can point any number of workers at the same database and they will share the load without stepping on each other. To add capacity, start another container.

Run a worker

A worker is the standard Checkmate image with QUEUE_MODE=worker. It connects to the same MongoDB as the primary, processes checks and serves health endpoints. It does not start the API or the web client.

The environment variables that matter:

DB_CONNECTION_STRING — the same MongoDB connection string as the primary. This is how workers find their jobs.

QUEUE_MODE — primary (default) runs the API, web client and scheduler. worker runs a job-processing worker only.

QUEUE_PRIMARY_PROCESSES — set to false on the primary once dedicated workers handle all checks, so the primary only schedules. Defaults to true. Ignored in worker mode.

JWT_SECRET and CLIENT_HOST — required by configuration validation on every node. Reuse the primary's values.

ENCRYPTION_KEY — only needed if you use Docker monitors with TLS. Must be identical on the primary and every worker, since any worker may pick up those checks.

HEALTH_PORT — port for the worker's health and metrics endpoints. Defaults to 52346.

Add a worker to Docker Compose

Starting from the standard Docker Compose setup, add a worker service:

services:
  checkmate:
    image: ghcr.io/bluewave-labs/checkmate:latest
    restart: always
    ports:
      - "52345:52345"
    environment:
      - DB_CONNECTION_STRING=mongodb://mongodb:27017/uptime_db
      - CLIENT_HOST=${CLIENT_HOST:-http://localhost:52345}
      - JWT_SECRET=${JWT_SECRET:?set JWT_SECRET in your environment}
      - QUEUE_PRIMARY_PROCESSES=false
    depends_on:
      mongodb:
        condition: service_healthy

  checkmate-worker:
    image: ghcr.io/bluewave-labs/checkmate:latest
    restart: always
    environment:
      - DB_CONNECTION_STRING=mongodb://mongodb:27017/uptime_db
      - CLIENT_HOST=${CLIENT_HOST:-http://localhost:52345}
      - JWT_SECRET=${JWT_SECRET:?set JWT_SECRET in your environment}
      - QUEUE_MODE=worker
    stop_grace_period: 60s
    healthcheck:
      test:
        [
          "CMD",
          "node",
          "-e",
          "require('http').get('http://localhost:52346/livez',r=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))",
        ]
      interval: 15s
      timeout: 3s
      start_period: 60s
      retries: 3
    depends_on:
      mongodb:
        condition: service_healthy

  mongodb:
    image: mongo:8.0
    restart: always
    command: ["mongod", "--quiet", "--bind_ip_all"]
    volumes:
      - mongo-data:/data/db
    healthcheck:
      test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/27017"]
      interval: 5s
      retries: 30

volumes:
  mongo-data:

Scale on the same host with docker compose up -d --scale checkmate-worker=3, or run the worker service on separate machines pointed at the same MongoDB. Workers on other machines need network access to the database and to the endpoints they will check, nothing else.

Note

The example sets QUEUE_PRIMARY_PROCESSES=false, which keeps the primary node out of check processing entirely. Leave it at the default true if you want the primary to keep pulling its share of checks alongside the workers.

Run workers on Kubernetes

The same split applies on Kubernetes: one deployment with the default environment for the primary and a second deployment with QUEUE_MODE=worker that you scale with replicas. Point the worker pods' liveness probe at /livez and the readiness probe at /readyz on port 52346. Give worker pods a termination grace period of 60 seconds so in-flight checks can drain.

Check worker health

Every process that runs a worker (a dedicated worker, or a primary with QUEUE_PRIMARY_PROCESSES=true) serves three endpoints on HEALTH_PORT:

/livez — returns 200 while the process is alive. Use it for container restarts.

/readyz — returns 200 once the worker is connected and accepting jobs. Use it for readiness probes during rollouts.

/metrics — Prometheus-format gauges covering in-flight jobs and drain state, ready to scrape.

On shutdown a worker stops claiming new jobs and drains in-flight checks for up to 25 seconds. Any check still running after that is not lost: its lock expires and another worker reclaims it.

Size the fleet

Each worker claims checks up to a fixed concurrency: 500 standard checks in flight at once, plus separate smaller pools for evaluation and cleanup jobs. How many monitors that supports depends on check duration. As a rough guide, if your average check takes 2 seconds, one worker sustains around 250 checks per second, which covers 15,000 monitors on one-minute intervals.

In practice the first bottleneck is usually MongoDB, not worker count. Watch database CPU and disk latency as you add monitors. Give MongoDB its own resources before adding a third or fourth worker.

Distributed workers