Queues & Jobs - SoliDB Documentation

Overview

SoliDB provides a built-in, distributed queue system backed by the database itself. Jobs are stored as documents in the _jobs collection, ensuring persistence and reliability.

Key features include:

Distributed Workers: Every node in a SoliDB cluster runs background workers that automatically pick up and execute jobs.
Optimistic Locking: Uses SoliDB's revision check (_rev) to ensure that a job is executed by exactly one worker, even in a large cluster.
Priorities & Retries: Jobs support priority levels and automatic retries with exponential backoff.
Script-based Execution: Jobs execute server-side Lua scripts, allowing complex logic close to your data.

API Reference

POST /_api/database/:db/queues/:queue/enqueue

Enqueue a new job.

Request Body

Field	Type	Description
script	string	Path/Name of the Lua script to execute.
params	object	JSON arguments passed to the script.
priority	integer	Higher values execute first.
max_retries	integer	Max attempts on failure on failure.

{"script": "send_email", "params": {"to": "[email protected]", "body": "Hello"}, "priority": 10}

GET /_api/database/:db/queues

Get statistics for all queues.

Lua Driver

You can enqueue jobs directly from your application using the SoliDB Lua driver.

-- Enqueue a job in the 'default' queue
db:queue("default"):enqueue("process_image", {
  image_id = "img_123",
  format = "webp"
})

-- With options
db:queue("emails"):enqueue("send_welcome", { user_id = "u1" }, {
  priority = 100,
  max_retries = 5
})

Dashboard

Monitor your queues visually using the dedicated Queues tab in the database dashboard.

View pending, running, and failed job counts.
Inspect job details and parameters.
Cancel pending jobs.
Manually enqueue jobs for testing.

Cron Jobs (Schedules)

SoliDB supports scheduled jobs via cron expressions. Schedules are stored in the _cron_jobs collection and are automatically managed by the cluster.

Cluster-Safe Execution: SoliDB uses optimistic locking to ensure that even in a multi-node cluster, only one node will spawn a job for a given schedule tick.

Cron Expression Format

SoliDB uses a 7-field cron expression format (including seconds and year):

sec min hour day month day_of_week year

Examples:

0 */5 * * * * * - Every 5 minutes
0 0 * * * * * - Every hour
0 0 0 * * * * - Every day at midnight

Job Identification

When a cron job spawns a regular job, the new job will contain a cron_job_id field. In the Dashboard, these jobs are identified by a clock icon.

API Management

Endpoints for managing schedules:

POST /_api/database/:db/cron
GET /_api/database/:db/cron
DELETE /_api/database/:db/cron/:id

Configuration

By default, each SoliDB node spawns 4 concurrent workers. You can adjust this globally via environment variables (either directly or via a .env file).

QUEUE_WORKERS=8 ./solidb-server ...

Queue Management