SoliDB 1.0: closed by default, set operations, and no more job queue

What 1.0 is

The changelog describes this release as security hardening, SDBQL function completeness, and one product breaking change. Application background jobs now belong to the Soli framework, not to the database.

Several defaults change to fail closed, and each of them can break a deployment that relied on the old behaviour. The one that fails quietly is the queue removal: a configured cron schedule stops firing, with no error. The upgrade section lists what to act on.

Set operations, recursive CTEs, RETURN DISTINCT

Comparing two result sets used to mean two queries and a merge in application code. Query blocks combine with UNION [ALL], INTERSECT and EXCEPT. Rows are compared by value — the same equality the UNION() and INTERSECTION() functions use — and duplicates are removed except with UNION ALL. Which newsletter subscribers are also customers:

FOR n IN newsletter RETURN n.email
INTERSECT
FOR c IN customers RETURN c.email

["bo@example.com", "cy@example.com"]

And which customers have never subscribed:

FOR c IN customers RETURN c.email
EXCEPT
FOR n IN newsletter RETURN n.email

["di@example.com"]

Either side may be parenthesized, and chains follow SQL precedence: INTERSECT binds tighter than UNION and EXCEPT, which chain left to right. Each operand is a full query block, and permission checks see through the operands.

Hierarchies were the other gap. WITH RECURSIVE name AS (<anchor> UNION ALL <step>) runs the anchor once, then runs the step until it stops producing rows. Inside the step, the CTE name holds the rows of the previous iteration, so each pass goes one level deeper. Everyone under Bob in an org chart, at any depth:

WITH RECURSIVE reports AS (
  -- Anchor: the root of the hierarchy
  FOR e IN employees
    FILTER e._key == "bob"
    RETURN e._key
  UNION ALL
  -- Step: direct reports of the previous level
  FOR m IN employees
    FILTER m.manager IN reports
    RETURN m._key
)
FOR x IN reports
  RETURN x

["bob", "dan", "erin"]

Dan reports to Bob and Erin to Dan; Alice, Carol and Frank are outside that branch. Recursion is capped at 1,000 iterations and 1M rows, so cyclic data (a → b → a) ends in an iteration-limit error rather than a hung query — filter out visited rows in the step, or carry a depth and stop at it. The CTE page has the full syntax.

RETURN DISTINCT deduplicates result rows, first occurrence wins. Five orders from three cities:

FOR o IN orders
  RETURN DISTINCT o.city

["Lyon", "Nantes", "Paris"]

The same release adds COLLECT … INTO g KEEP v1, v2, which limits which variables are stored in the group arrays (an unknown name is an error); the NONE quantifier, as NONE(x IN arr SATISFIES cond) or NONE(arr, x -> cond); and OFFSET, alone or as LIMIT n OFFSET m. Functions the reference already listed now exist, and string LENGTH counts Unicode scalars. See SDBQL syntax.

Indexes created by the queries that need them

A FILTER on an unindexed field is a full scan. 1.0 can create the index itself, on collections that opt in with autoIndex: true — or leave the property unset and set SOLIDB_AUTO_INDEX=1; an explicit autoIndex: false wins over the variable. The first time a FOR + FILTER equality or range comparison would have used an index and none exists, SoliDB creates a persistent _auto_{field} index, backfills it, and uses it:

# collection created with {"name": "events", "autoIndex": true}, 500 documents
# then: FOR e IN events FILTER e.kind == "signup" COLLECT WITH COUNT INTO n RETURN n
curl -s -u admin:$PASS localhost:6745/_api/database/blog/index/events \
  | jq -c '.indexes[] | {name, fields, index_type}'

{"name": "_auto_kind", "fields": ["kind"], "index_type": "Persistent"}

Creating an index is a write, so only a caller holding Write or Admin triggers one, on every query route; a query with no principal — a view refresh, a stream task — never does. There is a cap of 16 per collection. SORT, null comparisons, _key/_id/_rev, sharded collections, filters a composite index already covers, fields no document carries and collections above SOLIDB_AUTO_INDEX_MAX_DOCS (default 1,000,000) are left alone. EXPLAIN reports auto_index_candidate without creating anything. Details on the indexes page.

A fresh node starts closed

A 0.34 node started with no flags listened on every interface, accepted replication connections without a keyfile, and served /metrics to anyone. Each was reasonable on a laptop and a finding on a server. In 1.0 all three default the other way:

Listeners bind 127.0.0.1 unless --host or SOLIDB_HOST says otherwise. Use 0.0.0.0 only when something in front of the process terminates TLS — or when the server does, below.

For replication, the HTTP cluster bus already required a secret in 0.34; the multiplexed sync socket still skipped HMAC when no keyfile existed. Unauthenticated replication is now refused unless SOLIDB_ALLOW_UNAUTHENTICATED_SYNC=true, meant for local tests.

/metrics requires SOLIDB_METRICS_TOKEN (as X-Metrics-Token or Bearer) or an admin JWT, unless SOLIDB_METRICS_PUBLIC=1.

The permission model tightened the same way. Creating or changing _scripts and _services needed only Write, so a collection editor could publish an unauthenticated /api/{db}/{service}/… handler; installing Lua is now Admin, and new services default to require_auth: true. Cluster control-plane endpoints now require Admin. Webhook URLs are checked against loopback, RFC1918, link-local and metadata hosts. API keys must declare at least one role. Passwords must be at least 12 characters. A JWT in ?token= is accepted only on the three WebSocket endpoints. The security page covers each.

HTTPS and rate limiting in the server

Serving TLS used to mean a reverse proxy. --tls-cert and --tls-key now terminate it in the server, through rustls. Both are required together; one without the other refuses to start rather than listening in plaintext.

# HTTPS on all interfaces, TLS terminated by SoliDB itself
solidb --host 0.0.0.0 --tls-cert /etc/solidb/cert.pem --tls-key /etc/solidb/key.pem

On the multiplexed port the listener sniffs for a TLS ClientHello and handshakes only when one is offered. HTTPS clients get the tunnel, with HTTP and the native driver protocol both working inside it, while plaintext peers keep connecting — the shipped SDKs' driver protocol and the sync and cluster transports do not speak TLS yet. SOLIDB_TLS_REQUIRE=1 refuses plaintext on that port anyway, which is safe only on a single node with no native-protocol clients.

Before 1.0 only /auth/login was throttled. The whole router now has a sliding-window limiter answering 429 with Retry-After before any handler runs. It is opt-in: SOLIDB_API_RATE_LIMIT defaults to 0, because a database usually sits behind an application tier it trusts, and throttling that tier turns a capacity problem into an availability one. When on, the budget is keyed on the request's credential where there is one and on the address otherwise; SOLIDB_API_RATE_LIMIT_PER_IP (default 10× the budget) still caps the address. The window is SOLIDB_API_RATE_WINDOW_SECS, 60 by default.

The job and cron queue moves to Soli

Application jobs now live in the Soli framework, which claims and runs them in its own process. The queue SoliDB exposed for that work is removed: ten HTTP endpoints, eight driver commands, the Lua db:enqueue global, the jobs and cron sub-clients in six SDKs, and the admin Queues and Cron pages.

Triggers keep working. A trigger still fires by inserting a row into _jobs, and the worker still claims it and runs its Lua script or posts its signed webhook. Embedding generation and materialized-view refresh are untouched.

Smaller changes worth knowing

LENGTH("users") used to return the document count when the string happened to name a collection. It is now the character length, always. Use COLLECTION_COUNT for the count:

RETURN [LENGTH("orders"), COLLECTION_COUNT("orders")]

[[6, 5]]

FOR i IN 1..10 LIMIT 3 OFFSET 4 RETURN i

[5, 6, 7]

--no-lua (or SOLIDB_NO_LUA=1) skips the Lua VM pool on nodes that never run Lua, dropping the idle memory of the pre-warmed VMs — at least four states. Custom scripts, service routes and the REPL return 501, and a trigger whose action is a Lua script fails its job immediately, without retries.

Freed memory now goes back to the operating system. jemalloc was linked but never configured, so every Tokio worker that handled a burst grew an arena and went idle holding it. Measured: 3200 queries at 32× concurrency took RSS from 662 MB to 1296 MB, and it was still 954 MB five minutes later; with a background purger the same burst comes back to 625 MB.

Driver queries now run under the same 30-second cap as HTTP queries, and carry the session principal, so row policies filter driver reads too.

Upgrading from 0.34

These are the changes that need action. Everything else in 1.0 is additive.

The complete list is in the changelog.