Offline Sync
Build resilient applications that work anywhere. SoliDB's offline sync enables local-first data with automatic conflict resolution.
Introduction
Offline Sync is SoliDB's solution for building local-first applications. It allows clients to maintain a local replica of their data, work offline, and seamlessly synchronize changes when connectivity is restored.
Offline-First
Applications work without network connectivity. All operations are performed locally and synced automatically when online.
Instant Response
Zero-latency reads and writes from local storage. No waiting for network round-trips.
Auto Conflict Resolution
Sophisticated conflict resolution using version vectors ensures data consistency across all devices.
Why Offline Sync Matters
Modern applications need to work everywhere - from subway tunnels to airplanes. Offline sync ensures your users have a consistent experience regardless of connectivity. It's essential for mobile apps, field tools, and any application where reliability is critical.
Architecture
SoliDB's offline sync uses a sophisticated architecture combining version vectors, delta encoding, and CRDT principles to ensure data consistency across distributed clients.
Sync Flow
Version Vectors
Version vectors track the history of each document across all nodes. Unlike simple timestamps, they can detect concurrent modifications and establish causal relationships between changes.
{
"_key": "user-123",
"name": "Alice",
"_version": {
"client-A": 5, // Client A has made 5 edits
"client-B": 2, // Client B has made 2 edits
"server": 8 // Server has processed 8 total changes
},
"_timestamp": "2024-01-15T10:30:00Z"
}
Push Sync
Client-initiated sync sending local changes to the server:
- 1. Collect pending changes from local store
- 2. Send delta to server with version vector
- 3. Server validates and merges changes
- 4. Server returns updated version vector
- 5. Client marks changes as synced
Pull Sync
Client retrieves changes from the server:
- 1. Send current version vector to server
- 2. Server computes changes since last sync
- 3. Server returns delta with new version vector
- 4. Client applies changes to local store
- 5. Client resolves any local conflicts
Key Features
Delta Sync
Only changed fields are transmitted. Bandwidth-efficient sync for mobile and low-bandwidth environments.
CRDT Types
Built-in CRDT types (counters, registers, sets, maps) that merge automatically without conflicts.
Automatic Retry
Failed syncs are automatically retried with exponential backoff. No manual intervention required.
Local Storage
Uses SQLite or RocksDB locally. Full query capabilities on local data, even when offline.
Selective Sync
Sync only the data you need. Filter by user, date range, or any query criteria to save storage.
Change Notifications
Subscribe to local and remote changes. Real-time UI updates as data syncs across devices.
Quick Start
Get started with offline sync in minutes using the Rust client SDK. This example shows basic setup, local operations, and sync.
use solidb_client::{SyncManager, SyncConfig, HttpClient, LocalStore};
use serde_json::json;
use std::collections::HashMap;
#[tokio::main]
async fn main() -> Result<(), Box> {
// 1. Set up local storage and HTTP client
let local_store = LocalStore::open_default("myapp", "device-123".to_string())?;
let client = HttpClient::new("localhost:6745", 5000);
// 2. Configure and create sync manager
let config = SyncConfig {
sync_interval_secs: 30,
batch_size: 100,
max_retries: 5,
auto_sync: true,
collections: vec!["users".to_string(), "orders".to_string()],
};
let mut sync_manager = SyncManager::new(local_store, client, config);
let cmd_tx = sync_manager.start().await; // Start background sync
// 3. Work offline - changes stored locally
sync_manager.save_document("users", "user-1", &json!({
"name": "Alice",
"email": "[email protected]",
"status": "active",
})).await?;
println!("Document saved locally");
// 4. Query local data with full SDBQL (works offline!)
let mut bind_vars = HashMap::new();
bind_vars.insert("name".to_string(), json!("Alice"));
let users = sync_manager.query(
"FOR u IN users FILTER u.name == @name RETURN u",
Some(bind_vars)
).await?;
println!("Found {} local users", users.len());
// 5. Trigger sync when online
let sync_result = sync_manager.sync_now().await?;
println!("Synced: {} pushed, {} pulled", sync_result.pushed, sync_result.pulled);
// 6. Stop sync when done
cmd_tx.send(SyncCommand::Stop).await?;
Ok(())
}
That's It!
With just a few lines of code, your application can now work offline, automatically sync when online, and handle conflicts intelligently. The sync manager handles all the complexity for you.
Rust Client SDK
The OfflineSyncManager provides a comprehensive API for offline-first applications. All operations are async and work seamlessly with Tokio.
Configuration
use solidb_client::{SyncManager, SyncConfig, HttpClient, LocalStore};
// Local storage configuration
let local_store = LocalStore::open(
"/path/to/sync_data.db", // SQLite database path
"device-unique-id".to_string()
)?;
// Or use default path (~/.solidb/app_name/)
let local_store = LocalStore::open_default(
"my_app", // Application name
"device-uuid".to_string() // Unique device identifier
)?;
// HTTP client for server communication
let client = HttpClient::new(
"localhost:6745", // Server address
5000 // Timeout in milliseconds
);
// Sync configuration
let config = SyncConfig {
// Sync interval when online (seconds)
sync_interval_secs: 30,
// Maximum changes to push/pull per batch
batch_size: 100,
// Retry attempts for failed syncs
max_retries: 5,
// Enable automatic background sync
auto_sync: true,
// Collections to sync (empty = all collections)
collections: vec![
"users".to_string(),
"orders".to_string(),
],
};
// Create and start sync manager
let mut sync_manager = SyncManager::new(local_store, client, config);
let cmd_tx = sync_manager.start().await;
Local Operations
All CRUD operations work on local data. Changes are automatically queued for sync.
use serde_json::json;
// Save document locally (insert or update)
sync_manager.save_document(
"users", // collection
"user-456", // document key
&json!({
"name": "Bob",
"email": "[email protected]",
"created_at": "2024-01-15T10:00:00Z",
})
).await?;
// Get document from local store
let user = sync_manager.get_document("users", "user-456").await?;
// Returns: Option<Value> - None if not found
// Update existing document
sync_manager.save_document(
"users",
"user-456",
&json!({"email": "[email protected]", "status": "active"})
).await?;
// Delete document locally (queued for sync)
sync_manager.delete_document("users", "user-456").await?;
// List all documents in a collection
let docs = sync_manager.query_documents("users").await?;
// Returns: Vec<(String, Value)> - (key, document) pairs
| Method | Returns | Description |
|---|---|---|
save_document(col, key, doc) | Result<()> | Insert or update document locally |
get_document(col, key) | Result<Option<Value>> | Get document from local store |
delete_document(col, key) | Result<()> | Delete document locally |
query_documents(col) | Result<Vec<(String, Value)>> | List all documents in collection |
query(sdbql, bind_vars) | Result<Vec<Value>> | Execute SDBQL query on local data |
Querying Local Data
Run full SDBQL queries against local data using the query() method.
The query engine works identically whether online or offline, powered by the sdbql-core crate.
use std::collections::HashMap;
use serde_json::json;
// Simple query without bind variables
let active_users = sync_manager.query(
"FOR u IN users FILTER u.status == 'active' RETURN u",
None
).await?;
// Query with bind variables (recommended for security)
let mut bind_vars = HashMap::new();
bind_vars.insert("min_age".to_string(), json!(18));
bind_vars.insert("status".to_string(), json!("premium"));
let premium_users = sync_manager.query(
r#"FOR u IN users
FILTER u.age >= @min_age AND u.status == @status
SORT u.created_at DESC
LIMIT 50
RETURN { name: u.name, email: u.email }"#,
Some(bind_vars)
).await?;
// Aggregation queries with COLLECT
let stats = sync_manager.query(
r#"FOR u IN users
COLLECT status = u.status WITH COUNT INTO count
RETURN { status, count }"#,
None
).await?;
// Joins across local collections
let orders_with_users = sync_manager.query(
r#"FOR o IN orders
FOR u IN users FILTER u._key == o.user_id
RETURN { order: o, customer: u.name }"#,
None
).await?;
// Higher-order functions (FILTER, MAP, etc.)
let filtered = sync_manager.query(
r#"LET adults = FILTER(users, u -> u.age >= 18)
FOR u IN adults
RETURN u.name"#,
None
).await?;
Performance tip: Local queries execute instantly with no network latency. All data is stored in SQLite and queries use the same SDBQL syntax as server queries.
Manual Sync Control
While auto-sync runs in the background, you can manually trigger syncs and send commands.
use solidb_client::SyncCommand;
// Start the sync manager (returns command channel)
let cmd_tx = sync_manager.start().await;
// Trigger immediate sync
let result = sync_manager.sync_now().await?;
println!("Pushed: {}, Pulled: {}", result.pushed, result.pulled);
println!("Conflicts: {}, Errors: {}", result.conflicts, result.errors.len());
// Or send sync command through channel
cmd_tx.send(SyncCommand::SyncNow).await?;
// Subscribe to a specific collection
cmd_tx.send(SyncCommand::Subscribe {
collection: "users".to_string(),
filter: Some("status == 'active'".to_string()),
}).await?;
// Unsubscribe from a collection
cmd_tx.send(SyncCommand::Unsubscribe {
collection: "old_collection".to_string(),
}).await?;
// Set online/offline status manually
cmd_tx.send(SyncCommand::SetOnline(false)).await?; // Go offline
cmd_tx.send(SyncCommand::SetOnline(true)).await?; // Go online
// Check if currently syncing
let is_syncing = sync_manager.is_syncing().await;
// Check if online
let is_online = sync_manager.is_online().await;
// Stop the sync manager
cmd_tx.send(SyncCommand::Stop).await?;
| Command | Description |
|---|---|
SyncCommand::SyncNow | Trigger immediate bidirectional sync |
SyncCommand::Subscribe { collection, filter } | Start syncing a collection |
SyncCommand::Unsubscribe { collection } | Stop syncing a collection |
SyncCommand::SetOnline(bool) | Set online/offline status |
SyncCommand::Stop | Stop the sync manager |
LocalStore Direct Access
For advanced use cases, you can work directly with the underlying SQLite store.
use solidb_client::LocalStore;
// Open store at specific path
let store = LocalStore::open(
"/path/to/data.db",
"device-id".to_string()
)?;
// Or use default path
let store = LocalStore::open_default("app_name", "device-id".to_string())?;
// Direct document operations
store.put_document("users", "key", &json!({...}), &version_vector)?;
let doc = store.get_document("users", "key")?;
store.delete_document("users", "key", &version_vector)?;
// List documents in collection
let docs = store.list_documents("users")?;
// Returns: Vec<(key, document, version_vector)>
// List all collections
let collections = store.list_collections()?;
// Pending changes (for sync)
store.add_pending_change(collection, key, operation, data, vector)?;
let pending = store.get_pending_changes()?;
store.mark_change_synced(change_id)?;
// Subscriptions (for selective sync)
store.add_subscription(collection, filter)?;
let subs = store.list_subscriptions()?;
store.remove_subscription(collection)?;
SyncManager Method Reference
| Method | Returns | Description |
|---|---|---|
new(store, client, config) | Self | Create new sync manager |
start() | Sender<SyncCommand> | Start background sync, returns command channel |
save_document(col, key, doc) | Result<()> | Save (insert/update) document locally |
get_document(col, key) | Result<Option<Value>> | Get document from local store |
delete_document(col, key) | Result<()> | Delete document locally |
query_documents(col) | Result<Vec<(String, Value)>> | List all documents in collection |
query(sdbql, bind_vars) | Result<Vec<Value>> | Execute full SDBQL query on local data |
sync_now() | Result<SyncResult> | Trigger immediate bidirectional sync |
is_syncing() | bool | Check if sync is currently running |
is_online() | bool | Check if client is online |
SyncResult Structure
pub struct SyncResult {
pub success: bool, // Whether sync completed successfully
pub pulled: usize, // Number of documents pulled from server
pub pushed: usize, // Number of documents pushed to server
pub conflicts: usize, // Number of conflicts detected
pub errors: Vec<String>, // Error messages if any
}
API Endpoints
The sync protocol uses standard HTTP endpoints for compatibility. You can also use these endpoints directly for custom sync implementations.
/_api/sync/delta
Push local changes and receive server delta in one request.
{
"database": "mydb",
"client_id": "client-A",
"version_vector": {
"client-A": 5,
"server": 8
},
"changes": [
{
"collection": "users",
"key": "user-123",
"operation": "insert",
"document": { "name": "Alice", "email": "[email protected]" },
"timestamp": "2024-01-15T10:30:00Z"
}
]
}
{
"server_version": {
"client-A": 6,
"client-B": 2,
"server": 9
},
"server_changes": [
{
"collection": "users",
"key": "user-456",
"operation": "update",
"delta": { "status": "active" },
"version_vector": { "client-B": 2, "server": 9 }
}
],
"conflicts": []
}
/_api/sync/pull/:database
Get changes from server since last sync.
?client_id=client-A
&version_vector={"client-A":5,"server":8}
&collections=users,orders // Optional: filter by collections
&limit=100 // Optional: max changes to return
/_api/sync/push/:database
Send local changes to server.
{
"client_id": "client-A",
"changes": [
{
"collection": "users",
"key": "user-123",
"operation": "update",
"delta": { "email": "[email protected]" },
"base_version": { "client-A": 5, "server": 8 }
}
]
}
/_api/sync/status/:database
Get server-side sync status for a client.
{
"client_id": "client-A",
"server_version": {
"client-A": 10,
"client-B": 5,
"server": 15
},
"pending_for_client": 3,
"last_sync": "2024-01-15T10:30:00Z",
"collections_synced": ["users", "orders"]
}
Conflict Resolution
When the same document is modified on multiple clients simultaneously, SoliDB uses version vectors to detect conflicts and applies configurable resolution strategies.
Conflict Detection
A conflict occurs when two changes have version vectors that are neither descendants nor ancestors of each other (concurrent modifications).
Client A
Version: {A: 2, S: 1}
Change: name = "Alice"
Client B
Version: {B: 2, S: 1}
Change: name = "Alicia"
Conflict detected: Neither version dominates the other. Both clients modified the same document based on the same server version.
Resolution Strategies
Last Write Wins (LWW)
The most recent change based on server timestamp wins. Simple but may lose data.
ConflictStrategy::LastWriteWins
First Write Wins (FWW)
The first change to arrive at the server wins. Later changes are rejected.
ConflictStrategy::FirstWriteWins
Merge Fields
Automatically merge non-conflicting fields. Conflicting fields use LWW or are marked for manual resolution.
ConflictStrategy::MergeFields
Custom Resolver
Provide your own conflict resolution function for full control.
ConflictStrategy::Custom(Box::new(|local, remote, base| {
// Custom merge logic
let mut resolved = local.clone();
if remote["timestamp"] > local["timestamp"] {
resolved["status"] = remote["status"].clone();
}
Ok(resolved)
}))
Handling Conflicts in Code
// Check for conflicts after sync
let result = sync_manager.sync().await?;
if !result.conflicts.is_empty() {
for conflict in &result.conflicts {
println!("Conflict in {}:{}",
conflict.collection, conflict.document_key);
// Access conflicting versions
let local_version = &conflict.local_version;
let remote_version = &conflict.remote_version;
let base_version = &conflict.base_version;
// Manual resolution
let resolved = resolve_manually(
local_version,
remote_version,
base_version
);
// Apply resolution
sync_manager.resolve_conflict(
&conflict.collection,
&conflict.document_key,
resolved
).await?;
}
}
// Or handle conflicts via callback
sync_manager.on_conflict(|conflict| {
// Show UI for user to choose
// Or auto-resolve based on business logic
ConflictResolution::AcceptRemote // or AcceptLocal, Merge(doc)
});
Local SDBQL Queries
The Rust client supports full SDBQL query execution against local data, even when completely offline.
This is powered by the sdbql-core crate, which provides a storage-independent query engine
that works with SQLite, RocksDB, or any custom data source.
Why This Matters
Unlike simple key-value lookups, full SDBQL support means you can run complex queries with filters, sorts, aggregations, and joins - all locally. Your application logic doesn't need to change between online and offline modes.
Query API
use std::collections::HashMap;
use serde_json::json;
// Simple query - no bind variables
let users = sync_manager.query(
"FOR u IN users FILTER u.status == 'active' RETURN u",
None
).await?;
// Query with bind variables (recommended)
let mut bind_vars = HashMap::new();
bind_vars.insert("min_age".to_string(), json!(18));
bind_vars.insert("status".to_string(), json!("premium"));
let premium_users = sync_manager.query(
r#"FOR u IN users
FILTER u.age >= @min_age AND u.status == @status
SORT u.created_at DESC
LIMIT 50
RETURN { name: u.name, email: u.email }"#,
Some(bind_vars)
).await?;
// Aggregations work locally
let stats = sync_manager.query(
r#"FOR u IN users
COLLECT status = u.status WITH COUNT INTO count
RETURN { status, count }"#,
None
).await?;
// Joins between local collections
let orders_with_users = sync_manager.query(
r#"FOR o IN orders
FOR u IN users FILTER u._key == o.user_id
RETURN { order: o, customer: u.name }"#,
None
).await?;
Supported SDBQL Features
Fully Supported
FOR doc IN collection- Collection iterationFOR item IN array- Array iterationFOR i IN 1..10- Numeric rangesFILTER condition- All comparison operatorsSORT field ASC/DESC- Multi-field sortingLIMIT offset, count- PaginationRETURN expression- Any expressionLET var = expr- Variable bindingsCOLLECT/GROUP BY- AggregationsJOIN / LEFT JOIN- Local collections onlyNested FOR loops- Full supportBind variables- @var syntax
Server-Only Features
INSERT/UPDATE/DELETE- Use save_document() insteadUPSERT- Use save_document() insteadGraph traversal- OUTBOUND/INBOUND/ANYGEO_* functions- Requires geo indexFULLTEXT()- Requires fulltext indexSEARCH- Requires server searchDOCUMENT()- Cross-collection lookupsCrypto functions- Server keys required
These features require server-side resources. Use them when online or sync first.
Available Builtin Functions
Over 60 builtin functions are available for offline queries across these categories:
String
Array
Math
Date & Time
Type Checking
JSON / Object
Higher-Order Functions
Lambda expressions are supported for FILTER, MAP, REDUCE, ANY, ALL, and NONE:
FILTER(users, u -> u.age > 18 AND u.status == 'active')
Advanced Features
CRDT Data Types
Conflict-free Replicated Data Types (CRDTs) provide automatic conflict resolution for specific data structures. Use these when you need guaranteed convergence without conflicts.
PNCounter (Counter)
Increment/decrement counter that never loses operations.
// Increment counter locally
sync_manager.crdt_increment(
"users",
"user-123",
"login_count",
1
).await?;
// Decrement
sync_manager.crdt_decrement(
"inventory",
"item-456",
"stock",
1
).await?;
LWWRegister
Last-write-wins register for single values.
// Set value with automatic LWW semantics
sync_manager.crdt_set(
"users",
"user-123",
"last_seen",
json!("2024-01-15T10:00:00Z")
).await?;
GSet (Grow-only Set)
Set that only grows - elements can be added but never removed.
// Add to set
sync_manager.crdt_add_to_set(
"users",
"user-123",
"tags",
json!("premium")
).await?;
ORMap (Observed-Remove Map)
Map with observable add/remove operations.
// Put value in map
sync_manager.crdt_map_put(
"users",
"user-123",
"preferences",
"theme",
json!("dark")
).await?;
// Remove from map
sync_manager.crdt_map_remove(
"users",
"user-123",
"preferences",
"theme"
).await?;
Delta Encoding
Delta encoding reduces bandwidth by transmitting only changed fields. SoliDB automatically computes and applies deltas.
// Enable delta compression for a collection
sync_manager.enable_delta_sync("users", true).await?;
// Delta sync is automatic, but you can access delta info
let delta_info = sync_manager.get_last_delta_stats().await?;
println!("Last sync: {} bytes delta vs {} bytes full",
delta_info.delta_size, delta_info.full_size);
// Configure delta threshold (sync full doc if delta > 50%)
sync_manager.set_delta_threshold(0.5).await?;
Selective Sync
Sync only the data you need. Useful for large datasets where clients only need a subset.
// Sync only current user's data
sync_manager.set_sync_filter(
"FOR u IN users FILTER u._key == @user_id RETURN u",
vec![("user_id", json!("user-123"))].into_iter().collect()
).await?;
// Sync recent orders only
sync_manager.set_sync_filter(
"FOR o IN orders FILTER o.created_at > @since RETURN o",
vec![("since", json!("2024-01-01T00:00:00Z"))].into_iter().collect()
).await?;
// Multi-collection sync with different filters
sync_manager.set_collection_sync_filters(vec![
("users", None, None), // Sync all users
("orders",
Some("FOR o IN orders FILTER o.user_id == @id RETURN o"),
Some(vec![("id", json!("user-123"))].into_iter().collect())),
]).await?;
Local Storage Encryption
Encrypt local storage to protect data at rest. Uses AES-256-GCM with your provided key.
// Configure encryption in SyncConfig
let config = SyncConfig {
database: "mydb".to_string(),
local_storage_path: "./encrypted_data".to_string(),
encryption_key: Some("your-32-byte-secret-key-here!!".to_string()),
..Default::default()
};
// Or enable encryption on existing store
sync_manager.enable_encryption("your-32-byte-secret-key-here!!").await?;
// Rotate encryption key
sync_manager.rotate_encryption_key("new-32-byte-secret-key-here").await?;
// Disable encryption (decrypts all data)
sync_manager.disable_encryption().await?;
Best Practices
1. Design for Offline-First
Build your application assuming it's offline by default. Use local data for all UI rendering and treat server sync as a background optimization. This ensures your app works smoothly regardless of connectivity.
2. Sync Selectively
Don't sync everything. Use selective sync filters to limit data to what the user actually needs. This reduces storage usage, sync time, and bandwidth consumption.
3. Handle Conflicts Gracefully
Don't ignore conflicts. Either use automatic resolution strategies that match your use case, or implement UI flows for manual resolution when necessary. Always log conflicts for debugging.
4. Encrypt Sensitive Data
Always enable local storage encryption for sensitive data. Mobile devices can be lost or stolen, and local storage should be treated as untrusted.
5. Monitor Sync Health
Track sync metrics: pending changes, sync duration, conflict rates, and error counts. Set up alerts for sync failures and investigate recurring issues.
6. Clean Up Old Data
Implement data retention policies. Use TTL indexes or periodic cleanup to remove old local data that no longer needs to be synced. This keeps local storage performant.
7. Test Offline Scenarios
Test your app in airplane mode, with intermittent connectivity, and with high latency. Use network link conditioners to simulate real-world conditions during development.
8. Handle Storage Limits
Monitor local storage usage and implement eviction policies when approaching limits. Inform users when local storage is full and provide options to free space.
Troubleshooting
Sync Stuck or Hanging
- • Check network connectivity with
ping() - • Verify authentication hasn't expired
- • Check for large payloads that may timeout
- • Review server logs for errors
High Conflict Rates
- • Reduce sync interval to catch changes faster
- • Use CRDT types for frequently modified fields
- • Review data model for concurrent write hotspots
- • Consider implementing field-level locking in UI
Local Storage Errors
- • Check available disk space
- • Verify write permissions for storage path
- • Reset local storage with
full_resync() - • Enable encryption only after storage initialization
Data Not Syncing
- • Verify sync filter isn't too restrictive
- • Check that collections are in sync list
- • Ensure version vectors are incrementing
- • Verify server has sync endpoint enabled
Next Steps
Rust Client SDK
Complete reference for the Rust client with offline sync.
SDBQL Reference
Full query language documentation for online and offline queries.
Live Queries
Combine offline sync with real-time updates when online.
sdbql-core Crate
The offline query engine is powered by sdbql-core, a storage-independent SDBQL
parser and executor. It's available as a standalone crate for building custom offline-first applications.
cargo add sdbql-core