Offline Sync
Build resilient applications that work anywhere. SoliDB's offline sync enables local-first data with automatic conflict resolution.
Introduction
Offline Sync is SoliDB's solution for building local-first applications. It allows clients to maintain a local replica of their data, work offline, and seamlessly synchronize changes when connectivity is restored.
Offline-First
Applications work without network connectivity. All operations are performed locally and synced automatically when online.
Instant Response
Zero-latency reads and writes from local storage. No waiting for network round-trips.
Auto Conflict Resolution
Sophisticated conflict resolution using version vectors ensures data consistency across all devices.
Why Offline Sync Matters
Modern applications need to work everywhere - from subway tunnels to airplanes. Offline sync ensures your users have a consistent experience regardless of connectivity. It's essential for mobile apps, field tools, and any application where reliability is critical.
Architecture
SoliDB's offline sync uses a sophisticated architecture combining version vectors, delta encoding, and CRDT principles to ensure data consistency across distributed clients.
Sync Flow
Version Vectors
Version vectors track the history of each document across all nodes. Unlike simple timestamps, they can detect concurrent modifications and establish causal relationships between changes.
{
"_key": "user-123",
"name": "Alice",
"_version": {
"client-A": 5, // Client A has made 5 edits
"client-B": 2, // Client B has made 2 edits
"server": 8 // Server has processed 8 total changes
},
"_timestamp": "2024-01-15T10:30:00Z"
}
Push Sync
Client-initiated sync sending local changes to the server:
- 1. Collect pending changes from local store
- 2. Send delta to server with version vector
- 3. Server validates and merges changes
- 4. Server returns updated version vector
- 5. Client marks changes as synced
Pull Sync
Client retrieves changes from the server:
- 1. Send current version vector to server
- 2. Server computes changes since last sync
- 3. Server returns delta with new version vector
- 4. Client applies changes to local store
- 5. Client resolves any local conflicts
Key Features
Delta Sync
Only changed fields are transmitted. Bandwidth-efficient sync for mobile and low-bandwidth environments.
CRDT Types
Built-in CRDT types (counters, registers, sets, maps) that merge automatically without conflicts.
Automatic Retry
Failed syncs are automatically retried with exponential backoff. No manual intervention required.
Local Storage
Uses SQLite or RocksDB locally. Full query capabilities on local data, even when offline.
Selective Sync
Sync only the data you need. Filter by user, date range, or any query criteria to save storage.
Change Notifications
Subscribe to local and remote changes. Real-time UI updates as data syncs across devices.
Quick Start
Get started with offline sync in minutes using the Rust client SDK. This example shows basic setup, local operations, and sync.
use solidb_client::{SoliDBClient, OfflineSyncManager, SyncConfig};
use serde_json::json;
#[tokio::main]
async fn main() -> Result<(), Box> {
// 1. Create client and enable offline sync
let client = SoliDBClient::connect("localhost:6745").await?;
let sync_manager = OfflineSyncManager::new(client)
.with_config(SyncConfig {
database: "mydb".to_string(),
local_storage_path: "./local_data".to_string(),
auto_sync: true,
sync_interval_seconds: 30,
})
.initialize()
.await?;
// 2. Authenticate (credentials cached for reconnection)
sync_manager.authenticate("admin", "password").await?;
// 3. Work offline - changes stored locally
let local_doc = sync_manager.insert_local("users", None, json!({
"name": "Alice",
"email": "[email protected]",
"offline_created": true,
})).await?;
println!("Local document created: {}", local_doc["_key"]);
// 4. Query local data (works offline!)
let users = sync_manager.query_local(
"FOR u IN users FILTER u.name == @name RETURN u",
Some(vec![("name", json!("Alice"))].into_iter().collect())
).await?;
println!("Found {} local users", users.len());
// 5. Trigger sync when online
let sync_result = sync_manager.sync().await?;
println!("Synced: {} up, {} down", sync_result.pushed, sync_result.pulled);
// 6. Listen for changes
sync_manager.on_change(|change| {
match change.change_type {
ChangeType::LocalInsert => println!("Local insert: {}", change.document_key),
ChangeType::RemoteInsert => println!("Remote insert: {}", change.document_key),
ChangeType::Conflict => println!("Conflict resolved: {}", change.document_key),
_ => {}
}
});
Ok(())
}
That's It!
With just a few lines of code, your application can now work offline, automatically sync when online, and handle conflicts intelligently. The sync manager handles all the complexity for you.
Rust Client SDK
The OfflineSyncManager provides a comprehensive API for offline-first applications. All operations are async and work seamlessly with Tokio.
Configuration
use solidb_client::{SyncConfig, ConflictStrategy, StorageEngine};
let config = SyncConfig {
// Required settings
database: "production".to_string(),
local_storage_path: "./sync_data".to_string(),
// Sync behavior
auto_sync: true,
sync_interval_seconds: 60,
push_on_change: true, // Sync immediately on local changes
// Conflict resolution strategy
conflict_strategy: ConflictStrategy::LastWriteWins,
// Other options:
// - ConflictStrategy::FirstWriteWins
// - ConflictStrategy::MergeFields
// - ConflictStrategy::Custom(Box::new(my_resolver))
// Storage engine (default: SQLite)
storage_engine: StorageEngine::Sqlite,
// Alternative: StorageEngine::RocksDB
// Selective sync - only sync matching documents
sync_filter: Some(
"FOR doc IN users FILTER doc.user_id == @my_id RETURN doc".to_string()
),
sync_filter_params: Some(vec![("my_id", json!("user-123"))].into_iter().collect()),
// Encryption for local storage
encryption_key: Some("my-secret-key".to_string()),
};
Local Operations
All CRUD operations work on local data. Changes are queued for sync automatically.
// Insert document locally
let doc = sync_manager.insert_local(
"users",
Some("user-456"), // Optional key (auto-generated if None)
json!({
"name": "Bob",
"email": "[email protected]",
"created_at": "2024-01-15T10:00:00Z",
})
).await?;
// Get document from local store
let user = sync_manager.get_local("users", "user-456").await?;
// Update document (merge mode)
let updated = sync_manager.update_local(
"users",
"user-456",
json!({"email": "[email protected]"}),
true // merge = true (partial update)
).await?;
// Update document (replace mode)
let replaced = sync_manager.update_local(
"users",
"user-456",
json!({
"name": "Bob",
"email": "[email protected]",
"status": "active",
}),
false // merge = false (full replacement)
).await?;
// Delete document locally
sync_manager.delete_local("users", "user-456").await?;
// List documents with pagination
let (docs, total) = sync_manager.list_local(
"users",
Some(50), // limit
Some(0), // offset
Some("name ASC"), // sort
).await?;
Querying Local Data
Run SDBQL queries against local data. The query engine works identically whether online or offline.
// Simple query
let active_users = sync_manager.query_local(
"FOR u IN users FILTER u.status == 'active' RETURN u",
None
).await?;
// Query with bind variables
let mut params = HashMap::new();
params.insert("min_age".to_string(), json!(18));
params.insert("status".to_string(), json!("premium"));
let premium_users = sync_manager.query_local(
r#"FOR u IN users
FILTER u.age >= @min_age AND u.status == @status
SORT u.created_at DESC
RETURN { name: u.name, email: u.email }"#,
Some(params)
).await?;
// Aggregation queries
let stats = sync_manager.query_local(
r#"FOR u IN users
COLLECT status = u.status WITH COUNT INTO count
RETURN { status, count }"#,
None
).await?;
// Joins (local data only)
let orders_with_users = sync_manager.query_local(
r#"FOR o IN orders
FOR u IN users FILTER u._key == o.user_id
RETURN { order: o, user: u.name }"#,
None
).await?;
Manual Sync Control
While auto-sync handles most cases, you have full control over the synchronization process.
// Trigger full bidirectional sync
let result = sync_manager.sync().await?;
println!("Pushed: {}, Pulled: {}", result.pushed, result.pulled);
println!("Conflicts: {}", result.conflicts.len());
// Push only (send local changes to server)
let push_result = sync_manager.push().await?;
// Pull only (get server changes)
let pull_result = sync_manager.pull().await?;
// Sync specific collections only
let partial_result = sync_manager.sync_collections(vec!["users", "orders"]).await?;
// Check sync status
let status = sync_manager.sync_status().await;
println!("Pending changes: {}", status.pending_changes);
println!("Last sync: {:?}", status.last_sync_time);
println!("Is syncing: {}", status.is_syncing);
// Pause/resume auto-sync
sync_manager.pause_sync().await?;
// ... do work without syncing ...
sync_manager.resume_sync().await?;
// Force full re-sync (download everything)
sync_manager.full_resync().await?;
Change Subscriptions
Subscribe to local and remote changes to update your UI in real-time.
use solidb_client::{Change, ChangeType};
// Subscribe to all changes
sync_manager.on_change(|change: Change| {
match change.change_type {
ChangeType::LocalInsert => {
println!("📱 Local insert: {} in {}",
change.document_key, change.collection);
}
ChangeType::LocalUpdate => {
println!("✏️ Local update: {} in {}",
change.document_key, change.collection);
}
ChangeType::LocalDelete => {
println!("🗑️ Local delete: {} in {}",
change.document_key, change.collection);
}
ChangeType::RemoteInsert => {
println!("☁️ Remote insert: {} in {}",
change.document_key, change.collection);
}
ChangeType::RemoteUpdate => {
println!("🔄 Remote update: {} in {}",
change.document_key, change.collection);
}
ChangeType::RemoteDelete => {
println!("❌ Remote delete: {} in {}",
change.document_key, change.collection);
}
ChangeType::Conflict => {
println!("⚠️ Conflict resolved: {} in {}",
change.document_key, change.collection);
}
ChangeType::SyncComplete => {
println!("✅ Sync completed: {} up, {} down",
change.pushed_count, change.pulled_count);
}
}
});
// Subscribe to specific collection changes
sync_manager.on_collection_change("users", |change| {
println!("User changed: {:?}", change);
});
// Subscribe to specific document changes
sync_manager.on_document_change("users", "user-123", |change| {
println!("User-123 changed: {:?}", change);
});
Method Reference
| Method | Returns | Description |
|---|---|---|
initialize() | Result<Self> | Initialize local storage |
authenticate(user, pass) | Result<()> | Authenticate and cache credentials |
insert_local(col, key, doc) | Result<Value> | Insert document locally |
get_local(col, key) | Result<Value> | Get document from local store |
update_local(col, key, doc, merge) | Result<Value> | Update document locally |
delete_local(col, key) | Result<()> | Delete document locally |
list_local(col, limit, offset, sort) | Result<(Vec, usize)> | List documents with pagination |
query_local(query, params) | Result<Vec<Value>> | Execute SDBQL on local data |
sync() | Result<SyncResult> | Bidirectional sync |
push() | Result<SyncResult> | Push local changes to server |
pull() | Result<SyncResult> | Pull server changes to local |
sync_status() | SyncStatus | Get current sync status |
API Endpoints
The sync protocol uses standard HTTP endpoints for compatibility. You can also use these endpoints directly for custom sync implementations.
/_api/sync/delta
Push local changes and receive server delta in one request.
{
"database": "mydb",
"client_id": "client-A",
"version_vector": {
"client-A": 5,
"server": 8
},
"changes": [
{
"collection": "users",
"key": "user-123",
"operation": "insert",
"document": { "name": "Alice", "email": "[email protected]" },
"timestamp": "2024-01-15T10:30:00Z"
}
]
}
{
"server_version": {
"client-A": 6,
"client-B": 2,
"server": 9
},
"server_changes": [
{
"collection": "users",
"key": "user-456",
"operation": "update",
"delta": { "status": "active" },
"version_vector": { "client-B": 2, "server": 9 }
}
],
"conflicts": []
}
/_api/sync/pull/:database
Get changes from server since last sync.
?client_id=client-A
&version_vector={"client-A":5,"server":8}
&collections=users,orders // Optional: filter by collections
&limit=100 // Optional: max changes to return
/_api/sync/push/:database
Send local changes to server.
{
"client_id": "client-A",
"changes": [
{
"collection": "users",
"key": "user-123",
"operation": "update",
"delta": { "email": "[email protected]" },
"base_version": { "client-A": 5, "server": 8 }
}
]
}
/_api/sync/status/:database
Get server-side sync status for a client.
{
"client_id": "client-A",
"server_version": {
"client-A": 10,
"client-B": 5,
"server": 15
},
"pending_for_client": 3,
"last_sync": "2024-01-15T10:30:00Z",
"collections_synced": ["users", "orders"]
}
Conflict Resolution
When the same document is modified on multiple clients simultaneously, SoliDB uses version vectors to detect conflicts and applies configurable resolution strategies.
Conflict Detection
A conflict occurs when two changes have version vectors that are neither descendants nor ancestors of each other (concurrent modifications).
Client A
Version: {A: 2, S: 1}
Change: name = "Alice"
Client B
Version: {B: 2, S: 1}
Change: name = "Alicia"
Conflict detected: Neither version dominates the other. Both clients modified the same document based on the same server version.
Resolution Strategies
Last Write Wins (LWW)
The most recent change based on server timestamp wins. Simple but may lose data.
ConflictStrategy::LastWriteWins
First Write Wins (FWW)
The first change to arrive at the server wins. Later changes are rejected.
ConflictStrategy::FirstWriteWins
Merge Fields
Automatically merge non-conflicting fields. Conflicting fields use LWW or are marked for manual resolution.
ConflictStrategy::MergeFields
Custom Resolver
Provide your own conflict resolution function for full control.
ConflictStrategy::Custom(Box::new(|local, remote, base| {
// Custom merge logic
let mut resolved = local.clone();
if remote["timestamp"] > local["timestamp"] {
resolved["status"] = remote["status"].clone();
}
Ok(resolved)
}))
Handling Conflicts in Code
// Check for conflicts after sync
let result = sync_manager.sync().await?;
if !result.conflicts.is_empty() {
for conflict in &result.conflicts {
println!("Conflict in {}:{}",
conflict.collection, conflict.document_key);
// Access conflicting versions
let local_version = &conflict.local_version;
let remote_version = &conflict.remote_version;
let base_version = &conflict.base_version;
// Manual resolution
let resolved = resolve_manually(
local_version,
remote_version,
base_version
);
// Apply resolution
sync_manager.resolve_conflict(
&conflict.collection,
&conflict.document_key,
resolved
).await?;
}
}
// Or handle conflicts via callback
sync_manager.on_conflict(|conflict| {
// Show UI for user to choose
// Or auto-resolve based on business logic
ConflictResolution::AcceptRemote // or AcceptLocal, Merge(doc)
});
Advanced Features
CRDT Data Types
Conflict-free Replicated Data Types (CRDTs) provide automatic conflict resolution for specific data structures. Use these when you need guaranteed convergence without conflicts.
PNCounter (Counter)
Increment/decrement counter that never loses operations.
// Increment counter locally
sync_manager.crdt_increment(
"users",
"user-123",
"login_count",
1
).await?;
// Decrement
sync_manager.crdt_decrement(
"inventory",
"item-456",
"stock",
1
).await?;
LWWRegister
Last-write-wins register for single values.
// Set value with automatic LWW semantics
sync_manager.crdt_set(
"users",
"user-123",
"last_seen",
json!("2024-01-15T10:00:00Z")
).await?;
GSet (Grow-only Set)
Set that only grows - elements can be added but never removed.
// Add to set
sync_manager.crdt_add_to_set(
"users",
"user-123",
"tags",
json!("premium")
).await?;
ORMap (Observed-Remove Map)
Map with observable add/remove operations.
// Put value in map
sync_manager.crdt_map_put(
"users",
"user-123",
"preferences",
"theme",
json!("dark")
).await?;
// Remove from map
sync_manager.crdt_map_remove(
"users",
"user-123",
"preferences",
"theme"
).await?;
Delta Encoding
Delta encoding reduces bandwidth by transmitting only changed fields. SoliDB automatically computes and applies deltas.
// Enable delta compression for a collection
sync_manager.enable_delta_sync("users", true).await?;
// Delta sync is automatic, but you can access delta info
let delta_info = sync_manager.get_last_delta_stats().await?;
println!("Last sync: {} bytes delta vs {} bytes full",
delta_info.delta_size, delta_info.full_size);
// Configure delta threshold (sync full doc if delta > 50%)
sync_manager.set_delta_threshold(0.5).await?;
Selective Sync
Sync only the data you need. Useful for large datasets where clients only need a subset.
// Sync only current user's data
sync_manager.set_sync_filter(
"FOR u IN users FILTER u._key == @user_id RETURN u",
vec![("user_id", json!("user-123"))].into_iter().collect()
).await?;
// Sync recent orders only
sync_manager.set_sync_filter(
"FOR o IN orders FILTER o.created_at > @since RETURN o",
vec![("since", json!("2024-01-01T00:00:00Z"))].into_iter().collect()
).await?;
// Multi-collection sync with different filters
sync_manager.set_collection_sync_filters(vec![
("users", None, None), // Sync all users
("orders",
Some("FOR o IN orders FILTER o.user_id == @id RETURN o"),
Some(vec![("id", json!("user-123"))].into_iter().collect())),
]).await?;
Local Storage Encryption
Encrypt local storage to protect data at rest. Uses AES-256-GCM with your provided key.
// Configure encryption in SyncConfig
let config = SyncConfig {
database: "mydb".to_string(),
local_storage_path: "./encrypted_data".to_string(),
encryption_key: Some("your-32-byte-secret-key-here!!".to_string()),
..Default::default()
};
// Or enable encryption on existing store
sync_manager.enable_encryption("your-32-byte-secret-key-here!!").await?;
// Rotate encryption key
sync_manager.rotate_encryption_key("new-32-byte-secret-key-here").await?;
// Disable encryption (decrypts all data)
sync_manager.disable_encryption().await?;
Best Practices
1. Design for Offline-First
Build your application assuming it's offline by default. Use local data for all UI rendering and treat server sync as a background optimization. This ensures your app works smoothly regardless of connectivity.
2. Sync Selectively
Don't sync everything. Use selective sync filters to limit data to what the user actually needs. This reduces storage usage, sync time, and bandwidth consumption.
3. Handle Conflicts Gracefully
Don't ignore conflicts. Either use automatic resolution strategies that match your use case, or implement UI flows for manual resolution when necessary. Always log conflicts for debugging.
4. Encrypt Sensitive Data
Always enable local storage encryption for sensitive data. Mobile devices can be lost or stolen, and local storage should be treated as untrusted.
5. Monitor Sync Health
Track sync metrics: pending changes, sync duration, conflict rates, and error counts. Set up alerts for sync failures and investigate recurring issues.
6. Clean Up Old Data
Implement data retention policies. Use TTL indexes or periodic cleanup to remove old local data that no longer needs to be synced. This keeps local storage performant.
7. Test Offline Scenarios
Test your app in airplane mode, with intermittent connectivity, and with high latency. Use network link conditioners to simulate real-world conditions during development.
8. Handle Storage Limits
Monitor local storage usage and implement eviction policies when approaching limits. Inform users when local storage is full and provide options to free space.
Troubleshooting
Sync Stuck or Hanging
- • Check network connectivity with
ping() - • Verify authentication hasn't expired
- • Check for large payloads that may timeout
- • Review server logs for errors
High Conflict Rates
- • Reduce sync interval to catch changes faster
- • Use CRDT types for frequently modified fields
- • Review data model for concurrent write hotspots
- • Consider implementing field-level locking in UI
Local Storage Errors
- • Check available disk space
- • Verify write permissions for storage path
- • Reset local storage with
full_resync() - • Enable encryption only after storage initialization
Data Not Syncing
- • Verify sync filter isn't too restrictive
- • Check that collections are in sync list
- • Ensure version vectors are incrementing
- • Verify server has sync endpoint enabled