Offline Sync

Build resilient applications that work anywhere. SoliDB's offline sync enables local-first data with automatic conflict resolution.

Introduction

Offline Sync is SoliDB's solution for building local-first applications. It allows clients to maintain a local replica of their data, work offline, and seamlessly synchronize changes when connectivity is restored.

Offline-First

Applications work without network connectivity. All operations are performed locally and synced automatically when online.

Instant Response

Zero-latency reads and writes from local storage. No waiting for network round-trips.

Auto Conflict Resolution

Sophisticated conflict resolution using version vectors ensures data consistency across all devices.

Why Offline Sync Matters

Modern applications need to work everywhere - from subway tunnels to airplanes. Offline sync ensures your users have a consistent experience regardless of connectivity. It's essential for mobile apps, field tools, and any application where reliability is critical.

Architecture

SoliDB's offline sync uses a sophisticated architecture combining version vectors, delta encoding, and CRDT principles to ensure data consistency across distributed clients.

Sync Flow

Local Write
SQLite/RocksDB
Version Vector
{node: counter}
Delta Sync
Changes only
Merge & Resolve
Server reconciles

Version Vectors

Version vectors track the history of each document across all nodes. Unlike simple timestamps, they can detect concurrent modifications and establish causal relationships between changes.

{
  "_key": "user-123",
  "name": "Alice",
  "_version": {
    "client-A": 5,    // Client A has made 5 edits
    "client-B": 2,    // Client B has made 2 edits
    "server": 8       // Server has processed 8 total changes
  },
  "_timestamp": "2024-01-15T10:30:00Z"
}

Push Sync

Client-initiated sync sending local changes to the server:

  • 1. Collect pending changes from local store
  • 2. Send delta to server with version vector
  • 3. Server validates and merges changes
  • 4. Server returns updated version vector
  • 5. Client marks changes as synced

Pull Sync

Client retrieves changes from the server:

  • 1. Send current version vector to server
  • 2. Server computes changes since last sync
  • 3. Server returns delta with new version vector
  • 4. Client applies changes to local store
  • 5. Client resolves any local conflicts

Key Features

Delta Sync

Only changed fields are transmitted. Bandwidth-efficient sync for mobile and low-bandwidth environments.

CRDT Types

Built-in CRDT types (counters, registers, sets, maps) that merge automatically without conflicts.

Automatic Retry

Failed syncs are automatically retried with exponential backoff. No manual intervention required.

Local Storage

Uses SQLite or RocksDB locally. Full query capabilities on local data, even when offline.

Selective Sync

Sync only the data you need. Filter by user, date range, or any query criteria to save storage.

Change Notifications

Subscribe to local and remote changes. Real-time UI updates as data syncs across devices.

Quick Start

Get started with offline sync in minutes using the Rust client SDK. This example shows basic setup, local operations, and sync.

use solidb_client::{SoliDBClient, OfflineSyncManager, SyncConfig};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box> {
    // 1. Create client and enable offline sync
    let client = SoliDBClient::connect("localhost:6745").await?;
    
    let sync_manager = OfflineSyncManager::new(client)
        .with_config(SyncConfig {
            database: "mydb".to_string(),
            local_storage_path: "./local_data".to_string(),
            auto_sync: true,
            sync_interval_seconds: 30,
        })
        .initialize()
        .await?;

    // 2. Authenticate (credentials cached for reconnection)
    sync_manager.authenticate("admin", "password").await?;

    // 3. Work offline - changes stored locally
    let local_doc = sync_manager.insert_local("users", None, json!({
        "name": "Alice",
        "email": "[email protected]",
        "offline_created": true,
    })).await?;
    
    println!("Local document created: {}", local_doc["_key"]);

    // 4. Query local data (works offline!)
    let users = sync_manager.query_local(
        "FOR u IN users FILTER u.name == @name RETURN u",
        Some(vec![("name", json!("Alice"))].into_iter().collect())
    ).await?;
    
    println!("Found {} local users", users.len());

    // 5. Trigger sync when online
    let sync_result = sync_manager.sync().await?;
    println!("Synced: {} up, {} down", sync_result.pushed, sync_result.pulled);

    // 6. Listen for changes
    sync_manager.on_change(|change| {
        match change.change_type {
            ChangeType::LocalInsert => println!("Local insert: {}", change.document_key),
            ChangeType::RemoteInsert => println!("Remote insert: {}", change.document_key),
            ChangeType::Conflict => println!("Conflict resolved: {}", change.document_key),
            _ => {}
        }
    });

    Ok(())
}

That's It!

With just a few lines of code, your application can now work offline, automatically sync when online, and handle conflicts intelligently. The sync manager handles all the complexity for you.

Rust Client SDK

The OfflineSyncManager provides a comprehensive API for offline-first applications. All operations are async and work seamlessly with Tokio.

Configuration

use solidb_client::{SyncConfig, ConflictStrategy, StorageEngine};

let config = SyncConfig {
    // Required settings
    database: "production".to_string(),
    local_storage_path: "./sync_data".to_string(),
    
    // Sync behavior
    auto_sync: true,
    sync_interval_seconds: 60,
    push_on_change: true,           // Sync immediately on local changes
    
    // Conflict resolution strategy
    conflict_strategy: ConflictStrategy::LastWriteWins,
    // Other options:
    // - ConflictStrategy::FirstWriteWins
    // - ConflictStrategy::MergeFields
    // - ConflictStrategy::Custom(Box::new(my_resolver))
    
    // Storage engine (default: SQLite)
    storage_engine: StorageEngine::Sqlite,
    // Alternative: StorageEngine::RocksDB
    
    // Selective sync - only sync matching documents
    sync_filter: Some(
        "FOR doc IN users FILTER doc.user_id == @my_id RETURN doc".to_string()
    ),
    sync_filter_params: Some(vec![("my_id", json!("user-123"))].into_iter().collect()),
    
    // Encryption for local storage
    encryption_key: Some("my-secret-key".to_string()),
};

Local Operations

All CRUD operations work on local data. Changes are queued for sync automatically.

// Insert document locally
let doc = sync_manager.insert_local(
    "users",
    Some("user-456"),           // Optional key (auto-generated if None)
    json!({
        "name": "Bob",
        "email": "[email protected]",
        "created_at": "2024-01-15T10:00:00Z",
    })
).await?;

// Get document from local store
let user = sync_manager.get_local("users", "user-456").await?;

// Update document (merge mode)
let updated = sync_manager.update_local(
    "users",
    "user-456",
    json!({"email": "[email protected]"}),
    true                        // merge = true (partial update)
).await?;

// Update document (replace mode)
let replaced = sync_manager.update_local(
    "users",
    "user-456",
    json!({
        "name": "Bob",
        "email": "[email protected]",
        "status": "active",
    }),
    false                       // merge = false (full replacement)
).await?;

// Delete document locally
sync_manager.delete_local("users", "user-456").await?;

// List documents with pagination
let (docs, total) = sync_manager.list_local(
    "users",
    Some(50),                   // limit
    Some(0),                    // offset
    Some("name ASC"),           // sort
).await?;

Querying Local Data

Run SDBQL queries against local data. The query engine works identically whether online or offline.

// Simple query
let active_users = sync_manager.query_local(
    "FOR u IN users FILTER u.status == 'active' RETURN u",
    None
).await?;

// Query with bind variables
let mut params = HashMap::new();
params.insert("min_age".to_string(), json!(18));
params.insert("status".to_string(), json!("premium"));

let premium_users = sync_manager.query_local(
    r#"FOR u IN users
       FILTER u.age >= @min_age AND u.status == @status
       SORT u.created_at DESC
       RETURN { name: u.name, email: u.email }"#,
    Some(params)
).await?;

// Aggregation queries
let stats = sync_manager.query_local(
    r#"FOR u IN users
       COLLECT status = u.status WITH COUNT INTO count
       RETURN { status, count }"#,
    None
).await?;

// Joins (local data only)
let orders_with_users = sync_manager.query_local(
    r#"FOR o IN orders
       FOR u IN users FILTER u._key == o.user_id
       RETURN { order: o, user: u.name }"#,
    None
).await?;

Manual Sync Control

While auto-sync handles most cases, you have full control over the synchronization process.

// Trigger full bidirectional sync
let result = sync_manager.sync().await?;
println!("Pushed: {}, Pulled: {}", result.pushed, result.pulled);
println!("Conflicts: {}", result.conflicts.len());

// Push only (send local changes to server)
let push_result = sync_manager.push().await?;

// Pull only (get server changes)
let pull_result = sync_manager.pull().await?;

// Sync specific collections only
let partial_result = sync_manager.sync_collections(vec!["users", "orders"]).await?;

// Check sync status
let status = sync_manager.sync_status().await;
println!("Pending changes: {}", status.pending_changes);
println!("Last sync: {:?}", status.last_sync_time);
println!("Is syncing: {}", status.is_syncing);

// Pause/resume auto-sync
sync_manager.pause_sync().await?;
// ... do work without syncing ...
sync_manager.resume_sync().await?;

// Force full re-sync (download everything)
sync_manager.full_resync().await?;

Change Subscriptions

Subscribe to local and remote changes to update your UI in real-time.

use solidb_client::{Change, ChangeType};

// Subscribe to all changes
sync_manager.on_change(|change: Change| {
    match change.change_type {
        ChangeType::LocalInsert => {
            println!("📱 Local insert: {} in {}", 
                change.document_key, change.collection);
        }
        ChangeType::LocalUpdate => {
            println!("✏️  Local update: {} in {}", 
                change.document_key, change.collection);
        }
        ChangeType::LocalDelete => {
            println!("🗑️  Local delete: {} in {}", 
                change.document_key, change.collection);
        }
        ChangeType::RemoteInsert => {
            println!("☁️  Remote insert: {} in {}", 
                change.document_key, change.collection);
        }
        ChangeType::RemoteUpdate => {
            println!("🔄 Remote update: {} in {}", 
                change.document_key, change.collection);
        }
        ChangeType::RemoteDelete => {
            println!("❌ Remote delete: {} in {}", 
                change.document_key, change.collection);
        }
        ChangeType::Conflict => {
            println!("⚠️  Conflict resolved: {} in {}", 
                change.document_key, change.collection);
        }
        ChangeType::SyncComplete => {
            println!("✅ Sync completed: {} up, {} down", 
                change.pushed_count, change.pulled_count);
        }
    }
});

// Subscribe to specific collection changes
sync_manager.on_collection_change("users", |change| {
    println!("User changed: {:?}", change);
});

// Subscribe to specific document changes
sync_manager.on_document_change("users", "user-123", |change| {
    println!("User-123 changed: {:?}", change);
});

Method Reference

Method Returns Description
initialize()Result<Self>Initialize local storage
authenticate(user, pass)Result<()>Authenticate and cache credentials
insert_local(col, key, doc)Result<Value>Insert document locally
get_local(col, key)Result<Value>Get document from local store
update_local(col, key, doc, merge)Result<Value>Update document locally
delete_local(col, key)Result<()>Delete document locally
list_local(col, limit, offset, sort)Result<(Vec, usize)>List documents with pagination
query_local(query, params)Result<Vec<Value>>Execute SDBQL on local data
sync()Result<SyncResult>Bidirectional sync
push()Result<SyncResult>Push local changes to server
pull()Result<SyncResult>Pull server changes to local
sync_status()SyncStatusGet current sync status

API Endpoints

The sync protocol uses standard HTTP endpoints for compatibility. You can also use these endpoints directly for custom sync implementations.

POST /_api/sync/delta

Push local changes and receive server delta in one request.

// Request
{
  "database": "mydb",
  "client_id": "client-A",
  "version_vector": {
    "client-A": 5,
    "server": 8
  },
  "changes": [
    {
      "collection": "users",
      "key": "user-123",
      "operation": "insert",
      "document": { "name": "Alice", "email": "[email protected]" },
      "timestamp": "2024-01-15T10:30:00Z"
    }
  ]
}
// Response
{
  "server_version": {
    "client-A": 6,
    "client-B": 2,
    "server": 9
  },
  "server_changes": [
    {
      "collection": "users",
      "key": "user-456",
      "operation": "update",
      "delta": { "status": "active" },
      "version_vector": { "client-B": 2, "server": 9 }
    }
  ],
  "conflicts": []
}
GET /_api/sync/pull/:database

Get changes from server since last sync.

// Query Parameters
?client_id=client-A
&version_vector={"client-A":5,"server":8}
&collections=users,orders  // Optional: filter by collections
&limit=100                 // Optional: max changes to return
POST /_api/sync/push/:database

Send local changes to server.

// Request Body
{
  "client_id": "client-A",
  "changes": [
    {
      "collection": "users",
      "key": "user-123",
      "operation": "update",
      "delta": { "email": "[email protected]" },
      "base_version": { "client-A": 5, "server": 8 }
    }
  ]
}
GET /_api/sync/status/:database

Get server-side sync status for a client.

// Response
{
  "client_id": "client-A",
  "server_version": {
    "client-A": 10,
    "client-B": 5,
    "server": 15
  },
  "pending_for_client": 3,
  "last_sync": "2024-01-15T10:30:00Z",
  "collections_synced": ["users", "orders"]
}

Conflict Resolution

When the same document is modified on multiple clients simultaneously, SoliDB uses version vectors to detect conflicts and applies configurable resolution strategies.

Conflict Detection

A conflict occurs when two changes have version vectors that are neither descendants nor ancestors of each other (concurrent modifications).

Client A

Version: {A: 2, S: 1}
Change: name = "Alice"

Client B

Version: {B: 2, S: 1}
Change: name = "Alicia"

Conflict detected: Neither version dominates the other. Both clients modified the same document based on the same server version.

Resolution Strategies

Last Write Wins (LWW)

The most recent change based on server timestamp wins. Simple but may lose data.

ConflictStrategy::LastWriteWins

First Write Wins (FWW)

The first change to arrive at the server wins. Later changes are rejected.

ConflictStrategy::FirstWriteWins

Merge Fields

Automatically merge non-conflicting fields. Conflicting fields use LWW or are marked for manual resolution.

ConflictStrategy::MergeFields

Custom Resolver

Provide your own conflict resolution function for full control.

ConflictStrategy::Custom(Box::new(|local, remote, base| {
    // Custom merge logic
    let mut resolved = local.clone();
    if remote["timestamp"] > local["timestamp"] {
        resolved["status"] = remote["status"].clone();
    }
    Ok(resolved)
}))

Handling Conflicts in Code

// Check for conflicts after sync
let result = sync_manager.sync().await?;

if !result.conflicts.is_empty() {
    for conflict in &result.conflicts {
        println!("Conflict in {}:{}", 
            conflict.collection, conflict.document_key);
        
        // Access conflicting versions
        let local_version = &conflict.local_version;
        let remote_version = &conflict.remote_version;
        let base_version = &conflict.base_version;
        
        // Manual resolution
        let resolved = resolve_manually(
            local_version, 
            remote_version,
            base_version
        );
        
        // Apply resolution
        sync_manager.resolve_conflict(
            &conflict.collection,
            &conflict.document_key,
            resolved
        ).await?;
    }
}

// Or handle conflicts via callback
sync_manager.on_conflict(|conflict| {
    // Show UI for user to choose
    // Or auto-resolve based on business logic
    ConflictResolution::AcceptRemote  // or AcceptLocal, Merge(doc)
});

Advanced Features

CRDT Data Types

Conflict-free Replicated Data Types (CRDTs) provide automatic conflict resolution for specific data structures. Use these when you need guaranteed convergence without conflicts.

PNCounter (Counter)

Increment/decrement counter that never loses operations.

// Increment counter locally
sync_manager.crdt_increment(
    "users", 
    "user-123", 
    "login_count", 
    1
).await?;

// Decrement
sync_manager.crdt_decrement(
    "inventory",
    "item-456",
    "stock",
    1
).await?;

LWWRegister

Last-write-wins register for single values.

// Set value with automatic LWW semantics
sync_manager.crdt_set(
    "users",
    "user-123",
    "last_seen",
    json!("2024-01-15T10:00:00Z")
).await?;

GSet (Grow-only Set)

Set that only grows - elements can be added but never removed.

// Add to set
sync_manager.crdt_add_to_set(
    "users",
    "user-123",
    "tags",
    json!("premium")
).await?;

ORMap (Observed-Remove Map)

Map with observable add/remove operations.

// Put value in map
sync_manager.crdt_map_put(
    "users",
    "user-123",
    "preferences",
    "theme",
    json!("dark")
).await?;

// Remove from map
sync_manager.crdt_map_remove(
    "users",
    "user-123",
    "preferences",
    "theme"
).await?;

Delta Encoding

Delta encoding reduces bandwidth by transmitting only changed fields. SoliDB automatically computes and applies deltas.

// Enable delta compression for a collection
sync_manager.enable_delta_sync("users", true).await?;

// Delta sync is automatic, but you can access delta info
let delta_info = sync_manager.get_last_delta_stats().await?;
println!("Last sync: {} bytes delta vs {} bytes full", 
    delta_info.delta_size, delta_info.full_size);

// Configure delta threshold (sync full doc if delta > 50%)
sync_manager.set_delta_threshold(0.5).await?;

Selective Sync

Sync only the data you need. Useful for large datasets where clients only need a subset.

// Sync only current user's data
sync_manager.set_sync_filter(
    "FOR u IN users FILTER u._key == @user_id RETURN u",
    vec![("user_id", json!("user-123"))].into_iter().collect()
).await?;

// Sync recent orders only
sync_manager.set_sync_filter(
    "FOR o IN orders FILTER o.created_at > @since RETURN o",
    vec![("since", json!("2024-01-01T00:00:00Z"))].into_iter().collect()
).await?;

// Multi-collection sync with different filters
sync_manager.set_collection_sync_filters(vec![
    ("users", None, None),  // Sync all users
    ("orders", 
     Some("FOR o IN orders FILTER o.user_id == @id RETURN o"),
     Some(vec![("id", json!("user-123"))].into_iter().collect())),
]).await?;

Local Storage Encryption

Encrypt local storage to protect data at rest. Uses AES-256-GCM with your provided key.

// Configure encryption in SyncConfig
let config = SyncConfig {
    database: "mydb".to_string(),
    local_storage_path: "./encrypted_data".to_string(),
    encryption_key: Some("your-32-byte-secret-key-here!!".to_string()),
    ..Default::default()
};

// Or enable encryption on existing store
sync_manager.enable_encryption("your-32-byte-secret-key-here!!").await?;

// Rotate encryption key
sync_manager.rotate_encryption_key("new-32-byte-secret-key-here").await?;

// Disable encryption (decrypts all data)
sync_manager.disable_encryption().await?;

Best Practices

1. Design for Offline-First

Build your application assuming it's offline by default. Use local data for all UI rendering and treat server sync as a background optimization. This ensures your app works smoothly regardless of connectivity.

2. Sync Selectively

Don't sync everything. Use selective sync filters to limit data to what the user actually needs. This reduces storage usage, sync time, and bandwidth consumption.

3. Handle Conflicts Gracefully

Don't ignore conflicts. Either use automatic resolution strategies that match your use case, or implement UI flows for manual resolution when necessary. Always log conflicts for debugging.

4. Encrypt Sensitive Data

Always enable local storage encryption for sensitive data. Mobile devices can be lost or stolen, and local storage should be treated as untrusted.

5. Monitor Sync Health

Track sync metrics: pending changes, sync duration, conflict rates, and error counts. Set up alerts for sync failures and investigate recurring issues.

6. Clean Up Old Data

Implement data retention policies. Use TTL indexes or periodic cleanup to remove old local data that no longer needs to be synced. This keeps local storage performant.

7. Test Offline Scenarios

Test your app in airplane mode, with intermittent connectivity, and with high latency. Use network link conditioners to simulate real-world conditions during development.

8. Handle Storage Limits

Monitor local storage usage and implement eviction policies when approaching limits. Inform users when local storage is full and provide options to free space.

Troubleshooting

Sync Stuck or Hanging

  • • Check network connectivity with ping()
  • • Verify authentication hasn't expired
  • • Check for large payloads that may timeout
  • • Review server logs for errors

High Conflict Rates

  • • Reduce sync interval to catch changes faster
  • • Use CRDT types for frequently modified fields
  • • Review data model for concurrent write hotspots
  • • Consider implementing field-level locking in UI

Local Storage Errors

  • • Check available disk space
  • • Verify write permissions for storage path
  • • Reset local storage with full_resync()
  • • Enable encryption only after storage initialization

Data Not Syncing

  • • Verify sync filter isn't too restrictive
  • • Check that collections are in sync list
  • • Ensure version vectors are incrementing
  • • Verify server has sync endpoint enabled