Storine v0.3.0

Build with Storine

Embedding the engine

Use Storine as a library inside a Rust program, with a document store or typed zero-copy records, and configure its durability, memory, encryption and backups.

The engine is the storage-core crate. A program that links it reads and writes a data directory directly, with no server and no network. It is the same engine a server runs.

[dependencies]
storage-core = { git = "https://github.com/twilight-techy/storine" }
storage-codec = { git = "https://github.com/twilight-techy/storine" }   # values and field operations
storage-types = { git = "https://github.com/twilight-techy/storine" }   # Decimal, errors, ids

There are two kinds of store:

StoreHoldsUse it for
DocumentStoreDocuments in collections, as a server doesMost programs. The same data model as the clients
StorageEngine<T>Records of one Rust type T, stored as a zero-copy archiveThe lowest latency: reads decode the record in place with no allocation

Two processes cannot open one data directory at once: the first holds documents.lock while it has it open.

A document store#

This program is in the repository as crates/storage-core/examples/documents.rs, and the test suite compiles it:

use std::collections::BTreeMap;

use storage_codec::field_ops::FieldOp;
use storage_codec::PropertyValue;
use storage_core::{
    Direction, DocOp, DocQuery, DocWrite, DocumentStore, EngineConfig, Filter, Precondition,
};
use storage_types::Decimal;

fn money(s: &str) -> PropertyValue {
    PropertyValue::Decimal(s.parse::<Decimal>().expect("a decimal"))
}

fn main() -> storage_types::Result<()> {
    let dir = std::env::temp_dir().join(format!("storine-example-{}", std::process::id()));
    let store = DocumentStore::open(&dir, EngineConfig::default())?;

    let mut alice = BTreeMap::new();
    alice.insert("owner".to_string(), PropertyValue::Text("Alice".into()));
    alice.insert("balance".to_string(), money("100.00"));
    store.insert("accounts", "alice", alice)?;
    store.put(
        "accounts",
        "bob",
        BTreeMap::from([("owner".to_string(), PropertyValue::Text("Bob".into()))]),
    )?;

    // Server-side, checked arithmetic: fails instead of going below zero.
    store.patch(
        "accounts",
        "alice",
        vec![FieldOp::Add {
            path: "balance".into(),
            delta: money("-25.00"),
            min: Some(money("0")),
            max: None,
        }],
    )?;

    // Several writes, all or nothing, applied at most once for this request id.
    let alice_version = store.get("accounts", "alice")?.expect("exists").version;
    let summary = store.write(
        vec![
            DocWrite {
                collection: "accounts".into(),
                key: "alice".into(),
                op: DocOp::Patch(vec![FieldOp::Add {
                    path: "balance".into(),
                    delta: money("-10.00"),
                    min: Some(money("0")),
                    max: None,
                }]),
                precondition: Precondition::Version(alice_version),
            },
            DocWrite {
                collection: "accounts".into(),
                key: "bob".into(),
                op: DocOp::Patch(vec![FieldOp::Add {
                    path: "balance".into(),
                    delta: money("10.00"),
                    min: None,
                    max: None,
                }]),
                precondition: Precondition::Exists,
            },
        ],
        Some(*b"transfer-0000001"),
    )?;
    assert!(!summary.duplicate);

    store.create_index("accounts", "owner")?;
    let page = store.query(&DocQuery {
        collection: "accounts".into(),
        filter: Some(Filter::Gt("balance".into(), money("50"))),
        order_by: vec![("balance".into(), Direction::Desc)],
        limit: Some(10),
        ..DocQuery::default()
    })?;
    for doc in &page.documents {
        println!(
            "{} v{} {:?}",
            doc.key,
            doc.version.0,
            doc.fields.get("balance")
        );
    }

    // An embedded store does not checkpoint on its own: this cuts the write-ahead log.
    store.engine().checkpoint()?;
    drop(store);
    let _ = std::fs::remove_dir_all(&dir);
    Ok(())
}

Run it with cargo run -p storage-core --example documents.

Its methods#

MethodDoes
open(dir, config)Opens or creates a store in dir, replaying the write-ahead log if the last run did not close cleanly
get(collection, key)The Document (key, version, fields), or None
get_fields(collection, key, &paths)Only those dotted paths, up to 256
read_bytes(collection, key, field, offset, length)A range of up to 16 MiB of a bytes or text field
put, insert, deleteCreate or replace; create only; delete (returns whether it existed)
patch(collection, key, ops)Apply FieldOps (Set, Unset, Add, Append) to the current version
write(writes, request_id)Many DocWrites across collections, atomically, each with a Precondition (None, Absent, Exists, Version). A request id is applied at most once
query(&DocQuery)Filter, sort, limit, offset or after_key, and nearest for vector search. Returns a DocPage
scan_collection(collection)Every document of a collection
create_index, drop_indexSecondary indexes
create_vector_index(collection, field, dims), drop_vector_indexVector indexes
create_columnar, drop_columnar, columnarColumnar storage
aggregate(&AggregateQuery), scan_columns(&ScanQuery)Analytics
list_collections(), collection_info(name)Collections with metadata or documents, and one collection's indexes and metadata
set_collection_extra(collection, map)Store metadata on a collection
drop_collection(name)Delete a collection's documents, indexes and metadata. Returns the number of documents
engine()The underlying StorageEngine<DocumentRecord>, for checkpoint, stats, compact_arenas, snapshots and backups

Important

An embedded store does not checkpoint on its own. Call store.engine().checkpoint() from time to time (a server does it every 10 seconds while written), or the write-ahead log grows until the store is dropped, and the next open replays all of it.

Typed records#

StorageEngine<T> stores one Rust type. T implements the engine's ZeroCopyRecord and Indexable traits, which describe its archived layout and which fields are indexed. AgentObjectRecord in crates/storage-core/src/model.rs is a complete example with numeric and tag indexes, relationships and an embedding.

Reads are zero-copy: with_ref(id, |view| ...) and get_ref(id) give a view into the memory-mapped record, with no heap allocation on a warm read. Writes include insert, put, put_with_ttl, update, put_cas and update_if_version (compare-and-swap on the version), delete, and transaction for atomic batches.

Relationships#

Typed records can link to each other one-to-one, one-to-many and many-to-many (OneToOne<T>, OneToMany<T>, ManyToMany<T>). Link updates lock keys in a fixed order, so concurrent updates cannot deadlock. update_pair_atomic changes two records together, get_related_refs follows links, and verify_relationships reports links to records that no longer exist.

Schema migrations#

Give a record type a SCHEMA_VERSION. Records written by an older version are upgraded when they are read or replayed, by migrations you register with register_migration or register_step_upcaster. upgrade_legacy_schemas rewrites every old record at once.

Snapshots and time to live#

  • begin_read_snapshot() gives a consistent view (MVCC) while writes continue.
  • Records written with a TTL read as missing once expired. Call sweep_expired() to delete them; nothing does it for an embedded engine.

Configuration#

EngineConfig::default() suits most programs. DocumentStore::open sets the document defaults itself.

FieldDefaultMeaning
collection_name"default" ("documents" for a DocumentStore)The store's file stem
layout_modeDedicatedFiles (the document layout for a DocumentStore)How records are laid out on disk, below
wal_sync_policyGroupCommit { max_batch_entries: 64 }When a write is durable, below
enable_waltrueWithout a WAL, buffered writes are lost on a crash
background_flush, flush_interval_mstrue, 10A background thread writes buffered records to their files every 10 ms. Several writes to one key in that time are written once
max_hot_dirty_bytes64 MiBWrites not yet flushed may use this much memory. Beyond it, writers flush themselves
mmap_pool16,384 mappings, 4 GiB mappedCold mappings are unmapped beyond these
wal_archive, wal_archive_dirfalse, noneKeep the WAL that checkpoints cut, for point-in-time recovery. Setting a directory turns it on
index_tierMemoryPaged keeps secondary indexes in memory-mapped pages on disk
primary_tierMemoryCold keeps the primary index as of the last checkpoint on disk
encryption, decrypted_cache_bytesnone, 256 MiBEncryption at rest, below
expected_records0About how many records the store will hold. The primary index is sized for that many at open, so a bulk load doesn't pause while its tables double. Costs 150–300 bytes per expected record up front, the memory the index would reach at that size anyway. reserve_records(n) does the same on an open store, before an import

Layouts#

StorageLayoutModeHow
AdaptiveArenaRecords are appended to memory-mapped segment files that start at 4 KiB and double up to the segment size. Records above a threshold get their own file. Compaction reclaims dead space. Documents use it with a 64 KiB threshold and 64 MiB segments
DedicatedFilesOne file per record, replaced atomically through a temporary file and a rename
PingPongFiles { slot_stride }One file per record with two slots, written in turn. For large records rewritten often

Sync policies#

SyncPolicyA write returns afterLost on a crash
Immediateits own fsyncnothing
GroupCommit { max_batch_entries }an fsync shared with writers arriving at the same timenothing
AsyncMemoryFirstit is staged in memory. The flusher hands it to the OS every flush_interval_ms, with no fsynca process crash loses the last flush interval; a power failure, what the OS had not written back

Range-cluster nodes use AsyncMemoryFirst, because their Raft logs are the durable record.

Cold tiers#

For stores larger than memory:

  • primary_tier: PrimaryTier::Cold keeps the primary index as of the last checkpoint in a page-structured file, and holds only later changes in memory, with a Bloom filter in front. A lookup that misses memory reads one page.
  • index_tier: IndexTier::Paged keeps secondary indexes as memory-mapped B+tree pages and bitmap blocks, reused at open after a clean shutdown.

Either can be switched in both directions; the next checkpoint converts the files. The server does not expose these tiers yet.

Encryption at rest#

Set encryption: Some(EncryptionConfig { keys, active_key, accept_unencrypted }). Records, the WAL and index files are sealed with ChaCha20-Poly1305 under the active key, bound to each record's id and metadata. File names and sizes, and WAL frame headers (ids, versions), stay visible.

To rotate keys while running, add a key to the ring, call set_encryption with the new active key, then reseal_all(), which reseals every record with it and scrubs superseded copies. The server does this for you at start: see encryption at rest.

Backups and point-in-time recovery#

FunctionDoes
create_online_backup(engine, dir)A consistent copy of a running store, with a manifest of the WAL position it covers
archive_wal_for_pitr(engine, backup_dir)Copies archived WAL into a backup. Needs wal_archive
restore_point_in_time_with(backup, logs, restore_dir, config, time_ns)Restores a backup and replays WAL files up to a moment, then opens the result

The steps are the same as for a server: see backups and recovery.