Storine v0.3.0

Operations

Monitoring

The health report, Prometheus metrics and slow-request log, what each field means, and what to alert on.

Health#

storage-cli remote-health <addr> prints a node's health report as JSON. Any role may read it. A standalone server:

{"status":"ok","live_records":0,"index_capacity":3584,"total_reads":0,"total_writes":0,"hot_dirty_bytes":0,"arena":{"segments":1,"used":0,"dead":0,"live":0,"compactions":0,"removed":0,"errors":0,"last_error":""},"encryption":"off"}
FieldMeaning
statusok while the node serves
live_recordsDocuments stored
index_capacityDocuments the index holds before it next grows. reserve and STORAGE_EXPECTED_DOCUMENTS raise it
total_reads, total_writesSince the server started
hot_dirty_bytesWritten but not yet flushed to the record files. Stays low unless the disk falls behind
arena.segments, used, dead, liveSegment files and their bytes. dead is space that compaction will reclaim
arena.compactions, removed, errors, last_errorCompactions run, segments they removed, and failures
encryption"off", or the active key, the keys held, and accept_plaintext
wal_archiveThe archive folder, when STORAGE_WAL_ARCHIVE is set

A range node reports instead mode: "ranges", ranges, log_entries, log_bytes, queue and batch sizes, dropped_batches, and:

FieldMeaning
placementEach range the node holds: its span, its voters, and whether this node leads it
placement_rulesWith --zones: the rules file, and the error that kept its last change from applying, or null

Metrics#

storage-cli remote-metrics <addr> prints Prometheus text format, on standalone and single-group nodes:

MetricTypeMeaning
storine_live_recordsgaugeDocuments stored
storine_total_reads, storine_total_writescounterOperations since start
storine_coalesced_flushescounterWrites to one key merged before reaching disk
storine_hot_dirty_bytesgaugeUnflushed bytes
storine_arena_segmentsgaugeSegment files open
storine_arena_used_bytes, storine_arena_dead_bytesgaugeSegment bytes written, and dead
storine_arena_compactions, storine_arena_compacted_segments, storine_arena_compaction_errorscounterCompactions, segments they removed, failures
storine_read_latency_ns, storine_write_latency_nssummaryQuantiles 0.5, 0.99 and 0.999
storine_slow_rpcsgaugeRequests over 10 ms held in the slow-request log, which keeps the last 256

The server has no HTTP endpoint. To feed Prometheus, run the scrape from a small exporter script or a textfile collector, for example every 15 seconds:

storage-cli remote-metrics 127.0.0.1:7878 > /var/lib/node_exporter/storine.prom.tmp && \
  mv /var/lib/node_exporter/storine.prom.tmp /var/lib/node_exporter/storine.prom

Clusters#

storage-cli cluster <addr> members prints the members, their roles, the leader, the term and the commit index. The console's overview shows each node's commit and applied indexes live.

What to alert on#

SignalWhy
remote-health fails or status is not okThe node is down or unreachable
No leader, or the term rising quicklyA cluster that keeps re-electing has a network or overload problem
A follower's applied index falling behind the commit indexThe node is slow or partitioned
hot_dirty_bytes near 64 MiBThe disk cannot keep up; writers will start to flush themselves
arena.errors or storine_arena_compaction_errors risingCompaction is failing; disk space will not be reclaimed
arena.dead large and growingCompaction has not run; see storage
Disk free spaceThe WAL, segments and archive all need room
p99 write latency risingDisk or replication slowdown
placement_rules holds an errorA rules change was refused

Logs#

The server prints only its start line and messages that need attention, such as a reseal report or a failure to prune archived WAL. It has no log file: run it under systemd or another supervisor that keeps its output. The SQL gateway logs every statement with STORINE_SQL_LOG=1.