Store and Forward
When a destination goes away — the historian's QuestDB, a logging database, an MQTT broker — QUBIQ does not drop the data and does not block the producer. It buffers to durable storage and replays in order once the destination returns.
What is covered
Store-and-forward protects outgoing data writes. Each kind has its own family of buffers:
| Buffer family | What it holds |
|---|---|
| Historian | Every value on its way into the historian's QuestDB |
| Pipeline QuestDB writes | QuestDB writes made by pipeline nodes |
| Pipeline SQL writes · Pipeline MongoDB writes | Database writes made by pipeline nodes |
| Script SQL writes · Script MongoDB writes | Named-query and script writes to a database |
| Alarm journal | Alarm journal rows on their way to the alarm store |
| MQTT broker egress | Values a pipeline publishes to an external broker |
The audit journal has a durable buffer of its own, held until the Audit service writes it. It is shown beside the others 1.0.8 — see The audit journal's buffer.
It does not apply to reads, nor to control writes to devices — a tag write to a PLC that fails must fail visibly, not be replayed minutes later against a process that has moved on. Outbound REST requests are not buffered either: a REST call that fails, fails.
How it works
- One durable, file-backed buffer per connection. Buffers are independent, so a failing database cannot stall the historian.
- A single forwarder drains each buffer in batches: fetch, write, flush, then acknowledge. Acknowledging only after a confirmed flush is what makes a crash mid-drain replay rather than lose data.
- A latch enforces ordering. Once any write for a connection has been buffered, every subsequent write for it is buffered too — until the backlog fully drains. Without this, a live write taken on the fast path overtakes older buffered ones and the destination ends up out of order. Each connection has its own latch 1.0.8, so one full buffer no longer slows the writes of every other connection.
- A failed batch is retried in place. 1.0.8 The forwarder keeps the batch it could not write and retries those same items ahead of anything newer, so strict order holds through any number of failures. Earlier builds handed the batch back to the buffer, and newer items could occasionally be written before it.
- Failures are classified. A connection/transport failure is transient: retried indefinitely, head-of-line, and never counted toward the quarantine — an outage must not dead-letter good data. A query or data error is permanent: replaying it would only fail again or duplicate, so the item goes to quarantine.
A plain timeout is not treated as a connection failure, because a timed-out write may in fact have committed. Treating it as transient and replaying it would duplicate rows.
Replays are not written twice
A buffered database write carries an identity that the destination records in a small ledger table
(_sf_dedup) when the write is applied, so a replay of something already applied is skipped.
On a MongoDB replica set (or a mongos) 1.0.8, the identity and the write
commit in one transaction — both or neither — so a crash at any moment can neither lose the write
nor apply it twice. A standalone MongoDB server cannot run transactions and keeps the ordered path:
never lost, but a crash between the write and recording it can replay that one write once.
Durability on disk
The buffers live in the bus's file store, which reaches the operating system at once but is forced
to disk (fsync) every 5 seconds 1.0.8. A process crash loses
nothing; a power cut can lose at most the last few seconds of buffered data. Earlier builds used the
underlying server's default of two minutes. QUBIQ_JETSTREAM_SYNC sets a tighter interval
(environment variables) — always fsyncs every
write and, on ordinary disks, slows buffered writes to roughly ten a second.
The store is kept in the data directory (nats_data/), with the rest of the plant data, not
beside the program. 1.0.8 See
Where the buffers live.
Where you see it
Settings → Store & Forward shows every buffer (needs View Connections):
| Column | Shows |
|---|---|
| Connection · Type | Whose buffer, and which family |
| State | Idle, Buffering, Full, Quarantined — or Paused / Purging while an operator action is in force |
| Buffered | Items waiting. Hover for the rate in and out per second 1.0.8 |
| Oldest | Age of the oldest waiting item |
| Used 1.0.8 | Share of the buffer's size limit in use; amber from 80 % |
| Fills in 1.0.8 | Time until full at the current growth — shown only while the buffer is growing, so a draining one does not keep warning |
| Refused 1.0.8 | Values a full Refuse new buffer has turned away |
| Quarantined | Writes awaiting an operator |
A summary line above the table leads with the most serious condition: the disk floor, then quarantined writes, a full buffer, a paused buffer, and finally ordinary draining.
Buffer limits New in 1.0.8
The Buffer limits section sets, per buffer family, the Size (MB), the Age limit (h) —
0 means no age limit — and When full:
| When full | Behaviour |
|---|---|
| Refuse new (keep backlog) | Everything already buffered is kept; new values are refused until the buffer drains. |
| Drop oldest | The oldest buffered values are discarded to make room. |
Only the historian, pipeline QuestDB writes and MQTT broker egress offer the choice. Database and alarm-journal writes always refuse new: dropping a buffered write would lose it and break the write order. Saving needs Edit Connection.
Save limits applies them to every buffer that already exists, at once, whichever service created it. One exception protects data: a Drop oldest buffer is never shrunk below what it currently holds, because shrinking it would delete its oldest items on the spot. The smaller size waits until that buffer drains, and the section lists the buffers being held back.
Only values you change are stored; a family you leave alone keeps following the built-in default. Sizes run from 16 MB to 1 TiB, and age limits to one year.
Built-in defaults
| Family | Size | Age limit | When full |
|---|---|---|---|
| Historian | 25 % of the data volume (whole GiB, at least 2 GiB, at most 1 TiB) | none | Refuse new |
| Pipeline QuestDB writes | 512 MB | 72 h | Drop oldest |
| Pipeline SQL / MongoDB writes | 512 MB | none | Refuse new |
| Script SQL / MongoDB writes | 512 MB | none | Refuse new |
| Alarm journal | 512 MB | none | Refuse new |
| MQTT broker egress | 256 MB | 24 h | Drop oldest |
- It no longer discards its oldest history when full. It refuses new samples and keeps the backlog, because the start of an outage is the stretch an operator most needs. Choose Drop oldest in Buffer limits to restore the old behaviour.
- Its default size follows the disk: 25 % of the volume holding the data directory, instead of a fixed 2 GiB. On a 2 TB server that is 500 GiB of buffer.
- It has no age limit. An outage of any length is kept, bounded by size and the free-disk floor. You are alarmed instead when the oldest buffered history approaches the historian's own retention, since samples older than retention are deleted as soon as they are written.
Existing buffers pick all three up on the first sweep after the upgrade. HISTORIAN_SF_MAX_BYTES,
HISTORIAN_SF_MAX_AGE_HOURS and HISTORIAN_SF_FULL_POLICY in .env are still honoured, and a value
saved in Buffer limits overrides them.
The free-disk floor New in 1.0.8
The buffers share the data volume with the configuration database. When free space on that volume drops below the floor — 5 % of the volume, at least 1 GiB and at most 20 GiB — every buffer is frozen at what it holds: new values are refused, and nothing already buffered is deleted. The freeze lifts once free space is back above 1.25 × the floor, so a disk hovering at the line does not flap. Saving buffer limits while frozen does not unfreeze them.
A full disk would stop the whole gateway, the configuration database included; the floor trades new buffered values for keeping the gateway running.
Alarms New in 1.0.8
Store-and-forward trouble raises system alarms in the ordinary active-alarm list, so it is seen even when nobody has Settings open:
| Condition | Alarm | Priority |
|---|---|---|
| Data disk below the free-disk floor | Data disk low — store-and-forward frozen | Critical |
| A buffer at its size limit (98 % and above) | Store-and-forward buffer full — says whether new values are refused or the oldest discarded | High |
| A buffer at 80 % of its size limit | Store-and-forward buffer filling | Medium |
| Writes in quarantine | Store-and-forward writes quarantined | Medium |
| Oldest buffered history older than 80 % of the historian's retention | Buffered history nearing retention | Medium |
Each alarm clears itself once its condition is gone, including after a restart.
Operator actions on a buffer New in 1.0.8
Each row's ⋯ menu (needs Edit Connection) holds:
| Item | Does |
|---|---|
| Pause forwarding / Resume forwarding | Holds the buffer: it keeps accepting and keeps everything, but delivers nothing until resumed. For maintenance on the destination. |
| Forwarding rate | Opens a popover with Max forwarding rate (items/s) — blank is no cap — and, on a historian buffer, the Live first switch. |
| Purge | Permanently discards what the buffer holds now, after a confirmation that says how many items will never arrive. Values arriving afterwards are kept. Disabled on an empty buffer. |
The ⋯ is highlighted while a rate cap or live first is in force, and its tooltip says which. Every one of these actions is recorded in the audit journal.
Forwarding rate
A destination that has just come back can be flattened by hours of backlog arriving at once. The cap paces the forwarder to at most that many items a second. It applies to live values too, so set it above the normal rate. It takes effect without a restart.
Replay itself is batched 1.0.8: the forwarder fetches up to 100 items at a time and still writes them one at a time, in order, so a database buffer drains faster without giving up ordering.
Live first
After an outage the historian's backlog drains oldest-first, so a new sample waits behind hours of old ones and trends show nothing current until the drain finishes. Live first lets new samples overtake: while the drain is more than 30 seconds behind, new samples are written straight away and the backlog fills in behind them. Rows keep their own timestamps.
It is offered only on historian buffers and is off by default. Reordering database, alarm or MQTT writes could corrupt the destination, so the server refuses it for any other buffer.
Which recent rows live first has already stored is held in memory. If the gateway restarts mid-drain, some of those rows may be written a second time. Deduplication on the QuestDB connection absorbs them.
Quarantine (dead-letter queue)
An item that fails permanently is moved to quarantine rather than being retried forever or silently dropped. Quarantine is an operator queue, not a graveyard, and nothing in it expires on its own 1.0.8 — earlier builds deleted a quarantined write three days after it failed. When the quarantine itself is full, a new failure stays at the head of its buffer instead of evicting an older quarantined write, and the buffer's fill alarms take over.
Historian samples that QuestDB rejects are quarantined too 1.0.8; earlier builds discarded them with a log line.
| Action | Permission |
|---|---|
| List quarantined items | View Connections |
| View the whole write (eye icon) — the full payload, as text, or base64 for binary 1.0.8 | Edit Connection |
| Retry — put items back on their buffer | Edit Connection |
| Drop — discard permanently | Edit Connection |
| Export every quarantined write to a JSON file 1.0.8 | Edit Connection |
| Import such a file 1.0.8 | Manage Backup & Restore (super admin) |
An imported file's writes go back into the quarantine, never straight into a buffer, to be reviewed and retried or dropped there. Each keeps the time it was first quarantined, and importing the same file twice adds nothing. Import needs Manage Backup & Restore rather than Edit Connection because an imported write is executable — Retry runs it, and a script SQL write carries its own statement. Export and import are both recorded in the audit journal.
A retried historian sample goes back only into that historian's buffer. It is not republished as live telemetry, so alarms, expressions and screens never mistake an old value for a current one.
Investigate before retrying. A quarantined batch usually means a schema mismatch, a constraint violation or a credential change — retrying without fixing the cause just re-quarantines it.
The audit journal's buffer New in 1.0.8
Audit events from every service wait in a durable buffer until the Audit service writes them to the audit database. It appears in the table as Audit Journal, with the same fill, age, rate and time-to-full readouts and the same full and filling alarms as the other buffers.
It has no ⋯ menu and no entry in Buffer limits: purging it would destroy audit records, and the Audit service owns its configuration. → Audit journal
Liveness gating
Forwarders consult the connection monitor's state. While a connection is known to be down, the forwarder idles instead of hammering it with retries. An unknown connection is treated as up — the sink's own error is the authority, not a stale assumption.
This is why connection health matters operationally: it is not only an indicator, it throttles retry traffic.
Historian specifics
The historian's sink adapts its QuestDB writer to the same forwarder, so history inherits ordered, ack-after-flush, no-drop semantics. Two extra behaviours are historian-only:
- A live tail buffer holds the most recent numeric samples in memory and is merged into live chart queries, closing the sub-second lag between ingestion and QuestDB's flush. Charts look live even though storage is batched.
- A write-ahead-log supervisor watches for QuestDB tables suspended by a failed WAL apply. It tries one plain resume and otherwise raises a critical alarm, leaving any decision to skip transactions to an operator 1.0.8 — skipping one deletes the history it held, and the buffer has already handed that history over. → Suspended tables
Where the buffers live
Every buffer, the quarantine and the runtime state the services share are kept in one store,
nats_data/ in the data directory — on Windows under ProgramData, which an upgrade or an
uninstall leaves alone.
Earlier builds kept this store beside the program (<install folder>\nats_data), so a reinstall into
another folder, or a cleaned install folder, silently lost unsent history and quarantined writes. On
its first start, 1.0.8 moves the store into the data directory. If a copy already exists there,
the one beside the program wins — it is the one the running installation has been writing — and the
other is set aside as nats_data.stale-<number>, never deleted. If the move fails, the gateway
keeps using the old location and logs why.
The first start can take longer than usual while a large store is moved, and after an unclean stop
the bus rebuilds its indexes before anything else starts. The supervisor waits for that — up to 15
minutes by default (QUBIQ_NATS_READY_TIMEOUT) — and logs its progress every 10 seconds, instead of
giving up after 10 seconds as earlier builds did.
What this means operationally
- A short outage is invisible. Data lands late, in order, complete.
- A long outage is visible early — through the filling and retention alarms — while there is still time to act.
- A bad payload is isolated, in quarantine, instead of blocking every good write behind it, and stays there until somebody decides.
- Nothing is silently lost. Every drop is an explicit operator action (Purge, Drop), a Drop oldest buffer at its limit, or an age limit you set — and each of those raises an alarm or is recorded in the audit journal.
Next
→ Connections · Historian