Historian Configuration
1. Check the QuestDB version New in 1.0.8
The historian requires QuestDB 9.4.0 or later. Upgrade QuestDB before you upgrade QUBIQ to 1.0.8.
- Designating a store older than 9.4.0 is refused. Settings → Historian lists QuestDB x.y.z is below the minimum supported version 9.4.0 among the reasons the store is not safe to adopt. Designate anyway overrides it, as it overrides every pre-flight finding, but charts that need 9.4 features will fail. Don't use it for this.
- A store that is already the historian keeps ingesting, so no data is lost. Every start logs a critical error naming the version until you upgrade.
- A build whose version cannot be read, such as QuestDB Enterprise numbering, is a warning, never a refusal.
2. Create a QuestDB connection
The historian needs a QuestDB instance with both of these reachable:
| Port | Protocol | Used for |
|---|---|---|
| 8812 | Postgres wire | Queries |
| 9000 | Line protocol (ILP) over HTTP 1.0.8 | Ingestion |
Over TCP, QuestDB cannot report a rejected row. It just closes the connection, so a row could be lost after it had left the store-and-forward buffer. Over HTTP it reports the row and retries. A rejected row is set aside in the quarantine and the rest of the batch is stored. New connections use HTTP on port 9000. A connection saved before 1.0.8 with no transport, on port 9009, keeps TCP and logs a warning on every connect. Open it, set ILP Transport to HTTP (recommended), check the port, and save.
3. Designate it as the historian
The designation lives in Settings → Historian, under Historian connection. Reading it needs View Connections; changing it needs Edit Connection.
Exactly one connection can hold the designation. Changing it opens a new epoch rather than truncating existing history.
Before accepting a store, QUBIQ pre-flights it. It checks the QuestDB version and looks for an
existing historian table and its shape, rather than issuing a CREATE TABLE IF NOT EXISTS that
would silently do nothing to a table it did not create. A blocking finding stops the designation
until you confirm Designate anyway (I understand the existing data will be taken over).
UDT members with history on are filled in with the designated historian when an instance is created or synced. 1.0.8 Up to 1.0.7 a type with a historised member could not be instantiated at all. → UDTs
4. Tune ingestion
On the QuestDB connection, under Ingestion Performance:
| Setting | Default | Effect |
|---|---|---|
| Deduplication | On | UPSERT KEYS(ts, tag_id). Leave it on: it makes replays idempotent. |
| Auto Flush Rows | 5000 | Rows buffered before a commit |
| Auto Flush Interval (ms) | 3000 | Longest wait before a commit |
| Parallel Workers | 1 | Ingestion threads (1–16) |
| Partition By | Day | Raw table partition unit |
| Rollup Partition | Hour | Partition unit of the 1-minute rollup |
| Rollup Refresh Every | 5m | How often the 1-minute rollup re-aggregates |
Deduplication matters more than it looks. Store-and-forward can replay a batch that was flushed but not acknowledged, after a crash in exactly that window. With dedup on, the replay overwrites identical rows; with it off, it duplicates them.
Throughput Changed in 1.0.8
Historised values travel to the buffer in batches (every 100 ms, or at 1,000 values), and the historian drains them with one unfiltered consumer. On a 4-core test machine also running the broker, QuestDB and a browser, 50,000 tags (25,000 historised, about 25,000 values a second) were stored as they arrived, about 0.2 s from reading to stored row. A buffered backlog drained at 13,000–19,000 rows a second. Up to 1.0.7 the same box fell minutes behind at 10,000 tags.
The batching has a cost: a value waits up to 100 ms in its producer's memory before it reaches the durable buffer, so a process crash in that window loses it.
5. Set each tag's storage rules
On the tag's History tab, or for a selection or a folder through the tree's Enable History… menu:
| Field | Effect |
|---|---|
| Enable History | Record this tag's values. Not available for DateTime and Array tags. |
| Storage Provider | Read-only: the designated historian |
| Deadband | Store a value only when it moved at least this much. A quality change is always stored. 0 stores every change. |
| Deadband unit 1.0.8 | Units (engineering units), or % of range of the tag's engineering range |
| Min time between samples (ms) 1.0.8 | The shortest time between two stored values; faster changes are not stored. Blank or 0 = no minimum. |
| Store interval (ms) | The longest time without a stored value. While the tag is Good and its producer runs, the current value is stored again after this long. Blank or 0 = off. |
| Interpolation 1.0.8 | Auto (by data type), Linear or Step: how stored values connect on trends and in time-weighted statistics |
Percent deadband New in 1.0.8
% of range reads the tag's engineering range, set on the Scaling tab; the range no longer
requires scaling to be on. The field shows what the percentage means: 1 % of 0 – 200 bar = 2 bar.
A percent band with no valid range resolves to 0, so every change is stored and nothing is
silently dropped.
An older build would read a percent band as engineering units: 2 % of 0–1000 bar would become 2 bar. A backup that contains a tag with a percent deadband is therefore refused by a build older than 1.0.8, with that reason. Backups without one are unaffected.
How the rules combine
For each reading, in this order:
- Minimum time between samples, judged on event time. Quality changes and readings that arrive out of order are never throttled.
- Deadband, within one quality.
- The reading before an exception is kept. New in 1.0.8 When a value holds inside the band and then steps, the last reading the band dropped is stored too, at its own time, beside the step. Without it, a hold-then-step drew as a slow ramp across the whole quiet spell.
- Store interval: a heartbeat row when nothing else was stored for that long.
A failed flush rolls the deadband and throttle state back with the batch, so a redelivered batch is judged exactly as it was the first time. 1.0.8 Up to 1.0.7 a redelivered row was compared with its own failed copy and dropped.
Be deliberate. For each tag ask:
- Will anyone trend this, or is the live value enough?
- What deadband makes the trend honest without storing noise? 0.5–1 % of range typically removes 90 % or more of the rows and changes no chart a human can read.
- Does a slow, steady signal need a store interval to prove it was alive?
Interpolation New in 1.0.8
| Tag type | Auto means |
|---|---|
| Real numbers | Linear |
| Integers | Settings → Historian → Integer Tag Interpolation: Linear (default) or Step |
| Booleans, strings, JSON | Step: the value holds until the next one |
A tag's own Linear or Step always wins. Charts use the historian's resolved interpolation, so changing the integer default reaches every viewer, LiveView operators included, on their next load, with no restart. Changing that setting needs Manage System Services.
Choose Step for integers that are states, modes or codes, where a ramp from 3 to 4 would draw values that never existed. Keep Linear for counters, ramps and scaled raw values.
6. Set retention Changed in 1.0.8
On the historian connection:
| Field | Applies to | Kept in |
|---|---|---|
| Data Retention (days) | Raw history | Whole partitions |
| 1-hour Rollup Retention (days) 1.0.8 | historian_1h | Whole months |
| 1-day Rollup Retention (days) 1.0.8 | historian_1d | Whole years |
0 or blank keeps history forever. Raw retention is at least 1 day. Each is applied as a QuestDB
TTL, counted from the newest sample, so a row stamped in the future cannot expire the table
early. The rollups keep their own history whatever the raw retention is.
Lowering any of them on the historian asks you to confirm first, because it deletes history. Setting retention for the first time on a store that already holds older data removes that data without asking.
On the QuestDB form, Data Retention (days) was never saved: whatever you typed was dropped, and the store kept its earlier setting. From 1.0.8 the field is saved and loaded. If you set it on an earlier release, open the connection, check the value and save it.
Sizing:
raw rows ≈ historised tags × stored rows per second × 86,400 × retention-days
Apply QuestDB's compression to that, then add headroom. If the number is uncomfortable, revisit storage rules before retention. A value you never stored costs nothing to keep.
The store-and-forward buffer
Between the collectors and QuestDB, history waits in its own store-and-forward buffer. The historian's buffer differs from the others in three ways 1.0.8:
- When full, it refuses new samples and keeps the backlog. Up to 1.0.7 it dropped the oldest, which is the stretch of an outage you most need.
- It has no age limit. An outage of any length is kept, bounded by size and by the data disk's free-space floor. An alarm warns when the oldest buffered history nears the raw retention.
- Its default size is 25 % of the data volume (at least 2 GiB, at most 1 TiB), instead of a fixed 2 GiB.
Live first, available on historian buffers only, lets new samples overtake a draining backlog, so trends show the present while the past fills in. → Store and forward
Repairing a store
Everything below is in Settings → Historian and needs Edit Connection.
Schema drift
QuestDB auto-creates a table when an ILP line names one that does not exist, with a schema
QUBIQ did not choose. Missing columns, deduplication or the tag_id index are repaired in place
automatically.
A table whose designated timestamp is named timestamp instead of ts cannot be: QuestDB refuses to
rename a designated timestamp column, and every read fails with Invalid column: ts. The historian
logs an error saying the table must be rebuilt. The rebuild copies the rows into a correctly shaped
table and stops ingestion while it runs. This release has no button for it, so ask your QUBIQ
support contact to run it.
Rollup rebuild Changed in 1.0.8
Rebuild rollup (historian_1m) re-aggregates the 1-minute rollup from the raw table, and rebuilds any 1-hour or 1-day tier that QuestDB has marked invalid. Tiers that are valid are left alone.
A rebuild re-aggregates from raw. Once retention has expired raw rows that a rollup still holds, a rebuild deletes that older rollup history. So:
- an invalid rollup is rebuilt automatically only when that loses nothing. When the loss cannot be measured, it waits and tries again later.
- otherwise the historian raises Historian rollup needs a rebuild (High) and waits. Press the button when you have accepted the loss and can accept the load.
- a rollup whose definition is out of date after an upgrade stays as it is, still serving, unless the rebuild is both lossless and small (raw table under 10 million rows).
Suspended tables
When QuestDB's write-ahead-log apply job fails, it suspends the table: rows land in the WAL, nothing is applied, and charts stop at that point while ingestion looks healthy. The store-and-forward buffer keeps holding new history meanwhile.
| Cause | What happens |
|---|---|
| A resource problem: disk full, memory, file limits | A Historian table suspended alarm (Critical). Free the resource and the table resumes by itself; it cannot be skipped. |
| Anything else (a corrupt or missing WAL segment, say) | One plain resume. If that fails, a Historian table suspended alarm (Critical) names the table, the stuck transaction and QuestDB's error. |
Up to 1.0.7 the supervisor resumed a poisoned table past the stuck transaction, up to 50 times an episode. Every skip destroyed history the buffer had already handed over and could not resend. In 1.0.8 it never skips on its own. Settings → Historian shows each suspended table with its error. Skip the stuck transactions… resumes it and loses that history for good, behind a confirmation that says so. The skip is audited as a deletion. Do it only when the cause cannot be repaired.
The alarm clears by itself once the table resumes, including after a restart. Look at QuestDB's own log for the cause.
Ownership conflict
The ownership banner in Settings → Historian means another installation claims this store. While it does, history is still written, but schema changes, rollup maintenance and retention are off. Confirm the previous owner is retired, then press Take ownership of this store. This is audited. Do not do it to silence the banner on a store another live system is writing to.
Migrating the historian
- Create and test the new QuestDB connection (9.4.0 or later).
- Change Historian connection in Settings → Historian.
- QUBIQ closes the old epoch and opens a new one; History timeline lists them.
- Charts spanning the changeover query both stores and stitch the timeline.
- Retire the old store only when its epoch is no longer inside any window you need to read.
There is no export or import step, and trends are not truncated.
Query concurrency
Settings → Historian → Historian — Query Concurrency sets Max parallel queries the historian
runs against QuestDB (default 16, 1–256). Saving it restarts QBQ-Historian. It needs
Manage System Services.
Alarms and monitoring
| Alarm or signal | Priority | Means |
|---|---|---|
| Historian table suspended 1.0.8 | Critical | QuestDB stopped applying writes to a historian table → Suspended tables |
| Historian rollup needs a rebuild 1.0.8 | High | A rollup tier is invalid and an automatic rebuild would lose history → Rollup rebuild |
| Device clock skew 1.0.8 | Medium | A connection's device clock is more than 2 minutes off → Clock skew |
| Store-and-forward alarms | Medium to Critical | Buffer filling, full, frozen at the disk floor, or holding history near retention → Store and forward |
| QuestDB version below 9.4.0 | System log, critical | → Version |
| Schema cannot be repaired | System log, error | → Schema drift |
Buffer status, fill, drain rate and time-to-full are in Settings → Store & Forward (needs View Connections).
Tuning by environment
Rarely needed; set in .env and restart. The full list, including the rollup and cache switches,
is in Environment variables.
| Variable | Default | Controls |
|---|---|---|
HISTORIAN_CARRY_MAX_HOLD | 8h | How long a chart carries an on-change value without a newer row |
HISTORIAN_CLOCK_SKEW_ALARM | 2m | Device clock-skew alarm threshold; 0 = off |
QUBIQ_HISTORIAN_HEARTBEAT_MAX_SILENCE | 24h | Heartbeats stop after this long without a real reading |
HISTORIAN_TWA_RAW_MAX_DAYS | 7 | Longer ranges compute time-weighted statistics from the 1-hour tier |
HISTORIAN_ROLLUP_MIN_WINDOW_SEC | 43200 (12 h) | Smallest chart window, in seconds, served from the rollups; shorter windows read raw |
HISTORIAN_ROLLUP_REBUILD_MAX_ROWS | 10000000 | Raw-table rows above which an out-of-date rollup is not rebuilt automatically at start-up; 0 removes the guard |
Troubleshooting
| Symptom | Cause | Action |
|---|---|---|
| No data for a tag | History not enabled on the tag | Enable it |
| No data at all | No designated historian, or ILP unreachable | Check the designation and the ILP port (9000 for HTTP, 9009 for TCP) |
| Designation refused, below the minimum supported version | QuestDB older than 9.4.0 | Upgrade QuestDB |
Invalid column: ts | Schema drift with the wrong designated timestamp | The table must be rebuilt → Schema drift |
| Charts stop, ingestion looks fine | Table suspended | Historian table suspended alarm; check QuestDB's log |
| A stretch shaded Bad, Shutdown or Collector down | Nothing was collected then | Expected → Outage markers |
| A device's history lands minutes off | Its clock is wrong | Device clock skew alarm; fix the device's clock (NTP) |
| Ownership banner | Another installation claims the store | Verify, then take ownership |
| Disk filling faster than expected | No deadband, or history on chattering tags | Set storage rules |
| Long-range charts slow | Rollup behind or invalid | Check for the rollup alarm; rebuild it |