Command-line tools
request-nats-export.mjs
Makes one export request through central NATS, exactly as the webapp does, and
saves the CSV. Use it to script downloads or to test the export path without a
browser. It uses the webapp’s NATS libraries, so run pnpm install in
webapp/ once first.
node scripts/request-nats-export.mjs \
--subject avenars.i69.i69-mu1.i69-lj2.export.request \
--asset 1001 --channels 8,9,10,11 \
--start 2026-09-22T12:00:00Z --end 2026-09-22T12:30:00Z \
--output out/mu1.csv --creds apt.creds
| Option | Required | Meaning |
|---|---|---|
--subject | yes | The box’s export request subject |
--asset | yes | Asset number |
--channels | yes | Comma-separated channel numbers |
--start, --end | yes | RFC 3339 times, inclusive |
--output | yes | CSV file to write; its folder is created if needed |
--servers | no | Comma-separated NATS URLs. Default nats://nats1.oats:4222,nats://nats2.oats:4222 |
--creds | no | Credentials file. Default /etc/avena-rs/apt.creds |
--report | no | Also write the JSON summary to this file |
--timeout-seconds | no | Give up after this long. Default 120; raise it for large exports |
It prints a JSON report with the bytes, chunks and rows received and any channels with no data.
recompress
Rewrites archive files written before September 2026 in the current format (zstd, delta timestamps, one row group). Values do not change, and each file is checked before it replaces the original.
cd rust-ljm
cargo build --release --bin recompress
./target/release/recompress <parquet_root> --dry-run # report only
./target/release/recompress <parquet_root>
For each part-*.parquet file under the root, it writes a compressed copy next
to it, syncs it, reads it back and compares every timestamp and value bit for
bit, along with the metadata. Only then does it rename the copy over the
original. At any moment every file is either the complete original or the
verified copy, so exports keep working during a run, and a run can be stopped
and started again. It skips files that are already compressed, files with no
rows, unfinished .inprogress files and quarantined files.
It prints progress every 1,000 files and a summary at the end, and exits with
an error if any file failed; a failed file keeps its original. Run it with
nice -n 19 on a box that is recording.
dedupe
Removes duplicate samples from the archive. Before the archiver had its replay
guard, JetStream sometimes fed it the whole stream again, and it wrote the same
windows again into new part files (up to twelve copies on MU1). dedupe
rewrites each affected channel folder so every sample is stored once, in one
file per aligned window (and per calibration, if a folder mixes them), the
layout the archiver writes today. The exporter already skips exact copies, so
this mostly saves disk space and brings each window back to one file.
cd rust-ljm
cargo build --release --bin dedupe
./target/release/dedupe <parquet_root> --dry-run # report only
./target/release/dedupe <parquet_root>
| Option | Default | Meaning |
|---|---|---|
--dry-run | off | Build and check every new folder, then delete it and keep the original |
--rotate-secs N | 300 | Window length; must match the configuration’s rotate_secs |
Two samples are duplicates only when the timestamp, the value (bit for bit) and the file’s metadata all match. Samples with the same timestamp but different values are all kept and reported as conflicts. One rule crosses metadata: when the same sample is stored both under an identity (or missing) calibration and under a real one, only the calibrated copy is kept, because an identity calibration only means the channel had not been configured yet. A sample stored only under identity is kept.
For each folder it writes the new files to ch<NN>.dedupe-tmp, reads them
back, checks that every row of every original file is present, then swaps the
folders (the original becomes ch<NN>.dedupe-old and is deleted after the
swap). It skips folders for today and yesterday (UTC), which the archiver may
still be writing, folders holding .inprogress or quarantined files, and
folders that are already clean. A run can be stopped and started again: a
leftover staging folder is deleted and an interrupted swap is finished or
rolled back. It prints one line per rewritten folder and a summary, and exits
with an error if any folder failed; a failed folder is left as it was.
subscriber
Captures live samples to CSV without the archiver, for checking a stream by
eye. It subscribes to the live wildcard for one source and appends one file
per channel, labjack_<asset>_<channel>.csv, with the columns
sequence,timestamp,raw_value. It only sees messages published while it runs.
cd rust-ljm
NATS_SUBJECT=avenars SITE_ID=i69 BOX_ID=i69-mu1 SOURCE_ID=i69-lj2 \
ASSET_NUMBER=1001 OUTPUT_DIR=/tmp/capture NATS_CREDS_FILE=apt.creds \
cargo run --release --bin subscriber
Set SITE_ID, BOX_ID and SOURCE_ID. With none of them it falls back to the
legacy wildcard, which matches every asset and names each file after the asset
in its subject.
render-edge-config.py
Turns a box profile into the box’s generated files.
./shared/render-edge-config.py --config shared/edge-boxes/<box>.json \
--output-dir target/edge-config/<box>
| Option | Default | Meaning |
|---|---|---|
--config | shared/edge-box.config.json | The profile to render |
--output-dir | none | Write the files here. Without it, the files are written into shared/ and rust-ljm/ in the repository. |
--repo-root | . | Repository root that relative paths are resolved against |
install-edge-services.sh
Renders a profile, builds the release binaries, installs them with their configuration and credentials, and enables the systemd units. Run it on the edge node.
./scripts/install-edge-services.sh --profile shared/edge-boxes/<box>.json [--start]
It also installs wait-for-local-nats, the 10-nats-ready.conf drop-in for
each Rust service, the health-metrics timer and Alloy’s container unit. It does
not install nats-leaf.conf; see Everyday
tasks.
--start restarts the services and Alloy after installing. Without it the
units are enabled but left as they were. It expects the client credentials in
rust-ljm/apt.creds.
wait-for-local-nats.sh
Waits until the local NATS server reports JetStream healthy
(http://127.0.0.1:8222/healthz?js-enabled-only=true), polling once a second
for up to 600 seconds. The installer puts it in
/usr/local/libexec/avena-rs/wait-for-local-nats and runs it before each Rust
service through the 10-nats-ready.conf drop-ins (see Services on an edge
node).
edge-status.sh
Prints the state of every unit, clock synchronization, the local NATS server
and its leaf connection, the newest LabJack and camera files, disk space and
failed units. Exit status 0 means every check passed. On a box it is installed
as /home/user/avena-status.
LabJack examples
Small programs in rust-ljm/examples for checking the LabJack without the
services. They read LABJACK_IP (and optionally LABJACK_SERIAL) from the
environment, or from the file named by CONFIG_FILE, or from a
streamer.env.json in the current folder.
| Example | What it does |
|---|---|
info | Opens the device and prints its type, address and connection |
read | Polls the analog inputs and prints their values |
stream | Runs an LJM stream and prints the samples, without NATS |
stop | Stops a stream left running by a crashed process |
cd rust-ljm
LABJACK_IP=192.168.1.111 cargo run --example info
info works while the streamer is running. read and stream do not: the
LabJack runs one stream at a time and refuses analog reads while it streams, so
stop avena-streamer first. stop ends whatever stream is running, including
the streamer’s, so only use it when the streamer is already stopped.