Crate dedupe 

Offline tool that removes duplicate samples from archived channel folders.

When the archiver is fed the same JetStream messages again, it writes the same samples a second time into new part files. Folders written that way hold two or more copies of every sample, sometimes split at different window boundaries, and the exporter returns every copy. This tool rewrites each affected channel folder so that every distinct sample is stored once, in one file per aligned rotation window, the same layout the archiver writes today. It does not talk to NATS.

Usage

dedupe <parquet_root> [--dry-run] [--rotate-secs N]

• <parquet_root> - Archive root holding asset<NNN>/<YYYY-MM-DD>/ch<NN>/ folders. The first argument that does not start with -- is used. Required.
• --dry-run - Build and verify each new folder, then delete it and keep the original. Reports what a real run would change.
• --rotate-secs N - Window length in seconds (default 300). Must match the archiver’s rotate_secs.

Design

A sample is a (timestamp_unix_ns, value) pair plus the file’s key-value metadata. Two samples are duplicates only when all three match exactly, with values compared by their bit patterns. Samples with the same timestamp but different values are all kept and counted as conflicts.

One exception crosses metadata: when the same sample is stored both under an identity (or missing) calibration and under a real one, only the calibrated copy is kept. An identity calibration means no sensor had been configured yet, and the raw value is the same in both copies, so the identity copy carries no extra information. A sample stored only under identity is kept.

Each channel folder is handled on its own:

1. Every part file is read once to find its metadata and which windows its rows fall in.
2. Window by window, the rows of every file touching that window are merged, sorted, and exact duplicates dropped. The result is written to a staging folder ch<NN>.dedupe-tmp next to the original, one file per window (and per distinct metadata, if a folder mixes calibrations). Each file is read back and compared with what was meant to be written.
3. An independent check then reads every original file again and confirms each of its rows is present in the staged file for its window.
4. The original folder is renamed to ch<NN>.dedupe-old, the staging folder takes its name, the day folder is synced, and the old folder is deleted.

The exporter only reads ch<NN> folders, so it never sees the staging or old folders. Folders for today and yesterday (UTC) are skipped because the archiver may still be writing to them. Folders holding anything other than finished part files (such as .inprogress or quarantined files) are skipped, as are folders that have no duplicate rows and already hold one file per window. A rerun resumes safely: a leftover staging folder is deleted, and a leftover old folder is either deleted (swap finished) or renamed back (swap interrupted).

Modules

archive_format 🔒

On-disk format shared by the archiver and the recompress tool.

archive_io 🔒

Reading and writing whole archived sample files.

Structs

FolderStats 🔒

Counts for one channel folder.

PartCache 🔒

Loads part files on demand and drops them once no later window needs them.

PartInfo 🔒

What was learned about one original part file in the first pass.

Enums

Outcome 🔒

Result of processing one channel folder.

Constants

OLD_SUFFIX 🔒

Suffix the original folder is renamed to while the staging folder takes its place.

STAGING_SUFFIX 🔒

Suffix of the staging folder that holds the rewritten files before the swap.

Functions

channel_folders 🔒

Lists asset<NNN>/<YYYY-MM-DD>/ch<NN> folders under root older than cutoff.

dedupe_folder 🔒

Deduplicates one channel folder.

drop_superseded 🔒

Removes identity-calibration rows whose sample is also kept under a real calibration.

is_clean 🔒

Returns whether a folder is already clean: one file per window, no empty files, and no file holding the same row twice.

is_identity 🔒

Returns whether the metadata has no real calibration.

main 🔒

Parses the command line, deduplicates every eligible channel folder and prints a summary.

metadata_key 🔒

Returns the metadata as comparable (key, value) pairs, in order.

read_part 🔒

Reads a whole part file.

recover 🔒

Restores a channel folder left half-done by an interrupted run.

rows_in_window 🔒

Returns the rows of archive that fall in window, tagged with group.

sort_unique 🔒

Sorts rows and removes exact duplicates.

suffixed 🔒

Returns path with suffix appended to its last component.

sync_dir 🔒

Syncs a directory so renames and new entries in it are durable.

window_of 🔒

Returns the rotation window a timestamp falls in.

Type Aliases

Row 🔒

One sample, keyed by metadata group so different calibrations never merge.