Crate recompress 

Offline tool that rewrites older archived Parquet files in the current archive format.

The archiver writes one Parquet file per channel and rotation window under its Parquet root. Files written before the archiver switched to zstd with delta-encoded timestamps (see archive_format::writer_properties) are much larger than they need to be. This tool walks an archive tree and rewrites those files in place without changing a single stored value. It does not talk to NATS.

Usage

recompress <parquet_root> [--dry-run]

• <parquet_root> - Directory to scan recursively. The first argument that does not start with -- is used. Required.
• --dry-run - Write and verify each rewritten copy, then delete it and leave the original in place. Reports the sizes a real run would produce.

Other -- arguments are ignored. No environment variables are read. The tool prints a progress line every 1000 files and a summary at the end, and exits with an error if any file failed.

Design

Each part-*.parquet file that is not already zstd with delta-encoded timestamps is rewritten next to itself as *.parquet.recompress-tmp, synced, read back and compared value by value (bit for bit, so NaN counts as equal only to the identical NaN) together with its key-value metadata. Only then is the original replaced by an atomic rename, and the directory is synced, so every file is at all times either the complete old version or the verified new one. A failed file keeps its original.

Unfinished (.inprogress) and quarantined files are never touched, and files with no row groups are counted as empty and left alone. The tool can be interrupted and rerun: finished files are skipped, and temporary copies left by an interrupted run are deleted on the next scan.

Modules

archive_format 🔒

On-disk format shared by the archiver and the recompress tool.

archive_io 🔒

Reading and writing whole archived sample files.

Enums

Outcome 🔒

Result of processing one file with recompress_file.

Constants

TMP_SUFFIX 🔒

Suffix appended to a file’s full name for its temporary rewritten copy.

Functions

collect_files 🔒

Lists archived part-*.parquet files and removes stale temporary copies.

is_current_format 🔒

Returns whether a file already uses the current archive format.

main 🔒

Parses the command line, recompresses every archived file under the root and prints a summary.

recompress_file 🔒

Rewrites one file in the current format after verifying the copy.

verify_identical 🔒

Checks that two archives hold exactly the same rows and metadata.