Skip to main content

recompress_file

recompress

Function recompress_file 

Source 
pub(crate) fn recompress_file(path: &Path, dry_run: bool) -> Result<Outcome>
Expand description

Rewrites one file in the current format after verifying the copy.

Steps, in order:

  1. If the file has no row groups, return Outcome::Empty. If it is already current, return Outcome::AlreadyCurrent. Neither touches the file.
  2. Read all rows and metadata into memory.
  3. Write them to <path>.recompress-tmp with write_archive (which syncs it).
  4. Reopen the copy, check it is in the current format, and compare it with the original using verify_identical.
  5. In a dry run, delete the copy. Otherwise rename it over the original and sync the parent directory so the rename is durable.

If any step from 3 to 4 fails, the temporary copy is deleted (best effort) and the original is left as it was.

§Arguments

  • path - Path of a finished part-*.parquet file.
  • dry_run - When true, verify the rewrite but keep the original file.

§Returns

Outcome::Rewritten with the before and after sizes and row count (also in a dry run), Outcome::AlreadyCurrent or Outcome::Empty.

§Errors

Returns an error if the file cannot be opened or read, if writing or verifying the copy fails, or if deleting the copy (dry run), renaming it or syncing the directory fails.