Skip to main content

spawn_csv

Function spawn_csv 

Source
pub(crate) fn spawn_csv(
    job: CsvJob,
    chunk_size: usize,
) -> (Receiver<Vec<u8>>, JoinHandle<Result<Vec<u8>>>)
Expand description

Starts building an export’s CSV on a blocking thread.

The CSV thread formats the rows that a reader thread (read_rows) reads one merge slice ahead, so decoding the next file overlaps formatting the current one, also across channels. The channels come one after another, in the given order, and the CSV is cut into chunks: the first starts with the header line, and every chunk except the last is at least chunk_size bytes, exceeding it by less than one row. The chunks are sent in order on the returned receiver, up to CSV_CHUNKS_AHEAD ahead of the caller, so building the next chunks overlaps sending the current one and waiting for acks, and none of it blocks the async runtime. Memory stays bounded by two merge slices (one being formatted, one read ahead) and, besides the chunk being sent, CSV_CHUNKS_AHEAD ready chunks and the one being built. The caller stops both threads by dropping the receiver: each ends at its next hand-over, the reader after the slice it is reading.

§Returns

The chunk receiver, and the CSV thread’s handle, which yields the channels that produced no rows in the range, in request order, once every chunk is sent. On the first error from listing a channel’s day folder, the handle yields it, prefixed with channel NN:, after the chunks completed before it.