02 / Application design
Stage build: a bounded streaming library
Build a library that parses records as it reads them and limits buffer growth.
By Robert DeVore · Download Markdown
Build brief
Turn the first counter into a reusable library. It must accept any BufRead, preserve structured errors, and retain at most 4096 bytes for an unfinished record. Empty input succeeds. LF and CRLF records are accepted. A final record without a newline is accepted. Stop at the first malformed record and return an error, not a partial summary.
Run the example
cargo run --locked --example 12_stage_two
use std::io::Cursor;
fn main() -> Result<(), audit_core::AuditError> {
let summary = audit_core::summarize(Cursor::new("INFO started\nWARN retry\nERROR stopped\n"))?;
assert_eq!((summary.info, summary.warn, summary.error), (1, 1, 1));
println!("{summary:?}");
Ok(())
}
The example calls the completed library with an in-memory cursor. Read crates/audit-core/src/lib.rs alongside this lesson; it is the complete reference, not pseudocode.
Why the reader and parser are separate
parse(&str) handles syntax and returns a borrowed Event. summarize(impl BufRead) reads the input and adds a line number to parser errors. The parser does not know about files; the reader does not need to own a path. Tests can choose tiny buffer capacities to simulate fragmented input without relying on a real filesystem.
The reader calls fill_buf to inspect available bytes, finds the first newline if any, checks whether adding those bytes exceeds the record limit, copies the allowed portion, then calls consume. It must stop using the borrowed reader buffer before the next mutable reader operation. The borrow ends before the reader advances its buffer.
Check the limit before copying
Calling read_line into a String and checking its length afterward can already have allocated excessive memory. Our loop checks end > MAX_LINE_BYTES - bytes.len() before extending. The invariant is that bytes.len() never exceeds the limit, making the subtraction safe. The input reader may have its own buffer; the guarantee concerns the library's retained record storage, not all memory used by an arbitrary BufRead implementation.
UTF-8 can cross buffer boundaries. Decode only when you have a complete record or reach the end of the input. Reject invalid UTF-8 with a line number. The limit includes a newline when present, so the boundary is explicit and testable rather than a vague “roughly 4 KB”.
Exercise
Implement a simpler version first, then test reader capacities from 1 through 16. Include INFO café\n, a final unterminated record, a record exactly at the limit, and one byte beyond it. Inject an I/O failure. Ensure none of those errors produce a successful partial summary.
Solution and acceptance check
Use BufReader::with_capacity over a Cursor to force boundary splits. The UTF-8 input must parse regardless of where the two-byte character crosses a read. An exactly 4096-byte record is accepted; 4097 bytes are rejected before extension. A custom failing Read wrapped in BufReader preserves an AuditError::Io. The library's tests provide executable reference cases.
Review the API
Accepting BufRead lets the same function read files, stdin, and test buffers. The message borrow avoids allocation during parsing. Errors distinguish syntax from I/O. Three counters avoid a dynamic map. No trait framework, async runtime, or shared mutex is necessary for this stage.