go - Most efficient way to read Zlib compressed file in Golang? -

i'm reading in , @ same time parsing (decoding) file in custom format, compressed zlib. question how can efficiently uncompress , parse uncompressed content without growing slice? parse whilst reading reusable buffer.

this speed-sensitive application , i'd read in efficiently possible. ioutil.readall , loop again through data parse it. time i'd parse it's read, without having grow buffer read, maximum efficiency.

basically i'm thinking if can find buffer of perfect size can read this, parse it, , write on buffer again, parse that, etc. issue here zlib reader appears read arbitrary number of bytes each time read(b) called; not fill slice. because of don't know perfect buffer size be. i'm concerned might break of data wrote 2 chunks, making difficult parse because 1 uint64 split 2 reads , therefore not occur in same buffer read - or perhaps can never happen , it's read out in chunks of same size written?

1. what optimal buffer size, or there way calculate this?
2. if have written data zlib writer f.write(b []byte) possible same data split 2 reads when reading compressed data (meaning have have history during parsing), or come in same read?

you can wrap zlib reader in bufio reader, implement specialized reader on top rebuild chunks of data reading bufio reader until full chunk read. aware bufio.read calls read at most once on underlying reader, need call readbyte in loop. bufio take care of unpredictable size of data returned zlib reader you.

if not want implement specialized reader, can go bufio reader , read many bytes needed readbyte() fill given data type. optimal buffer size @ least size of largest data structure, whatever can shove memory.

if read directly zlib reader, there no guarantee data won't split between 2 reads.

another, maybe cleaner, solution implement writer data, use io.copy(your_writer, zlib_reader).