--------------------------------------------------------------------------- The plan - There're multiple levels of dependencies actually. 0. nlen,ndist,ncode - sizes of tables in [1] and [2] 1. secondary length table: (up to) 19 random 3-bit codes 2. primary length table: compressed using huffman+RLE, with codes based on [1] 3. literals/matches, until EOF symbol 4. uncompressed data So we need to parse [4] into [3], then build [2] based on [3] (while RLE makes it complicated - with RLE+huffman, RLE doesn't necessarily improve the compression), also compute ndist and ncode based on [3]. Then build [1] based on [2], and compute nlen - at least here there's only one way to do it optimally... although then again, there're multiple equivalent huffman codes. Plan: 1. Reconstruct nlen+secondary-len-table based on primary-len-table. 2. Reconstruct ndist+ncode+primary-len-table based on block LZ tokens. 3. Reconstruct block's .rec structure - Make an hc3 matchfinder with up to 64k window - Do greedy/lazy parsing - If necessary, refactor zlib encoding (likely the most reasonable) 4. Decoding/encoding using a separate file for metainfo 5. Interleaving the metainfo. - Interleaving with a common rangecoder - Interleaving with signatures in uncompressed data - Simple interleaving in compressed data (with rc flushes) (ie plzma compressed data interleaved with separately compressed metainfo) 6. deflate block detection 7. interleaving of deflate/non-deflate data (Uncompressed deflate data has to be the same stream with outside data, with additional out-of-band deflate metainfo) Questions: 1. How to use plzma's rangecoder (in other thread!) to also encode (and especially decode) the metainfo? Only psrc? 2. Is it really necessary to detect non-byte-aligned deflate blocks? 3. How do we know where to put deflate blocks at decoding? > First there goes some indefinite block of "outside data", then we have to stop and flush the main codec (so EOF has to be stored in the stream), then read the metainfo and do some blockwise processing (decode a block with plzma, encode with deflate, read more metainfo) 4. Does it really have to be a stream? (deflate+plzma) 5. How to sync in cases like when there's direct output in a function, and also a multi-thread chain with output to the same buffer? > Instead of EOF, make it a special state where plzma thread flushes its rc, waits for some event, then pushes a block from some buffer into output? Sounds kinda too specific for plzma. But still its not a good idea to let the lzmarec thread to fully quit each time... and then again, afaik lzma backend is delayed compared to frontend, so its necessary to also flush the frontend somehow