1. Optimization of a "real" state machine for counters (not freq-based) (see http://nishi.dreamhosters.com/u/fsm-213203-depth13.txt ) Actually I have an interesting approach for initial SM approximation (very redundant), but not much progress with state number optimization - trying to merge the best possible pairs of states is "a little" slow, and there're potentially other nested loops (like choice of transitions from the merged state). Also an interesting possibility is fsm with randomized transitions, but it adds even more parameters for optimization. 2. Design of a fsm mixer - mix2 to be specific. A mixer should have two counters and its own state as 3 inputs, which is clearly too much for a LUT, which means that we have to add some kind of a reduction step - c1,c2 - counter states, w1 - mixer state a) (perfect, but impractical) c = T1[w1][c1][c2] w1 = T2[w1][c1][c2][bit] c1 = U1[c1][bit] c2 = U2[c2][bit] b) (merge counters) c12 = T0[c1][c2] c = T1[w1][c12] w1 = T2[w1][c12][bit] // 6+8+1=32k is barely ok, probably need even less c) (partial mixing, then fixed merge) d1 = T1[w1][c1] // can be stretch + multiplication + quantization d2 = T2[w1][c2] c = T3[d1][d2] w1 = T4[d1][d2][bit] // 128k with normal fsm counters; would be slow 3. Further experiments with logistic mixers + optimal parsing For now I only had some positive results in http://www.ctxmodel.net/files/mix_test/mix_test_v4.rar ("likelihood" coder), where weight estimation is based on comparison of smoothed codelengths. This topic is actually all questions - the specifics of the switching scheme for parsing, how to apply changes after reparsing of recent history, whether its possible to do "update exclusion" by parsing results, etc. Actually I suspect that after adding optimal parsing to CM the usual mixer update methods might become obsolete... 4. Structures for storage of statistics - isn't there some way to improve usual hashtables? For example, we can use http://en.wikipedia.org/wiki/Bloom_filter to filter out the low-count contexts - how to complete the design of the tree with bitmask compression. (Or maybe there's a better idea about data structures w/o hashing? or a better idea to speed up symbol lookups in tree nodes?) 5. Intermediate compression - it'd be really cool to use entropy coding as much as possible in the internal data structures - as hash functions or anything. a) we still don't really have a list of possible applications (hashing, dictionary storage, optimization of radix sorting,...) b) the main issue is data non-stationarity. One workaround is a completely static order4 or so CM model tuned to compress any normal compressible data (see "msb"). Another is low-order blockwise CM, but there're problems: remapping of stats and inter-block string comparisons. 6. psrc (Probability Sorting + Rangecoding; parallel rangecoding method) - How to resolve the quantization asymmetry without branches? - How to reduce the number of cache misses on bit lookups in decoder? - Apply probability symmetry or not? - Efficient sort implementation? - Efficient model for sorted bits? (secondary model) - How to handle "direct bits" (ones with p=~0.5)? - Is there a different method for parallel entropy coding, separate from model? 7. Stream interleaving issues in complex recompression schemes For example, deflate recompressor would have the following parts: - stream detector (encodes the stream start and detected encoder options) - huffman header model (encodes block headers + huffman trees in context of block's LZ tokens) - token model (encodes LZ tokens in context of uncompressed data) - postcoder (main codec that encodes non-stream + extracted stream data) The trick here is that the encoding and decoding orders of various rc streams would differ considerably, so we can't use independent rcs in all modules (well, without adding noticeable redundancy due to flushes).