*** complogger has joined the channel2009-11-24 10:53:31
<toffer> yes i saw it2009-11-24 11:06:16
 but that's nothing new2009-11-24 11:06:49
<Shelwien> well, after that i also messed with my extrapolation formula a little2009-11-24 11:07:14
<toffer> and?2009-11-24 11:07:35
<Shelwien> and after using n*(1-wr)+wr for update instead of n+12009-11-24 11:07:41
 i've got p^wr/(p^wr+(1-p)^wr) for extrapolation2009-11-24 11:08:01
<toffer> you'd better place some bars on top of your pc2009-11-24 11:08:11
<Shelwien> and that's exactly what i added to paq mixer too2009-11-24 11:08:28
<toffer> n*(1-wr)+wr = exponential aging?2009-11-24 11:08:54
<Shelwien> yeah, somewhat2009-11-24 11:10:40
 anyway, that p^wr mapping looks right on the plot too2009-11-24 11:11:09
 with this, maybe i'll have all the necessary components now2009-11-24 11:11:54
 rolling freqs, counter, mixer, and SSE2009-11-24 11:12:30
<toffer> that is supposed ot be a counter probability estimation via liklihood?2009-11-24 11:13:09
 p^wr/(p^wr+(1-p)^wr)2009-11-24 11:13:24
*** Shelwien has left the channel2009-11-24 11:13:30
*** Guest9968193 has joined the channel2009-11-24 11:13:35
<Shelwien> yeah2009-11-24 11:13:56
<toffer> and p is?2009-11-24 11:14:08
<Shelwien> initial probability estimation2009-11-24 11:14:21
<toffer> i mean how do you derive it?2009-11-24 11:14:22
 ok2009-11-24 11:14:24
 so basically the primary estimate, via linear coutners i guess?2009-11-24 11:14:38
<Shelwien> and with that it becomes a posterior probability approximation2009-11-24 11:14:44
 yeah, but with freqs2009-11-24 11:15:01
<toffer> well, liklihoods2009-11-24 11:15:08
 how does p^wr/(p^wr+(1-p)^wr) look alike using stretch/squash?2009-11-24 11:15:29
<Shelwien> i mean, i'd like to make it all freq-based this time2009-11-24 11:15:31
<toffer> sq(wr*st(p)) ?2009-11-24 11:15:38
<Shelwien> counter, mixer, and sse2009-11-24 11:15:39
 yeah ;)2009-11-24 11:15:46
 that's what i said - i added it to my version of paq mixer2009-11-24 11:16:01
<toffer> now you still need a practical evaluation2009-11-24 11:16:27
<Shelwien> afair there was no such thing there originally, at least not explicit2009-11-24 11:16:29
<toffer> yes, you're right there2009-11-24 11:16:45
 but it can be implicitly contained within mixibg2009-11-24 11:17:03
 mixing2009-11-24 11:17:05
 sq(w0*s0 + ... = sq(wr*(w0'*s0 + ...))2009-11-24 11:17:28
<Shelwien> yeah, but that's not very compatible with optimizing2009-11-24 11:17:28
 also maybe i'd finally build my fsm with this2009-11-24 11:18:07
 if you remember, that thing with lossy compression2009-11-24 11:18:35
<toffer> yeah2009-11-24 11:18:41
<Shelwien> this extrapolation is exactly what it lacked before ;)2009-11-24 11:19:08
 bit histories there would be fairly short2009-11-24 11:19:34
 so that part becomes very important2009-11-24 11:19:46
<toffer> i'd appreciate results with "real" data2009-11-24 11:24:09
 e.g. book12009-11-24 11:24:18
<Shelwien> well, book1.bwt maybe2009-11-24 11:24:35
<toffer> so it's just a direct model for context histories?2009-11-24 11:26:01
 why not integrate it into a cm compressor?2009-11-24 11:26:17
*** Shelwien has left the channel2009-11-24 11:26:27
*** Guest9968193 has joined the channel2009-11-24 11:26:30
 reboot again?2009-11-24 11:26:44
<Guest9968193> no, irc server disconnects somehow2009-11-24 11:26:57
<toffer> too quick i guess2009-11-24 11:26:58
 it must be the illuminates2009-11-24 11:27:06
<Guest9968193> sure2009-11-24 11:27:11
<toffer> ^^2009-11-24 11:27:18
<Shelwien> i don't have a good coder template for direct model2009-11-24 11:27:25
 and i don't like to do it with hashes2009-11-24 11:27:31
<toffer> you could use something like the data structure of dmc2009-11-24 11:27:50
 well with some path compression2009-11-24 11:27:58
<Shelwien> ideally it has to be sliding window + bytewise tree + unary model + bitwise entropy model2009-11-24 11:28:19
<toffer> or use the model on multi symbol alphabets2009-11-24 11:28:23
<Shelwien> but i didn't quite solve it neither with the tree, nor with unary model2009-11-24 11:28:59
<toffer> unary model... reminds me of some older source where i started to write a unary coder2009-11-24 11:29:02
 trees reuqire some work compared to hashes. especially when doing them "nicely" or efficient2009-11-24 11:29:31
<Shelwien> unary model is most problematic with bitwise models2009-11-24 11:29:48
 Shkarin's approach with mtf ranking is good enough, but only works with plain freqs2009-11-24 11:30:00
<toffer> erm?2009-11-24 11:30:11
 i thought he actually used frequency sorting2009-11-24 11:30:19
 (or prob. sorting)2009-11-24 11:30:25
 for decomposition2009-11-24 11:30:34
<Shelwien> he does that sometimes too - on rescales etc2009-11-24 11:30:48
 but its more of mtf most of the time2009-11-24 11:31:08
<toffer> well one can optimize a decomposition model which minimizes the number of guesses ^^2009-11-24 11:31:33
 reminds me of guessing game2009-11-24 11:31:38
<Shelwien> anyway, its very hard to reorder the symbols with bitwise statistics2009-11-24 11:31:43
<toffer> i mean shannon's2009-11-24 11:31:47
<Shelwien> sure, that's what i'm talking about basically2009-11-24 11:32:18
 but its impossible to implement with bitwise counters etc2009-11-24 11:32:29
<toffer> in my eyes bitwise coding is pretty attractive (i mean using flat deco.) since you don't need escapes2009-11-24 11:32:33
 on the other hand you cannot mask symbols2009-11-24 11:32:50
<Shelwien> its relatively ok until we keep the order static2009-11-24 11:32:53
<toffer> so both cm and ppm coding have some redundancy2009-11-24 11:33:18
 order?2009-11-24 11:33:36
<Shelwien> guessing order or whatever2009-11-24 11:33:45
 if its freqs like in ppmd, we can just swap them and all stats remain precise2009-11-24 11:34:10
 but swapping two unary probabilities... its very messy2009-11-24 11:35:00
 division etc2009-11-24 11:35:10
 and imprecise even while very slow2009-11-24 11:35:26
<toffer> why'd you need to swap unary probs?2009-11-24 11:35:52
<Shelwien> because order of appearance or whatever else2009-11-24 11:36:17
 won't be efficient forever2009-11-24 11:36:24
<toffer> i thought these unary probs would be constructed adaptivly2009-11-24 11:36:56
<Shelwien> that's the problem2009-11-24 11:37:09
 if its a:Na b:Nb ...2009-11-24 11:37:32
 then probs are Na/(Na+Nb+...) etc2009-11-24 11:37:41
<toffer> and where's the problem?2009-11-24 11:37:59
<Shelwien> and we can just swap the symbols in the order2009-11-24 11:38:02
 but, for example, if instead of Na2009-11-24 11:38:17
 we'd use a linear counter there2009-11-24 11:38:24
<toffer> ok now i now what you mean2009-11-24 11:38:31
<Shelwien> it would be a:Pa b:Pb2009-11-24 11:38:32
 and would require some crazy renormalization to swap2009-11-24 11:38:50
<toffer> but that's not just related to probs - it depends on how symbols are processed2009-11-24 11:39:32
<Shelwien> ?2009-11-24 11:39:49
<toffer> that implies you process multi-symbol alphabets2009-11-24 11:40:40
 i mean not decomposing the symbol as a basis2009-11-24 11:41:11
 (like flat deco. in cm does)2009-11-24 11:41:31
<Shelwien> text compression would never be completely precise2009-11-24 11:41:39
 unless we start estimating atleast the byte probabilities2009-11-24 11:41:58
<toffer> well, yes2009-11-24 11:42:07
 but as i said ppm has redundancy in escapes2009-11-24 11:42:14
<Shelwien> not really2009-11-24 11:42:29
 ppm is redundant by whole idea2009-11-24 11:42:35
<toffer> ?2009-11-24 11:42:44
<Shelwien> coding with one context2009-11-24 11:42:50
 well, its a cool speed optimization in the end2009-11-24 11:43:07
<toffer> nah - that's a question of probability estimation2009-11-24 11:43:14
<Shelwien> basically, the thing Shkarin tried to do in ppmonstr2009-11-24 11:43:20
<toffer> i mean the fact that it requires to encode artificial symbols2009-11-24 11:43:29
<Shelwien> is implementing a cached mixing of some kind2009-11-24 11:43:39
<toffer> bitwise cm has its redundancy in the unavailability of masking2009-11-24 11:43:56
 well afair ppmii is static mixing - as you said2009-11-24 11:44:18
<Shelwien> there's no redundancy in coding escapes2009-11-24 11:44:27
<toffer> why?2009-11-24 11:44:34
<Shelwien> with exclusions2009-11-24 11:44:34
 the only question is to properly compute the stats2009-11-24 11:44:51
<toffer> normally one wouldn't require to encode symbols not actually present in the message. thus inserting these symbols adds redundancy2009-11-24 11:45:07
<Shelwien> if you encode escape from a context2009-11-24 11:45:21
 and means that symbols present in that context2009-11-24 11:45:41
<toffer> i'm not talking about probabilty estimation of escapes, etc. but the fact that ppm needs to code artificial symbols2009-11-24 11:45:50
<Shelwien> would be masked out and unavailable for encoding anymore2009-11-24 11:45:54
<toffer> i know2009-11-24 11:46:04
 but that's still artificial data to code2009-11-24 11:46:12
 added by the algorithm2009-11-24 11:46:18
 not present in the data2009-11-24 11:46:21
<Shelwien> escapes are no more artificial that some msbs in bitwise coding2009-11-24 11:46:29
 they're used to select a symbol set2009-11-24 11:46:50
<toffer> i didn't say that cm has no additional redundancy2009-11-24 11:46:52
 i just said that ppm has2009-11-24 11:47:00
<Shelwien> well, i still don't agree that its a redundancy2009-11-24 11:47:18
 it only makes it harder to properly compute the probabilities in some cases2009-11-24 11:47:46
 while bitwise CM has a real redundancy2009-11-24 11:48:04
<toffer> i think it's simply bias of a certain viewpoint2009-11-24 11:48:26
<Shelwien> because it has to assign probabilities to unused symbols2009-11-24 11:48:45
<toffer> lz requires to code e.g. flags to distingush between a (non)-match2009-11-24 11:48:53
 that's redundancy generated by the algorithm, too2009-11-24 11:49:01
<Shelwien> well, LZ is not a compression algorithm at all2009-11-24 11:49:13
 its an ad hoc transform of data into a match sequence2009-11-24 11:49:32
 redundant of course2009-11-24 11:49:47
<toffer> if the resulting sequence is shorter than it's compression2009-11-24 11:49:49
 anyway2009-11-24 11:49:51
<Shelwien> its not always shorter2009-11-24 11:50:06
<toffer> that was just an example of redundancy added by a compression algorithm2009-11-24 11:50:07
<Shelwien> and its unknown whether it would be shorter2009-11-24 11:50:15
 sure2009-11-24 11:50:21
<toffer> compression is bijective, so no algorithm is guaranteed to compress everything2009-11-24 11:50:36
<Shelwien> and as I said, bitwise coding has some redundancy too2009-11-24 11:50:37
 due to precision limits2009-11-24 11:50:45
 while PPM doesn't actually have any2009-11-24 11:50:57
<toffer> and i never said that it doesn't contain redundancy - i just sad ppm has2009-11-24 11:51:02
<Shelwien> as there it all depends on implementation2009-11-24 11:51:05
 well, but bytewise PPM is worse for direct compression of binaries2009-11-24 11:51:38
 and as to bijectivity2009-11-24 11:52:38
 i mean, there's no certain way to know that LZ transform would allow to compress anything2009-11-24 11:53:05
 there're only some very approximate metrics2009-11-24 11:53:18
 while redundancy is much easier to limit with PPM/CM2009-11-24 11:53:58
<toffer> erm wasn't there a proof somewhere that lz is equivalent to statistical compression in the infinite case?!2009-11-24 11:53:59
<Shelwien> sure, but who cares about that2009-11-24 11:54:25
 we don't have enough memory to assign a symbol to every encountered substring2009-11-24 11:54:58
<toffer> conclusion about statistical algorithms thus can be ported2009-11-24 11:54:59
<Shelwien> not really, that's not practical2009-11-24 11:55:20
 lz optimizers still have to use ad hoc metrics and multiple passes2009-11-24 11:55:41
<toffer> despite not beeing practical it's still possible2009-11-24 11:56:22
 anyway - it's nice to have some more time for talks like that ^^2009-11-24 11:56:37
*** Shelwien has left the channel2009-11-24 11:57:06
*** Guest9968193 has joined the channel2009-11-24 11:57:09
 and some more coffee 2009-11-24 11:57:14
 these discs are really annoying2009-11-24 11:57:28
<Shelwien> yeah, dunno wtf. i said this2009-11-24 11:57:52
 <Shelwien> not really, that's not practical2009-11-24 11:57:57
 <Shelwien> lz optimizers still have to use ad hoc metrics and multiple passes2009-11-24 11:57:57
 <Shelwien> ...2009-11-24 11:57:57
 <Shelwien> also, masking with bitwise coding is still possible2009-11-24 11:57:57
<toffer> toffer> despite not beeing practical it's still possible2009-11-24 11:59:37
  anyway - it's nice to have some more time for talks like that ^^2009-11-24 11:59:39
<Shelwien> err... so you would have it now? ;)2009-11-24 11:59:57
 not working or anything? ;)2009-11-24 12:00:14
<toffer> well i got at least 6 weeks of vacation now2009-11-24 12:00:57
 afterwards i gonna work at the fraunhofer institute here2009-11-24 12:01:19
<Shelwien> cool2009-11-24 12:01:24
<toffer> and find a project for phd2009-11-24 12:01:27
<Shelwien> maybe you'd be able to motivate me enough to actually write something ;)2009-11-24 12:01:38
 for now i'm mainly working on that conexware backup2009-11-24 12:02:05
<toffer> the thing which is really cool about that is that when i'm 26 i'd have my phd. most people finishing their studies are at lest 25-26 here 2009-11-24 12:02:11
<Shelwien> yeah, i guess2009-11-24 12:02:39
 unfortunately phd or anything doesn't really mean anything here atm2009-11-24 12:03:10
<toffer> i got around army - since i was in hospital durint draft. and in east germany there're 12 years of school compared to 13 in west germany2009-11-24 12:03:41
 why?2009-11-24 12:03:50
<Shelwien> no good jobs i guess2009-11-24 12:04:13
<toffer> ah2009-11-24 12:04:16
<Shelwien> these phd guys trade on the flea market here2009-11-24 12:04:57
<toffer> >.<2009-11-24 12:05:19
 i'd leave the country than2009-11-24 12:05:27
 having a phd raises your salary2009-11-24 12:06:02
<Shelwien> well, on other hand, that's what they're worth too2009-11-24 12:06:09
<toffer> and provides more possibilities2009-11-24 12:06:12
<Shelwien> because smart people already left, yeah2009-11-24 12:06:21
<toffer> alltogetehr that's really weird2009-11-24 12:07:13
*** Shelwien has left the channel2009-11-24 12:08:01
*** Guest9968193 has joined the channel2009-11-24 12:08:05
<Shelwien> well, anyway, are there any modelling jobs? ;)2009-11-24 12:09:57
<toffer> yes2009-11-24 12:11:57
<Shelwien> and does it really look like nobody knows about CM there? ;)2009-11-24 12:12:28
<toffer> yep2009-11-24 12:13:29
<Shelwien> huh. i guess you have a bright future then ;)2009-11-24 12:13:55
<toffer> ^^2009-11-24 12:16:05
*** Shelwien has left the channel2009-11-24 12:17:22
*** Guest9968193 has joined the channel2009-11-24 12:17:26
<Shelwien> damn, that's on my side i guess2009-11-24 12:17:53
 luckily i have chanlog here ;)2009-11-24 12:18:06
<toffer> sorry, had to phone with my family2009-11-24 12:18:31
 anyway that cm principle is mostly unknown2009-11-24 12:18:43
 or present in an ad hoc manner2009-11-24 12:18:51
<Shelwien> yeah, thing is, i discussed it with some academic people here too2009-11-24 12:19:07
<toffer> not taking a proper optimization of the whole forecast system into accoutn2009-11-24 12:19:08
<Shelwien> same thing2009-11-24 12:19:09
<toffer> well i'd characterize cm in the following way:2009-11-24 12:19:42
<Shelwien> err, not only optimization2009-11-24 12:19:43
<toffer> - combine several predictions adaptively2009-11-24 12:20:01
 - have specialized, adaptive, independent modeling components2009-11-24 12:20:29
 - separate the data via discrete states (contexts)2009-11-24 12:20:52
<Shelwien> well, for me the main point is the shannon metric2009-11-24 12:21:29
<toffer> but a fuzzy system for example can, under certain assumptations, be interpreted as a cm system, too2009-11-24 12:21:34
<Shelwien> that's one thing which they tend to always forget2009-11-24 12:21:40
 yeah, that's what i mean too2009-11-24 12:22:10
 and another quirk is caring too much about precision and theory2009-11-24 12:22:47
<toffer> well i'd think dropping theory would be helpful2009-11-24 12:23:46
<Shelwien> somehow it looks like they never believe that the modelled data can be imprecise or partly random2009-11-24 12:23:48
<toffer> mh2009-11-24 12:24:05
<Shelwien> not, dropping theory is wrong too2009-11-24 12:24:14
<toffer> i'd not agree - there actual models for random processes2009-11-24 12:24:34
<Shelwien> but its important to have a right image in mind2009-11-24 12:24:36
 i mean, what we do is really simple math-wise2009-11-24 12:25:33
<toffer> something i often encouter while looking at some papers2009-11-24 12:25:35
<Shelwien> like linear interpolation etc2009-11-24 12:25:52
<toffer> was that just assumptations were made without analysis2009-11-24 12:25:55
 yes2009-11-24 12:25:57
 that's simple indeed2009-11-24 12:26:01
<Shelwien> well, i mean, they try to overcomplicate things right away2009-11-24 12:26:26
 to make it look smart ;)2009-11-24 12:26:31
<toffer> depends2009-11-24 12:26:36
 i guess it looks like that if i'd post something on the forum sometimes2009-11-24 12:26:50
 but to some degree that is just more compact and simpler2009-11-24 12:27:07
 math wise2009-11-24 12:27:09
<Shelwien> err... your posts are just hard to understand most of the time2009-11-24 12:27:44
 because you probably just type them and send as is ;)2009-11-24 12:27:53
<toffer> ?!2009-11-24 12:28:10
<Shelwien> formatting etc ;)2009-11-24 12:28:28
<toffer> i'm used to write tex2009-11-24 12:28:40
* Shelwien wonders if vbulletin has a tex plugin2009-11-24 12:29:03
* Shelwien thinks they should for their $2002009-11-24 12:29:11
 ^^2009-11-24 12:29:24
 that'd be nice2009-11-24 12:29:28
 well but i guess me and probably you would be the only guys using it2009-11-24 12:29:40
 or osman maybe, too2009-11-24 12:29:49
<Shelwien> shkarin was the only one who actually used it until now ;)2009-11-24 12:29:59
<toffer> haven't seen him for a while2009-11-24 12:30:02
<Shelwien> so i had to render it and post instead of him ;)2009-11-24 12:30:12
<toffer> when writing stuff properly i mostly usei t2009-11-24 12:30:28
 it2009-11-24 12:30:30
<Shelwien> osman? he appeared here2009-11-24 12:30:43
 had a talk with him about molecular biology ;)2009-11-24 12:30:54
 apparently he tries to write a good dna model2009-11-24 12:31:08
<toffer> interesting, too2009-11-24 12:31:17
 but i don't got any knowledge there2009-11-24 12:31:27
<Shelwien> yeah, he found some interesting molecular dependencies there2009-11-24 12:31:36
<toffer> i could hardly help2009-11-24 12:31:38
 and i guess understanding everything would take some effort2009-11-24 12:31:57
 but i remember to read something about linear functions2009-11-24 12:32:10
<Shelwien> and came here asking how to approximate a data set with lines ;)2009-11-24 12:32:16
<toffer> that's what i mean 2009-11-24 12:32:40
<Shelwien> yeah2009-11-24 12:32:42
 apparently cumulative phases there are linear2009-11-24 12:32:58
<toffer> phases of what?2009-11-24 12:33:27
<Shelwien> dna triplets are mapped onto some complex numbers2009-11-24 12:33:29
 and their phases summed2009-11-24 12:33:33
<toffer> ah2009-11-24 12:33:33
 ok2009-11-24 12:33:35
 didn't know about that2009-11-24 12:33:44
<Shelwien> and somehow there're long smooth lines2009-11-24 12:33:52
 like millions of nucleotids2009-11-24 12:34:02
<toffer> at least i could help there - since it's a quite easy prediction task2009-11-24 12:34:17
<Shelwien> but that's fairly hard to use in a model imho2009-11-24 12:34:24
<toffer> (or modeling)2009-11-24 12:34:27
<Shelwien> as its only linear on average2009-11-24 12:34:40
<toffer> well if you can decompose the modeled entity into several subentities2009-11-24 12:34:50
<Shelwien> it doesn't block any given symbol from appearing at any place2009-11-24 12:34:58
 afair osman was going to use it as context though2009-11-24 12:35:21
<toffer> you can use probability densities and the decomposition to merge the densities of the subentities back2009-11-24 12:35:24
 guess to completely understand i'd need to have a look at all of that2009-11-24 12:36:08
<Shelwien> well, he didn't post any code2009-11-24 12:36:22
<toffer> but i didn't know that dna triplets are mapped onto the complex plane2009-11-24 12:36:35
<Shelwien> !grep nucleotid2009-11-24 12:36:42
 hmm, that was a month ago i guess2009-11-24 12:37:03
 he does read the forum though2009-11-24 12:37:18
<toffer> exactly a month ago2009-11-24 12:37:19
 looks like the channel still isn't that busy2009-11-24 12:39:24
 what about inviting some people like malcom tylor?2009-11-24 12:39:33
 or charles bloom2009-11-24 12:39:41
 i accidently read some blog entries from bloom2009-11-24 12:40:07
<Shelwien> won't work with bloom i think2009-11-24 12:40:11
<toffer> why?2009-11-24 12:40:17
<Shelwien> he was cool though2009-11-24 12:40:18
 well, the same reason why he doesn't write compressors anymore ;)2009-11-24 12:40:46
 some people like taylor, or shkarin, or mahoney could be nice though2009-11-24 12:41:30
<toffer> i still don't know why2009-11-24 12:41:49
 is he dead or something?!2009-11-24 12:41:57
<Shelwien> but in fact, i don't really know what to discuss with them2009-11-24 12:42:21
 bloom? no, alive2009-11-24 12:42:31
 just has a life or something2009-11-24 12:42:34
<toffer> i do too2009-11-24 12:42:40
<Shelwien> and he's a physicist anyway, not a programmer2009-11-24 12:42:45
<toffer> but still i'm messing with all of that2009-11-24 12:42:49
<Shelwien> well, he did too, for quite some time2009-11-24 12:43:03
<toffer> tylor seems to be a very weird person2009-11-24 12:43:25
<Shelwien> too paranoid, like many2009-11-24 12:44:09
 but talks at least2009-11-24 12:44:13
 but the problem, as i said, is that i don't know what to talk to them atm2009-11-24 12:45:03
 i don't really have any questions anymore2009-11-24 12:45:18
<toffer> just some basic stuff can produce new ideas, etc2009-11-24 12:45:29
 more like interchanging opinions2009-11-24 12:45:47
<Shelwien> at least, such that i can go asking about without thinking2009-11-24 12:45:47
 yeah, but they're too old for that or something2009-11-24 12:46:22
 and some are also too paranoid2009-11-24 12:46:29
 anyway, for example, Matt seems only interested in some philosophic questions ;)2009-11-24 12:47:14
 well, whatever2009-11-24 12:48:22
 they'd probably appear sometimes2009-11-24 12:48:28
 i just have to decompile something - that's easy ;)2009-11-24 12:48:40
<toffer> ^^2009-11-24 12:48:44
 anyway that ccm decompilation was a nice toss2009-11-24 12:49:08
<Shelwien> yeah, i even implemented the parallel rc in that wave2009-11-24 12:49:36
 btw, instead of making a direct CM compressor2009-11-24 12:50:43
 i'm thinking more about hutter prize thing2009-11-24 12:51:02
 if you remember, there was my multipass idea2009-11-24 12:51:18
 kinda an extended unary coding2009-11-24 12:51:24
 where we write down a set of replace regexps2009-11-24 12:51:46
 and encode the information lost in replacements, to make them reversible2009-11-24 12:52:09
 so, its still a CM, of course2009-11-24 12:52:43
 but we can encode whole words or anything2009-11-24 12:53:07
 using bidirectional contexts2009-11-24 12:53:18
 and of any complexity too2009-11-24 12:53:22
 including stuff like distances to other words etc2009-11-24 12:53:34
 doesn't it sound interesting?2009-11-24 12:54:12
<toffer> not yet2009-11-24 12:54:20
 ^^2009-11-24 12:54:22
 at least at the moment2009-11-24 12:54:28
 i'm mostly thinking about some m1 fixups and extensions2009-11-24 12:54:46
<Shelwien> yeah, somebody should finally beat ccm at least ;)2009-11-24 12:55:19
 there's also a lot to do about rangecoding in fact2009-11-24 12:55:59
<toffer> somehow i decided to re-use a paq style vectorizable mixer. since with too may models the sse chains get too large resulting in a slowdown2009-11-24 12:56:06
<Shelwien> %)2009-11-24 12:56:20
<toffer> i mean2009-11-24 12:56:24
 i found out that having a 0.5 or 1 k stretch table and a 4k squash table is sufficient2009-11-24 12:56:40
 precision wise2009-11-24 12:56:50
 thus alltogether i'd need 9 or 10k of auxilary memory2009-11-24 12:57:20
<Shelwien> however, in my case (mix_Test) it didn't visibly affect the speed2009-11-24 12:57:26
<toffer> but a single sse chain requires ~ 80x80x2 bytes ~ 12k2009-11-24 12:57:44
<Shelwien> when i increased the tables to 64k from 12-bit ones2009-11-24 12:57:45
 well, caches are big etc ;)2009-11-24 12:58:01
<toffer> the most important thing for caching is the context models2009-11-24 12:58:21
 having more than ~ 4 sse maps results in a slowdown2009-11-24 12:58:34
 m1x2 with a match model already used 42009-11-24 12:58:43
 and that paq mixing can be vectorized2009-11-24 12:58:55
 and i wonder why noone ever tried to vectorize counter updates2009-11-24 12:59:06
 i'm just writing an inline asm piece for that2009-11-24 12:59:29
<Shelwien> i tried to explain that to "encode"2009-11-24 12:59:50
 he even has these dual counters2009-11-24 13:00:00
 really easy to vectorize2009-11-24 13:00:06
<toffer> ^^2009-11-24 13:00:09
<Shelwien> well, result is expected ;)2009-11-24 13:00:23
<toffer> maybe rewriting as a loop is enough to let the compiler do the work2009-11-24 13:00:23
<Shelwien> no, its not that easy2009-11-24 13:00:42
<toffer> i tried to rewrite things as loops and looked at the vectorizer output2009-11-24 13:00:42
 but mostly things weren't vectorized2009-11-24 13:00:55
 so i need to write that on my own2009-11-24 13:01:01
<Shelwien> you have to pay attention to types and pointers2009-11-24 13:01:03
<toffer> i know2009-11-24 13:01:12
 but there's ftree-vectorizer-verbose=# in gcc2009-11-24 13:01:24
<Shelwien> types = SSE2 has short ints, but not bytes or words somehow2009-11-24 13:01:32
<toffer> for me mmx is sufficient2009-11-24 13:01:45
<Shelwien> well, intelc doesn't support mmx anymore ;)2009-11-24 13:02:00
<toffer> sse seems to be mostly related to floats2009-11-24 13:02:02
<Shelwien> its good for ints too2009-11-24 13:02:12
 twice the vector size2009-11-24 13:02:31
<toffer> yep 128 bit2009-11-24 13:02:38
 but in fact on a 64 bit cpu even "manual" vectorization can be possible2009-11-24 13:02:55
<Shelwien> don't forget that vectorized mixing in ccm ;)2009-11-24 13:03:23
<toffer> well in m1 there's something like that too. several things packed into a uin322009-11-24 13:04:02
 uint322009-11-24 13:04:06
<Shelwien> btw, i still want to make that probability mapping model for jpeg2009-11-24 13:06:09
 i mean, completely unpack it into pixels, and models these pixels with CM2009-11-24 13:06:33
 and then transform the pixel p.d's into DCT coef p.d's2009-11-24 13:07:06
 should be nice compression-wise2009-11-24 13:07:22
 but i really wonder about an efficient implementation for that transform2009-11-24 13:07:39
<toffer> well i got no hint. what is the pdf of cos X for the random variable X ^^2009-11-24 13:08:27
 guess i'll go to gym than2009-11-24 13:09:00
*** Shelwien has left the channel2009-11-24 13:09:04
*** Shelwien has joined the channel2009-11-24 13:09:09
<Shelwien> ;)2009-11-24 13:09:19
<toffer> i need some more kilos to approach my physical limit which would be around 86kg ^^2009-11-24 13:09:34
 and somehow that is really a nice variation2009-11-24 13:10:09
<Shelwien> %)2009-11-24 13:10:21
 adding kilos is not a compression, yeah ;)2009-11-24 13:10:29
<toffer> ^^2009-11-24 13:10:35
 adding non-fat kilos became pretty hard the more you approach your physical limit2009-11-24 13:10:57
 in 1 year i gained 20 kilos in the next just 42009-11-24 13:11:13
 and now i got stuck at around 822009-11-24 13:11:22
 so guess it'd take at least 1-2years2009-11-24 13:11:36
 to reach the limit2009-11-24 13:11:40
<Shelwien> ...and then you won't need any trains or cars anymore, because you would be strong enough to just run anywhere? ;)2009-11-24 13:14:52
<toffer> not really2009-11-24 13:15:37
 but it's fun2009-11-24 13:15:40
 like others go cycling or stuff like that2009-11-24 13:15:48
 and matt participates at marathons2009-11-24 13:16:14
<Shelwien> well, yeah2009-11-24 13:17:44
 just that i don't have any friends with that hobby here, somehow ;)2009-11-24 13:17:59
 which just makes you more exotic ;)2009-11-24 13:18:46
<toffer> don't know about the availability of stuff like that at your side2009-11-24 13:18:59
 but here're quite alot guys doing that2009-11-24 13:19:07
<Shelwien> here too, of course2009-11-24 13:19:57
 just not anyone i know2009-11-24 13:20:05
<toffer> ^^2009-11-24 13:20:12
 maybe it`s because i'm a bit younger than you are2009-11-24 13:20:31
<Shelwien> nah, i was like that all the time2009-11-24 13:20:44
<toffer> dunnot know than2009-11-24 13:21:18
 what else do you do? any sports?2009-11-24 13:21:27
<Shelwien> well, shoot things with a bow ;)2009-11-24 13:21:42
<toffer> bow?2009-11-24 13:21:48
 shooting?2009-11-24 13:21:53
<Shelwien> climb trees and eat apples ;)2009-11-24 13:21:55
<toffer> shoot apples off people's heads ^^2009-11-24 13:22:19
<Shelwien> nah, they don't allow me somehow ;)2009-11-24 13:22:46
<toffer> ... might be caused by the possibility of weird "side effects" 2009-11-24 13:23:27
<Shelwien> nah, they're just paranoid ;)2009-11-24 13:23:48
<toffer> sure ^^2009-11-24 13:24:30
<Shelwien> then, cracking passwords (atm)... or it doesn't count as sport? ;)2009-11-24 13:25:10
<toffer> i meant physical activities2009-11-24 13:25:22
<Shelwien> then sleeping ;)2009-11-24 13:26:02
<toffer> i mostly just like that cause it can be challenging and you can switch off your mind2009-11-24 13:26:03
<Shelwien> yeah, i understand2009-11-24 13:26:29
<toffer> just a physical compensation2009-11-24 13:26:33
<Shelwien> i try some things sometimes too, but don't keep for long2009-11-24 13:26:39
<toffer> what for example?2009-11-24 13:27:29
<Shelwien> well, cycling (until bike died) or running around2009-11-24 13:28:18
<toffer> would you try some training than?2009-11-24 13:28:48
 to me it looks like you appreciate such activities, too2009-11-24 13:29:02
 i mean in the intention i explained2009-11-24 13:29:13
<Shelwien> well, i'd prefer to try some martial arts more likely ;)2009-11-24 13:29:56
<toffer> that'd be interesting, too2009-11-24 13:30:33
<Shelwien> tried some fencing before2009-11-24 13:30:49
<toffer> hey, me too. but just for 2 or 3 weeks as a tryout2009-11-24 13:31:10
<Shelwien> well, just didn't find a good enough company anywhere i tried, i guess2009-11-24 13:31:16
<toffer> liked it2009-11-24 13:31:21
<Shelwien> yeah2009-11-24 13:31:27
<toffer> no company - come on i can hardly imagine that2009-11-24 13:31:42
<Shelwien> there're people, but i don't quite like them ;)2009-11-24 13:32:01
<toffer> but what i didn't like is that people mostly targeted a area which wasn't armored2009-11-24 13:32:24
 why=2009-11-24 13:32:28
 ?2009-11-24 13:32:30
 like right below your arm2009-11-24 13:32:45
<Shelwien> good instincts? ;)2009-11-24 13:33:08
<toffer> not everybody is evil or an illuminate2009-11-24 13:33:27
 ^^2009-11-24 13:33:28
<Shelwien> nah, i mean, its more healthy to go for a kill right away, right? ;)2009-11-24 13:34:03
 especially, comparing to hitting the "armored areas" ;)2009-11-24 13:34:22
<toffer> erm i dunnot understand2009-11-24 13:34:27
<Shelwien> nevermind ;)2009-11-24 13:34:38
<toffer> well i was talking about the "company"2009-11-24 13:34:54
<Shelwien> i understand2009-11-24 13:35:03
 no, i my case its more about vodka i guess ;)2009-11-24 13:35:15
 *in2009-11-24 13:35:21
<toffer> vodka?2009-11-24 13:35:44
<Shelwien> its like part of every sport here or i dunno ;)2009-11-24 13:35:56
<toffer> erm?!2009-11-24 13:36:03
 so boozing is part of every sport?!2009-11-24 13:36:27
<pinc> he is definitely kidding )) vodka itself is a national sport2009-11-24 13:36:32
<toffer> ^^2009-11-24 13:36:40
<Shelwien> well, like, if you meet some people regularly, that means you have to celebrate stuff with them ;)2009-11-24 13:36:41
<toffer> yeah but that'd not be a problem2009-11-24 13:37:01
 and afair you drink alcohol2009-11-24 13:37:13
<pinc> we ought to %)2009-11-24 13:37:34
<Shelwien> i do, but don't like it to become regular2009-11-24 13:37:43
<toffer> and i don't think things get too messy2009-11-24 13:37:44
 that certainly is different in here2009-11-24 13:38:14
<Shelwien> well, i'm sure its possible to find something here too2009-11-24 13:38:23
 not that i really tried2009-11-24 13:38:27
 frankly, i don't have enough time even for computer-specific hobbies2009-11-24 13:39:21
<toffer> but your work time is pretty flexible2009-11-24 13:39:47
*** Shelwien has left the channel2009-11-24 13:40:44
*** Guest9968193 has joined the channel2009-11-24 13:40:47
<Guest9968193> <Shelwien> sure, but somehow always something happens2009-11-24 13:41:02
 <Shelwien> well, at least now i'm staying at home most of the time2009-11-24 13:41:03
<Shelwien> like, in 2004-2007 i didn't have any time for compression at all2009-11-24 13:41:35
<toffer> anyway you have to know what to do2009-11-24 13:43:09
 i'm lucky that i've got the time for that stuff2009-11-24 13:43:28
<Shelwien> hopefully i still have time to leave it to chance ;)2009-11-24 13:44:30
<toffer> i think i gonna leave now for gym2009-11-24 13:47:04
<Shelwien> well, i'll sleep then ;)2009-11-24 13:47:19
<toffer> ok2009-11-24 13:47:27
<Shelwien> !next2009-11-24 13:47:30