*** scott___ has joined the channel2010-01-22 23:55:48
<scott___> anybody here2010-01-23 00:30:11
*** scott___ has left the channel2010-01-23 00:32:04
<Shelwien> no patience :)2010-01-23 00:33:36
*** STalKer-X has joined the channel2010-01-23 04:21:42
*** STalKer-Y has left the channel2010-01-23 04:23:16
*** pinc has joined the channel2010-01-23 07:26:12
*** M4ST3R has joined the channel2010-01-23 12:20:54
<M4ST3R> hi2010-01-23 12:20:57
*** pmcontext has joined the channel2010-01-23 12:55:46
<pmcontext> hi2010-01-23 12:56:22
*** mike_____ has joined the channel2010-01-23 13:21:31
*** M4ST3R has left the channel2010-01-23 13:50:42
*** mike_____ has left the channel2010-01-23 14:29:40
 o.o2010-01-23 14:52:46
*** mike_____ has joined the channel2010-01-23 15:03:38
*** pinc has left the channel2010-01-23 15:25:08
*** encode has joined the channel2010-01-23 15:36:02
*** encode has left the channel2010-01-23 15:49:28
*** Shelwien has left the channel2010-01-23 17:17:40
*** Guest9968193 has joined the channel2010-01-23 17:17:43
*** mike_____ has left the channel2010-01-23 18:15:39
<Shelwien> respawn!2010-01-23 19:07:57
 pmcontext: seen http://encode.dreamhosters.com/showpost.php?p=10875&postcount=10 ?2010-01-23 19:08:30
<pmcontext> hi shelwien :D2010-01-23 19:09:02
<Shelwien> hi2010-01-23 19:09:07
<pmcontext> seeing now2010-01-23 19:09:54
*** scott___ has joined the channel2010-01-23 19:15:11
<Shelwien> hi2010-01-23 19:15:22
<pmcontext> it seems do very well even on enwik8 21MB !!2010-01-23 19:15:25
 tree is also fixed weights ?2010-01-23 19:15:52
<Shelwien> well, BWT does 20.6M on enwik, CM has to be better2010-01-23 19:15:58
 tree has not only fixed weights, but completely same mixing2010-01-23 19:16:12
<pmcontext> i see2010-01-23 19:16:34
<Shelwien> better results are due to DEPTH=8192 cutoff in green2010-01-23 19:16:35
 freqs are not completely the same2010-01-23 19:16:46
 ...actually it looks like i'm still running green on enwik8 :)2010-01-23 19:18:23
 since like 10 hours ago :)2010-01-23 19:18:28
<pmcontext> o.o omg2010-01-23 19:18:43
<scott___> it takes 10 hours to run?2010-01-23 19:18:54
<Shelwien> well, its slow even with DEPTH=81922010-01-23 19:18:56
 more like 20 i guess :)2010-01-23 19:19:14
 its only 65.5M of 100 now :)2010-01-23 19:19:26
 also not that its a 3.52Ghz Q9450 :)2010-01-23 19:19:56
 and it runs on ramdrive too :)2010-01-23 19:20:11
 *also note2010-01-23 19:20:36
<pmcontext> tree byte based mixer ?2010-01-23 19:22:19
<scott___> !Shelwien I got my best compresion on a modifiyes arb255 your fpaq0mw is better since its nonstationary. Today I am going to try to put the same modifications in that code and test on DNA to see how good the combo is2010-01-23 19:22:32
<Shelwien> %)2010-01-23 19:22:43
 pmcontext: what? :)2010-01-23 19:22:52
<scott___> bad typing hope you could follow it2010-01-23 19:23:02
<pmcontext> im having trouble implementing gradient descent2010-01-23 19:24:16
 i have one weight for each model2010-01-23 19:24:37
 example w0 , w1 ,w2 and 2010-01-23 19:25:05
<Shelwien> well, its not completely smooth2010-01-23 19:25:27
<pmcontext> and each model gives n0 n1 2010-01-23 19:25:34
<Shelwien> just try sampling the function (codelength of weights) with some step by single weight2010-01-23 19:26:06
 and plot it2010-01-23 19:26:14
<pmcontext> coz is use the n1/(n1+n0) counter , i cant get the prediction directly wihtout division2010-01-23 19:26:32
<Shelwien> i don't see how's that a problemn2010-01-23 19:26:49
<pmcontext> my problem is to adjust a weight w' = w + correction2010-01-23 19:27:11
<Shelwien> well, the coef for that correction is important2010-01-23 19:27:29
 and maybe you lost some sign2010-01-23 19:27:53
<pmcontext> yes i think correction = K*(y-p)2010-01-23 19:27:56
 where y-p is the prediction error2010-01-23 19:28:06
<Shelwien> and maybe it gives weird results with |p1-p2| near 02010-01-23 19:28:14
<pmcontext> but i cant have p , coz i use n1 , n0 counts 2010-01-23 19:28:41
<Shelwien> again, did you maybe forget about fixed-point shifts?2010-01-23 19:28:51
 like, if you multiple two N-bit values2010-01-23 19:29:06
<pmcontext> yes i rember the shifts , k is power of 22010-01-23 19:29:11
<Shelwien> you're supposed to shift it right by N2010-01-23 19:29:15
<pmcontext> i mean inverse power2010-01-23 19:29:32
 so that (y-p) >> 42010-01-23 19:29:47
 while updating i dont have P for each model . i have n0 n1 , to get P i need to do n1/(n1+n0) but this will become too many division2010-01-23 19:30:21
<Shelwien> you need to compute it before anyway, right? for mixing?2010-01-23 19:30:52
 why don't you just save the probs at that point?2010-01-23 19:31:04
<pmcontext> coz while mixing i dont do divison per model2010-01-23 19:31:15
<scott___> gn82010-01-23 19:31:36
<pmcontext> i combine n1 n0 from all models by multipling weights , then i do one division2010-01-23 19:31:50
 to get combined P2010-01-23 19:32:02
*** scott___ has left the channel2010-01-23 19:32:07
<Shelwien> well, if you don't, its already a different kind of mixing, it has a different gradient function :)2010-01-23 19:32:24
<pmcontext> o.o wat do i do, its been puzzling me2010-01-23 19:32:41
<Shelwien> well, i'd suggest to stop trying to make it faster when you didn't make it work yet :)2010-01-23 19:33:28
<pmcontext> with fixed weights 2391162010-01-23 19:34:58
<Shelwien> is that o3 or what?2010-01-23 19:35:12
<pmcontext> it has 5 orders2010-01-23 19:35:42
<Shelwien> well, that's not quite good for o4 if that's what you mean (o0-o4)2010-01-23 19:36:17
<pmcontext> i use hashtable so some contexts map to the same counter2010-01-23 19:36:50
 i was thinking i could see how it does with dynamic weights but2010-01-23 19:37:24
 i got stuck with the weight update problem2010-01-23 19:37:38
<Shelwien> well, i suppose you can write a log and see what's your problem where2010-01-23 19:38:43
 *there2010-01-23 19:38:46
 1. the weight is supposed to increase for the submodel with best prediction for current symbol2010-01-23 19:39:29
 2. the repeated mixing for the same inputs with updates weights is supposed to give better probability to current bit2010-01-23 19:40:05
<pmcontext> isnt it bad to have 4 divisions ? one per model 2010-01-23 19:44:44
<Shelwien> it is2010-01-23 19:44:55
 but why don't you care about that after you get better compression?2010-01-23 19:45:19
 there're multiple ways to get rid of divisions2010-01-23 19:45:34
<pmcontext> ok2010-01-23 19:45:48
<Shelwien> btw all these ways generally work at the cost of somewhat worse compression :)2010-01-23 19:46:29
<pmcontext> shelwien: have u done gradient based mixer ? any simple one2010-01-23 20:32:16
<Shelwien> not linear2010-01-23 20:32:43
 and not n-ary anyway2010-01-23 20:32:48
 toffer didn't do it either btw :)2010-01-23 20:33:03
 its just a theory :)2010-01-23 20:33:07
<pmcontext> not linear means ?2010-01-23 20:33:09
<Shelwien> well, paq mixer in mix_test is gradient-based2010-01-23 20:33:33
*** toffer has joined the channel2010-01-23 20:43:03
<toffer> hi 2010-01-23 20:43:08
 i just wanted to comment something quickly.2010-01-23 20:43:15
 gonna go downtown in a few mins2010-01-23 20:43:22
<pmcontext> hi2010-01-23 20:43:48
<toffer> in the log i've read that you got trouble with gradient descent...2010-01-23 20:43:52
 well2010-01-23 20:43:54
<pmcontext> yes2010-01-23 20:43:59
<toffer> to clarify - i implemented it several times2010-01-23 20:44:03
 in cmm32010-01-23 20:44:08
 for n inputs2010-01-23 20:44:12
 and for two2010-01-23 20:44:22
 but i used the n input variant only2010-01-23 20:44:33
 anyway2010-01-23 20:44:36
<pmcontext> i have 5 input2010-01-23 20:44:51
<toffer> i don't see why you insist on your counter outputting n0 and n1 to be a problem2010-01-23 20:44:53
 just calculate the probability2010-01-23 20:45:02
 and use the update formula we derived2010-01-23 20:45:10
<Shelwien> i think he mixes the freqs instead2010-01-23 20:45:22
<toffer> w' = w + L (p2-p1) (y-p) 1/(p(1-p))2010-01-23 20:45:26
<Shelwien> with a single division in the end2010-01-23 20:45:27
 so there's no separate probabilities2010-01-23 20:45:38
<toffer> p = (p2-p1)w + p12010-01-23 20:46:03
<Shelwien> well, he said that divisions are slow :)2010-01-23 20:46:28
<pmcontext> p2 and p1 what is it2010-01-23 20:46:33
<toffer> input probs2010-01-23 20:46:39
<pmcontext> but we have only one p for each model2010-01-23 20:47:08
<toffer> and?2010-01-23 20:47:14
 in cm you want to combine multiple models predictions2010-01-23 20:47:25
<Shelwien> mix( p3, mix(p1,p2) ) 2010-01-23 20:47:38
<toffer> yes2010-01-23 20:47:42
 in that case p = mix(p1,p2) = (p2-p1)w + p12010-01-23 20:47:57
<pmcontext> yes p = w0 * p0 + w1 * p1 + w2 * p2 + w3 * p32010-01-23 20:48:15
<toffer> well2010-01-23 20:48:27
 as i said2010-01-23 20:48:30
<pmcontext>  p = (w0 * p0 + w1 * p1 + w2 * p2 + w3 * p3) / ( w0 + w1 + w2 + w3) ,2010-01-23 20:49:01
<toffer> ...2010-01-23 20:49:13
 !grep (1-u)2010-01-23 20:49:26
 that is what these weights look like after cascading several n input mixers2010-01-23 20:49:56
 and u and v are limited to [0, 1]2010-01-23 20:50:12
<pmcontext> oh i see2010-01-23 20:50:19
<toffer> to calrify2010-01-23 20:50:28
 mix(p1,p2) = q2010-01-23 20:50:41
 mix(q,p3) = final prediction 2010-01-23 20:50:48
 q = (1-u) p1 + u p2, right?2010-01-23 20:51:05
<pmcontext> yes2010-01-23 20:51:21
<toffer> final prediction - we'd call it p'2010-01-23 20:51:29
 p' = (1-v) q + v p32010-01-23 20:51:42
 right?2010-01-23 20:51:43
<pmcontext> yes2010-01-23 20:51:55
<toffer> now substitute q into p' and compare the weight coeffs to p' = w1 p1 + w2 p2 + w3 p32010-01-23 20:52:24
<pmcontext> in weight update2010-01-23 20:53:59
 w' = w + L (p2-p1) (y-p) 1/(p(1-p)) , here i dont understand2010-01-23 20:54:01
 this we do for each update ?2010-01-23 20:54:02
 w0 = w0 + L( ? - ? ) * 1/ p(1-p)2010-01-23 20:54:04
<toffer> you're fine in using the recursive formulation2010-01-23 20:54:40
 you got2010-01-23 20:54:43
 q = mix(p1,p2)2010-01-23 20:54:48
 q = (p2-p1)u + p12010-01-23 20:54:57
 right?2010-01-23 20:55:00
<pmcontext> yes2010-01-23 20:55:11
<toffer> thus you compute u' = u + (p2-p1) (y-q) 1/(q(1-q))2010-01-23 20:55:32
 and you got p' = mix(q,p3)2010-01-23 20:55:47
 now hod you update the weight v in there?2010-01-23 20:55:54
<pmcontext> w' = w + L ( p2-p1) * 1/p(1-P)2010-01-23 20:56:53
<toffer> no2010-01-23 20:57:01
 maybe the terming confuses you2010-01-23 20:57:17
 say2010-01-23 20:57:39
 c = mix(a, b)2010-01-23 20:58:01
 c = (b-a) w + b2010-01-23 20:58:11
<pmcontext> ok2010-01-23 20:58:27
<toffer> c' = c + L (b-a) (y-c) 1/(c(1-c))2010-01-23 20:58:37
 now you got the mixer2010-01-23 20:58:56
 q = mix(p1,p2)2010-01-23 20:59:06
 mistake2010-01-23 20:59:21
 w' = w + L (b-a) (y-c) 1/(c(1-c))2010-01-23 20:59:24
 now you got, q = mix(p1,p2) - thus c=q, a=p1, b=p22010-01-23 20:59:48
 w' = w + L (b-a) (y-c) 1/(c(1-c)) becomes?2010-01-23 21:00:05
<pmcontext> w' = w + L (p2 - p1) (y-q) 1/q(1-q)2010-01-23 21:00:50
<toffer> ok2010-01-23 21:00:57
<pmcontext> i was trying to update the weights independently 2010-01-23 21:01:20
<toffer> and now you got p' = mix(q, p2) and that mixer got the weigh v (not to confuse it with w from the previous level)2010-01-23 21:01:31
<pmcontext> ok new is v2010-01-23 21:01:55
<toffer> and why?2010-01-23 21:01:59
 just try the current soluiton2010-01-23 21:02:05
 and improve afterwards2010-01-23 21:02:10
<pmcontext> i duno >< , ok2010-01-23 21:02:14
<toffer> i have to leave now2010-01-23 21:02:29
 gn8 and good luck2010-01-23 21:02:32
<pmcontext> thank you2010-01-23 21:02:38
*** toffer has left the channel2010-01-23 21:02:41
<Shelwien> btw, you can try a simpler thing first2010-01-23 21:10:13
 adaptive mix only for 2 inputs of 52010-01-23 21:10:56
 it should improve compression anyway2010-01-23 21:11:30
 and when it'd work, you'd be able to extend it2010-01-23 21:11:48
<pmcontext> ok2010-01-23 21:17:20
 i g2g bye and thanks2010-01-23 21:25:22
*** pmcontext has left the channel2010-01-23 21:25:44
*** STalKer-Y has joined the channel2010-01-24 04:21:18
*** STalKer-X has left the channel2010-01-24 04:22:21
*** pinc has joined the channel2010-01-24 10:57:24
*** mike_____ has joined the channel2010-01-24 11:00:11
*** STalKer-Y has left the channel2010-01-24 13:25:24
*** STalKer-X has joined the channel2010-01-24 13:26:30
*** Shelwien has left the channel2010-01-24 13:27:26
*** Guest9968193 has joined the channel2010-01-24 13:27:30
*** pinc has left the channel2010-01-24 13:39:06
*** pmcontext has joined the channel2010-01-24 14:22:38
*** scott___ has joined the channel2010-01-24 14:45:16
*** scott___ has left the channel2010-01-24 14:55:14
*** Krugz has left the channel2010-01-24 16:18:15
*** toffer has joined the channel2010-01-24 16:24:14
<toffer> hi2010-01-24 16:28:14
<pmcontext> hii2010-01-24 16:29:44
 toffer im working on the mixer , and jst got back again2010-01-24 16:30:43
*** ARTHURIUSS has joined the channel2010-01-24 16:36:43
<toffer> os you're doing cascaded two input mixers now?2010-01-24 16:37:33
 hi ARTHURIUSS - are you new here?2010-01-24 16:40:05
<ARTHURIUSS> yeah..2010-01-24 16:40:29
<toffer> are you on the forum, too?2010-01-24 16:40:39
<ARTHURIUSS> nooo..just a random passerby looking for a matlab program for an IEEE paper2010-01-24 16:41:24
<toffer> compression relaetd?2010-01-24 16:42:27
 related2010-01-24 16:42:28
<ARTHURIUSS> hmm..actually it's image encryption and sharing....2010-01-24 16:43:07
 using shadow images..2010-01-24 16:43:32
<toffer> dunnot know about that2010-01-24 16:44:39
 and how'd you came accross that channel than?2010-01-24 16:44:47
*** toffer has left the channel2010-01-24 16:45:38
*** toffer has joined the channel2010-01-24 16:47:01
*** ARTHURIUSS has left the channel2010-01-24 16:56:42
<pmcontext> yes 2 input mixer2010-01-24 17:00:27
*** scott___ has joined the channel2010-01-24 17:01:58
<scott___> hello where is !Shelwien today2010-01-24 17:02:32
 while toffer maybe you can anwser2010-01-24 17:03:47
 well toffer2010-01-24 17:03:53
<pmcontext> toffer: w' = w + L (b-a) (y-c) 1/(c(1-c)) , do i really multiply by 1/(c(1-c))2010-01-24 17:04:13
 or can that term be removed ?2010-01-24 17:04:14
<toffer> you need to...2010-01-24 17:04:36
 otherwise you're not following the entropy metric2010-01-24 17:04:50
<pmcontext> ok2010-01-24 17:05:31
<scott___> is Ilia Muraviev the same as Shelwien?2010-01-24 17:08:19
<toffer> erm2010-01-24 17:08:45
 pretty funny2010-01-24 17:08:49
 no, of course no2010-01-24 17:08:52
 t2010-01-24 17:08:54
 not2010-01-24 17:08:56
<scott___> ok I was wondering since I have been looking at fpaq0mw that Shel... gave me but it has the other guys name who did it?2010-01-24 17:10:29
<toffer> shelwien just inserted a liner mixer implementatio2010-01-24 17:10:46
 inplementation2010-01-24 17:10:51
 implementation2010-01-24 17:10:54
 damn!2010-01-24 17:10:56
 guess i'm still drunk2010-01-24 17:11:00
<scott___> from beer?2010-01-24 17:11:08
<toffer> yesterday i officially celebrated my graduation2010-01-24 17:12:19
 now i'm officially an engineer2010-01-24 17:12:29
<pmcontext> congrats !!!2010-01-24 17:12:34
<toffer> dipl.-ing. in german2010-01-24 17:12:39
 thanks2010-01-24 17:12:43
<pmcontext> :D2010-01-24 17:12:45
<scott___> any way trying to modify it to cpmparess this DAN packed file file and it does not seem to compress better with mixing 2010-01-24 17:12:53
 now you can drive a train2010-01-24 17:13:07
<pmcontext> shelwien inserted on website ?2010-01-24 17:13:27
<toffer> not really2010-01-24 17:13:36
<scott___> congrats what kind of enginer2010-01-24 17:13:37
<toffer> for modelling, automation and control2010-01-24 17:13:59
<scott___> my BSEE was in fields and waves while MSEE was in control theory2010-01-24 17:14:22
<toffer> sounds like theoretical electectrical engineering2010-01-24 17:15:25
 your bachelor2010-01-24 17:15:36
 control theory is equivalent to automation and control i guess2010-01-24 17:16:15
 and in addition i got the best possible graduation2010-01-24 17:16:34
 ^^2010-01-24 17:16:37
<scott___> yes I wsitched from math to electronis since less english and bs humnaity classed I went to school an a maht scholar ship but got more real math in engineering2010-01-24 17:16:45
 what level BS MS or Phd2010-01-24 17:17:16
*** Shelwien has joined the channel2010-01-24 17:17:19
<toffer> it's equal to ms2010-01-24 17:17:37
<pmcontext> darn it giving some big number2010-01-24 17:17:53
<toffer> i gonna do my phd than. probably at the local fraunhofer2010-01-24 17:17:57
<scott___> good did you have to writw thesis2010-01-24 17:18:00
<toffer> of course2010-01-24 17:18:07
 it was about modelling and predicting stochastic processes with pattern characteristics2010-01-24 17:18:36
 i made some context mixing predictor2010-01-24 17:18:42
 which outpreformed the previous solution2010-01-24 17:19:12
<scott___> I was luck at USC that had two options. One was to take a test and pass an MS level and wirte a paper the other option was to pass the test ast the equivalent of Phd level I choose that path sa as not to wirte anything2010-01-24 17:19:19
<pmcontext> o.o awesome!!2010-01-24 17:19:27
<scott___> lucky2010-01-24 17:19:34
<toffer> that's pretty weird2010-01-24 17:20:05
<scott___> they offered my to stay in school to get a Phd in Plasma Physics but I shoose not to.2010-01-24 17:20:09
 chose not too.2010-01-24 17:20:26
*** Guest9968193 has left the channel2010-01-24 17:20:38
<toffer> is that a lang time ago?2010-01-24 17:20:42
<scott___> that was in the early 70's for ms2010-01-24 17:21:09
<toffer> really?2010-01-24 17:21:38
 i thought you're younger2010-01-24 17:21:43
<scott___> USC thorugh rated high. I hated it compared to ASU2010-01-24 17:21:51
<toffer> you must be like 60 now?2010-01-24 17:21:51
<scott___> yest in my 60's2010-01-24 17:22:01
 thats not my picture at your site2010-01-24 17:22:17
 USC univeristy of southern california ASU arizonia state university2010-01-24 17:23:12
<toffer> "not my picture on your site"?2010-01-24 17:23:35
<scott___> not my picture on any site.2010-01-24 17:23:47
<toffer> that image on your profile?2010-01-24 17:24:38
<scott___> is just an avator who uses there real picture.2010-01-24 17:25:28
<toffer> well i do2010-01-24 17:25:52
<scott___> well youlook better than me2010-01-24 17:26:03
<toffer> i'm just younger2010-01-24 17:26:12
<pmcontext> im more younger2010-01-24 17:26:26
 :D2010-01-24 17:26:29
<toffer> ^^2010-01-24 17:26:40
<pmcontext> -----------------------------------------------------2010-01-24 17:26:46
 strange i got w = 5 , p1 = 4 , p2 = 5 2010-01-24 17:26:48
 p = (w*(p1-p2)+p2) = 0 , resulted in 02010-01-24 17:26:49
 , my mixer update bombed due to division by zero2010-01-24 17:26:51
 dw = ((p2-p1)*((b<<12)-p)/(p*(4096-p)))>>4 , since p is 0 it blowed division with 1/p(1-p)2010-01-24 17:26:52
<scott___> I had a fu manchu mostache in ealy college years with my motorcle if I ever find it may use that some as my pic2010-01-24 17:27:15
<toffer> 1. p = (1-w) p1 + w p2 = (p2-p1) w + p12010-01-24 17:28:19
 2. i told you that for p<t, p>=1-t you should make g(p) = 1/(p(1-p)) equal to zero for some small threshold t2010-01-24 17:29:00
 you mean that guy from some movie?2010-01-24 17:29:30
 dr. fu manchu2010-01-24 17:29:33
 such a picture would be a good avatar i guess2010-01-24 17:29:49
<pmcontext> oh my bad2010-01-24 17:31:07
<scott___> I don't have a Phd in fact when working with Gil his the one that wrote the paper I just BSed on phoe after he got it to work he asked about my Phd and where I taught2010-01-24 17:31:15
 The only teaching was at a juniot college where the kids never learn to add factions and I did not want that scholl even mentioned2010-01-24 17:32:11
 I have never talked to any on voice the forum but Mike boss and I talked on the phone.2010-01-24 17:33:05
<toffer> somehow i feel kids get dumber the more time passes by2010-01-24 17:33:07
<pmcontext> p=(w*(p1-p2)+p2); for this should i do a check for the range of p ?2010-01-24 17:34:11
 after p is computed2010-01-24 17:34:22
<toffer> 1. p = (1-w) p1 + w p2 = (p2-p1) w + p12010-01-24 17:35:32
 your formula is still wrong2010-01-24 17:35:39
 see?2010-01-24 17:36:00
 you just need to make sure w is in [0, 1]2010-01-24 17:36:44
<scott___> it makes no difference his formula same as yours2010-01-24 17:37:26
<toffer> no2010-01-24 17:37:41
 (p2-p1) w + p1 vs. w*(p1-p2)+p22010-01-24 17:37:57
 weight update relies on the order of p1 and p22010-01-24 17:38:22
 it must be p2-p1 but not p1-p22010-01-24 17:38:31
 anyway gonna go for dinner now2010-01-24 17:38:39
<scott___> sorry2010-01-24 17:39:11
<pmcontext> coz in mix (mix( p1 , p2 ) , p3 ) ?2010-01-24 17:39:13
 or mix (p3 , mix( p1 , p2 ) ) 2010-01-24 17:39:15
<toffer> because it's mix(a,b) but not mix(b,a) with a=p1, b=p22010-01-24 17:39:45
<pmcontext> oh ok2010-01-24 17:40:14
<toffer> same for a=p3, b=mix(p1,p2)2010-01-24 17:40:17
 and so on...2010-01-24 17:40:20
<pmcontext> yes i und2010-01-24 17:40:23
<toffer> you just need to make a single mixer class2010-01-24 17:40:28
 with something like int Mix(int p1, int p2)2010-01-24 17:40:45
<scott___> yes I was wrong i really am not into this yet2010-01-24 17:40:48
<toffer> anyway dinner calls 2010-01-24 17:41:03
 good luck2010-01-24 17:41:04
<pmcontext> i made one2010-01-24 17:41:18
 class MIXER {2010-01-24 17:41:19
 public :2010-01-24 17:41:21
  int w;2010-01-24 17:41:22
  int p,p1,p2;2010-01-24 17:41:24
  MIXER(){w=32;}2010-01-24 17:41:25
  int mix(int _p1, int _p2){2010-01-24 17:41:27
  p1=_p1;2010-01-24 17:41:28
  p2=_p2;2010-01-24 17:41:30
  p=(w*(p2-p1)+p1);2010-01-24 17:41:31
  p = (p<0 ? 0 : (p>4095 ? 4095 : p));2010-01-24 17:41:33
  return p;2010-01-24 17:41:34
  }2010-01-24 17:41:36
  void update(int b){2010-01-24 17:41:37
  working on this ...2010-01-24 17:41:39
  w += dw;2010-01-24 17:41:40
  w = (w<0 ? 0 : (w>64 ? 64 : w));2010-01-24 17:41:42
  }2010-01-24 17:41:44
  void printw(){printf("w=%d\n",w);}2010-01-24 17:41:45
 };2010-01-24 17:41:47
 ok see you after dinner2010-01-24 17:42:03
<scott___> how you determine between to models that which to make P1 and which to make P2 just try both and see what happens or what?2010-01-24 17:42:12
<pmcontext> we use this order mix ( mix (p1 , p2 ) , p3 )2010-01-24 17:42:51
<scott___> put aren;t the Ps different models?2010-01-24 17:43:26
<pmcontext> yes2010-01-24 17:44:33
<scott___> if I hav 2 different model P1 and P2 your saying that the mixing depenfs on which I call P1 and P2 if I reverse the order I get a different anwser for the final guess of P2010-01-24 17:45:08
*** mike_____ has left the channel2010-01-24 17:47:13
*** mike_____ has joined the channel2010-01-24 17:52:47
<toffer> storing p1 and p2 is redundant2010-01-24 18:05:27
 int mix( const int dp, const int p2 ) { return w*dp+p1; } call it with mix(p2-p1,p1).2010-01-24 18:05:58
 int update(int dp, int e, int p) { if (p>T && p<=ONE-T) w=max(0, min(W_ONE, (int)w + dp*e*p/(p*(1-p))>>L )); }2010-01-24 18:09:17
 and add some symbolic constants2010-01-24 18:09:23
 like mixer { static const int ONE=1<<6; // 6 bit weights ... }2010-01-24 18:09:49
 gn8 guys2010-01-24 18:15:43
*** toffer has left the channel2010-01-24 18:15:57
*** pmcontext has left the channel2010-01-24 18:38:27
<scott___> hello !Shelwien2010-01-24 18:45:21
<Shelwien> too late today, i guess :)2010-01-24 18:59:19
<scott___> still here2010-01-24 19:04:45
<Shelwien> hi :)2010-01-24 19:04:53
<scott___> I have some questions about fpaq0wm2010-01-24 19:05:24
<Shelwien> ?2010-01-24 19:05:51
<scott___> well for one thing when I use the dna file it seems to expand2010-01-24 19:06:28
 how dow you pick the P1 equation and the P2 equation2010-01-24 19:06:54
 one you shif 4 and the other 82010-01-24 19:07:21
<Shelwien> more or less random2010-01-24 19:07:44
 it was just a demo of dynamic mixing2010-01-24 19:07:56
<scott___> so one would have to play with a bunch of several shitf for p1 and p2 to get the best2010-01-24 19:08:49
<Shelwien> i'd prefer something like in http://www.ctxmodel.net/rem.pl?-6 instead2010-01-24 19:09:42
<scott___> here is the thing I was playing and lost the modifed arb255 which uses a single P based on 64bit count registers for ones and zeros2010-01-24 19:10:03
 and thought since static your would do much better2010-01-24 19:10:22
 fpaq0pv4B is this what you mean?2010-01-24 19:11:24
<Shelwien> specifically the version linked there2010-01-24 19:11:46
<scott___> that is the version lined there2010-01-24 19:12:17
<Shelwien> well, anyway, the update function is described in the text, along with optimal parameter values for different types of data2010-01-24 19:12:35
 wbich "that"? http://ctxmodel.net/files/fpaq0pv4B_wr.rar is linked there2010-01-24 19:12:58
 and there're a few other branches of the same coder2010-01-24 19:13:10
<scott___> but then you would need to now the int wr_mw[][2] = { valuse which are different for each file.2010-01-24 19:13:58
<Shelwien> why, you do need to know these2010-01-24 19:15:21
 of course you can select some parameters (for example, by the table there) with good average results2010-01-24 19:15:57
<scott___> well for one think I am using the compressed 2 bits / symbol DNA file so would have to guess which one is corrent2010-01-24 19:16:30
<Shelwien> try them all? :)2010-01-24 19:17:19
<scott___> well if it compresses good would you later tune it to a better set2010-01-24 19:17:57
 you have a fast way of tuning you models.2010-01-24 19:18:22
 your models2010-01-24 19:19:03
<Shelwien> well, there's no better way than trying all the parameter sets anyway :)2010-01-24 19:19:44
 but in the end we have to use some heuristics2010-01-24 19:21:13
<scott___> you mean guess2010-01-24 19:22:10
<Shelwien> more or less2010-01-24 19:22:21
 its a guess, but its based on some assumptions about the behavior of the codelength function2010-01-24 19:23:22
<scott___>  Predictor(): cxt(1) { this line confuses me I no C better but what does the " cxt(1) " do I had to create other varibles and just later did the x = 1; type of thing but why no commas or ; stuff2010-01-24 19:25:23
 i know c2010-01-24 19:25:39
 I can edit matts and your code but not really familiar with just what the line above does. 2010-01-24 19:26:24
<Shelwien> Predictor(): cxt(1) {...} is equal to Predictor() { cxt=1; ... } in C++2010-01-24 19:26:42
<scott___> so its designed to confuse c coders2010-01-24 19:28:39
 can you do then Preictor(): ctx(1) : xzy(2) to mean ctx = 1; xzy = 2;2010-01-24 19:29:48
<Shelwien> yea, but Preictor(): ctx(1), xzy(2)2010-01-24 19:34:17
 and both ctx and xzy have to be class members2010-01-24 19:34:34
<scott___> ok thanks I moded your code and had to the xyz = 1; thing in my mind I was worried there may have been side affects that I could not follow but you have cleared it up thank you?2010-01-24 19:35:54
 I will play with the way I have changed fpaq0mw then if it fails to get very good I think I will go back to kluding up arb255 tohandle the mess un a stationay way when done May try to add windowing probabiliy2010-01-24 19:38:01
<Shelwien> not that fpaq0mw is still order02010-01-24 19:38:31
 and it has to encode 8 bits per symbol for DNA data2010-01-24 19:38:49
<scott___> I don't think I have a good enough so handle nonstionary2010-01-24 19:38:58
  no i changed it hadles more2010-01-24 19:39:15
 yes I got got rsults from arb255 modifed though order 0 to2010-01-24 19:39:53
 will go to order one later2010-01-24 19:40:17
 the numbers I give you the other day for a order 0 stationary code of the e.col2010-01-24 19:40:52
<Shelwien> well, with bits 7..0 it encodes e.coli to 11976552010-01-24 19:41:26
 but with only bits 2..1 it encodes it to 11471302010-01-24 19:41:40
<scott___> what do you mean two bits?2010-01-24 19:42:49
 mix 6 8 1143894 in sa2010-01-24 19:43:08
<Shelwien> i don't know what you mean either :)2010-01-24 19:44:00
 but bits 2..1 of acgt are enough to identify all symbols2010-01-24 19:44:22
<scott___> by two 2 do you mean regular compacted to 4 per character 2010-01-24 19:44:57
 a 00 c 01 g10 t 112010-01-24 19:45:16
<Shelwien> a 61 = 01100<00>12010-01-24 19:46:13
 c 63 = 01100<01>12010-01-24 19:46:13
 g 67 = 01100<11>12010-01-24 19:46:13
 t 74 = 01110<10>02010-01-24 19:46:13
<scott___> and2010-01-24 19:47:13
 look I compress e,col to a file of 1159673 use a = 00 c =01 g= 10 t = 112010-01-24 19:48:45
 then compress that with a modifed fpaq0wm2010-01-24 19:49:08
 this what I meant what do yo mean what did you compress2010-01-24 19:49:34
<Shelwien> a=00 c=01 g=11 t=10 i guess2010-01-24 19:50:49
<scott___> ok so whne you do that you end up doing zero order on two bit fields2010-01-24 19:52:27
<Shelwien> sure, and there is some compression2010-01-24 19:53:16
<scott___> you only need like int p[4;2010-01-24 19:53:23
 p[4]2010-01-24 19:53:38
<Shelwien> but obviously fpaq0 parameters are not optimized for dna data2010-01-24 19:53:42
<scott___> yes that why you need so other mods2010-01-24 19:54:00
 some ohter mods to it 2010-01-24 19:54:14
 to start with p[4] is to small p[64] works better2010-01-24 19:54:56
 why grap 2 bits whne 6 is better2010-01-24 19:56:06
<Shelwien> well, sure you need a higher order context there2010-01-24 19:57:25
<scott___> using the fpaqowm mdoel I changed the bits grapped need to make a symbol. in your model 6 bits the best. When I did it no arb 12 bits the best.2010-01-24 20:03:43
 so I am father changing model and working with 12 till I add in all the features I had in arb2010-01-24 20:04:27
 like it runs I compress and uncompress back and do a fc /b on the files.2010-01-24 20:04:56
<Shelwien> you can also try assigning various permutations of 2 bits to symbols2010-01-24 20:05:04
<scott___> can show you the code at this point if you wnat to see it2010-01-24 20:06:16
<Shelwien> i mean, there're 24 permutations of acgt2010-01-24 20:06:25
<scott___> I can show ...2010-01-24 20:06:25
<Shelwien> and you can map any of them to 00 01 10 11 2010-01-24 20:06:40
 and that would affect compression with a bitwise coder2010-01-24 20:07:05
<scott___> your right any would have worked2010-01-24 20:07:20
 only if you count with one p there would me two answers if you p[2] there are 4 seperate leaves so no real difference in compressed size2010-01-24 20:08:24
 in short if 100 a's 1 c 1 g 100 t then a 00c 01g 10 t 11 gives a bad answes2010-01-24 20:09:57
  bad chocie2010-01-24 20:10:56
<Shelwien> not really... compression is determined by entropy model2010-01-24 20:11:30
<scott___> better 100 a 200 c 300 g 400 t the ratio of 1 ot 0 would depend on how you assing a c g t2010-01-24 20:12:31
 but the relative ratiors of a c g t the same not matter who you assing it2010-01-24 20:12:58
<Shelwien> and in dna A and T, C and G are "complementary"2010-01-24 20:13:05
<scott___> how you assign it2010-01-24 20:13:12
 on oppostie strands2010-01-24 20:13:44
<Shelwien> so for a context AATCCG you can sometimes find a corresponding TTAGGC2010-01-24 20:14:00
<scott___>  but if you look at e.col you find all 4 a different number2010-01-24 20:14:11
 that could be account for in the model 2010-01-24 20:15:21
 accounted for in the model2010-01-24 20:15:34
 in fact i think any combinatu possible2010-01-24 20:15:54
 look at the fisrt order model (which is not done) I can see that at grouping of 3 condons and 6 very good when I varied the number of bits in a symbol at least in arb I got min 3 symbol 6 bits and a bigger one at groupising od 6 12 bits2010-01-24 20:18:09
 on hte fpaq model 3 was best but 6 a local best which is 12 bits 10,11. 13.14 all bad2010-01-24 20:19:28
 so decided to make basci symbol size 6 units or 12 bits of the packed dna2010-01-24 20:20:15
 does this make sense2010-01-24 20:20:28
 now I have not done the frame reset in the fp version that made a big difference in the arb version2010-01-24 20:21:22
 any questions of any of the butchered lines I wrote need claifing2010-01-24 20:27:13
 here is what I think it is you see a file as characters.2010-01-24 20:28:22
 I see a file as ones and zeroes.2010-01-24 20:28:34
 i see the normal DNA acgt characater stuff as a waste of ones and zeroes,2010-01-24 20:29:06
<Shelwien> whatever... the point is that you can assign different binary codes to input symbols2010-01-24 20:29:41
<scott___> so I chage it to a file of oneze and zeroes where very 2 bits meant a chanacter in the old file set2010-01-24 20:29:42
 however I then group so that when viewed from the outsise I use 6 charcter as a basic symbol 2010-01-24 20:31:17
 so know 64 leaves while running is the whole space of symbols.2010-01-24 20:32:03
<Shelwien> but with bitwise coding2010-01-24 20:32:34
<scott___> now and know are alwasy a bitch I get the two confused in typeing2010-01-24 20:32:39
<Shelwien> you assign probabilities not only to symbols2010-01-24 20:32:42
 but to sets of symbols as well2010-01-24 20:32:46
<scott___> are you asking a question or correcting what I said or what?2010-01-24 20:33:33
<Shelwien> that's why your binary decompositions of symbols affect the compression2010-01-24 20:33:51
 unrelated to efficiently of bitcode2010-01-24 20:34:09
<scott___> I am actaully only looking at large symbol sets. but I always making them binary2010-01-24 20:34:52
 sort of the way fpaq0 is only looking at 257 symbols2010-01-24 20:35:29
<Shelwien> as i said, there're different binary codes for the same symbols2010-01-24 20:35:46
 like a:00 c:01 g:10 t:112010-01-24 20:35:48
<scott___> they ise 8 bits for the bytes and anothe wastefully for the EOF2010-01-24 20:35:55
<Shelwien> and a:00 c:10 g:11 t:012010-01-24 20:35:58
 and that affects the following compression with a bitwise model like fpaq02010-01-24 20:36:42
<scott___> yes and in this case of the symbol tree is even in length you get the same length compressions no matter how you assing them2010-01-24 20:37:00
<Shelwien> that "length" only affects compression speed, not ratio2010-01-24 20:37:38
<scott___> take any file swap character codes and compress again with fpap0 you should get the same length except for round offs2010-01-24 20:38:15
 bwt of character makes a big difference but zero order makes no difference. 2010-01-24 20:39:08
<Shelwien> there's no difference only with alphabet coding2010-01-24 20:40:10
 but with bitwise coding there's a significant difference2010-01-24 20:40:24
<scott___> your write is will make a difference if you group based on closeness2010-01-24 20:41:09
 your right I was wrong2010-01-24 20:41:18
*** scott____ has joined the channel2010-01-24 20:43:37
<scott____> you are right I was wrong bad memory2010-01-24 20:43:57
 the browser crashed so I was off2010-01-24 20:44:11
<Shelwien> ...2010-01-24 20:44:28
<scott____> any way I forgot I wondered why arb255 did get different compression with different chacters sets2010-01-24 20:44:53
<Shelwien> :)2010-01-24 20:45:32
<scott____> I found that if you give each charter a weight of one on the leaves and then adde as you go back up so top cell has a weight of 256 then they compress to same lenght2010-01-24 20:46:03
 it gets the same compression when you have pernutaions of the input file.2010-01-24 20:46:39
<Shelwien> http://compressionratings.com/bwt.html#transformation2010-01-24 20:46:39
 [6] there uses the alphabet permutation which i optimized for enwik2010-01-24 20:47:19
<scott____> permutiations2010-01-24 20:47:29
 bad speilling2010-01-24 20:47:40
<Shelwien> you can see that it improves compression for most of the coders2010-01-24 20:48:03
<scott____> what improves it?2010-01-24 20:48:24
 sorry I see it2010-01-24 20:48:39
 you are correct since I use a binary tree I would need to do that2010-01-24 20:49:30
 i suspect when done the easy way is to check all permutaions and see what gives the best2010-01-24 20:50:25
<Shelwien> unfortunately that's not easy at all2010-01-24 20:50:53
<scott____> but since I group in units of 9 it would be a real mess since 2**9 ways2010-01-24 20:51:19
 sorry 6 coden unit for 12 bits for 2**132010-01-24 20:51:52
 2**122010-01-24 20:51:58
 typo after typo2010-01-24 20:52:13
 and I have not had any beer2010-01-24 20:52:25
<Shelwien> think it could help? :)2010-01-24 20:52:55
<scott____> it always seems to my body is made for beer2010-01-24 20:53:13
 by2010-01-24 20:57:20
*** scott____ has left the channel2010-01-24 20:57:26
*** maniscalco has joined the channel2010-01-24 21:51:41
<Shelwien> hi :)2010-01-24 21:52:11
*** Krugz has joined the channel2010-01-24 21:57:20
<maniscalco> so, shelwien, you have been doing work lately to simply CM for new people 2010-01-24 21:59:09
 ?2010-01-24 21:59:11
<Shelwien> in a way, i guess2010-01-24 21:59:38
 not that it really can be called "work"2010-01-24 21:59:58
<maniscalco> i also always wondered about ACB. its a bit of a legend really2010-01-24 22:00:05
 ha2010-01-24 22:00:13
 if you are having fun ....2010-01-24 22:00:18
 you can't call it work2010-01-24 22:00:22
<Shelwien> no, i mean it didn't really take much time2010-01-24 22:00:32
<maniscalco> well, sure2010-01-24 22:00:39
<Shelwien> i was experimenting before with data structures2010-01-24 22:00:55
<maniscalco> but making something that can be followed easily .... that takes effort2010-01-24 22:00:57
<Shelwien> and made that "tree" parser2010-01-24 22:01:10
 then added some simple coding to it2010-01-24 22:01:18
 well, that was long ago2010-01-24 22:01:34
 and recently a new guy appeared here, and started asking questions about CM2010-01-24 22:02:05
<maniscalco> i have not looked at the code, just the posts. "easily followed" russin style coding ... or for the rest of us? (^:2010-01-24 22:02:05
 (I know, you are not russian)2010-01-24 22:02:14
<Shelwien> err, i am :)2010-01-24 22:02:19
<maniscalco> i thought you are ukranian!2010-01-24 22:02:29
<Shelwien> its kinda the same2010-01-24 22:02:36
<maniscalco> hmm2010-01-24 22:02:41
 tell my wife that2010-01-24 22:02:47
 she is "ot bulgaria"2010-01-24 22:02:59
<Shelwien> well, that's different2010-01-24 22:03:10
<maniscalco> how so ?2010-01-24 22:03:16
<Shelwien> but russia and ukraine was parts of the same USSR not that long before :)2010-01-24 22:03:25
<maniscalco> she gave me a daughter by the way2010-01-24 22:03:31
 yesterday2010-01-24 22:03:33
<Shelwien> yeah, i've seen your blog :)2010-01-24 22:03:40
 congratulations :)2010-01-24 22:03:46
<maniscalco> thank you2010-01-24 22:03:50
 but russia cut of gas to ukraine 2010-01-24 22:04:02
 a year ago2010-01-24 22:04:08
 how close can they be ?2010-01-24 22:04:13
<Shelwien> not really2010-01-24 22:04:13
<maniscalco> oh, well in bulgaria, they were cold as a result2010-01-24 22:04:28
<Shelwien> err, its 12 hours to moscow by train from here2010-01-24 22:04:30
<maniscalco> I was there2010-01-24 22:04:33
<Shelwien> and anyway i'm russian2010-01-24 22:04:45
<maniscalco> we bostonians were not so cold,but the bulgarians were2010-01-24 22:04:49
 (^:2010-01-24 22:04:51
 ok, i don't mean to offend2010-01-24 22:05:03
 i just try to understand2010-01-24 22:05:07
 (^:2010-01-24 22:05:11
<Shelwien> no problem, but ukraine is still kinda an artifical country2010-01-24 22:05:41
<maniscalco> so, how did you come to know ACB ?2010-01-24 22:05:41
 oh.2010-01-24 22:05:51
<Shelwien> well, you can say that it was what got me into compression 2010-01-24 22:06:24
<maniscalco> but i didnt think that anyone knew his algortihm2010-01-24 22:06:43
 its in the books 2010-01-24 22:06:46
 but not really described2010-01-24 22:06:52
<Shelwien> there's even a source2010-01-24 22:06:59
<maniscalco> and i never saw any results2010-01-24 22:07:03
<Shelwien> and original article(s)2010-01-24 22:07:05
 its just too old2010-01-24 22:07:10
<maniscalco> in russian, maybe?2010-01-24 22:07:13
<Shelwien> its from 19972010-01-24 22:07:13
 there's a link in the thread, did you see it?2010-01-24 22:07:26
<maniscalco> i was busy having a daughter2010-01-24 22:07:42
 i have new results for M03 btw2010-01-24 22:07:58
 but they are not "proven"2010-01-24 22:08:04
 as i have not had time to match the decoder model to the changes in the encoder2010-01-24 22:08:15
 I know not to announce results before the decoder2010-01-24 22:08:24
 but2010-01-24 22:08:25
 easy change2010-01-24 22:08:30
 I predict 206 ,000 for book12010-01-24 22:08:42
 same speed2010-01-24 22:08:47
 well.... 206,xxx2010-01-24 22:08:59
 that is2010-01-24 22:09:01
 it turns out that # of symbols in a parent context are closely related to the distribution patern in the children2010-01-24 22:09:41
 if there are more children to a single parent etc2010-01-24 22:09:53
 that is2010-01-24 22:09:55
 if a parent has N symbols2010-01-24 22:10:02
 and there are K children contexts to that one parent context2010-01-24 22:10:13
 add (K/N) to the model2010-01-24 22:10:27
 and you get big results2010-01-24 22:10:34
 (3 bits)2010-01-24 22:10:38
 im not sure i underatnd why yet2010-01-24 22:10:52
 and my daughter was born so i have not had time to work on it2010-01-24 22:11:06
*** Shelwien has left the channel2010-01-24 22:12:15
*** Guest9968193 has joined the channel2010-01-24 22:12:22
<Shelwien> sorry, got disconnected2010-01-24 22:12:41
<maniscalco> np2010-01-24 22:12:45
 can you read the previous posts?2010-01-24 22:12:59
 or should i repost2010-01-24 22:13:08
<Shelwien> i have the log, yeah2010-01-24 22:13:16
 btw2010-01-24 22:13:28
 http://nishi.dreamhosters.com/chantailh.cgi?100 is the log2010-01-24 22:13:31
 and also http://nishi.dreamhosters.com/log/2010-01-24 22:13:43
<maniscalco> ah, 2010-01-24 22:14:20
 good thanks2010-01-24 22:14:22
 "/log/" is useful2010-01-24 22:14:53
 didnt know about that2010-01-24 22:14:59
<Shelwien> there's also search here2010-01-24 22:15:14
 !grep maniscalco>2010-01-24 22:15:19
<maniscalco> (^:2010-01-24 22:15:32
 im not so good with computers2010-01-24 22:15:39
 i just know the algorithms2010-01-24 22:15:45
<Shelwien> that's kinda unrelated to computers :)2010-01-24 22:15:52
<maniscalco> yes2010-01-24 22:15:57
<Shelwien> i mean, its my script :)2010-01-24 22:16:04
<maniscalco> anyhow, I have found that there is a relationship between the distribution of symbols in the child context with regard to the parent context2010-01-24 22:16:42
 more or less symbols in the parent2010-01-24 22:17:03
<Shelwien> well, sure, it can be seen in PPM/CM as well2010-01-24 22:17:14
<maniscalco> right. BWT is like PPM in this way2010-01-24 22:17:33
<Shelwien> like, there're a few kinds of contexts2010-01-24 22:17:37
<maniscalco>  i just cant quite account for the big imrpovements just by adding this to the models2010-01-24 22:17:55
<Shelwien> most important are deterministic, with 1 or ~1 symbol in the alphabet2010-01-24 22:18:10
<maniscalco> i guess it suggets that my original models were not respecting this phenomemon2010-01-24 22:18:13
 right2010-01-24 22:18:21
 i think the same2010-01-24 22:18:25
 its more of a special case2010-01-24 22:18:30
<Shelwien> but there're more special cases2010-01-24 22:18:34
<maniscalco> which happens to be common at high order2010-01-24 22:18:39
<Shelwien> for example, considered dictionary files2010-01-24 22:18:50
<maniscalco> like shkarin does with "binary contexts"2010-01-24 22:18:58
 its really just for special high order cases2010-01-24 22:19:08
<Shelwien> sure, these binary contexts are deterministic2010-01-24 22:19:18
<maniscalco> keeps the general model from getting messed up2010-01-24 22:19:21
<Shelwien> they're only binary because of escape :)2010-01-24 22:19:27
 anyway, i was talking about wordlists2010-01-24 22:19:53
 like english.dic2010-01-24 22:19:57
<maniscalco> well, i think they reflect the special syntax of the language rather then the general rules of the language2010-01-24 22:19:58
 go ahead ...2010-01-24 22:20:06
 there is a thorn for BWT2010-01-24 22:20:15
 im sure i can get this to work for M03 someday2010-01-24 22:20:27
<Shelwien> they're a special case for BWT (and PPM/CM too, but less so)2010-01-24 22:20:32
<maniscalco> but its a special case2010-01-24 22:20:34
<Shelwien> because there's a lot of "garbage" contexts2010-01-24 22:21:00
<maniscalco> i never really thought about it because i know its detectable and a filter can adjust to help bwt2010-01-24 22:21:04
 yes2010-01-24 22:21:07
 and you loose the significance of garbage vs important with BWT2010-01-24 22:21:23
 it cant tell what is important before the transform2010-01-24 22:21:37
 so it mashes them all together2010-01-24 22:21:44
 where as2010-01-24 22:21:46
<Shelwien> yeah, though i never heard of a PPM/CM which would specifically handle such cases2010-01-24 22:22:02
<maniscalco> PPM understands that garbage is at the lower orders2010-01-24 22:22:04
<Shelwien> err, no2010-01-24 22:22:13
 as i said, wordlists are a good example, though the same behavior appears anywhere2010-01-24 22:22:30
<maniscalco> sure, the statistics at lower orders are less effects by the oddities2010-01-24 22:22:34
 because they are normalized more frequently2010-01-24 22:22:46
<Shelwien> i mean, you have repeatable suffixes in words2010-01-24 22:22:47
<maniscalco> that is only part of it though2010-01-24 22:22:54
 yes2010-01-24 22:22:57
<Shelwien> like -ment or -ness, lots of even longer ones2010-01-24 22:22:58
 and with wordlists, most models try to predict the prefix of the next word2010-01-24 22:23:24
 in context of <suffix><LF>2010-01-24 22:23:34
 and its a big mistake2010-01-24 22:23:47
<maniscalco> and do you think that the models tend to default "escape" to lower orders each time they hit "ing" "est"etc2010-01-24 22:23:49
 so they reach the root and re-model quickly ?2010-01-24 22:24:01
<Shelwien> no, that'd be good instead, if they could escape to low orders there2010-01-24 22:24:09
 but normally they see some statistics in high orders2010-01-24 22:24:27
<maniscalco> it would be intersting to have secondary models2010-01-24 22:24:39
<Shelwien> and try to use them, but distributions don't match at all2010-01-24 22:24:48
<maniscalco> which are intended to model "garbage"2010-01-24 22:24:50
<Shelwien> well, ppmonstr2010-01-24 22:25:01
<maniscalco> and find ways of mapping back into sensical posisitions in the normal model2010-01-24 22:25:07
 that is "ing" happens2010-01-24 22:25:13
<Shelwien> has a sparse submodel which kinda can handle that i guess2010-01-24 22:25:13
<maniscalco> but tends to map back to lower order 4-5 contexts2010-01-24 22:25:30
 regardless of how long xxxxxxxxx-ing is 2010-01-24 22:25:40
 but this is language based2010-01-24 22:25:56
 i guess im more interested in solving abitrary signals2010-01-24 22:26:12
 there, bwt is usually god2010-01-24 22:26:19
 good2010-01-24 22:26:20
<Shelwien> well, i think this requires a different approach2010-01-24 22:26:30
<maniscalco> someday, i plan on adding ppm like statistics to M032010-01-24 22:26:43
 its trivial to gather the statistics2010-01-24 22:26:52
<Shelwien> like, specifically detecting garbage contexts by their alphabet size/distributions/entropy2010-01-24 22:27:00
<maniscalco> but its complex (memory wise) to know what context you are currently modelling2010-01-24 22:27:16
 that's why i say a different small model is god2010-01-24 22:27:34
 good2010-01-24 22:27:36
 for english2010-01-24 22:27:47
 an order 3 model would pick up ing, ed(blank) etc2010-01-24 22:28:04
 and if the standard model could respect the predicted changes in the small model2010-01-24 22:28:26
 ....2010-01-24 22:28:29
 then the standard model might predict "escape" better2010-01-24 22:28:46
 regardless of if the state in the standard model is fairly new2010-01-24 22:29:02
 or not2010-01-24 22:29:05
 of dict .... imagine a new context "sort"2010-01-24 22:29:37
 then comes2010-01-24 22:29:41
 "i"2010-01-24 22:29:44
 then "n"2010-01-24 22:29:47
 etc....2010-01-24 22:29:49
 this short context model would have a high indicator that "g" and EOL2010-01-24 22:30:05
 are comming next2010-01-24 22:30:08
<Shelwien> sure2010-01-24 22:30:18
<maniscalco> even though the main model has never seen "sort"2010-01-24 22:30:22
<Shelwien> but the problem is after that :)2010-01-24 22:30:24
<maniscalco> i dont think so2010-01-24 22:30:32
<Shelwien> like, in a sorted wordlist2010-01-24 22:30:35
<maniscalco> localized errors are the problem2010-01-24 22:30:40
 well, for BWT at least2010-01-24 22:30:45
<Shelwien> by history of context "ing<LF>" you would never be able to predict the next symbol2010-01-24 22:31:08
<maniscalco> sorted lists are a good example of this2010-01-24 22:31:08
 right2010-01-24 22:31:18
 but the cost happens at the esacpe2010-01-24 22:31:25
 i think2010-01-24 22:31:27
 with a sorted list2010-01-24 22:31:31
 there is no prediction after LF2010-01-24 22:31:40
<Shelwien> not really, and that's the problem too2010-01-24 22:31:49
<maniscalco> unless you have a human brain2010-01-24 22:31:52
<Shelwien> no, i mean there're not that much different letters2010-01-24 22:32:04
<maniscalco> wait, im wrong2010-01-24 22:32:11
<Shelwien> so after a few escapes2010-01-24 22:32:11
<maniscalco> the order 3 model would pick up on <LF>2010-01-24 22:32:22
<Shelwien> you'd have all the alphabet in the "=ing" context2010-01-24 22:32:25
<maniscalco> and could predict that the next symbol is similar to the last following <LF>2010-01-24 22:32:36
<Shelwien> but probability distribution would be completely wrong2010-01-24 22:32:40
 ...yeah, that's the problem2010-01-24 22:33:03
 the context model _would_ predict that next symbol is similar to the last in context history2010-01-24 22:33:21
<maniscalco> i think that no model will be good at this2010-01-24 22:33:22
 we only see it because we have many years of sampling in our heads2010-01-24 22:33:34
<Shelwien> but there're some2010-01-24 22:33:36
<maniscalco> we have good prediction2010-01-24 22:33:42
 because we are keyed to find patterns2010-01-24 22:33:51
 and language models would require the same advantage of experience to find the same patterns2010-01-24 22:34:11
<Shelwien> http://www.maximumcompression.com/data/dict.php2010-01-24 22:34:12
<maniscalco> what i dont quite understand here is how limited order BWT does so much better than full BWT2010-01-24 22:35:45
<Shelwien> yeah2010-01-24 22:35:57
<maniscalco> usually szip order 4 does much better on these types of data2010-01-24 22:36:02
<Shelwien> due to data specifics :)2010-01-24 22:36:04
<maniscalco> where it is like ppm2010-01-24 22:36:09
 more than bwt2010-01-24 22:36:11
*** pinc has joined the channel2010-01-24 22:36:22
 but M03 is like PPM in that it is context based2010-01-24 22:36:26
 and it doesn't get this advantage2010-01-24 22:36:32
 (its not fair!)2010-01-24 22:36:38
 (^:2010-01-24 22:36:40
<Shelwien> well, i think you can still detect it...2010-01-24 22:36:49
<maniscalco> obviously, because it makes full order predictions 2010-01-24 22:36:58
 sure2010-01-24 22:37:08
 a simple filter can find and destroy2010-01-24 22:37:16
 but that's not the point2010-01-24 22:37:22
 adaptive models are what we want2010-01-24 22:37:32
 not bandaids2010-01-24 22:37:38
<Shelwien> yeah2010-01-24 22:37:57
<maniscalco> which is why i dont have any particular respect for stuff like nanozip2010-01-24 22:38:02
 big deal2010-01-24 22:38:05
 you wrote some filters2010-01-24 22:38:10
 modeling language is the trick2010-01-24 22:38:26
 how do you adapt to localized syntax quickly2010-01-24 22:38:44
 ?2010-01-24 22:38:45
 and here, ppm has the edge2010-01-24 22:38:58
 and cm2010-01-24 22:39:00
<Shelwien> yeah2010-01-24 22:39:10
<maniscalco> because they are not block based2010-01-24 22:39:13
 they adapt2010-01-24 22:39:15
 maybe i could make a BWT like CM2010-01-24 22:39:28
 with respect to lower order BWT2010-01-24 22:39:37
 and full2010-01-24 22:39:39
<Shelwien> i think there're were ideas about weird BWT versions2010-01-24 22:39:45
<maniscalco> M03 isnt "weird" (^:2010-01-24 22:40:02
<Shelwien> like coding a BWT for block of N2010-01-24 22:40:06
 and then coding a "patch" to BWT of block of 2*N2010-01-24 22:40:24
 ...or N+k2010-01-24 22:40:31
<maniscalco> well, certainly M03 would work for this2010-01-24 22:40:41
 since it always knows what the current context is2010-01-24 22:40:49
 but that would require more memory2010-01-24 22:40:54
<Shelwien> yeah2010-01-24 22:41:02
<maniscalco> and one of the main reasons for BWT is limited memory2010-01-24 22:41:05
 so that is the point?2010-01-24 22:41:09
<Shelwien> speed too 2010-01-24 22:41:15
<maniscalco> if you have to use more, use PPM/CM2010-01-24 22:41:17
 sure2010-01-24 22:41:20
 but PPM can do that2010-01-24 22:41:24
 and speed and memory are becoming cheaper by the day2010-01-24 22:41:49
<Shelwien> well, PPM is kinda dead at this point :)2010-01-24 22:41:56
<maniscalco> bzzzt!!!!!2010-01-24 22:42:12
<Shelwien> all thats to Matt :)2010-01-24 22:42:15
<maniscalco> but thanks for playing the game2010-01-24 22:42:22
 come on2010-01-24 22:42:36
 shkarins work is still very strong 2010-01-24 22:42:47
<Shelwien> i mean, we don't have anything beside ppmd2010-01-24 22:42:57
<maniscalco> and fast compared to cm2010-01-24 22:43:00
 why do you need more ?2010-01-24 22:43:18
<Shelwien> because there better CMs now2010-01-24 22:43:31
<maniscalco> as we say in the states he "knocked it out of the park"2010-01-24 22:43:35
 still slow2010-01-24 22:43:40
<Shelwien> no2010-01-24 22:43:46
 ccm is faster than ppmd2010-01-24 22:43:50
 and has better overall compression2010-01-24 22:44:01
<maniscalco> really, i would have to look2010-01-24 22:44:10
 and not at a single file bench mark like large text file2010-01-24 22:44:25
 that's an easy file to compress2010-01-24 22:44:32
<Shelwien> well, considering texts, ccm is relatively bad2010-01-24 22:44:46
 like 220k on book12010-01-24 22:44:51
<maniscalco> that's the only way i have seen it compared, i confess2010-01-24 22:45:08
<Shelwien> but its really good on binaries2010-01-24 22:45:09
<maniscalco> really2010-01-24 22:45:15
 i will look into that2010-01-24 22:45:19
 that is a problem for BWT2010-01-24 22:45:30
<Shelwien> well, there're reasons for that :)2010-01-24 22:45:35
<maniscalco> again, short contexts2010-01-24 22:45:39
 BWT is a great big thing, taken down by small things2010-01-24 22:45:56
 which is where CM comes in2010-01-24 22:46:16
 ... pick you battles wisely ...2010-01-24 22:46:32
 got to go for a bit. cat is hungry2010-01-24 22:47:12
<Shelwien> :)2010-01-24 22:47:16
*** maniscalco has left the channel2010-01-24 22:47:26
*** pinc has left the channel2010-01-24 23:11:30
*** mike_____ has left the channel2010-01-24 23:57:38
*** Shelwien has left the channel2010-01-25 01:18:18
*** Shelwien has joined the channel2010-01-25 01:32:32
*** STalKer-Y has joined the channel2010-01-25 04:19:56
*** STalKer-X has left the channel2010-01-25 04:21:07
*** scott___ has left the channel2010-01-25 05:55:48
*** pinc has joined the channel2010-01-25 08:01:44
*** Shelwien has left the channel2010-01-25 10:01:47
*** Guest9968193 has joined the channel2010-01-25 10:01:50
*** mondragon has joined the channel2010-01-25 11:17:52
*** mondragon has left the channel2010-01-25 11:23:59
*** pinc has left the channel2010-01-25 12:26:55
*** pinc has joined the channel2010-01-25 12:27:03
*** pinc has left the channel2010-01-25 12:27:26
*** pinc has joined the channel2010-01-25 12:27:36
*** pmcontext has joined the channel2010-01-25 15:24:29
<pmcontext> hi ;)2010-01-25 15:26:16
 finally i think i got the mixer working2010-01-25 15:28:47
 yesterday it expanded files2010-01-25 15:29:11
 today its compressing 2010-01-25 15:29:17
 i used o0 + o1 to test it2010-01-25 15:30:13
 and the w is [0,64] , is this ok ? or should it have more range2010-01-25 15:30:51
 currently on book1 it is 358591 , i guess it need more tuning2010-01-25 15:34:48
 ok after little tuning book1 3462242010-01-25 15:40:24
*** mike_____ has joined the channel2010-01-25 15:44:54
 strangly , dw = (( dp*error(b) ) / (p*(ONE-p)) ); gives me bad compression2010-01-25 15:51:33
 compared to dw = (( dp*error(b) )>>L)2010-01-25 15:51:46
 i am running out of tuning ideas , darn 3462242010-01-25 15:56:56
*** scott___ has joined the channel2010-01-25 16:10:03
<scott___> hi2010-01-25 16:11:10
*** mike_____ has left the channel2010-01-25 16:26:07
*** toffer has joined the channel2010-01-25 16:48:56
<toffer> hi2010-01-25 16:49:04
<pmcontext> hi2010-01-25 16:57:41
 at the moment i use dw = (( dp*error(b) )>>L) ;2010-01-25 16:58:09
 there was a small bug , now it is 3518882010-01-25 17:00:12
*** pinc has left the channel2010-01-25 17:00:36
 increased range of w , now 3504572010-01-25 17:05:06
 the mixer seems to be working ,2010-01-25 17:11:44
 using only o0 + o12010-01-25 17:12:00
<toffer> your update formula is still wrong...2010-01-25 17:12:20
<pmcontext> but its not giving good result yet i guess2010-01-25 17:12:31
<toffer> what's plain o1 w/o mixing?2010-01-25 17:13:36
 and what is L2010-01-25 17:13:42
 and you need to use more precision for the weights2010-01-25 17:13:58
<pmcontext> o1 + o0 mix2010-01-25 17:14:00
 L = 17 . and w [0 , 512]2010-01-25 17:14:31
<toffer> thus w=9 bit2010-01-25 17:14:39
 and probs are?2010-01-25 17:14:44
 16 bit?2010-01-25 17:14:48
<pmcontext> 12 bit2010-01-25 17:14:54
<toffer> as i said2010-01-25 17:14:58
 you can increase2010-01-25 17:15:00
 to 162010-01-25 17:15:02
<pmcontext> ok i will do now2010-01-25 17:15:05
<toffer> and weights to 122010-01-25 17:15:08
 erm2010-01-25 17:15:10
 152010-01-25 17:15:12
 thus you have2010-01-25 17:15:22
 dp*error(b) = 24 bit, signed2010-01-25 17:15:31
<pmcontext> increasing w to 12 bit i can 2010-01-25 17:16:08
 but im not sure how to change p2010-01-25 17:16:17
<toffer> and 24-17 = 7 bit2010-01-25 17:16:25
<pmcontext> the probs are stored as n0 , n12010-01-25 17:16:26
<toffer> ...2010-01-25 17:16:43
 these are counts2010-01-25 17:16:46
 not probs2010-01-25 17:16:48
 N bit precision2010-01-25 17:16:55
 P(y=1) = (n1<<N)/(n0+n1)2010-01-25 17:17:08
<pmcontext> dam ok i got it2010-01-25 17:17:11
 i forgot about that2010-01-25 17:17:19
<toffer> i'm away for dinner2010-01-25 17:17:43
<pmcontext> ok2010-01-25 17:19:21
 with w increased to 12 bit 3491232010-01-25 17:20:16
 after p is scaled to 16 bit , since my coder needs 12 bit , should i p>>4 ?2010-01-25 17:23:41
  3488252010-01-25 17:32:31
 working on increasing p2010-01-25 17:36:46
 after i try to increase p 365693, may be i did something bad2010-01-25 17:40:17
 back to 12 bit and 3485302010-01-25 17:49:55
 w is 12 bit , p is 12 bit now2010-01-25 17:50:16
 p 16 bit 3657592010-01-25 17:58:13
 p 15 bit 3558582010-01-25 17:58:14
 p 14 bit 3494912010-01-25 17:58:16
 p 12 bit 3485302010-01-25 17:58:17
*** scott___ has left the channel2010-01-25 18:21:46
<toffer> higher precision give better compression2010-01-25 18:28:58
 the worse compression is due to overflows2010-01-25 18:29:08
 for sure2010-01-25 18:29:12
 *gives better ...2010-01-25 18:29:21
 set weight precision to 15 bit2010-01-25 18:30:05
<pmcontext> ok2010-01-25 18:30:10
<toffer> and probability precision to 16 bit2010-01-25 18:30:18
 in some old m1 i used 20 bit probs2010-01-25 18:30:28
 overflows can happen like that:2010-01-25 18:30:44
 (p2-p1) w + p12010-01-25 18:30:53
 p2-p1 will be 16 bit, signed -> thus 17 bit2010-01-25 18:31:04
 and (p2-p1) w will be 16bit, 1 bit sign + 15 bit (weight) = 32 bit2010-01-25 18:31:35
<pmcontext> w 15 bit 3464552010-01-25 18:31:40
<toffer> so check for overflows in weighting and update2010-01-25 18:31:55
 you can post the implementation, too2010-01-25 18:32:04
<pmcontext> im sure the w is has no overflow problem2010-01-25 18:32:17
 but not sure when i change p2010-01-25 18:32:42
 how do i scale the P after mixing for my encoder , it needs 12 bit p2010-01-25 18:35:16
 if p is now 16 bit . after mixing p = ? bit , and encoder need p = 12 bit2010-01-25 18:35:59
<toffer> your mixing equation, please2010-01-25 18:47:10
<pmcontext> dp=p2-p1;2010-01-25 18:47:42
 p = p1 + ((dp*w)>>W_BIT);2010-01-25 18:47:44
 W_BIT=15;2010-01-25 18:48:01
 int mix(int p1, int p2){2010-01-25 18:48:48
  dp=p2-p1;2010-01-25 18:48:50
  p = p1 + ((dp*w)>>W_BIT);2010-01-25 18:48:51
<toffer> w is initialized to 2^(W_BIT-1)2010-01-25 18:48:53
<pmcontext>  return p;2010-01-25 18:48:53
  }2010-01-25 18:48:54
<toffer> ?2010-01-25 18:49:17
 and the mixer update?2010-01-25 18:49:35
<pmcontext> ----- about w2010-01-25 18:49:48
 W_BIT=15;2010-01-25 18:49:50
 W_ONE=(1<<W_BIT)2010-01-25 18:49:51
<toffer> looks ok precisionwise2010-01-25 18:49:52
<pmcontext> w=W_ONE/2;2010-01-25 18:49:53
 ----------- update2010-01-25 18:50:30
 void update(int b){2010-01-25 18:50:31
  if(p>T && p<(ONE-T)){2010-01-25 18:50:33
  int dw = (( dp*error(b) )>>L) ;2010-01-25 18:50:35
  w = max( 0 , min( w+dw , W_ONE ) );2010-01-25 18:50:36
  }2010-01-25 18:50:38
 }2010-01-25 18:50:39
<toffer> i can tell you what blows higher precision p up2010-01-25 18:51:25
<pmcontext> -------------------------- const2010-01-25 18:51:40
 T=1;2010-01-25 18:51:42
 PRANGE=12;2010-01-25 18:51:44
 ONE=(1<<PRANGE);2010-01-25 18:51:45
 W_BIT=15;2010-01-25 18:51:47
 W_ONE=(1<<W_BIT);2010-01-25 18:51:48
 o.o anything blow ?2010-01-25 18:52:21
<toffer> think about it yourself2010-01-25 18:54:06
 dp*error(b)2010-01-25 18:54:13
 with 32 bit precision2010-01-25 18:54:17
<pmcontext> L=14 currently2010-01-25 18:55:10
 i tried 16 bit for p but it seem to 365778 , with 12 bit it giving3464552010-01-25 18:55:11
<toffer> just think about it2010-01-25 18:56:26
 dp = p2-p1 -> 16 bit with 1 bit sign2010-01-25 18:56:37
<pmcontext> and error is ((b<<PRANGE)-p) 2010-01-25 18:57:31
 where PRANGE=12;2010-01-25 18:57:33
 so it be 12 bit ?2010-01-25 18:57:35
<toffer> you don't understand, i guess2010-01-25 18:57:58
 w82010-01-25 18:58:01
 phoen2010-01-25 18:58:03
 phone2010-01-25 18:58:04
 if p1 and p2 have 16 bit precision2010-01-25 19:16:17
 how much bits does dp=p2-p1 need?2010-01-25 19:16:27
<pmcontext> 16 bit ?2010-01-25 19:16:36
<toffer> 0-2^16 = 2010-01-25 19:16:51
<pmcontext> hm it can also be negtive 2010-01-25 19:17:11
<toffer> -2^16, thus you get a sign bit2010-01-25 19:17:13
 alltogether 17 bit2010-01-25 19:17:21
<pmcontext> so it need 17 bit2010-01-25 19:17:23
<toffer> same for error2010-01-25 19:17:30
 thus2010-01-25 19:17:32
 dp*error = 34 bit2010-01-25 19:17:38
 overflow!2010-01-25 19:17:43
 same for 15 bit2010-01-25 19:17:50
 but ok for 14 -> 30 bit alltogether2010-01-25 19:17:58
 you see it's just an overflow issue2010-01-25 19:18:03
<pmcontext> o.o oh2010-01-25 19:18:27
 34 bit is more then my int 2010-01-25 19:18:43
<toffer> that's why you get worse results with precision higher than 142010-01-25 19:18:44
 my advise2010-01-25 19:19:59
 use 16 bit for p2010-01-25 19:20:03
 and calculate it like that2010-01-25 19:20:14
 the most you can get is 31 bit with a sign bit2010-01-25 19:21:17
<pmcontext> do u think this is bad 2010-01-25 19:21:57
 in mixing 2010-01-25 19:21:59
 p = p1 + ((dp*w)>>W_BIT);2010-01-25 19:22:01
 dp = 17 bit , w = 15 bit 2010-01-25 19:22:02
 so it is 32 bit before i shift with W_BIT;; 2010-01-25 19:22:04
<toffer> mixing is ok2010-01-25 19:22:11
 but the update goes wrong2010-01-25 19:22:21
 (dp+1>>1)*(error+1>>1)2010-01-25 19:22:23
 or (dp+2>>2)*error2010-01-25 19:22:39
 you still need to add g(p)2010-01-25 19:22:47
<pmcontext> but when i added g(p) it gave bad result2010-01-25 19:23:07
<toffer> because you implement it in a wrong fashion2010-01-25 19:23:31
<pmcontext> i used g(p) = 1/(p*(ONE-p))2010-01-25 19:23:57
<toffer> and again2010-01-25 19:24:07
 how much bits of precision do you need for dp*e*g(p)2010-01-25 19:24:23
 you got 322010-01-25 19:24:25
 and if you used 12 bit2010-01-25 19:24:30
 you get2010-01-25 19:24:34
 36 bit alltogether...2010-01-25 19:24:44
 plus a sign bit2010-01-25 19:24:49
<pmcontext> oh o.o2010-01-25 19:24:50
<toffer> and2010-01-25 19:24:55
 your threshold T=12010-01-25 19:25:00
 is far too less2010-01-25 19:25:04
<pmcontext> T=2?2010-01-25 19:25:13
<toffer> ...2010-01-25 19:25:17
 you got 15 bit weights2010-01-25 19:25:27
 i guess something like 2^5 or ...2^8 is reasonable2010-01-25 19:25:43
 you have to try2010-01-25 19:25:52
<pmcontext> oh ok2010-01-25 19:25:58
<toffer> as i said2010-01-25 19:26:08
 first implement it properly instead of using such improper fixes2010-01-25 19:26:20
 than2010-01-25 19:26:27
 you can plot 1/(p(1-p)) and see if it can be approximated with 12010-01-25 19:26:44
 what you did2010-01-25 19:26:48
<pmcontext> ok i will add g(p) and fix the precision2010-01-25 19:27:48
<toffer> ok2010-01-25 19:28:17
 good luck2010-01-25 19:28:39
 i gonna be here some more time2010-01-25 19:28:44
 if you got any questions2010-01-25 19:28:47
<pmcontext> ok thank you, no question yet2010-01-25 19:30:22
<toffer> and keep in mind that (15 bit mantissa, 1 bit sign) * (15 bit mantissa, 1 bit sign) = 30 bit mantissa, 1 bit sign (not 2)2010-01-25 19:32:01
<pmcontext> ok 30 bit and 1 sign bit 2010-01-25 19:32:40
 oh and when the p is 16 bit , i do p>>4 to get 12 bit for encoder ?2010-01-25 19:35:56
<toffer> yes2010-01-25 19:36:28
 but you can use higher precisions, too2010-01-25 19:36:40
 which helps2010-01-25 19:36:44
 but first make the calculations work2010-01-25 19:36:49
<pmcontext> ok2010-01-25 19:36:54
<toffer> and compare to order1 only2010-01-25 19:39:01
<pmcontext> dw = (( dp*error(b) ) / (p*(ONE-p)) ) >> L2010-01-25 19:45:23
 is this look ok ? i mean the formula2010-01-25 19:45:25
 im still working on precision2010-01-25 19:45:26
<toffer> dp is?2010-01-25 19:48:21
 p2-p1?2010-01-25 19:48:38
<pmcontext> dp is p2 - p12010-01-25 19:48:44
<toffer> ok2010-01-25 19:48:48
 error is?2010-01-25 19:48:52
<pmcontext> ((b<<PRANGE)-p)2010-01-25 19:49:02
<toffer> your terminology confuses me a bit2010-01-25 19:49:13
 it'd be better if you make it a bit clearer2010-01-25 19:49:26
 like P_BITS, P_ONE=1<<P_BITS2010-01-25 19:49:37
 and so one2010-01-25 19:49:39
 W_BITS, ...2010-01-25 19:49:42
 ONE is?2010-01-25 19:49:47
<pmcontext> b is bit2010-01-25 19:49:54
 prange , precision of P2010-01-25 19:49:55
 and p is prediction we got after mix2010-01-25 19:49:57
 PRANGE=14;2010-01-25 19:50:10
<toffer> i mean ONE is 1<<PRANGE?2010-01-25 19:50:10
<pmcontext> ONE=(1<<PRANGE);2010-01-25 19:50:11
<toffer> ok than2010-01-25 19:50:14
<pmcontext> yes2010-01-25 19:50:16
<toffer> so T is2010-01-25 19:50:17
<pmcontext> 1<<52010-01-25 19:50:26
 T = 322010-01-25 19:50:45
<toffer> looks ok2010-01-25 19:50:49
 and did you made some sign checks2010-01-25 19:50:58
 i.e. not w - dw 2010-01-25 19:51:08
<pmcontext> w + dw2010-01-25 19:51:19
<toffer> did we define dw = dH/dw or dw=-dH/dw2010-01-25 19:51:48
 it hast to read w - dH/dw for minimization2010-01-25 19:52:27
 so check your signs2010-01-25 19:53:21
<pmcontext> i duno but i rember this2010-01-25 19:54:06
 dw = (p2-p1) * (y-p) * g(p) , its value is positive if i get y=1 and p2 is greater, 2010-01-25 19:54:08
 that means w should increase to give more weight to p2 model2010-01-25 19:54:09
 then i have to add , w + dw2010-01-25 19:54:11
<toffer> i checked it again2010-01-25 19:58:34
 dH/dw = - dp/dw (y-p) 1/(p(1-p))2010-01-25 19:58:53
 and dw = -dH/dw2010-01-25 19:59:02
 thus it's corect2010-01-25 19:59:08
<pmcontext> ok i pass :D2010-01-25 19:59:27
<toffer> H = -( y ln p + (1-y) ln (1-p) -> dH/dw = dp/dw (y-p) 1/(p(1-p)) and dp/dw = (p2-p1), iff p=(p2-p1)w + p12010-01-25 20:00:31
 H = -( y ln p + (1-y) ln (1-p) -> dH/dw = - dp/dw (y-p) 1/(p(1-p)) and dp/dw = (p2-p1), iff p=(p2-p1)w + p12010-01-25 20:00:49
 anyway2010-01-25 20:00:53
 your formula is correct2010-01-25 20:00:57
 w' = w - L dH/dw = w - L( - dp/dw (y-p) 1/(p(1-p)) )2010-01-25 20:01:21
<pmcontext> w' = w + L dp/dw (y-p) 1/(p(1-p)) :D2010-01-25 20:02:50
 fixing presicion is hard , may be i increase one bit at a time2010-01-25 20:03:42
 when p is 13 bit , it improved compression2010-01-25 20:04:06
 when i put 14 bit it went lil bad2010-01-25 20:04:15
 p 13 bit 3651512010-01-25 20:06:27
<toffer> no wonder...2010-01-25 20:06:52
 your update formula again,please2010-01-25 20:07:10
<pmcontext> int dw = (( dp*error(b) ) / (p*(ONE-p)) );2010-01-25 20:07:22
 w = max( 0 , min( w+dw , W_ONE ) );2010-01-25 20:07:42
<toffer> p(1-p) has 26 bit precision2010-01-25 20:08:21
<pmcontext> yes2010-01-25 20:08:34
 but isnt it 27 bit ?2010-01-25 20:08:50
<toffer> no2010-01-25 20:08:53
 it's always positive2010-01-25 20:08:58
 and dp*error(b) has 27 bit2010-01-25 20:09:03
<pmcontext> oh yes2010-01-25 20:09:03
<toffer> 26 bit mantissa2010-01-25 20:09:11
 thus 26bit - 26 bit = how many bit of adjustment?2010-01-25 20:09:30
 you see2010-01-25 20:09:37
<pmcontext> 0 bit o.o ?2010-01-25 20:09:46
<toffer> right2010-01-25 20:09:48
 maybe due to roundin 1 bit2010-01-25 20:09:54
 rounding2010-01-25 20:09:57
<pmcontext> yes could be2010-01-25 20:10:10
<toffer> as you remember2010-01-25 20:10:17
 T = 1<<52010-01-25 20:10:22
<pmcontext> yes2010-01-25 20:10:27
<toffer> thus p(1-p) is guaranteed to have 10 bit, at least2010-01-25 20:10:35
 right?2010-01-25 20:10:37
<pmcontext> yes2010-01-25 20:10:45
<toffer> si you can scale it down a bit2010-01-25 20:10:58
 so2010-01-25 20:11:00
<pmcontext> dw >> L ?2010-01-25 20:11:25
<toffer> ... / ( (p*(ONE-p)) >> 8 )2010-01-25 20:11:26
 no2010-01-25 20:11:29
<pmcontext> oh2010-01-25 20:11:29
<toffer> proper rounding can be important, too2010-01-25 20:12:00
 (... + (1<<7)) >> 82010-01-25 20:12:11
<pmcontext> ah i see2010-01-25 20:12:11
<Shelwien> hi2010-01-25 20:12:58
 also i think its better to use divisions in experimental formulas2010-01-25 20:13:39
 like <pmcontext> p = p1 + ((dp*w)>>W_BIT);2010-01-25 20:14:01
 with negative dp it would become -1 max2010-01-25 20:14:34
<pmcontext> hi shelwien2010-01-25 20:15:04
<Shelwien> and 0 with positive, which is asymmetric2010-01-25 20:15:08
<pmcontext> OMG awsome i did p*(ONE-p)) >> 8 and suddenly it is at 346355, came down from 365k2010-01-25 20:15:23
 ((p*(ONE-p)) >> 8)2010-01-25 20:15:49
 p*(ONE-p)) >> (7+1)2010-01-25 20:16:07
 sorry bad typing2010-01-25 20:16:30
<toffer> hi eugene2010-01-25 20:17:51
 i got o1 decomposition working2010-01-25 20:17:56
 with proper decompression2010-01-25 20:18:04
 speed increased by 700kb/s from 1.8mb/s to 2.5mb/s2010-01-25 20:18:15
<Shelwien> good i guess... though did I benchmark that before? guess not...2010-01-25 20:19:20
<toffer> nope2010-01-25 20:19:30
 compression on big files improved2010-01-25 20:19:37
 the o1 decomposition virtually replaces an o1 moel2010-01-25 20:19:44
 model2010-01-25 20:19:46
 but on short files saving the codelength hurts2010-01-25 20:19:56
 i'm working on that2010-01-25 20:20:01
<Shelwien> maybe disable it for short files? :)2010-01-25 20:20:17
<toffer> i can still drop to o02010-01-25 20:22:28
 or to flat decomposition2010-01-25 20:22:34
 currently i'm just storing the code lengths2010-01-25 20:22:51
*** pinc has joined the channel2010-01-25 20:22:54
 i experimented with masked decompositions2010-01-25 20:23:19
 0x40df gives 0.1 bpc better "decomposition compression" for text on e7/e82010-01-25 20:23:45
<Shelwien> err... like partial o1 or what?2010-01-25 20:23:46
 ah2010-01-25 20:23:56
<toffer> the speedup is about 2..3%2010-01-25 20:23:59
 compared to plain o12010-01-25 20:24:03
 on the other hand2010-01-25 20:24:05
 o1 is better elsewhere2010-01-25 20:24:16
<Shelwien> well, surely its more general2010-01-25 20:24:53
 there're russian (and finnish) texts in my benchmark, so 40DF might not be that good...2010-01-25 20:25:16
<toffer> atm i'd just keep o12010-01-25 20:25:34
 and maybe change the coder structure to add a parameter for that2010-01-25 20:25:45
 currently the decomposition is constructed outside of the actual compressor2010-01-25 20:26:00
 as to speedup2010-01-25 20:26:05
<Shelwien> "Francesco used Huffman coding before arithmetic coding is his Rings compressor. That increased both speed and strength of compression."2010-01-25 20:26:28
<toffer> plain: 1.8mb/s, o0: 2.2mb/s, o1: 2.5,b/s2010-01-25 20:26:30
<Shelwien> did you see that? :)2010-01-25 20:26:32
<toffer> i read it a while ago2010-01-25 20:26:43
 code greatly simplified,too2010-01-25 20:28:36
 since the previous nibble caching is highly distorted with variable length codes2010-01-25 20:28:52
 the hit rate dropped fromo 50..70% to 10%2010-01-25 20:29:05
 i abandoned the 1% speedup2010-01-25 20:29:18
 which was left from that2010-01-25 20:29:23
<Shelwien> btw, maybe o1 decomposition can help with hashing too?2010-01-25 20:29:37
<toffer> virtually the hash table is twice as large2010-01-25 20:29:49
 since it stores 3.92 bits per char2010-01-25 20:30:00
 for e72010-01-25 20:30:02
 instead of 82010-01-25 20:30:05
 bpc2010-01-25 20:30:19
 with just 32 mb i can already compress e8 to 21 4xx xxx2010-01-25 20:32:00
 previously i needed something like 100mb for that2010-01-25 20:32:15
 and o1 helps with hashing greatly, since almost every symbol is coded with just a single random memory access2010-01-25 20:32:59
<Shelwien> well, why don't we just benchmark it?2010-01-25 20:35:13
 or does it require more tuning first?2010-01-25 20:35:28
<toffer> it's untuned2010-01-25 20:35:33
 and i first want to make code lengths compressed2010-01-25 20:35:44
<Shelwien> err... do you store them uncompressed now? %)2010-01-25 20:36:06
<toffer> at the moment they require 256*128 bit2010-01-25 20:36:09
 as i said2010-01-25 20:36:13
 i just worked on the algoritihms2010-01-25 20:36:18
 yes2010-01-25 20:36:27
 algorithms2010-01-25 20:36:32
<Shelwien> well, for me some o0 coding is usually easier that bit packing :)2010-01-25 20:37:07
 because i already have rc and counter classes anyway, but no bit i/o :)2010-01-25 20:37:37
<toffer> i know2010-01-25 20:38:04
 i didn't want to do something like that2010-01-25 20:38:12
 i wanted to store lengths hierarchically at least2010-01-25 20:38:31
 and i guess i'd use ordinary stationary counters there2010-01-25 20:39:16
<Shelwien> well, i wonder whether a binary tree would be better there2010-01-25 20:39:56
 or some coding of length array2010-01-25 20:40:03
 as to that, jpeg has some kinda funny lengthtable coding2010-01-25 20:40:42
 they sorted all the lengths there, from 0 to max2010-01-25 20:41:12
 and for each length a number of symbols encoded instead (runlength)2010-01-25 20:41:52
 of course, 0 if length is not used too2010-01-25 20:42:10
 nice idea imho, but implies an alphabet permutation2010-01-25 20:42:56
<pmcontext> i g2g and thank you2010-01-25 21:12:53
*** pmcontext has left the channel2010-01-25 21:13:11
<Shelwien> still, i wonder what would be the effect2010-01-25 21:35:00
 on a file with random blocks in it2010-01-25 21:35:40
<toffer> somehow it badly works2010-01-25 21:38:48
 guess i gonna do some basic o1 coding2010-01-25 21:38:57
 or something like that2010-01-25 21:39:00
<Shelwien> no, i mean how it would affect your o1 decomposition2010-01-25 21:39:19
<toffer> but i just found that directly skipping deterministic o1 contexts improves speed2010-01-25 21:39:29
 no wonder2010-01-25 21:39:43
 but i didn't expect o1 contexts to have determinism2010-01-25 21:39:53
<Shelwien> even o0 can :)2010-01-25 21:40:13
<toffer> very unlikely2010-01-25 21:40:43
 but yes2010-01-25 21:40:45
<Shelwien> anyway, i was thinking about blockwise-adaptive decomposition2010-01-25 21:41:51
 like, you can store freqs for some kinda blocks2010-01-25 21:42:55
<toffer> first i need to get storage working good enough2010-01-25 21:42:57
<Shelwien> ...and then reencode the contexts on access or something2010-01-25 21:43:34
 ...guess that won't make much sense because of speed issues2010-01-25 21:44:05
 which means that you'd need some segmentation instead2010-01-25 21:44:31
<toffer> anyway for me the priority is like that2010-01-25 21:44:49
 1. get things woring2010-01-25 21:44:52
 working2010-01-25 21:44:57
 2. make them pretty2010-01-25 21:44:59
<Shelwien> ...and a match model unaffected by segmentation2010-01-25 21:45:01
<toffer> i could post a list of code lengths2010-01-25 21:46:19
 and see if anybody has some good ideas2010-01-25 21:46:28
 but well...2010-01-25 21:46:33
<Shelwien> you can do that, but any feedback is unlikely :)2010-01-25 21:46:50
<toffer> yep2010-01-25 21:47:13
 guess i gonna do some weight training now2010-01-25 21:47:56
 and i think the compressor for code lengths will be more difficult than the main moel2010-01-25 21:48:08
 model2010-01-25 21:48:10
 ^^2010-01-25 21:48:12
<Shelwien> "weight training" always sounds like running the optimizer to me :)2010-01-25 21:48:41
<toffer> well maybe it's flushing the optimizer between the ears2010-01-25 21:49:07
 ^^2010-01-25 21:49:11
<Shelwien> and as to codelengths... it could be more troublesome if you really had to encode blockwise tables2010-01-25 21:49:44
 but for a single table there's too liitle stats to make anything complex2010-01-25 21:50:16
<toffer> there's much correlation betwenn adjacent contexts2010-01-25 21:50:18
 thus i gonna take care of it2010-01-25 21:50:30
<Shelwien> err... don't forget about symmetry :)2010-01-25 21:51:01
 i mean, number of XY context affects the number of YZ contexts etc :)2010-01-25 21:51:29
 for o1 it should be still manageable... but i really got stuck with high-order context in ctx :)2010-01-25 21:52:11
 btw toffer2010-01-25 22:35:28
 i've got a new interesting testfile :)2010-01-25 22:35:42
 maybe two, even2010-01-25 22:35:48
 there's a 40M exe from microsoft sql server2010-01-25 22:36:09
 and a 70M pdb file for it :)2010-01-25 22:37:41
 unlike acrord32 etc, there're no images or deflate streams, just lots of various tables and x86 code :)2010-01-25 22:38:53
*** pinc has left the channel2010-01-25 23:03:38
<toffer> good leg training with just dumbbells is really annoying2010-01-25 23:17:41
 but the optimzier flush seemed to work2010-01-25 23:18:24
<Shelwien> any new ideas?2010-01-25 23:18:58
<toffer> well some good contexts for storing code lengths2010-01-25 23:19:57
 even if i'd just compress o1 stats to 1/4 of their original size2010-01-25 23:20:13
 it'd pay off compression-wise already2010-01-25 23:20:19
<Shelwien> well, it'd still be redundant2010-01-25 23:20:46
<toffer> i can upload a sample 2010-01-25 23:21:06
<Shelwien> if you won't use that info somehow to reduce the codelength of actual data2010-01-25 23:21:07
<toffer> if you want2010-01-25 23:21:08
<Shelwien> well, why not2010-01-25 23:23:51
 did you try compressing them with paq8 for redundancy estimation btw?2010-01-25 23:24:11
<toffer> no2010-01-25 23:24:15
 it's in the /dcc folder2010-01-25 23:27:23
<Shelwien> oh, you remember where it is now? :)2010-01-25 23:27:49
<toffer> as i said2010-01-25 23:27:57
 it is in the log of my univeristies pc2010-01-25 23:28:04
 (explorer2010-01-25 23:28:09
 )2010-01-25 23:28:11
<Shelwien> now, what about uploading it again in the form of 64k byte table? :)2010-01-25 23:29:15
<toffer> i usually use texts since i make some plots with octave2010-01-25 23:30:57
 w82010-01-25 23:30:58
 i guess you have a hex editor2010-01-25 23:31:28
 any idea why that doesn't work2010-01-25 23:38:50
  putc('a'+tmp[i], to);2010-01-25 23:38:54
  putc('a'+tmp[i+1], to);2010-01-25 23:38:55
 // printf( "%2u %2u ", tmp[i], tmp[i+1] );2010-01-25 23:38:57
 // if (((i+1)&31)==31) { putchar('\n'); }2010-01-25 23:38:58
 it should output a, b, ...2010-01-25 23:39:06
 but it doesn't2010-01-25 23:39:11
<Shelwien> what's the file mode?2010-01-25 23:39:48
<toffer> well finally2010-01-25 23:41:08
 it was just written to the compressed file2010-01-25 23:41:16
 instead of stdout2010-01-25 23:41:20
 it's there now2010-01-25 23:42:32
<Shelwien> ok... meanwhile i nearly finished a perl script to do the same :)2010-01-25 23:43:03
<toffer> that was just a pretty dumb mistake2010-01-25 23:43:30
 reminds me of reimplementing hashing2010-01-25 23:43:43
 and comparing u8 to u162010-01-25 23:43:47
 wondering why the comparsion failed almost always2010-01-25 23:43:57
 ^^2010-01-25 23:43:59
<Shelwien> well, i can't see any problem in your quote :)2010-01-25 23:44:41
<toffer> to is the destination file2010-01-25 23:44:52
 not stdout2010-01-25 23:44:57
<Shelwien> sure, but isn't that what you posted?2010-01-25 23:46:01
<toffer> i was dumping it like that2010-01-25 23:46:48
 m1 3 ..\..\enwik7.txt \testset\enwik\enwik7 nul > enwik7_len.bin2010-01-25 23:46:54
 even with just o0 coding2010-01-25 23:47:22
 that stuff can be compressed alot2010-01-25 23:47:30
 guess some simple model will be sufficient2010-01-25 23:47:58
<Shelwien> undef $/;2010-01-25 23:51:46
 open I, "<enwik7_len.txt";2010-01-25 23:51:47
 binmode I;2010-01-25 23:51:47
 $a = <I>;2010-01-25 23:51:47
 close I;2010-01-25 23:51:47
 $a =~ s/[\x00-\x20]+$//;2010-01-25 23:51:47
 $a =~ s/^[\x00-\x20]+//;2010-01-25 23:51:48
 @b = split /[\x00-\x20]+/, $a;2010-01-25 23:51:50
 $c = join "", map chr, @b;2010-01-25 23:51:53
 open O, ">enwik7_len0.bin";2010-01-25 23:51:54
 binmode O;2010-01-25 23:51:57
 print O $c;2010-01-25 23:51:58
 close O;2010-01-25 23:52:00
<toffer> i hate perl2010-01-25 23:54:18
 as you might remember2010-01-25 23:54:24
 i tried to learn it for code generation2010-01-25 23:54:32
<Shelwien> well, i don't really understand it myself2010-01-25 23:54:56
 also i had to use some windows perl version here2010-01-25 23:55:28
 which required these "binmode" lines because of LF/CRLF stuff2010-01-25 23:55:49
 but still i don't know anything more efficient for text manipulation2010-01-25 23:58:42
<toffer> afair it was intended exactly for that2010-01-25 23:59:47
<Shelwien> $c = join "", map chr($_+ord('a')), @b;2010-01-25 23:59:54
 for example, this produces exactly the same file like you uploaded2010-01-26 00:00:15
*** scott___ has joined the channel2010-01-26 00:05:37
 anyway, somehow your version with +'a' really results in better compression with paq2010-01-26 00:05:46
 6634 vs 6655 using paq8p -72010-01-26 00:06:10
 6291 vs 6334 using paq8p -7 and with added grayscale bmp headers2010-01-26 00:06:36
<toffer> so that's already 1/52010-01-26 00:08:09
 i guess reaching 1/4 with specialised compression will be possible2010-01-26 00:08:50
<Shelwien> actually paq is usually fairly bad at table compression2010-01-26 00:16:16
 so a tuned simple specialized coder should do better than paq2010-01-26 00:16:39
*** scott___ has left the channel2010-01-26 00:25:13
<toffer> hm just storing it hierarchically2010-01-26 00:26:16
 already gets it down to 10k2010-01-26 00:26:20
<Shelwien> note that paq just sequentially compresses it2010-01-26 00:26:42
<toffer> yep2010-01-26 00:27:12
 guess there's really alot of room for improvement2010-01-26 00:27:21
<Shelwien> ...and bmp looks symmetric, as expected...2010-01-26 00:27:35
<toffer> symmetric?2010-01-26 00:28:05
<Shelwien> http://nishi.dreamhosters.com/u/enwik7_le.bmp2010-01-26 00:28:24
<toffer> top left is codelen[0][0] and bottom right is [255][255] ?2010-01-26 00:29:14
 i don'T see alot of symmetry2010-01-26 00:30:35
 but i see what i saw when looking at the dumps already2010-01-26 00:30:44
<Shelwien> diagonal symmetry2010-01-26 00:30:46
<toffer> nearby contexts are correlated2010-01-26 00:30:49
 diagonal?2010-01-26 00:31:14
<Shelwien> XY vs YX2010-01-26 00:33:03
<toffer> locally maybe2010-01-26 00:33:19
<Shelwien> its 0;0 is likely bottom left btw2010-01-26 00:33:33
<toffer> a scanline left to right is a context?2010-01-26 00:35:24
 now i know why i didn't see the symmetry2010-01-26 00:36:07
 wrong layout interpretation2010-01-26 00:36:14
 well when looking at the output2010-01-26 00:38:04
 i found correlations at i-256 and i-12010-01-26 00:38:14
 i mean just by looking at the text output2010-01-26 00:38:35
 guess it's pretty clear in that image2010-01-26 00:38:41
<Shelwien> :)2010-01-26 00:38:49
<toffer> anyway2010-01-26 00:54:19
 i gonna sleep now2010-01-26 00:54:23
 guess tomorrow there gonna be something to tune/test2010-01-26 00:54:33
 but there's no match model in the decomposition branch2010-01-26 00:54:51
 yet2010-01-26 00:55:00
 gn82010-01-26 00:55:11
*** toffer has left the channel2010-01-26 00:55:38
<Shelwien> !next2010-01-26 01:24:27