* Shelwien wonders if it would crash now2009-08-22 15:48:06
*** Skymmer has joined the channel2009-08-22 16:08:16
<Shelwien> hi2009-08-22 16:09:26
 there's sh1c to test ;)2009-08-22 16:09:34
<Skymmer> Hello!2009-08-22 16:11:26
 Tested it already. Good results!!!2009-08-22 16:11:37
  CCM v1.30c = 71.7092009-08-22 16:12:00
  ccm_sh1c = 71.9402009-08-22 16:12:00
<Shelwien> ah, ok2009-08-22 16:12:19
<Skymmer> Time is equal. Very nice :)2009-08-22 16:12:23
<Shelwien> there're still ugly references left2009-08-22 16:12:42
 so it should be even faster next time2009-08-22 16:13:01
<Skymmer> Hope so.2009-08-22 16:13:29
*** FuSiyuan has left the channel2009-08-22 16:54:07
<Simon|B> oh nice result ;)2009-08-22 16:59:58
<osman> hi everyone2009-08-22 17:04:12
 just came from the dinner :)2009-08-22 17:04:21
 what's up? :)2009-08-22 17:04:32
<Shelwien> what's up is up ;)2009-08-22 17:19:13
<osman> i'm working on your 1c2009-08-22 17:24:42
 seems it's much better2009-08-22 17:24:47
 i'm merging them together2009-08-22 17:24:53
<Shelwien> %)2009-08-22 17:25:10
<osman> i feel that we've almost done it2009-08-22 17:25:12
<Shelwien> do you know how that log2 table is used there?2009-08-22 17:26:12
<osman> still working :)2009-08-22 17:29:15
 you mean log2rem?2009-08-22 17:53:49
 shelwien?2009-08-22 17:53:52
 it's intend for match quantization.2009-08-22 17:56:45
 btw, how can i detect ICC with #ifdef?2009-08-22 17:57:00
 #ifdef ICC?2009-08-22 17:57:05
<Shelwien>  #if (__INTEL_COMPILER)2009-08-22 17:57:27
 #if defined(__INTEL_COMPILER)2009-08-22 17:57:55
<osman> seems msvc like __restrict form ;)2009-08-22 18:00:08
 so, i need this #ifdef block2009-08-22 18:00:24
 @shelwien: seems all variables have a new name now ;)2009-08-22 19:17:36
 btw, what's that useless table???2009-08-22 19:17:50
 dword_4092F42009-08-22 19:17:51
 it's initialized only for two possible values2009-08-22 19:18:11
 0 and 0xFFF0002009-08-22 19:18:15
 and it's used as a look-up for each bit2009-08-22 19:18:32
 do we really need it?2009-08-22 19:18:45
 i've changed it to x?y:z block2009-08-22 19:19:00
<Shelwien> as a table? dunno2009-08-22 19:19:01
<sami> I edited the wikipedia: http://en.wikipedia.org/wiki/Comparison_of_file_archivers#Comparison_of_efficiency and yes, comp.ratings is mentioned first because I didn't want to be a hypocrite, I think the most useful site of these. I just copy/pasted the rest of the links to there from the bottom of the page and added the original comparison at the top2009-08-22 19:19:13
<Shelwien> well, not sure about your idea2009-08-22 19:19:19
 i'd use -bit&0xFFF000 or something2009-08-22 19:19:32
<osman> look at ccm_sh2009-08-22 19:19:32
 look at ccm_sh.cpp2009-08-22 19:19:36
 i'm leaving as logic block for better understanding2009-08-22 19:20:18
 we could change it later2009-08-22 19:20:21
 now renaming completed :)2009-08-22 19:23:12
 there are still some redundant code2009-08-22 19:23:12
 i'll clean them more2009-08-22 19:23:17
 and upload it tonight2009-08-22 19:23:23
<Shelwien> ...2009-08-22 19:23:34
*** Shelwien has left the channel2009-08-22 19:45:37
*** Skymmer has left the channel2009-08-22 19:52:58
*** Shelwien has joined the channel2009-08-22 20:04:00
<osman> only small amount of code has left %)2009-08-22 20:05:45
 approximately 50 lines at most2009-08-22 20:05:51
<Shelwien> not quite small...2009-08-22 20:06:07
<chornobyl> did you lost the rest:)2009-08-22 20:06:24
<osman> no :)2009-08-22 20:07:02
 i didn't get currenly match window indexing2009-08-22 20:07:18
 there are two different pointer2009-08-22 20:07:25
 which are addressed to almost same location (+0 and +4)2009-08-22 20:07:49
 what's is intend for?2009-08-22 20:08:01
 *what is2009-08-22 20:08:11
<toffer> win[offset-4]==win[matchpos-4] -> match o4 context2009-08-22 20:11:40
<osman> ah...ok... got it now2009-08-22 20:12:59
*** chornobyl has left the channel2009-08-22 20:21:03
<Surfer> 662009-08-22 20:26:32
*** Surfer has left the channel2009-08-22 20:26:36
<toffer> well i got an improvement for you (ccm)2009-08-22 20:56:17
 which doesn't effect compatibility2009-08-22 20:56:29
 instead of using lzp pointers2009-08-22 20:56:36
 replace these uint32 with actual 4 byte contex2009-08-22 20:56:46
 context2009-08-22 20:56:48
 since ccm only uses 4 byte context confirumation, thus you can drop the window2009-08-22 20:57:04
 in my experience doing a full string comparison isn't any slower but gives notably better results2009-08-22 20:57:35
 you may want to store an offet there in addition to detect distance > 0x2002009-08-22 20:58:16
 i tried similar stuff some time ago, wo success... it turned out that using a window would be faster - maybe it's not the case this time2009-08-22 20:59:12
 w/o2009-08-22 20:59:18
*** Skymmer has joined the channel2009-08-22 21:12:23
*** Skymmer has left the channel2009-08-22 21:12:36
<osman> 13 lines left %)2009-08-22 21:29:55
*** osman has left the channel2009-08-22 22:22:05
<Simon|B> last line killed him :-D2009-08-22 22:29:10
*** Simon|B has left the channel2009-08-22 22:35:13
<Shelwien> btw sami2009-08-22 22:37:00
 i had to do a listening test today2009-08-22 22:37:28
 and there were 32-bit uncompressed wavs for some reason2009-08-22 22:38:02
 so, i've got an idea about another kind of testsets for compressors2009-08-22 22:38:26
<sami> a listening test?2009-08-22 22:38:55
<Shelwien> well, we're developing a audio codec if you remember2009-08-22 22:39:24
<sami> ok2009-08-22 22:39:32
<Shelwien> anyway, my idea is not directly related2009-08-22 22:39:48
 its just that i don't really know any testsets intended for format support testing2009-08-22 22:40:26
 and i think that at least wav and bmp corpora would be relevant already2009-08-22 22:41:31
 maybe even jpeg, as there're at least 4 implementations already (stuffit, winzip, paq8, packjpg)2009-08-22 22:42:35
 i mean, files with various format settings2009-08-22 22:43:13
 but with content to compress too2009-08-22 22:43:36
<sami> you mean wav with 8-bit, 16, 24, etc?2009-08-22 22:43:53
*** osman has joined the channel2009-08-22 22:44:03
<Shelwien> yeah, and there're also big-endian2009-08-22 22:44:15
<sami> to my knowledge wav cannot be big-endian2009-08-22 22:44:28
<Shelwien> tell that to apple2009-08-22 22:44:35
<sami> apple switched to little-endian2009-08-22 22:44:55
<Shelwien> yeah, but there're still occasional big-endian files2009-08-22 22:45:18
<sami> anyway. I think it is ok idea. although I still lack a special audio benchmark (with audio only compressors included)2009-08-22 22:45:50
<Shelwien> sure. I probably could even write something for that ;)2009-08-22 22:46:39
<sami> perhaps a jpg test is ok as well. although winzip and stuffit cannot be tested. I've never tested packjpg but if it has cli interface it can be tested2009-08-22 22:47:09
<Shelwien> winzip has a working commandline version afaik2009-08-22 22:47:42
 at least that's what i was decompiling ;)2009-08-22 22:47:52
<sami> http://compressionratings.com/i_winzip.html2009-08-22 22:48:01
*** Skymmer has joined the channel2009-08-22 22:48:04
<Shelwien> huh? it doesn't require anything2009-08-22 22:48:31
 well, some cracking maybe, to make it work without installing2009-08-22 22:48:50
<sami> winzip sent me a eval key but I told them I can use it if I can mirror it in the site, and they refused2009-08-22 22:49:29
<Shelwien> afair there was a separate CLI package or something2009-08-22 22:49:42
<sami> I mean, the program, cli, after installed or unzipped, whatever, asks for user input when you compress files2009-08-22 22:50:22
 I mean always, and only the registered version doesn't have that2009-08-22 22:50:44
<Shelwien> wzunzip doesn't for sure...2009-08-22 22:51:03
 well, i can crack it if that would help ;)2009-08-22 22:51:18
<sami> ok I might recall incorrectly, but the compression did ask that for sure2009-08-22 22:51:27
<Shelwien> though i guess i installed a torrentbay version anyway ;)2009-08-22 22:51:57
 ah, btw, maybe it would work with echo something | wzzip ?2009-08-22 22:53:57
<sami> I have given some thought about it before, but I guess it would only pose problems for me and I don't really care that much if some programs are intentionally made such that they cannot be tested2009-08-22 22:54:27
<Shelwien> well, i don't really care about testing of GUI archivers anyway ;)2009-08-22 22:55:03
<sami> probably I would not support such echo hack anyway, but I bet it doesn't work2009-08-22 22:55:34
<Skymmer> @sami: By the way, latest StuffIt has CLI2009-08-22 22:55:46
*** Simon|B has joined the channel2009-08-22 22:55:49
 You can test it if you want2009-08-22 22:55:57
 http://my.smithmicro.com/downloads/files/StuffIt2009-Multi.exe2009-08-22 22:55:59
<sami> Skymmer, thanks, I'll test that tomorrow2009-08-22 22:56:33
<Skymmer> You're welcome :)2009-08-22 22:56:50
 Though its CLI is ugly for my taste and bad documented2009-08-22 22:57:27
<Simon|B> has the complogger any functionality?2009-08-22 22:57:32
<sami> squeez has weird problems as well: http://compressionratings.com/i_squeez.html but at least it can be tested2009-08-22 22:57:52
<Shelwien> Simon: tell me what you want ;)2009-08-22 22:58:01
<Simon|B> something like requesting last 10,50 comments would be fine2009-08-22 22:58:20
<Shelwien> by PM2009-08-22 22:58:34
 ?2009-08-22 22:58:35
<Simon|B> my browser died and I wanted to see what was written2009-08-22 22:58:47
 yes2009-08-22 22:58:49
 I think that would be easiest one?!?2009-08-22 22:59:06
 ok there is a link to a up to date text file right?2009-08-22 23:00:05
<Shelwien> there's a small problem - there're more than one file2009-08-22 23:00:42
 current one is http://nishi.dreamhosters.com/log/2009-08-22_15-47-52.txt2009-08-22 23:01:03
 i also added it to the bot and sent the packlist to you ;)2009-08-22 23:01:39
*** toffer has left the channel2009-08-22 23:02:04
<Simon|B> could you create a static link for the last X comments? That would be perfect too2009-08-22 23:03:08
<Shelwien> probably can write a separate script for that, yeah2009-08-22 23:03:35
 wait a moment2009-08-22 23:03:53
 ok, what about this: http://nishi.dreamhosters.com/chantail.pl?102009-08-22 23:14:21
*** sami has left the channel2009-08-22 23:18:55
*** Simon|B has left the channel2009-08-23 00:10:16
<Skymmer> Guys, what is the strongest PPMII realisation? I mean compression ratio?2009-08-23 00:55:23
 I've just been little bit interested in PPM2009-08-23 00:55:40
<Shelwien> ppmonstr/durilc2009-08-23 00:55:45
 there's also a decompiled source of ppmonstr vI2009-08-23 00:56:16
<Skymmer> And compiled one? :)2009-08-23 00:56:49
<Shelwien> yeah, same as ccm2009-08-23 00:57:02
<Skymmer> You mean ppmd_Jr1_sh82009-08-23 00:57:13
<Shelwien> nah, that's a hacked ppmd2009-08-23 00:57:23
 http://shelwien.googlepages.com/deppm_20080704.rar2009-08-23 00:57:56
<Skymmer> Ahh, yes. I see that sources for PPMD J rev.1 are is in2009-08-23 00:57:58
 I've just tested a couple of PPM based compressors2009-08-23 00:58:36
 on goddamned enwik8 ;)2009-08-23 00:58:54
 Want a results?2009-08-23 00:59:09
 no matter. Here they are:2009-08-23 01:00:11
  enwik82009-08-23 01:00:16
 ========2009-08-23 01:00:16
 PPMD_J_rev12009-08-23 01:00:16
 -----------2009-08-23 01:00:16
  PPMd -m128 -o8 -r0 = 30.598 = 22 753 3762009-08-23 01:00:16
  PPMd -m128 -o8 -r1 = 41.617 = 21 791 9882009-08-23 01:00:17
  PPMonstr -m128 -o8 -r0 = 204.457 = 20 969 7812009-08-23 01:00:19
  PPMonstr -m128 -o8 -r1 = 207.222 = 20 164 6082009-08-23 01:00:21
 PPMD_Jr1_sh82009-08-23 01:00:23
 ------------2009-08-23 01:00:25
  PPMd /c0 /m128 /o8 = 37.332 = 22 729 6792009-08-23 01:00:27
  PPMd /c0 /m128 /o8 /r = 49.013 = 21 764 1482009-08-23 01:00:29
  PPMd /c1 /m128 /o8 = 32.882 = 22 729 9012009-08-23 01:00:31
  PPMd /c1 /m128 /o8 /r = 44.661 = 21 764 4222009-08-23 01:00:33
 Slim v0.23:2009-08-23 01:00:35
 -----------2009-08-23 01:00:37
  slim a -o8 -m128 = 442.768 = 20 066 5112009-08-23 01:00:39
 But why ppmonstr is decompiled version. Or I guess the sources in ppmd_j are for PPMd only? Not for ppmonstr?2009-08-23 01:03:12
<Shelwien> deppm link is what I just pasted above2009-08-23 01:04:16
 and ppmd_sh is a rewrite of ppmd_vJ public source2009-08-23 01:04:45
 to make a usable library from it etc2009-08-23 01:05:06
<Skymmer> thats what I mean. Why I wonder Sources for ppmonstr have not been provided by Shkarin?2009-08-23 01:05:40
<Shelwien> for the same reason as Christian not providing the source for ccm2009-08-23 01:06:18
 also actually Shkarin was much more angry than Chris when I posted the deppm source ;)2009-08-23 01:07:10
<Skymmer> Yeah? I wonder what kind of words he used to tell it :)))2009-08-23 01:08:10
<Shelwien> nothing interesting, but he demanded to immediately remove the file which i posted on compression.ru2009-08-23 01:09:01
<Skymmer> Maybe even something like "Gad", "Êàzel", "Skotina" :)2009-08-23 01:09:33
<Shelwien> well, he didn't say anything like that2009-08-23 01:10:05
 and I didn't really do anything to make him even think that2009-08-23 01:10:23
<Skymmer> Eugene, call me dumb here but I didn't completely understood.2009-08-23 01:12:15
 deppm_dec is clear enough2009-08-23 01:12:27
 But what is diff. between deppm and deppm-enc ?2009-08-23 01:12:50
<Shelwien> there's an integrated codec2009-08-23 01:14:24
 and separate encoder and decoder2009-08-23 01:14:29
 dunno what's the problem2009-08-23 01:14:36
 originally I decompiled only the decoder2009-08-23 01:15:16
 and then made an encoder for it2009-08-23 01:15:25
 and then integrated them2009-08-23 01:15:32
<Skymmer> Ok, thanks2009-08-23 01:17:09
 Tell me, when you developed FLT32 ?2009-08-23 01:17:25
<Shelwien> 28.10.2001?2009-08-23 01:18:29
<Skymmer> I'm asking because I seen some strange text on Shkarin's page at compression.ru2009-08-23 01:19:13
 ïðåïðîöåññîð E8 äëÿ ýêçåøíèêîâ, íàãëî ñòûðåííûé ó Å.Øåëâèíà2009-08-23 01:19:30
<Shelwien> well, he rewritten it2009-08-23 01:19:51
 somehow, it was popular enough2009-08-23 01:20:00
<Skymmer> So, there was sources for FLT32 ?2009-08-23 01:20:11
<Shelwien> like, 7z's bcj2 is based on that2009-08-23 01:20:11
 they're still there2009-08-23 01:20:21
 http://compression.ru/sh/flt32.rar2009-08-23 01:20:30
<Skymmer> I just thought that such action was his answer on what you did :)2009-08-23 01:20:53
<Shelwien> not really, and he meant something different too2009-08-23 01:21:26
 in fact, i don't think i ever posted anything without sources2009-08-23 01:22:38
<Skymmer> mp3zip :)))2009-08-23 01:22:54
<Shelwien> i didn't post that ;)2009-08-23 01:23:08
<Skymmer> OK. Silence here ;)2009-08-23 01:23:21
 Pity that de_ppm is based on I revision. Need to retest everything.2009-08-23 01:24:22
<Shelwien> there's almost no difference2009-08-23 01:24:37
 except that deppm allows for orders higher than 128 2009-08-23 01:24:53
 and others not supported by ppmonstr2009-08-23 01:25:06
 but then, deppm also doesn't have any memory overflow handling2009-08-23 01:25:29
 that stuff is really complicated, like the same amount of code as the model2009-08-23 01:26:07
 so i just skipped it2009-08-23 01:26:14
 anyway, I only wanted to understand why its results were so good2009-08-23 01:26:50
 of course, we asked, but still couldn't understand anything for at least two years2009-08-23 01:27:28
<osman> %)2009-08-23 01:27:58
<Skymmer> two years 8-| ???2009-08-23 01:28:15
<Shelwien> maybe more actually2009-08-23 01:28:33
 like 2001-20042009-08-23 01:28:36
 ppmonstr had a first place with large gap2009-08-23 01:28:49
<osman> what is the main strong part of it? still secret?2009-08-23 01:29:30
<Shelwien> no2009-08-23 01:29:38
 basically, its secondary estimation (SSE), and Shkarin was repeatedly explaining that2009-08-23 01:30:12
 even written a couple of articles2009-08-23 01:30:26
 s/was/had/2009-08-23 01:30:45
<osman> ahh...ok... i remember where APM comes from %)2009-08-23 01:31:04
<Shelwien> yeah, from deppm in fact ;)2009-08-23 01:31:24
 Serge Osnach was the main supporter for deppm development ;)2009-08-23 01:31:48
 and then for SSE popularization ;)2009-08-23 01:32:07
<osman> btw, i've remembered ccm right now %) 13 lines are still as they are2009-08-23 01:32:31
* Shelwien is messing with comptrace anyway2009-08-23 01:32:48
<Shelwien> so, Shkarin was explaning about SSE, but not how to properly use it ;)2009-08-23 01:33:49
<osman> http://pastebin.com/m6f0aedac2009-08-23 01:33:58
<Shelwien> we listened, and tried to apply the simple 1D probability mapping2009-08-23 01:34:13
<osman> yeah. you had mentioned about that problem2009-08-23 01:34:20
<Shelwien> and it helped a little, but was slow, and it was too little2009-08-23 01:34:28
 and only with deppm we understood finally2009-08-23 01:34:59
<osman> actually SSE is really useful. for example we can even integrate simple match model to a ppm coder with SSE2009-08-23 01:35:07
<Shelwien> that there's unary coding2009-08-23 01:35:08
<osman> and it will virtually unlimited order statistics2009-08-23 01:35:29
<Shelwien> and the main benefit of SSE is actually its context clustering ability2009-08-23 01:35:35
<osman> *will have2009-08-23 01:35:40
<Shelwien> but Shkarin was always talking only about "probability correction" with SSE2009-08-23 01:36:48
 and that didn't really help ;)2009-08-23 01:37:01
 in that sense, I really wonder if that "TSE" is actually worth investigation too ;)2009-08-23 01:37:30
<osman> nah. only imprecise models can benefit in here2009-08-23 01:37:32
<Shelwien> but at least durilca didn't get any visible improvement since ppmonstr vI2009-08-23 01:37:52
<osman> yeah. i revisited it many times. but, didn't understand %)2009-08-23 01:37:58
 shkarin claims that TSE is really helpful for bit-wise coders2009-08-23 01:38:31
 i really wonder the "measurement"2009-08-23 01:38:45
 btw, TSE means in turkish is "Turkish Standards Institue" :)2009-08-23 01:39:38
<Shelwien> well, whatever... we still didn't find the way to use likelihood optimization in the models2009-08-23 01:39:51
 ...except for logistic mixer though ;)2009-08-23 01:40:10
 i've got a working likelihood-based counter in that argument with Shkarin2009-08-23 01:41:04
 but no progress since then2009-08-23 01:41:12
 speed issues, i guess2009-08-23 01:41:34
<osman> i still didn't fully understand "likelihood" term :/2009-08-23 01:42:33
 http://shelwien.googlepages.com/tex1.png2009-08-23 01:46:05
 what's that DeNom and Nom???2009-08-23 01:46:24
<Shelwien> "likelihood" is a string probability2009-08-23 01:51:32
 and likelihood optimization works like this2009-08-23 01:52:11
 suppose you have a coin with some statistics on its tossing2009-08-23 01:53:16
 X heads and Y tails2009-08-23 01:53:28
 and the task is to find the actual probability2009-08-23 01:53:57
 then the idea is to find the probability with "maximum likelihood"2009-08-23 01:54:39
 p^X*(1-p)^Y2009-08-23 01:54:48
 basically, with _any_ probability you can have _any_ statistics2009-08-23 01:55:52
 but likelihood determines the probability that you'd have these specific stats2009-08-23 01:57:12
 ...its not _equal_ to that probability though ;)2009-08-23 01:57:43
<osman> that example basically a bit statistics: (p1^n1)*((1-p1)^n0)2009-08-23 01:57:58
<Shelwien> you have to collect all the likelihoods and normalize2009-08-23 01:58:01
*** Skymmer has left the channel2009-08-23 01:58:20
 but for finding the best parameter value the demoninator doesn't matter2009-08-23 01:58:30
 you just take the most probable case2009-08-23 01:58:41
 *denominator2009-08-23 01:58:56
 and as to bits2009-08-23 01:59:51
 the problem is that the optimal probability approximation for p^n1*(1-p)^n0 is known2009-08-23 02:00:40
 and its (n1+1)/(n0+n1+2)2009-08-23 02:00:49
 you can just maximize that likelihood analytically and get that2009-08-23 02:01:23
 but it becomes a little more interesting2009-08-23 02:01:42
 if there's a skewed probability probability distribution ;)2009-08-23 02:02:00
 like, i investigated a case where p was limited to (0..r)+(1-r..1) interval2009-08-23 02:02:41
 and found that the solution formula becomes completely different2009-08-23 02:03:16
 (and really complex)2009-08-23 02:03:25
<osman> hm...seems i have to revisit it again ;)2009-08-23 02:05:09
 btw, what about TSE description2009-08-23 02:05:29
 i think it's not clear enough2009-08-23 02:05:42
<Shelwien> its clear, just a little arbitrary2009-08-23 02:07:39
 Shkarin first decides that SSE output should be in frequency form2009-08-23 02:08:13
 and then decides to use the specific transformation on that formula2009-08-23 02:09:00
 (linear freq mixing)2009-08-23 02:09:03
 and then uses a likelihood optimization to determine the optimal weight value2009-08-23 02:09:28
 for that mixing2009-08-23 02:09:52
 so its completely reasonable2009-08-23 02:10:17
 and also completely arbitrary2009-08-23 02:10:26
 i mean, yeah, that's how you improve a specific model2009-08-23 02:11:39
 but first you have to use that specific model2009-08-23 02:11:52
 and nobody beside Shkarin does that atm ;)2009-08-23 02:12:03
 ...2009-08-23 02:12:30
 it doesn't mean though that a similar approach can't be used in other circumstances2009-08-23 02:12:54
<osman> it seems bernoulli distrubution: http://mathworld.wolfram.com/MaximumLikelihood.html2009-08-23 02:15:52
<Shelwien> better check the wikipedia article on that2009-08-23 02:16:34
 it doesn't really require any explanations though, as its fairly straightforward2009-08-23 02:17:38
 its the same as own parameter optimizers work ;)2009-08-23 02:17:56
 *our2009-08-23 02:18:22
 *as how our2009-08-23 02:18:34
<osman> i've almost fallen into sleep 8)2009-08-23 02:20:02
 seems it's better to sleep2009-08-23 02:20:21
<Shelwien> you select the parameter value which gives you the minimum codelength - that's all of the maximum likelihood method2009-08-23 02:20:22
<osman> cya2009-08-23 02:20:22
<Shelwien> ;)2009-08-23 02:20:26
 some people like to be cryptic though2009-08-23 02:20:59
<osman> but, still that can't explain TSE for me2009-08-23 02:21:09
<Shelwien> so they use "log-likelihood" instead of codelength ;)2009-08-23 02:21:12
<osman> Nom, DeNom !!??2009-08-23 02:21:25
<Shelwien> that's frequencies2009-08-23 02:21:37
 from SSE counters2009-08-23 02:21:43
<osman> Nom -> n1, DeNom -> n0+n1 ?2009-08-23 02:21:56
<Shelwien> yeah2009-08-23 02:22:00
<osman> so, TSE is kind of small effect of each sub model on final frequencies?2009-08-23 02:22:26
<Shelwien> TSE is optimizing the weight for mixing with static value in the freq space2009-08-23 02:23:21
 if we drop the freqs it would be just linear mixing with constant2009-08-23 02:23:49
<osman> for example, E_i indicates a model effect ratio then it's something like that p = (n1 + M[i].n1*E_i) / (n + M[i].n*E_i)2009-08-23 02:24:07
<Shelwien> kinda2009-08-23 02:24:29
<osman> it's not a magic as i thought :/2009-08-23 02:24:58
<Shelwien> some people really like to encrypt trivial things like that2009-08-23 02:31:09
 no wonder that Shkarin respects Shtarkov ;)2009-08-23 02:31:31
 Shkarkov is even "cooler" - he can't write a single formula without stirling approximation ;)2009-08-23 02:32:17
 (that turns any formula into a huge scary monster)2009-08-23 02:32:35
 (Shkarkov is the theorist behind CTW etc)2009-08-23 02:33:17
*** osman has left the channel2009-08-23 03:57:14
*** Skymmer has joined the channel2009-08-23 06:52:51
* Shelwien was trying to write a generator of DLLs with specified export forwards, but apparently didn't manage to complete it today2009-08-23 07:05:26
*** chornobyl has joined the channel2009-08-23 07:23:11
<Skymmer> Such generator can be usefull I think but I suppose at very specific area :)2009-08-23 07:27:43
 So, Its possible to create as so called "fake" dlls?2009-08-23 07:30:28
<Shelwien> what's really useful there is the library for PE structure manipulation2009-08-23 07:31:36
 and I'm not sure what you call a "fake dll"2009-08-23 07:32:28
<Skymmer> You mean realigning, header squeezing\cleaning ?2009-08-23 07:32:43
<Shelwien> but for comptrace to run on different windows version I need different stub dll versions2009-08-23 07:33:07
 and instead of recompiling these with MSC for each platform2009-08-23 07:33:27
 it'd be much more convenient to just add the export forwards2009-08-23 07:33:54
 i implemented adding of a single name forward today2009-08-23 07:34:55
 but extracting the exports and then adding corresponding forwards would still take a considerable time2009-08-23 07:35:24
 ...2009-08-23 07:35:58
 and as to that PE library... did you see my dllmerge?2009-08-23 07:36:10
<Skymmer> Understood. So "fake" dll term is not right for this situation2009-08-23 07:36:36
 No, I haven't seen it2009-08-23 07:36:45
 *not right term2009-08-23 07:37:12
<Shelwien> http://shelwien.googlepages.com/dllmerge.rar2009-08-23 07:37:44
 its a tool to merge a .exe and .dll linked to it into a single .exe2009-08-23 07:38:17
 like see an example with m1/pthread.dll there2009-08-23 07:38:33
<Skymmer> Cool. Its a bundler shortly speaking2009-08-23 07:39:27
<Shelwien> no2009-08-23 07:39:43
 bundlers work by hooking the APIs2009-08-23 07:39:54
 and I needed a tool to create a real static executable, without runtime patching2009-08-23 07:40:23
 to be able to load it into IDA and debug it there etc ;)2009-08-23 07:40:43
<Skymmer> So after merging the resulting file doesn't extract original files somewhere? The code is now inside new file?2009-08-23 07:41:21
<Shelwien> yes2009-08-23 07:41:30
<Skymmer> Nice :)2009-08-23 07:41:37
<Shelwien> it resolves imports and exports and merges the sections2009-08-23 07:41:45
<Skymmer> Need to try it surely.2009-08-23 07:42:14
<Shelwien> it has a lot of quirks2009-08-23 07:42:24
 but i managed to merge wzunzip.exe+wz_cline.dll+wz_cl32.dll into a single executable with it ;)2009-08-23 07:42:54
 and decompiled the jpeg decoder from it then ;)2009-08-23 07:43:05
<Skymmer> Ay-ay-ay ;)2009-08-23 07:44:22
<Shelwien> in this case it shouldn't be a problem2009-08-23 07:46:00
 as that algorithm is supposedly public2009-08-23 07:46:15
 they even have a spec for it online2009-08-23 07:46:25
 but it'd take a crazy amount of trial-and-error to implement it by spec2009-08-23 07:46:48
* Shelwien is gone2009-08-23 07:48:44
<Skymmer> Just successfully merged precomp.exe and packjpg_dll.dll2009-08-23 07:53:16
 and tested on one jpeg file. It works!2009-08-23 07:53:47
*** pinc has left the channel2009-08-23 08:20:15
*** Skymmer has left the channel2009-08-23 08:33:08
*** Skymmer has joined the channel2009-08-23 08:33:09
*** Skymmer has left the channel2009-08-23 08:44:36
*** Skymmer has joined the channel2009-08-23 08:47:14
*** Skymmer has left the channel2009-08-23 08:58:24
*** Skymmer has joined the channel2009-08-23 09:02:32
*** Skymmer has left the channel2009-08-23 09:02:54
*** Skymmer has joined the channel2009-08-23 09:05:22
*** Skymmer has left the channel2009-08-23 09:08:39
*** Skymmer has joined the channel2009-08-23 09:09:16
*** osman has joined the channel2009-08-23 09:38:38
*** chornobyl has left the channel2009-08-23 10:17:59
*** chornobyl has joined the channel2009-08-23 10:24:39
*** chornobyl has left the channel2009-08-23 10:26:16
*** chornobyl has joined the channel2009-08-23 10:33:24
*** toffer has joined the channel2009-08-23 11:23:44
<toffer> hi2009-08-23 11:23:49
 finished ccm?2009-08-23 11:23:56
<Skymmer> Hi!2009-08-23 11:24:18
 I thought that's quite far to completition2009-08-23 11:24:41
<toffer> @osman did you add my comments of the o1 context quantisation table?2009-08-23 11:24:44
 not really - at least the main algorithm2009-08-23 11:25:03
 and the comments of the bit coding loop sse updates?2009-08-23 11:25:25
<Skymmer> Oh, damn, sorry I'm quit2009-08-23 11:26:03
*** Skymmer has left the channel2009-08-23 11:26:07
<toffer> anybody here?2009-08-23 11:44:34
*** Surfer has joined the channel2009-08-23 11:44:49
<osman> yes i'm :)2009-08-23 11:56:31
 toffer let me introduce just latest ccm main loop. the other files already finished2009-08-23 11:57:03
 main loop has only 13 redundant lines2009-08-23 11:57:25
 http://pastebin.com/m364e3aff2009-08-23 11:58:01
 also i've placed match model into a structure2009-08-23 11:59:45
 http://pastebin.com/m4341fe22009-08-23 11:59:46
 i think it's much easier to understand2009-08-23 11:59:59
 what do you think of about them?2009-08-23 12:00:46
<toffer> good so far, sicne the main loop is cleaner2009-08-23 12:11:27
 isn't that a mistake?2009-08-23 12:12:33
  else // if (curbyte>=48 && curbyte<=122) { // text2009-08-23 12:12:34
 commented that out2009-08-23 12:12:40
<osman> seems i've forgotten it :)2009-08-23 12:18:04
 no no. not a mistake2009-08-23 12:19:00
 it was your comment2009-08-23 12:19:03
<toffer> well, yes2009-08-23 12:19:11
 but the comparison is missing2009-08-23 12:19:17
 btw i managed to get m1 to 23 28x xxx bytes now :)2009-08-23 12:19:43
 w/o additional tuning2009-08-23 12:20:00
<osman> btw, there is no actually. it's else in original2009-08-23 12:20:17
 by what?2009-08-23 12:20:33
<toffer> trying to find a bit better context2009-08-23 12:20:53
 for sse(coutner1,counter2,context)2009-08-23 12:21:02
 i'm using a quantisation of the coding error2009-08-23 12:21:11
 it gave .2ompression gain2009-08-23 12:21:23
 232866972009-08-23 12:21:37
 with optimization i might reac 23200 xxx2009-08-23 12:21:57
<osman> hmm...good :)2009-08-23 12:22:23
<toffer> reach2009-08-23 12:22:46
 well yes2009-08-23 12:22:50
 w/o additional complexity2009-08-23 12:22:57
 i mean run time2009-08-23 12:23:01
 ccm gave something like 24 6xx xxx with the same configuration :)2009-08-23 12:23:20
 and was slower2009-08-23 12:23:22
 that's 1.4 % worse2009-08-23 12:23:37
<osman> it has some really weird modelling way2009-08-23 12:24:52
<toffer> ccm?2009-08-23 12:25:08
<osman> yep2009-08-23 12:25:58
<toffer> i don't think so - at least now2009-08-23 12:26:10
 it lacks proper quantisation2009-08-23 12:26:17
 i mean the probabilities are mapped linearily2009-08-23 12:26:27
 which is bad2009-08-23 12:26:32
 and the match model actually does some kind of classification of text vs non-text2009-08-23 12:28:31
 which works better than these character range checks2009-08-23 12:28:41
<osman> do you have any "good" name for dword_14CF5E0 table?2009-08-23 12:35:12
 it's something like another SSE stage with "access" count like paq does2009-08-23 12:35:32
 btw it seems each entry is 64 bits2009-08-23 12:36:58
 first 32 bits is somehow counter and 32-40 bits is "access count"2009-08-23 12:38:42
 the rest of seems unused2009-08-23 12:38:51
 maybe i can merge it together without breaking the compatibility2009-08-23 12:39:06
<toffer> the final counter uses an access cout2009-08-23 12:39:36
 so that's just an sse table2009-08-23 12:39:42
 with a dfiferent counter implementation2009-08-23 12:39:53
<osman> i'll rename it as finalSSE then2009-08-23 12:41:08
<toffer> just have a look at line 1302009-08-23 12:42:10
 there's the sse update2009-08-23 12:42:15
<osman> seems it uses 24 bits at most. so having 64 bits is unnecessary2009-08-23 12:42:32
 because access count is limited at 632009-08-23 12:42:44
 so, it can fit in 32 bits2009-08-23 12:42:53
<toffer> yeah2009-08-23 12:50:28
 @osman: you once used rhc for bit optimization?2009-08-23 13:07:06
 i've made it multi-threaded, too2009-08-23 13:07:14
<osman> yeah. i'm using rhc as a final pass for bit2009-08-23 13:15:54
 it's good to hear multithreading capabilities2009-08-23 13:16:28
 btw, last 13 lines translated into 2 lines :)2009-08-23 13:16:39
 now i'm removing eax, edx, like registers2009-08-23 13:16:50
 :)2009-08-23 13:16:51
 *registers -> variables2009-08-23 13:16:59
<toffer> ok, so it's almost complete2009-08-23 13:21:20
 i accidently ran m1 w/o counter updates... and it's 8% and loses 0.04ompression :)2009-08-23 13:31:30
<osman> testing latest changes...2009-08-23 13:51:26
 CCM CRACKING IS FINISHED COMPLETELY!!! :-)2009-08-23 13:53:13
 www.osmanturan.com/ccm_sh_v5.zip2009-08-23 13:59:39
 it's final :)2009-08-23 13:59:48
 though several optimizations can be applied without breaking compatibility2009-08-23 14:00:10
 i will glad to see any comments about it2009-08-23 14:00:56
*** sami has joined the channel2009-08-23 14:32:24
<toffer> i hate that linear prob. quantisation2009-08-23 14:32:36
 it just drops comrpession2009-08-23 14:32:42
 definitely a week point2009-08-23 14:32:49
 weak2009-08-23 14:32:51
 and probably one should rename permutationtable to something like ctx2idx, since it doesn't do any permutation actually2009-08-23 14:33:27
 or nextCtx or something like that2009-08-23 14:34:30
 @osman what about applying a proper quantisation now?2009-08-23 14:41:14
*** chornobyl has left the channel2009-08-23 14:55:45
 -.-2009-08-23 15:04:25
*** Simon|B has joined the channel2009-08-23 15:20:33
<Simon|B> no new release guys? I am shocked :p2009-08-23 15:40:00
<toffer> ^^2009-08-23 15:40:51
 well osman cleaned his version, merged it with eugenes and added all of my comments - thus i think were're pretty much complete2009-08-23 15:41:24
<Simon|B> oh ok I cancel my comment^^2009-08-23 15:43:29
 did your ideas bring a good effect?2009-08-23 15:44:54
<toffer> kommentare im wahrsten sinne des wortes, nachdem ich vor einer woche das disassemblat gecrackt hab, hab ich mit kommentaren den algorithmus beschrieben. osman alle sh versionen und meine kommentare in eine neue version übertragen2009-08-23 15:51:49
<osman> someone called me? :)2009-08-23 15:56:21
 btw, i have to go for dinner because of ramadan2009-08-23 15:57:02
 toffer why did you apply some small optimizations to ccm_sh_v5?2009-08-23 15:57:33
<toffer> i didn't2009-08-23 15:57:43
<osman> like non-linear quantization and linear counters?2009-08-23 15:57:47
*** chornobyl has joined the channel2009-08-23 15:57:58
 or any other small improvements which won't keep you busy2009-08-23 15:58:09
<toffer> it's just a proposal2009-08-23 15:58:10
 erm do you actually mean why i "didn't"?2009-08-23 15:58:34
<osman> in here i=toffer :)2009-08-23 15:58:53
<toffer> :)2009-08-23 15:59:16
<osman> you have a optimizer right? then it will be very nice if you create another branch of ccm_sh for small improvements2009-08-23 15:59:44
 as i said, i'll vote unless it won't keep you busy2009-08-23 15:59:59
 just for fun2009-08-23 16:00:02
 for example, i can eliminate most of data structures easily (there are some redundant structures. i think, they are left for CCMX)2009-08-23 16:00:56
 what about calling this new branch as "ccm_shot" :)2009-08-23 16:01:46
 shot -> SHelwien + Osman + Toffer :)2009-08-23 16:02:01
<toffer> we could do that right2009-08-23 16:02:47
 but atm i'm busy with m12009-08-23 16:02:52
 :)2009-08-23 16:02:53
*** Skymmer has joined the channel2009-08-23 16:02:56
 but i can integrate the optimzier2009-08-23 16:03:00
<osman> if you can integrate optimizer that would be enough for me2009-08-23 16:03:20
*** Skymmer has left the channel2009-08-23 16:03:30
 i can change some parameters based on it2009-08-23 16:03:33
*** Skymmer has joined the channel2009-08-23 16:03:38
*** Skymmer has left the channel2009-08-23 16:03:43
 anyway. i have to go now. but, i'll keep irc client as open.2009-08-23 16:05:16
 you can write anything you want. i'll read when i came back2009-08-23 16:05:30
 cya now2009-08-23 16:05:33
*** Skymmer has joined the channel2009-08-23 16:06:12
<toffer> bye2009-08-23 16:08:13
*** Skymmer has left the channel2009-08-23 16:39:36
*** Skymmer has joined the channel2009-08-23 16:39:38
<Skymmer> Anybody here uses Mozilla Firefox ?2009-08-23 16:40:04
 hmm... strange. Seems everybody are the IE fans here :)2009-08-23 16:43:16
<chornobyl> i use rarely2009-08-23 16:49:22
<Skymmer> Well, i just wanted to report the Easter egg which I found by myself2009-08-23 16:54:37
<chornobyl> robots?2009-08-23 16:54:45
<Skymmer> What?2009-08-23 16:54:51
<chornobyl> about:robots2009-08-23 16:55:03
<Skymmer> Thanks. Cool :))2009-08-23 16:55:37
<chornobyl> what is yours2009-08-23 16:55:43
<Skymmer> about:mozilla2009-08-23 16:55:53
 I suppose its alredy known ?2009-08-23 16:56:11
<chornobyl> creepy2009-08-23 16:56:22
<Shelwien> ...2009-08-23 16:56:32
<Skymmer> yes. a little bit2009-08-23 16:56:58
 But now I know where this fox comes from2009-08-23 16:57:21
<chornobyl> tell us2009-08-23 16:58:00
<Skymmer> Well, everything written there2009-08-23 16:58:21
 By the way first phrase from about:robots is the 1st rule of robototechnics by Isaac Asimov2009-08-23 17:01:20
 http://en.wikipedia.org/wiki/Three_Laws_of_Robotics2009-08-23 17:01:31
 By the way, I haven't told you but I'm robot too2009-08-23 17:05:28
<chornobyl> and i ama banana2009-08-23 17:05:49
<Skymmer> Don't joke please. I'm serious.2009-08-23 17:06:18
<chornobyl> i'm listening2009-08-23 17:06:36
<Skymmer> Well, I can send you the dump of my logic2009-08-23 17:07:18
* Shelwien thinks that robots should be able to compress enwik8 "manually"2009-08-23 17:07:36
<chornobyl> ok2009-08-23 17:07:38
<Skymmer> I'm model RX-37_sk2009-08-23 17:07:59
 And yes, I can do it2009-08-23 17:08:07
<Shelwien> then please do and lets close that case ;)2009-08-23 17:08:21
<Skymmer> But using quantum logic :))2009-08-23 17:08:35
 OK. Sorry.2009-08-23 17:08:44
<Shelwien> you'd have to run away from Hutter after that, though ;)2009-08-23 17:08:49
 i found this: http://www.mirtepla.com.ua/index.php?productID=2212009-08-23 17:11:18
 guess that won't be able to compress enwiks2009-08-23 17:12:10
 its a pity as I hoped that it would be some kind of war mecha2009-08-23 17:12:47
<Simon|B> Internet explorer egg res://mshtml.dll/about.moz2009-08-23 17:21:56
 in earlier version it could be found by typing about:mozilla ;)2009-08-23 17:22:22
<Shelwien> not iE62009-08-23 17:23:05
<chornobyl> totally2009-08-23 17:23:13
<Simon|B> it was 42009-08-23 17:23:21
 but the new link works or didn't it?2009-08-23 17:23:50
<Shelwien> neither works in IE62009-08-23 17:26:33
 !next2009-08-23 17:28:02