*** chornobyl has joined the channel2009-08-26 06:37:27
*** pinc|mirror has joined the channel2009-08-26 06:39:39
*** pinc has left the channel2009-08-26 06:40:05
*** FuSiyuan has left the channel2009-08-26 07:25:43
*** chornobyl has left the channel2009-08-26 08:57:16
*** pinc|mirror has left the channel2009-08-26 09:05:53
*** Simon|B2 has joined the channel2009-08-26 09:07:47
*** chornobyl has joined the channel2009-08-26 09:09:42
<Shelwien> ...2009-08-26 09:33:43
<chornobyl> dots2009-08-26 09:34:02
<Shelwien> can you confirm Skymmer's result that sh1d is slower than sh1c?2009-08-26 09:34:25
<chornobyl> give links ill try2009-08-26 09:34:37
<Shelwien> http://shelwien.googlepages.com/ccm_sh1c.rar2009-08-26 09:34:57
 http://shelwien.googlepages.com/ccm_sh1d.rar2009-08-26 09:34:59
 http://shelwien.googlepages.com/ccm_sh1d1.rar2009-08-26 09:35:02
<osman> hi chorbobyl2009-08-26 09:36:37
 hi shelwien2009-08-26 09:36:49
<Shelwien> hi osman ;)2009-08-26 09:36:59
<osman> @chornobyl: yesterday when i finish your tool. you had gone already :)2009-08-26 09:37:20
<chornobyl> nice to hear it2009-08-26 09:37:40
<osman> @shelwien: how are you?2009-08-26 09:37:42
<Shelwien> sleepy. fighting bugs.2009-08-26 09:39:50
<osman> :)2009-08-26 09:40:24
 still dejpg?2009-08-26 09:40:32
<Shelwien> no, other job this time2009-08-26 09:40:42
 PA with dejpg is out btw2009-08-26 09:40:47
<osman> you mean you completely finished dejpg?2009-08-26 09:41:13
<Shelwien> ...i did before starting this ccm project ;)2009-08-26 09:41:35
<osman> @chornobyl: grap it here: www.osmanturan.com/sizeondisk.zip2009-08-26 09:41:44
 @shelwien: it's good then. what's it this time?2009-08-26 09:42:11
<chornobyl> thank you2009-08-26 09:42:24
<Shelwien> alien bugs2009-08-26 09:42:31
 ...wanted to try the interleaved rc idea on ccm2009-08-26 09:42:48
 but no time yet2009-08-26 09:43:01
<osman> @chornobyl: you're welcome :) source code included btw. and i think it's enough easy to use.2009-08-26 09:43:09
 @shelwien: alien bugs!? :)2009-08-26 09:43:24
<Shelwien> well, they're made by other developers2009-08-26 09:44:12
<chornobyl> eugene what is commandline2009-08-26 09:44:32
<Shelwien> for ccm?2009-08-26 09:44:41
<chornobyl> yepp2009-08-26 09:44:47
<Shelwien> ccm c/d input output ;)2009-08-26 09:44:51
 there's t.bat for test2009-08-26 09:45:00
 but skymmer used enwik82009-08-26 09:45:12
<osman> i really hate this kind of jobs. at a time, i have to hunt my big brother program's bugs. they were huge scale database applications and it was not easy task2009-08-26 09:45:19
<Shelwien> these multithreading bugs are not really nice either ;)2009-08-26 09:45:59
 what do you think about MS RDC btw?2009-08-26 09:46:32
<osman> i didn't properly read it. i've another job which have to be done today.2009-08-26 09:47:09
<Shelwien> "The RDC library's FilterMax signature generator "slides" the hash window across the entire file by adding the byte at the leading edge and subtracting the byte at the trailing edge of the window. Meanwhile, the generator continually examines the sequence of fingerprint function values over a given range of bytes, called the horizon size. If a fingerprint function value is a local maximum within the range, its byte position is chos2009-08-26 09:47:31
 After the file has been divided into chunks, the signature generator computes a strong hash value (an MD4 hash), called a signature, for each chunk. The signatures can be used to compare the contents of two arbitrarily different versions of a file. "2009-08-26 09:47:31
<chornobyl> what is mem usage2009-08-26 09:49:03
<Shelwien> now compare it to this2009-08-26 09:49:07
 <Shelwien> i don't want to mess with it now anyway ;)2009-08-26 09:49:08
 <Shelwien> anyway, an archiver based on CM/ppmd requires a far match model2009-08-26 09:49:08
 <Shelwien> like bulat's rep2009-08-26 09:49:08
 <Shelwien> and i've got one interesting idea about block hashes2009-08-26 09:49:08
 <Shelwien> i call it "anchored hashes"2009-08-26 09:49:08
 <Shelwien> like, normally we have to calculate a sliding hash for all strings of some fixed length2009-08-26 09:49:09
 <Shelwien> and look them up on each symbol2009-08-26 09:49:11
 <Shelwien> but only store the hashes on aligned offsets2009-08-26 09:49:13
 <Shelwien> (well, reverse is also possible - aligned lookups, but all stored, but it seems worse)2009-08-26 09:49:15
 <sami> i've tried some "anchoring"2009-08-26 09:49:17
 <sami> like byte0<byte1<byte2 is an anchor etc2009-08-26 09:49:19
 <Shelwien> yeah, something along that line2009-08-26 09:49:21
 <Shelwien> but i'm thinking to use something else2009-08-26 09:49:23
*** toffer has joined the channel2009-08-26 09:49:24
 <Shelwien> as symbol combinations are unstable2009-08-26 09:49:25
 <Shelwien> like, a minimal hash value in some area2009-08-26 09:49:27
 <sami> hmm. yeah. that's better idea2009-08-26 09:49:29
 <sami> i like it2009-08-26 09:49:31
 <Shelwien> ;)2009-08-26 09:49:33
 <sami> :-D2009-08-26 09:49:35
<toffer> hi, what's that?2009-08-26 09:49:49
<Shelwien> hi, what's what?2009-08-26 09:50:01
<osman> :)2009-08-26 09:50:25
<chornobyl> can mem usage for ccm be lower2009-08-26 09:50:45
<Shelwien> you don't have 500M or what?2009-08-26 09:51:04
 its a pity then as ccm_sh requires recompiling to lower memory usage2009-08-26 09:51:22
<chornobyl> i dont have 530 free which it whants to eat2009-08-26 09:51:26
<Shelwien> forget it then ;)2009-08-26 09:51:37
 the question was about comparing this mode anyway ;)2009-08-26 09:51:59
<toffer> and does your templated memorylevel pay off?2009-08-26 09:53:44
<Shelwien> well, it considerably simplifies the source2009-08-26 09:54:08
<toffer> mh not sure about that2009-08-26 09:54:19
 i think it's just too stiff2009-08-26 09:54:40
 not flexible2009-08-26 09:54:43
<Shelwien> why? i just declared the tables2009-08-26 09:54:49
 its simpler than new (more so malloc that was there)2009-08-26 09:55:09
<toffer> there's no big deal in having an allocation function with one call to malloc2009-08-26 09:55:14
<Shelwien> there's some deal in having to use pointers2009-08-26 09:55:31
 instead of direct offsets in the code2009-08-26 09:55:45
 anyway, sh1d is faster here2009-08-26 09:56:28
<toffer> well yes, but how's the speed difference?2009-08-26 09:56:31
<Shelwien> http://shelwien.googlepages.com/results.txt2009-08-26 09:56:46
<toffer> a lead ballon :)2009-08-26 09:56:54
<Shelwien> what?2009-08-26 09:57:07
<toffer> that's an idiom i learned some time ago2009-08-26 09:57:22
 unsuccessful try2009-08-26 09:57:34
<Shelwien> what is?2009-08-26 09:57:53
<toffer> if the old version is faster2009-08-26 09:59:53
 i don't know what you mean2009-08-26 10:00:09
<Shelwien> new one is faster for me2009-08-26 10:00:13
 but not for skymmer: http://skymmer.narod.ru/data/deccm_speed.txt2009-08-26 10:00:55
<toffer> 32.109s 32.734s ccm_sh1c vs 31.703s 32.766s ccm_sh1d2009-08-26 10:01:08
<Shelwien> so that's the question2009-08-26 10:01:09
 so? encoding is faster, decoding is the same2009-08-26 10:01:31
<toffer> that's just ~2 %2009-08-26 10:01:50
 i'd not call it significant - and you lose the flexibility2009-08-26 10:02:09
 i'd not accept that change than2009-08-26 10:02:20
<Shelwien> well, i hoped for more, but IntelC is hard to control2009-08-26 10:02:24
 it wasn't the only change anyway2009-08-26 10:02:33
 i think that speed problem2009-08-26 10:02:41
<toffer> even worse than i guess - but compilers are weird2009-08-26 10:02:46
<Shelwien> is actually because of added align directive2009-08-26 10:02:51
<toffer> you know better than me but i had weird stuff these days too2009-08-26 10:02:56
<Shelwien> *directives2009-08-26 10:03:00
 ...anyway, i also added uncompressed data buffering in 1d12009-08-26 10:06:52
 and it helped a little too2009-08-26 10:07:03
<toffer> 1d1?2009-08-26 10:07:42
<Shelwien> yeah ;)2009-08-26 10:07:58
*** pinc has joined the channel2009-08-26 10:08:09
<toffer> what's that2009-08-26 10:11:07
<Shelwien> ?2009-08-26 10:12:03
<toffer> 1d1?2009-08-26 10:13:12
<Shelwien> http://shelwien.googlepages.com/ccm_sh1d1.rar2009-08-26 10:13:45
 only one function is modified so it doesn't matter2009-08-26 10:15:41
 ...2009-08-26 10:16:20
 anyway, i want to test interleaved rcs there, but didn't have time yet2009-08-26 10:16:35
 i make, to make 8 rcs there, and move all the renorm stuff out of the bit loop2009-08-26 10:17:15
<toffer> as i said it helped2009-08-26 10:17:22
 and the fact that a renorm emits one byte only is useful, too2009-08-26 10:17:44
 i tried to buffer 16 bits in the rc state2009-08-26 10:17:54
 which increased speed, too2009-08-26 10:18:01
 as i said, i really like that idead :)2009-08-26 10:18:18
<Shelwien> somehow i don't think that we're talking about the same thing ;)2009-08-26 10:18:49
<toffer> the resulting compression loss was something like 100..200 bytes2009-08-26 10:18:53
 well as you know i triedthat yesterday2009-08-26 10:19:08
 just to have a function call2009-08-26 10:19:18
 to renorm2009-08-26 10:19:21
<Shelwien> yeah2009-08-26 10:19:26
<toffer> after a check for renorm possible2009-08-26 10:19:30
 and buffering 16 bits2009-08-26 10:19:34
 halves the number of calls efficiently2009-08-26 10:19:40
 so it became faster2009-08-26 10:19:45
<Shelwien> sure2009-08-26 10:19:59
<toffer> instead of range < (1<<24), range < (1<<16)2009-08-26 10:20:06
<Shelwien> but i was talking about using 8 separate rcs there ;)2009-08-26 10:20:11
<toffer> it doesn't change the idea of using function calls2009-08-26 10:20:31
<Shelwien> yeah, its just completely unrelated ;)2009-08-26 10:20:48
<toffer> not really2009-08-26 10:20:55
 i make, to make 8 rcs there, and move all the *renorm stuff out of the bit loop*2009-08-26 10:21:28
<Shelwien> yes, but i talked about processing all of it out of the bit loop2009-08-26 10:22:14
 including the first checks2009-08-26 10:22:21
 so it would be possible to leave rc parts completely without branches in the loop ;)2009-08-26 10:23:23
<toffer> basically because one rc state per bit is likely to not to cause a precision loss?2009-08-26 10:23:27
<Shelwien> still looks like a misunderstanding...2009-08-26 10:24:06
<toffer> i think i got that2009-08-26 10:24:18
<Shelwien> imagine that we made 8 output files for compressed data2009-08-26 10:24:20
 and I call rc[k].BProcess( bit, prob ) instead of same rc2009-08-26 10:24:55
<toffer> i mean you can get the checks out becasue there's one rc for each of the 8 bits2009-08-26 10:24:57
<Shelwien> yeah, they don't overlap2009-08-26 10:25:08
<toffer> and just one coding step can drop at most N bits of precision2009-08-26 10:25:11
 instead of dropping at most 8*N (for 1 rc)2009-08-26 10:25:21
 you just drop at most N2009-08-26 10:25:25
<Shelwien> i don't really see how this is related to precision2009-08-26 10:25:29
<toffer> per byte coding step2009-08-26 10:25:32
<Shelwien> only redundancy there would be because of multiple rc flushes2009-08-26 10:25:48
<toffer> too less precision = bad coding performance2009-08-26 10:25:50
<Shelwien> there's no redundancy other than that2009-08-26 10:26:00
<toffer> well what is a rc flush?2009-08-26 10:26:31
 it increased precision2009-08-26 10:26:35
 by rescaliong2009-08-26 10:26:38
<Shelwien> its what rc does at the end of file2009-08-26 10:26:45
<toffer> ok i meant renorm there2009-08-26 10:26:56
<Shelwien> each rc would still have a renorm per bit2009-08-26 10:27:13
 i'm talking about using 8 different rcs at once here2009-08-26 10:27:29
 one per each bit2009-08-26 10:27:32
 and then doing renorm for all of them, in a separate loop2009-08-26 10:27:55
*** osman has left the channel2009-08-26 10:33:22
<toffer> well i got that all the time2009-08-26 10:33:57
 but you didn't understand me - i said if you lower the precision you have even less renormalizations in total2009-08-26 10:34:30
*** osman has joined the channel2009-08-26 10:34:40
 for example if i have 1 renorm every 4..6 bytes in a sequential rc with that range < (1<<16)2009-08-26 10:35:12
 you'd have'd have the same for the parallel one2009-08-26 10:35:36
 on of these parallel coders will renorm every (4..6)*8 bytes2009-08-26 10:36:02
 and all at least one of them will renorm every 4..6 bytes2009-08-26 10:36:20
<Shelwien> and as i said these things are orthogonal2009-08-26 10:36:26
<toffer> it's just some confusion again2009-08-26 10:36:51
 ^^2009-08-26 10:36:55
<Shelwien> you can separately balance the precision and renorm frequency2009-08-26 10:37:07
 and rc interleaving is another thing completely2009-08-26 10:37:22
 its not about reducing the number of renorms or anything2009-08-26 10:37:42
 but about separating the renorm+i/o code2009-08-26 10:38:00
 from the model code2009-08-26 10:38:07
 and btw, in fpaq0pv4B I experimented with 16bit i/o and skipped renorms2009-08-26 10:39:41
 at it really helped, at the cost of some imprecision2009-08-26 10:39:56
<toffer> for me it helped, too2009-08-26 10:40:50
 gonna try to incorporate that into the m1 coding scheme2009-08-26 10:41:00
<Shelwien> i'm also wondering about multiplication-free thingie2009-08-26 10:41:31
 like, rnew=exp(log(range)+log(prob)) or something2009-08-26 10:42:01
 (with lookups)2009-08-26 10:42:10
<toffer> mh2009-08-26 10:42:31
 three lookups instead of a single mult...2009-08-26 10:42:44
<Shelwien> not really2009-08-26 10:42:56
 you can precalculate log(range)2009-08-26 10:43:07
 at the end of previous step2009-08-26 10:43:19
 ah2009-08-26 10:43:54
 or actually it would be just that log sum2009-08-26 10:44:02
<toffer> i just noticed there can be a problem with the parallel rc2009-08-26 10:57:04
<Shelwien> ?2009-08-26 10:57:14
<toffer> if .e.g bit 7 will be compressed to almost nothing, and bit 1 is somehow pretty much more than you cannot easily store interleaved output2009-08-26 10:57:54
<Shelwien> i can ;)2009-08-26 10:58:14
<toffer> ?2009-08-26 10:58:23
<Shelwien> i guess you didn't check parcoder ;)2009-08-26 10:58:35
<toffer> no, i didn't2009-08-26 10:58:47
<Shelwien> well, i have a working interleaved rc implementation since long ago ;)2009-08-26 10:59:08
<toffer> i see2009-08-26 11:00:34
 you don't directly interleave, but add a list of next bytes2009-08-26 11:00:50
<Shelwien> not exactly2009-08-26 11:01:06
 the trick is to allocate a place for a byte where decoder would need it2009-08-26 11:01:31
<toffer> wouldn't just one ptr be enough?2009-08-26 11:02:19
 since norm. is syncron in encoding/decoding2009-08-26 11:02:30
 only init would need actual interleaving2009-08-26 11:02:38
<Shelwien> still, there's a delay of 4-5 bytes2009-08-26 11:03:33
<toffer> i guess you mean the one caused by decoder init.2009-08-26 11:06:42
<Shelwien> yeah2009-08-26 11:07:02
*** Simon|B2 has left the channel2009-08-26 11:07:17
 but again, that's a renorm-only stuff, so pointer juggling would be only present in tenorm loop2009-08-26 11:07:19
 *renorm2009-08-26 11:07:23
 at the time, i also tried to fully synchronize enc and dec2009-08-26 11:08:25
 by computing the decode intervals2009-08-26 11:08:36
 and only actually fetching the bytes necessary to compute code<rnew2009-08-26 11:09:39
 but it quickly became too complicated2009-08-26 11:09:59
 ...2009-08-26 11:10:18
 in fact, there's even better possibility for encoding2009-08-26 11:10:33
 there we can cache the probabilities and do the rangecoding separately for the whole buffer, right?2009-08-26 11:11:09
 so with 8 rcs2009-08-26 11:11:27
 it would be possible to also vectorize the {range;low} update in all rcs2009-08-26 11:12:11
 which includes the multiplication2009-08-26 11:12:21
<toffer> but that's only acceptable for encoding2009-08-26 11:15:39
<Shelwien> yeah, unfortunately2009-08-26 11:15:48
<toffer> there're several things like that2009-08-26 11:16:03
 e.g. prefetching the known memory locations on encoding2009-08-26 11:16:13
<Shelwien> well, decoding won't become slower from that either2009-08-26 11:16:29
<toffer> true2009-08-26 11:16:39
 but still that's somehow weird somehow :)2009-08-26 11:16:49
<Shelwien> just that its hard to think of anything to optimize it with unknown data2009-08-26 11:16:49
<toffer> but good for optimization btw2009-08-26 11:16:58
<Shelwien> well, there was that runlength idea, which works for decoding2009-08-26 11:17:13
 guess there might be a sense to check it with ccm too2009-08-26 11:17:35
<toffer> afair the speed gain wasn't that notable?2009-08-26 11:17:51
<Shelwien> yeah, and mainly in encoding again ;)2009-08-26 11:18:06
 but still, that was some mistake ;)2009-08-26 11:18:28
 i mean, millions of rc calls skipped, and no effect? ;)2009-08-26 11:19:26
<toffer> ^^2009-08-26 11:20:06
<Shelwien> i guess that rc/RLE branch was really slow...2009-08-26 11:33:45
 so its probably necessary to completely split the probability range into several intervals2009-08-26 11:34:25
 i mean, so there won't be any rc stuff in decoding - only looking up bits by probability2009-08-26 11:36:51
<toffer> well i thinkt for a parallel rc a convention for the arrangement of the first 4 bytes per rc would be enough 2009-08-26 11:38:30
 thnik2009-08-26 11:38:56
 think2009-08-26 11:38:58
 that might add redundancy for small files2009-08-26 11:39:14
<Shelwien> unfortunately there's no other way than queuing the byte locations for decoder2009-08-26 11:40:27
 that 4-5 byte delay is just that significant2009-08-26 11:40:59
<toffer> isn't the only problem that the decoder is exactly 4 bytes in advance (paq rc)2009-08-26 11:41:04
<Shelwien> yeah, but its 4 _byte fetches_ in advanced, but just 4 bytes2009-08-26 11:41:31
<toffer> yes2009-08-26 11:41:46
<Shelwien> *advanced, not just 4 bytes2009-08-26 11:41:55
 ...2009-08-26 11:41:59
 *advance, not just 4 bytes2009-08-26 11:42:08
<toffer> that's why i say that placing the first 4 bytes in a deterministic way per rc (that'd be the first 32 bytes for 8 rcs)2009-08-26 11:42:11
<Shelwien> that won't help at all2009-08-26 11:42:31
<toffer> why?2009-08-26 11:42:36
<Shelwien> that is, of course the layout for initial decoder bytes would be known and fixed2009-08-26 11:42:54
 but it doesn't matter2009-08-26 11:43:02
<toffer> erm the encoder and decoder preform coding operations in sync2009-08-26 11:43:14
<Shelwien> yeah, but there're multiple rcs2009-08-26 11:43:25
 which are not in sync2009-08-26 11:43:31
<toffer> and still all of them behave identically on encoding and decoding2009-08-26 11:43:42
<Shelwien> they don't, at all2009-08-26 11:43:52
<toffer> so they're in sync as longas the model is in sync2009-08-26 11:43:56
<Shelwien> imagine processing english text like that2009-08-26 11:44:03
 with bit7 always zero2009-08-26 11:44:13
 and different rc for each bit2009-08-26 11:44:20
<toffer> i know what you mean2009-08-26 11:44:36
 but that's not the point2009-08-26 11:44:41
<Shelwien> the point is2009-08-26 11:44:50
<toffer> since if the model behaves indentical in encoding and decoding2009-08-26 11:44:52
<Shelwien> that you can't just calculate where each rc's bytes would be located2009-08-26 11:45:10
<toffer> all of the rcs will behave identical for encoding and decoding (rc7 encode = rc7 decode, rc6 encode = rc6 decode and so on)2009-08-26 11:45:20
 that's implicitly done by processing2009-08-26 11:45:29
<Shelwien> yeah, but delay changes it all2009-08-26 11:45:36
<toffer> and the dealy is fixed by a deterministic inital alignment2009-08-26 11:45:54
<Shelwien> the number of delay bytes is, but not distances between them2009-08-26 11:46:24
 and the distances depend only on the behavior of other rcs2009-08-26 11:46:43
<toffer> well the distances are deterministic for the first 4 bytes per rc2009-08-26 11:46:47
 and for the rest they're implicitly known by coding ops?!2009-08-26 11:46:56
<Shelwien> for the first bytes they are, but not for all others2009-08-26 11:47:06
 ...2009-08-26 11:47:16
 i'd try to clarify2009-08-26 11:47:28
 the idea is2009-08-26 11:47:31
 that storing the bytes in encoding2009-08-26 11:47:43
 can be delayed2009-08-26 11:47:49
 first byte locations queued2009-08-26 11:47:58
 and then bytes stored there2009-08-26 11:48:04
 but with that2009-08-26 11:48:14
 decoders won't need any fixing at all2009-08-26 11:48:24
 they would be completely the same as in normal rc2009-08-26 11:48:36
<toffer> i'm still unsure about that2009-08-26 11:49:38
<Shelwien> about what?2009-08-26 11:50:01
<toffer> guess i'll have to get it on my own to get what i might be missing2009-08-26 11:50:04
 you need to store stream lengths?2009-08-26 11:50:55
<Shelwien>  void StartEncode( void ) { 2009-08-26 11:51:26
  low=0; range=(uint)-1; 2009-08-26 11:51:26
  DO(8) QueueReserve();2009-08-26 11:51:26
  }2009-08-26 11:51:26
  void Encode (uint cumFreq, uint freq, uint totFreq) {2009-08-26 11:51:26
  low += cumFreq * (range/= totFreq);2009-08-26 11:51:27
  range*= freq;2009-08-26 11:51:29
  while( range<TOP ) {2009-08-26 11:51:31
  if ( uc((low^low+range)>>56) ) 2009-08-26 11:51:33
  range = ((uint(low)|(TOP-1))-uint(low));2009-08-26 11:51:35
  OutByte( low>>56 ), range<<=8, low<<=8;2009-08-26 11:51:37
  QueueReserve();2009-08-26 11:51:39
  }2009-08-26 11:51:41
  }2009-08-26 11:51:43
 (its a weird 64-bit carryless, but don't mind that)2009-08-26 11:52:02
 OutByte() stores a byte into the first location in the queue2009-08-26 11:53:05
 http://compression.ru/sh/parcoder.rar2009-08-26 11:53:28
*** chornobyl has left the channel2009-08-26 11:59:38
*** Simon|B2 has joined the channel2009-08-26 12:10:00
*** FuSiyuan has joined the channel2009-08-26 12:31:56
*** FuSiyuan has left the channel2009-08-26 12:46:39
*** FuSiyuan has joined the channel2009-08-26 13:20:59
*** pinc has left the channel2009-08-26 14:43:15
*** Skymmer has joined the channel2009-08-26 14:52:48
<Simon|B> hi Skymmer ;)2009-08-26 15:01:33
<Skymmer> Hello :)2009-08-26 15:03:02
 How do you do?2009-08-26 15:04:08
 I'm somehow have been caught by DOS last days2009-08-26 15:05:48
 Now I'm collecting different screenshot tools for DOS2009-08-26 15:06:15
 I need to capture some program running2009-08-26 15:06:44
 Never thought that screenshoting is possible under DOS 8)2009-08-26 15:07:14
<Simon|B> hehe funny I played DOS games via ScummVm on my handy today^^2009-08-26 15:07:51
<Skymmer> nice :)2009-08-26 15:09:16
 collected so far: GRABBER V3.98, PCXDUMP v9.30, SCREEN Thief v1.58, VideoThief v0.072009-08-26 15:09:25
<Simon|B> It's so heavy to me what's possible on such a SMALL HANDY2009-08-26 15:09:27
 video? nice^^2009-08-26 15:10:22
<Skymmer> Of cource I could use the DOSBox software for my task but there is no atmosphere in such case ;)2009-08-26 15:12:00
 There is also Ruxgulo's paq8px_61 compilation which runs in DOS !2009-08-26 15:13:17
 Fascinating :)2009-08-26 15:13:53
*** FuSiyuan has left the channel2009-08-26 15:14:49
*** pinc has joined the channel2009-08-26 15:16:14
 I'm also become interested in FireFox browser last days.2009-08-26 15:16:17
 Damn! Those guys (I mean developers) just rock2009-08-26 15:16:42
 I mean overall idea and principles2009-08-26 15:17:21
 For example if somebody complaining about too less options in it then...2009-08-26 15:18:46
 about:config2009-08-26 15:18:58
 8)2009-08-26 15:19:13
 Also I like their humor2009-08-26 15:20:06
 For example you can test FireFox v3.7 right now2009-08-26 15:20:29
 Seriously2009-08-26 15:20:37
 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/2009-08-26 15:20:44
 But its called not FireFox2009-08-26 15:20:57
 MineField2009-08-26 15:21:03
<Simon|B> hmm dunno I am a bit strange (also for myself). For some reason I dislike Firefox and this will never change :-D2009-08-26 15:34:43
 It's the same for Mercedes, WOW, Aldi..... :D2009-08-26 15:35:49
<Skymmer> Sure2009-08-26 15:36:26
 I loved Opera in a days when traffic coasts me too much2009-08-26 15:37:06
<Simon|B> Yeah but most times I have no real arguments ;)2009-08-26 15:37:10
<Skymmer> So I was turning off all images and was happy :)2009-08-26 15:37:38
<Simon|B> ah I forgot winamp. I also hate it^^2009-08-26 15:37:44
<Skymmer> I dropped it after I met foobar2009-08-26 15:38:14
 Foobar2000 forever for me :)2009-08-26 15:38:36
<Simon|B> Yes it now has a proxy server which can lossy reduce images heavily before sending2009-08-26 15:38:43
 aimp for me ;)2009-08-26 15:38:53
 tested foobar but it was too freaky2009-08-26 15:39:11
*** Surfer has joined the channel2009-08-26 15:40:34
<Skymmer> Well, design for audio player is the last I'm looking at2009-08-26 15:40:43
<Simon|B> hi Surfer2009-08-26 15:40:44
<Surfer> hi2009-08-26 15:40:52
<Skymmer> after all, you listen to music.2009-08-26 15:41:12
<Simon|B> indeed2009-08-26 15:41:20
<Skymmer> But foobar's possibilities just fascinating2009-08-26 15:41:41
<Surfer> songbird great :)2009-08-26 15:42:15
<Skymmer> foobar for me its first of all functionality2009-08-26 15:43:46
 Great converter with possibility to plug any command line encoder to it f.e.2009-08-26 15:44:22
 tagger with custom scripting2009-08-26 15:44:41
<Simon|B> yeah foobar is famous for it's features. 2009-08-26 15:44:44
 It's for me the same like miranda and firefox2009-08-26 15:45:04
 it's nothing without plugins2009-08-26 15:45:17
<Skymmer> Sure. With foobar I can also rip CDs with CD-Paranoia method and even write something to CD-Rs\CD-RWs. I'm not talking about great playlist manipulation system2009-08-26 15:48:51
<Surfer> proper cd-rips can only be done with ExactAudioCopy2009-08-26 15:49:43
<Skymmer> After all we all have out prefernces :)2009-08-26 15:49:54
 *our2009-08-26 15:50:04
 No2009-08-26 15:50:21
 EAC is not the only2009-08-26 15:50:30
<Surfer> eac is standart2009-08-26 15:51:03
<Skymmer> CD-paranoia (which is also used in CDex) gives me better results sometimes2009-08-26 15:51:11
 on scratched CDs2009-08-26 15:51:22
<Simon|B> "only be done" -.-2009-08-26 15:53:59
 there are hundreds of good mucic rip tools2009-08-26 15:54:13
<Skymmer> But I'm agree. EAC is just great2009-08-26 15:56:13
 Actually there are only two good ripping tools2009-08-26 15:56:33
 EAC and CDex2009-08-26 15:56:40
 They're care about such things as your device offset, error handling, if your device uses audio caching or not, gaps and etc.2009-08-26 15:57:51
 Also you can plug any command line encoder of your choice to it, use presets2009-08-26 15:58:46
 I'm also have one story with EAC2009-08-26 15:59:09
 Once I've found one undocumented command-line for ir2009-08-26 15:59:28
 *it2009-08-26 15:59:32
 And published it on forum with the question: What its for?2009-08-26 16:00:00
*** pinc has left the channel2009-08-26 16:00:28
 My post have been quickly edited and author of EAC contacted me2009-08-26 16:00:30
 He asked me NOT to ask about this option anymore2009-08-26 16:00:59
 I've pushed on him2009-08-26 16:01:11
<Surfer> what option ? :D2009-08-26 16:01:30
<Skymmer> And he said that this option is for paying customer2009-08-26 16:01:45
 Well, I don't know if its fair to publish it ???2009-08-26 16:02:39
*** chornobyl has joined the channel2009-08-26 16:39:41
<osman> hi chorbobyl2009-08-26 16:51:18
 did you test the program?2009-08-26 16:51:26
<chornobyl> just woke up2009-08-26 16:51:38
<osman> %)2009-08-26 16:51:51
<chornobyl> i'll try i in small scale2009-08-26 16:51:57
<osman> are you a bat-man :)2009-08-26 16:51:58
<chornobyl> not me Eugene are2009-08-26 16:52:25
<osman> just woke up for feeding by sucking blood from humans :)2009-08-26 16:52:29
 eugene has a really "time-shift" problem. he's "out of scope" for me :)2009-08-26 16:53:03
 anyway. i have to eat something. just drop a comment when you test it2009-08-26 16:53:33
<chornobyl> i'm not going ti bite anyone2009-08-26 16:53:49
 ok2009-08-26 16:53:49
*** sami has joined the channel2009-08-26 17:12:16
*** Skymmer has left the channel2009-08-26 17:15:07
<osman> I wanna buy a new DDR2 notebook ram set (2x2 Gib or more). single question: what's your offer? OCZ vs kingston?2009-08-26 17:23:14
<Surfer> kingston more PR'ed brand :)2009-08-26 17:26:45
<osman> i know but most of people claim that ocz is obviously better2009-08-26 17:27:32
 note that my current rams are already kingston :) you can see it's performance in the forum :)2009-08-26 17:28:08
 the problem is i have 2 GiB in total but it's not useful for Vista x642009-08-26 17:28:30
 it eats more than 1 GiB by itself2009-08-26 17:28:41
 you may say "install xp then"2009-08-26 17:28:55
 but it's not an option. because vista x64 really boosts my core duo performance. i've tested xp, xp 64, vista x86 and vista x642009-08-26 17:29:36
 vista x64 seems best. but it's a really memory sucker2009-08-26 17:29:54
 in fact, it can efficiently reorganize memory fragments. but, as i said it really need ~4 GiB memory2009-08-26 17:30:24
<Shelwien> hi btw2009-08-26 17:41:11
<sami> hi. so have you yet tried a faster rc for the ccmsh?2009-08-26 17:43:57
<Shelwien> no, only discussed the ideas with toffer2009-08-26 17:44:27
 well, in fact i tried something in sh1d2009-08-26 17:44:37
 moved the rc renorm2009-08-26 17:44:46
 and replaced the range update with gnu inline asm2009-08-26 17:45:01
 and added some templates2009-08-26 17:45:08
 ...but performance maybe actually degraded because of that %)2009-08-26 17:45:34
 anyway, what do you think about2009-08-26 17:46:09
<sami> hmm. your rc had an unsigned mul right? how about getting rid of that2009-08-26 17:46:10
<Shelwien> i don't think that (range>>LOG)*freq is any better2009-08-26 17:47:06
<sami> yes it is2009-08-26 17:47:18
<Shelwien> well, it can be, but because of other optimizations mainly2009-08-26 17:47:39
<sami> I mean discarding the low bits for speed2009-08-26 17:47:42
<Shelwien> i mean, such multiplication itself isn't any faster imho2009-08-26 17:48:10
 i think it computes whole 64 bits anyway2009-08-26 17:48:25
 just discards the high dword2009-08-26 17:48:36
 <Shelwien> anyway, what do you think about2009-08-26 17:48:51
 1. exp(log(range)+log(prob)) approximation (with lookups)2009-08-26 17:49:10
 2. runlength coding of bits sorted by quantized probability2009-08-26 17:49:37
 3. multiple rc (8 or more) interleaving (like in parcoder)2009-08-26 17:50:06
 4. caching the probabilities for data block and rangecoding in another loop (for encoder)2009-08-26 17:51:25
 3+4. splitting encoding process into 3 loops (model,low/range updates,renorms) and vectorizing low/range updates2009-08-26 17:52:55
 5. 16-bit i/o in rc (I have a 32-bit implementation too, but its much slower)2009-08-26 17:54:51
<Surfer> osman: try Win7 x64, it eats lower memory and have better perfomance2009-08-26 17:55:21
<osman> @Surfer: Thanks. I'll try when I get ms partner dvd from my brother (=valid license). is it still in RC state, right?2009-08-26 17:56:43
 @Shelwien: option 1 is frequently used in FPGAs :)2009-08-26 17:57:43
<Surfer> i think you can upgrage from vista2009-08-26 17:57:55
<Shelwien> its used in damned wz_jpeg2009-08-26 17:58:01
 but it should be possible to make a better version - simpler and without losing that much precision2009-08-26 17:58:35
<sami> I don't think 2. will will be any good, it will not work. but I don't have strong feelings for any of those. of course none of the optimizations will bring much speedup, but would have to see them implemented and tested to say are they interesting. I wouldn't expect 4. and 5. to give any consistent results (might be compressor/implementation specific)2009-08-26 18:03:09
<Shelwien> 5 is implemented in fpaq0pv4B and gives results2009-08-26 18:03:48
*** osman has left the channel2009-08-26 18:03:51
 2 is implemented too, and allows to skip 300f rc calls on enwik2009-08-26 18:04:32
 but unfortunately in my implementation it seems well compensated by a probability branch2009-08-26 18:05:23
<sami> yes I remember you told me about 2, but you all you said back then was "run length coding on bits". I expect that to add lot's of brances and slowdown2009-08-26 18:05:28
<Shelwien> so only encoding is somewhat faster2009-08-26 18:05:37
 no, my implementation is actually faster, just not that much as one expects given 30ut in rc calls2009-08-26 18:06:24
 and also it even improves compression a little %)2009-08-26 18:06:42
<sami> could you give me the parcoder link again, so I could check up what the multiple rcs was about?2009-08-26 18:07:17
<Shelwien> http://compression.ru/sh/parcoder.rar2009-08-26 18:07:52
<sami> thanks2009-08-26 18:08:00
<Shelwien> you can also find its discussion with toffer ~11:51 in http://nishi.dreamhosters.com/log/2009-08-26_06-10-52.txt2009-08-26 18:08:51
 anyway, its dumb and really old, but idea remains relevant imho2009-08-26 18:09:33
 in fact, it should much more important these days because of vectors and cores2009-08-26 18:10:50
 *should be2009-08-26 18:11:04
*** osman has joined the channel2009-08-26 18:13:55
<osman> electricity problem... :/2009-08-26 18:14:05
 btw, @Shelwien: i mean instead of multiplication, log via lookups. FPGAs supports lookups by their natures.2009-08-26 18:14:14
<sami> about the parcoder. you open multiple input files for compressing? is this just because this is a simple test or is there somethng I don't understand?2009-08-26 18:19:31
<Shelwien> its a "parallel rangecoder"2009-08-26 18:19:53
 it opens multiple input files2009-08-26 18:19:58
 and encodes the bytes from each2009-08-26 18:20:08
 into a single code stream2009-08-26 18:20:15
 in a single loop2009-08-26 18:20:17
<sami> right, but you could use a single file instead2009-08-26 18:20:25
<Shelwien> which can be (partly) vectorized2009-08-26 18:20:30
 input? yeah, but it won't be that easy to understand imhop2009-08-26 18:20:52
 *imho2009-08-26 18:20:56
<sami> well, ok, it would have been for me :-)2009-08-26 18:22:34
<Shelwien> ok, i guess maybe a complete and really vectorized implementation would be more usable2009-08-26 18:23:34
 but i only wanted to demonstrate the rc delay sync2009-08-26 18:23:56
 instead of messing with many technical details of how to really vectorize it2009-08-26 18:24:25
<sami> anyway, perhaps having multiple rcs for the current cms might be interesting/fun. even though I wouldn't be optimistic about any speedup2009-08-26 18:25:54
<Shelwien> ok, i'd demonstrate it soon (i hope) ;)2009-08-26 18:27:56
<Simon|B> shelwien I have this second schroeder book if you want2009-08-26 18:29:56
<Shelwien> i found it in ed2k too ;)2009-08-26 18:30:36
<Simon|B> ok2009-08-26 18:30:46
<Shelwien> and paid a person today to OCR and then translate it into russian ;)2009-08-26 18:31:06
<Simon|B> finding isn't hard but only a few people still using it ;)2009-08-26 18:31:12
 lol? Not really? :-D2009-08-26 18:31:33
<Shelwien> really2009-08-26 18:31:38
<Simon|B> the one you wanted or this?2009-08-26 18:31:53
<Shelwien> the "computer speech" obviously2009-08-26 18:32:11
<Simon|B> yes2009-08-26 18:32:22
<Shelwien> its highly relevant for my current job (audio codec development)2009-08-26 18:32:26
<Simon|B> don't you understand english well??2009-08-26 18:32:32
<Shelwien> i do, and bought the book, and read it already2009-08-26 18:32:44
 but other developers are in other cities2009-08-26 18:32:57
 and I think they don't have enough patience even to read a one specific chapter in english2009-08-26 18:33:28
 (even if they can read it at all, which i doubt)2009-08-26 18:33:40
*** Simon|B has left the channel2009-08-26 18:49:43
*** Simon|B has joined the channel2009-08-26 18:53:03
<Simon|B> how good/bad is this http://www.hex-rays.com/decompiler.shtml ?2009-08-26 19:07:04
<Shelwien> unfortunately i didn't see the new version2009-08-26 19:07:34
 but i posted a link with ida 5.2 including that decompiler yesterday2009-08-26 19:07:58
<Simon|B> so you never tried it?2009-08-26 19:08:17
<Shelwien> not the new one2009-08-26 19:08:25
 there're no pirated versions, and they ask $2k for it2009-08-26 19:08:41
 and for $2k i'd write such a thing myself2009-08-26 19:08:50
 anyway, its the best available x86 decompiler2009-08-26 19:09:26
<Simon|B> 1.1 is so much better?2009-08-26 19:09:49
<Shelwien> i don't know, and really doubt in fact2009-08-26 19:10:08
 it has float-point support though2009-08-26 19:10:19
 at least i know that 5.5 is not noticeably better than 5.22009-08-26 19:10:44
 although it has some nice external features2009-08-26 19:11:19
 like python etc scripting2009-08-26 19:11:28
 and iphone debugging2009-08-26 19:11:35
 (remote)2009-08-26 19:11:41
 and alternate debugger support2009-08-26 19:12:24
<Simon|B> 5.5?2009-08-26 19:13:36
 only saw 5.42009-08-26 19:13:50
 indeed 5.52009-08-26 19:14:04
 hex rays page says 5.5 releases and ida pro page says latest news 5.4 .... -.-2009-08-26 19:14:45
<Shelwien> well, i have a real 5.5 license2009-08-26 19:15:43
<Simon|B> nice2009-08-26 19:15:50
 I also buy all good software2009-08-26 19:16:02
<Shelwien> but can't say its worth the money2009-08-26 19:16:03
 except that i had to hack it right away2009-08-26 19:16:11
<Simon|B> If it's anywhere near my budget2009-08-26 19:16:16
<Shelwien> to load my old databases2009-08-26 19:16:17
<Simon|B> for example I owned hexworkshop^^2009-08-26 19:16:23
<Shelwien> "owned" has a few meanings ;)2009-08-26 19:16:54
<Simon|B> hehe2009-08-26 19:16:58
 bought it ;)2009-08-26 19:17:02
 I use this software so much and didn't find any other good one2009-08-26 19:17:29
<Shelwien> btw, did you see the contest there?2009-08-26 19:17:43
 i can participate2009-08-26 19:17:48
<Simon|B> yes I read it2009-08-26 19:17:55
 try it ;)2009-08-26 19:18:04
<Shelwien> so if you have any good ideas on what to write2009-08-26 19:18:07
 i'd like to hear it ;)2009-08-26 19:18:10
<Simon|B> you seem to know very mmuch about revers engeneering why don't you write something groundbreaking direct to c++ or similar ;)2009-08-26 19:19:07
<Shelwien> i did write what i needed ;)2009-08-26 19:19:28
<Simon|B> write something to earn money^^2009-08-26 19:20:16
<Shelwien> for example, this asm2c script http://shelwien.googlepages.com/sub_1008DCA0.txt2009-08-26 19:20:23
 was the main tool for ccm reversing2009-08-26 19:20:36
 and well, i have a few ideas of my own for that contest2009-08-26 19:22:54
 but somehow they are all not that ida-specific2009-08-26 19:23:06
<Simon|B> hehe2009-08-26 19:23:26
<Shelwien> more like just some standalone reversing tools2009-08-26 19:23:32
<Simon|B> did you ever used ollydbg?2009-08-26 19:24:20
<Shelwien> not really ;)2009-08-26 19:24:30
*** Shelwien has left the channel2009-08-26 19:29:23
*** Shelwien has joined the channel2009-08-26 19:29:41
 e,e2009-08-26 19:29:50
*** Skymmer has joined the channel2009-08-26 19:35:01
<Surfer> я вот думаю, а почему nosso не декомпилировать или там проблемы есть ? :)2009-08-26 19:43:38
<Shelwien> а зачем? ;)2009-08-26 19:43:52
<Skymmer> Там же только decompressor2009-08-26 19:44:02
<Shelwien> ну там же ничего нетривиального нет ;)2009-08-26 19:44:23
<Surfer> ну ну по декомпрессору тоже вроде2009-08-26 19:44:27
<Shelwien> по крайней мере, такого, что проще декомпилировать, чем написать ;)2009-08-26 19:44:36
<Skymmer> Лучше уж ROLZ3 из WinRK2009-08-26 19:44:59
<Surfer> ну разьве есть такой компрессор, арчивер, который может адобе ридер ужат ?2009-08-26 19:45:02
<Shelwien> хз, я не пробовал2009-08-26 19:45:23
<Surfer> winrk v3.11 normal 39.8mg2009-08-26 19:45:36
 7zip 40.6mg2009-08-26 19:45:36
 stuffit12 52.1mg2009-08-26 19:45:36
 nosso 33.4mg2009-08-26 19:45:36
<Shelwien> ну это надо смотреть, что там за форматы файлов, и как их лучше сжать2009-08-26 19:46:09
 но известно, что там ничего круче LZMA не применяется2009-08-26 19:46:20
<Surfer> ну там в основном екзешки и дллки, но по идее BCJ2 должен был справиться лушче2009-08-26 19:46:41
<Shelwien> а, ну это не факт2009-08-26 19:46:57
 ты дурилку пробовал?2009-08-26 19:47:02
<Surfer> нее, это вообще из темы http://encode.ru/forum/showthread.php?t=1402009-08-26 19:47:17
 но думаю данные совпадут2009-08-26 19:47:30
<Shelwien> в дурилке полный дизассемблер, а не bcj22009-08-26 19:48:00
<Skymmer> В Nosso заточенные фильтры, пере-создавание CAB архивов и прочие хитрости. Все это вроде уже обсасывалось :)2009-08-26 19:48:41
<Shelwien> а кроме того, если там есть картинки или звук (например, в ресурсах), то их лучше соответствующими средствами сжимать2009-08-26 19:49:07
 не говоря уже о рекомпрессии кабов и т.п.2009-08-26 19:49:22
<Surfer> ну это понятно, но не думаю что прекомп например даст существенный прирост например2009-08-26 19:49:24
 притом замедлит сильно2009-08-26 19:49:40
<Shelwien> ну в принципе я же и не говорю, что там нечего делать2009-08-26 19:50:07
<Skymmer> Там в теме где-то Nanoflooder все что только можно попробывал. Поищи ссылку на XLS файл.2009-08-26 19:50:30
<Shelwien> я говорю, что есть масса очевидных вещей, которые проще сделать самому, чем декомпилировать2009-08-26 19:50:40
<Surfer> ну я видел этот файл, щя ещё посмотрю2009-08-26 19:51:49
 рядом стоит только нанозип, но скорость распаковки в 2 раза ниже :)2009-08-26 19:53:15
 и на 15 секунд дольше2009-08-26 19:53:31
<Shelwien> ну так тем более... и дурилку имхо никто не проверял...2009-08-26 19:53:38
<Skymmer> Там в этой дурилке свихнуться можно от настройки триков2009-08-26 19:55:39
 А если все перебирать то вообще весла суши2009-08-26 19:56:03
<Simon|B> lol2009-08-26 19:56:31
 looks funny for my eyes2009-08-26 19:56:38
<Shelwien> да оно просто на самом деле, только старая дурилка нужна ;)2009-08-26 19:56:44
<Skymmer> :) Но так да. Надо попробывать2009-08-26 19:56:54
<Shelwien> запускаешь с -l, подбираешь order и тип отдельно для сегментов2009-08-26 19:57:00
 а потом собираешь командную строку2009-08-26 19:57:10
 в новой версии он просто -l отломал2009-08-26 19:57:25
<Skymmer> А какая именно? 0.5 если не ошибаюсь ?2009-08-26 19:57:45
<Shelwien> http://shelwien.googlepages.com/durilca2_002a.rar2009-08-26 19:58:15
<Surfer> Simon|B: http://www.microsofttranslator.com/ :D2009-08-26 19:58:43
<Simon|B> it detected italian nice start... :D2009-08-26 20:05:28
<Shelwien> %)2009-08-26 20:05:34
<Simon|B> it seems to translate absolutely nothing2009-08-26 20:06:27
<Surfer> or google translate :)2009-08-26 20:07:19
<Shelwien> http://translate.google.com/translate?hl=en&sl=ru&tl=en&u=http%3A0.0000000.000000nishi.dreamhosters.com0.000000log0.0000002009-08-26_06-10-52.txt2009-08-26 20:07:30
 its funny2009-08-26 20:07:34
<Simon|B> there were anything wrong with what I have here2009-08-26 20:08:01
 google also showed no valid result2009-08-26 20:08:11
<Shelwien> why, it translated something ;)2009-08-26 20:08:38
<Skymmer> :D2009-08-26 20:09:12
<Simon|B> this one is fine. I meant mine :p2009-08-26 20:09:19
<Shelwien> its not fine actually2009-08-26 20:10:28
 it didn't translate half of the words ;)2009-08-26 20:10:34
<chornobyl> just watched some really GOOD movie2009-08-26 20:15:50
<Skymmer> Double Air Bags 28 ?2009-08-26 20:16:34
<chornobyl> 2007 Du levande2009-08-26 20:17:17
 http://www.imdb.com/title/tt0445336/2009-08-26 20:17:44
*** osman has left the channel2009-08-26 20:20:38
<Surfer> гуд найт всем :)2009-08-26 20:32:35
*** Surfer has left the channel2009-08-26 20:32:39
*** osman has joined the channel2009-08-26 20:35:30
<osman> chornobyl did you try the program?2009-08-26 20:42:06
<Skymmer> What program if its not a secret ?2009-08-26 20:48:14
<chornobyl> how to get errorcode used2009-08-26 20:48:47
<osman> hehe :) secret!? :) not really2009-08-26 20:48:48
<Skymmer> Ahh, understood2009-08-26 20:49:22
<osman> yesterday chornobyl demand a program which compares NTFS compressed size vs [original size]*expectedRatio2009-08-26 20:49:26
<chornobyl> "demand"2009-08-26 20:49:50
<osman> it returns -1, 0 and +1 for each cases2009-08-26 20:50:03
<Skymmer> And purpose?2009-08-26 20:50:22
<chornobyl> so there should be some ot i guess2009-08-26 20:50:27
 all written there2009-08-26 20:50:36
 http://nishi.dreamhosters.com/log/2009-08-25_04-27-49.txt2009-08-26 20:50:39
<osman> @chornobyl: it may seems rude. seems it's better to use "wishes" :)2009-08-26 20:50:52
<Skymmer> I thought Bit v0.08 ;)2009-08-26 20:52:01
<osman> @chornobyl: look at here for error levels: http://www.robvanderwoude.com/errorlevel.php2009-08-26 20:52:04
<chornobyl> Давайте жить, во всём друг другу потакая,2009-08-26 20:52:16
 тем более, что жизнь короткая такая...2009-08-26 20:52:16
 i dont know how to explain it2009-08-26 20:52:40
<osman> @skymmer: there won't be any update until middle of September :/2009-08-26 20:52:40
<Skymmer> Давайте мы не будем потакать2009-08-26 20:53:26
 А просто веселится и порхать2009-08-26 20:53:29
<chornobyl> is this yours?2009-08-26 20:53:46
<Skymmer> Yes2009-08-26 20:53:54
 Just assembled2009-08-26 20:54:01
<chornobyl> cause i quoted Omar Hayam2009-08-26 20:54:13
<osman> @Skymmer: but still collecting ideas...from constructing my own version of state table (for CM part), faster and better LWCX, and better LZ compressor (with fast & slow variant) etc etc2009-08-26 20:54:17
 do you know Omar Hayam ? :)2009-08-26 20:54:35
<Skymmer> I heard this name but never read it2009-08-26 20:54:57
<chornobyl> just two shorts2009-08-26 20:55:12
 this one where written in my literature class at school2009-08-26 20:55:35
<osman> he's the inventor of binomial series :)2009-08-26 20:55:44
<chornobyl> what arm how?2009-08-26 20:56:02
<osman> ?2009-08-26 20:56:17
<chornobyl> are you serious2009-08-26 20:56:36
<osman> yep2009-08-26 20:56:40
<chornobyl> i ma y mistake but i think we talk about different people2009-08-26 20:57:31
<osman> http://en.wikipedia.org/wiki/Г–mer_Hayyam2009-08-26 20:57:41
<chornobyl> guess not2009-08-26 20:58:06
<Skymmer> I write some poetry for myself. Quite rarely and basicly some pessimistic stuff2009-08-26 20:58:31
<chornobyl> good for you (молодец)2009-08-26 20:59:08
<Skymmer> Well, I can give you a couple if you want2009-08-26 21:00:04
<osman> in turkish his name is written like that: "Г–mer Hayyam" and he is a persian mathematician, poet, astronomer, philosopher. And "Omar Hayam" name seems very familiar with it2009-08-26 21:00:11
<chornobyl> шеы ршь ш сруслув цшлш2009-08-26 21:00:30
 its him i checked wiki2009-08-26 21:00:40
<osman> seems you wrote last sentence with russian alphabet :)2009-08-26 21:01:17
<chornobyl> btw irc support international symbols badly2009-08-26 21:01:27
 yes2009-08-26 21:01:28
 i told already2009-08-26 21:01:38
 i mess with 3 keybord layouts2009-08-26 21:01:50
<toffer> what about panufti lwowitsch chebychow - that russian mathematican saved me today :)2009-08-26 21:01:58
<chornobyl> english/russian/ukrainian2009-08-26 21:02:14
<toffer> and hi btw2009-08-26 21:02:15
<Skymmer> hi!2009-08-26 21:02:21
<chornobyl> hi toffer2009-08-26 21:02:25
<osman> hi2009-08-26 21:02:35
<toffer> just stopped writing on my thesis for today2009-08-26 21:02:38
<chornobyl> how shebyshev saved you?2009-08-26 21:02:40
<Skymmer> Probably Chebyshev2009-08-26 21:02:57
<toffer> well it's transcripted in may ways2009-08-26 21:03:06
<chornobyl> i mistype, my bad2009-08-26 21:03:20
<osman> btw, toffer i just looked up your FSM state construction but i was a bit shocked when i see you don't use entropic metric to find "nearest" state as next state. am i right?2009-08-26 21:03:39
<toffer> i remembered some law about the pdf mass location of random variables w/o making assumptations about a specifiy distribution2009-08-26 21:03:42
 it`s the chebychow inequality2009-08-26 21:03:48
 pertty useful2009-08-26 21:03:53
 you are right2009-08-26 21:04:16
 how do you define entropy metric?2009-08-26 21:04:27
 and states have more in common with run lengths than probabilites2009-08-26 21:04:56
<Skymmer> That Chebyshev (if we talk about same person) is the author of one filtering method used in Scientific Filters in Adobe Audition sound editor I use2009-08-26 21:05:10
<osman> you used TK estimator to get their static probabilities. then why don't you use code lengths to detect "similar" states?2009-08-26 21:05:18
<toffer> erm no2009-08-26 21:05:21
 chebychew polynomals can be used in filter construction2009-08-26 21:05:35
 define code length2009-08-26 21:06:03
<osman> %)2009-08-26 21:06:15
<toffer> it's at as simple as it seems2009-08-26 21:06:24
<osman> seems "you are woke up on your wrong side" ( a turkish idiom)2009-08-26 21:06:58
<toffer> not as simple2009-08-26 21:06:59
<osman> *you woke2009-08-26 21:07:18
<toffer> these mistypings really tend to overwhelm me 2009-08-26 21:08:08
<chornobyl> here i sounds "from the wrong leg'2009-08-26 21:08:18
<toffer> yeah in german, too2009-08-26 21:08:40
 so - where's the beef?2009-08-26 21:08:48
<Skymmer> @osman: In Russia its: "Stand up from the another leg" :)2009-08-26 21:08:56
<osman> if you have a probability for a symbol then we know it's "optimal" code length with shannon formula. I bet that you already know. else you wouldn't write any CM %)2009-08-26 21:08:57
<toffer> ok2009-08-26 21:10:35
 so tell me what do the states with p=0.8 and p=0.2 in common?2009-08-26 21:10:47
 they share the same entropy2009-08-26 21:10:51
 and the metric i use2009-08-26 21:11:00
 is not probabiltiy actually2009-08-26 21:11:05
 but run length2009-08-26 21:11:07
<osman> it depends on "seen" n1 and n0. that's another point of view of run length2009-08-26 21:11:43
<toffer> what i do for an update is2009-08-26 21:11:56
 decrease max(n1,n0)2009-08-26 21:12:01
 increase n(1-y)2009-08-26 21:12:08
 and find the corresponding best match w.r.t max(n1,n0)2009-08-26 21:12:21
 and if it doesn't fit in n(1-y) other run length contexts are probed for2009-08-26 21:12:45
 i was rather surprised when i found out that matt did it the same way2009-08-26 21:12:57
 but it's not about finding the state with the closest prob2009-08-26 21:13:10
 i tried to simulate a linear counter, too2009-08-26 21:13:19
 it was much worse2009-08-26 21:13:22
 that's cause of ordinary bit histories seem to be non-combinatoric2009-08-26 21:13:57
<osman> ok2009-08-26 21:13:59
<toffer> as i said it's not as simple as it seems2009-08-26 21:14:19
 and i found out that three run length contexts (the small counter = 0, 1, 2) is actually enough2009-08-26 21:14:53
 are2009-08-26 21:14:56
 do you have any other ideas?2009-08-26 21:15:17
 i mean i tried alot2009-08-26 21:15:24
 and that one turned out to be best2009-08-26 21:15:31
 (better than paq)2009-08-26 21:15:35
<osman> probably you are right at that limit. because most of context history looks like that: 0000000000000010000000000001000000000002009-08-26 21:15:44
<toffer> actually in my m1 0.4 candidate2009-08-26 21:16:25
<osman> seems including last 2 bits can be good too2009-08-26 21:16:39
<toffer> there's support for that paq-ish states which remember the last bit seen2009-08-26 21:16:44
 but that optimizer turns that off2009-08-26 21:16:55
 but the optimal state allocation seems to be similar to paq2009-08-26 21:18:13
 these "limit" pairs2009-08-26 21:18:30
 so do you have any other ideas there to try?2009-08-26 21:18:54
<osman> not yet. just found that including 2 bits can be good but there is not any proof about it2009-08-26 21:19:32
 so, there will be 64 states left2009-08-26 21:19:47
 btw, how do you quantize those static probabilities? seems you use a lookup table.2009-08-26 21:21:03
 but, there is a question about it2009-08-26 21:21:12
 suppose we have a probability, P2009-08-26 21:21:24
 and we want to quantize it into Q2009-08-26 21:21:36
<toffer> yeaj2009-08-26 21:22:00
 yeah2009-08-26 21:22:02
<osman> you optimize a bit string which hold N bits2009-08-26 21:22:02
 which is much smaller than P range2009-08-26 21:22:12
<toffer> i don't use a table for quantisation btw2009-08-26 21:22:13
<osman> so?2009-08-26 21:22:22
<toffer> guess it still used that pqmap2009-08-26 21:22:49
 ok2009-08-26 21:22:57
 so you're right2009-08-26 21:23:01
<osman> then it uses lookup :)2009-08-26 21:23:07
 anyway2009-08-26 21:23:11
<toffer> it's just that the last release was somewhere in april2009-08-26 21:23:12
<osman> i mean in short2009-08-26 21:23:19
<toffer> the quantisation in there works like that2009-08-26 21:23:28
<osman> Q = tableq[P]2009-08-26 21:23:30
 but P range is much greater than N2009-08-26 21:23:38
<toffer> yeah2009-08-26 21:23:43
 let me explain2009-08-26 21:23:46
<osman> so, you probably use Q=tableq[P>>S];2009-08-26 21:23:55
<toffer> that "lookup" table2009-08-26 21:24:07
<osman> then how do you determine initial S value?2009-08-26 21:24:07
<toffer> i don't use any2009-08-26 21:24:18
 that lookup table2009-08-26 21:24:21
 contains a monotnous increasing function2009-08-26 21:24:38
 initially2009-08-26 21:24:42
 with a high slope where alot resolution is required2009-08-26 21:24:56
 and a low slope where less precision is used2009-08-26 21:25:06
 it is encoded as a picewise linear function2009-08-26 21:25:21
 the bits in that bitmask encode the positions of vertices2009-08-26 21:25:41
<osman> so kind of sum of linear functions in different ranges?2009-08-26 21:25:45
<toffer> not sum2009-08-26 21:25:51
<osman> i meant for different ranges there are different linear functions right?2009-08-26 21:26:25
<toffer> not exactly2009-08-26 21:26:36
 that function maps from P -> [0, 1]2009-08-26 21:27:01
 and each range in between two bits2009-08-26 21:27:25
 maps to a constant portion of output space2009-08-26 21:27:35
 thus if two adjacent bits are far away2009-08-26 21:27:51
 the slope will be low2009-08-26 21:27:57
 and than there's another parameter, which is used to decide in how man values the mapped domain [0 1] is to be divided2009-08-26 21:28:39
 but some time ago i just had the idea to use tha actual fsm state allocation for quantisation2009-08-26 21:30:19
 since it represents the distribution of input2009-08-26 21:30:32
 worked even better and requires less parameters2009-08-26 21:30:40
 -.-2009-08-26 21:36:33
<osman>  p = pqMap(p>>counter1::P_LOG-MIX_P_DOMAIN_LOG); 2009-08-26 21:42:27
 i think Q=table[P>>shift]; is same as above2009-08-26 21:42:48
<toffer> it's just to avoid alot of bits for that pdomain2009-08-26 21:43:09
 anyway i've got a better solution now2009-08-26 21:43:18
*** Skymmer has left the channel2009-08-26 21:43:20
 that's obsolete2009-08-26 21:43:21
<osman> initially i had Q=table[P>>shift] idea. and i've wanted to see your idea to determine which one is better. And I saw they are same :)2009-08-26 21:45:36
 and what's new?2009-08-26 21:45:47
<toffer> the problem can be roduced to the computation of a quantisation function, right?2009-08-26 21:49:01
 reduced2009-08-26 21:49:05
 and you can map states to probabilities, thus you have a denstribution density(p)2009-08-26 21:49:28
 so you can compute a quantisation function which has a high resolution where the density is high, etc.2009-08-26 21:50:26
<osman> in the other word, you don't use lookup table anymore, right?2009-08-26 21:52:41
 but what about quantisation function complexity?2009-08-26 21:52:59
 could you give it's formula?2009-08-26 21:53:11
<toffer> it's no formula there2009-08-26 21:55:04
 it's just learned by the optimizer2009-08-26 21:55:11
<osman> at least it must have a form right? :)2009-08-26 21:55:41
<toffer> but in the end it looks pretty much like stretch2009-08-26 21:55:46
 with higher slopes near 0/12009-08-26 21:55:58
<osman> @toffer: i didn't get the point. could you explain in different way?2009-08-26 22:01:44
*** chornobyl has left the channel2009-08-26 22:02:03
<toffer> i just made a plot of the function2009-08-26 22:03:46
 which is implicitly generated2009-08-26 22:03:54
 https://ftp.tu-ilmenau.de/index.php?id=c96fd2e41a90b6abc8cbfa7bf3362157&login=ee715ee44e81ef00dec0d3691150e931&file=9ec13a913acdc29d352b96330e1e78632009-08-26 22:05:01
 as i said it looks like stretch2009-08-26 22:05:09
 did you get it?2009-08-26 22:06:42
<osman> i understood function's output. but, the question is how did you "generate" that function from optimization?2009-08-26 22:08:12
<toffer> simply by enumeration2009-08-26 22:08:38
 sort the states by max(p,1-p)2009-08-26 22:09:10
 to cluster predictions near 0/1 together2009-08-26 22:09:21
 and assign a quantisation level to each adjacent elements2009-08-26 22:09:54
 each N adjacent2009-08-26 22:09:58
<osman> seems you're only talking about state probability quantization. what about another kind of inputs?2009-08-26 22:11:34
<toffer> that's the implementation of integrating and dividing the density weighted input into elemtns of equal mass2009-08-26 22:11:38
 i don't need any other quantisation 2009-08-26 22:11:53
 and there's still the quantisation mapping in m1 0.3b2009-08-26 22:12:06
<osman> ok. then2009-08-26 22:12:14
<toffer> for a general solution2009-08-26 22:12:18
<osman> here is the same question again. in m1 0.3b how do you determine "shift" value? (q=map[p>>shift])2009-08-26 22:12:55
 or in the another word qmap length?2009-08-26 22:13:07
<toffer> it's fixed2009-08-26 22:13:18
 at 8 bits afair2009-08-26 22:13:23
 i mean p>>N = 8 bits2009-08-26 22:13:32
 or maybe 92009-08-26 22:13:34
<osman> i know it's fixed. so, that's why i'm asking that question :)2009-08-26 22:13:45
 i mean why not "say 10 bits" ?2009-08-26 22:13:57
<toffer> i use the highest value which doesn't cause an enormous amout of parameters2009-08-26 22:14:11
<osman> so, in the another word it's ad-hoc for simplifying optimization, right?2009-08-26 22:14:40
*** sami has left the channel2009-08-26 22:14:50
<toffer> i'd not write an optimizer which allows variable sized bitstrings2009-08-26 22:15:28
 yes2009-08-26 22:15:28
<osman> yes. it'll be less stable2009-08-26 22:15:45
 it's funny that i was thinking exactly same things for quantization and i thought you had better solution for all the time %)2009-08-26 22:16:39
<toffer> as i said i got a better solution2009-08-26 22:17:46
<osman> but i'm asking for "general" solution2009-08-26 22:18:03
<toffer> the most obvious thing is to write a permutation optimizer2009-08-26 22:18:23
 since discrete quantisation is just a mapping from N to M<N2009-08-26 22:18:45
<osman> yeah. it's really hard for me. ( at least for now)2009-08-26 22:18:47
<toffer> you may want to have a look at GAs for the travelling samlesmen problem2009-08-26 22:19:12
 there was something similar afair2009-08-26 22:19:28
<osman> i roughly know them already2009-08-26 22:19:32
 btw, thanks for the briefing :)2009-08-26 22:22:10
<toffer> briefing?2009-08-26 22:22:50
<osman> just a joke. i meant detailed explanation of the quantization.2009-08-26 22:23:26
 thanks2009-08-26 22:23:29
<toffer> i just know that word from some old games2009-08-26 22:23:39
 ^^2009-08-26 22:23:41
<osman> "roger sir", "i'm in position", "i want to secure this area" :) from unreal tournament series :)2009-08-26 22:24:20
 btw, i really love unreal tournament 3, it has somehow most "realistic" environment IMO %)2009-08-26 22:25:38
<toffer> i just played that cry-stuff2009-08-26 22:26:07
 well for something like 10 minutes to test my new hardware2009-08-26 22:26:19
<osman> 3 turkish are founder of that cry-tek company :)2009-08-26 22:26:51
<toffer> ^^2009-08-26 22:27:41
<osman> btw, what was your "20 bits counter"? they are rewritten form of 16 bits linear counters? what are their performance comparing to 16 bits one?2009-08-26 22:28:41
<toffer> i get something like +0.02 0etter compression2009-08-26 22:30:13
 and speed is the same2009-08-26 22:30:21
<osman> what about speed?2009-08-26 22:30:25
 ah ok2009-08-26 22:30:35
 they are same as 16 bits linear counters?2009-08-26 22:30:47
<toffer> instead of p:w = 16:16 i use 20:122009-08-26 22:30:48
 and instead of arithmetic coding with 12 bits of precision i use 202009-08-26 22:30:59
 but it was - like bob ross would say - just a little happy accident2009-08-26 22:32:07
<osman> what about pm? it should be 12 IMO?2009-08-26 22:33:13
<toffer> 202009-08-26 22:33:55
 but log-coded2009-08-26 22:33:59
 pm = 1<<z2009-08-26 22:34:06
 i made some sensitivity analysis2009-08-26 22:34:25
 to reduce the parameter spae2009-08-26 22:34:31
 space2009-08-26 22:34:33
<osman> so, p' = w*p + (1-w)*pm[y] -> [20 bits] = [12 bits]*[20 bits] + [32-12=20 bits] * [20 bits] ?2009-08-26 22:34:43
<toffer> [32-12=20 bits] that's w2009-08-26 22:35:10
 and it's 12 bit2009-08-26 22:35:15
<osman> (1-w)2009-08-26 22:35:16
<toffer> yes2009-08-26 22:35:29
 but why shall it be 20 bits 2009-08-26 22:35:39
 ^^2009-08-26 22:35:54
<osman> ah ok. it was my bad2009-08-26 22:35:58
<toffer> maybe i can extend p to be 32 bits2009-08-26 22:36:07
 w/o 64 bit arithmetic2009-08-26 22:36:15
 using the fact that every 32 bit intel cpu does return eax*somereg in eax:edx (64 bit)2009-08-26 22:36:42
 yesterday eugene provided a little piece of asm code2009-08-26 22:36:54
 gcc inline asm2009-08-26 22:37:01
 well i thought that gcc would be clever enough to convert something line r=uint64(a)*b>>32 into r=edx2009-08-26 22:37:36
 but it didn't2009-08-26 22:37:42
<osman> as i expected ;)2009-08-26 22:38:08
 anyway. it's still 64-bit multiply in 20 bits form. because only 16 bits values will be placed on eax2009-08-26 22:39:29
 else it'll also affect edx2009-08-26 22:39:40
 in the another word edx:eax pair2009-08-26 22:39:50
<toffer> ?2009-08-26 22:40:25
 well ax*something results in eax2009-08-26 22:40:40
<osman> i mean if inputs are 16 bits. then "mul" produces the result in eax only2009-08-26 22:40:53
<toffer> yeah2009-08-26 22:41:02
<osman> else results will be placed in edx:eax pair2009-08-26 22:41:07
<toffer> but all multiplications used in here are 32 bit2009-08-26 22:41:10
<osman> so, you counter pair placed in edx:eax2009-08-26 22:41:24
 *your counter2009-08-26 22:41:39
<toffer> not really2009-08-26 22:41:40
 u just use edx2009-08-26 22:41:44
 well the counter uses just eax2009-08-26 22:42:02
 since it's a 32 bit result2009-08-26 22:42:08
 but the rangecoder uses edx2009-08-26 22:42:13
 the high 32 bit2009-08-26 22:42:19
<osman> more or less, edx:eax pair is calculated ;)2009-08-26 22:42:26
<toffer> to overcome2009-08-26 22:42:26
 always, yes2009-08-26 22:42:33
<osman> that's the point what i meant ;)2009-08-26 22:42:42
 multiplication is a really hard work for low level system design. and AMD performance on multiplication is not really shock me :)2009-08-26 22:43:54
 as you know AMD's has a really bad performance on multiplications and divisons2009-08-26 22:44:20
<toffer> well ma recent x2 245 is as fast as my q66002009-08-26 22:44:41
 my2009-08-26 22:44:45
<osman> that's why mostly engineers use log tables instead of multiplication in FPGAs2009-08-26 22:44:51
<toffer> but it hast just two cores2009-08-26 22:44:53
<osman> just run BIT 0.2b and compare them again ;)2009-08-26 22:45:41
<toffer> you use divisons?2009-08-26 22:46:11
<osman> yep2009-08-26 22:46:15
 :)2009-08-26 22:46:17
<toffer> erm really?2009-08-26 22:46:20
 where?2009-08-26 22:46:21
<osman> BIT 0.2b. it's enough old. it was first attempt to write a CM :)2009-08-26 22:46:38
 in counters2009-08-26 22:46:47
<toffer> ah, ok2009-08-26 22:46:56
 ah more recent version had a number like 0.7 or something2009-08-26 22:48:25
 i just wondered2009-08-26 22:48:28
<osman> yep. latest one is 0.72009-08-26 22:48:41
 anyway. i'll go now. cya. good night2009-08-26 22:50:03
<toffer> ok2009-08-26 22:50:51
 gn82009-08-26 22:50:52
*** Skymmer has joined the channel2009-08-26 23:07:02
 does anybody here have a 64 bit linux box?2009-08-26 23:08:08
*** Simon|B has left the channel2009-08-26 23:08:38
*** toffer has left the channel2009-08-26 23:19:49
*** Skymmer has left the channel2009-08-26 23:25:42
*** FuSiyuan has joined the channel2009-08-26 23:39:44
*** Skymmer has joined the channel2009-08-27 00:12:48
* Shelwien in fact has something alike to a 64-bit linux box. its a macbook which kinda a 64-bit freebsd box ;)2009-08-27 01:03:11
<Shelwien> !next2009-08-27 01:03:16