*** chornobyl has joined the channel | 2009-08-26 06:37:27 |
*** pinc|mirror has joined the channel | 2009-08-26 06:39:39 |
*** pinc has left the channel | 2009-08-26 06:40:05 |
*** FuSiyuan has left the channel | 2009-08-26 07:25:43 |
*** chornobyl has left the channel | 2009-08-26 08:57:16 |
*** pinc|mirror has left the channel | 2009-08-26 09:05:53 |
*** Simon|B2 has joined the channel | 2009-08-26 09:07:47 |
*** chornobyl has joined the channel | 2009-08-26 09:09:42 |
<Shelwien> | ... | 2009-08-26 09:33:43 |
<chornobyl> | dots | 2009-08-26 09:34:02 |
<Shelwien> | can you confirm Skymmer's result that sh1d is slower than sh1c? | 2009-08-26 09:34:25 |
<chornobyl> | give links ill try | 2009-08-26 09:34:37 |
<Shelwien> | http://shelwien.googlepages.com/ccm_sh1c.rar | 2009-08-26 09:34:57 |
| http://shelwien.googlepages.com/ccm_sh1d.rar | 2009-08-26 09:34:59 |
| http://shelwien.googlepages.com/ccm_sh1d1.rar | 2009-08-26 09:35:02 |
<osman> | hi chorbobyl | 2009-08-26 09:36:37 |
| hi shelwien | 2009-08-26 09:36:49 |
<Shelwien> | hi osman ;) | 2009-08-26 09:36:59 |
<osman> | @chornobyl: yesterday when i finish your tool. you had gone already :) | 2009-08-26 09:37:20 |
<chornobyl> | nice to hear it | 2009-08-26 09:37:40 |
<osman> | @shelwien: how are you? | 2009-08-26 09:37:42 |
<Shelwien> | sleepy. fighting bugs. | 2009-08-26 09:39:50 |
<osman> | :) | 2009-08-26 09:40:24 |
| still dejpg? | 2009-08-26 09:40:32 |
<Shelwien> | no, other job this time | 2009-08-26 09:40:42 |
| PA with dejpg is out btw | 2009-08-26 09:40:47 |
<osman> | you mean you completely finished dejpg? | 2009-08-26 09:41:13 |
<Shelwien> | ...i did before starting this ccm project ;) | 2009-08-26 09:41:35 |
<osman> | @chornobyl: grap it here: www.osmanturan.com/sizeondisk.zip | 2009-08-26 09:41:44 |
| @shelwien: it's good then. what's it this time? | 2009-08-26 09:42:11 |
<chornobyl> | thank you | 2009-08-26 09:42:24 |
<Shelwien> | alien bugs | 2009-08-26 09:42:31 |
| ...wanted to try the interleaved rc idea on ccm | 2009-08-26 09:42:48 |
| but no time yet | 2009-08-26 09:43:01 |
<osman> | @chornobyl: you're welcome :) source code included btw. and i think it's enough easy to use. | 2009-08-26 09:43:09 |
| @shelwien: alien bugs!? :) | 2009-08-26 09:43:24 |
<Shelwien> | well, they're made by other developers | 2009-08-26 09:44:12 |
<chornobyl> | eugene what is commandline | 2009-08-26 09:44:32 |
<Shelwien> | for ccm? | 2009-08-26 09:44:41 |
<chornobyl> | yepp | 2009-08-26 09:44:47 |
<Shelwien> | ccm c/d input output ;) | 2009-08-26 09:44:51 |
| there's t.bat for test | 2009-08-26 09:45:00 |
| but skymmer used enwik8 | 2009-08-26 09:45:12 |
<osman> | i really hate this kind of jobs. at a time, i have to hunt my big brother program's bugs. they were huge scale database applications and it was not easy task | 2009-08-26 09:45:19 |
<Shelwien> | these multithreading bugs are not really nice either ;) | 2009-08-26 09:45:59 |
| what do you think about MS RDC btw? | 2009-08-26 09:46:32 |
<osman> | i didn't properly read it. i've another job which have to be done today. | 2009-08-26 09:47:09 |
<Shelwien> | "The RDC library's FilterMax signature generator "slides" the hash window across the entire file by adding the byte at the leading edge and subtracting the byte at the trailing edge of the window. Meanwhile, the generator continually examines the sequence of fingerprint function values over a given range of bytes, called the horizon size. If a fingerprint function value is a local maximum within the range, its byte position is chos | 2009-08-26 09:47:31 |
| After the file has been divided into chunks, the signature generator computes a strong hash value (an MD4 hash), called a signature, for each chunk. The signatures can be used to compare the contents of two arbitrarily different versions of a file. " | 2009-08-26 09:47:31 |
<chornobyl> | what is mem usage | 2009-08-26 09:49:03 |
<Shelwien> | now compare it to this | 2009-08-26 09:49:07 |
| <Shelwien> i don't want to mess with it now anyway ;) | 2009-08-26 09:49:08 |
| <Shelwien> anyway, an archiver based on CM/ppmd requires a far match model | 2009-08-26 09:49:08 |
| <Shelwien> like bulat's rep | 2009-08-26 09:49:08 |
| <Shelwien> and i've got one interesting idea about block hashes | 2009-08-26 09:49:08 |
| <Shelwien> i call it "anchored hashes" | 2009-08-26 09:49:08 |
| <Shelwien> like, normally we have to calculate a sliding hash for all strings of some fixed length | 2009-08-26 09:49:09 |
| <Shelwien> and look them up on each symbol | 2009-08-26 09:49:11 |
| <Shelwien> but only store the hashes on aligned offsets | 2009-08-26 09:49:13 |
| <Shelwien> (well, reverse is also possible - aligned lookups, but all stored, but it seems worse) | 2009-08-26 09:49:15 |
| <sami> i've tried some "anchoring" | 2009-08-26 09:49:17 |
| <sami> like byte0<byte1<byte2 is an anchor etc | 2009-08-26 09:49:19 |
| <Shelwien> yeah, something along that line | 2009-08-26 09:49:21 |
| <Shelwien> but i'm thinking to use something else | 2009-08-26 09:49:23 |
*** toffer has joined the channel | 2009-08-26 09:49:24 |
| <Shelwien> as symbol combinations are unstable | 2009-08-26 09:49:25 |
| <Shelwien> like, a minimal hash value in some area | 2009-08-26 09:49:27 |
| <sami> hmm. yeah. that's better idea | 2009-08-26 09:49:29 |
| <sami> i like it | 2009-08-26 09:49:31 |
| <Shelwien> ;) | 2009-08-26 09:49:33 |
| <sami> :-D | 2009-08-26 09:49:35 |
<toffer> | hi, what's that? | 2009-08-26 09:49:49 |
<Shelwien> | hi, what's what? | 2009-08-26 09:50:01 |
<osman> | :) | 2009-08-26 09:50:25 |
<chornobyl> | can mem usage for ccm be lower | 2009-08-26 09:50:45 |
<Shelwien> | you don't have 500M or what? | 2009-08-26 09:51:04 |
| its a pity then as ccm_sh requires recompiling to lower memory usage | 2009-08-26 09:51:22 |
<chornobyl> | i dont have 530 free which it whants to eat | 2009-08-26 09:51:26 |
<Shelwien> | forget it then ;) | 2009-08-26 09:51:37 |
| the question was about comparing this mode anyway ;) | 2009-08-26 09:51:59 |
<toffer> | and does your templated memorylevel pay off? | 2009-08-26 09:53:44 |
<Shelwien> | well, it considerably simplifies the source | 2009-08-26 09:54:08 |
<toffer> | mh not sure about that | 2009-08-26 09:54:19 |
| i think it's just too stiff | 2009-08-26 09:54:40 |
| not flexible | 2009-08-26 09:54:43 |
<Shelwien> | why? i just declared the tables | 2009-08-26 09:54:49 |
| its simpler than new (more so malloc that was there) | 2009-08-26 09:55:09 |
<toffer> | there's no big deal in having an allocation function with one call to malloc | 2009-08-26 09:55:14 |
<Shelwien> | there's some deal in having to use pointers | 2009-08-26 09:55:31 |
| instead of direct offsets in the code | 2009-08-26 09:55:45 |
| anyway, sh1d is faster here | 2009-08-26 09:56:28 |
<toffer> | well yes, but how's the speed difference? | 2009-08-26 09:56:31 |
<Shelwien> | http://shelwien.googlepages.com/results.txt | 2009-08-26 09:56:46 |
<toffer> | a lead ballon :) | 2009-08-26 09:56:54 |
<Shelwien> | what? | 2009-08-26 09:57:07 |
<toffer> | that's an idiom i learned some time ago | 2009-08-26 09:57:22 |
| unsuccessful try | 2009-08-26 09:57:34 |
<Shelwien> | what is? | 2009-08-26 09:57:53 |
<toffer> | if the old version is faster | 2009-08-26 09:59:53 |
| i don't know what you mean | 2009-08-26 10:00:09 |
<Shelwien> | new one is faster for me | 2009-08-26 10:00:13 |
| but not for skymmer: http://skymmer.narod.ru/data/deccm_speed.txt | 2009-08-26 10:00:55 |
<toffer> | 32.109s 32.734s ccm_sh1c vs 31.703s 32.766s ccm_sh1d | 2009-08-26 10:01:08 |
<Shelwien> | so that's the question | 2009-08-26 10:01:09 |
| so? encoding is faster, decoding is the same | 2009-08-26 10:01:31 |
<toffer> | that's just ~2 % | 2009-08-26 10:01:50 |
| i'd not call it significant - and you lose the flexibility | 2009-08-26 10:02:09 |
| i'd not accept that change than | 2009-08-26 10:02:20 |
<Shelwien> | well, i hoped for more, but IntelC is hard to control | 2009-08-26 10:02:24 |
| it wasn't the only change anyway | 2009-08-26 10:02:33 |
| i think that speed problem | 2009-08-26 10:02:41 |
<toffer> | even worse than i guess - but compilers are weird | 2009-08-26 10:02:46 |
<Shelwien> | is actually because of added align directive | 2009-08-26 10:02:51 |
<toffer> | you know better than me but i had weird stuff these days too | 2009-08-26 10:02:56 |
<Shelwien> | *directives | 2009-08-26 10:03:00 |
| ...anyway, i also added uncompressed data buffering in 1d1 | 2009-08-26 10:06:52 |
| and it helped a little too | 2009-08-26 10:07:03 |
<toffer> | 1d1? | 2009-08-26 10:07:42 |
<Shelwien> | yeah ;) | 2009-08-26 10:07:58 |
*** pinc has joined the channel | 2009-08-26 10:08:09 |
<toffer> | what's that | 2009-08-26 10:11:07 |
<Shelwien> | ? | 2009-08-26 10:12:03 |
<toffer> | 1d1? | 2009-08-26 10:13:12 |
<Shelwien> | http://shelwien.googlepages.com/ccm_sh1d1.rar | 2009-08-26 10:13:45 |
| only one function is modified so it doesn't matter | 2009-08-26 10:15:41 |
| ... | 2009-08-26 10:16:20 |
| anyway, i want to test interleaved rcs there, but didn't have time yet | 2009-08-26 10:16:35 |
| i make, to make 8 rcs there, and move all the renorm stuff out of the bit loop | 2009-08-26 10:17:15 |
<toffer> | as i said it helped | 2009-08-26 10:17:22 |
| and the fact that a renorm emits one byte only is useful, too | 2009-08-26 10:17:44 |
| i tried to buffer 16 bits in the rc state | 2009-08-26 10:17:54 |
| which increased speed, too | 2009-08-26 10:18:01 |
| as i said, i really like that idead :) | 2009-08-26 10:18:18 |
<Shelwien> | somehow i don't think that we're talking about the same thing ;) | 2009-08-26 10:18:49 |
<toffer> | the resulting compression loss was something like 100..200 bytes | 2009-08-26 10:18:53 |
| well as you know i triedthat yesterday | 2009-08-26 10:19:08 |
| just to have a function call | 2009-08-26 10:19:18 |
| to renorm | 2009-08-26 10:19:21 |
<Shelwien> | yeah | 2009-08-26 10:19:26 |
<toffer> | after a check for renorm possible | 2009-08-26 10:19:30 |
| and buffering 16 bits | 2009-08-26 10:19:34 |
| halves the number of calls efficiently | 2009-08-26 10:19:40 |
| so it became faster | 2009-08-26 10:19:45 |
<Shelwien> | sure | 2009-08-26 10:19:59 |
<toffer> | instead of range < (1<<24), range < (1<<16) | 2009-08-26 10:20:06 |
<Shelwien> | but i was talking about using 8 separate rcs there ;) | 2009-08-26 10:20:11 |
<toffer> | it doesn't change the idea of using function calls | 2009-08-26 10:20:31 |
<Shelwien> | yeah, its just completely unrelated ;) | 2009-08-26 10:20:48 |
<toffer> | not really | 2009-08-26 10:20:55 |
| i make, to make 8 rcs there, and move all the *renorm stuff out of the bit loop* | 2009-08-26 10:21:28 |
<Shelwien> | yes, but i talked about processing all of it out of the bit loop | 2009-08-26 10:22:14 |
| including the first checks | 2009-08-26 10:22:21 |
| so it would be possible to leave rc parts completely without branches in the loop ;) | 2009-08-26 10:23:23 |
<toffer> | basically because one rc state per bit is likely to not to cause a precision loss? | 2009-08-26 10:23:27 |
<Shelwien> | still looks like a misunderstanding... | 2009-08-26 10:24:06 |
<toffer> | i think i got that | 2009-08-26 10:24:18 |
<Shelwien> | imagine that we made 8 output files for compressed data | 2009-08-26 10:24:20 |
| and I call rc[k].BProcess( bit, prob ) instead of same rc | 2009-08-26 10:24:55 |
<toffer> | i mean you can get the checks out becasue there's one rc for each of the 8 bits | 2009-08-26 10:24:57 |
<Shelwien> | yeah, they don't overlap | 2009-08-26 10:25:08 |
<toffer> | and just one coding step can drop at most N bits of precision | 2009-08-26 10:25:11 |
| instead of dropping at most 8*N (for 1 rc) | 2009-08-26 10:25:21 |
| you just drop at most N | 2009-08-26 10:25:25 |
<Shelwien> | i don't really see how this is related to precision | 2009-08-26 10:25:29 |
<toffer> | per byte coding step | 2009-08-26 10:25:32 |
<Shelwien> | only redundancy there would be because of multiple rc flushes | 2009-08-26 10:25:48 |
<toffer> | too less precision = bad coding performance | 2009-08-26 10:25:50 |
<Shelwien> | there's no redundancy other than that | 2009-08-26 10:26:00 |
<toffer> | well what is a rc flush? | 2009-08-26 10:26:31 |
| it increased precision | 2009-08-26 10:26:35 |
| by rescaliong | 2009-08-26 10:26:38 |
<Shelwien> | its what rc does at the end of file | 2009-08-26 10:26:45 |
<toffer> | ok i meant renorm there | 2009-08-26 10:26:56 |
<Shelwien> | each rc would still have a renorm per bit | 2009-08-26 10:27:13 |
| i'm talking about using 8 different rcs at once here | 2009-08-26 10:27:29 |
| one per each bit | 2009-08-26 10:27:32 |
| and then doing renorm for all of them, in a separate loop | 2009-08-26 10:27:55 |
*** osman has left the channel | 2009-08-26 10:33:22 |
<toffer> | well i got that all the time | 2009-08-26 10:33:57 |
| but you didn't understand me - i said if you lower the precision you have even less renormalizations in total | 2009-08-26 10:34:30 |
*** osman has joined the channel | 2009-08-26 10:34:40 |
| for example if i have 1 renorm every 4..6 bytes in a sequential rc with that range < (1<<16) | 2009-08-26 10:35:12 |
| you'd have'd have the same for the parallel one | 2009-08-26 10:35:36 |
| on of these parallel coders will renorm every (4..6)*8 bytes | 2009-08-26 10:36:02 |
| and all at least one of them will renorm every 4..6 bytes | 2009-08-26 10:36:20 |
<Shelwien> | and as i said these things are orthogonal | 2009-08-26 10:36:26 |
<toffer> | it's just some confusion again | 2009-08-26 10:36:51 |
| ^^ | 2009-08-26 10:36:55 |
<Shelwien> | you can separately balance the precision and renorm frequency | 2009-08-26 10:37:07 |
| and rc interleaving is another thing completely | 2009-08-26 10:37:22 |
| its not about reducing the number of renorms or anything | 2009-08-26 10:37:42 |
| but about separating the renorm+i/o code | 2009-08-26 10:38:00 |
| from the model code | 2009-08-26 10:38:07 |
| and btw, in fpaq0pv4B I experimented with 16bit i/o and skipped renorms | 2009-08-26 10:39:41 |
| at it really helped, at the cost of some imprecision | 2009-08-26 10:39:56 |
<toffer> | for me it helped, too | 2009-08-26 10:40:50 |
| gonna try to incorporate that into the m1 coding scheme | 2009-08-26 10:41:00 |
<Shelwien> | i'm also wondering about multiplication-free thingie | 2009-08-26 10:41:31 |
| like, rnew=exp(log(range)+log(prob)) or something | 2009-08-26 10:42:01 |
| (with lookups) | 2009-08-26 10:42:10 |
<toffer> | mh | 2009-08-26 10:42:31 |
| three lookups instead of a single mult... | 2009-08-26 10:42:44 |
<Shelwien> | not really | 2009-08-26 10:42:56 |
| you can precalculate log(range) | 2009-08-26 10:43:07 |
| at the end of previous step | 2009-08-26 10:43:19 |
| ah | 2009-08-26 10:43:54 |
| or actually it would be just that log sum | 2009-08-26 10:44:02 |
<toffer> | i just noticed there can be a problem with the parallel rc | 2009-08-26 10:57:04 |
<Shelwien> | ? | 2009-08-26 10:57:14 |
<toffer> | if .e.g bit 7 will be compressed to almost nothing, and bit 1 is somehow pretty much more than you cannot easily store interleaved output | 2009-08-26 10:57:54 |
<Shelwien> | i can ;) | 2009-08-26 10:58:14 |
<toffer> | ? | 2009-08-26 10:58:23 |
<Shelwien> | i guess you didn't check parcoder ;) | 2009-08-26 10:58:35 |
<toffer> | no, i didn't | 2009-08-26 10:58:47 |
<Shelwien> | well, i have a working interleaved rc implementation since long ago ;) | 2009-08-26 10:59:08 |
<toffer> | i see | 2009-08-26 11:00:34 |
| you don't directly interleave, but add a list of next bytes | 2009-08-26 11:00:50 |
<Shelwien> | not exactly | 2009-08-26 11:01:06 |
| the trick is to allocate a place for a byte where decoder would need it | 2009-08-26 11:01:31 |
<toffer> | wouldn't just one ptr be enough? | 2009-08-26 11:02:19 |
| since norm. is syncron in encoding/decoding | 2009-08-26 11:02:30 |
| only init would need actual interleaving | 2009-08-26 11:02:38 |
<Shelwien> | still, there's a delay of 4-5 bytes | 2009-08-26 11:03:33 |
<toffer> | i guess you mean the one caused by decoder init. | 2009-08-26 11:06:42 |
<Shelwien> | yeah | 2009-08-26 11:07:02 |
*** Simon|B2 has left the channel | 2009-08-26 11:07:17 |
| but again, that's a renorm-only stuff, so pointer juggling would be only present in tenorm loop | 2009-08-26 11:07:19 |
| *renorm | 2009-08-26 11:07:23 |
| at the time, i also tried to fully synchronize enc and dec | 2009-08-26 11:08:25 |
| by computing the decode intervals | 2009-08-26 11:08:36 |
| and only actually fetching the bytes necessary to compute code<rnew | 2009-08-26 11:09:39 |
| but it quickly became too complicated | 2009-08-26 11:09:59 |
| ... | 2009-08-26 11:10:18 |
| in fact, there's even better possibility for encoding | 2009-08-26 11:10:33 |
| there we can cache the probabilities and do the rangecoding separately for the whole buffer, right? | 2009-08-26 11:11:09 |
| so with 8 rcs | 2009-08-26 11:11:27 |
| it would be possible to also vectorize the {range;low} update in all rcs | 2009-08-26 11:12:11 |
| which includes the multiplication | 2009-08-26 11:12:21 |
<toffer> | but that's only acceptable for encoding | 2009-08-26 11:15:39 |
<Shelwien> | yeah, unfortunately | 2009-08-26 11:15:48 |
<toffer> | there're several things like that | 2009-08-26 11:16:03 |
| e.g. prefetching the known memory locations on encoding | 2009-08-26 11:16:13 |
<Shelwien> | well, decoding won't become slower from that either | 2009-08-26 11:16:29 |
<toffer> | true | 2009-08-26 11:16:39 |
| but still that's somehow weird somehow :) | 2009-08-26 11:16:49 |
<Shelwien> | just that its hard to think of anything to optimize it with unknown data | 2009-08-26 11:16:49 |
<toffer> | but good for optimization btw | 2009-08-26 11:16:58 |
<Shelwien> | well, there was that runlength idea, which works for decoding | 2009-08-26 11:17:13 |
| guess there might be a sense to check it with ccm too | 2009-08-26 11:17:35 |
<toffer> | afair the speed gain wasn't that notable? | 2009-08-26 11:17:51 |
<Shelwien> | yeah, and mainly in encoding again ;) | 2009-08-26 11:18:06 |
| but still, that was some mistake ;) | 2009-08-26 11:18:28 |
| i mean, millions of rc calls skipped, and no effect? ;) | 2009-08-26 11:19:26 |
<toffer> | ^^ | 2009-08-26 11:20:06 |
<Shelwien> | i guess that rc/RLE branch was really slow... | 2009-08-26 11:33:45 |
| so its probably necessary to completely split the probability range into several intervals | 2009-08-26 11:34:25 |
| i mean, so there won't be any rc stuff in decoding - only looking up bits by probability | 2009-08-26 11:36:51 |
<toffer> | well i thinkt for a parallel rc a convention for the arrangement of the first 4 bytes per rc would be enough | 2009-08-26 11:38:30 |
| thnik | 2009-08-26 11:38:56 |
| think | 2009-08-26 11:38:58 |
| that might add redundancy for small files | 2009-08-26 11:39:14 |
<Shelwien> | unfortunately there's no other way than queuing the byte locations for decoder | 2009-08-26 11:40:27 |
| that 4-5 byte delay is just that significant | 2009-08-26 11:40:59 |
<toffer> | isn't the only problem that the decoder is exactly 4 bytes in advance (paq rc) | 2009-08-26 11:41:04 |
<Shelwien> | yeah, but its 4 _byte fetches_ in advanced, but just 4 bytes | 2009-08-26 11:41:31 |
<toffer> | yes | 2009-08-26 11:41:46 |
<Shelwien> | *advanced, not just 4 bytes | 2009-08-26 11:41:55 |
| ... | 2009-08-26 11:41:59 |
| *advance, not just 4 bytes | 2009-08-26 11:42:08 |
<toffer> | that's why i say that placing the first 4 bytes in a deterministic way per rc (that'd be the first 32 bytes for 8 rcs) | 2009-08-26 11:42:11 |
<Shelwien> | that won't help at all | 2009-08-26 11:42:31 |
<toffer> | why? | 2009-08-26 11:42:36 |
<Shelwien> | that is, of course the layout for initial decoder bytes would be known and fixed | 2009-08-26 11:42:54 |
| but it doesn't matter | 2009-08-26 11:43:02 |
<toffer> | erm the encoder and decoder preform coding operations in sync | 2009-08-26 11:43:14 |
<Shelwien> | yeah, but there're multiple rcs | 2009-08-26 11:43:25 |
| which are not in sync | 2009-08-26 11:43:31 |
<toffer> | and still all of them behave identically on encoding and decoding | 2009-08-26 11:43:42 |
<Shelwien> | they don't, at all | 2009-08-26 11:43:52 |
<toffer> | so they're in sync as longas the model is in sync | 2009-08-26 11:43:56 |
<Shelwien> | imagine processing english text like that | 2009-08-26 11:44:03 |
| with bit7 always zero | 2009-08-26 11:44:13 |
| and different rc for each bit | 2009-08-26 11:44:20 |
<toffer> | i know what you mean | 2009-08-26 11:44:36 |
| but that's not the point | 2009-08-26 11:44:41 |
<Shelwien> | the point is | 2009-08-26 11:44:50 |
<toffer> | since if the model behaves indentical in encoding and decoding | 2009-08-26 11:44:52 |
<Shelwien> | that you can't just calculate where each rc's bytes would be located | 2009-08-26 11:45:10 |
<toffer> | all of the rcs will behave identical for encoding and decoding (rc7 encode = rc7 decode, rc6 encode = rc6 decode and so on) | 2009-08-26 11:45:20 |
| that's implicitly done by processing | 2009-08-26 11:45:29 |
<Shelwien> | yeah, but delay changes it all | 2009-08-26 11:45:36 |
<toffer> | and the dealy is fixed by a deterministic inital alignment | 2009-08-26 11:45:54 |
<Shelwien> | the number of delay bytes is, but not distances between them | 2009-08-26 11:46:24 |
| and the distances depend only on the behavior of other rcs | 2009-08-26 11:46:43 |
<toffer> | well the distances are deterministic for the first 4 bytes per rc | 2009-08-26 11:46:47 |
| and for the rest they're implicitly known by coding ops?! | 2009-08-26 11:46:56 |
<Shelwien> | for the first bytes they are, but not for all others | 2009-08-26 11:47:06 |
| ... | 2009-08-26 11:47:16 |
| i'd try to clarify | 2009-08-26 11:47:28 |
| the idea is | 2009-08-26 11:47:31 |
| that storing the bytes in encoding | 2009-08-26 11:47:43 |
| can be delayed | 2009-08-26 11:47:49 |
| first byte locations queued | 2009-08-26 11:47:58 |
| and then bytes stored there | 2009-08-26 11:48:04 |
| but with that | 2009-08-26 11:48:14 |
| decoders won't need any fixing at all | 2009-08-26 11:48:24 |
| they would be completely the same as in normal rc | 2009-08-26 11:48:36 |
<toffer> | i'm still unsure about that | 2009-08-26 11:49:38 |
<Shelwien> | about what? | 2009-08-26 11:50:01 |
<toffer> | guess i'll have to get it on my own to get what i might be missing | 2009-08-26 11:50:04 |
| you need to store stream lengths? | 2009-08-26 11:50:55 |
<Shelwien> | void StartEncode( void ) { | 2009-08-26 11:51:26 |
| low=0; range=(uint)-1; | 2009-08-26 11:51:26 |
| DO(8) QueueReserve(); | 2009-08-26 11:51:26 |
| } | 2009-08-26 11:51:26 |
| void Encode (uint cumFreq, uint freq, uint totFreq) { | 2009-08-26 11:51:26 |
| low += cumFreq * (range/= totFreq); | 2009-08-26 11:51:27 |
| range*= freq; | 2009-08-26 11:51:29 |
| while( range<TOP ) { | 2009-08-26 11:51:31 |
| if ( uc((low^low+range)>>56) ) | 2009-08-26 11:51:33 |
| range = ((uint(low)|(TOP-1))-uint(low)); | 2009-08-26 11:51:35 |
| OutByte( low>>56 ), range<<=8, low<<=8; | 2009-08-26 11:51:37 |
| QueueReserve(); | 2009-08-26 11:51:39 |
| } | 2009-08-26 11:51:41 |
| } | 2009-08-26 11:51:43 |
| (its a weird 64-bit carryless, but don't mind that) | 2009-08-26 11:52:02 |
| OutByte() stores a byte into the first location in the queue | 2009-08-26 11:53:05 |
| http://compression.ru/sh/parcoder.rar | 2009-08-26 11:53:28 |
*** chornobyl has left the channel | 2009-08-26 11:59:38 |
*** Simon|B2 has joined the channel | 2009-08-26 12:10:00 |
*** FuSiyuan has joined the channel | 2009-08-26 12:31:56 |
*** FuSiyuan has left the channel | 2009-08-26 12:46:39 |
*** FuSiyuan has joined the channel | 2009-08-26 13:20:59 |
*** pinc has left the channel | 2009-08-26 14:43:15 |
*** Skymmer has joined the channel | 2009-08-26 14:52:48 |
<Simon|B> | hi Skymmer ;) | 2009-08-26 15:01:33 |
<Skymmer> | Hello :) | 2009-08-26 15:03:02 |
| How do you do? | 2009-08-26 15:04:08 |
| I'm somehow have been caught by DOS last days | 2009-08-26 15:05:48 |
| Now I'm collecting different screenshot tools for DOS | 2009-08-26 15:06:15 |
| I need to capture some program running | 2009-08-26 15:06:44 |
| Never thought that screenshoting is possible under DOS 8) | 2009-08-26 15:07:14 |
<Simon|B> | hehe funny I played DOS games via ScummVm on my handy today^^ | 2009-08-26 15:07:51 |
<Skymmer> | nice :) | 2009-08-26 15:09:16 |
| collected so far: GRABBER V3.98, PCXDUMP v9.30, SCREEN Thief v1.58, VideoThief v0.07 | 2009-08-26 15:09:25 |
<Simon|B> | It's so heavy to me what's possible on such a SMALL HANDY | 2009-08-26 15:09:27 |
| video? nice^^ | 2009-08-26 15:10:22 |
<Skymmer> | Of cource I could use the DOSBox software for my task but there is no atmosphere in such case ;) | 2009-08-26 15:12:00 |
| There is also Ruxgulo's paq8px_61 compilation which runs in DOS ! | 2009-08-26 15:13:17 |
| Fascinating :) | 2009-08-26 15:13:53 |
*** FuSiyuan has left the channel | 2009-08-26 15:14:49 |
*** pinc has joined the channel | 2009-08-26 15:16:14 |
| I'm also become interested in FireFox browser last days. | 2009-08-26 15:16:17 |
| Damn! Those guys (I mean developers) just rock | 2009-08-26 15:16:42 |
| I mean overall idea and principles | 2009-08-26 15:17:21 |
| For example if somebody complaining about too less options in it then... | 2009-08-26 15:18:46 |
| about:config | 2009-08-26 15:18:58 |
| 8) | 2009-08-26 15:19:13 |
| Also I like their humor | 2009-08-26 15:20:06 |
| For example you can test FireFox v3.7 right now | 2009-08-26 15:20:29 |
| Seriously | 2009-08-26 15:20:37 |
| http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/ | 2009-08-26 15:20:44 |
| But its called not FireFox | 2009-08-26 15:20:57 |
| MineField | 2009-08-26 15:21:03 |
<Simon|B> | hmm dunno I am a bit strange (also for myself). For some reason I dislike Firefox and this will never change :-D | 2009-08-26 15:34:43 |
| It's the same for Mercedes, WOW, Aldi..... :D | 2009-08-26 15:35:49 |
<Skymmer> | Sure | 2009-08-26 15:36:26 |
| I loved Opera in a days when traffic coasts me too much | 2009-08-26 15:37:06 |
<Simon|B> | Yeah but most times I have no real arguments ;) | 2009-08-26 15:37:10 |
<Skymmer> | So I was turning off all images and was happy :) | 2009-08-26 15:37:38 |
<Simon|B> | ah I forgot winamp. I also hate it^^ | 2009-08-26 15:37:44 |
<Skymmer> | I dropped it after I met foobar | 2009-08-26 15:38:14 |
| Foobar2000 forever for me :) | 2009-08-26 15:38:36 |
<Simon|B> | Yes it now has a proxy server which can lossy reduce images heavily before sending | 2009-08-26 15:38:43 |
| aimp for me ;) | 2009-08-26 15:38:53 |
| tested foobar but it was too freaky | 2009-08-26 15:39:11 |
*** Surfer has joined the channel | 2009-08-26 15:40:34 |
<Skymmer> | Well, design for audio player is the last I'm looking at | 2009-08-26 15:40:43 |
<Simon|B> | hi Surfer | 2009-08-26 15:40:44 |
<Surfer> | hi | 2009-08-26 15:40:52 |
<Skymmer> | after all, you listen to music. | 2009-08-26 15:41:12 |
<Simon|B> | indeed | 2009-08-26 15:41:20 |
<Skymmer> | But foobar's possibilities just fascinating | 2009-08-26 15:41:41 |
<Surfer> | songbird great :) | 2009-08-26 15:42:15 |
<Skymmer> | foobar for me its first of all functionality | 2009-08-26 15:43:46 |
| Great converter with possibility to plug any command line encoder to it f.e. | 2009-08-26 15:44:22 |
| tagger with custom scripting | 2009-08-26 15:44:41 |
<Simon|B> | yeah foobar is famous for it's features. | 2009-08-26 15:44:44 |
| It's for me the same like miranda and firefox | 2009-08-26 15:45:04 |
| it's nothing without plugins | 2009-08-26 15:45:17 |
<Skymmer> | Sure. With foobar I can also rip CDs with CD-Paranoia method and even write something to CD-Rs\CD-RWs. I'm not talking about great playlist manipulation system | 2009-08-26 15:48:51 |
<Surfer> | proper cd-rips can only be done with ExactAudioCopy | 2009-08-26 15:49:43 |
<Skymmer> | After all we all have out prefernces :) | 2009-08-26 15:49:54 |
| *our | 2009-08-26 15:50:04 |
| No | 2009-08-26 15:50:21 |
| EAC is not the only | 2009-08-26 15:50:30 |
<Surfer> | eac is standart | 2009-08-26 15:51:03 |
<Skymmer> | CD-paranoia (which is also used in CDex) gives me better results sometimes | 2009-08-26 15:51:11 |
| on scratched CDs | 2009-08-26 15:51:22 |
<Simon|B> | "only be done" -.- | 2009-08-26 15:53:59 |
| there are hundreds of good mucic rip tools | 2009-08-26 15:54:13 |
<Skymmer> | But I'm agree. EAC is just great | 2009-08-26 15:56:13 |
| Actually there are only two good ripping tools | 2009-08-26 15:56:33 |
| EAC and CDex | 2009-08-26 15:56:40 |
| They're care about such things as your device offset, error handling, if your device uses audio caching or not, gaps and etc. | 2009-08-26 15:57:51 |
| Also you can plug any command line encoder of your choice to it, use presets | 2009-08-26 15:58:46 |
| I'm also have one story with EAC | 2009-08-26 15:59:09 |
| Once I've found one undocumented command-line for ir | 2009-08-26 15:59:28 |
| *it | 2009-08-26 15:59:32 |
| And published it on forum with the question: What its for? | 2009-08-26 16:00:00 |
*** pinc has left the channel | 2009-08-26 16:00:28 |
| My post have been quickly edited and author of EAC contacted me | 2009-08-26 16:00:30 |
| He asked me NOT to ask about this option anymore | 2009-08-26 16:00:59 |
| I've pushed on him | 2009-08-26 16:01:11 |
<Surfer> | what option ? :D | 2009-08-26 16:01:30 |
<Skymmer> | And he said that this option is for paying customer | 2009-08-26 16:01:45 |
| Well, I don't know if its fair to publish it ??? | 2009-08-26 16:02:39 |
*** chornobyl has joined the channel | 2009-08-26 16:39:41 |
<osman> | hi chorbobyl | 2009-08-26 16:51:18 |
| did you test the program? | 2009-08-26 16:51:26 |
<chornobyl> | just woke up | 2009-08-26 16:51:38 |
<osman> | %) | 2009-08-26 16:51:51 |
<chornobyl> | i'll try i in small scale | 2009-08-26 16:51:57 |
<osman> | are you a bat-man :) | 2009-08-26 16:51:58 |
<chornobyl> | not me Eugene are | 2009-08-26 16:52:25 |
<osman> | just woke up for feeding by sucking blood from humans :) | 2009-08-26 16:52:29 |
| eugene has a really "time-shift" problem. he's "out of scope" for me :) | 2009-08-26 16:53:03 |
| anyway. i have to eat something. just drop a comment when you test it | 2009-08-26 16:53:33 |
<chornobyl> | i'm not going ti bite anyone | 2009-08-26 16:53:49 |
| ok | 2009-08-26 16:53:49 |
*** sami has joined the channel | 2009-08-26 17:12:16 |
*** Skymmer has left the channel | 2009-08-26 17:15:07 |
<osman> | I wanna buy a new DDR2 notebook ram set (2x2 Gib or more). single question: what's your offer? OCZ vs kingston? | 2009-08-26 17:23:14 |
<Surfer> | kingston more PR'ed brand :) | 2009-08-26 17:26:45 |
<osman> | i know but most of people claim that ocz is obviously better | 2009-08-26 17:27:32 |
| note that my current rams are already kingston :) you can see it's performance in the forum :) | 2009-08-26 17:28:08 |
| the problem is i have 2 GiB in total but it's not useful for Vista x64 | 2009-08-26 17:28:30 |
| it eats more than 1 GiB by itself | 2009-08-26 17:28:41 |
| you may say "install xp then" | 2009-08-26 17:28:55 |
| but it's not an option. because vista x64 really boosts my core duo performance. i've tested xp, xp 64, vista x86 and vista x64 | 2009-08-26 17:29:36 |
| vista x64 seems best. but it's a really memory sucker | 2009-08-26 17:29:54 |
| in fact, it can efficiently reorganize memory fragments. but, as i said it really need ~4 GiB memory | 2009-08-26 17:30:24 |
<Shelwien> | hi btw | 2009-08-26 17:41:11 |
<sami> | hi. so have you yet tried a faster rc for the ccmsh? | 2009-08-26 17:43:57 |
<Shelwien> | no, only discussed the ideas with toffer | 2009-08-26 17:44:27 |
| well, in fact i tried something in sh1d | 2009-08-26 17:44:37 |
| moved the rc renorm | 2009-08-26 17:44:46 |
| and replaced the range update with gnu inline asm | 2009-08-26 17:45:01 |
| and added some templates | 2009-08-26 17:45:08 |
| ...but performance maybe actually degraded because of that %) | 2009-08-26 17:45:34 |
| anyway, what do you think about | 2009-08-26 17:46:09 |
<sami> | hmm. your rc had an unsigned mul right? how about getting rid of that | 2009-08-26 17:46:10 |
<Shelwien> | i don't think that (range>>LOG)*freq is any better | 2009-08-26 17:47:06 |
<sami> | yes it is | 2009-08-26 17:47:18 |
<Shelwien> | well, it can be, but because of other optimizations mainly | 2009-08-26 17:47:39 |
<sami> | I mean discarding the low bits for speed | 2009-08-26 17:47:42 |
<Shelwien> | i mean, such multiplication itself isn't any faster imho | 2009-08-26 17:48:10 |
| i think it computes whole 64 bits anyway | 2009-08-26 17:48:25 |
| just discards the high dword | 2009-08-26 17:48:36 |
| <Shelwien> anyway, what do you think about | 2009-08-26 17:48:51 |
| 1. exp(log(range)+log(prob)) approximation (with lookups) | 2009-08-26 17:49:10 |
| 2. runlength coding of bits sorted by quantized probability | 2009-08-26 17:49:37 |
| 3. multiple rc (8 or more) interleaving (like in parcoder) | 2009-08-26 17:50:06 |
| 4. caching the probabilities for data block and rangecoding in another loop (for encoder) | 2009-08-26 17:51:25 |
| 3+4. splitting encoding process into 3 loops (model,low/range updates,renorms) and vectorizing low/range updates | 2009-08-26 17:52:55 |
| 5. 16-bit i/o in rc (I have a 32-bit implementation too, but its much slower) | 2009-08-26 17:54:51 |
<Surfer> | osman: try Win7 x64, it eats lower memory and have better perfomance | 2009-08-26 17:55:21 |
<osman> | @Surfer: Thanks. I'll try when I get ms partner dvd from my brother (=valid license). is it still in RC state, right? | 2009-08-26 17:56:43 |
| @Shelwien: option 1 is frequently used in FPGAs :) | 2009-08-26 17:57:43 |
<Surfer> | i think you can upgrage from vista | 2009-08-26 17:57:55 |
<Shelwien> | its used in damned wz_jpeg | 2009-08-26 17:58:01 |
| but it should be possible to make a better version - simpler and without losing that much precision | 2009-08-26 17:58:35 |
<sami> | I don't think 2. will will be any good, it will not work. but I don't have strong feelings for any of those. of course none of the optimizations will bring much speedup, but would have to see them implemented and tested to say are they interesting. I wouldn't expect 4. and 5. to give any consistent results (might be compressor/implementation specific) | 2009-08-26 18:03:09 |
<Shelwien> | 5 is implemented in fpaq0pv4B and gives results | 2009-08-26 18:03:48 |
*** osman has left the channel | 2009-08-26 18:03:51 |
| 2 is implemented too, and allows to skip 300f rc calls on enwik | 2009-08-26 18:04:32 |
| but unfortunately in my implementation it seems well compensated by a probability branch | 2009-08-26 18:05:23 |
<sami> | yes I remember you told me about 2, but you all you said back then was "run length coding on bits". I expect that to add lot's of brances and slowdown | 2009-08-26 18:05:28 |
<Shelwien> | so only encoding is somewhat faster | 2009-08-26 18:05:37 |
| no, my implementation is actually faster, just not that much as one expects given 30 ut in rc calls | 2009-08-26 18:06:24 |
| and also it even improves compression a little %) | 2009-08-26 18:06:42 |
<sami> | could you give me the parcoder link again, so I could check up what the multiple rcs was about? | 2009-08-26 18:07:17 |
<Shelwien> | http://compression.ru/sh/parcoder.rar | 2009-08-26 18:07:52 |
<sami> | thanks | 2009-08-26 18:08:00 |
<Shelwien> | you can also find its discussion with toffer ~11:51 in http://nishi.dreamhosters.com/log/2009-08-26_06-10-52.txt | 2009-08-26 18:08:51 |
| anyway, its dumb and really old, but idea remains relevant imho | 2009-08-26 18:09:33 |
| in fact, it should much more important these days because of vectors and cores | 2009-08-26 18:10:50 |
| *should be | 2009-08-26 18:11:04 |
*** osman has joined the channel | 2009-08-26 18:13:55 |
<osman> | electricity problem... :/ | 2009-08-26 18:14:05 |
| btw, @Shelwien: i mean instead of multiplication, log via lookups. FPGAs supports lookups by their natures. | 2009-08-26 18:14:14 |
<sami> | about the parcoder. you open multiple input files for compressing? is this just because this is a simple test or is there somethng I don't understand? | 2009-08-26 18:19:31 |
<Shelwien> | its a "parallel rangecoder" | 2009-08-26 18:19:53 |
| it opens multiple input files | 2009-08-26 18:19:58 |
| and encodes the bytes from each | 2009-08-26 18:20:08 |
| into a single code stream | 2009-08-26 18:20:15 |
| in a single loop | 2009-08-26 18:20:17 |
<sami> | right, but you could use a single file instead | 2009-08-26 18:20:25 |
<Shelwien> | which can be (partly) vectorized | 2009-08-26 18:20:30 |
| input? yeah, but it won't be that easy to understand imhop | 2009-08-26 18:20:52 |
| *imho | 2009-08-26 18:20:56 |
<sami> | well, ok, it would have been for me :-) | 2009-08-26 18:22:34 |
<Shelwien> | ok, i guess maybe a complete and really vectorized implementation would be more usable | 2009-08-26 18:23:34 |
| but i only wanted to demonstrate the rc delay sync | 2009-08-26 18:23:56 |
| instead of messing with many technical details of how to really vectorize it | 2009-08-26 18:24:25 |
<sami> | anyway, perhaps having multiple rcs for the current cms might be interesting/fun. even though I wouldn't be optimistic about any speedup | 2009-08-26 18:25:54 |
<Shelwien> | ok, i'd demonstrate it soon (i hope) ;) | 2009-08-26 18:27:56 |
<Simon|B> | shelwien I have this second schroeder book if you want | 2009-08-26 18:29:56 |
<Shelwien> | i found it in ed2k too ;) | 2009-08-26 18:30:36 |
<Simon|B> | ok | 2009-08-26 18:30:46 |
<Shelwien> | and paid a person today to OCR and then translate it into russian ;) | 2009-08-26 18:31:06 |
<Simon|B> | finding isn't hard but only a few people still using it ;) | 2009-08-26 18:31:12 |
| lol? Not really? :-D | 2009-08-26 18:31:33 |
<Shelwien> | really | 2009-08-26 18:31:38 |
<Simon|B> | the one you wanted or this? | 2009-08-26 18:31:53 |
<Shelwien> | the "computer speech" obviously | 2009-08-26 18:32:11 |
<Simon|B> | yes | 2009-08-26 18:32:22 |
<Shelwien> | its highly relevant for my current job (audio codec development) | 2009-08-26 18:32:26 |
<Simon|B> | don't you understand english well?? | 2009-08-26 18:32:32 |
<Shelwien> | i do, and bought the book, and read it already | 2009-08-26 18:32:44 |
| but other developers are in other cities | 2009-08-26 18:32:57 |
| and I think they don't have enough patience even to read a one specific chapter in english | 2009-08-26 18:33:28 |
| (even if they can read it at all, which i doubt) | 2009-08-26 18:33:40 |
*** Simon|B has left the channel | 2009-08-26 18:49:43 |
*** Simon|B has joined the channel | 2009-08-26 18:53:03 |
<Simon|B> | how good/bad is this http://www.hex-rays.com/decompiler.shtml ? | 2009-08-26 19:07:04 |
<Shelwien> | unfortunately i didn't see the new version | 2009-08-26 19:07:34 |
| but i posted a link with ida 5.2 including that decompiler yesterday | 2009-08-26 19:07:58 |
<Simon|B> | so you never tried it? | 2009-08-26 19:08:17 |
<Shelwien> | not the new one | 2009-08-26 19:08:25 |
| there're no pirated versions, and they ask $2k for it | 2009-08-26 19:08:41 |
| and for $2k i'd write such a thing myself | 2009-08-26 19:08:50 |
| anyway, its the best available x86 decompiler | 2009-08-26 19:09:26 |
<Simon|B> | 1.1 is so much better? | 2009-08-26 19:09:49 |
<Shelwien> | i don't know, and really doubt in fact | 2009-08-26 19:10:08 |
| it has float-point support though | 2009-08-26 19:10:19 |
| at least i know that 5.5 is not noticeably better than 5.2 | 2009-08-26 19:10:44 |
| although it has some nice external features | 2009-08-26 19:11:19 |
| like python etc scripting | 2009-08-26 19:11:28 |
| and iphone debugging | 2009-08-26 19:11:35 |
| (remote) | 2009-08-26 19:11:41 |
| and alternate debugger support | 2009-08-26 19:12:24 |
<Simon|B> | 5.5? | 2009-08-26 19:13:36 |
| only saw 5.4 | 2009-08-26 19:13:50 |
| indeed 5.5 | 2009-08-26 19:14:04 |
| hex rays page says 5.5 releases and ida pro page says latest news 5.4 .... -.- | 2009-08-26 19:14:45 |
<Shelwien> | well, i have a real 5.5 license | 2009-08-26 19:15:43 |
<Simon|B> | nice | 2009-08-26 19:15:50 |
| I also buy all good software | 2009-08-26 19:16:02 |
<Shelwien> | but can't say its worth the money | 2009-08-26 19:16:03 |
| except that i had to hack it right away | 2009-08-26 19:16:11 |
<Simon|B> | If it's anywhere near my budget | 2009-08-26 19:16:16 |
<Shelwien> | to load my old databases | 2009-08-26 19:16:17 |
<Simon|B> | for example I owned hexworkshop^^ | 2009-08-26 19:16:23 |
<Shelwien> | "owned" has a few meanings ;) | 2009-08-26 19:16:54 |
<Simon|B> | hehe | 2009-08-26 19:16:58 |
| bought it ;) | 2009-08-26 19:17:02 |
| I use this software so much and didn't find any other good one | 2009-08-26 19:17:29 |
<Shelwien> | btw, did you see the contest there? | 2009-08-26 19:17:43 |
| i can participate | 2009-08-26 19:17:48 |
<Simon|B> | yes I read it | 2009-08-26 19:17:55 |
| try it ;) | 2009-08-26 19:18:04 |
<Shelwien> | so if you have any good ideas on what to write | 2009-08-26 19:18:07 |
| i'd like to hear it ;) | 2009-08-26 19:18:10 |
<Simon|B> | you seem to know very mmuch about revers engeneering why don't you write something groundbreaking direct to c++ or similar ;) | 2009-08-26 19:19:07 |
<Shelwien> | i did write what i needed ;) | 2009-08-26 19:19:28 |
<Simon|B> | write something to earn money^^ | 2009-08-26 19:20:16 |
<Shelwien> | for example, this asm2c script http://shelwien.googlepages.com/sub_1008DCA0.txt | 2009-08-26 19:20:23 |
| was the main tool for ccm reversing | 2009-08-26 19:20:36 |
| and well, i have a few ideas of my own for that contest | 2009-08-26 19:22:54 |
| but somehow they are all not that ida-specific | 2009-08-26 19:23:06 |
<Simon|B> | hehe | 2009-08-26 19:23:26 |
<Shelwien> | more like just some standalone reversing tools | 2009-08-26 19:23:32 |
<Simon|B> | did you ever used ollydbg? | 2009-08-26 19:24:20 |
<Shelwien> | not really ;) | 2009-08-26 19:24:30 |
*** Shelwien has left the channel | 2009-08-26 19:29:23 |
*** Shelwien has joined the channel | 2009-08-26 19:29:41 |
| e,e | 2009-08-26 19:29:50 |
*** Skymmer has joined the channel | 2009-08-26 19:35:01 |
<Surfer> | ÿ âîò äóìàþ, à ïî÷åìó nosso íå äåêîìïèëèðîâàòü èëè òàì ïðîáëåìû åñòü ? :) | 2009-08-26 19:43:38 |
<Shelwien> | à çà÷åì? ;) | 2009-08-26 19:43:52 |
<Skymmer> | Òàì æå òîëüêî decompressor | 2009-08-26 19:44:02 |
<Shelwien> | íó òàì æå íè÷åãî íåòðèâèàëüíîãî íåò ;) | 2009-08-26 19:44:23 |
<Surfer> | íó íó ïî äåêîìïðåññîðó òîæå âðîäå | 2009-08-26 19:44:27 |
<Shelwien> | ïî êðàéíåé ìåðå, òàêîãî, ÷òî ïðîùå äåêîìïèëèðîâàòü, ÷åì íàïèñàòü ;) | 2009-08-26 19:44:36 |
<Skymmer> | Ëó÷øå óæ ROLZ3 èç WinRK | 2009-08-26 19:44:59 |
<Surfer> | íó ðàçüâå åñòü òàêîé êîìïðåññîð, àð÷èâåð, êîòîðûé ìîæåò àäîáå ðèäåð óæàò ? | 2009-08-26 19:45:02 |
<Shelwien> | õç, ÿ íå ïðîáîâàë | 2009-08-26 19:45:23 |
<Surfer> | winrk v3.11 normal 39.8mg | 2009-08-26 19:45:36 |
| 7zip 40.6mg | 2009-08-26 19:45:36 |
| stuffit12 52.1mg | 2009-08-26 19:45:36 |
| nosso 33.4mg | 2009-08-26 19:45:36 |
<Shelwien> | íó ýòî íàäî ñìîòðåòü, ÷òî òàì çà ôîðìàòû ôàéëîâ, è êàê èõ ëó÷øå ñæàòü | 2009-08-26 19:46:09 |
| íî èçâåñòíî, ÷òî òàì íè÷åãî êðó÷å LZMA íå ïðèìåíÿåòñÿ | 2009-08-26 19:46:20 |
<Surfer> | íó òàì â îñíîâíîì åêçåøêè è äëëêè, íî ïî èäåå BCJ2 äîëæåí áûë ñïðàâèòüñÿ ëóø÷å | 2009-08-26 19:46:41 |
<Shelwien> | à, íó ýòî íå ôàêò | 2009-08-26 19:46:57 |
| òû äóðèëêó ïðîáîâàë? | 2009-08-26 19:47:02 |
<Surfer> | íåå, ýòî âîîáùå èç òåìû http://encode.ru/forum/showthread.php?t=140 | 2009-08-26 19:47:17 |
| íî äóìàþ äàííûå ñîâïàäóò | 2009-08-26 19:47:30 |
<Shelwien> | â äóðèëêå ïîëíûé äèçàññåìáëåð, à íå bcj2 | 2009-08-26 19:48:00 |
<Skymmer> |  Nosso çàòî÷åííûå ôèëüòðû, ïåðå-ñîçäàâàíèå CAB àðõèâîâ è ïðî÷èå õèòðîñòè. Âñå ýòî âðîäå óæå îáñàñûâàëîñü :) | 2009-08-26 19:48:41 |
<Shelwien> | à êðîìå òîãî, åñëè òàì åñòü êàðòèíêè èëè çâóê (íàïðèìåð, â ðåñóðñàõ), òî èõ ëó÷øå ñîîòâåòñòâóþùèìè ñðåäñòâàìè ñæèìàòü | 2009-08-26 19:49:07 |
| íå ãîâîðÿ óæå î ðåêîìïðåññèè êàáîâ è ò.ï. | 2009-08-26 19:49:22 |
<Surfer> | íó ýòî ïîíÿòíî, íî íå äóìàþ ÷òî ïðåêîìï íàïðèìåð äàñò ñóùåñòâåííûé ïðèðîñò íàïðèìåð | 2009-08-26 19:49:24 |
| ïðèòîì çàìåäëèò ñèëüíî | 2009-08-26 19:49:40 |
<Shelwien> | íó â ïðèíöèïå ÿ æå è íå ãîâîðþ, ÷òî òàì íå÷åãî äåëàòü | 2009-08-26 19:50:07 |
<Skymmer> | Òàì â òåìå ãäå-òî Nanoflooder âñå ÷òî òîëüêî ìîæíî ïîïðîáûâàë. Ïîèùè ññûëêó íà XLS ôàéë. | 2009-08-26 19:50:30 |
<Shelwien> | ÿ ãîâîðþ, ÷òî åñòü ìàññà î÷åâèäíûõ âåùåé, êîòîðûå ïðîùå ñäåëàòü ñàìîìó, ÷åì äåêîìïèëèðîâàòü | 2009-08-26 19:50:40 |
<Surfer> | íó ÿ âèäåë ýòîò ôàéë, ùÿ åù¸ ïîñìîòðþ | 2009-08-26 19:51:49 |
| ðÿäîì ñòîèò òîëüêî íàíîçèï, íî ñêîðîñòü ðàñïàêîâêè â 2 ðàçà íèæå :) | 2009-08-26 19:53:15 |
| è íà 15 ñåêóíä äîëüøå | 2009-08-26 19:53:31 |
<Shelwien> | íó òàê òåì áîëåå... è äóðèëêó èìõî íèêòî íå ïðîâåðÿë... | 2009-08-26 19:53:38 |
<Skymmer> | Òàì â ýòîé äóðèëêå ñâèõíóòüñÿ ìîæíî îò íàñòðîéêè òðèêîâ | 2009-08-26 19:55:39 |
| À åñëè âñå ïåðåáèðàòü òî âîîáùå âåñëà ñóøè | 2009-08-26 19:56:03 |
<Simon|B> | lol | 2009-08-26 19:56:31 |
| looks funny for my eyes | 2009-08-26 19:56:38 |
<Shelwien> | äà îíî ïðîñòî íà ñàìîì äåëå, òîëüêî ñòàðàÿ äóðèëêà íóæíà ;) | 2009-08-26 19:56:44 |
<Skymmer> | :) Íî òàê äà. Íàäî ïîïðîáûâàòü | 2009-08-26 19:56:54 |
<Shelwien> | çàïóñêàåøü ñ -l, ïîäáèðàåøü order è òèï îòäåëüíî äëÿ ñåãìåíòîâ | 2009-08-26 19:57:00 |
| à ïîòîì ñîáèðàåøü êîìàíäíóþ ñòðîêó | 2009-08-26 19:57:10 |
| â íîâîé âåðñèè îí ïðîñòî -l îòëîìàë | 2009-08-26 19:57:25 |
<Skymmer> | À êàêàÿ èìåííî? 0.5 åñëè íå îøèáàþñü ? | 2009-08-26 19:57:45 |
<Shelwien> | http://shelwien.googlepages.com/durilca2_002a.rar | 2009-08-26 19:58:15 |
<Surfer> | Simon|B: http://www.microsofttranslator.com/ :D | 2009-08-26 19:58:43 |
<Simon|B> | it detected italian nice start... :D | 2009-08-26 20:05:28 |
<Shelwien> | %) | 2009-08-26 20:05:34 |
<Simon|B> | it seems to translate absolutely nothing | 2009-08-26 20:06:27 |
<Surfer> | or google translate :) | 2009-08-26 20:07:19 |
<Shelwien> | http://translate.google.com/translate?hl=en&sl=ru&tl=en&u=http%3A0.0000000.000000nishi.dreamhosters.com0.000000log0.0000002009-08-26_06-10-52.txt | 2009-08-26 20:07:30 |
| its funny | 2009-08-26 20:07:34 |
<Simon|B> | there were anything wrong with what I have here | 2009-08-26 20:08:01 |
| google also showed no valid result | 2009-08-26 20:08:11 |
<Shelwien> | why, it translated something ;) | 2009-08-26 20:08:38 |
<Skymmer> | :D | 2009-08-26 20:09:12 |
<Simon|B> | this one is fine. I meant mine :p | 2009-08-26 20:09:19 |
<Shelwien> | its not fine actually | 2009-08-26 20:10:28 |
| it didn't translate half of the words ;) | 2009-08-26 20:10:34 |
<chornobyl> | just watched some really GOOD movie | 2009-08-26 20:15:50 |
<Skymmer> | Double Air Bags 28 ? | 2009-08-26 20:16:34 |
<chornobyl> | 2007 Du levande | 2009-08-26 20:17:17 |
| http://www.imdb.com/title/tt0445336/ | 2009-08-26 20:17:44 |
*** osman has left the channel | 2009-08-26 20:20:38 |
<Surfer> | ãóä íàéò âñåì :) | 2009-08-26 20:32:35 |
*** Surfer has left the channel | 2009-08-26 20:32:39 |
*** osman has joined the channel | 2009-08-26 20:35:30 |
<osman> | chornobyl did you try the program? | 2009-08-26 20:42:06 |
<Skymmer> | What program if its not a secret ? | 2009-08-26 20:48:14 |
<chornobyl> | how to get errorcode used | 2009-08-26 20:48:47 |
<osman> | hehe :) secret!? :) not really | 2009-08-26 20:48:48 |
<Skymmer> | Ahh, understood | 2009-08-26 20:49:22 |
<osman> | yesterday chornobyl demand a program which compares NTFS compressed size vs [original size]*expectedRatio | 2009-08-26 20:49:26 |
<chornobyl> | "demand" | 2009-08-26 20:49:50 |
<osman> | it returns -1, 0 and +1 for each cases | 2009-08-26 20:50:03 |
<Skymmer> | And purpose? | 2009-08-26 20:50:22 |
<chornobyl> | so there should be some ot i guess | 2009-08-26 20:50:27 |
| all written there | 2009-08-26 20:50:36 |
| http://nishi.dreamhosters.com/log/2009-08-25_04-27-49.txt | 2009-08-26 20:50:39 |
<osman> | @chornobyl: it may seems rude. seems it's better to use "wishes" :) | 2009-08-26 20:50:52 |
<Skymmer> | I thought Bit v0.08 ;) | 2009-08-26 20:52:01 |
<osman> | @chornobyl: look at here for error levels: http://www.robvanderwoude.com/errorlevel.php | 2009-08-26 20:52:04 |
<chornobyl> | Äàâàéòå æèòü, âî âñ¸ì äðóã äðóãó ïîòàêàÿ, | 2009-08-26 20:52:16 |
| òåì áîëåå, ÷òî æèçíü êîðîòêàÿ òàêàÿ... | 2009-08-26 20:52:16 |
| i dont know how to explain it | 2009-08-26 20:52:40 |
<osman> | @skymmer: there won't be any update until middle of September :/ | 2009-08-26 20:52:40 |
<Skymmer> | Äàâàéòå ìû íå áóäåì ïîòàêàòü | 2009-08-26 20:53:26 |
| À ïðîñòî âåñåëèòñÿ è ïîðõàòü | 2009-08-26 20:53:29 |
<chornobyl> | is this yours? | 2009-08-26 20:53:46 |
<Skymmer> | Yes | 2009-08-26 20:53:54 |
| Just assembled | 2009-08-26 20:54:01 |
<chornobyl> | cause i quoted Omar Hayam | 2009-08-26 20:54:13 |
<osman> | @Skymmer: but still collecting ideas...from constructing my own version of state table (for CM part), faster and better LWCX, and better LZ compressor (with fast & slow variant) etc etc | 2009-08-26 20:54:17 |
| do you know Omar Hayam ? :) | 2009-08-26 20:54:35 |
<Skymmer> | I heard this name but never read it | 2009-08-26 20:54:57 |
<chornobyl> | just two shorts | 2009-08-26 20:55:12 |
| this one where written in my literature class at school | 2009-08-26 20:55:35 |
<osman> | he's the inventor of binomial series :) | 2009-08-26 20:55:44 |
<chornobyl> | what arm how? | 2009-08-26 20:56:02 |
<osman> | ? | 2009-08-26 20:56:17 |
<chornobyl> | are you serious | 2009-08-26 20:56:36 |
<osman> | yep | 2009-08-26 20:56:40 |
<chornobyl> | i ma y mistake but i think we talk about different people | 2009-08-26 20:57:31 |
<osman> | http://en.wikipedia.org/wiki/Ömer_Hayyam | 2009-08-26 20:57:41 |
<chornobyl> | guess not | 2009-08-26 20:58:06 |
<Skymmer> | I write some poetry for myself. Quite rarely and basicly some pessimistic stuff | 2009-08-26 20:58:31 |
<chornobyl> | good for you (ìîëîäåö) | 2009-08-26 20:59:08 |
<Skymmer> | Well, I can give you a couple if you want | 2009-08-26 21:00:04 |
<osman> | in turkish his name is written like that: "Ömer Hayyam" and he is a persian mathematician, poet, astronomer, philosopher. And "Omar Hayam" name seems very familiar with it | 2009-08-26 21:00:11 |
<chornobyl> | øåû ðøü ø ñðóñëóâ öøëø | 2009-08-26 21:00:30 |
| its him i checked wiki | 2009-08-26 21:00:40 |
<osman> | seems you wrote last sentence with russian alphabet :) | 2009-08-26 21:01:17 |
<chornobyl> | btw irc support international symbols badly | 2009-08-26 21:01:27 |
| yes | 2009-08-26 21:01:28 |
| i told already | 2009-08-26 21:01:38 |
| i mess with 3 keybord layouts | 2009-08-26 21:01:50 |
<toffer> | what about panufti lwowitsch chebychow - that russian mathematican saved me today :) | 2009-08-26 21:01:58 |
<chornobyl> | english/russian/ukrainian | 2009-08-26 21:02:14 |
<toffer> | and hi btw | 2009-08-26 21:02:15 |
<Skymmer> | hi! | 2009-08-26 21:02:21 |
<chornobyl> | hi toffer | 2009-08-26 21:02:25 |
<osman> | hi | 2009-08-26 21:02:35 |
<toffer> | just stopped writing on my thesis for today | 2009-08-26 21:02:38 |
<chornobyl> | how shebyshev saved you? | 2009-08-26 21:02:40 |
<Skymmer> | Probably Chebyshev | 2009-08-26 21:02:57 |
<toffer> | well it's transcripted in may ways | 2009-08-26 21:03:06 |
<chornobyl> | i mistype, my bad | 2009-08-26 21:03:20 |
<osman> | btw, toffer i just looked up your FSM state construction but i was a bit shocked when i see you don't use entropic metric to find "nearest" state as next state. am i right? | 2009-08-26 21:03:39 |
<toffer> | i remembered some law about the pdf mass location of random variables w/o making assumptations about a specifiy distribution | 2009-08-26 21:03:42 |
| it`s the chebychow inequality | 2009-08-26 21:03:48 |
| pertty useful | 2009-08-26 21:03:53 |
| you are right | 2009-08-26 21:04:16 |
| how do you define entropy metric? | 2009-08-26 21:04:27 |
| and states have more in common with run lengths than probabilites | 2009-08-26 21:04:56 |
<Skymmer> | That Chebyshev (if we talk about same person) is the author of one filtering method used in Scientific Filters in Adobe Audition sound editor I use | 2009-08-26 21:05:10 |
<osman> | you used TK estimator to get their static probabilities. then why don't you use code lengths to detect "similar" states? | 2009-08-26 21:05:18 |
<toffer> | erm no | 2009-08-26 21:05:21 |
| chebychew polynomals can be used in filter construction | 2009-08-26 21:05:35 |
| define code length | 2009-08-26 21:06:03 |
<osman> | %) | 2009-08-26 21:06:15 |
<toffer> | it's at as simple as it seems | 2009-08-26 21:06:24 |
<osman> | seems "you are woke up on your wrong side" ( a turkish idiom) | 2009-08-26 21:06:58 |
<toffer> | not as simple | 2009-08-26 21:06:59 |
<osman> | *you woke | 2009-08-26 21:07:18 |
<toffer> | these mistypings really tend to overwhelm me | 2009-08-26 21:08:08 |
<chornobyl> | here i sounds "from the wrong leg' | 2009-08-26 21:08:18 |
<toffer> | yeah in german, too | 2009-08-26 21:08:40 |
| so - where's the beef? | 2009-08-26 21:08:48 |
<Skymmer> | @osman: In Russia its: "Stand up from the another leg" :) | 2009-08-26 21:08:56 |
<osman> | if you have a probability for a symbol then we know it's "optimal" code length with shannon formula. I bet that you already know. else you wouldn't write any CM %) | 2009-08-26 21:08:57 |
<toffer> | ok | 2009-08-26 21:10:35 |
| so tell me what do the states with p=0.8 and p=0.2 in common? | 2009-08-26 21:10:47 |
| they share the same entropy | 2009-08-26 21:10:51 |
| and the metric i use | 2009-08-26 21:11:00 |
| is not probabiltiy actually | 2009-08-26 21:11:05 |
| but run length | 2009-08-26 21:11:07 |
<osman> | it depends on "seen" n1 and n0. that's another point of view of run length | 2009-08-26 21:11:43 |
<toffer> | what i do for an update is | 2009-08-26 21:11:56 |
| decrease max(n1,n0) | 2009-08-26 21:12:01 |
| increase n(1-y) | 2009-08-26 21:12:08 |
| and find the corresponding best match w.r.t max(n1,n0) | 2009-08-26 21:12:21 |
| and if it doesn't fit in n(1-y) other run length contexts are probed for | 2009-08-26 21:12:45 |
| i was rather surprised when i found out that matt did it the same way | 2009-08-26 21:12:57 |
| but it's not about finding the state with the closest prob | 2009-08-26 21:13:10 |
| i tried to simulate a linear counter, too | 2009-08-26 21:13:19 |
| it was much worse | 2009-08-26 21:13:22 |
| that's cause of ordinary bit histories seem to be non-combinatoric | 2009-08-26 21:13:57 |
<osman> | ok | 2009-08-26 21:13:59 |
<toffer> | as i said it's not as simple as it seems | 2009-08-26 21:14:19 |
| and i found out that three run length contexts (the small counter = 0, 1, 2) is actually enough | 2009-08-26 21:14:53 |
| are | 2009-08-26 21:14:56 |
| do you have any other ideas? | 2009-08-26 21:15:17 |
| i mean i tried alot | 2009-08-26 21:15:24 |
| and that one turned out to be best | 2009-08-26 21:15:31 |
| (better than paq) | 2009-08-26 21:15:35 |
<osman> | probably you are right at that limit. because most of context history looks like that: 000000000000001000000000000100000000000 | 2009-08-26 21:15:44 |
<toffer> | actually in my m1 0.4 candidate | 2009-08-26 21:16:25 |
<osman> | seems including last 2 bits can be good too | 2009-08-26 21:16:39 |
<toffer> | there's support for that paq-ish states which remember the last bit seen | 2009-08-26 21:16:44 |
| but that optimizer turns that off | 2009-08-26 21:16:55 |
| but the optimal state allocation seems to be similar to paq | 2009-08-26 21:18:13 |
| these "limit" pairs | 2009-08-26 21:18:30 |
| so do you have any other ideas there to try? | 2009-08-26 21:18:54 |
<osman> | not yet. just found that including 2 bits can be good but there is not any proof about it | 2009-08-26 21:19:32 |
| so, there will be 64 states left | 2009-08-26 21:19:47 |
| btw, how do you quantize those static probabilities? seems you use a lookup table. | 2009-08-26 21:21:03 |
| but, there is a question about it | 2009-08-26 21:21:12 |
| suppose we have a probability, P | 2009-08-26 21:21:24 |
| and we want to quantize it into Q | 2009-08-26 21:21:36 |
<toffer> | yeaj | 2009-08-26 21:22:00 |
| yeah | 2009-08-26 21:22:02 |
<osman> | you optimize a bit string which hold N bits | 2009-08-26 21:22:02 |
| which is much smaller than P range | 2009-08-26 21:22:12 |
<toffer> | i don't use a table for quantisation btw | 2009-08-26 21:22:13 |
<osman> | so? | 2009-08-26 21:22:22 |
<toffer> | guess it still used that pqmap | 2009-08-26 21:22:49 |
| ok | 2009-08-26 21:22:57 |
| so you're right | 2009-08-26 21:23:01 |
<osman> | then it uses lookup :) | 2009-08-26 21:23:07 |
| anyway | 2009-08-26 21:23:11 |
<toffer> | it's just that the last release was somewhere in april | 2009-08-26 21:23:12 |
<osman> | i mean in short | 2009-08-26 21:23:19 |
<toffer> | the quantisation in there works like that | 2009-08-26 21:23:28 |
<osman> | Q = tableq[P] | 2009-08-26 21:23:30 |
| but P range is much greater than N | 2009-08-26 21:23:38 |
<toffer> | yeah | 2009-08-26 21:23:43 |
| let me explain | 2009-08-26 21:23:46 |
<osman> | so, you probably use Q=tableq[P>>S]; | 2009-08-26 21:23:55 |
<toffer> | that "lookup" table | 2009-08-26 21:24:07 |
<osman> | then how do you determine initial S value? | 2009-08-26 21:24:07 |
<toffer> | i don't use any | 2009-08-26 21:24:18 |
| that lookup table | 2009-08-26 21:24:21 |
| contains a monotnous increasing function | 2009-08-26 21:24:38 |
| initially | 2009-08-26 21:24:42 |
| with a high slope where alot resolution is required | 2009-08-26 21:24:56 |
| and a low slope where less precision is used | 2009-08-26 21:25:06 |
| it is encoded as a picewise linear function | 2009-08-26 21:25:21 |
| the bits in that bitmask encode the positions of vertices | 2009-08-26 21:25:41 |
<osman> | so kind of sum of linear functions in different ranges? | 2009-08-26 21:25:45 |
<toffer> | not sum | 2009-08-26 21:25:51 |
<osman> | i meant for different ranges there are different linear functions right? | 2009-08-26 21:26:25 |
<toffer> | not exactly | 2009-08-26 21:26:36 |
| that function maps from P -> [0, 1] | 2009-08-26 21:27:01 |
| and each range in between two bits | 2009-08-26 21:27:25 |
| maps to a constant portion of output space | 2009-08-26 21:27:35 |
| thus if two adjacent bits are far away | 2009-08-26 21:27:51 |
| the slope will be low | 2009-08-26 21:27:57 |
| and than there's another parameter, which is used to decide in how man values the mapped domain [0 1] is to be divided | 2009-08-26 21:28:39 |
| but some time ago i just had the idea to use tha actual fsm state allocation for quantisation | 2009-08-26 21:30:19 |
| since it represents the distribution of input | 2009-08-26 21:30:32 |
| worked even better and requires less parameters | 2009-08-26 21:30:40 |
| -.- | 2009-08-26 21:36:33 |
<osman> | p = pqMap(p>>counter1::P_LOG-MIX_P_DOMAIN_LOG); | 2009-08-26 21:42:27 |
| i think Q=table[P>>shift]; is same as above | 2009-08-26 21:42:48 |
<toffer> | it's just to avoid alot of bits for that pdomain | 2009-08-26 21:43:09 |
| anyway i've got a better solution now | 2009-08-26 21:43:18 |
*** Skymmer has left the channel | 2009-08-26 21:43:20 |
| that's obsolete | 2009-08-26 21:43:21 |
<osman> | initially i had Q=table[P>>shift] idea. and i've wanted to see your idea to determine which one is better. And I saw they are same :) | 2009-08-26 21:45:36 |
| and what's new? | 2009-08-26 21:45:47 |
<toffer> | the problem can be roduced to the computation of a quantisation function, right? | 2009-08-26 21:49:01 |
| reduced | 2009-08-26 21:49:05 |
| and you can map states to probabilities, thus you have a denstribution density(p) | 2009-08-26 21:49:28 |
| so you can compute a quantisation function which has a high resolution where the density is high, etc. | 2009-08-26 21:50:26 |
<osman> | in the other word, you don't use lookup table anymore, right? | 2009-08-26 21:52:41 |
| but what about quantisation function complexity? | 2009-08-26 21:52:59 |
| could you give it's formula? | 2009-08-26 21:53:11 |
<toffer> | it's no formula there | 2009-08-26 21:55:04 |
| it's just learned by the optimizer | 2009-08-26 21:55:11 |
<osman> | at least it must have a form right? :) | 2009-08-26 21:55:41 |
<toffer> | but in the end it looks pretty much like stretch | 2009-08-26 21:55:46 |
| with higher slopes near 0/1 | 2009-08-26 21:55:58 |
<osman> | @toffer: i didn't get the point. could you explain in different way? | 2009-08-26 22:01:44 |
*** chornobyl has left the channel | 2009-08-26 22:02:03 |
<toffer> | i just made a plot of the function | 2009-08-26 22:03:46 |
| which is implicitly generated | 2009-08-26 22:03:54 |
| https://ftp.tu-ilmenau.de/index.php?id=c96fd2e41a90b6abc8cbfa7bf3362157&login=ee715ee44e81ef00dec0d3691150e931&file=9ec13a913acdc29d352b96330e1e7863 | 2009-08-26 22:05:01 |
| as i said it looks like stretch | 2009-08-26 22:05:09 |
| did you get it? | 2009-08-26 22:06:42 |
<osman> | i understood function's output. but, the question is how did you "generate" that function from optimization? | 2009-08-26 22:08:12 |
<toffer> | simply by enumeration | 2009-08-26 22:08:38 |
| sort the states by max(p,1-p) | 2009-08-26 22:09:10 |
| to cluster predictions near 0/1 together | 2009-08-26 22:09:21 |
| and assign a quantisation level to each adjacent elements | 2009-08-26 22:09:54 |
| each N adjacent | 2009-08-26 22:09:58 |
<osman> | seems you're only talking about state probability quantization. what about another kind of inputs? | 2009-08-26 22:11:34 |
<toffer> | that's the implementation of integrating and dividing the density weighted input into elemtns of equal mass | 2009-08-26 22:11:38 |
| i don't need any other quantisation | 2009-08-26 22:11:53 |
| and there's still the quantisation mapping in m1 0.3b | 2009-08-26 22:12:06 |
<osman> | ok. then | 2009-08-26 22:12:14 |
<toffer> | for a general solution | 2009-08-26 22:12:18 |
<osman> | here is the same question again. in m1 0.3b how do you determine "shift" value? (q=map[p>>shift]) | 2009-08-26 22:12:55 |
| or in the another word qmap length? | 2009-08-26 22:13:07 |
<toffer> | it's fixed | 2009-08-26 22:13:18 |
| at 8 bits afair | 2009-08-26 22:13:23 |
| i mean p>>N = 8 bits | 2009-08-26 22:13:32 |
| or maybe 9 | 2009-08-26 22:13:34 |
<osman> | i know it's fixed. so, that's why i'm asking that question :) | 2009-08-26 22:13:45 |
| i mean why not "say 10 bits" ? | 2009-08-26 22:13:57 |
<toffer> | i use the highest value which doesn't cause an enormous amout of parameters | 2009-08-26 22:14:11 |
<osman> | so, in the another word it's ad-hoc for simplifying optimization, right? | 2009-08-26 22:14:40 |
*** sami has left the channel | 2009-08-26 22:14:50 |
<toffer> | i'd not write an optimizer which allows variable sized bitstrings | 2009-08-26 22:15:28 |
| yes | 2009-08-26 22:15:28 |
<osman> | yes. it'll be less stable | 2009-08-26 22:15:45 |
| it's funny that i was thinking exactly same things for quantization and i thought you had better solution for all the time %) | 2009-08-26 22:16:39 |
<toffer> | as i said i got a better solution | 2009-08-26 22:17:46 |
<osman> | but i'm asking for "general" solution | 2009-08-26 22:18:03 |
<toffer> | the most obvious thing is to write a permutation optimizer | 2009-08-26 22:18:23 |
| since discrete quantisation is just a mapping from N to M<N | 2009-08-26 22:18:45 |
<osman> | yeah. it's really hard for me. ( at least for now) | 2009-08-26 22:18:47 |
<toffer> | you may want to have a look at GAs for the travelling samlesmen problem | 2009-08-26 22:19:12 |
| there was something similar afair | 2009-08-26 22:19:28 |
<osman> | i roughly know them already | 2009-08-26 22:19:32 |
| btw, thanks for the briefing :) | 2009-08-26 22:22:10 |
<toffer> | briefing? | 2009-08-26 22:22:50 |
<osman> | just a joke. i meant detailed explanation of the quantization. | 2009-08-26 22:23:26 |
| thanks | 2009-08-26 22:23:29 |
<toffer> | i just know that word from some old games | 2009-08-26 22:23:39 |
| ^^ | 2009-08-26 22:23:41 |
<osman> | "roger sir", "i'm in position", "i want to secure this area" :) from unreal tournament series :) | 2009-08-26 22:24:20 |
| btw, i really love unreal tournament 3, it has somehow most "realistic" environment IMO %) | 2009-08-26 22:25:38 |
<toffer> | i just played that cry-stuff | 2009-08-26 22:26:07 |
| well for something like 10 minutes to test my new hardware | 2009-08-26 22:26:19 |
<osman> | 3 turkish are founder of that cry-tek company :) | 2009-08-26 22:26:51 |
<toffer> | ^^ | 2009-08-26 22:27:41 |
<osman> | btw, what was your "20 bits counter"? they are rewritten form of 16 bits linear counters? what are their performance comparing to 16 bits one? | 2009-08-26 22:28:41 |
<toffer> | i get something like +0.02 0etter compression | 2009-08-26 22:30:13 |
| and speed is the same | 2009-08-26 22:30:21 |
<osman> | what about speed? | 2009-08-26 22:30:25 |
| ah ok | 2009-08-26 22:30:35 |
| they are same as 16 bits linear counters? | 2009-08-26 22:30:47 |
<toffer> | instead of p:w = 16:16 i use 20:12 | 2009-08-26 22:30:48 |
| and instead of arithmetic coding with 12 bits of precision i use 20 | 2009-08-26 22:30:59 |
| but it was - like bob ross would say - just a little happy accident | 2009-08-26 22:32:07 |
<osman> | what about pm? it should be 12 IMO? | 2009-08-26 22:33:13 |
<toffer> | 20 | 2009-08-26 22:33:55 |
| but log-coded | 2009-08-26 22:33:59 |
| pm = 1<<z | 2009-08-26 22:34:06 |
| i made some sensitivity analysis | 2009-08-26 22:34:25 |
| to reduce the parameter spae | 2009-08-26 22:34:31 |
| space | 2009-08-26 22:34:33 |
<osman> | so, p' = w*p + (1-w)*pm[y] -> [20 bits] = [12 bits]*[20 bits] + [32-12=20 bits] * [20 bits] ? | 2009-08-26 22:34:43 |
<toffer> | [32-12=20 bits] that's w | 2009-08-26 22:35:10 |
| and it's 12 bit | 2009-08-26 22:35:15 |
<osman> | (1-w) | 2009-08-26 22:35:16 |
<toffer> | yes | 2009-08-26 22:35:29 |
| but why shall it be 20 bits | 2009-08-26 22:35:39 |
| ^^ | 2009-08-26 22:35:54 |
<osman> | ah ok. it was my bad | 2009-08-26 22:35:58 |
<toffer> | maybe i can extend p to be 32 bits | 2009-08-26 22:36:07 |
| w/o 64 bit arithmetic | 2009-08-26 22:36:15 |
| using the fact that every 32 bit intel cpu does return eax*somereg in eax:edx (64 bit) | 2009-08-26 22:36:42 |
| yesterday eugene provided a little piece of asm code | 2009-08-26 22:36:54 |
| gcc inline asm | 2009-08-26 22:37:01 |
| well i thought that gcc would be clever enough to convert something line r=uint64(a)*b>>32 into r=edx | 2009-08-26 22:37:36 |
| but it didn't | 2009-08-26 22:37:42 |
<osman> | as i expected ;) | 2009-08-26 22:38:08 |
| anyway. it's still 64-bit multiply in 20 bits form. because only 16 bits values will be placed on eax | 2009-08-26 22:39:29 |
| else it'll also affect edx | 2009-08-26 22:39:40 |
| in the another word edx:eax pair | 2009-08-26 22:39:50 |
<toffer> | ? | 2009-08-26 22:40:25 |
| well ax*something results in eax | 2009-08-26 22:40:40 |
<osman> | i mean if inputs are 16 bits. then "mul" produces the result in eax only | 2009-08-26 22:40:53 |
<toffer> | yeah | 2009-08-26 22:41:02 |
<osman> | else results will be placed in edx:eax pair | 2009-08-26 22:41:07 |
<toffer> | but all multiplications used in here are 32 bit | 2009-08-26 22:41:10 |
<osman> | so, you counter pair placed in edx:eax | 2009-08-26 22:41:24 |
| *your counter | 2009-08-26 22:41:39 |
<toffer> | not really | 2009-08-26 22:41:40 |
| u just use edx | 2009-08-26 22:41:44 |
| well the counter uses just eax | 2009-08-26 22:42:02 |
| since it's a 32 bit result | 2009-08-26 22:42:08 |
| but the rangecoder uses edx | 2009-08-26 22:42:13 |
| the high 32 bit | 2009-08-26 22:42:19 |
<osman> | more or less, edx:eax pair is calculated ;) | 2009-08-26 22:42:26 |
<toffer> | to overcome | 2009-08-26 22:42:26 |
| always, yes | 2009-08-26 22:42:33 |
<osman> | that's the point what i meant ;) | 2009-08-26 22:42:42 |
| multiplication is a really hard work for low level system design. and AMD performance on multiplication is not really shock me :) | 2009-08-26 22:43:54 |
| as you know AMD's has a really bad performance on multiplications and divisons | 2009-08-26 22:44:20 |
<toffer> | well ma recent x2 245 is as fast as my q6600 | 2009-08-26 22:44:41 |
| my | 2009-08-26 22:44:45 |
<osman> | that's why mostly engineers use log tables instead of multiplication in FPGAs | 2009-08-26 22:44:51 |
<toffer> | but it hast just two cores | 2009-08-26 22:44:53 |
<osman> | just run BIT 0.2b and compare them again ;) | 2009-08-26 22:45:41 |
<toffer> | you use divisons? | 2009-08-26 22:46:11 |
<osman> | yep | 2009-08-26 22:46:15 |
| :) | 2009-08-26 22:46:17 |
<toffer> | erm really? | 2009-08-26 22:46:20 |
| where? | 2009-08-26 22:46:21 |
<osman> | BIT 0.2b. it's enough old. it was first attempt to write a CM :) | 2009-08-26 22:46:38 |
| in counters | 2009-08-26 22:46:47 |
<toffer> | ah, ok | 2009-08-26 22:46:56 |
| ah more recent version had a number like 0.7 or something | 2009-08-26 22:48:25 |
| i just wondered | 2009-08-26 22:48:28 |
<osman> | yep. latest one is 0.7 | 2009-08-26 22:48:41 |
| anyway. i'll go now. cya. good night | 2009-08-26 22:50:03 |
<toffer> | ok | 2009-08-26 22:50:51 |
| gn8 | 2009-08-26 22:50:52 |
*** Skymmer has joined the channel | 2009-08-26 23:07:02 |
| does anybody here have a 64 bit linux box? | 2009-08-26 23:08:08 |
*** Simon|B has left the channel | 2009-08-26 23:08:38 |
*** toffer has left the channel | 2009-08-26 23:19:49 |
*** Skymmer has left the channel | 2009-08-26 23:25:42 |
*** FuSiyuan has joined the channel | 2009-08-26 23:39:44 |
*** Skymmer has joined the channel | 2009-08-27 00:12:48 |
* Shelwien in fact has something alike to a 64-bit linux box. its a macbook which kinda a 64-bit freebsd box ;) | 2009-08-27 01:03:11 |
<Shelwien> | !next | 2009-08-27 01:03:16 |