*** complogger has joined the channel | 2009-12-09 15:57:02 |
*** mike_____ has joined the channel | 2009-12-09 17:44:49 |
*** mike_____ has left the channel | 2009-12-09 18:31:20 |
*** mike_____ has joined the channel | 2009-12-09 18:42:10 |
*** scott___ has joined the channel | 2009-12-09 19:18:58 |
<toffer> | hi! noticed that tests/tuning results -> thanks! cannot write much now, i'm preparing food. something like that:http://en.wikipedia.org/wiki/Rouladen | 2009-12-09 19:23:59 |
<Shelwien> | %) | 2009-12-09 19:24:55 |
*** scott___ has left the channel | 2009-12-09 19:40:18 |
*** scott___ has joined the channel | 2009-12-09 19:48:54 |
<scott___> | testing isthis thing wokring | 2009-12-09 19:49:25 |
<Shelwien> | maybe | 2009-12-09 19:50:02 |
*** scott___ has left the channel | 2009-12-09 20:01:36 |
*** scott__ has joined the channel | 2009-12-09 20:28:11 |
<scott__> | is it working | 2009-12-09 20:28:45 |
| hello | 2009-12-09 20:29:14 |
<toffer> | @shelwien: how do you calculate your "metric" ? i can modify my optimization target to optimize for "efficiency". | 2009-12-09 20:29:27 |
| hi | 2009-12-09 20:29:29 |
<scott__> | do your guys have trouble staying online more than a few minutes | 2009-12-09 20:30:13 |
| I tried several times to get on and then when I do only one or to lines before its dead | 2009-12-09 20:30:47 |
*** scott__ has left the channel | 2009-12-09 20:32:07 |
*** mike_____ has left the channel | 2009-12-09 20:37:56 |
<Shelwien> | my metric is the time of data distribution to 10 people | 2009-12-09 20:40:58 |
| 1 * compression time | 2009-12-09 20:41:11 |
| + compressed size / upload speed // 512kbit in my tests | 2009-12-09 20:41:32 |
| + 10 * compressed size / download speed // 512kbit too | 2009-12-09 20:42:01 |
| + 10 * decompression time | 2009-12-09 20:42:07 |
| Sami has the same metric implemented here: http://compressionratings.com/calc.cgi?file=app1.full.html&id=10&cc=1&cd=10&cu=1&cw=10&bu=4096&bd=512&bt=1 | 2009-12-09 20:43:17 |
| but because of time measurements in there | 2009-12-09 20:43:49 |
| it would be troublesome to tune the coder to it | 2009-12-09 20:44:05 |
*** mike_____ has joined the channel | 2009-12-09 20:49:27 |
<STalKer-Y> | meh, compression is so slow :x | 2009-12-09 21:19:51 |
<Shelwien> | ? | 2009-12-09 21:30:27 |
<STalKer-Y> | 7zip on an athlon xp 2500+ :) | 2009-12-09 21:34:41 |
<Shelwien> | it has different modes too | 2009-12-09 21:34:56 |
<STalKer-Y> | which don't compress that well :) | 2009-12-09 21:35:21 |
<Shelwien> | well, try paq8 to understand what is slow ;) | 2009-12-09 21:35:54 |
<STalKer-Y> | no thanks ;-) | 2009-12-09 21:36:23 |
<Shelwien> | it also compresses much better than 7z ;) | 2009-12-09 21:36:45 |
<STalKer-Y> | sadly it is not quite practical yet :x | 2009-12-09 21:37:16 |
<Shelwien> | i don't think that "yet" is applicable here ;) | 2009-12-09 21:37:43 |
<STalKer-Y> | maybe with quantum computing | 2009-12-09 21:38:02 |
<Shelwien> | not really | 2009-12-09 21:38:39 |
<STalKer-Y> | heh | 2009-12-09 21:38:42 |
<Shelwien> | i guess we can make a FPGA paq coprocessor though | 2009-12-09 21:38:57 |
<STalKer-Y> | compressing 500gb of data with paq8? =/ | 2009-12-09 21:39:09 |
<Shelwien> | that's why we need all these new CPUs and GPUs ;) | 2009-12-09 21:39:59 |
<STalKer-Y> | do you know any compression programs that use GPU? :) | 2009-12-09 21:41:00 |
<Shelwien> | some h264 codecs and flac encoder | 2009-12-09 21:41:30 |
| but its really easy to make a "compression program which would use GPU" from nearly any open source compressor | 2009-12-09 21:42:15 |
| the problem is that it would be most of the time slower than with x86 | 2009-12-09 21:42:38 |
<STalKer-Y> | some distributed computing programs can use the GPU now :D | 2009-12-09 21:43:14 |
| but GPUs are very different from "normal" CPUs o.o | 2009-12-09 21:43:36 |
<Shelwien> | yeah, didn't see any useful program yet though | 2009-12-09 21:43:42 |
| for example, i have CoreAVC and 8800GT here | 2009-12-09 21:44:42 |
| but i had to disable the GPU mode after some testing | 2009-12-09 21:45:09 |
| because its buggy and slower than my cpu in complex cases | 2009-12-09 21:45:32 |
| ... | 2009-12-09 21:45:50 |
| and GPUs are different, yeah | 2009-12-09 21:45:54 |
| but with CUDA toolkit its really easy to compile a program to run on GPU | 2009-12-09 21:46:17 |
| optimizing it for speed is a completely different thing though | 2009-12-09 21:46:32 |
*** mike_____ has left the channel | 2009-12-09 22:06:26 |
*** STalKer-X has joined the channel | 2009-12-09 22:13:32 |
*** STalKer-Y has left the channel | 2009-12-09 22:14:12 |
*** mike_____ has joined the channel | 2009-12-09 22:17:04 |
*** mike_____ has left the channel | 2009-12-09 22:58:08 |
*** pinc has joined the channel | 2009-12-09 23:12:09 |
*** pinc has left the channel | 2009-12-10 00:04:41 |
<toffer> | gn8 | 2009-12-10 00:23:33 |
*** toffer has left the channel | 2009-12-10 00:23:39 |
<Shelwien> | ... | 2009-12-10 00:23:44 |
*** Krugz has joined the channel | 2009-12-10 00:31:33 |
*** schnaader has joined the channel | 2009-12-10 02:18:27 |
*** scott___ has joined the channel | 2009-12-10 02:22:59 |
<scott___> | I uploaded the time tests for BWTMIX with qsort and BWT BWTS | 2009-12-10 02:23:34 |
<Shelwien> | hi, ok | 2009-12-10 02:23:51 |
<scott___> | the BWT yutta fastest then BWTS then your qsort | 2009-12-10 02:24:19 |
<Shelwien> | i see | 2009-12-10 02:24:47 |
<scott___> | the source code and one set of timmings with MinGW execuatbles in the file | 2009-12-10 02:24:59 |
<Shelwien> | well, its as i expected, except that i couldn't know the actual timings | 2009-12-10 02:25:16 |
| but as you can see, there's no sense to use BWTS there at this point | 2009-12-10 02:25:35 |
<scott___> | I used zip instead of 7z since better with pkzip sadly its does not support the long names so shortened them | 2009-12-10 02:25:52 |
<Shelwien> | you can use 7z to create zip actually ;) | 2009-12-10 02:26:18 |
<scott___> | well BWTS faster than your qsort | 2009-12-10 02:26:28 |
<Shelwien> | yeah, but its easy to even write a new faster sort from scratch | 2009-12-10 02:27:03 |
<scott___> | well I don't know all the ins and outs of 7z but use it for your rar files | 2009-12-10 02:27:15 |
<Shelwien> | well, you can use something like 7z a -tzip -mx 1.zip * | 2009-12-10 02:27:46 |
| to create a zip archive | 2009-12-10 02:27:52 |
<scott___> | BWT will always be faster then BWTS all things equal | 2009-12-10 02:27:52 |
| I think I only have the windows version of 7z not a command line version | 2009-12-10 02:28:26 |
<Shelwien> | well, i'd suggest to also test decompression - results there could be even more interesting | 2009-12-10 02:28:33 |
| 7z always includes the command-line version | 2009-12-10 02:28:56 |
<scott___> | I tested both compression and decompression | 2009-12-10 02:29:03 |
<Shelwien> | also with GUI version you can create zip archives too | 2009-12-10 02:29:06 |
<scott___> | I would prefer non gui version | 2009-12-10 02:29:24 |
<Shelwien> | ah, so these are total times? | 2009-12-10 02:29:36 |
| i'd prefer to see them separately | 2009-12-10 02:29:44 |
<scott___> | the files BMI*.DAT show times of each phase as well as compression decompression | 2009-12-10 02:30:17 |
| your decompression slightly faster | 2009-12-10 02:30:44 |
<Shelwien> | %) | 2009-12-10 02:30:51 |
<scott___> | its compression that is slow | 2009-12-10 02:31:09 |
| I am not sure its easy to write a faster sort. Especially one that works for worst case type of files. | 2009-12-10 02:32:47 |
| but a lot of people are working on it. | 2009-12-10 02:33:18 |
<Shelwien> | i didn't say anything about writing faster sort than yuta's | 2009-12-10 02:33:21 |
| though that should be possible too, if we won't care about memory usage | 2009-12-10 02:33:39 |
| but beating qsort is easy | 2009-12-10 02:33:54 |
| even simply adding a 2-byte radix sort and doing qsorts after that | 2009-12-10 02:34:27 |
| would be already a considerable improvement | 2009-12-10 02:34:36 |
<scott___> | anyway they all compress to about the same size file regardless and that is what i wanted to show about BWTS since I think most don't belive it works at all | 2009-12-10 02:36:06 |
| and I am still hung up on bijective file compression and that can't be done with any sort of BWT you need BWTS to do that | 2009-12-10 02:37:11 |
<Shelwien> | well, now you're wrong about that | 2009-12-10 02:37:31 |
<scott___> | wrong about what? | 2009-12-10 02:37:51 |
<Shelwien> | BWT with EOF symbol doesn't require index coding even | 2009-12-10 02:38:02 |
| so its not any different from BWTS in that sense | 2009-12-10 02:38:13 |
<scott___> | no its not the same. Since you are increacing file by the lengh of EOF. But try to do the UNBWT where you use an EOF its not bijective in fact most UNBWT of arbitary file not possible | 2009-12-10 02:39:27 |
<Shelwien> | well, I understand that BWT loses the information about original string rotation | 2009-12-10 02:40:09 |
| so all the string shifts would produce the same BWT | 2009-12-10 02:40:22 |
<scott___> | no its more than that | 2009-12-10 02:40:23 |
| thats trute in the forward direction | 2009-12-10 02:40:43 |
| but whne you try UNBWT you can't do it for most files regardless of any index value | 2009-12-10 02:41:16 |
<Shelwien> | well, its probably possible to fix unbwt to avoid loops | 2009-12-10 02:42:34 |
<scott___> | no its not possible sorry | 2009-12-10 02:43:11 |
<Shelwien> | wby, it certainly is | 2009-12-10 02:43:31 |
<scott___> | not possible if you want the forward bwt to fo back to the file | 2009-12-10 02:43:42 |
<Shelwien> | at least you can discard the already used pointers in unBWT | 2009-12-10 02:43:48 |
| and find the next available | 2009-12-10 02:44:03 |
| thus all pointers would be only used once | 2009-12-10 02:44:19 |
<scott___> | well you could define a transform like that. I played with many version. The problem then is to define a forward transform to be the reverse of this function you just created | 2009-12-10 02:45:30 |
| just becage you can create some other permutation doesn't mean that it will be close to a real BWT or that it would have any meaning | 2009-12-10 02:46:25 |
<Shelwien> | it would produce same outputs for BWT strings | 2009-12-10 02:47:05 |
| and different outputs for non-BWT strings | 2009-12-10 02:47:12 |
<scott___> | I know ir sounds easy but I don't think it is | 2009-12-10 02:47:15 |
<Shelwien> | well, i'm just talking as i think | 2009-12-10 02:47:34 |
| of course i didn't spend much time thinking about how to make a bijective BWT ;) | 2009-12-10 02:47:59 |
<scott___> | well UNBWTS produces the same value for any BWT of a sting. | 2009-12-10 02:48:16 |
<Shelwien> | but anyway my point is that there's surely an "easier" workaround than BWTS | 2009-12-10 02:48:24 |
| one where we could use the available optimized BWT implementations | 2009-12-10 02:48:45 |
<scott___> | meaning if the sring is a result of a BWT tthen UNBWTS would be the same string but most likely rotated to another position | 2009-12-10 02:48:59 |
| well maybe know that BWTS exists someone will find a simple way to do what you wnat | 2009-12-10 02:51:38 |
<Shelwien> | err... its what _you_ want ;) | 2009-12-10 02:51:57 |
<scott___> | yes | 2009-12-10 02:52:11 |
| when I type fast I spell very poorly | 2009-12-10 02:52:28 |
<Shelwien> | ell, | 2009-12-10 02:52:33 |
| well, the problem for now | 2009-12-10 02:52:42 |
<scott___> | but evwn when I type slow my spelling sucks | 2009-12-10 02:52:52 |
<Shelwien> | is that you didn't suggest any applications for BWTS or any bijective stuff in fact | 2009-12-10 02:53:12 |
<scott___> | will Gil suggested one to me. | 2009-12-10 02:53:42 |
<Shelwien> | as is, BWTS doesn't have any benefits in compression or speed | 2009-12-10 02:53:57 |
| or memory usage | 2009-12-10 02:54:32 |
<scott___> | he calimed google compress a lot of lines of text to short segments each with an index. If you have a lot of short compression the removal of an index will save space in long run | 2009-12-10 02:54:40 |
| if you try BWTmix wiht short buffers you will see a big difference if you have many buffers | 2009-12-10 02:55:21 |
<Shelwien> | there's no sense to use BWT on short blocks | 2009-12-10 02:55:43 |
| in such cases even zlib would be better than any BWT | 2009-12-10 02:56:12 |
<scott___> | appearntly google does somewhere in there vast storage of things | 2009-12-10 02:56:16 |
*** scott___ has left the channel | 2009-12-10 03:02:20 |
*** scott___ has joined the channel | 2009-12-10 03:11:36 |
<Shelwien> | i have another idea | 2009-12-10 03:11:59 |
<scott___> | sorry I guess my connection to here is very poor | 2009-12-10 03:12:05 |
| it kicks me off after a short time | 2009-12-10 03:12:22 |
| it decodes tthere is to much delay in pinging or something like that | 2009-12-10 03:12:52 |
| what is your idea | 2009-12-10 03:13:04 |
<Shelwien> | BWT of a string with EOF doesn't have the rotation lossiness | 2009-12-10 03:13:15 |
| because string starting with EOF would always have fixed position | 2009-12-10 03:13:45 |
<scott___> | thats ture | 2009-12-10 03:14:00 |
<Shelwien> | so, it becomes just a question of finding the EOF position | 2009-12-10 03:14:03 |
| without encoding it | 2009-12-10 03:14:16 |
<scott___> | sadly if you take a random string even a very short one you can pretend the EOF is any where you like many string will still not have an inverse | 2009-12-10 03:14:46 |
| and some will actually jave more than one inverse | 2009-12-10 03:15:11 |
| i tried that kind of thing years ago | 2009-12-10 03:15:25 |
*** scott___ has left the channel | 2009-12-10 03:21:27 |
*** scott___ has joined the channel | 2009-12-10 03:28:46 |
| I think this is last time on tonight the connect to net itself iis not workin to well | 2009-12-10 03:29:29 |
| its not just an irc thing | 2009-12-10 03:29:49 |
| i look at packet sent and recieved and the nuber just stops | 2009-12-10 03:30:21 |
<Shelwien> | well, maybe i'd think more about this | 2009-12-10 03:36:47 |
| i still believe that it should be possible to somehow reverse the usual BWT without information loss | 2009-12-10 03:37:49 |
*** scott___ has left the channel | 2009-12-10 03:39:34 |
*** Shelwien has left the channel | 2009-12-10 04:02:53 |
*** schnaader_afk has left the channel | 2009-12-10 04:20:53 |
*** STalKer-Y has joined the channel | 2009-12-10 04:56:24 |
*** STalKer-X has left the channel | 2009-12-10 04:57:47 |
*** Shelwien has joined the channel | 2009-12-10 07:05:42 |
*** pinc has joined the channel | 2009-12-10 09:26:59 |
*** toffer has joined the channel | 2009-12-10 11:13:20 |
*** compbooks has left the channel | 2009-12-10 11:27:57 |
<toffer> | @shelwien: what dcc shall i download again? <2005 ? | 2009-12-10 11:59:50 |
*** toffer has left the channel | 2009-12-10 13:50:25 |