*** complogger has joined the channel2009-12-09 15:57:02
*** mike_____ has joined the channel2009-12-09 17:44:49
*** mike_____ has left the channel2009-12-09 18:31:20
*** mike_____ has joined the channel2009-12-09 18:42:10
*** scott___ has joined the channel2009-12-09 19:18:58
<toffer> hi! noticed that tests/tuning results -> thanks! cannot write much now, i'm preparing food. something like that:http://en.wikipedia.org/wiki/Rouladen2009-12-09 19:23:59
<Shelwien> %)2009-12-09 19:24:55
*** scott___ has left the channel2009-12-09 19:40:18
*** scott___ has joined the channel2009-12-09 19:48:54
<scott___> testing isthis thing wokring2009-12-09 19:49:25
<Shelwien> maybe2009-12-09 19:50:02
*** scott___ has left the channel2009-12-09 20:01:36
*** scott__ has joined the channel2009-12-09 20:28:11
<scott__> is it working2009-12-09 20:28:45
 hello2009-12-09 20:29:14
<toffer> @shelwien: how do you calculate your "metric" ? i can modify my optimization target to optimize for "efficiency".2009-12-09 20:29:27
 hi2009-12-09 20:29:29
<scott__> do your guys have trouble staying online more than a few minutes2009-12-09 20:30:13
 I tried several times to get on and then when I do only one or to lines before its dead2009-12-09 20:30:47
*** scott__ has left the channel2009-12-09 20:32:07
*** mike_____ has left the channel2009-12-09 20:37:56
<Shelwien> my metric is the time of data distribution to 10 people2009-12-09 20:40:58
 1 * compression time2009-12-09 20:41:11
 + compressed size / upload speed // 512kbit in my tests2009-12-09 20:41:32
 + 10 * compressed size / download speed // 512kbit too2009-12-09 20:42:01
 + 10 * decompression time2009-12-09 20:42:07
 Sami has the same metric implemented here: http://compressionratings.com/calc.cgi?file=app1.full.html&id=10&cc=1&cd=10&cu=1&cw=10&bu=4096&bd=512&bt=12009-12-09 20:43:17
 but because of time measurements in there2009-12-09 20:43:49
 it would be troublesome to tune the coder to it2009-12-09 20:44:05
*** mike_____ has joined the channel2009-12-09 20:49:27
<STalKer-Y> meh, compression is so slow :x2009-12-09 21:19:51
<Shelwien> ?2009-12-09 21:30:27
<STalKer-Y> 7zip on an athlon xp 2500+ :)2009-12-09 21:34:41
<Shelwien> it has different modes too2009-12-09 21:34:56
<STalKer-Y> which don't compress that well :)2009-12-09 21:35:21
<Shelwien> well, try paq8 to understand what is slow ;)2009-12-09 21:35:54
<STalKer-Y> no thanks ;-)2009-12-09 21:36:23
<Shelwien> it also compresses much better than 7z ;)2009-12-09 21:36:45
<STalKer-Y> sadly it is not quite practical yet :x2009-12-09 21:37:16
<Shelwien> i don't think that "yet" is applicable here ;)2009-12-09 21:37:43
<STalKer-Y> maybe with quantum computing2009-12-09 21:38:02
<Shelwien> not really2009-12-09 21:38:39
<STalKer-Y> heh2009-12-09 21:38:42
<Shelwien> i guess we can make a FPGA paq coprocessor though2009-12-09 21:38:57
<STalKer-Y> compressing 500gb of data with paq8? =/2009-12-09 21:39:09
<Shelwien> that's why we need all these new CPUs and GPUs ;)2009-12-09 21:39:59
<STalKer-Y> do you know any compression programs that use GPU? :)2009-12-09 21:41:00
<Shelwien> some h264 codecs and flac encoder2009-12-09 21:41:30
 but its really easy to make a "compression program which would use GPU" from nearly any open source compressor2009-12-09 21:42:15
 the problem is that it would be most of the time slower than with x86 2009-12-09 21:42:38
<STalKer-Y> some distributed computing programs can use the GPU now :D2009-12-09 21:43:14
 but GPUs are very different from "normal" CPUs o.o2009-12-09 21:43:36
<Shelwien> yeah, didn't see any useful program yet though2009-12-09 21:43:42
 for example, i have CoreAVC and 8800GT here2009-12-09 21:44:42
 but i had to disable the GPU mode after some testing2009-12-09 21:45:09
 because its buggy and slower than my cpu in complex cases2009-12-09 21:45:32
 ...2009-12-09 21:45:50
 and GPUs are different, yeah2009-12-09 21:45:54
 but with CUDA toolkit its really easy to compile a program to run on GPU2009-12-09 21:46:17
 optimizing it for speed is a completely different thing though2009-12-09 21:46:32
*** mike_____ has left the channel2009-12-09 22:06:26
*** STalKer-X has joined the channel2009-12-09 22:13:32
*** STalKer-Y has left the channel2009-12-09 22:14:12
*** mike_____ has joined the channel2009-12-09 22:17:04
*** mike_____ has left the channel2009-12-09 22:58:08
*** pinc has joined the channel2009-12-09 23:12:09
*** pinc has left the channel2009-12-10 00:04:41
<toffer> gn82009-12-10 00:23:33
*** toffer has left the channel2009-12-10 00:23:39
<Shelwien> ...2009-12-10 00:23:44
*** Krugz has joined the channel2009-12-10 00:31:33
*** schnaader has joined the channel2009-12-10 02:18:27
*** scott___ has joined the channel2009-12-10 02:22:59
<scott___> I uploaded the time tests for BWTMIX with qsort and BWT BWTS2009-12-10 02:23:34
<Shelwien> hi, ok2009-12-10 02:23:51
<scott___> the BWT yutta fastest then BWTS then your qsort2009-12-10 02:24:19
<Shelwien> i see2009-12-10 02:24:47
<scott___> the source code and one set of timmings with MinGW execuatbles in the file2009-12-10 02:24:59
<Shelwien> well, its as i expected, except that i couldn't know the actual timings2009-12-10 02:25:16
 but as you can see, there's no sense to use BWTS there at this point2009-12-10 02:25:35
<scott___> I used zip instead of 7z since better with pkzip sadly its does not support the long names so shortened them2009-12-10 02:25:52
<Shelwien> you can use 7z to create zip actually ;)2009-12-10 02:26:18
<scott___> well BWTS faster than your qsort2009-12-10 02:26:28
<Shelwien> yeah, but its easy to even write a new faster sort from scratch2009-12-10 02:27:03
<scott___> well I don't know all the ins and outs of 7z but use it for your rar files2009-12-10 02:27:15
<Shelwien> well, you can use something like 7z a -tzip -mx 1.zip * 2009-12-10 02:27:46
 to create a zip archive2009-12-10 02:27:52
<scott___>  BWT will always be faster then BWTS all things equal2009-12-10 02:27:52
 I think I only have the windows version of 7z not a command line version2009-12-10 02:28:26
<Shelwien> well, i'd suggest to also test decompression - results there could be even more interesting2009-12-10 02:28:33
 7z always includes the command-line version2009-12-10 02:28:56
<scott___> I tested both compression and decompression2009-12-10 02:29:03
<Shelwien> also with GUI version you can create zip archives too2009-12-10 02:29:06
<scott___> I would prefer non gui version2009-12-10 02:29:24
<Shelwien> ah, so these are total times?2009-12-10 02:29:36
 i'd prefer to see them separately2009-12-10 02:29:44
<scott___> the files BMI*.DAT show times of each phase as well as compression decompression2009-12-10 02:30:17
 your decompression slightly faster2009-12-10 02:30:44
<Shelwien> %)2009-12-10 02:30:51
<scott___> its compression that is slow2009-12-10 02:31:09
 I am not sure its easy to write a faster sort. Especially one that works for worst case type of files.2009-12-10 02:32:47
 but a lot of people are working on it.2009-12-10 02:33:18
<Shelwien> i didn't say anything about writing faster sort than yuta's2009-12-10 02:33:21
 though that should be possible too, if we won't care about memory usage2009-12-10 02:33:39
 but beating qsort is easy2009-12-10 02:33:54
 even simply adding a 2-byte radix sort and doing qsorts after that2009-12-10 02:34:27
 would be already a considerable improvement2009-12-10 02:34:36
<scott___> anyway they all compress to about the same size file regardless and that is what i wanted to show about BWTS since I think most don't belive it works at all2009-12-10 02:36:06
 and I am still hung up on bijective file compression and that can't be done with any sort of BWT you need BWTS to do that2009-12-10 02:37:11
<Shelwien> well, now you're wrong about that2009-12-10 02:37:31
<scott___> wrong about what?2009-12-10 02:37:51
<Shelwien> BWT with EOF symbol doesn't require index coding even2009-12-10 02:38:02
 so its not any different from BWTS in that sense2009-12-10 02:38:13
<scott___> no its not the same. Since you are increacing file by the lengh of EOF. But try to do the UNBWT where you use an EOF its not bijective in fact most UNBWT of arbitary file not possible2009-12-10 02:39:27
<Shelwien> well, I understand that BWT loses the information about original string rotation2009-12-10 02:40:09
 so all the string shifts would produce the same BWT2009-12-10 02:40:22
<scott___> no its more than that2009-12-10 02:40:23
 thats trute in the forward direction2009-12-10 02:40:43
 but whne you try UNBWT you can't do it for most files regardless of any index value2009-12-10 02:41:16
<Shelwien> well, its probably possible to fix unbwt to avoid loops2009-12-10 02:42:34
<scott___> no its not possible sorry2009-12-10 02:43:11
<Shelwien> wby, it certainly is2009-12-10 02:43:31
<scott___> not possible if you want the forward bwt to fo back to the file2009-12-10 02:43:42
<Shelwien> at least you can discard the already used pointers in unBWT2009-12-10 02:43:48
 and find the next available2009-12-10 02:44:03
 thus all pointers would be only used once2009-12-10 02:44:19
<scott___> well you could define a transform like that. I played with many version. The problem then is to define a forward transform to be the reverse of this function you just created2009-12-10 02:45:30
 just becage you can create some other permutation doesn't mean that it will be close to a real BWT or that it would have any meaning2009-12-10 02:46:25
<Shelwien> it would produce same outputs for BWT strings2009-12-10 02:47:05
 and different outputs for non-BWT strings2009-12-10 02:47:12
<scott___> I know ir sounds easy but I don't think it is2009-12-10 02:47:15
<Shelwien> well, i'm just talking as i think2009-12-10 02:47:34
 of course i didn't spend much time thinking about how to make a bijective BWT ;)2009-12-10 02:47:59
<scott___> well UNBWTS produces the same value for any BWT of a sting.2009-12-10 02:48:16
<Shelwien> but anyway my point is that there's surely an "easier" workaround than BWTS2009-12-10 02:48:24
 one where we could use the available optimized BWT implementations2009-12-10 02:48:45
<scott___> meaning if the sring is a result of a BWT tthen UNBWTS would be the same string but most likely rotated to another position2009-12-10 02:48:59
 well maybe know that BWTS exists someone will find a simple way to do what you wnat2009-12-10 02:51:38
<Shelwien> err... its what _you_ want ;)2009-12-10 02:51:57
<scott___> yes2009-12-10 02:52:11
 when I type fast I spell very poorly2009-12-10 02:52:28
<Shelwien> ell,2009-12-10 02:52:33
 well, the problem for now2009-12-10 02:52:42
<scott___> but evwn when I type slow my spelling sucks2009-12-10 02:52:52
<Shelwien> is that you didn't suggest any applications for BWTS or any bijective stuff in fact2009-12-10 02:53:12
<scott___> will Gil suggested one to me.2009-12-10 02:53:42
<Shelwien> as is, BWTS doesn't have any benefits in compression or speed2009-12-10 02:53:57
 or memory usage2009-12-10 02:54:32
<scott___> he calimed google compress a lot of lines of text to short segments each with an index. If you have a lot of short compression the removal of an index will save space in long run2009-12-10 02:54:40
 if you try BWTmix wiht short buffers you will see a big difference if you have many buffers2009-12-10 02:55:21
<Shelwien> there's no sense to use BWT on short blocks2009-12-10 02:55:43
 in such cases even zlib would be better than any BWT2009-12-10 02:56:12
<scott___> appearntly google does somewhere in there vast storage of things2009-12-10 02:56:16
*** scott___ has left the channel2009-12-10 03:02:20
*** scott___ has joined the channel2009-12-10 03:11:36
<Shelwien> i have another idea2009-12-10 03:11:59
<scott___> sorry I guess my connection to here is very poor2009-12-10 03:12:05
 it kicks me off after a short time2009-12-10 03:12:22
 it decodes tthere is to much delay in pinging or something like that2009-12-10 03:12:52
 what is your idea2009-12-10 03:13:04
<Shelwien> BWT of a string with EOF doesn't have the rotation lossiness2009-12-10 03:13:15
 because string starting with EOF would always have fixed position2009-12-10 03:13:45
<scott___> thats ture2009-12-10 03:14:00
<Shelwien> so, it becomes just a question of finding the EOF position2009-12-10 03:14:03
 without encoding it2009-12-10 03:14:16
<scott___> sadly if you take a random string even a very short one you can pretend the EOF is any where you like many string will still not have an inverse2009-12-10 03:14:46
 and some will actually jave more than one inverse2009-12-10 03:15:11
 i tried that kind of thing years ago2009-12-10 03:15:25
*** scott___ has left the channel2009-12-10 03:21:27
*** scott___ has joined the channel2009-12-10 03:28:46
  I think this is last time on tonight the connect to net itself iis not workin to well2009-12-10 03:29:29
 its not just an irc thing2009-12-10 03:29:49
 i look at packet sent and recieved and the nuber just stops2009-12-10 03:30:21
<Shelwien> well, maybe i'd think more about this2009-12-10 03:36:47
 i still believe that it should be possible to somehow reverse the usual BWT without information loss2009-12-10 03:37:49
*** scott___ has left the channel2009-12-10 03:39:34
*** Shelwien has left the channel2009-12-10 04:02:53
*** schnaader_afk has left the channel2009-12-10 04:20:53
*** STalKer-Y has joined the channel2009-12-10 04:56:24
*** STalKer-X has left the channel2009-12-10 04:57:47
*** Shelwien has joined the channel2009-12-10 07:05:42
*** pinc has joined the channel2009-12-10 09:26:59
*** toffer has joined the channel2009-12-10 11:13:20
*** compbooks has left the channel2009-12-10 11:27:57
<toffer> @shelwien: what dcc shall i download again? <2005 ?2009-12-10 11:59:50
*** toffer has left the channel2009-12-10 13:50:25