*** toffer has joined the channel2009-12-12 13:42:30
<toffer> hi2009-12-12 13:45:52
*** mike_____ has joined the channel2009-12-12 14:21:41
*** scott___ has joined the channel2009-12-12 15:16:01
<scott___> whats new2009-12-12 15:17:09
*** scott___ has left the channel2009-12-12 15:38:26
<toffer> >.< everyone is sleeping2009-12-12 15:59:07
*** schnaader has joined the channel2009-12-12 16:59:01
<schnaader> Na, sind sie immernoch alle am Schlafen? ;)2009-12-12 17:00:20
<toffer> richtig2009-12-12 17:10:30
<schnaader> btw, tried to translate my sentence on google translate, which gives the result "Well, they're all still awake?" which is quite wrong... :) After removing "am" it works, though2009-12-12 17:12:58
*** mike_____ has left the channel2009-12-12 17:24:50
*** mike_____ has joined the channel2009-12-12 17:28:09
<toffer> gonna go downtown now, bye2009-12-12 17:30:28
*** toffer has left the channel2009-12-12 17:30:34
*** Shelwien has left the channel2009-12-12 18:28:13
*** Guest9968193 has joined the channel2009-12-12 18:28:16
*** schnaader_afk has left the channel2009-12-12 18:36:46
*** schnaader has joined the channel2009-12-12 18:55:20
*** schnaader has left the channel2009-12-12 20:08:27
*** chornobl has joined the channel2009-12-12 20:36:43
*** Krugz has joined the channel2009-12-12 20:39:27
<Shelwien> ...2009-12-12 21:09:01
<Krugz> lol sorry, I have nothing to say! :P2009-12-12 21:10:06
<Shelwien> what about http://ps16893.dreamhost.com/log/2009-12-11_03-53-30.txt2009-12-12 21:11:41
 like from 03:092009-12-12 21:11:57
<Krugz> ?2009-12-12 21:12:13
<Shelwien> paper storage ;)2009-12-12 21:12:38
<Krugz> O.o well it sounds interesting, but I really don't know anything about it :O2009-12-12 21:13:18
<Shelwien> i posted a log link where we discussed it ;)2009-12-12 21:13:49
<Krugz> ya I see it, but what could I say about it? :P2009-12-12 21:15:13
<Shelwien> dunno, but maybe you know how print stuff on laserjet printer without dithering? ;)2009-12-12 21:15:55
 *how to2009-12-12 21:16:02
<Krugz> not likely, I hardly do anything interesting with printers 2009-12-12 21:16:42
 I would never have thought to use pixels on paper to store data, what's the objective anyways?2009-12-12 21:17:15
<Shelwien> it seems possible to put 6-12M of data onto one side of A4 sheet2009-12-12 21:17:59
 that's not much maybe, but has some applications2009-12-12 21:18:47
<Krugz> I can't think of any, tbh 2009-12-12 21:18:56
<Shelwien> you can sell your program ;)2009-12-12 21:19:05
<chornobl> as a paper copy?2009-12-12 21:19:22
<Krugz> You can't already sell a program?2009-12-12 21:19:27
<Shelwien> its much cheaper with paper ;)2009-12-12 21:19:40
<Krugz> O.o wouldn't electronic files be much cheaper??2009-12-12 21:19:51
<Shelwien> and easier in fact2009-12-12 21:19:53
 electronic files are files2009-12-12 21:20:13
<Krugz> well I mean2009-12-12 21:20:38
<chornobl> how it compares to tape2009-12-12 21:20:40
<Shelwien> but you won't be able to easily copy that stuff on paper2009-12-12 21:20:41
 not by xerox stuff anyway2009-12-12 21:20:57
 anyway, obviously its just for fun2009-12-12 21:21:43
<Krugz> I'm confused, what are you talking about? why would you need to copy your program on paper? If you can store your program as pixels on a piece of paper, wouldn't it be easier to do a similiar virtualized process in like pdf or something?2009-12-12 21:21:51
 lol well just for fun is totally different :P2009-12-12 21:22:03
<Shelwien> its not any similar as pdf is not lossy2009-12-12 21:22:35
<chornobl> you can't put pdf into your desk2009-12-12 21:22:58
<Krugz> yes but2009-12-12 21:23:12
 in order to read the paper again, you'd have to go to a computer2009-12-12 21:23:23
 so you might as well just have it at the computer already -.-2009-12-12 21:23:32
<Shelwien> yeah2009-12-12 21:23:36
 i just suggested that it can be used for copy protection2009-12-12 21:23:49
<chornobl> and how reliable that storage (time wise)2009-12-12 21:24:09
<Krugz> ah I see.. so using this method to "encrypt" sort of?2009-12-12 21:24:11
<Shelwien> its reliability depends on added redundancy2009-12-12 21:24:34
 thats kinda the point of the whole experiment2009-12-12 21:24:46
<Krugz> well it is an odd idea, and it sounds kinda fun lol2009-12-12 21:25:23
<Shelwien> unfortunately its also nothing new - http://ollydbg.de/Paperbak/2009-12-12 21:26:16
 but that thing only used standard codes and libs2009-12-12 21:26:35
 which is not fun at all imho2009-12-12 21:26:42
<Krugz> lolol2009-12-12 21:28:00
<Shelwien> btw, chornobl, can you test something?2009-12-12 21:36:00
<chornobl> like what2009-12-12 21:39:55
<Shelwien> something like tar ;)2009-12-12 21:40:07
<chornobl> what exactly2009-12-12 21:40:24
<Shelwien> i just want to know what kind of copy speed it would show for you2009-12-12 21:40:53
<chornobl> so what to do2009-12-12 21:53:44
<Shelwien> this: http://nishi.dreamhosters.com/u/scan7_v3.rar2009-12-12 21:54:15
 just run some scan7 c:2009-12-12 21:54:44
 scan7 c:\windows2009-12-12 21:54:53
 or whatever else you have to test2009-12-12 21:55:05
<chornobl> with timer?2009-12-12 21:55:16
<Shelwien> and tell me what kind of speed it shows ;)2009-12-12 21:55:19
 you can, but it reports the speed2009-12-12 21:55:38
 though also you can compare with rar a -m0 and a timer, i guess2009-12-12 21:55:57
*** chornobl has left the channel2009-12-12 22:16:08
*** chornobl has joined the channel2009-12-12 22:19:50
<chornobl> it didn work well2009-12-12 22:20:20
 free space is ended2009-12-12 22:21:33
<Shelwien> well, you can find some smaller directory ;)2009-12-12 22:22:31
<chornobl> how smaller2009-12-12 22:23:31
<Shelwien> to fit in your free space? ;)2009-12-12 22:28:04
 as i said, its kinda an archival tool without compression2009-12-12 22:28:27
 similar to rar -m0 etc2009-12-12 22:28:32
<chornobl> i got 413mb, should i scan bigger directory?2009-12-12 22:29:31
<Shelwien> well, i think it would work somehow even if your free space ends ;)2009-12-12 22:30:04
<chornobl> _somehow_2009-12-12 22:30:16
<Shelwien> ok, find something smaller then ;)2009-12-12 22:31:09
<Krugz> I'm trying to do some homework that's regarding Caches, and I'm doing some reading but I've run into something I have a question about.. anyone who might know?2009-12-12 22:51:18
<Shelwien> ?2009-12-12 22:53:36
<Krugz> I think I might have found my answer actually..2009-12-12 22:53:56
*** schnaader has joined the channel2009-12-12 23:08:49
<chornobl> http://www.sendspace.com/file/pr4sb12009-12-12 23:11:26
<Shelwien> hi schnaader2009-12-12 23:13:33
<schnaader> hi2009-12-12 23:14:30
 just tested streaming some music with VLC - if anyone wants to listen: http://92.228.56.102:1234 :) just streaming some test pop megamix, though ;)2009-12-12 23:15:15
 but beware, only 100 KB/s upstream here, so up to 4 listeners only :D2009-12-12 23:15:34
<Shelwien> %)2009-12-12 23:15:52
<schnaader> Last time I set up some IceCast server and nothing worked, now I saw they put the functionality direct in VLC...2009-12-12 23:16:20
<Shelwien> for me icecast worked, but VLC didn't ;)2009-12-12 23:16:47
<chornobl> huh popmusic of the 80-90s2009-12-12 23:17:15
<schnaader> Well, I think icecast could've worked for me, too, but I just couldn't figured out how it all works2009-12-12 23:17:19
 yes, actually that's a ~100 MB MP2 testfile I found :)2009-12-12 23:17:32
<Shelwien> so2009-12-12 23:17:50
 <\\?\D:\*>2009-12-12 23:17:52
 dirs=3419 files=48130 bytes=12342081041 outbytes=439803904 speed=10081832009-12-12 23:17:52
 <\\?\C:\WINDOWS\system32>2009-12-12 23:17:52
 dirs=211 files=3349 bytes=864180479 outbytes=443830265 speed=44802822009-12-12 23:17:52
 <\\?\c:\windows>2009-12-12 23:17:52
 dirs=556 files=8542 bytes=1460409691 outbytes=443699197 speed=35585042009-12-12 23:17:53
 <\\?\c:\Program Files>2009-12-12 23:17:55
 dirs=309 files=4742 bytes=394493886 outbytes=394493886 speed=64047452009-12-12 23:17:57
 timetest system\winrar\rar a -m0 1 "c:\program files"2009-12-12 23:17:59
 Tested program has wasted 57.828s2009-12-12 23:18:01
 394493886/57.828 = 6821849 bytes/s2009-12-12 23:18:03
 the point of this, actually2009-12-12 23:18:09
 is that people keep talking about 100M/s compression algorithms2009-12-12 23:18:24
<chornobl> that is not a new pc2009-12-12 23:18:39
<Shelwien> you'd have it even slower on a new notebook2009-12-12 23:18:54
<chornobl> plus free space is highly limited2009-12-12 23:18:55
<Shelwien> and that applies to anybody2009-12-12 23:19:06
<chornobl> and everythig fragmented2009-12-12 23:19:17
<schnaader> Is it just scanning or doing something with the data?2009-12-12 23:19:23
<Shelwien> of course i know that it can show 200Mb/s on ramdrive2009-12-12 23:19:25
 my util similar to tar2009-12-12 23:19:46
<schnaader> you might gain some performance with prebuffering and other optimizations perhaps2009-12-12 23:19:49
<Shelwien> but with unicode support and attr/time saving2009-12-12 23:19:58
 sure, that's why we have rar for comparison2009-12-12 23:20:12
<schnaader> but I noticed that problem, too. that's why my first step before compressing some directories is tarring them, after that you get the 100 MB/s you want here2009-12-12 23:20:48
<Shelwien> anyway, it seems that 7M/s (de)compression speed is ok actually2009-12-12 23:20:56
 and you get 100M/s probably only because of caching ;)2009-12-12 23:21:23
<schnaader> yeah, 100 M/s on a nice good PC with SSD or very good HD, but saw ~40 MB/s with THOR already2009-12-12 23:21:57
<Shelwien> still, i think strong compression might actually have more applications than they say2009-12-12 23:23:09
<schnaader> the 7M/s could really get better after defrag, try that, too - also, do you have some graph over time, should get faster at big files, shouldn't it?2009-12-12 23:23:29
<Shelwien> why don't you test it yourself? ;)2009-12-12 23:24:17
 http://nishi.dreamhosters.com/u/scan7_v3.rar2009-12-12 23:24:26
 vs rar -m02009-12-12 23:24:29
<schnaader> yeah, just saw the URL in the log, think will test a bit later2009-12-12 23:24:34
 what I don't like about the VLC streaming is that you can't adjust playback speed when streaming anymore like with local playback - that makes that mix a lot better ;)2009-12-12 23:26:25
<Shelwien> %)2009-12-12 23:27:05
<schnaader> some better music streaming now :)2009-12-12 23:32:35
 actually, that scanning is a thing I'll have to do for precomp soon to support multiple files and directories2009-12-12 23:34:16
 although it might get a bit more complex to prevent that f.e. using "*.*" and ".\*.*" together doesn't add everything twice - and perhaps there'll have to be some file sorting to do2009-12-12 23:35:48
*** scott___ has joined the channel2009-12-12 23:41:00
 Testing scan7 on two different HDs (in: local HD, out: external USB drive) boosts speed from 1,8 MB/s to 9,6 MB/s2009-12-12 23:52:02
 still a bit slow, yeah2009-12-12 23:52:21
<Shelwien> you can also test extracting speed btw2009-12-12 23:56:29
 for decompression speed limit2009-12-12 23:56:40
 hi David2009-12-12 23:56:43
<schnaader> (in: external drive, out: local HD) gives 2,9 MB/s... strange, would've expected reading from external drive to be faster...2009-12-12 23:57:22
<scott___> hello2009-12-12 23:57:28
<Shelwien> if "local HD" is system HD, that's troublesome in windows2009-12-12 23:58:14
<schnaader> ah, I see2009-12-12 23:58:23
<Shelwien> i mean, it always writes something2009-12-12 23:58:40
<schnaader> just one HD available here, that's why I'm using the external as a second one :)2009-12-12 23:58:50
<Shelwien> anyway, extraction speed is interesting too2009-12-12 23:59:07
<schnaader> seems the cache has recognized me now - local->local is at 4,4 MB/s now, testing extraction now2009-12-12 23:59:57
 extraction local->local 4,0 MB/s2009-12-13 00:00:41
 ouch... extraction local->HD 500 KB/s2009-12-13 00:01:44
 extraction USB->local 4,7 MB/s2009-12-13 00:03:29
 so yes, nothing over 10 MB/s here, blame seek time I guess, although not getting more even for big files is a bit sad, too2009-12-13 00:04:33
<Shelwien> i guess that's it for fast LZ and multithreaded compression ;)2009-12-13 00:06:22
<schnaader> is there something wrong with the code? testing storing a big 600 MB file USB->local now, speed rises slowly from ~6 MB/s to ~20 MB/s and still increasing...2009-12-13 00:07:23
<Shelwien> you can verify it with rar -m0 or 7z or anything2009-12-13 00:08:27
<schnaader> 7z gives 15 MB/s right away and pretty much stays there2009-12-13 00:10:25
<Shelwien> about 24-27MB/s for usb2 is ok though2009-12-13 00:10:30
<schnaader> RAR is quite slow, 1 minute for 400 MB, so only 6 MB/s2009-12-13 00:11:52
 yes, I think a read test gave me ~30 MB/s, although I might just retest, was long ago :)2009-12-13 00:12:32
 yes, USB disk ~27 MB/s, local disk 30-50 MB/s2009-12-13 00:13:57
<Shelwien> local sata could be easily >100MB/s on large straight files2009-12-13 00:14:55
 but does it really mean that 100MB/s compression algos are the most useful? ;)2009-12-13 00:15:52
<schnaader> well, newest SSD disks should give you 150-200 MB/s :) and seek time is muuch better there2009-12-13 00:16:00
<Shelwien> sure2009-12-13 00:16:10
 and then they die in a month2009-12-13 00:16:15
<schnaader> not dying completely, but going down to 50-100 MB/s, yes2009-12-13 00:16:38
<Shelwien> might die too, for all i know2009-12-13 00:17:01
 i have hdds crashing on me when i download torrents for a few month to the same place2009-12-13 00:17:26
<schnaader> if you transfer some TB per day, it might, yes, but unless this case, modern SSDs have quite clever algorithms that prevent such things2009-12-13 00:17:55
<Shelwien> unfortunately it doesn't seem like they do any real tests2009-12-13 00:18:27
<schnaader> there are some nice tests, f.e. read about testing a 2 GB Flash stick (7 MB/s) 1 month 24/7 - 23,5 TB data transfer without errors2009-12-13 00:21:25
 and this should be transferable to SSD, error rate should be even less with SSDs, as you don't tend to fill them completely 24/7 :)2009-12-13 00:22:10
<Shelwien> you know, not all the samples have the same quality ;)2009-12-13 00:23:09
<schnaader> yes, better use an Intel one2009-12-13 00:23:49
 or wait some years until SSD hopefully gets mainstream2009-12-13 00:24:37
<Shelwien> hdds are more interesting somehow ;)2009-12-13 00:25:16
 same as with CRT vs LCD ;)2009-12-13 00:25:24
<schnaader> HD is one of the biggest bottleneck at the moment - the scanning experiment shows this: you might be faster when putting all your game data in a big container, even slightly compressed, because you get >50 MB/s this way instead of 7 MB/s2009-12-13 00:25:52
<Shelwien> sure, and game developers know that ;)2009-12-13 00:26:18
<schnaader> of course :)2009-12-13 00:26:24
<Shelwien> asmodean here btw2009-12-13 00:26:36
<schnaader> that's the main reason for DAEMON Tools, too, isn't it ;)2009-12-13 00:26:54
<Shelwien> http://asmodean.reverse.net/2009-12-13 00:27:00
<schnaader> Nice page - added to favorites :)2009-12-13 00:28:09
<Shelwien> ;)2009-12-13 00:28:23
<schnaader> one last thing about scan7 - perhaps it would get faster by preoccupying the space, so it doesn't has to enlarge the file constantly but just write the data there2009-12-13 00:29:40
<Shelwien> that's only important for multithreaded stuff imho2009-12-13 00:30:33
<schnaader> ok2009-12-13 00:30:48
<Shelwien> and there's a lot of improvement required anyway2009-12-13 00:30:54
 like, async i/o should be more important imho2009-12-13 00:31:16
 and i'm even sure that there's nothing to improve in the scanning2009-12-13 00:31:48
 ah2009-12-13 00:33:25
 btw, that scanner also does some calculations too2009-12-13 00:33:50
 crc32 x22009-12-13 00:34:01
 btw, i've decided to use crc32 everywhere ;)2009-12-13 00:34:35
 but to reduce the collision probability2009-12-13 00:34:50
 i extend it with crc32 of incremented bytes ;)2009-12-13 00:35:03
<schnaader> is the crc calculated constantly or only after each file?2009-12-13 00:35:16
<Shelwien> so there's a 64-bit hash ;)2009-12-13 00:35:17
 2x crc32 is calculated for each file and stored in $fsnodes2009-12-13 00:35:42
<schnaader> but that's the same for 7z/rar, so it's kind of comparable2009-12-13 00:36:38
 although 7z/rar use more complex hashes2009-12-13 00:36:48
<Shelwien> huh?2009-12-13 00:37:32
<schnaader> doesn't it? actually I'm not sure :)2009-12-13 00:37:45
<Shelwien> never heard of rar using anything other than crc322009-12-13 00:37:59
 and 7z should do that by default as well2009-12-13 00:38:12
<schnaader> Ah yes, both using crc32 :)2009-12-13 00:38:54
 sorry for that :)2009-12-13 00:38:58
 well, shouldn't matter anyway, even md5sum is bottlenecked by HD speed here2009-12-13 00:39:17
<Shelwien> i'm just saying that people should make new CM/PPM coders instead of all these LZ ;)2009-12-13 00:40:58
 because even if its hard to reach 100M/s with CM2009-12-13 00:41:28
 but it still can work with practically required speed2009-12-13 00:41:50
<schnaader> oh I think we aren't that far away from 100 M/s with CM, some parallel algorithms and 16 or 32 cores in some years and you have it :)2009-12-13 00:42:43
<Shelwien> nah, won't work2009-12-13 00:43:03
 well, if only BWT maybe2009-12-13 00:43:31
 but BWT has even more troubles with i/o2009-12-13 00:43:53
 because blockwise i/o with 100M blocks is slow2009-12-13 00:44:00
 and even stuff like separately compressing the segments in parallel2009-12-13 00:46:31
 is not any compatible with i/o either2009-12-13 00:46:43
<schnaader> anyway, I think the main disadvantage and reason people are still doing LZ algorithms is the asymmetrical decompression - most people won't care about ~1 MB/s compression, but if they're offered ~1 MB/s or 10-20 MB/s decompression, they'll choose the latter2009-12-13 00:47:04
<Shelwien> i rarely decompress stuff which i compress ;)2009-12-13 00:47:42
 and nobody said that asymmetric CM can't exist - people are just lazy to make them2009-12-13 00:48:18
 well, there're only a few CM developers so its not really a wonder2009-12-13 00:48:55
<schnaader> the main problem I'm facing with CM is that its hard to find sources that describe how it all works and how one can put up a simple CM to start with2009-12-13 00:49:58
<Shelwien> huh... http://ctxmodel.net http://ctxmodel.net/files ?2009-12-13 00:50:35
 fpaq0p?2009-12-13 00:50:46
 m1?2009-12-13 00:50:55
<schnaader> yes, but some people get scared of the source code and like pretty diagrams that describe the algorithm :)2009-12-13 00:52:23
<Shelwien> but there's nothing to describe kinda %) 2009-12-13 00:52:48
 well, i mean, arithmetic coding is described everywhere2009-12-13 00:53:08
 and CM is just adding a statistical probability estimator to the arithmetic coding2009-12-13 00:53:48
 people even try to write NN-based compressors and the like2009-12-13 00:54:10
 or bayesian2009-12-13 00:54:17
 or CTW (which has a lot of documentation btw)2009-12-13 00:54:28
 ...what do you think about my rc explaination btw? http://www.ctxmodel.net/rem.pl?-372009-12-13 00:54:55
*** mike_____ has left the channel2009-12-13 00:57:49
<schnaader> that seems to explain it pretty well, nice work2009-12-13 00:58:35
<Shelwien> thanks, but kinda reminds me of http://www.ctxmodel.net/rem.pl?-8 ;)2009-12-13 00:59:34
<schnaader> ??2009-12-13 01:01:35
<Shelwien> its a spam trap ;)2009-12-13 01:01:44
<schnaader> so? is the rc explanation one too? don't get it...2009-12-13 01:02:44
<Shelwien> "I agree with auther. Article rather interesting and usefull" etc2009-12-13 01:03:47
 there're lots of bot comments like that2009-12-13 01:04:15
 i guess its an indirect spam with authors profile or something2009-12-13 01:04:34
 and your comment just reminded me of that - very abstract and neutral ;)2009-12-13 01:05:35
<schnaader> well, at least you can be pretty sure I'm not a spambot ;) - BUY VIARGA FOR FREE CIALI$ VERY LOW PRICES2009-12-13 01:06:27
<Shelwien> yeah ;)2009-12-13 01:06:55
<schnaader> and I just kind of overread the explanation, can't say much more about it until I start experimenting with deriving my own first rangecoder from it :)2009-12-13 01:07:28
<Shelwien> well, my recent example is this2009-12-13 01:07:48
 !grep brc2009-12-13 01:07:52
 it also includes coroutines though ;)2009-12-13 01:08:18
<schnaader> :) you're lucky complogger didn't think that last one was another comment and recursed2009-12-13 01:08:29
 yes, that was the Intel/GCC thing :)2009-12-13 01:09:37
<Shelwien> coder.inc contains the actual coder there2009-12-13 01:10:37
 and you can replace "uint freq = hSCALE + 1000;" with anything2009-12-13 01:11:06
 and get it to work like paq8 if you want2009-12-13 01:11:19
 (using paq8 model)2009-12-13 01:11:23
* Shelwien thinks about adding a big "TL;DR" button to all ctxmodel articles2009-12-13 01:15:57
<schnaader> TL;DR?2009-12-13 01:17:25
 OK, just had a look at fpaq0.cpp - this is indeed a well documented and readable start point, perhaps I should have a look at it and its variants to understand things better2009-12-13 01:19:10
<Shelwien> %)2009-12-13 01:19:25
 i'd still suggest fpaq0p though2009-12-13 01:19:39
 http://en.wiktionary.org/wiki/TLDR2009-12-13 01:20:05
<schnaader> :) ah, I see2009-12-13 01:21:00
<Shelwien> i didn't mean my version ;)2009-12-13 01:21:35
<schnaader> fpaq0p.zip link here ( http://mattmahoney.net/dc/#fpaq0 ) doesn't work2009-12-13 01:21:37
<Shelwien> maybe this: http://www.ctxmodel.net/rem.pl?-6 ?2009-12-13 01:23:28
 ah, no2009-12-13 01:23:49
 guess i meant this: http://shelwien.googlepages.com/fpaq0p-sb_sh.cpp2009-12-13 01:25:18
<schnaader> that's better, yes :)2009-12-13 01:26:01
<Shelwien> though thats actually was a demo of why counter updates like2009-12-13 01:27:31
 p -= p>>52009-12-13 01:27:43
 are randomized2009-12-13 01:27:55
*** schnaader has left the channel2009-12-13 01:51:56
*** scott___ has left the channel2009-12-13 01:53:30
*** chornobl has left the channel2009-12-13 01:57:09
*** STalKer-X has joined the channel2009-12-13 04:52:28
*** STalKer-Y has left the channel2009-12-13 04:54:47
*** Scientist_encode has joined the channel2009-12-13 05:25:06
 !next2009-12-13 05:51:55