Slow scanning even with cache

[DocMAX]

Hi,

1st scan is slow, i know that.
But shouldn't the 2nd be faster (because cached).
To me it's almost the same speed.
It's like the cache isn't used...

(I'm scanning a network share, is this the problem?)

Cheers,
DocMAX

[Admin]

It must be faster, how many files do you have ?
Similarity always checks file attributes of files to check changes, maybe it slows.
Does counter of cached files increase after second scan ?

[DocMAX]

Hi thanks for the answer.
Cache counter doesn't increase.
File attribs are the same.

It starts quickly till around 1.000 (of 12.000) and slows down more and more. In the end it's like without cache, CPU to the max.

[Admin]

What CPU do you have ? Is Task Manager shows almost 100% cpu loading ?

[DocMAX]

I have an Intel Core 2 Duo 2.4 Ghz. But i have the same issue on an iCore 5 CPU.
Around 500 it's getting slower.
Similarity has 100% cpu usage right from the start.

[Admin]

Please send us logs from "%appdata%\Similarity\logs" folder to our email.

[DocMAX]

There is not so much in it...

2013-05-26 19:09:13   
2013-05-26 19:09:13   Similarity version: 64-bit (x64) 1.8.4 build 1694 (13.05.2013)
2013-05-26 19:09:13   Windows version: 64-bit (x64) 6.1 SP1 (build 7601)
2013-05-26 19:09:13   Config init...
2013-05-26 19:09:13   Config init complete
2013-05-26 19:09:13   Module init...
2013-05-26 19:09:13   Module init complete
2013-05-26 19:09:13   Translator init...
2013-05-26 19:09:13   Translator init complete
2013-05-26 19:09:13   Start program
2013-05-26 19:09:13   State init...
2013-05-26 19:09:13   State init complete
2013-05-26 19:09:13   Adding worker threads...
2013-05-26 19:09:13   Adding worker threads complete
2013-05-26 19:09:13   Thread started
2013-05-26 19:09:13   Thread started
2013-05-26 19:09:13   Menu creating...
2013-05-26 19:09:13   Menu creating complete
2013-05-26 19:09:13   Toolbar creating...
2013-05-26 19:09:13   Toolbar creating complete
2013-05-26 19:09:13   Statusbar creating...
2013-05-26 19:09:13   Statusbar creating complete
2013-05-26 19:09:13   Tabs creating(Folders)...
2013-05-26 19:09:14   Tabs creating(AudioResults)...
2013-05-26 19:09:14   Tabs creating(ImageResults)...
2013-05-26 19:09:14   Tabs creating(AudioAnalysis)...
2013-05-26 19:09:14   Tabs creating complete
2013-05-26 19:09:14   Adding tab pages complete
2013-05-26 19:09:14   Loading cache...
2013-05-26 19:09:16   Loading cache complete
2013-05-26 19:09:16   Loading ignores...
2013-05-26 19:09:16   Loading ignores complete
2013-05-26 19:09:16   Taskbar init complete
2013-05-26 19:09:16   Mainframe initialized...
2013-05-26 21:09:45   Thread stopped
2013-05-26 21:09:45   Thread stopped
2013-05-26 21:09:45   Shutdown program

[DocMAX]

anybody else? i really can replicate this problem on multiple pc's.
some versions before (i think without image comparison) it worked fine.

[DocMAX]

any news??? scanning takes whole days if > 3000!!!

[Admin]

1. In you logs only 2 threads started do you have 2 cores CPU ?
2. What size of images/audio do you have (largest) ?
3. Try disable unncessary decoders (DirectShow, MMF, QuickTime, ACM)
4. You can disable fully image comparing algorithm just uncheck image content based algorithm and experemental in Options
4. Just scanned 13000 photo (up to 6Mb JPG) images, took 40 minutes on i7-2600

[Springdream]

The slowing down also applies to me. First some 1000 are quite slow, then at let's say 10000 its 1 song per second.
I guess similarity has to compare each file against each other  meaning time effort is proportional to faculity of n (N!)?

I also have the feeling that the cache does not speed up too much.

ALSO another question on the cache:
1) does it slow down if there are too many item in the cache that do not exist anymore? AND
2) if so maybe a function like "delete cache items that do not exist anymore" would be nice
3) I have the feeling that the cache is relating to absolute paths of the files. Everytime there is a changed folder name all files below that folder are lost in the cache. Maybe the cache could refer to some other file properties like a hash, checksum ect. or name, size and date?

[Admin]

Not N! it's N^2. N! very big values.
1) no, cache used as binary tree search time always log(N), it's only take time to load data and take RAM.
2) reasonable suggestion, we think about "purge cache" button.
3) yes cache uses absolutly pathes, because how we can found file equivalence ? For calculation hash or checksum we need disk access that takes time.

[Igreenzevb]

Hi,
I do not understand why clicking a tag in review mode is so slow, while clicking the same tag in the folder view shows me the actions in under a second.

e.g. I have one tag that has 19 items. In folder view 1 second in review mode it takes 14 seconds.

What I get to see is exactly the same data but why does it take such a different time to load the list?

Regards
Manu

[Admin]

Hi,
I do not understand why clicking a tag in review mode is so slow, while clicking the same tag in the folder view shows me the actions in under a second.

e.g. I have one tag that has 19 items. In folder view 1 second in review mode it takes 14 seconds.

What I get to see is exactly the same data but why does it take such a different time to load the list?

Regards
Manu

What do you mean by review mode and folder view (first tab with folders doesn't show tags) ?

[Springdream]

I guess you have simply too less RAM.
Consider 8GB+ for your collection...