rejetto forum

Software => HFS ~ HTTP File Server => Topic started by: maxmouse on May 01, 2009, 07:19:58 PM

Title: HFS and Large Hard Drives
Post by: maxmouse on May 01, 2009, 07:19:58 PM
I've been trying to share a folder on a 750 GB external USB 2.0 drive that contains about 500 GB worth of files in various folders. The issue is when a user tries to access the folder using HFS, the server freezes trying to retrieve the directory listing. Sometimes it will eventually retrieve it but it can literally take 10 minutes or something ridiculous to get the data. It doesn't seem to lag as bad over FTP for example. I was wondering if there was anything I could do to the drive or HFS to make it load faster. For example, would it be possible to index the contents of the drive, so HFS wouldn't have to perform a full directory listing every time a user tries to access it.

Thanks so much for your help.
Title: Re: HFS and Large Hard Drives
Post by: Kremlin on May 01, 2009, 10:57:41 PM
Maybe it's a NTFS/FAT32 problem? Anyway rejetto might have a better clue.
Title: Re: HFS and Large Hard Drives
Post by: rejetto on May 03, 2009, 11:13:39 PM
what action does take 10 minutes?
entering a folder to see the list?
if yes, how many files are in this folder?
how many files are listed after this long waiting?

you can try disabling: menu > virtual file system > use system icons
Title: Re: HFS and Large Hard Drives
Post by: maxmouse on May 06, 2009, 01:59:33 AM
Thanks for your reply. Yeah, I have a directory on my external hard drive that I've added to my server as a real folder. The root directory contains about 4,000 individual files and something like 7,000 subdirectories. When I click on the folder from within HFS the browser shows that it's loading something and when I check HFS the status indicates it's, "thinking." If I wait long enough, the browser ends up freezing. After consuming something like 100 megabytes of RAM, the connection is dropped. Other times, HFS becomes unresponsive. I've gotten it to list a couple times, but I can't even get it to do it at all at the moment. I'm accessing the server from my local machine. The only thing I can figure is it's just overwhelmed with the number of files and folders. But, it seems like if I can access it from windows explorer at a reasonable speed the server should be able to mimic that, but I don't really understand enough about how it works.

I just added the folder as a virtual folder instead and the result seems to be the same. I tried disabling the option you suggested and toggled some of the others to no avail. It says it added something like 122,000 files. It seems to freeze HFS on access. Maybe it's just too many files at once, but it seems like you could index the folder somehow or get it to list faster. Windows explorer seems to be able to access it within 15-20 seconds.

Thanks for your help.
Title: Re: HFS and Large Hard Drives
Post by: rejetto on May 06, 2009, 10:04:46 AM
did you try real folder + icons disabled ?
i'm sorry you are hitting this problem. HFS was not designed to handle such situations, and sadly with so much things to do i must focus on what most people need.
but HFS is an open source project, and would any other programmer want to deal with it...

keeping so much items in a single folder does not scale (http://en.wikipedia.org/wiki/Scalability) well.
organizing so much stuff i always consider to not get over 2000 items/folder.
Title: Re: HFS and Large Hard Drives
Post by: Kremlin on May 06, 2009, 12:32:13 PM
Actually I don't think it's from the number os folders/files since I use roughly 4000+ also and have no such behaviour from the server (2.5TB Hard Drive + 3TB External Hard Drive). When you say the server freezes and reaches 100mb of RAM does the processor also pike up to 100% (50% in case of duo, 25% in case of quad)?

In order to confirm it's a scale problem try only loading 2000 folders in your HFS instead of the 7000 and see how the server reacts, afterwards gradually add 500 more and test until it starts freezing up again. If so then rejetto's thought is right.
Title: Re: HFS and Large Hard Drives
Post by: maxmouse on May 06, 2009, 06:48:57 PM
So I opened the folder using firefox and after about 10 minutes it finally loaded. HFS has the following stats for it:

This folder contains:
[5666] Folder(s), [4028] File(s)
Total size: [14.52 GB]

It looks fine when it's loaded it's just annoying because if you navigate into a subfolder and go back to the root folder it has to reload for another 10 minutes all over again. The browser just waits and HFS uses about 48% (with a duo) of the CPU and around 127 MB of ram and when it's finally loaded usages goes back to normal.

I tried disabling the icons but it doesn't seem to make a difference, so I'm stumped.

Maybe it's an issue with the hard drive itself. It's NTFS so I wouldn't think it would be a problem but I've used HFS with 300 GB hard drives before and never encountered this problem. It's a western digital mybook usb 2.0, but maybe there's something wrong with the file system.
Title: Re: HFS and Large Hard Drives
Post by: bacter on May 06, 2009, 08:19:36 PM
maxmouse

I propose you to organize your files and folders in a more browsable way, or perhaps even allow only the search function.

It may take 10 minutes to build the page.. but have you measured the time users are waiting to load such a huge page with more than 9500 items? And even worse: How many time will they spend scrolling through the page to see where the file of their interest is ! ;)

It's only a proposition  :)
Title: Re: HFS and Large Hard Drives
Post by: rejetto on May 06, 2009, 11:55:13 PM
how long does it take if you access the file list instead?
http://myserver/myfolder/~files.lst
Title: Re: HFS and Large Hard Drives
Post by: maxmouse on May 08, 2009, 01:49:08 PM
Rejetto I tried that and it only took about 2 minutes maybe a little less. Bacter makes a good point though even when you do get it to load it's too many folders on one page. I think I'm just going to break it up by letters of the alphabet and see if that doesn't help. I'll let you know.

Okay so I broke my folders up into 6 smaller groupings alphabetically, and it made all the difference in the world. A folder only takes 12 seconds to load as opposed to 10 minutes for the whole thing. So, this is the way to go I think and as Bacter pointed out it's too hard to look at that many folders on one page anyways. Thanks for all the help guys. Good work on HFS rejetto works wonders.
Title: Re: HFS and Large Hard Drives
Post by: MarkV on May 11, 2009, 01:58:32 PM
Multi paged lists would help in this case. Like breaking up large listings in pages of 1000 items each and retrieve only the current page.

I know it's possible with the scripting language, but maybe it's time to put it in HFS default template... I'd hard-code it to allow max 2000 items per page.
Title: Re: HFS and Large Hard Drives
Post by: rejetto on May 11, 2009, 02:47:20 PM
yes, paging is a quite important feature.