Download Accelerator Prevention

[raybob]

When HFS is set to prevent Download Accelerators, it responds with a 503 - Overload.  Can you make a way to have these events not show in the log?  At the same time I do want to see other not-served requests, such as 403s and 404s.  Reason I ask is because ~95% of my hits in the past 2 days were refused because they were duplicates from download managers.  Obviously this makes my log very difficult to read.

Thanks,
Ray

[r][m]

Does your template have a [overload] section in error-page section?
If so would [overload|no log] work?

Actually, the error pages don't appear to log here at all, now I'm
trying figure out why  

[raybob]

Nope... lines like this keep appearing over and over again....

10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Requested GET /.(username)/shared/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----file.extension
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Not served: 503 - Overload
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Requested GET /.(username)/shared/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----file.extension
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Not served: 503 - Overload
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Requested GET /.(username)/shared/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----file.extension
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Not served: 503 - Overload
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Requested GET /.(username)/shared/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----file.extension
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Not served: 503 - Overload
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Requested GET /.(username)/shared/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----file.extension
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Not served: 503 - Overload
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Requested GET /.(username)/shared/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----file.extension
10/6/2011 7:34:22 PM XX.XX.XX.XX:38696 Not served: 503 - Overload

[r][m]

What tpl are you using?
Or do you mean [overload|no log] doesn't work?

[raybob]

Using [overload|no log] has not effect.

One thing that would be nice and would solve this problem as well as save a lot of CPU would be to simply disconnect w/o reply when multiple connections like that are opened up.... I find that my GUI only freezes up when some download manager opens up like 50 connections per second and HFS refuses 49 of them but still responds to all...

I'm sure there's some way to do it with macros (most of my other problems have been solving with scripting)  


EDIT:

Just a quick thought (I'll test this later) perhaps something along the lines of this would work.... (I'm sure there's a much better way to do it)

When a new download opens, HFS saves a file saying that that person is downloading that file.  Upon completion it deletes it.  If the file already exists (is being downloaded) then they get disconnected.

Code:

[download]
{.if|{.exists|%ip%-%item-name%.txt.}|{:{.disconnect.}:}|
{:{.save|%ip%-%item-name%.txt|.}:}.}

[download completed]
{.delete|%ip%-%item-name%.txt.}

One problem I thought of though is that if the user cuts off the download half way through the file would not be deleted and they would never be able to access their download again...  I'm sure I could somehow use the [disconnected] section to fix this somehow...

[r][m]

OK, I see. The logging is just a nuisance, the real problem are the connections.
I guess the Prevent leaching (download accelerators) feature in Limits doesn't help either?

Myself I wouldn't spend a lot of time developing a script, I'd just notify the user
to stop, or I'd ban their IP with a disconnect. I'd tell them they do have the option of using Archive.

[raybob]

I can't very well block them in any way since it's not just DL managers that do it but also apparently IE9 and IE8 and possibly Chrome too...


EDIT:

I'm gonna give this a test-run:

Code:

[download]
{.if|{.exists|C:\documents and settings\admin\desktop\Current Downloads\%ip%-%item-name%.txt.}|{:{.disconnect.}:}|
{:{.save|C:\documents and settings\admin\desktop\Current Downloads\%ip%-%item-name%.txt|.}:}.}

[download completed]
{.delete|C:\documents and settings\admin\desktop\Current Downloads\%ip%-%item-name%.txt.}

[disconnected]
{.if|{.exists|C:\documents and settings\admin\desktop\Current Downloads\%ip%-%item-name%.txt.}|{:{.delete|C:\documents and settings\admin\desktop\Current Downloads\%ip%-%item-name%.txt.}:}.}

[raybob]

Hmmm well that script does it job but it apparently doesn't lessen HFS' CPU load or help responsiveness at all nor does it remove the GET lines from the log, only the 503 reponses.

This might need to be coded into the software to work right.

BTW, based on my log and logging the user-agents, it seems as if IE8 and IE9 are the only ones that automatically try to open up a ton of download connections at once, every other browser does it normally.  I think it might only be that it does it when loading files embedded into web-pages though (like JS, CSS, etc.)

Just a side note, today when I walked over to my computer I looked and saw that there were 55 download icons in the system tray and that HFS would not respond.... I went into Firefox and refreshed my page and it said it couldn't connect.  Sound like an overload crash?

Perhaps I should limit the amount of hot-linking each user gets (my ratio of downloads to user accounts is well in excess of 300:1)

EDIT:
It seems as if the extreme connection acceleration in IE only happens when trying to download a certain SWF files...

EDIT 2:
I think the problem might just be that the user embedded the SWF file into their page multiple times... I'm gonna go investigate their code.


I guess my main problem here is not how annoying the log is but that since HFS takes the time to respond to the repetitive connections and execute the macros under [request] for them rather than just disconnecting without reply, it takes up a lot of CPU time and slows down the server, a lot.

What tpl are you using?

Not that it's relevant but I actually use several different templates based on section... the Global Template is >200KB in size  
VFS, Events, all Templates, TXT Files in use by HFS, etc., amounts to over 700KB 

[rejetto]

instead of using the files, documentation says
Example: {.current downloads|user=%user%|file=this.} will tell you how many connections this user is using to download this same file.

see if you can use this in [request] that comes first
otherwise in [download]