Log to db

[r][m]

Could a delimiter character (maybe -) be added to the log similar to the example below?
And leave out the @.

(origional) 7/12/08 6:57:21 PM user@192.168.1.35:2433 Fully uploaded -  14.01 KB @ 0 KB/s
(changed) 7/12/08 - 6:57:21 PM - user - 192.168.1.35:2433 - Fully uploaded -  14.01 KB @ 0 KB/s

This would allow linking the log text file to a database for sorting things like most downloaded
file, or anything else needed. This info might eventually be available as a html file via load macro
for a stats page for remote viewing.
Apache style log is pretty lame. Aparrently it wont list uploads or downloads properly,
and does show some info of (from my point of view) questionable value.
r][m         

HFS=no 3rd party !

[rejetto]

at the moment, customization of log lines is supported through the apache-style log.
i can add more symbols easily if needed.

the change you are asking for the log is easily achieved through a script, python, php, anything.

here is for you a one-line solution in php, to run it you need to install php.exe (not the apache php module)

Code: [Select]

<? foreach (file('myfile.log') as $l) echo preg_replace('_(\S+) (.+ (AM|PM)) (([^@]+)@)?(\S+) (.+)_', '$1 - $2 - $5 - $6 - $7', $l) ?>

If you don't want to install php, you can try Sed, a command line utility, but it requires some syntax adjusting.

Code: [Select]

s/\([^ ]\+\) \(.\+ \(AM\|PM\)\) \(\([^@]\+\)@\)\?\([^ ]\+\) \(.\+\)/\1 - \2 - \5 - \6 - \7/g
I spent 30+ minutes on this solution, because i had to learn sed syntax. It seems quite shitty, only little better than my template macros.
For your convenience, i will attach sed (the program itself)

[r][m]

So it can't be added?
Is there a configuration for apache log to show what files were
uploaded and downloaded?
So far I haven't found them listed anywhere.
Anyway, on second look, character " - " wouldn't work.
 "|" would be best.

Sorry, didn't realiaze it would be a big problem to add a
seperator character.
Thanks for the reply and suggestions.
r][m

[rejetto]

what's the problem with the "sed" solution?

the apache log problem is that it expects one line for one request. (am i wrong?)
uploading 3 files together is done through one request, so i could put all the files on the same line.
would this fit your needs?

[r][m]

Don't think I can cope with the sed solution.
PHP would likely be more useful, but I can't see installing it for just
this one thing.
Putting all uploaded files in one line might work if there was blank space
between each.
Apache log appears to be intended to track a web site, not a file server.
I find no info on how to set up the apache log to show uploads at all,
not just in HFS, but in general, and apache log for downloads leaves much
to be desired. I'd want to change the time format also.

The regular log format is far more useful. It does show all files uploaded
even several at once. Thats the one I'd prefer to use. If a "|" can't be
used to define sections, what about putting a blank space between them?
r][m

[r][m]

Actually, I could make this work well enough to be of some use if there
were just a blank space added before and after the @ like
"user @ 192.168.1.35:1443"
Is that possible?

Or...does anyone know of a log analizer prog that will read the regular
log? So far I've found none?
r][m

[rejetto]

Don't think I can cope with the sed solution.

put the sed script in a file "script".
then run: sed -f script hfs_apache_log_file > new_format_log_file

Putting all uploaded files in one line might work if there was blank space
between each.

i could add a symbol that will expand to such list, so that you include this symbol in your apache-style log

I find no info on how to set up the apache log to show uploads at all, not just in HFS, but in general, and apache log for downloads leaves much to be desired.

indeed it doesn't, it would be a feature i would add (as %symbol)

I'd want to change the time format also.

it's likely you can do it by changing the sed script, but you'll have to learn regular expressions (if you don't know them yet)

The regular log format is far more useful. It does show all files uploaded even several at once. Thats the one I'd prefer to use. If a "|" can't be used to define sections, what about putting a blank space between them?

at the moment the "regular" log cannot be customized, but you can build a solution similar to the one i suggested above.
if you don't have necessary knowledge you can ask a programmer friend.

[Rarst]

This would allow linking the log text file to a database for sorting things like most downloaded
file, or anything else needed.

I think it's problem for database, not HFS. What if rejetto adds deliimters and next days someone says that he needs to feed it to different database and they must be removed for that?

What db and how are you trying to use exactly? In my opinion regexp solutions from above are pretty fine (except I'd use languages I am more faimiliar with but that's going to be few lines of code anyway).

[r][m]

i could add a symbol that will expand to such list, so that you include this symbol in your apache-style log

If you will do this I'd be happy to give it a try.
Rarst
Thanks for your reply.
The problem is with hfs log. Most db prog I've used like M$ Access will link to
external data. Most I'm familar with use pretty much the same criteria to delimit
or seperate the sections for sorting such as "blank space" or various characters
that the developer may choose to use out of necessity. This criteria is then applied
to all data. Is not selectable by block.
This link is dynamic, so that any time a query or report is opened what you
see is up to date as of that moment.  I have this type of setup running for other
types of programs that output logs or other data.
Trying to use something like sed to convert to a useable format would likely be static.
Calling sed by script from the db is possible, but in my experience, once the db link is
activated, the file linked to is locked. Once the link closes the log up dates on next hit.
Obviously the db can't be left open, but then it doesn't need to be.
Addition of a space or character to delimit shouldn't impact those who log hfs by other
criteria like user or date.
 So far I've found only one log analyzer program that will read hfs's apache log and
its static. On a scale of 1 to 10 its usefullness compared to what I'm trying to do
is a 3 !
r][m

[Rarst]

i could add a symbol that will expand to such list, so that you include this symbol in your apache-style log

If you will do this I'd be happy to give it a try.
Rarst
Thanks for your reply.
The problem is with hfs log. Most db prog I've used like M$ Access will link to
external data. Most I'm familar with use pretty much the same criteria to delimit
or seperate the sections for sorting such as "blank space" or various characters
that the developer may choose to use out of necessity. This criteria is then applied
to all data. Is not selectable by block.

Reading by separator only seems to be very limited functionality. I am not experienced in databases but most of programming languages have regular expressions support that allows highly customizable actions with text.

Addition of a space or character to delimit shouldn't impact those who log hfs by other
criteria like user or date.

Spaces won't work because they are possible in URLs and it is going to ruin reading by separators. Other characters are going to add clutter.

[r][m]

Reading by separator only seems to be very limited functionality. I am not experienced in databases but most of programming languages have regular expressions support that allows highly customizable actions with text.

Reading by delimiter character or even a blank works very well. I'm already doing this with data
output from various log files and data out put like down time from machines, and yes your are right,
the db has customizeable actions with text, and even via programing language, but like hfs
aparrently,theres only so much you can do.

Spaces won't work because they are possible in URLs and it is going to ruin reading by separators. Other characters are going to add clutter.

Spaces would work, because if you'll look, they already exist in the hfs log in places and I'm
presently using that method. One problem area is the @ that joins ip and user in the regular log.
I'm sure you must know that in the apache style log you can format it with delimiter now, as
%h | %u | %t | \"%r\" | %>s | %b | %{user-agent}i | "%{Referer}i -> %U
Regards
r][m

[rejetto]

in next build you will be able to specify %z to get extra information about uploaded files.
What exactly are your needs about it?
i'm going to print filenames separated by pipes, but only those who have been successfully uploaded.
we can parameterize this by using the %{parameters}z standard syntax.

[Mars]

rejetto , just a question about uploaded files

actually , if we upload a very big file and if the file can not be stored, we must wait the end of upload before know this.

Is there a possiblity with the header to compare the upload-filename with hfs-internal mask, and stop the current upload for this file only, if the filename is forbiden, then ask to the next upload file sended by the browser?

[r][m]

Continued testing indicates apache log works best.
What would work best, is log output that indicates real downloads and
uploads, different from the pages. Kinda like the "fully uploaded/downloaded"
in the regular log.
Would it be possible to add

Code: [Select]

%{format}t and maybe failed logins and bans to apache?

Also, could someone check the ban to regular log? It doesn't seem to be working here.

[rejetto]

Is there a possiblity with the header to compare the upload-filename with hfs-internal mask, and stop the current upload for this file only, if the filename is forbiden, then ask to the next upload file sended by the browser?

not with the current method, based on a single http request.
in the future i can use something else, based on javascript or flash.