Jump to content











Photo
- - - - -

Javascript external logging! Why?


  • Please log in to reply
7 replies to this topic

#1 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 14 October 2008 - 02:12 PM

I'm sure there must be some reason, but I don't understand why boot-land.net is wasting its internal resources with javascript click-throughs to an external logger on every page/visitor access.

External counters like www.statcounter.com may be a decent solution for people who cannot access and analyse their own raw server logs, but, for those who can, it's just a wasteful duplication of that on-board logging process at best. In most cases, it doesn't even come close to providing the analytical detail that is possible without any duplication of the on-board server logging that occurs automatically anyhow.

As just one example, take a look at the information provided by AWStats analysis of the raw server logs at http://virtech.org/logs/ . Note, in particular, its detailed accounting of robots/spiders activity which seems to be something of an obsession here and which uses nearly 25% of the same database that is used by these forums. (Triplication?!)

For anyone who may not like AWStats for some reasom, there are plenty of other on-board analysis options (e.g., Webalyzer) that are nearly as good, but mostly harder to set up initially. In any case, any of them have wasteful javascript click-throughs to external logging duplication beat hands down IMO.

#2 Brito

Brito

    Platinum Member

  • .script developer
  • 10616 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 14 October 2008 - 02:38 PM

Hi Arvy.

Statcounter is used for keeping all stats in one place, it's been my favorite stat interpretation tool mostly because I don't need to visit each site to keep track on their evolution and so far the "Error 500" messages have been quite less frequent so it wouldn't likely justify dropping it's use in favor of awstats at this moment as our past history from the last years is stored statcounter and I'd like to keep things together rather than scattered around.

I'm sure there must be some reason, but I don't understand why boot-land.net is wasting its internal resources with javascript click-throughs to an external logger on every page/visitor access.


The main reason is to provide more information about people who visit the site, like the topic that was posted recently - http://www.boot-land...?showtopic=5949

We can later use this information to account the site popularity which is important to predict our future growth rate.

Let's look on other optimizations that are likely more important in terms of performance.

On the topic I mentioned above you've also said:

Actually, the logging methodology used on this site is one of its heaviest burdens. Just for the forums dB alone, a single logging table (spider_logs) uses nearly 25 percent (42 of 174 Megabytes) of the total MySQL database resources.


I don't know where this can be disabled, would you care to do this modification?


:confused1:

#3 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 14 October 2008 - 03:15 PM

Statcounter is used for keeping all stats in one place, ...

Dunno about your total "system spread", but I use AWStats for tracking multiple subdomains, and FTP access as well, on my server as per the selections list at http://virtech.org/logs/select.php. (Ignore the Webalyzer items. They're only there because my hosting service runs that log analyzer themselves.)

The main reason is to provide more information about people who visit the site, ...

I see nothing in the statcounter.com breakdown that isn't available in at least as much detail from AWStats or any other decent on-board log analyser. But you've obviously become "attached" to that external javascript logging methodology and that, of course, is entirely up to you.

I don't know where this can be disabled, would you care to do this modification?

No, but I'll have a look and let you know what I'm able to find out about it, if anything.

__
EDIT (Follow-up):

You'll find the IPB admin panel "on/off switch" under Tools and Settings > System Settings > Search Engine Spiders. There is also an option to prune the existing accumulated dB data under Tools and Settings > System Settings > Log Pruning. Or you can use the Boot Land PHP MySQL Manager that I left on-site for you to manage your entire database any way you wish.

The IPB help info reads as follows: "Enabling spider bot visits [logging] can greatly increase the amount of database storage space used. If this is enabled, it is recommended to enable Log Pruning for spider visits."

#4 Brito

Brito

    Platinum Member

  • .script developer
  • 10616 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 14 October 2008 - 04:50 PM

Hi again.

I see nothing in the statcounter.com breakdown that isn't available in at least as much detail from AWStats or any other decent on-board log analyser. But you've obviously become "attached" to that external javascript logging methodology and that, of course, is entirely up to you.

You're right, guess I've grow too attached to statcounter and maybe I'm getting a bit lazy too.. :confused1:

In either case, it's no excuse that my stat preferences should disturb the optimal forum performance, I'll leave statcounter as it is for the moment due to other priorities in terms of efficiency (need to remove mkportal from frontpage first) and then see if awstats can provide a quick and global view of the stats that I can work with and replace the current javascript call.

-----------------

The option to prune logs every 30 days was already selected but I wasn't aware of the search engine spider optimizations.

This was a really cool tip, thank you.

It even said:

If you're under heavy attack, switch this off as it may put a little extra load on the SQL server as it attempts to insert new data rows.


Excellent! :cheers:

#5 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 14 October 2008 - 05:13 PM

... I'll leave statcounter as it is for the moment due to other priorities in terms of efficiency (need to remove mkportal from frontpage first) and then see if awstats can provide a quick and global view of the stats that I can work with and replace the current javascript call.


Sounds logical. The initial AWStats set-up can be a bit tricky, especially for multi-domain and FTP analyses, and even more especially if you want both normal user viewing and admin/update access options. If you run into any problems, let me know. I'll help if I can.

#6 Brito

Brito

    Platinum Member

  • .script developer
  • 10616 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 14 October 2008 - 08:00 PM

Thank you! :cheers:

The site does seems to be behaving much better now!

:confused1:

#7 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 14 October 2008 - 08:11 PM

Thank you! :confused1:

The site does seems to be behaving much better now!

:cheers:

I read all the above posts. But as I'm not a server admin, I did not understand a lot.
Can you explain for an expert-dummy, what the issue is?

thanks

Peter

#8 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 14 October 2008 - 09:44 PM

Very simply, every process takes time and utilises resources. The process of logging site visits, crawlers/spiders, page hits, etc., etc. is not an exception to that rule. Logically, therefore, it's desirable to avoid duplicating (triplicating?) any process wherever possible. In this case, since the server itself is continually maintaining its own set of accessible "on-board" logs, maintenance of a second set elsewhere would not normally be considered necessary or desirable.

Even if one ignores all impacts on the web server's own performance and resources, where client side scripting (e.g., javascript) is involved, there is also some added "cost" to each browser access on each page visited.

Fundamentally, it can be thought of as basic "economics" or operational efficiency. Leaving aside the functional particulars, it's not really any more complicated than that.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users