Geeklog Forums

Trying to resolve some slow down and lock up issues I was looking at my server logs and noticed what looked like a user trying to what looked like submit events for every day of every month of every year. However doing it very fast like it was a BOT. Upon doing a Whois on the IP I found it belonged to Google. Has anyone noticed this. My guess is that somehow it is spidering everyday. Here are few items of the log file.
This can't be good for my bandwidth. I don't really want to stop Google from indexing the site, but I sure don't want it to do every day of every month of every year until it gives up or locks up the server. Any ideas folks.

if it is a bot, have a look at this article by Dirk.

Thanks, good info, I'll admit I failed to do a search on this one.

Maybe it would help to put the Calendar in it's own directory in future versions of Geeklog.

It also appears that google is ignoring the robots.txt file.

Why should that help? Search engine spiders follow links and you would have a link to the calendar directory then ...


It may take some hours before it recognizes that it has changed, but if it is syntactically correct, Googlebot will obey.
bye, Dirk

Ok Dirk thanks for the info.

I was thinking that if it was in a directory then you could just exclude that directory. I'm not sure what the googlebot is doing except filling up my log files.

Simply exclude the calendar.php script ...


I've seen this before - sometimes it just starts to index the calendar like wild, then it ignores it completely again for weeks. As long as it keeps the bot happy, I don't mind

bye, Dirk

I had to complety exclude MSNbot, It ate up gigabites of bandwidth and from what I've read, MS isn't even using it.