Geeklog Forums

I'll give the background first:

* whatismyipaddress.com
* 500+ concurrent users during the day
* Running on a dedicated box
* Local MySQL installation
* Mac OS X Server 10.5
* Multi-Processor (almost never more than 20% CPU usage)
* 6GB of RAM (almost never more than 2GB used)
* Mirrored Drive RAID
* Net connection usually at 50% utilization during peak traffic

What I am seeing is apache becoming non-responsive, memory usage climbs to 100%, and the drives start thrashing (swapping memory out assumedly). Once it's in this mode I've got three options: 1) change DNS entries to direct traffic to a different machine (different site), 2) power cycle the machine and usually have to repair the gl_sessions table, or 3) drop the network connection and wait 30 minutes+ for the machine to get back to normal.

Needing some help in trying to figure out what direction to start looking. I've tried to lower the MySQL queries by removing polls from the blocks and removing the forum centerblock from the homepage. Does not seem to help. Upped the MySQL querycache size to 500MB this morning. Any suggestion on other things to look at?

I wish I had your problem!

Do you think maybe it is one your IP tools instead of Geeklog?

Tom

---

One of the Geeklog Core Developers.

Yeah, it's a nice problem to have.

Honestly, I don't know where the problem arises so I'm going after this on multiple fronts (of which geeklog is one). If I knew that someone else was exceeding the traffic I have (without problems) then I'd focus more of my attention elsewhere.

I previously had a problem with one of the IP tools' db table reaching 1.5 million records and selects and inserts would suffer. That problem has been resolved.

Another thing that I cannot do is batch delete users. If I try to delete more than maybe ten people at a time the site hangs. That makes me think more along the lines of MySQL than geeklog itself.

I've spend tweaked some of the Apache settings today so I'll see if that helps.

As a side note, page creation times remain low up to the time where the site become unresponsive.

I have a similar problem, I run a popular Thai motorcycle website (motorcycle.in.th)

+2000 users a day
* Running on a shared hosting (but as I'm also the hosting it is only shared with fasthosting.in.th)
* Linux Hosting (RedHat) and lots of memory...

The website sometimes just doesn't react on clicks and response time is in the worst moments down to a minute...

I installed Bad Behavior2 to filter out the bad weed, but with Bad Behavior2 saying that it blocked 24,000 hits (I was wondering is that not also 24,000 SQL activities, or does Bad Behavior write it to a log file?)

I have to tell that it is based on glFusion, but as glFusion has the same background I follow this posting with lost of interest.

The first thing I'd do is to actually check the webserver's log and see where that traffic is coming from. You may be targetted by a spam bot or script kiddies. If so, I'd block IP addresses or typical requests at the webserver level, so that PHP and MySQL (and Geeklog) don't even come into play.

As I said in a recent discussion, Bad Behavior will help a little bit, but as far as actual server load goes, it's kicking in too late to make a difference.

bye, Dirk

Most of the hits that Bad Behavior2 is flagging as bad are trackbacks. I have the idea to switch of the trackback option... but then I'm sure they will target other parts of our website...

How long does it take before this problem appears after a server reboot (do you get at least a few days out of it)? I would think it would be a PHP bug or possibly a MySQL issue because of the memory usuage issues (leak).

Did this just start happening or has this always happened under these loads.

The deleting of users is a weird problem. Maybe concentrate on this since it is easy to reproduce. Any log information on this?

---

One of the Geeklog Core Developers.

It's a new issue the corresponds with a new advertising campaign that has bumped traffic. Issue seems to occur when traffic is peaking to historically new highs. It happened again this morning.

I'm trying to go through the logs but they're 500MB a day so it takes some time to be able to identify issues.

I've made some more tweaks to some code that is used for proxy server detection to see if that is the issue.

Argh, just happened again. This time I was able to get a mysql status before it system got unusable.

Before

After (restarted both Apache and MySQL)

When it's misbehaving and I run the SHOW PROCESSLIST I see a whole bunch of processes that all look the same with varying "time"



State is "Sending data".

That SQL request is from SEC_getUserGroups where it collects the list of groups the current user is in. Do you have a lot of groups? Or many groups that inherit from other groups?

bye, Dirk

I've made no group changes since installation.

admin block says (18 ) next to groups but when I click there are only 8 groups.

There are 30k users.

(could this be tied in with problems does batch deletes of users?)

Doubled all MySQL caches. Switched from using GeoIPCity DB to GeoIPCountry where applicable. Seems to have made a world of difference. Seeing record concurrent users and page creation time (when not using the GeoIPCity db) down to 0.07 seconds.

Looks like I need to switch over to the Apache2 mod_geoip to be able to push more performance out of it. Now to figure out how to do that on Mac OS X 10.5. What little reading I've found seems to indicate there may be issues there. (due to 64bit OS).

Thanks to everyone for their suggestions.

How about turning on the slow query log and seeing if there's just a few specific queries that are slow.