Geeklog Forums

Hi we are currently investigating a memory leak on our server which we have narrowed down to MySQL. We run 8 instances of Geeklog on the server and 1 instance of phpAdsNew which is linked to Adlink in Germany. We have rebooted a couple of times and within hours the memory usage climbs to 99% plus the virtual memory. None of the GL's have extra plugins apart from what comes out of the box.

Our server is an IBM with Xeon 3.02 1 Gb RAM and dual scsi running RAID. The Geeklogs running are a mixture of 1.3.8.x and 1.3.9.x all with latest patches as appropriate. MySQL is 4.0.20 and Apache 2.0.5x. OS is RedHat Enterprise.

We are not saying it is anything to do GL but has anyone had any experience of memory leaks.

- Tess -

I have a red hat (FEDORA CORE 2) server running 6 geeklog sites and there are no memory leak issues. My previous server (pentium 200 with 64 mb of memory!) ran 9 months once between boots, running http (6 geeklog sites) and mail servers.

Have you looked at the number of server instances running, etc?

If you are using an RPM version of Apache / PHP, you may want to compile your own. Usually RPM distros are "best fit" and "most generic" pre-compiles and whilie they work most of the time, in production they tend to have issues.

As for trouble shooting your mem leak, I would first take a look at what services you have running, turn off any services and modules not needed. Finally, check out your php.ini and make sure its the latest version, along with apache and mysql.

If you still are having problems after that, run a free unix utility called mem-check. You can google it. It will check the physical sectors of your hardwares memory, and report if any are bad, and possiblity block access to them.

If that still doesnt work, disable php-ads, see what happens.
If that still doesnt work, you may need to recompile your own webserver as a needed file may be broken

We built our own Apache, MySQL and PHP rather than using the RPM's. We have switched usages of MySQL off in bits and pieces including moving the phpadsnew to another rack.

When we reboot we have been starting 1 of the instances of GL running for 30 minutes, rebooting then starting a different instance of GL, switching of everything else no needed including Sendmail.

In each of our tests we have found the one instance builds up memory usage quite fast, the same in fact for each instance tested. When rebooting and starting 2 or three of the instances at the same time the memory build up is that much faster.

IBM who donated the unit are coming to check the hardware part and switch out the memory tomorrow.

At the moment it does look as if it is our install of GL that is the problem 3 of the site add 200 new items a day and receive about 10,000 visitors a day, the other sites are a 'little' quieter.

The server is managed by Verio-ATT and is only two months old, both of the Xeon's have been changed and senior engineers have examined the software installation .... we are all baffled.

I appreciate the responces from all - I do ot think it is a flaw with GL let me make that clear I feel it is some interaction with GL our software and hardware.

Regards

- Marites -

Marites,

it could simple be normal cache memory grabbing all it can. How are you "reporting" that memory is nearly full?
What's swap use like? - if this is nearly untouched, you don't have a problem.

Have a look at tools like orca, vmstat and friends

You mention the problem is apparently with MySQL - on what basis is that claim made? I'd be loking at hard at your mysql config. You may have simply allowed it to grab as much memory as it can.


Cheers!


- Steve

Steve the swap is also being eaten

We have been watching with ps and various other tools - I will hold tight until the memory is swapped out tomorrow and see if that makes any difference.

Many thanks

- Marites -

We have isolated the problem to 2 sites running GL 1.3.8.x we are not sure if this is an issue relating to the versions of GL/MySQL/PHP in combination or individual problem within the combination.

We have turned off these two instances prior to upgrading to 1.3.9.x, which the rest of the site use with no severe problems hopefully that will cure the problem.

For some reason on our server the 1.3.8 versions were using 3 times as many resources as the 1.3.9.

I have no idea where to try and track down this issue and pinpoint exactly what is/was causing it but if we find anything more I will report back.

- Marites -

It might have something to do with the fact we made some major improvements to how Geeklog queries the Database in 1.3.9 (by reducing DB calls and making the remaining ones more efficient). More of these (though probably not as significant) will be coming in 1.3.10.

You're memory problems sound like a MySQL caching/buffer issue. Check your config options looking specifically at places that deal with memmory. Check out mysql config options for more information.

Good Luck,
Vinny

We have had the server checked inside out by an IBM, RedHat and network engineer.

MySQL checks out fine we have the correct configs in my.cnf applicable for the memory we have.

It does sadly seem as if there is something in our hardware / applications / Geeklog that is not compatible. Everyone except our group is pointing the finger at GL

To reiterate we are running on an IBM x306 server with 3.02 GHz processor, normally 1 Mb RAM, (currently trying more), with RedHat Enterprise 3 Server, latest php, mysql.

I should say the memory has been replaced, along with the CPU - It is not so bad with 1.3.9 as it was with 1.3.8 but it is bad enough to cause concern. This is what mysql is using at this minute ...

1821 mysql 15 0 15540 15M 1944 S 0.4 1.5 0:14 0 mysqld

Just don't know where else to look.

Marites

I suggest adding more memory, as its cheap. 1 gig of mem sounds like a lot, but in a application server roll, thats nothing (I have a gig of mem in my laptop)
Really, if you have extensive use on that box, up the memory.

Second, if your IBM box is SMP, You may want to check their site for updates, as i have had experience with IBM 3XX SMP boxes and redhat choking, for simular issues.

Finally, if you have custom coded your 1.3.8 plugins. Check php. there was a bug with my_sql_connect, or simular statement , last year. Had everything to do with leaving connections open. (thus chewing up memory)

Hi thanks

We have exclusive and etensive use of the box. We have actually asked the server managers earlier today to upgrade the RAM to 2 Gb. It does take a maximum of 4 so still have leeway yet.

We are finding the sql_connect is effecting 1.3.8 and 1.3.9 but strangely it seems to have been closed in 1.3.10. Chancing our arm we have put 1.3.10 rc2 on a high usage site and the difference was noticable immediately with the amount the same site was using on 1.3.9.

So it looks like the problem may go away when 1.3.10 is released.

Many thanks

Marites