Jump to content


livezilla crashing php5-fpm with high CPU

php-fpm crash cpu nginx

1 reply to this topic

#1 imvu-e.com

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 16 May 2017 - 01:53 AM

I am running php5-fpm with nginx on a debian jessie server. The server has 16GB ram, RAID 0 SSDs, and highly optimized configs for MySQL and php5-fpm.

However, after some time (30 minutes to an hour) the server completely locks up and CPU raises to 100%. The number of active processes of php5-fpm maxes out to well over 100 (normally averages ~20).
There are normally 300 users on the site, and typically around 5 concurrent chats going on between ~2 operators. I have visitor monitoring off.

This system has been running without problems for over 2 years (without livezilla), and I've consistently got it to lock up when I enable the new widget v2 API chat on all my pages. Right now in order to avoid lock ups I have to only show the widget on my contact page (vs all pages).

I have done most of the tweaks suggested in the FAQ on speeding it up, and have tried setting the cache to both APC and shmop to no avail.

Typically my php5-fpm process list will be loaded with a bunch of server.php all blocking because of MySQL from what I can tell. A spam of requests like
/contact/livezilla/server.php?rqst=track&b=YzI4NjYxZWM2MQ__&pc=NQ__&i=ZDA1YzQ3ZDc3Ng__&di=dcca48101505dd86b703689a604fe3c4&ovlv=djI_&ovltwo=MQ__&ovlc=MQ__&esc=IzlGQzNFNw__&epc=IzU5NzBCMA__&ovlts=MA__&ovlapo=MQ__&eg=c3VwcG9ydA__&tc=MQ__&lmr=T05NMDE_&lp=c3lz&intid=SU1WVSBFbXBvcml1bQ__

I have mysql slow queries logging in order to detect whats going on and it considers almost all SELECT * queries done by livezilla are marked because they don't use an index. I highly doubt these queries would be the cause but I found it very odd that 4-5 similar queries going by every second or so-- especially when I have cache turned on (and i verified it was working via php var_dump). I noticed SELECT * FROM lz_config every 2-3 times a second, and this table barely changes. Seems like caching could be improved?

Just for the sake of completeness I am attaching some of settings for my.cnf and www.conf

(Wont let me attach so will link)

my.cnf: https://gist.github....86541cb3720eeab
www.conf: https://gist.github....c4d09d8a9ddc56e

I have attached my montioring system screenshot so you can see the problem that occurs in relation to system resources (mostly cpu). The gaps in CPU is me manually restarting the system to fix the issue. Restarting php5-fpm or mysql doesnt fix it alone. I believe the clients are still puesdo-connected so the requests keep coming in

System resources: http://i.imgur.com/KXMFoL2.png

I am not sure livezilla can do anything because its so specific to my use case but I just wanted to let you guys be aware of a potential problem with the high amount of MySQL requests you are doing and php5-fpm. There are so many factors at play that it could be a setting in php5-fpm or mysql that needs tweaked, but I am confident based on my testing that the catalyst in livezilla.

#2 Patrick Keil

    Administrator

  • Administrators
  • 3139 posts
  • LocationSingen, Germany

Posted 16 May 2017 - 07:31 AM

Hi,

Thanks for your throughts. We never worked with php5-fpm and I can't tell you what's causing the problems you have.

What I can tell you is that 2 ops, 300 visitors and VM turned off shouldn't be much of a challenge for your machine.

You said that there's a spam of tracking requests. If VM is turnded off, there should be one request per visitor per page load. Each chat and operators add 1 request every 4 seconds.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users