Jump to content


Load and stress test

Load test Stress test

18 replies to this topic

#1 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 29 June 2017 - 08:55 AM

Hi. For us LiveZilla is a important and mission critical piece of software, cause we use it to interact with our main customers and we have an average of 200 chats per day, with peaks of 3000 chats per day.
In peaks value day we experienced some leaks of performance, so we would like knowing if someone has done some load and/or stress tests on LiveZilla.
In the past we have 6.1.1.2 version, now we are on 7.0.5.0 in a testing environment.
Thanks and regards

#2 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 29 June 2017 - 10:00 AM

Hi,

We are trying to improve performance in almost any update.

Can you provide some more details on the performance issues you see?

- Client or server performance
- Area (Chat, Tickets, Visitor Monitoring, Archive etc)
- Amount of elements to deal with
- Specific instructions on how to reproduce the crash / slowdown etc.

Thanks for your help.

#3 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 12 July 2017 - 08:23 AM

Hi. We had several performance issue:
  • we use only chat and archive area;
  • on the client, we need to clean often the browser cache. The operator has slow down in writing chats;
  • on the server, we experience memory problems (OOM), as a workaround we added RAM and things go slightly better (but we have now 16GB RAM, as a panzer!!!);
  • we have 30-70 req/sec.
We are not able to reproduce the problem, cause it starst when the traffic is huge, so I was asking you if you have had load/stress test and if you share the test configuration/method.
Thanks and regards

#4 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 12 July 2017 - 09:51 AM

Thanks for your feedback!

Btw. please don't miss our performance optimization guideline.

#5 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 13 July 2017 - 10:17 AM

Hi. We made all the changes mentioned in the page and we were on a dedicated hosting, but the only definitely gaining solutions was adding RAM.

The main problem is the slowing down of operators' client; their client seems to freeeze when they write and/or they attempt to interact with LiveZilla web GUI.

Since we have many many chats, we limited visibility to only group chat of the operator, but the emprovement was very very slight.

Cleaning the browser cache helps, but is not the definitive solution of the problem.

As I said for us LiveZilla is a important and mission critical piece of software: there is something more to do? You have any suggestion?
Thanks and regards

#6 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 13 July 2017 - 10:35 AM

In order to reproduce your problems, we require more specific information. Terms like "traffic is huge" and "many many chats" are not helpful.

Quote

- Amount of elements to deal with (operators, overall chats, chats per operators, amount of knowledge base entries etc)
- Specific instructions on how to reproduce the crash / slowdown etc.

The more details you provide, the more likely that we can solve the problem.

Thanks.

#7 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 13 July 2017 - 01:11 PM

Hi. You are absolutely right, here more informations:
  • we have about 100 operators, 10/15 concurrent;
  • slowdown seems to be only for operatos (customers did not complain about this). When load (customers who wants to chat) grows up see what they write seconds after writing;
  • number of chats per day is growing up faster, with peaks of 3000 chats par day and 400 started in a hour;
  • our archive has more than 50K chats, the db is over 1GB.
Thanks and regards

#8 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 13 July 2017 - 02:19 PM

Thanks.

1.) 15 operators need to be online - check. No slowdown so far.
2.) 50K chats in archive - check. No slowdown so far.
3.) How many chats need to be running (overall + per operator)?
4.) How big is your knowledge base?
5.) ????

#9 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 14 July 2017 - 09:12 AM

We limit 5 chats concurrent per operator.
No limit about chats running (customers arrive and go in queue).
We do not use knowledge base.
Thanks and regards

#10 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 14 July 2017 - 09:15 AM

Thanks!. Can you estimate the total number of active chats when the slowdown occurs? It would be really helpful to have this number.

Alternatively, we can offer to troubleshoot this directly on your server if you can provide access.

#11 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 17 July 2017 - 10:51 AM

The worst slowdown occurred having 630 initiated chats in a hour, operators only took 252 chats (mainly for slowdown problems, it was impossible typing and working well).
You can check also this condition?

#12 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 17 July 2017 - 11:54 AM

Can you please estimate the total number of active chats when the slowdown occurs?

#13 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 17 July 2017 - 12:19 PM

Nevermind, we are able to reproduce the problem. We will try to improve performance if possible.

#14 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 17 July 2017 - 01:49 PM

Fix will be included in 7.0.6.0. Thanks for your help!

#15 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 18 July 2017 - 02:09 PM

Thanks to you, Patrick!!!
I will install 7.0.6.0, now we are not in peak moment, so I cannot test... but I trust you as a LiveZilla Dev!!!
Just a curiosity: which was the problem?

#16 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 19 July 2017 - 07:47 AM

Hi,

We found two problems:

1.) There was no pruning. All previous chats will be hold in memory which is okay for small amount of chats but a problem if you have 400 chats per hour. The history array is now limited to 50 entries. Older entries above this level will be pruned automatically.
2.) Data transmissions between client and server was very inefficient. We were able to reduce the amount of data to transmit and process significantly.

#17 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 22 August 2017 - 10:10 AM

Hi.
Tested today 7.0.6.0 in production environment.
On a queue of 2 operators with 50 chats in queue we experienced 5/10 minutes of slowdown while distributing chats (operators have a limit of 5 concurrent chats, they had 1 chats per operator and the system would not assign other chats).
The slowdown in distribution resolved spontaneously.
Thanks and regards

#18 motormadness

    Advanced Member

  • Members
  • PipPipPip
  • 36 posts

Posted 29 August 2017 - 08:45 AM

Hi.
We are having huge load on MySQL (500% CPU, 250 contemporaneous disk writings).
We see in slowlog many queries like
mysql> SELECT AVG(`duration`) AS `waitingtime` FROM `lz_chat_archive`
WHERE `chat_type`=1 AND `duration`>30 AND `duration`<3600;
+-------------+
| waitingtime |
+-------------+
| 753.5564 |
+-------------+
1 row in set (2.28 sec)
It is possible to do something to reduce MySQL activity?
Thanks and regards

#19 Patrick Keil

    Administrator

  • Administrators
  • 3187 posts
  • LocationSingen, Germany

Posted 08 September 2017 - 12:18 PM

Thanks for bringing this to our attention.

I forwarded this to the dev in charge.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users