Problem Statement :
There is php, apache , mysql based application, the stack is hosted on Linux.
This is a single server application, where PHP and apache are on an ubuntu-based Linux server and MySQL is on another server.
let’s suppose the application becomes unresponsive after a certain load. Now you want to RCA and optimize the bottleneck of the problem, the following will provide you a complete guide to debug LOAD issue in LAMP stack
To figure out the load issue, first, we need to figure out the dependent items of the stack, from where the problem may arise. We divide this into two parts, hardware, and software
- Linux server hosting apache2
- Linux server hosting MySQL
- Apache/Nginx webserver
- PHP processing [running as module/fpm/fastcgi]
- Mysql server
Considering the situation where the site is becoming unresponsive,
the very first thing that is coming to consideration is, Hardware limitation
Q1:- are the resources enough to handle so many requests?
To get the answer to this question, we analyze the system state. considering that we are on Linux OS [ubuntu], however, most *nix are same.
— check for web server machine
- If possible, put monitoring on CPU utilization. You can use htop/top command to check the process and load
- Put monitoring on ram utilization. htop/top command can be used to check the log
- In case of the machine become unresponsive, check for System logs [syslogs], mostly under var/log/syslog location
- tail -f /var/log/syslog
htop/top is only helpful if you monitor during load as they shows current state of the system otherwise put the monitoring by using any tool.
There are various tool like cloud instances also provide resource monitoring as Cloudwatch in AWS, or, one can setup open source tool like nagios, zabbix, opennms or there are handy utilitites like, glance, atop, etc https://www.cyberciti.biz/tips/top-linux-monitoring-tools.html
There are 2 main factors to analyze
Apart from these 2, one can also check for another two factors. As most of the cloud systems provide a good bandwidth allocation and that can also be checked from Network monitoring and Most systems are equipped with SSD, so these are not the major factory but can play a vital role if your application highly depends on that, like you are using firebase data to read and write which need higher I/O or delivering large media file
— check for the database server machine
If the database server is deployed on another machine, then, we need to follow the same steps as we covered in the earlier section to figure if the database server is enough or not.
In case, you are using AWS RDS service for the database, you can use AWS cloudwatch or RDS monitoring tool with CPU matrix or connection matrix to figure out the root cause.
Most of the cloud provider provides these type of system/instance/machine monitoring tool, where we check the hardware component dependency and bottleneck areas.
Now there may be two situations,
- Either server resources are NOT enough,
- Or server resources are available even at peak load.
Server resources are NOT enough :
* Scale Up / Scale Out the resources [Vertical Scaling / Horizontal Scaling]
* Validate the request are legtimate, means there is no bocus request flooding your server to keep busy
If hardware is the bottleneck, we need to upgrade the resource either by following Vertical Scaling or Horizontal Scaling
Server resources are enough :
If we are at this point, where resources are available like, servers are healthy and under the threshold limit, it means that hardware is not the bottleneck, we should check for the software level bottleneck.
Q2: if the server resources are enough, then what might be the root cause of the unresponsive web app?
In a typical PHP website, the software stack includes.
- Monitor and optimize Webserver [apache/nginx]
- Database server [mysql/MariaDB/postgre]
- PHP Monitoring and Optimization