Last night at 12:31am I got a notice saying Apache stopped running on this server. Me in my Texas Cedar Fever house had taken two Benadryl and was out cold. I have a blog so lets be honest its not mission critical, but it’d be nice to have things auto recover so you don’t have to wake up and start apache etc.
So below going to outline a simple solution to this issue that I just implemented to prevent this from happening again.
Welcome, monit - a package used to monitor your services and restart the service on failure. I am only going to cover mysqld and httpd because that is all that is running on this server.
Install monit
yum -y install monit
Configure monit
Open /etc/monit.conf in your favorite text editor and add
set httpd port 2812 and
allow localhost # allow localhost
allow 1.2.3.4 # allow your ip
allow admin:monit # basic auth login
This will allow a web administration panel to be available on port 2812 and only be accessed by the IP’s whitelisted in the configuration.
Add monitoring for apache
Open /etc/monit.d/httpd and add
check process httpd with pidfile /var/run/httpd/httpd.pid
start program = "/etc/init.d/httpd start" with timeout 60 seconds
stop program = "/etc/init.d/httpd stop"
Things to check when using the above configuration: The pid file is where you say it is and the init script is as well.
Add monitoring for mysql
Open /etc/monit.d/mysql and add
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"
Start the monit service
chkconfig monit on
/etc/init.d/monit start
Now, moving forward whenever the pid file is cleaned up the monit service will know and restart the service. You can view your monit admin page at http://yoursite.com:2812 and login with the creds you setup in the first part of this.