Node.js is all fun and games until...

As many of you know, I have been doing a ton of development. Some for my work at Cisco and some for fun. I like using PM2 to manage my node processes, logging, restarts, etc. As my integrations and bots have started getting more use I became fascinated/addicted with watching the logs for errors.

Here is a snapshot of the logs that were stealing my attention

This was all great until last week, I realized it was really distracting me from my real job! I decided it was time to take these my hobby bots/apps to the next level.

I set out to find a solution that would alert me if a PM2 process reset/stopped, PM2 died, my server went offline as well as capture stack traces if/when a node process threw an exception. Keymetrics.io Has an awesome option for this. If you are running node apps in production and are making money with them, this would be an excellent option. Since my apps are mostly running for free, I kept looking. I already have Zabbix set up to monitor the infrastructure for Douglas Automotive and was excited when I found a module was already written for Zabbix to monitor PM2!

Zabbix PM2 graph displaying node process memory usage

This was fairly painless to configure. I will post some details below. Zabbix has me covered for the process monitoring, server load, resets, etc. But I still needed a solution for stack traces.

There are lots of options out there for capturing and analyzing logs in the cloud. Many of them come with a hefty monthly price tag. After trying a couple options. I decided to use sentry.io, setting up an account and integrating into node was very simple.

Sample stack trace captured by sentry.io

Here is an example of a stack trace from some tests I was doing.

With these two solutions in place, I hope to maintain my sanity and continue to write more code that other people find useful! I am always looking for better ways to do things. If you have a logging/monitoring solution that is better, let me know!

Zabbix Config Detail:

You can find the Zabbix PM2 module here. I am running Zabbix on an EC2 Amazon Linux AMI. The example startup files were written for Debian where Amazon Linux is more like CentOS/RHEL. After lots of trial and error I was able to make the following script work.

Zabbix-pm2 AMI startup

#!/bin/sh

# Install this file as /etc/init.d/pm2-zabbix

### BEGIN INIT INFO
# Provides:        pm2-zabbix
# Required-Start:  $network $remote_fs $syslog
# Required-Stop:   $network $remote_fs $syslog
# Default-Start:   2 3 4 5
# Default-Stop:
# Short-Description: Start PM2 monitor for Zabbix
### END INIT INFO

. /etc/rc.d/init.d/functions

NAME="pm2-zabbix"
DAEMON=/usr/local/bin/pm2-zabbix
PIDFILE=/var/run/pm2-zabbix/pm2-zabbix.pid

PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Actual start/stop logic
[ -x "$DAEMON" ] || exit 0

start() {
        echo -n "Starting PM2 Zabbix: "
     daemon --user "ec2-user" --pidfile $PIDFILE $DAEMON "--monitor 2>&1 &"
     RETVAL=$?
     sleep 5
     PID=`ps aux|grep -vE 'grep|runuser|bash'|grep -w "$DAEMON"|awk '{print $2}'`
        if [ -z "$PID" ]; then
            printf "[ \e[31mFAIL\033[0m ]\n"
        else
            echo $PID > $PIDFILE
            printf "[ \e[32mOK\033[0m ]\n"
        fi
}

stop() {
        echo -n "Shutting down PM2 Zabbix: "
        echo
        killproc -p $PIDFILE pm2-zabbix
        echo
        rm -f /var/lock/subsys/pm2-zabbix
        return 0
}

case $1 in
     start)
          start
          ;;
     stop)
            stop
          ;;
     restart|force-reload)
          $0 stop && sleep 2 && $0 start
            ;;
     try-restart)
          if $0 status >/dev/null; then
               $0 restart
          else
               exit 0
          fi
          ;;
     reload)
          exit 3
          ;;
     status)
          status -p "$PIDFILE" "$DAEMON" "$NAME" && exit 0 || exit $?
          ;;
     *)
          echo "Usage: $0 {start|stop|restart|try-restart|force-reload|status}"
          exit 2
          ;;
esac

Node.js is all fun and games until...

Tags