Graphite on Nginx/uwsgi - Debian Wheezy

Graphite it an opensource application used to track metrics on a time series database. You can send metrics using almost any programming language and its way easier than other good options that may result a little more difficult to understand tools like OpenTSDB and RRDtool.

There are lots of tutorials and scripts that show how to install it,  most of them use apache to deploy the web interface. The graphite web app is written in Django and as such it's really easy to deploy using nginx.

Enough talking, let's get to work:

Install the required packages:

# Install the required packages for graphite
aptitude install -y python2.6 python-pip python-cairo python-django \
  python-django-tagging python-twisted python-memcache python-pysqlite2 \
  python-simplejson

# Install nginx and the required cgi modules
aptitude install -y nginx uwsgi-plugin-python uwsgi

# install graphite via PIP
pip install whisper carbon graphite-web

Create the configuration files:

cd /opt/graphite/conf/
# Create a directory with all configurations 
# It will allow us to keep this dir clean and use the original
# files as templates
mkdir examples; mv *.example examples

# create a copy of the required files
cp examples/storage-schemas.conf.example storage-schemas.conf
cp examples/storage-aggregation.conf.example storage-aggregation.conf
cp examples/carbon.conf.example carbon.conf
cp examples/graphite.wsgi.example wsgi.py

Create webapp configuration file

cp /opt/graphite/webapp/graphite/{local_settings.py.example,local_settings.py}

Uncomment the parameter "TIME_ZONE" on "/opt/graphite/webapp/graphite/local_settings.py" and replace with the result of "cat /etc/timezone". In my case the resulting edit looks like:

TIME_ZONE = 'America/Caracas'

Otherwise the webapp will assume the default time zone for Django apps: 'America/Chicago'.

Create nginx configuration file:
#/etc/nginx/sites-available/graphite
server {
  listen 8080;
  charset utf-8;
  access_log /var/log/nginx/graphite.access.log;
  error_log /var/log/nginx/graphite.error.log;

  location / {
  include uwsgi_params;
  uwsgi_pass 127.0.0.1:3031;
  }
}

Enable the virtualhost:

ln -s /etc/nginx/sites-available/graphite /etc/nginx/sites-enabled/

Create the uwsgi configuration file:

#/etc/uwsgi/apps-available/graphite.ini
[uwsgi]
processes = 2
socket = 127.0.0.1:3031
gid = www-data
uid = www-data
chdir = /opt/graphite/conf
module = wsgi:application

Final uwsgi application configurations:

# enable the uwsgi app 
ln -s /etc/uwsgi/apps-available/graphite.ini /etc/uwsgi/apps-enabled/

# create the local database for the webapp
python /opt/graphite/webapp/graphite/manage.py syncdb

# Allow access to  www-data to the webapp and storage  directories
chown -R www-data:www-data /opt/graphite/webapp/  /opt/graphite/storage/

Start carbon, the uwsgi and nginx daemons:

/opt/graphite/bin/carbon-cache.py start
/etc/init.d/uwsgi restart
/etc/init.d/nginx restart

Check http://yourIPAddress:8080 and you should see the graphite web interface.

If you have any issues, remember to check the following log files for possible errors:

/var/log/nginx/graphite.error.log
/var/log/uwsgi/app/graphite.log

As a last resource you can set the Django app into debug mode by setting "Debug = True" on "/opt/graphite/webapp/graphite/local_settings.py".It should provide enough information to give you a hint on what may be causing the problem.

A primer on runit using Debian Wheezy - process supervision

Runit is a program used to make process supervision and as an alternative for startup scripts.
For the purpose of our example, we're are not going to use runit as a compliment to init.d.

Why should I use it?

It's easy to use.
It's multiplatform.
It allows to to avoid having to fight with pid files.
Helps you to avoid reinventing the wheel.

Installing runit

aptitude install runit

The installer will add the following two lines at the end of your /etc/inittab:

SV:123456:respawn:/usr/sbin/runsvdir-start
SV:123456:respawn:/sbin/runsvdir-start

After installing, you'll have to options to start using runit: reboot o reread your /etc/inittab file by running:

init q

From now on, every time we start your OS it will run the runsvdir daemon. This process will be responsible for monitoring the /etc/service/ directory where we will configure our services. By each service running on this directory, the runsvdir daemon will spawn a new runsv process.

Configuring our first service.

Create the service directory

mkdir -p /etc/sv/test

Create the service script and the script to log everything that happens with the service:

touch /etc/sv/test/run /etc/sv/test/log/run

Give running persmission. This is very important. Otherwise the services won't run:

chmod u+x /etc/sv/test/run

Our really simple 'service' that writes to stdout and stderr every and die after 5 seconds:

#!/bin/sh
exec 2>&1
exec bash - <

Note: We need to use exec in order to make our command replace the current shell without creating a new process. Content of /etc/sv/test/log/run:

#!/bin/sh

exec chpst -u nobody svlogd -ttt /var/log/test/

We have our services configured but not under supervision. We need to create a symbolic link to the /etc/service/ directory:

pushd /etc/service/
ls -s ../sv/test/
popd

Show time

Everything its setup. Let's do some test:

# start the service
root@beta:/etc/service# sv start test
ok: run: test: (pid 26967) 0s

# lets check our service during 5 seconds to see how it behaves
root@beta:/etc/service# sv status test
run: test: (pid 26972) 0s; run: log: (pid 2643) 25259s
root@beta:/etc/service# sv status test
run: test: (pid 26972) 1s; run: log: (pid 2643) 25260s
root@beta:/etc/service# sv status test
run: test: (pid 26972) 2s; run: log: (pid 2643) 25261s
root@beta:/etc/service# sv status test
run: test: (pid 26972) 3s; run: log: (pid 2643) 25262s
root@beta:/etc/service# sv status test
run: test: (pid 26972) 4s; run: log: (pid 2643) 25263s
root@beta:/etc/service# sv status test
run: test: (pid 26972) 5s; run: log: (pid 2643) 25264s
root@beta:/etc/service# sv status test
run: test: (pid 26984) 0s; run: log: (pid 2643) 25264s
# the service got restarted automatically!!
root@beta:/etc/service# sv status test
down: test: 5s, normally up; run: log: (pid 2643) 25276s

# now we can stop out service
root@beta:/etc/service# sv stop test
ok: down: test: 1s, normally up

This is it for this first post on runit.

Simulating a filesystem with not space left

The situation:

This may sound curious, but yes I had to simulate a filesystem with no space left in order to reproduce a failure that caused problems with the uploads on an PHP application. The upload tmp dir got full and the application stopped uploading files.

The application was running on a VM and there was no LVM. This meant that I would had to add a disk to the VM, restart de VM, create a new partition... It was to many steps for a simple test. Besides that, I wouldn't use the partition again after the test.

The Solution:

Create a filesystem on a file, use it as a loopback device and fill the filesystem. I thought loopback devices were useful just for mounting ISOs and disk images, but no, they became really useful on this situation:
That's it. I updated the upload_tmp_dir PHP parameter to:
upload_tmp_dir = /mnt/php-full-device

The developer corrected the bug and I didn't have add another disk to the VM (I know, I'm lazy).

Bonus Track:

Here I leave some additional commands that might be useful:
As you may appreciate this is easy to implement and doesn't require any server restarts.