Getting "no space left on device" error on Ubuntu Linux

I was getting "no space left on device error" when I tried to restart my php5-fpm process. The server was a Ubuntu Linux on Amazon AWS (small instance). df command said I had plenty of space, so there must be another problem:

$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/xvda1       8256952  5401088   2436436  69% /
udev              838180       12    838168   1% /dev
tmpfs             338516      228    338288   1% /run
none                5120        0      5120   0% /run/lock
none              846284        0    846284   0% /run/shm
/dev/xvdb      153899044  6357108 139724312   5% /mnt

df command can also show the number of free nodes and I could be run out of free nodes:

$ df -i
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda1      524288  255729  0   100% /
udev            209545     390  209155    1% /dev
tmpfs           211571     282  211289    1% /run
none            211571       4  211567    1% /run/lock
none            211571       1  211570    1% /run/shm
/dev/xvdb      9773056   22975 9750081    1% /mnt

Voila! No free nodes left on my disk (the first entry). Let's check which directory is the culprit:

for i in /*; do echo $i; find $i |wc -l; done

The above script shows the number of files on / . We are searching for an unusual number of files. In my case it was /var. So issued the command for that directory:

for i in /var/*; do echo $i; find $i |wc -l; done

I got /var/lib . Then another run for /var/lib gave /var/lib/php5 that has more than 250,000 files! They were /var/lib/php5/sess_... files. I tried rm command:

rm -f /var/lib/php5/sess_*

I got "Argument list too long" error. find command with -delete option comes handy in this case as it does not deal with a long argument list:

find /var/lib/php5/sess_* -type f -delete

Here we go. If you check the number of free nodes now, you will see the difference.