CentOS: No space left on device

Encountering the error message “No space left on device” hits most CentOS system administrators by surprise. There can be a myriad of reasons for this error message. In this little HowTo we examine a specific case of “No space left on device”, we learn how to check if this cae applies and we learn about a (bad!) quick and dirty solution, as well as a professional (optimal) solution afterwards.

Reason 1: disk full

The most obvious reason is a full disk. Plain check your disk usage with

In case your disk is full… you gotta do something, like extending your partition in question.

Reason 2: insufficient inodes

Another issue can be a lack of inodes. Plain check your inodes:

In case you ran out of inodes, you got several options.

1. First of all some service might have create way too much files somewhere (e.g. tmp-files). Find it and fix it.
2. If all is ok, but you simply lack inodes, you can just increase the partition and reassign inodes in the case of lvm.
3. Other options are to remount the partition with inode64 or…
4. you have neither option and plain have to migrate to another disk or file system. Take care you have sufficient inodes: create a new partition, switch to a more fitting filesystem (eg. XFS) or
5. install a new system and do it right directly during installation.

Reason 3: insufficient inotify watches

Crashplan, fail2ban, systemd, NetworkManager and a lot more services use the inotify-kernel-module or put load on the number of inotify watches. Yet many other services may need the inotify-kernel-module at peak times too. While this is a nifty way to handle file system events, the default value of 8192 inotify watches usually does not cut it, once a service heavy on inotify-watches is started.

Insufficient inotify watches: Common symptoms

Insufficient inotify watches don’t show up in the system log in standard installations (INFO). Instead you have to be wary of the symptoms:

• Starting stopping services:
  Error: No space left on device
• On some command-line commands you get a:
  No space left on device
• log-files can not be automatically watched (esspecially obvious in the case of fail2ban) or services can’t write any longer to log-files
• Using tail -f on an “unused” file, leads to unexpected results:
  tail -f ~/.bash_profile
• incomplete log-files
• …

Bottom line: the system is fully operational (for the most part), but watching files doesn’t work any longer for excessive files.

Solution

You have to increase the standard inotify watches amount (which is normally 8192).

Short term (break-all) solution

This solution has issues, since it eats a lot of memory by plain setting inotify to a high value. So you increase the number of inotify watches like this:

In case this works, you can add this to sysctl.conf:

Or alternatively:

Correct solution

What did we just do in the rpevious chapter? We plain increased system limits without knowing what we just did. So now let us make this the “good way”. First of all determine the number of watches you need. For that look at the services, which need those watches and count the amount of files they are approximately watching. So let us consider the case of Crashplan watching the /var/www directory. As a matter of fact we can simply count the files, which Crashplan might be watching:

So 103862 files might be watched, when Crashplan is fully active.

Now let us look at the watches currently in use:

And we also like to know, which of those watches are in use by Crashplan currently:

So now we are good to go:

• My system as such needs: 98364 – 97998 = 366 watches.
• Crashplan right now uses 97998 watches, but might need 103862 watches.
• Knowing my site, I assume that there will be a maximum of 150000 files under /var/www within the next 2 years.

To be safe for two years, I would need 150000 + 366 = 150366 watches.

Knowing computer systems, we prefer numbers made of the 2^n pattern. As a result this would be 2^18 = 262144 watches. So now it is time to set it up:

Or alternatively:

Discussion

Why actually care and not plain set inotify-watches to maximum? For every watched file the system stores 540 Bytes on 32-bit systems and 1080 bytes on 64-bit systems. Indeed this memory is not reserved, but instead only allocated when needed. Then again none knows what kind of software bug or even malicious software may become active on our system. Professionals know that their resources are limited and allocate those in a smart way.

So if I had set it to 1048576, this makes:

1080 Bytes * 1048576 = 1132462080 Bytes
= 1132462.08 KByte 
= 1132.46208 MB 
= 1.13246208 GB

We plain increased a limit by almost 1.2 GB when using the quick and dirty solution. And we did not even know exactly why and for what.

In the professional solution it is:

1080 Bytes * 262144 = 283115520 Bytes
= 283115,52 KByte
= 283,11552 MB
= 0.28311552 GB

Less than 0.3 GB and less than 1/3 of the “short term” solution.

Bottom line

Don’t go out and wildly up system-limits. As soon as you do it in several cases, these add up and you may regret it.