When You’re The Server Wrangler

Though this blog belongs to me, it doesn’t live with me. www.geofffox.com resides on a computer somewhere in the suburbs west of Philadelphia. I rent a tiny sliver of the server which is shared with countless others.

I pay the hosting company to handle the operating system and play traffic cop if one of the tenants uses more than his fair share of the resources. I handle the rest, including the code that runs my site.

About two weeks ago the server stopped allowing me to upload new files.

Uh oh.

I poked around and found that though I have unlimited storage space (I’m using around 10 Gb of storage) there is a limit to the number of individual files. Sneaky.

I am allowed 262,144 files. That should be plenty. Unfortunately a quick check showed I was at the limit.

Why?

In a case like this there are a few go-to geeks I lean on. Not this time. This was going to be my challenge!

My server is headless, meaning there is no pretty graphical screen like you see on your computer. My only means of communication is through a terminal program.

Think Matthew Broderick in War Games!

This is where Google is indispensable as a tech support tool. A half dozen keywords hinting at what I wanted went in, a site with exactly the code I needed came out!

du ~/*|sort -n

Those characters commanded my server to count the number of files in each of its directories, then list them starting with the directory containing the most files.

Within seconds I found the problem. A caching plug-in, a tiny program to make this website faster for you, had gone nuts! It had spawned hundreds of thousands of files and never cleaned up after itself.

Two minutes later after an uninstall/reinstall I was 184,783 files short of a full load. Problem solved… at least for now.

8 thoughts on “When You’re The Server Wrangler”

  1. Unix,Linux, and BSD have some pretty awesome commands, and then the fact you can pipe the output of commands to the input of commands is cool too.

    For those unfamiliar I will break down the command.

    du ~/*|sort -n

    “du” is the command used to list the sizes of all files in a directory. It is followed by “~/*”. “~” means the home directory, or in this case the directory in which geoff’s user account is located. In this particular example “~/*” means all the files in the Geoff’s home directory. If you ran *Nix at home it would be all the files in your user’s home directory

    The output of du command is “piped” or sent to the input of another program, using “|” between the commands, in this case the program is sort.

    the sort command is used to sort items into a list. In this case, the list is the file list created by du. the “sort -n” command sorts the data from the list in numerical order.

    1. Well done, Michael.

      The commands available let me manipulate the website at a very basic level. These are powerful little code snippets often written on-the-fly.

      When I’m working on a website this obscure shell language is constantly in use. It allows me to easily move or copy files or, as was the case today, delete well over 150,000 of them in an instant!

      You’ve got to use the force wisely. A few web dev friends think I’m crazy to do it this way. Maybe.

      My copy of O’Reilly’s “Linux Pocket Guide” is never far away.

      1. Yeah, Sometimes it’s just the most effective way to solve a problem. I agree that you definitely have to be careful with every command. For example, piping data to a file if you accidentally typed a ‘>’ (overwrite) instead of a ‘>>’ (append).

        For the issue you had on wordpress, is this a custom plugin, or an issue with stock wordpress configuration?

        1. Michael,

          This is probably larger than most amateur websites. I’ve got 5,700 entries, 22,000 comments and 14,000 tags. It was the.w3tc caching plugin. Add on.

  2. I’ll look into it a bit more and let you know if I find anything. I found that other users are getting similar problems as well, but aren’t finding a solution other than stopping usage of the plugin.

    Another option may be to switch to another caching plugin. I saw some benchmarks that show that Hyper Cache outperforms W3-TC (source: http://www.tutorial9.net/tutorials/web-tutorials/wordpress-caching-whats-the-best-caching-plugin/), and one of the features is “autoclean system to reduce the disk usage” (source: http://wordpress.org/extend/plugins/hyper-cache/)

Leave a Reply

Your email address will not be published. Required fields are marked *