A little knowledge is a dangerous thing

When someone posts a comment to my blog, up to six hundred gifted monkeys put down their work on NaNoWriMoFoMo to scan the comment for bad html and verify the correct security code. If it passes muster, they store the comment in a relational database. Next, smoke is wafted and mirrors are carefully placed as the site updates itself.

The monkeys have learned that old content is preferable to no content. They try to create a temporary page. If that’s successful, they’ll move it onto the old copy. If not, they’ll send me a terse comment.

Two days ago, something went wrong and the Apache server on my shared hosting system was running as a different effective userid. In layperson terms, temporary pages couldn’t be created. The monkeys became very agitated. In their chattering frenzy, they forgot to let me know that the site was broken. A neighbor heard cacophony and wrote me to complain about the noise let me know something was broken.

I logged a trouble ticket with my ISP. Ten minutes later, I became impatient and made the web directory world-writable. The site worked again and the monkeys went back to work.

The Apache problem was fixed very early yesterday morning. I needed to do some editing of a previous feature and was chagrined that the web server was broken in a new and exciting way: every script was throwing exceptions when invoked from a browser. I logged a ticket, and tried to diagnose by looking at the log files. They were useless:

[Wed Jul 28 12:34:56 2004] [error] [client xxx.yyy.zzz.pdq] Premature end of script headers: /time/to/clean/the/guns.cgi

The monkeys refused to write more content until this was fixed. “Artistic license!” they said. “Sunny day to play outside?” I retorted.

I logged another trouble ticket. A few hours later, I logged a second one with some new information. Someone called me yesterday evening, thinking this was related to the first problem and hoping for a quick close. (When I worked at Oracle, we used to do the same thing. And call east coast customers after 4pm PST.) When it wasn’t that, he tried to help. Expecting that he’d want to blame my scripts, I had created a “hello, world” that should work anywhere. He gave it a good try, but couldn’t figure it out and left it in the queue.

By this morning, I was a bit frantic. I logged yet another trouble ticket, indicating my frantic(k)ness then, taking a cue from the monkeys, went biking. Someone called me back and knew that the problem was the permissions on the scripts directory. One Unix command later, the site was copacetic.

If I had left things alone, my site would have been down only one day instead of two.