8 Comments

Summary:

Working on the web is great – except when it’s not. One of the areas where our way of work is a two-edged sword is in accountability. When you do something great, it’s easy to get the credit, because you aren’t just one of an anonymous herd in an office. But when you mess something up, it’s equally easy to get the blame. And given the nature of the tools we use, opportunities to make mistakes are always around.

Working on the web is great – except when it’s not. One of the areas where our way of work is a two-edged sword is in accountability. When you do something great, it’s easy to get the credit, because you aren’t just one of an anonymous herd in an office. But when you mess something up, it’s equally easy to get the blame. And given the nature of the tools we use, opportunities to make mistakes are always around.

My own biggest screw-up is easy to remember: I ran a mistaken SQL statement and wiped out about 4GB of data in a production database – a couple of days before we were going to implement regular backups. I kept my job, but there was plenty of egg to clean off of my face. How about you? What have you managed to do in your web working career that was a huge mistake? Feel free to post anonymously if the story is particularly outrageous.

You’re subscribed! If you like, you can update your settings

By Mike Gunderloy

You're subscribed! If you like, you can update your settings

  1. Mr. Gunderloy’s post has the best solution: backups.

    “Backups cure blunders” (everybody say it with me).

  2. My worst: A few years back I was sending out bulk password updates by email… I created the email body by concatenating a string with database values of username and password. But as I was looping through the user records, I wasn’t resetting the string to an empty value so everyone on the list was getting their own password, and the username/password of everyone else I sent before them. There we some 20,000 users but fortunately the system bogged down after the first 1,000 as the email body string became unmanageable large for the VBScript which I was using at the time… This was for a pretty high-profile site and the blunder earned me a special place in IT lore at the company.

  3. A year ago we were switching a radio station over to use Joomla, and I was in the process of integrating a new updated Coppermine Photo Gallery install. It was the last task we had on the project and once the gallery was up the site would go live on schedule.

    I decided it was important to keep everything in one database and started moving the old image database into new tables inside the new joomla site’s DB. Well I ended up on autopilot inside phpMyAdmin, and ended up dropping the entire DB. The box even popped up to ask me if I was sure. I clicked yes because similar boxes had been popping up just like it.

    Needless to say, Joomla had to be reinstalled and the site rebuilt. I still remember the feeling of horror creep up my spine and down my arms, when I realized I had just dropped the DB and lost 3 weeks of work.

  4. And as a corollary to the original lesson: make sure your backups are good.

    Quite a few years ago, I took over administration of an OS 9 web server. I can’t remember the name of the actual web server app, but it had a well known memory leak, so the server rebooted itself every morning at 2:00.

    It had two internal hard drives, and one would synchronize with the other each night. On top of that, various files were backed up to a central file server as well.

    One night, the machine rebooted and never came back. In the morning, I rolled it over to boot off of the other internal drive — only the data on it was about 6 months old. The files backed up to the central file server were a little better, but still a couple of weeks out of date.

    I spent the next 3 days trying to rebuild/recover data from the original hard drive in order to “fill in the gaps”.

  5. With one misguided SQL statement I accidentally turned all the cars in a large national car rental company’s fleet system into white fords. All 13000 of them.

    Their IT team were slack arses and hadn’t been backing up. It took me 2.5 months to restore the data using some tricky data conversion techniques.

    Not long after that the IT manager was fired. And backups were at the top of the complaints list.

  6. Several years ago, I was working on a project with several other contractors on a goverment project. Some of the contractors were less than helpful at times and were actively working to undermine the work of others.

    I had sent an email to one company asking about an issue with their software and copied our customer on the email. The person who was working on the issue decided to forward the email to a coworker but, for some reason hit the Reply All button before adding his coworker’s address. His snide comments managed to reach myself and ALL of the top-level government managers. He was moved off the project shortly afterward.

    While I didn’t appreciate his remarks, I had to laugh when I saw what he had done to himself.

  7. Not really a web mistake, but it was on the web server (and I swear it wasn’t me that did it):

    rm -r /etc/*

  8. Hey thats a nice disclosure.

    Rajeev Vashisht
    http://tekno-world.blogspot.com

Comments have been disabled for this post