February 2018
« Aug    

Back Up? What Back Up?

It can be hard to face up to the obviously stupid things we do

The story goes that Saint Patrick was partway through baptizing an Irish king, when he looked down and noticed he had put the metal tip of his bishop’s staff through the poor fellow’s foot. When he apologized, the king replied that he had thought it was part of the ceremony, which just goes to show what we’ll put up with without complaining.

Collective stupidity is still stupid

Once upon a time, I worked for a tech savvy firm. Our office was littered with cool gizmos and everyone was super smart. Our collective wisdom, however, was low. Together, we did lots of stupid things – like putting our servers in the basement.

The inevitable happened as it invariably does. Our building was at the bottom of a hill and one weekend a heavy rain fell. The basement flooded.

“What’s the big deal?”

On Monday, the flood was the talk of the office. We had backup servers at our regional headquarters, but if you put servers in the basement, you probably also reassign backup servers without warning. At least we did. Our servers were backing up other people’s data.

Risk people like a good mess and I definitely enjoyed this one. Hours and dollars lost piled up in my head until a colleague stopped me.

“It’s no big deal. No one lost anything”, he said, “we have our own backups”.

Ignoring stupidity is not smart either

It turned out everyone was copying everything every day. Backups are fine, but we had way too much of a good thing. Uploading and downloading took forever. Rebooting took forever. The crowd at the espresso machine wasn’t slacking. They were waiting.

It was an obvious problem with an obvious fix. Put the servers somewhere sensible, back them up off site, and get the developers to trust the system enough to stop overcompensating.

Instead, the servers stayed in the basement where it was cool and they didn’t complain about not having windows. No one found it odd. We accepted our inability to secure the data.

Stupidity costs money

Intelligent people can be stupid. We certainly were. A good place to start managing your risks is with the perfectly obvious, but utterly stupid things you do without questioning.

2 comments to Back Up? What Back Up?

  • I once worked for an international securities dealer in Toronto; we had additional trading desks in Montreal and Vancouver. When I was asked to help with DR planning I learned from the IT guy that the back up server (than handled our trading system) was 6 ft away from the primary server. So if the building went down, both servers were down. And while people were thinking that the other two trading desks were natural backup sites, it turns out all trades in Montreal and Vancouver were routed through the Toronto server before going to the Exchange. So if Toronto went down we would have been SOL.

  • RiskContainment

    At least your servers were six feet apart! I’ve seen them side by side, on different floors in the same building (like the famous World Trade Center 1993 example only ten years later), and doubtless soon will see them mirrored virtually on the same physical machine.
    I almost hesitated to write this piece, but the more I thought about it, the more obvious it was. We all know how important our servers are, but we don’t care. If we did, we might do something about it.

Leave a Reply




You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>