<< Hoopla Part Three

The amazing story continues with...

Hoopla - Part Four

By Dean Webb
November 11, 2001

Conglomerate Assimilates makes LOTS of software suites, and all of them stink. They didn't use to, but once CA (No relation to Computer Associates, really! I just made that name up!) bought out a company that made good stuff, it soon went bad. If the company was bad to begin with, CA made it worse.

And we used their solutions for backups and antivirus... hoo boy... So, first to get the AV solution was the email server. We had the worst luck with it sending viruses to and fro, and something had to be done about it.

I followed the manual to the letter. Got everything up and running just fine with it. It scanned viruses and got rid of them. I started to wonder why CA had such a bad reputation in the industry.

Then, it happened!

We just hired a new email admin, Dan Dare, who really knew his stuff. He could even write code, something I tried to avoid doing in my admin career. The *day* we hired Dan, the email server tanked.

It was around 10 AM and he had just finished his orientation. All of a sudden, the email server was offline. We looked into it and tried starting it back up. It refused.

We had a choice. We could either run database recovery utilities on the email message stores, which could potentially wipe out huge chunks of the database, or restore from backups. Of course, we chose to restore from backups. All we needed to do was hook up a tape drive to the hapless server and run the CA restoration program. Less than an hour later, we'd have back our email and things would be back to normal.

So we go to the operations room to get the latest backup tape. Rockin' Otto was there. I called him "Rockin" because late at night, he'd jam out on his electric guitar in the ops center. He loved the night. He was cool, too.

DEAN: Hey, Otto, what's up?

OTTO: Hey.

DEAN: Otto, this is Dan Dare. He's our new email guy.

OTTO: Hey.

DAN: Hey, Otto.

OTTO: My mail's down, dude. You gonna fix that?

DEAN: That's *exactly* what we're here for! We need the tape from last night's backup.

OTTO: Suuuuure...

Otto got up and went to where the tapes from last night's backup were kept.

He dug through and got what he thought was the right tape. He put it in a tape drive, checked it out on the backup admin console and, sure enough, it was the right one. We took it and a spare tape drive and went to the server. An hour later, it was clear something was wrong. We had finally gotten the backup software installed on the server and got it to recognize the tape drive and found the correct driver for it and it kept rejecting the tape. It would start the restore, then quit suddenly, saying the backup wasn't a complete one.

Dan figured the tape was busted, which often happens on quarter-inch tapes. They get loose, snap, feel blue, you name it, they'll go out on you at the earliest opportunity.

So Dan goes back to get another tape. Whatta guy. I liked the fact he took the initiative on his first day. It meant I wouldn't have to take the initiative as often as I had in the past. That was so cool. Dan didn't come back right away, though. Rats. That meant I would have to take the initiative, myself.

I went back to the ops room and Otto was trying to get hold of our Ops manager, Boomhauer. Boomhauer was somewhere in the building and his pager battery was low again.

DEAN: What's up?

DAN: We need Boomhauer's approval to get a shipment of tapes from the archive.

DEAN: So we don't have any more recent ones here?

OTTO and DAN: Nope.

DEAN: Damn.

Otto continued dialing, to no avail.

DEAN: Well, shouldn't we get more than one day's worth if we're getting stuff back from the archives?

DAN: Yeah. They can pull a couple of tapes and we can test those. The only mail we'll lose will be the stuff from today.

OTTO: You sure?

DEAN: Is that a problem?

OTTO: Yep.

DEAN: Why?

OTTO: All the tapes are kept in these huge lock boxes. We can't get just one or two.

DEAN: Oh.

Boomhauer walks in.

OTTO: Hey, we were tryin' to reach you, boss.

BOOMHAUER: Yeahireckonyouwerethisdangolpageraintworthbeansitellyouwhat.

OTTO: Yeah, well...

BOOMHAUER: Oksowhaddyallneedoverhereicanhelpyallwith?

OTTO: We need a backup from the day before yesterday.

BOOMHAUER: Ohmangottacallupthedangolarchivecompnyandgettacouriertakethethingoverhere.

OTTO: Yeah. We need your approval.

BOOMHAUER: Okdowhatchagottadomanillbeinmyofficeifyouneedmeferanythingorwhatnot.

OTTO: OK, chief.

And with that, Boomhauer retired to his office.

While we waited for the tapes to show up, LaLa was going totally ape and demanding updates every two minutes. We did everything we could to avoid it, but our pagers had fresh batteries and we were never far from the phone, so we were doomed.

Finally, the tape showed up and we tried it out. Sadly, it was no good, either. Another incomplete backup. We went back to Rockin' Otto for answers. When we got there, our weekend tape operator Anne Heche (no relation) was there. Anne was a "free spirit." That's the nicest way we could say "totally airheaded floozy." She happened to be working today, so we asked her about the weekend's backups after we finished the introductions.

DAN: Anne, uh, all our backups are showing up as incomplete. You know anything about that? Were there any errors?

ANNE: Nope.

DAN: OK.

Dan and I didn't believe her 100%, though. Dan went to the backup admin console and checked through the backup log files. He didn't see any errors, but I noticed an "informational warning":

WARNING: Backup on server EMAIL001 cancelled.

That was our email server.

DAN: Anne, could you come here?

ANNE: What?

DAN: You cancelled the email server backup.

ANNE: Huh? Which server is that?

DAN: EMAIL001.

ANNE: Oh, that one? It always takes forever and we need the tape drives to back up the file servers so we always cancel that one.

DAN: (stunned silence)

DEAN: (stunned silence)

OTTO: Is that bad?

DEAN: Yes. Very bad.

OTTO: Oh. Sorry.

Dan scrolled back through the log files. Those backups had been cancelled for over a year.

We now had one hell of an update for when LaLa pestered us again. Meanwhile, Dan and I had to go run utilities to recover the email server. 15 hours later, we had mail again. LaLa told us Tinky-Winky said we were supposed to keep the mail server in a "spare" cubicle near us (really a tiny, malformed thing no admin wanted to use, so it was Dan's cubicle, as he was The New Guy) and run backups on it every three hours.

Good thing we ran backups every three hours like he said, because later that day, the email server went down again.

NEXT: We continue to discover why Conglomerate Assimilates products have such a rotten reputation.
(Coming soon)