Last week on Sunday my trip to Michigan started. I flew out to visit everyone at work, and do some TLF stuff that needed to get done.
As soon as I checked in at Sacramento International, I knew there were going to be problems. The nice lady at the counter told me to “expect an adventure”, as the weather in Chicago was quickly deteriorating and delays were inevitable. With lukewarm temperatures and the sun shining, it’s hard to imagine weather being bad enough to delay a plane. Without much choice, I plunge ahead.
As soon as the door shuts on the plane, the pilot gets on the PA and declares: “Sorry folks, but work out of O’Hare is that visibility is down to less than a mile, and they’re temporarily delaying flights until further notice.”
We sit on the tarmac in Sacramento for 45 minutes.
Finally, after watching a good portion of “The Guardian” on the ground, we take off. Both the flight and landing at O’Hare are smooth, and looking out the window it seems the weather isn’t quite as bad as I had thought. Outside, it seemed to be foggy and chilly. No snow on the ground.
We begin to taxi and come to a stop. The pilot breaks back in on the PA and announces, “Sorry folks, more bad news. Because of the weather planes are backed up in the gates getting de-iced. As soon as a gate opens up, we’ll get parked.”
About 30 minutes goes by, and the fellow next to me is getting agitated. He indicates to me that he only had about a 45-minute layover, and he’ll miss his next flight for certain now. He hails a stewardess, and she mildly explains to him that since we’re waiting, all other flights are too. She continues to explain that if a connection has already left, the right thing to do is to proceed to the gate at which the next flight to your final destination is leaving and talk to the people at the counter there. At first relieved that the two delays so far have reduced my layover to a mere 30 minutes, I too begin to worry that my flight has left and hassles are on the horizon.
We finally get to the gate, and deplane about an hour later than scheduled. I jog over to the flight information monitors, and not much to my surprise, my connecting flight is nowhere to be found. It left on time. To add insult to injury, the next flight to Detroit displays “Canceled”. Oh boy.
I go back to the gate from which we just left and stand in line for 10 minutes just to find out that I need to go see the customer service reps at the “desk across the way from gate B8″. I make haste to get to the counter before the other people on the plane.
It doesn’t matter. Approaching the desk from afar, I can already see the line of people waiting to talk to the people at the counter. As I get closer, I see that the line not only exceeds the zig-zag holding tape but extends about 7 gates further into the distance. At first I’m furious, but then I feel bad for the three people behind the counter that have close to 700 angry people to look forward to.
I’m not one for hanging around airports waiting for someone else to get their poop in a group, so I make the decision to head straight for Thrifty, upgrade to a 4×4 and drive to Detroit. The people at the Thrifty counter are all involved in the Bears game leading up to the Superbowl, and it was a welcome bit of lightheartedness compared to the stress in the airport.
I ask the fellow behind the counter if it is possible to move my reservation from Detroit Metro to their location. At first he gets confused and explains that why he can’t, but the funny thing is as soon as I request an upgrade to my reservation, they’re more than happy to help. Hmm.
I explain that I need something with four wheel drive and plenty of trunk space for the TLF co-lo move that is happening the next weekend. In a few minutes he drives around the corner of the building with something so hideous, massive, and purple that I can’t quite figure out how to explain it. Behold, the Jeep Commander:
In the dwindling light, I first think that they’ve rented me into a semi-retired SUV from 1995, but I then quickly realize that this is a new vehicle that somehow made it past legions of corporate types that somehow thought that a wheeled brick with exposed allen-head bolts in the dashboard was a good idea.
Sometime around the time I got in the Jeep I had the epiphany: My luggage is still on the plane! DUH! I was so frustrated with the trip so far that I had completely forgotten about getting my luggage. I quickly make a call to Phaedrah to get United’s baggage claim phone number. I call them up, and the conversation goes something like this:
“Hi, my connector flight from Chicago O’Hare to Detroit has been delayed indefinitely, and I’ve decided to drive to my final destination. How do I go about getting my luggage off the plane?”
“I’m sorry sir, you can only retrieve your luggage at your final destination. Why aren’t you flying there?”
“All your flights to Detroit have been canceled for the evening and I can’t standing waiting in airports.”
“I see. Well your only option is to pick up your luggage in Detroit. You said you’re driving?”
“Yes I did. But… my plane isn’t getting there. Is there no way to pick it up here in Chicago?”
“No sir, TSA regulations prohibit passengers from removing stowed luggage from the plane.”
At this time I can kind of see his point, but I figure they’re taking the luggage off the plane anyway and other than some minor inconvenience to a runway worker, it’s really not a big deal. But knowing how fun it is to argue with airlines, I decide to go with their plan and forgo the suffering.
“Ok. So in Detroit, do I just pick up my bag at the baggage office?”
“Yes. What is your claim number?”
I give him the ticket number and am told that I will get a call when my bag arrives.
About midway on I-294, I blow through a tollgate by going through the I-PASS lane by accident. I thought the tollbooth lane was an exit for another freeway: It hadn’t been marked. At the next tollbooth I ask the booth person what happens to someone if they miss a toll. “Your license plate is photographed and a ticket is issued. However, you have 7 days to pay online.”
I’m a wanted man in the city of Chicago, so I decide to get the nice card from the lady that explains how to pay online so my rap sheet doesn’t grow.
Other than it’s looks bringing it down, the Jeep handled crossing the State of Michigan in the nasty weather without a problem at all. In fact, right until I got to Plymouth via M-14, the weather was actually perfect. Not good enough to fly, but good enough to drive I guess. I stop at Wendy’s and have some nasty chicken sandwich for driving fuel. Before that I had been running on coffee.
Somewhere around Kalamazoo, I get worried and decide to call the baggage claim people back to confirm that my bag will in fact be waiting at DTW in the morning. The conversation was as follows:
“Hi there, I talked to someone earlier. My connection flight was canceled, and I’m driving to my final destination. I wanted to see if there was any updates on when my bag might arrive in Detroit.”
“You said you’re driving?”
“Yes.”
Why do people keep asking me that?
“What’s your claim number?”
“3016UA115589″
“Ok sir, one moment.”
The lady disappears behind hold music. Really LOUD hold music.
“We don’t have an update on your file yet. However, we will call you when we do.”
“Great, thank you.”
I continue driving, and end up at the hotel around 10:30 pm. I still haven’t heard anything from the airline, so I call back. I discover that somehow or other, my bag had made it on a flight to Miami, and will be back in Detroit around 1:00AM. When I ask the person if the baggage office will be open, she explains that they close at 10:00pm and the luggage will show up on the carousel with the other passengers’.
FARK! I have two laptops in that bag, and I’ll be damned if it’s going to float around the carousel for eight hours where anyone can pick it up.
I drive down to Detroit Metro and pick it up. I call Emmet to explain the complete muck up of the flights and the whole situation and that I’ll be into work late tomorrow. He says it’s no problem. I finally get to bed at 3:00AM EST, at this point I’d been up 24 hours.
Work at Quicken is calm, and very relaxing all week. I enjoyed hanging out with Brad, Keith, and Dan as well as seeing the folks that I haven’t seen in six months like Shiv and Kevin. Greg and I discover we can capture video and make movies with the iSight in my MBP, and make a few choice clips for YouTube.
Fast forward to Thursday. Bill from Waveform calls me to say that my nice Dell server-compatible rack has been delayed by weather in Texas. He says the racks that they do have are tight, and he’s not sure if my servers will fit. I’m a tad bit furious since I’ve been planning this move for well over a month and a half, and that the racks have been ordered a mere week prior to my arrival.
Bottom line from the conversation is that there’s not much we can do about it now, and all the cogs of the machine have been put into motion. Even if the doors have to come off the rack, TLF is moving on Saturday night.
Saturday during the day I head out to Brighton to help my mom install a wireless router and go through some more billing stuff with TLF. She’s thankfully taking it over and will easily do a better job of it than I. After that, I head over to Dan’s for a few hours and see Battlefield 2142 as well as watch part of Nacho Libre. That movie is funny.
As I’m getting ready to head out to Glen’s, Bob calls and says that he’d like to help out tonight. I’m super thankful, since moving 10 servers and their related network equipment to a new building is much easier with more people. Poor guy didn’t know what he was getting into.
When we walk into the co-lo in “Troy Classic” (as coined by Bill, the Waveform guy) the amount of servers is staggering. The room is stuffed with servers. The temperature was well over 90 degrees. Ouch. We begin to tear down around 8:30pm, and after doing all my switch and firewall configuration backups we finally have everything out of the old location at about 10:00pm and drive out to the new place with all the gear in two cars.
We arrive at the new place, and it’s instantly apparent that this facility is far better than the old. It has a high ceiling and the wiring is done overhead. The racks are all locking cabinets and there is plenty of room. Bill leads us to TLF’s new rack, and we open the doors. It’s completely empty, not even mounting rails are in it. Somewhat confused, he leads us over to a stack of milled aluminum and gives us the “go for it” look.
I’m nonplussed, but at this point our network is down and I need to get it back up. Glen, Bob, and I begin assembling the rack. After getting all the bolts in, I put in a pair of the Dell server rails to see how tight the fit will be. The mounting rig in the rack isn’t even close to fitting the rails. The front set of screws is about 1 inch away from the mounting holes in the front. We all stand around looking at it for a minute, and Glen comes up with the idea of drilling out the cabinet’s mounting points so the Dell rails would fit.
Two hours later at about 3:00am we finally have an assembled cabinet and start racking the servers and network equipment powered up. I know that as long as I can hit the inside of the network from the outside, I can do everything else back at the hotel. I plug my laptop into the Fortigate firewall and try to ping one of the servers. Nothing. Try another server, and nothing. I try to hit the gateway for their VLAN. That works, so I try to ping the switch. Nothing.
Let me explain a little bit about the Dell PowerConnect 5324. It’s actually a pretty decent switch for the price, and when it’s up and running it’s quite solid and a good performer. But don’t turn them off, they have a nasty habit of losing their configuration. It happened once before when Bill accidentally unplugged it while moving gear around in our shared rack at Troy Classic, and it took me almost 6 hours to finally get it up and running remotely. I made precautions at Troy Classic by purposely logging in and backing it up. Problem was that I backed it up by copying the Running Configuration to the Startup Configuration. Silly me, because apparently that doesn’t do what it says it will. I panic and realize that I left the serial configuration cord at home in Redding. Glen and Bob are digging for null modem cables in the massive cord heap in the colo’s garage, and I get desperate and decide to reboot the switch. Nothing.
Glen comes back with a Cisco, some black cable, and what appears to be a perfectly normal null modem cable. I hook up the switch to the Cisco console cable, fire up PuTTY and nothing. I reboot the switch again, and nothing. I plug in the null modem cable, bounce the switch again, and nothing. At this point, I’m starting to worry that the switch actually took a major dump and isn’t coming up at all. I notice that none of the activity LEDs are flashing at all, even when the servers themselves are sending traffic–evident by their own activity LEDs going crazy.
Oh boy. Out of desperation, I unplug the switch one more time, wait about a minute, and plug it back in. At first, same thing–nothing–but after about 20 seconds, all the LEDs on its ports flash, and activity starts bouncing around. I quickly try to ping our main server, and it responds.
Not sure what caused that little freakout, but I’m not ready to try and find out at that time.
After all the hassle with the switch, the servers are powered up and it’s after 4:30am. Glen and Bob are exhausted and for some reason (probably due to the stress) I’m wide awake. Glen takes off in his car for home, and I drive Bob back to his apartment in Plymouth since he got way worse than he bargained for, I’m sure.
I get back to the hotel and decide that it’s a good idea to just check everything one last time before I go to bed. It’s now 6:30am. I fire up the laptop and hit thelinuxfix.com. Joomla pops up the “out of service” page, which is encouraging… it means that at least the web server came up and the VM’s probably just fired up out of order.
I log into the network, and quickly realize something’s wrong. The main NFS server isn’t up and I can’t seem to get to it’s host. Logging into the DRAC reveals something horribly bad: The hard disks aren’t being recognized in the server. Oh crap! I power down the box a few times and reboot it, but same thing each time, the Dell PERC is coming back with a error saying the card isn’t responding.
I hop back in the Jeep and drive out to the colo. While I’m there, I pull the server out of the rack, take the cover off, push everything I can see back down into its slot. I take the hard drives out and put them back in. After I get the server back into the rack and power it up, it boots fine. I’m not happy, but definitely relieved that I didn’t have a long day of restoring from backups on my hands.
I drive back to the hotel and go to bed at exactly 8:45AM on Sunday. It was a long day.
The rest of the weekend I spent driving around and remembering why we moved in the first place. Though I miss my friends out here, I do not miss the horrid weather, insane drivers and crazy congestion of the area. After logging a few hours on Detroit’s freeways I discovered myself already getting back into the aggressive driving, and yelling expletives at the other morons on the road. I don’t miss it at all, but it is really sad that two points of happiness–friends and relaxedness–are so far apart. Luckily we’ll be back in about a month for the Gilbert Awards at work, and we’ll get to see everyone again.
I’m glad to be home.
