When Servers Go Down

It wasn’t until nearly 6:30, and after I’d done all my weather hits, that someone discovered what was wrong. Like flipping a switch the missing graphics magically appeared.

By the time I got to work Thursday Dave, one of our maintenance techs, was already replacing the “C:\” drive on the weather server. It’s a machine called Metline. It is our master filing cabinet fed by a satellite delivered firehose of data. Up and running 24/7/365 it is constantly storing new and deleting old info.

Let me take a step back. At work we are fed nearly every bit of weather data imaginable. Need to know the current condtions in Singapore? just open a small terminal window and enter “sa wsss.” It’s there. So is everything else.

When this system was first installed there was no Internet, so our vendor fed the maps, text and images to an uplink facility (actually two totally separate facilities via totally separate paths) where it bounced off a satellite and back to a few dishes on the roof of the TV station. The data pipe we see is about T-1 sized and constantly full of bits headed our way.

Unlike the Internet we don’t ask for specific data to be delivered. We get everything!

Back to the Metline. It had begun to act up. A request for data which should be filled on-the-spot started to take 20-30 seconds to show up. The machine was strangely incommunicado during that time. Computers never heal themselves. They are only capable of getting worse!

Bruce, in Madison, WI and working for the vendor that sold us the system, thought a hard drive might be going. It was sent to us overnight. Now you understand why Dave was swapping hardware.

The drive went in fine and the whole backup process went smoothly in not much more than 90 minutes. Unfortunately parts of the system that worked before the swap didn’t work now! The ability to take data and contour it was gone and airtime was approaching.

During our 90 minute newscast I was assisting Dave, who knew the hardware but not the software, and the folks in Madison, and also sporadically appearing on-the-air. Oh–I also had to work around the graphics I’d usually use which were unavailable.

It wasn’t until nearly 6:30, and after I’d done all my weather hits, that someone discovered what was wrong. Like flipping a switch the missing graphics magically appeared.

The more computers you have the more likely it is something will go wrong on any given day. At work I manage around 15 PCs. There’s always something not working. you’ve got to learn to adapt on-the-fly.