When PCs Fail

That’s part of what’s happened today and it’s causing me to tear my hair out.

computer racks.jpgI’ve been working with computers most of my life. My first/only computer course was 1968. For the past 25+ years they have been an integral part of my work life.

Nowadays I wrangle around a dozen machines (see photo) at work which let me produce a forecast and feed it to a bunch of different (buzzword coming) platforms.

Mostly, I get it. I understand how computers work. That gives me a leg up. Often it’s necessary to think along with the programmer to affect a fix.

There are two things which always surprise me.

1) There’s always something that’s not working!

It might be hardware or software or even a bad piece of data which should be a temperature or cloud but ends up being interpreted as a command. The computer stops what its doing. There’s never a time when I can depend on everything!

Google is well known for designing its software specifically to understand hardware will always fail. Those Google guys are right.

2) Computers often continue to work when something’s wrong–though it turns out they’re really waiting to fail at a time less convenient to me!

That’s part of what’s happened today and it’s causing me to tear my hair out.

A hardware failure late last week took out a two hard drive RAID array (two disks which act as one to provide constant backup or, in this case, additional speed). This particular piece of equipment was down for a day while we waited for FedEx to deliver the replacement. No problem. Like Google we understand working around bad hardware.

Once we replaced the drives we had to repopulate them with data. In this case it was an accurate rendition of the Earth’s surface–really. That meant nearly 200 GB of data had to move across our network. It took hours.

By late last Thursday evening we were up and running perfectly. We’d made some accommodations for the new hardware. No sweat.

Saturday was rainy and heavily tested this new configuration which worked nearly perfectly.

It failed this morning!

Why?

Who knows.

What was different between Saturday and today? As far as I can tell nothing!

The point is the computer was working just fine though it obviously wasn’t. There was something still wrong that needed just the right moment… the right set of circumstances… to fail

For whatever reason I was always under the (false) assumption that you needed perfection within these complex system for things to work. Obviously not. And, of course, it makes you wonder what’s next… or if you really can ever fix all the problems.

I’ve still got over two hours of data transfer to go this second time. Time to think about what might be next.

Corporate Kvetching

Pardon me for being a little skeptical, but I am. I’m not denying it’s tough to find educated people, and I’ll get to that in a moment, but Mr. Stephenson leaves out an important part of the equation.

The president of AT&T, the telephone giant that was SBC… was Cingular… was SNET… was speaking a few days ago, and he was upset.

As Reuters reported:

“We’re having trouble finding the numbers that we need with the skills that are required to do these jobs,” AT&T Chief Executive Randall Stephenson told a business group in San Antonio, where the company’s headquarters is located.

So far, only around 1,400 jobs have been returned to the United States of 5,000, a target it set in 2006, the company said, adding that it maintains the target.

Pardon me for being a little skeptical, but I am. I’m not denying it’s tough to find educated people, and I’ll get to that in a moment, but Mr. Stephenson leaves out an important part of the equation.

Isn’t this what he meant to say: “We’re having trouble finding the numbers that we need with the skills that are required to do these jobs at the salary we’re willing to pay.?

That’s a question asked yearly when H-1B visas are debated. H-1B is the ‘permission’ required by high tech companies to bring in foreign workers to perform highly skilled jobs.

Bill Gates was talking about this last week and is quoted on SearchCIO-Midmarket.com.

“We live in an economy that depends on the ability of innovative companies to attract and retain the very best talent, regardless of nationality or citizenship,” Gates said. “Unfortunately, the U.S. immigration system makes attracting and retaining high-skilled immigrants exceptionally challenging.”

The thing is, opponents of H-1B visas say, Microsoft and other companies aren’t paying “high-skilled” worker wages. H-1B rules do require that workers are paid the prevailing wage for their job.

Matloff said H-1B workers in the IT industry are “almost always programming of some sorts.”

“It could be a programmer, it could be a software engineer, it could be a system analyst,” Matloff said.

But Matloff and other H-1B critics contend there is no shortage of American workers for those jobs. H-1B workers, they say, just come cheaper and younger.

Quite honestly, is this any different than the complaints raised by farmers, looking to bring in migrant workers? They always say there aren’t enough US workers for these jobs. But is agricultural work more difficult or distasteful than construction or pumping septic systems. People do those jobs, even under difficult conditions.

Is the question finding workers or finding workers at what they’re willing to pay? Is that the fault of the business or the worker? Should American businesses pit Americans against foreigners when it comes to wages?

Back to what AT&T’s Stephenson and Bill Gates said about the state of American education. I couldn’t agree more.

Our high schools and colleges have become more like trade schools than institutions of higher learning. Where is the broad foundation which used to make up a high school or college education?

We live in an era where creative thought is required for more and more jobs. But is that creative thought being nurtured? Are we really well served by our education system?

A Brief Status Report

In other related news,, it now looks like the attack on my site produced over 69,000 spam pages! Mind boggling, isn’t it? That’s how many pages Google indexed before I cleaned things up

I guess I’m sort of pleased with how quickly the site has come back together. Movable Type 4.1 is very different, but I’m getting the hang of it. Not being a programmer or web professional, there are a lot of head scratching moments for me. So far, I’ve been able to do just about everything I’ve attempted (except getting the archives to display in proper chronological order).

My friend Peter has a weird text problem, where entries are truncated in such a way as to be unreadable. I haven’t been able to replicate the problem, though I’m sure it’s real.

My friend Wendie isn’t thrilled with the narrower columns, though the prevailing wisdom is, they make a site easier to read.

It looks like I will be staying with this reddish look. Once I built my new masthead with the grunge font, the color scheme was locked in.

I have made some minor tweaks to become more search engine friendly. My quote: Back when I was in school, teachers would always scare me with stories of how my exploits would end up in my permanent record. I believe this is it! – Geoff Fox,” was at the top of every page. It was the except you’d see in search engines.

Now the words are integrated in a graphic, not spelled out in text. The actual page content will now show up in the except. That’s as it should be.

If you have any problems with the site’s operation, suggestions or questions, I’d like to hear from you. You can leave a blog comment or send me an email.

In other related news,, it now looks like the attack on my site produced over 69,000 spam pages! Mind boggling, isn’t it? That’s how many pages Google indexed before I cleaned things up

After a few weeks off their site, Google has resumed indexing me. My traffic is way up – nearly twice normal numbers. Much of that is due to people looking for the pages posted by whoever broke into my system. I guess there’s a market for that stuff.

I’ve posted a robots.txt file, which tells search engines what they should and should not index. Hopefully that will clear away the bogus pages they’re still pointing to.

Blogger’s addendum: Tonight I created a new page, explaining to those who get here by accident, why they’re here. It shows up automatically to anyone who enters a bogus page on this site.

The Evil Within (My Website)

I have been complaining for a few weeks now about the disappearance of my website from Google. It was a problem that only appeared when a page was clicked directly from a search engine, not entered by hand. Instead of my blog, spammy pages of links appeared, looking like they were coming from geofffox.com.

I looked and looked and couldn’t find the content ascribed to me. Nothing.

A few times, I asked for assistance from the Google Groups Webmaster Help Forum. They didn’t solve my problem, but they did help me figure out where to look.

Tonight, I think the mystery is solved.

On November 23 at 9:04 PM and again on Dec 10 at 10:54 AM someone gained access to my server.

Whether it was a hack or exploit doesn’t matter. My host would like me to think it was an exploit – meaning it was through my doing. Whatever.

New .htaccess files were inserted in every directory on my web site. These files, which begin with a dot to make them normally invisible, control how certain web requests are handled.

These .htaccess files looked for 404 errors. That’s what you get when a page is missing.

This explains why I couldn’t find the spammy files on my website. This exploit only worked if a file name that DOESN’T exist was entered. Only then were they composed on-the-fly.

On top of that, a second file specified the trick should only worked if the request was coming from a search engine. No wonder I couldn’t make these spammy pages appear.

As awful as it is, I have respect for the programmer who accomplished this. It’s a very sneaky trick, and it sat on my site for a few months before I discovered it.

The .htaccess file called a php file, which is similar to the one this web page is served to your browser on. One set of these php files had an all numeric filename (002314.php, etc) and was 617 bytes long. The other used simple computer-ish names (server.php, command.php, etc.) and was 1260 bytes.

Every web attack has a weak spot and I had found it. Having the files all be the same length, and placed on my machine on specific days, made it easier to scout them out.

To make matters worse, permissions were changed all over the place. These are the rules that decide who can or cannot read, write or execute files. Lots of stuff was turned 777, meaning anyone could do anything!

The person who attacked my machine had opened all the doors. Now anyone could gain access and do anything.

Good grief!

I called on my friend Bob Hart to help.

Bob claims not be be a computer expert. Right.

His logical, organized, well exercised mind knew all the commands and tricks to remove thousands of files and reset an equal number of pointers without hurting anything. He dictated long strings of characters for me to type in… and they worked!

Is there an Emmy for computer assistance? I nominate Bob.

So, now you know the good news. The bad news is, it can happen again because I don’t know how the miscreants got in. I’m working on that next.

Hopefully, in deleting files, rewriting permissions and changing passwords, I will slow them down until a solution is found.

Once again, I’m begging Google to let me back into their good graces. Traffic on the blog is down about 75%. Geofffox.com has slid off the face of the Earth!

In The Trash It Goes

Today was my first ‘real’ day of dumpster duty, participating with Helaine as we tried to empty the house of more accumulated stuff. There’s a lot to choose from!

I suppose some of what we tossed today has resale value… but it’s so much to deal with, nickels and dimes at a time. After the entire job is done, there will be material for eBay – just not this stuff.

We started in the attic. Ours is quite large. It’s a whole floor above three of our bedrooms and a full bath.

I found old computers and circuit boards. It was tough to move them out, but some are so outmoded, it made no sense to continue to house them.

I tried to quickly calculate what they were worth ‘in the day.’ Too depressing. Then I tried to figure out how quickly they’d become outmoded. One computer monitor had a 1999 manufacturers stamp on it. Could you even get a 17″ CRT monitor today?

I did notice one thing that’s changed in computing. I threw out lots of manuals. An old printer had a ‘programmer’s manual.” Some programs had two or three separate printed books. I read those manuals. I miss the reassurance they brought.

I threw out some books related to Photoshop 3. Photoshop version, CS2, is really Photoshop 9!

As we carried load after load to the front door, for a wheelbarrow ride the rest of the way to the dumpster, I realized how nice a ranch home would be right now.

We headed to the garage next. Are you like us? Do you push things against the wall, hoping you’ll never have to deal with them again?

As with the attic, there was lots of stuff in the garage ripe for throwing. Each successive layer removed revealed cobwebs and dirt. Is there a place where dirt accumulates better than a garage? Those big doors open to the elements a few times a day, and then there’s whatever the tires track in.

I moved in with the shop vac. Everything was quickly scarfed up – though my back won’t soon forget the stooping I did. Growing up in an apartment, a shop vac was foreign to me. I learned quickly as a homeowner, this was no option!

Helaine called me over. She said she’d found a dead frog. Nope – dead mouse… long dead.

You can’t live in the woods without having a mouse in the house every once in a while. Luckily, there’s no evidence they’ve ever lived in the parts where we live.

After the garage, it was on to the basement. There was too much there for one afternoon, so Helaine will continue while I’m at work this week.

One thing we did do was clean the basement’s walk out steps. It’s possible they’ve never been vacuumed in the 16 years we’ve lived here. They look great now.

I threw out some more videocassettes – some from my days in Buffalo. It’s not that I didn’t want to cling, but even today there are few places to play these outmoded U-Matic format tapes.

I did keep a box of reel-to-reel audio tapes. I have a few marked 1973. I might have been in Cleveland, or maybe Phoenix, then. I’m not sure. I do know radio airchecks aren’t for tossing.

The dumpster is still a long way from being full. Thirty cubic yards is a heck of a challenge!

We’ve got one more week with it as a house guest. I told Helaine, when it’s picked up and the driver looks inside, we won’t be embarrassed.

It’s An Addiction – I’m Not Alone

Katie Haffner had an interesting story about blogging in this moring’s New York Times. I always thought (and Helaine will confirm) I’d gone off the deep end with blogging, but this article makes it seem like I’m not so bad. There are others who have been bitten far worse.

Thanks God for small favors.

Continue reading “It’s An Addiction – I’m Not Alone”

Who Is Your Tech Support?

A few years ago, my friend Kevin gave me a bumper sticker, “Friends Don’t Let Friends Do Tech Support.” Yet that’s what home computing today is built on.

Try getting support from someone who sold you hardware or software and you’ll find you’re the last person they want to hear from. Have you ever tried to get in touch with Microsoft?

To much of my family and many of my friends, I am tech support. Don’t understand what’s wrong, call Geoff. That’s good and I enjoy it… though it seems a shame that the company’s responsible aren’t carrying their own weight in this regard.

Who do I go to? For Linux and OS related problems, it’s my friend Bob in Florida. For Windows and hardware related problems (and, thankfully, I seldom have software problems I can’t solve on my own), I go to my friend Kevin.

I saw Kevin tonight.

This afternoon, as I was attempting to print 25 sheets of something for Helaine, the Epson Stylus Photo 785EPX connected to Steffie’s computer (but which I print to through our home network) decided to ingest about 25 sheets at once. As the paper jammed into a space much smaller than it could be compressed, the printer started to whine. Gears meshed. It wouldn’t stop. I swear the printer was crying.

I unplugged the it and removed the paper without much problem. But, when I turned the printer back on, I got a paper jam error message. Uh oh. I absolutely knew there was no paper there because the sheets that had been caught had come out whole, though somewhat creased.

After scouting around the net, I realized it was probably the paper jam sensor, not a jam itself. Three choices, new printer, printer service (at most of the cost of a new printer) or do it myself. I didn’t have much choice but the latter.

Being technically inept when it comes to mechanics, I called Kevin on the phone and asked real nice. There was never a question, because Kevin’s the kind of guy who would give you the shirt off his back and because he really enjoys the challenge of fixing something that’s not really built to be fixed.

I am so upset I didn’t bring the camera, because this printer is a mechanical work of art. As you peel away the layers of a mechanical system, you can quickly see how much thought went into doing it right. The cable runs were neatly held in place by guides. Most terminated in nicely keyed plugs. A few didn’t have plugs but seemed to end with exposed connectors and were stiff enough to insert cleanly in sockets.

The cover came off fairly easily. That didn’t get us to the problem. Next, a rear assembly which drives the paper as it is pushed into the path. Kevin saw this mechanical marvel intuitively and was immediately able to know how it worked and why everything was where it was. There were a few times when I pointed the way, but mostly it was Kevin.

The ability to see how something works is a gift. I think I have it as far as software is concerned. I can look at a program or even look at its code and understand what the programmer was trying to do. Kevin can do that too, and take it one step further by understanding hardware.

The problem was a tiny lever which was supposed to be held taught with a smaller spring. The lever blocked a light sensor from seeing an LED. That’s how it knew if the path was blocked by a paper jam. But, the spring, held by tension alone, had disconnected from the lever.

It required removing three separate assemblies and then, putting them back together. On the first try a cam wasn’t set right. The printer powered up to the sound of plastic gears gnashing. Kevin and I looked at each other. This could be the end of the repair.

We quickly figured out what the cam was supposed to do and where it should be on power up. Bingo! The printer fired up quietly and the indicator for a printer jam stayed dark.

Because we didn’t have the drivers for the printer, that’s as far as testing has gone until right now.

I’m going to plug it into the computer.

The computer has recognized it and is loading the drivers. Success. Now, to print.

Wow. No smoke and a perfectly executed print job.

Kevin would be a great friend even if he couldn’t fix anything. But, he can.