Smelling The Bull....

 I happened to catch a news story on CNN this morning that says HBO - an entertainment organization - is producing a "documentary" (CNN's words) on the "Time Bomb Y2K" about the Y2K problem in computers back then.

I haven't seen the show - obviously - and I'm not likely to, but from the write-up in CNN, it sure seems like the CNN writer and the the directors of this ... thing are saying that the Y2K Problem was all made up.  Given that I was an adult IT technician and manager at the time, I do think that their opinions are certainly their own, but sadly, the "entertainment" value of this show may be just about as much as the accurate content.

Why do I say that, you ask?  In late 1998 when I was asked to evaluate a few things, I had to develop a test plan and check out my organization's systems.  And I was working on what was rather widely acknowledged by the technical industry as the trailing edge of that evaluation.  

Many of our systems had been modified to handle the Y2K problem.  For those millennials and others who were not aware of the issue, many computers, both computerized systems that relied on software, and some that were integrated with what we used to call "hard coded" systems, being boxes that had computerized controllers built in, relied on a systems date that was usually represented in 6 digits - a two-digit day of the month, a two digit month, and a two-digit year.  

Why was this a problem?  Well, that list can go on, so I'll sum up.  In some systems, if you sort things out by date, 12/31/99 represents a pretty high number.  01/01/00 is a very low number.  Some systems picked an arbitrary day and counted forward from that.  The problem there is "arbitrary".  If you enter a number in a spreadsheet cell, then format it as a date, you may see some of the problem.  I personally rely on OpenOffice for my home use, and in that, if I enter 1 for a date, and format it, my computer now tells me "day 1" is Sunday, December 31, 1899.  If I provide a negative number, it will count backwards from that date.  Obviously, a larger number counts forwards.  Today, December 30, 2023, is 45,290 days from 12/31/1899.

What kind of problem could that cause?  Well, in my experience, it was going to generate a whole lot of invoices for the customers of my organization, who had already paid those bills.  But the computer saw them as past due.  Fortunately, we caught that one early, and then, after a fix was written, we went into the database, and fixed all the dates.  That's what we did then.  We wrote a program that read the information in one table, then wrote it into a separate table.  The original table was then modified.  Modification, in that system, was going to break a lot of things if it wasn't done correctly.  If we'd done it the easy way, all of those dates from 1985 or 1992 or 1999 would have become 8519, 9219, or 9919.  What we had to do was create a new table, which would then take every date (and the other information from that table), and push the last two digits back two characters, and insert a 19 in front of every single one of them.  Because, back then, we were all in the 20th century.  We could, then, assume that the dates were from that century.  I wasn't working with records - like documentation that had been computerized from, say, medical records, which quite legitimately could have seen dates from the 1800s.  

What?  Yes, in the 20th Century, there were people who had been born in the 1800s.  Not a lot of stuff was computerized in the 1950s, but when you ran into someone who was 80 years old when I was a child, they had to have been born before 1900.  So there's another piece.

Now, that accounting problem we discovered and resolved was a very early, very easy problem.  I stumbled across one one weekend when assisting my boss.  We had our own computer that handled voice mail.  Yeah, kids, take a seat if you haven't already, back then, most organizations did not have the option of using the phone company's voice mail.  They didn't have it.  And my organization was responsible for people who worked across the United States.  So here's a problem for you.

Joe, not his real name, lives in California.  He has investments that quite literally cross the country.  He has operations in Maine, Ohio, Iowa, Colorado, and Washington state.  Now, let's say, for whatever reason, someone leaves him a message - because, being any successful citizen, he's decided to visit friends and family in New York, and he's in Times Square back then.  He's got his brand new flip-phone - a new type of cellular phone - on him.  Tom, in Maine, leaves him a message wishing him Happy New Year.  Tom knows that Joe is in New York, so he leaves it at 11:30 pm on 12/31/1999.  The voice mail system stores the date and time as 9912312330 - 99 for the year, 12 for the month, 31 for the day, 23 from the hours from the previous midnight, and 30 as the minutes in the hour.

Stan didn't get the email from his boss that the team would be celebrating in Times Square.  Stan waited until after midnight in his location in Iowa to send that voice mail message.  It hits the voice mail system at 12:15 am.  Stan figures that Joe is in California, he's got an hour and 45 minutes yet.  Except the voice mail system stored the date as 0001010015,  Now Joe doesn't check his voice mails all that much on Holidays, his businesses are closed.  The next morning, he gets a message from Stan.  The system checks it's internal clock, and tell him that as of 10:30 am the message from Stan is 10 hours 15 minutes old.  It does not find the message from Tom.  Because when it looks for messages, it assumes the messages have been left in the past.  Why?  Well, aside from starships named Enterprise, Marty McFly is off somewhere with Doc Brown's DeLorean, and most of us are only leaving messages in the past.  We don't leave messages in the future.

So on 0001011030, there's no way messages from 9912312330 even exist.  Which is exactly the problem we discovered.  How?  Well, here's what we had to do.  We had a pretty high-end computer - a gigantic (for then) amount of storage - an entire terrabyte - set up.  It was compatible with the operating system the phone software needed.  So I installed the software, worked with the vendor to make sure I'd covered all of the obvious issues, and then restored the backup onto the new computer.  After I'd changed the date to January 10, 2000.  I was not able to find any messages.  None of them came up when I called the number - I'd plugged the new computer into my desk extension, and tried to get a message I'd left a few days before taking the backup.  I'd saved it on the system.  But it wasn't there.  I could see the information was stored, but when I tried to retrieve old messages, the system reported "you have no saved messages."  

This was from a system which my employer had purchased four years prior, and should have been configured to use a four-digit year.  The reason it hadn't, I presume, is because the folks writing the software had never looked all the way down into the nuts-and-bolts to find out how the date was stored.

And that was only one of the issues.  I was talking to one of the maintenance managers in the building about adding a new T1 line (yes, kids, it's a fancy phone line that can bring in 24 phone calls, or 1.5 megabytes of data - back then).  We were needing some of it (18 lines) for additional in-and-out-bound calls, and six of those lines were going to be bundled and used for the 75 people in my office to share for internet access.  99% of which was going to be email.  Though I did have to have a number of conversations with a few employees who thought the whole point of our internet access was to surf the net, and a few people had discovered these early "shopping" sites.  One employee threatened to quit when I told her that if she did not stop her non-work-related use of the internet from the office.  Her supervisor, who was right next to me as I explained the issue, told her that the reason she was there was because they needed her to do her job.  And the HR person who was also there (I did not leave myself uncovered, in those fun days) explained that nowhere in her job description was internet access mentioned or assured, and that I could prevent her from getting to the internet.  Which I could not do yet in those days, because it hadn't become a problem big enough for people to figure it out.  

These days it would not be a problem at all.  But then it was one of those calls you'd make to the software help desk for an organization and the tech support person would respond with a "you wanna do WHAT?"  

Anyway, when I was talking with the maintenance manager, I mentioned we had been doing Y2K testing, was there anything they saw?  I was also responsible for our disaster preparedness planning from an IT perspective - so knowing if the building was going to be open and working would be helpful.  He had mentioned that while the overall building security system, our access control system - was Y2K compliant, they had found a problem in the elevator controller which had caused the elevators to miss the floor by nearly a foot.  Why?  We could not figure it out.  But the elevator people had come out on a weekend, installed a couple new boxes in the elevator shafts, and the second test showed that the floor of the elevator was within 1/32nd of an inch of the floor when opened.  So yeah, expecting to walk out onto a level floor and finding out the floor was 16" below where you expected, well, the wall probably smacked you when you didn't notice it, I figured, but that would have been a nasty surprise.  The equivalent of going down stairs and then missing two steps.  Oops.

 Obviously, the phone call I had to make - my boss had left me in charge of the project - after I discovered the Y2K issue with the voice mail system - was one where I expected similar "what the heck did you do that for?"  I mean, the system was only about four years old.  We shouldn't have had a problem like that.  But it was pretty quick.  I called the vendor, spoke to the person who answered the phone, and the call was pretty much "Hi.  I've been doing some Y2K testing and - "  The rep stopped me.  "Oh good.  Thank you.  I've got you listed as [name, company, address].  We're sending updates to everyone, yours is scheduled to ship [in a month].  Can I move you up our priority list and send it to you directly?"  Uh, yes, please, but ...  "Will this resolve all issues?"  "We tested about a year ago, discovered the issues, and the fix is totally within the software, we didn't need new hardware, yours is still plenty new.  We have had to upgrade some older systems, but this fix was easy."  Of course it was, they assured me, because if it was difficult, well...  

I got it.  I put the upgrade on the new temporary system first, and determined that it would work right.  I then did all the backups and very late one Saturday night, after sending a mass email and voicemail to people a month before, took the voice mail system down, did the upgrade, and brought it back up.  I disconnected it from the phone system and external inputs while doing it, and then verified it worked.  And with three missed calls at 2:00 am, the total extent of the damage was minimal.  The system had been off-line for 19 minutes, I had projected three hours - the vendor recommended an hour, but I tripled that because my boss said "oh yeah, what happens if something goes wrong?"  And 4 of the 19 minutes were me making a phone call to the technician from the vendor who had promised to be available for support issues.  I called her and told her "everything went great, I'm good."  Seemed to be the courteous thing to do.  

So after all of that, the fancy new computer that had been used to back up the voice mail system, which we had figured was going to end up on my desk, because it was a good one, ended up being pressed into backup service for the accounting system, just in case.  So that was one less worry.  

But when people say the "Y2K problem is overblown" I am amazed.  I know some of those contractors were probably billing for vacation homes, because some systems were easy fixes, some didn't need any at all.  But there were definitely some that were just flat out dangerous.  Takes a fool combined with conspiracy theorist to say "ah, nah, the Y2K thing was overblown."  The great news is I am betting that there is an extremely high correlation between those fools and the people who thought the COVID-19 Pandemic was overrated.  You know, like the idiots who said on TV "Why didn't they fix it earlier, since this is the 19th one?"  Uh, sorry, dipshit.  It was COVID-19 because it was discovered and identified as a problem in 2019.  

But there's yet another example of the Y2K problem.  Had they called it the COVID-2019 version, then we wouldn't get the bullshit like "oh, it's been around forever."  

Yep, still here, still kicking - and after a quarter century, I guess I do have a pretty good memory.  Then again, it doesn't take a whole lot of brain bandwidth to remember that Cammie, short for Camille, was the person who took my 2:15 am call and chatted with me.  But I do suppose reviewing the process, making sure I did ALL the steps, made sure her organization was doing their due diligence.  And she was in California, so it was only a little after midnight.

But I did miss my then-new-semi-brother-in-law's wedding...  Because I had been asked to "remain available locally" in case issues came up.  My wife was not pleased.  And I do not recall any particular compensation of any sort for sticking around.  But that's the way it goes with IT.  We were just expected to be available.  I like the job I have now, mostly because of entirely reasonable hours.