Doing anything at the last minute is usually not a good idea. That seems obvious, but it’s something I had to learn the hard way.
Years ago, I used to work in the aerospace industry. It’s a lot of fun, but there’s also a lot of tedium and waiting. High dollar, single quantity products such as spacecraft, are not easy to build, test, and deliver. The problem is that everything has to be right the first time. This also applied to the systems used to test the flight systems.
There are many stages of testing, one of which is integration with a larger package, in our case the “ride”, or the satellite our instrument was piggybacking on. The satellite was built by GE Astrospace division, formerly RCA. This particular satellite system was quite old, it was primarily used by NOAA to measure cloud cover using infrared imaging.
Our instrument was bolted to the satellite after which GE would go through their automated test procedures. There are a lot of systems on a satellite, so the test takes a long time — usually hours. The test is monitored by GE and NASA staff in what is called the NAGE room. Our lead systems engineer was located there as well to monitor the test. Myself and another colleague were located in the NAGE annex. This was a small room far way from the NAGE room proper. We had a direct data link from the satellite telemetry stream which we would use to monitor our instrument. Our only communication was through that data link and a telephone.
The data link was a proprietary, synchronous serial stream designed in the 1960s. Nothing existed off shelf to interface to that stream, so we built our own interface. It was basically a couple of shift registers feeding a FIFO interfaced to a PC ISA bus. We had no idea if that interface would even work, this was the first time it had been used with a live system.
We were told that it would be several hours before our instrument was to be powered on and we would see data. In the mean time, we thought we might add a few features to our data capture and display software. It sounded like a good use of time. It didn’t really matter, we recorded everything on DAT tape in real time anyway.
We were given regular updates through phone calls from our systems engineer in the NAGE room. Eventually he called and told us we were about to be powered on. It was absolutely critical for our ground support (GSE) system to record all data during power up. If something went wrong, we could go back through the data and track it down. We sat patiently waiting, waiting, waiting for the bits to come down that cable in the wall.
Finally, it showed up. To our amazement, the instrument powered up normally, went through it’s built in test sequence and delivered good data. Our GSE interface worked flawlessly also. It was like the Eagle landing on the moon! We were jumping for joy along with the systems engineer in the NAGE room.
I thought I would switch screens to look at different telemetry streams. I hit one of the function keys. That was a mistake. The screen froze and the PC no longer responded. The tape drive write LED no longer flashed every half second indicating data was not recording.
Our GSE software was written in C, primarily by yours truly. As any C programmer knows, the C runtime does not have any runtime type safety. Simply put, a bad pointer caused the PC to crash. These were the DOS days, no hardware memory protection. Apparently, this was one of “features” I added while we were waiting in the annex. Things were not good, nothing was being recorded, bits were spilling on the floor. We were in shock. I handed the phone to my colleague who didn’t know what to do either. He hung up the phone without saying anything as we frantically tried to reboot as fast as possible.
Our systems engineer immediately called back, we felt like we were in the control room of Three mile island with the core temperature rising. Our systems engineer was very understanging, he tried to calm us down and said he was not going to yell at us, just to tell him what was going on. Eventully, we rebooted, started recording again and didn’t touch anything until the test was over.
Fortunately for us, nothing went wrong with the instrument. Everything we smoothly, but it was a hard learned lesson.