Remembering Steve – Part 3

While I was at Great Ape Trust of Iowa, I had built an XSan installation that I used to host user home directories off of. This wasn’t a very standard configuration, albeit one that was marketed and supported by Apple.

As is customary, I will wait to install a new operating system until at least one point release is distributed. This was the case with 10.5.1 client and server, which I installed one night in early 2008. I installed it, then went home and to bed.

The next morning, all hell breaks loose.

At some point during the morning, a particular user logs in, and all of the files on the server disappear.

I reboot the server, and the files are back. Then they disappear again. Reboot, and they’re back. Up, down, up, down.

I revert the server to a backup of the previous version, and then start a series of long nights trying to figure out why I couldn’t keep things stable.

At one point, I send a 2 AM rant to sjobs@apple.com, saying that with all the hell 10.5 has put me through regarding XSan, the least they could do is make iCal server not suck in 10.6.

The next day, I get a call.. “Hello, my name is so-and-so from Apple Executive Relations, we understand that you are having a problem with your XSan. I’m going to get you in touch with the XSan engineers to assist with the problem.”

It turns out that there is a bug in the server software that was only reliably triggered by my configuration. Once I found the parameters, and relayed them to the engineer, they were able to duplicate the bug in their test installation, and they sent me a patch to test. That patch was then included in 10.5.2.