So I thought to myself, “well doofus, how you going to do this?” Now Bubba and I had discussed in detail what needed to happen for the site to survive before I signed the papers, so there were no surprises going in. Matter-of-fact, the needs were part of what made me hesitate. SimHQ needed work done on it. Lots of work. There just hadn’t been the time, money or personnel to address things previously. Now it was at the critical stage.
Hosting needed to be changed. Those of you who were around back in 2002 can remember how dog slow the site was. Like latex paint drying slow because of nearly non-existent bandwidth. On the other hand, how the hard drive got too full because of the hosts config, and then lost data in the infamous SimHQ Crash of ’02. Then there were 72 hour downtimes on weekends because the host — which was located in a very rural area — closed the doors at 5 PM sharp and tech support was an answering machine they’d check “first thing” Monday AM. I could go on and on — and have — about the host back then. In fairness to them, SimHQ was their biggest client, and they liked the traffic, and probably did what they could. However, the final straw was the “Dear Doug” letter where they decided to double their rates on all customer sites the next month. It was (past) time to move.
The next “opportunity” was the server itself. Even though Bubba had built a new server the year previously, it was already wheezing trying to run the site, the message boards, the mail server, the SQL server, and God knows what else.
Then we come to the real eye-watering decisive moment. Rebuilding the message boards and migrating nearly 1.6 GB of data of flat file software. Field training for the local bomb squad could be no more tedious. Do you remember those “we’ll be back after maintenance which should take about 9 hours” weekends? That was why. If the diagnostics wheezed, part of the way through, it was start over time. If it corrupted a table, it was rebuild time. Not pretty.
Why not abandon the boards, go to new software, and start all over? It was considered. However, because we believe one board complete is better than two half-baked ones; the decision was made to wait until the successor of the board software was finished in development. Also because the migration to any other forum software we investigated — and we looked at all the leaders — would have been more of a nightmare than staying with what we had and waiting for the developers to finish. Therefore, we patched and updated the old board. Faithfully making backups “just in case,” and planning on more 9-hour maintenance sessions.
Server Building Time
The good news was I had built enough systems to spec and put together two nice new servers. The bad news was I was near clueless about networks. Therefore, I had to hire out the help to a friend who had just started a new tech support business. In 2005, SimHQ did what every fledgling new business does with a problem — throw money at it initially. The two new servers proved to be the easiest part of the journey. Coupled with the fact I retrieved the Bubba built server from the previous host, we were set hardware wise for the time being. Or so I thought.
Hosting Follies
You know when you get a “great” idea, and you check all the details out but it does not work? When I moved hosting from the rural location, I thought… “Check into hosting this at home”. I knew other people who hosted their sites at home. They put in dedicated power, added some ducting to a spare cool bedroom (i.e. server room) and with a healthy cable account and a UPS, what could go wrong? It was a hobby site after all and not an e-commerce site. Those of you who are sys admins are already shaking you are head. Yeah, and a few of you are near tears from laughter, right? Okay, enough with the suspense. Can you say heat? Can you say enough heat that we thought we had added a sauna? Oh yeah… and the pipe we got was 4 times bigger than was handling the site before… but it was not big enough now. Therefore, those nice new high-powered dual Xeon systems were sitting in a sauna and pulling bandwidth through a straw. We still no clue why the bandwidth was so far off unless the host gave us bad readings. That scenario lasted less than a month back in early 2005.
I searched around frantically for a collocation site that was affordable. No luck. Fortunately for me and SimHQ, Frank “Dart” Giger suggested a site in Dallas that he uses for his IL-2 server. I contacted them, struck an agreement, loaded up the car with hardware, and headed for Dallas on the weekend so we would be back online Monday (we hoped). The colo situation in Dallas worked great for quite a while except for one big problem: the distance. When something needed hand’s on, it was a 9-hour round trip drive. As much as I loved the folks at that colo, I had to look back towards something closer. Keep in mind the trick-or-treat forum software was acting-up periodically, and that made for some tense moments. So now SimHQ was about to move to a fourth hosting situation within two-and-a-half-years.
It was the first time I wished I would have said “no” to Bubba just one more time.
Before making the move back to a newly located colo in Houston, we did a few months of testing with the new SimHQ TeamSpeak and Game Server. After running near perfect for 4 months with no downtime, I decided it was time to make one last trip to Dallas to grab the site servers, and move them back to their new and final home. Hope springs eternal. Along with the move, we would also do the forum software switch to the new, finally released version of UBB Threads 7. Things were looking-up.
So a few months ago, we shut things down in Dallas, and moved the whole deal back south. The weekend was uneventful and we were back online Monday. Trouble is it went toosmooth. I’ve told this part of the story in the forums before, but for the 5 of you who don’t know, the very next week after the server was in-place, and after testing the site for 4 months with the game server, the HVAC system died, the backup cooling died, and servers started flopping like chickens. Of course, the high-tech host with the terrific creds had no windows to open in their modern high-rise office complex. The temperature outside was in the 90°F range. When the temperature rose to 140°F, the servers shut down. Rather, the SQL server shutdown and that corrupted the forums data, which took the server offline. All this within a few hours. Our tech was going crazy trying to secure data — ours was not the only customer he had there. The host advised they had to have the HVAC service ‘make some parts.’ This was becoming too much like the old joke of parts arriving by pack mule from a distant country. Through tenaciousness, our tech managed to get the site server back online and cool enough until we could find another hosting location. That now makes 5 in three years.
Two weeks ago, we relocated and some of the remaining bugs were fixed this past week (like sending members an email when a PM arrives). Still a couple more data items to resolve, but considering the heat related problems, the fact that every backup pulled of the forums was damaged from weeks back, and the fact that we almost lost PanzerMeyer’s 26,000+ post existence, things are once again looking-up. A special thanks to Rick Baker, UBB Threads developer and Carl Phillips of CJP Services. Marine boot camp could not build character any better than what we have been through.
Onward, Through the Fog
So there you have it. Did I make mistakes? You bet. Did we learn some new things? Yep. Did we have lousy luck? Absolutely. Did you suspect this many problems had occured? Hopefully not. That is what we wanted.
We are coming-up on another crossroads. A major update will arrive for the forums in the next couple of months that promises to fix the search engine. What to do? Patch or leave it alone? I honestly do not know right now. For the time being, I would just like to enjoy things working for a little while. As all of you sys admins know, there will be another downtime.
So we will look forward to the next goal before the end of 2007, the resurrected downloads server to go along with (at least as of this writing) a healthy site server and a new game server. Then in 2008, we will finish-up SimHQ.net with realtime chat and SimHQ.tv. Yes, I bought that domain. I don’t like to overpromise, but that’s the plan. Now if the current servers will just behave.
Oh and by the way, Bubba and I never have had the chance to play co-op Falcon 4. Yet.
– “guod”
