UPDATE 13/5/2008: The migration is all but finished. There will be another outage in the  early hours of Thursdy morning and perhaps some tuning over the next few days. Otherwise, we're up and running on the (third) new server and you should notice a huge improvement in performance plus a few of the previously disabled features are back. Huge thanks to Mark and Adrian for all their work on this! Apart from a hiccup between 4-6am  this morning the switchover went very smoothly last night. This update is just scratching the surface of major changes that will be following later this year. Stay tuned for more announcements! 

UPDATE 8/5/2008: Nearly a month since we started migrating the site and we're still having difficulties, I'm afraid. My apologies again to all concerned, some of the problems simply couldn't have been foreseen. If we had a little more wriggle room (time, cash, hardware on hand) we could have planned more redundancy into the process, but we're relying on highly skilled volunteer labour and scraping together hardware which doesn't leave any room for test, staging and failover environments. A third server (count 'em!) was ordered at the beginning of this week and was originally due to go in last night but didn't make it. Fingers crossed for tonight. Once it's installed in the machine room it will probably be a wait overnight while I migrate the actual site to the new server and get it running (again!).

What happened to server number two? I was on holiday on the south coast for two weeks and not much use with the hardware side but I gather it rather spectacularly took out an entire rack of servers and a power supply when it was plugged in. It's taken some time to clean up the mess and sort out a third new server.

I appreciate the patience people have demonstrated. The site is very frustrating to use at present, with much of the functionality disabled to try and improve performance and even then, it's excruciatingly slow. I guess after ten relatively reliable years with a very, very few outages of more than a hour or so we were bound to strike trouble.

But don't despair, far from going away, I expect we'll be bouncing back in the next week or so and can assure you that the we'll be taking things to a new level before the end of the year!

Did I mention that in the midst of all this and a fortnight away with the family that I'm tackling a field of 42 job applicants for 14 new positions in my regular day job managing 550 websites for a university? Yes, I have a day job too.

UPDATE 11/4/2008: Do I need to tell anyone it has been a rocky week? Apologies to all. I seriously underestimated the load that you all place on the web server hardware. As a result the site has been virtually unavailable for much of the week - struggling to serve a few pages per second when we regularly peak at 40+ pages per second.

My sincere thanks to Mark and Adrian who are beavering away behind the scenes trying to get a very much more capable server online - hopefully early next week. In the meantime, this evening I've made some fairly extensive modifications to the site that appear to have resulted in some marginal improvements. Keep your fingers crossed!

6/4/2008: We moved out of the old server and into the new at approximately 9pm (AWST) on Sunday 6th April 2008. There was a brief outage of half an hour or so. A couple of minor hiccups as we went across and there is some ongoing tuning happening as I speak - nervously checking server load between sentences!

Huge and sincere thanks to Mark Tearle for steering this tricky process towards a successul conclusion -  he's been at my place for the last 7 hours working on this. Special thanks to Stephen Darragh, ex-Informed Technology, who delighted us again by responding within minutes to an email request on a Sunday evening!

Keep your fingers crossed!

Cheers
Grant 

Grant Malcolm's blog | email this blog | 1196 reads