37 replies to this topic

[Saint]

We had better have a contingency plan in case the HostCell server disappears again.

All our communications seem to centre around the HostCell server. If that dies, we are all left in the dark. Nothing panics members more than being out of touch.

Perhaps the Support Team should keep members' email addresses, in a secure place apart from the HostCell server, so as to be able to email us and let us know what is happening if HostCell dies again.

[HCN-SeriesN]

We had the email address, Remember, it was new years eve, beside me and raul, No one else was working. So we did not have enough time to contact everyone. While I was trying to fix every thing back, Raul was working hard to keep other Nodes stable. Also We were busy with setting up some new nodes for business and semi-dedi hostings.
Our main mail system (For emergency use) is hosted on a separated node. How ever, We kept every one uptodate via twitter https://twitter.com/#!/hostcell and Facebook.
I am going to setup a dns redundancy and failover system for the main site by the end of this or next week. But Glad, No data was lost

[the_reaper]

HCN-SeriesN, on 03 January 2012 - 02:33 AM, said:

... How ever, We kept every one uptodate via twitter https://twitter.com/#!/hostcell and Facebook...

You should probably connect twitter and facebook (eg. get twitter to post your tweets to facebook). Some of us don't use twitter

[HCN-SeriesN]

I think I did changed the settings. How ever will check again.

[loremar]

Although I don't have a twitter, I was actually tuned in to hostcell's twitter for updates. When it was down the whole day I was like WHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATTTTTTTTTTTTTT!?!?!!!!!!!!!!!!!!!!!!!

Glad I found the twitter and know that seriesN's working on the issue.

[HCN-SeriesN]

Yes You guys can trust me on that. With me, Raul and David around, Things in no way can go wrong. (Well it did)

[HCN-Northern]

as i suggested to Series on FB yesterday a front page with info would have been good due to not many people realising hostcell has a fb page and twitter but MEH! i can only suggest

[Saint]

HCN-SeriesN, on 03 January 2012 - 02:33 AM, said:

... We kept every one uptodate via twitter https://twitter.com/#!/hostcell and Facebook...

Not everyone! Only those who thought to look there.

the_reaper, on 03 January 2012 - 04:35 AM, said:

... Some of us don't use twitter

Nor FaceBook. Especially those of us who have evaluated Google+ as better.

HCN-Northern, on 03 January 2012 - 09:54 AM, said:

as i suggested to Series on FB yesterday a front page with info would have been good due to not many people realising hostcell has a fb page and twitter but MEH! i can only suggest

And a properly professional suggestion, too! SeriesN must have been too stressed to realise the obvious merits of that idea.

As soon as I twigged that HostCell was going to be down for quite a while, I put up holding pages for my sites at another free host. It took less than an hour.

There is one thing I did learn that is useful. The old warnings that DNS changes can take 48-72 hours to propagate through the internet, now seem to be out of date. My DNS changes were effectively instant.

[mezzi]

sours, on 03 January 2012 - 02:22 AM, said:

We had better have a contingency plan in case the HostCell server disappears again.

All our communications seem to centre around the HostCell server. If that dies, we are all left in the dark. Nothing panics members more than being out of touch.

Perhaps the Support Team should keep members' email addresses, in a secure place apart from the HostCell server, so as to be able to email us and let us know what is happening if HostCell dies again.

Good idea..

[roderick young]

sours, on 03 January 2012 - 10:38 AM, said:

Not everyone! Only those who thought to look there.

My first impulse was not to look at twitter, but when I went to Google, I typed "hostcell" and one of the suggested terms was "hostcell.net down". That took me straight to twitter.

And by the way, whatever storage backup hostcell was using worked. I lost nothing, not even changes made the day before.

[loremar]

Wow so many people online.

. . .

Make the server down for almost two days to catch everyone's attention and make them actively post again.

Very clever.

[Saint]

loremar, on 03 January 2012 - 03:27 PM, said:

Wow so many people online.

. . .

Make the server down for almost two days to catch everyone's attention and make them actively post again.

Very clever.

Almost two days? And the rest! Correct me if I'm wrong, but member websites were down for over 68 hours.

[loremar]

sours, on 03 January 2012 - 03:58 PM, said:

Almost two days? And the rest! Correct me if I'm wrong, but member websites were down for over 68 hours.

I notice it was down a few hours after the end of new year's eve and back up on this afternoon January 3. Uhm, Ok you're right.

[HCN-SeriesN]

Hey look man, we are human too and taking the time into consideration It was really difficult for us to fix old server,setup new server and trying to keep the old servers stable.
I know communication could have been better, but we were not able to consider the options of setting up an custom page. Reason?
HostCell main site was created with DNS redundancy network. For some reason that also did not work.
I will try to setup some new system for the main site.

[gxjenkin]

roderick young, on 03 January 2012 - 03:07 PM, said:

My first impulse was not to look at twitter, but when I went to Google, I typed "hostcell" and one of the suggested terms was "hostcell.net down". That took me straight to twitter.

And by the way, whatever storage backup hostcell was using worked. I lost nothing, not even changes made the day before.

Yeah, I did the same thing and found you guys on Twitter. I raerly sign into twitter, but it gave me some info about the problems hostcell was having. You guys really rang in the New Year.

[Saint]

HCN-SeriesN, on 03 January 2012 - 05:07 PM, said:

Hey look man, we are human too...

We aren't doubting the work you put in to getting HostCell back! Thanks for your efforts.

We didn't really doubt that you were human either, despite your avatar. (A one-eyed penguin?)

There was a point to my suggesting a Contingency Plan. The first I knew that my websites were down was when somebody else emailed me to tell me. It doesn't matter so much with me, because my websites aren't earning money (yet), and I can take a joke, and over the years I have built up a loyalty to HostCell. With other people, it might be a Public Relations disaster. It might be an idea to have a plan to email all users in an emergency, so that at least HostCell is the first to let them know that their websites are down, and that action is in hand.

[HCN-SeriesN]

Yes, Unfortunately your email do not work a lot of time (something @london.com :?)

[loremar]

Yes emailing everyone would have been the best thing to do since everything is all tied up into one server. Twitter and Facebook is great but there's a good chance that people will miss it.

I suggest you do this next time. Maybe you can save a draft of some message informing everyone that you're working on the issue and just instantly send it to everyone in case a serious problem comes up. That would be like a professional thing that one like you can do.

[HCN-SeriesN]

Yes we are actually working with a script to automate such task. It is just that, we never had any issue in our 7 years in this industry.

[the_reaper]

roderick young, on 03 January 2012 - 03:07 PM, said:

And by the way, whatever storage backup hostcell was using worked. I lost nothing, not even changes made the day before.

I can second that. I scheduled a blog post for midnight (local time) not 2 hours before the server went down...that was still there once everything was back up.

sours, on 03 January 2012 - 03:58 PM, said:

Almost two days? And the rest! Correct me if I'm wrong, but member websites were down for over 68 hours.

When I last calculated it, it was 68 hours and a hand full of minutes.

Just to echo everyone others, sending out an email to us all would be very helpful. I usually notice outages pretty quickly (since I'm nearly always checking my sites and these forums), but if I'm asleep when it goes down, it'd be good to know in the morning (or early afternoon, in my case) that everything is/was down.
That aside, awesome work with getting it all running again seriesN. I'm sure it wasn't the easiest task with the pressure of potentially losing hundreds of potential customers.