Problems with this weekend's upgrade

[Davehudsn]

I am shocked that the entire BlueHost community is not up in arms (at least those affected by this weekend's server move). For the first time, i sent an email to Matt@bluehost.com and i haven't heard zip back. Below is my email. It looks like the user community needs to speak up on this one to get a response:

Subject: Feedback on this weekend's Server Upgrades

Matt,

I have been a customer of BlueHost for many years now and have multiple accounts with you. I have always admired your company, your support, and the way you proactively communicate and handle situations.

I have never sent in an email before, but I felt that I needed to based on what was performed this weekend with the “server/account moves”. There are a couple of aspects of this weekend that went horribly wrong:

1) Lack of communication: BlueHost sent out an email notification, literally 8 hours or so before the upgrade. This does not give us time to prepare/backup our sites.
2) Lack of Understanding of the changes: It is quite clear that your team did/does not have a clear understanding of what impact their changes had this weekend. There are three distinct examples:
a. Shell access broken: When the move was done, it turned off all shell access for those accounts moved.
b. “Home” Directories changed: For example, my root is now “/home6/hdscorp/” instead of “/home/hdscorp/”. This pretty much broke every site, file repository, etc that I have with EVERY domain I have hosted. I have spent all morning trying to clean this up as all my sites “relative path” changed.
c. Configuration Changes: During the move, it looks like you pushed out new (or overwrote our existing) php.ini files in our root directories. This has also caused major problems when applications start breaking.

In short, I think you owe us all an explanation of why this was so suddenly done and why it was so poorly thought out. As always, your support staff was very responsive and helpful. I am sure they will soon be overwhelmed, and they will not be able to fix all problems for everyone. I have always recommended BlueHost to everyone I meet, but if this is an indication of future operations, I will have to rethink my future hosting need.

Regards,
Dave Hudson

[felgall]

Firstly, BlueHost are moving servers progressively over a long period so only a few servers will have been moved on the same weekend as yours.

The server my site is on hasn't been moved yet (as far as I know unless they managed it without any downtime) but I read about the proposes upgrades on Matt's blog quite a while ago - which is the only reason I have any idea what you are talking about.

[JamesT2]

I didnt receive any communication and only realised the upgrade of the sql stuff when I checked the server status.

I have noticed that on box 346 it is unreliable and giving 503 errors for more than 15 minutes in any one time...

Its very frustrating having a site that a lot of people are complaining that they cannot get access....

I have sent tickets which never get answered and this never used to be the case....

found out that the outages are caused by 2 sites not behaving... on the same box ...

[Early Out]

For problems like this, do not submit tickets. Use Live Chat.

[markowe]

For problems like this, do not submit tickets. Use Live Chat.

I am no big-time user, but I have noticed a drastic increase in downtime lately. I certainly haven't received any information about any server moves/transfers in advance - isn't it normally customary to send an email in such cases? If anyone knows my email address it's Bluehost.

I recently recommended Bluehost to a friend - I certainly hope that BH is not going downhill and that I gave him the wrong advice...

[felgall]

With over 1,000,000 domains hosted BlueHost would have difficulty emailing everyone to tell them about changes. By the time that all the emails went out the changes would be long finished.

Matt does mention what sorts of things are happening/about to happen on his blog which is linked from the BlueHost homepage so if you read that every so often then you know much of what their plans are without their having to send over a million emails.

With the new faster servers BlueHost should hopefully be even better than it has been.

I haven't noticed any downtime at all in many many months so their service certainly isn't getting any worse than in the past. With over 1000 servers any problems with downtime would affect less than 0.1% of BlueHost customers even when the problem does affect everyone on the server.

[alligosh]

Even with Matt's blog mentioning it, the server migrations should not be having those issues. I'll talk to my guys about it; it really is unacceptable for these kinds of issues to be happening to customers, even for a major upgrade.

-Steve

[Bob Barr]

Even with Matt's blog mentioning it, the server migrations should not be having those issues. I'll talk to my guys about it; it really is unacceptable for these kinds of issues to be happening to customers, even for a major upgrade.

-Steve

Thanks for posting, Steve. It's really good to hear something official on the forums.

I encountered a minor issue on a recent server move. When the move was performed, I lost my shell access. While it only took a few minutes to get this issue resolved through the Live Chat support system, I wonder if there shouldn't be an item on some sort of checklist to restore shell access for all of the server's users who have previously had it enabled. (Had it not been for the shell access issue, I might not have known that the move had even taken place.)

[alligosh]

I wonder if there shouldn't be an item on some sort of checklist to restore shell access for all of the server's users who have previously had it enabled.

They have a checklist in place, and they even spent several weeks on developing a suite of scripts to test and check after a migration to make sure everything is still working as it should. This is the first I have heard of the shell not still being active after the move, and sounds like a bug that has either recently popped up or nobody has complained more than a short chat session to get it fixed. It will definitely be added to the list.

These are really big upgrades that the guys are doing, and they bring a lot of really new and fast hardware to the boxes (from two CPU cores to eight, four times the memory in the box, and in many cases faster I/O and a separate mysql spindle. Some will be getting SSD drives for the mysql space as well. At the same time, the upgrades reduce the energy footprint per user, which means less cooling needed and less cost to run (meaning that the upgrades will pay for themselves in the long run) and less impact per user on the environment. It is a great win all around.

At the same time, it is #1 importance to effect the users as little as possible. We are constantly updating the test suite to catch anything new we find is breaking.

-Steve

[Bob Barr]

Thanks for your response, Steve. Since I don't use shell access all that often, I only thought to check it after seeing another forum member mention that he had lost his shell access after a server move. Depending of the proportion of user accounts who have shell access enabled, this may appear as a random, sporadic issue to the support folks.