Serious problems with box 22, after IP migration

[SessionDrummer]

Hello, for almost two days now, ever since box 22 was migrated, I am having serious problems with my site, and forums. At times, it just competely stalls out. I have contacted BH, and supposedly this should have been resolved AT the time of completion, after the migration. I have been running quite a few trace routes and consistantly the last hop or two, is where I am getting massive latency. I am still waiting for a response to my new open tickets, which clearly show/reference the problem at BH's end. Has anyone else on box 22 experienced this, and/or got it resolved ??

Thank you.

S.D.

[Filmosity]

I don't know what box I'm on, but I've noticed a severe spike in the time it takes to not only load my site, but also ftp to it. What gives?

[Early Out]

I don't know what box I'm on, ....Since problems can be confined to one box, and since different boxes experience different problems, it's always good to know what box you're on. Login to the Control Panel, and take a look at your shared IP address. The box number is the last octet in that address. So, if your IP is (making something up) 70.123.35.92, you're on box 92.

[SessionDrummer]

Kudos to the president for taking an active stance on this issue. I'm quite sure that the following, has everything to do with my problems, and quite a few others.


Dear Bluehost.com Customer,

I am writing today to give you some important information
regarding your Bluehost.com account. As many of you know
Bluehost.com uses Cpanel, a third party product, as our control
panel to allow customers to manage their sites. While this is
normally a good thing, it also takes much of the control away
from us and sometimes forces us to live with deficiencies that
we would normally not tolerate. Lately Cpanel has become so slow
and cumbersome that despite buying some of the highest end hardware
available some sites are sluggish at best and extremely slow at
worst. This was never acceptable to us and we tried to work with
Cpanel to resolve our issues, but it is clear to us that our issues
will never be at their forefront. So what do we do?

Yesterday was my breaking point. A customer called in to complain
of a slow site. I checked it myself and the server load showed it
was running fine. Clearly it wasn't, and I wanted to know what was
causing his issues. To make a long story short Cpanel was hitting
every server's hard drive array close to 50,000 times per day more
often than it needed to (Based on my rough calculations). Our
server monitoring was not doing its job and our customers bore the
brunt of these issues. There wasn't an easy fix and our servers
were all grinding to a halt. We had to implement a custom fix and
it had to done immediately. The main issue causing these problems
was mail related. About 70% of all users had to have their mail
moved to a new system and converted to a new storage method. I
made the difficult decision to execute these fixes immediately
without notifying all our customers first because of a couple of
reasons. First, to mail all our users with our current system takes
approximately 18 hours (We could do faster, but then many emails
would be denied by large ISPs because too many emails come form a
single source too quickly), and second because it was causing such
a severe problem with all our servers that it simply couldn't be
delayed. It was causing so many sites to run poorly that we acted
as fast and effeciently as we could.

This made many users email temporarily unavailable. No emails should
have been lost in the transition. It simply wouldn't allow you to
log in while mail was transitioned from one platform to another.
Again, we greatly regret having to do this without prior notification,
but the benefits were exactly what we hoped for.

While all server issues aren't immediately resolved, the VAST majority
of problems we were seeing on all servers are virtually gone. You
should now see a SUBSTANTIAL improvement in almost all areas of your
site including mail issues, script execution time, and overall
responsiveness of your hosting experience.

Next week we will provide a link to a user monitoring service that will
allow you to monitor your server (From our perspective) to see basically
what we see, and know if issues you are experiencing originate on our
end or if some other factor is involved. We have more speed improvements
being worked on aggressively right now and in the near future those
details will be made available as well.

I know this email is lengthy, and I congratulate those that made it this
far :) I just wanted you to know that I care about what happens to
your sites, and not just because you pay me to care. I don't get to
be as technically involved as I used to be because of trying to run the
company, but I realize now that a change is needed. I may not be the
best admin out there, but I care the most and will always be your advocate
to making your hosting experience better.

All users (All 210,00 of you) can ALWAYS email me directly at -
matt@bluehost.com

Thanks for your support,
Matt Heaton / President Bluehost.com

Let's hope this fixes it. :)

S.D.

[SessionDrummer]

Have sent another message to tech. support, as the problem hasn't been fixed, and is getting worse.


Tracing route over a maximum of 30 hops:


13 12 ms 14 ms 13 ms 12.119.53.49
14 19 ms 17 ms 16 ms tbr1-p012401.phlpa.ip.att.net [12.123.137.202]
15 17 ms 21 ms 18 ms tbr1-cl8.n54ny.ip.att.net [12.122.2.17]
16 15 ms 16 ms 16 ms ggr1-p330.n54ny.ip.att.net [12.123.0.81]
17 17 ms 16 ms 16 ms 206.111.13.33.ptr.us.xo.net [206.111.13.33]
18 18 ms 16 ms 17 ms p5-0-0.RAR1.NYC-NY.us.xo.net [65.106.3.37]
19 40 ms 42 ms 41 ms p6-0-0.RAR2.Chicago-IL.us.xo.net [65.106.0.29]
20 39 ms 41 ms 40 ms p0-0-0d0.RAR1.Chicago-IL.us.xo.net [65.106.1.85]

21 63 ms 63 ms 64 ms p6-0-0.RAR2.Denver-CO.us.xo.net [65.106.0.25]
22 62 ms 111 ms 83 ms p0-0-0d0.RAR1.Denver-CO.us.xo.net [65.106.1.73]

23 89 ms 90 ms 92 ms 65.106.1.78
24 91 ms 91 ms 91 ms p0-0.CHR1.SaltLake-UT.us.xo.net [207.88.83.42]
25 88 ms 87 ms 88 ms 67.110.119.194.ptr.us.xo.net [67.110.119.194]
26 * 1115 ms 1174 ms box22.bluehost.com [69.89.18.22]

Trace complete.



Tracing route over a maximum of 30 hops:


13 12 ms 17 ms 14 ms 12.119.53.49
14 17 ms 15 ms 17 ms tbr1-p012401.phlpa.ip.att.net [12.123.137.202]
15 18 ms 16 ms 17 ms tbr1-cl8.n54ny.ip.att.net [12.122.2.17]
16 14 ms 14 ms 17 ms ggr1-p330.n54ny.ip.att.net [12.123.0.81]
17 17 ms 16 ms 16 ms 206.111.13.33.ptr.us.xo.net [206.111.13.33]
18 15 ms 19 ms 16 ms p5-0-0.RAR1.NYC-NY.us.xo.net [65.106.3.37]
19 44 ms 52 ms 39 ms p6-0-0.RAR2.Chicago-IL.us.xo.net [65.106.0.29]
20 42 ms 39 ms 42 ms p0-0-0d0.RAR1.Chicago-IL.us.xo.net [65.106.1.85]

21 71 ms 64 ms 63 ms p6-0-0.RAR2.Denver-CO.us.xo.net [65.106.0.25]
22 65 ms 70 ms 62 ms p0-0-0d0.RAR1.Denver-CO.us.xo.net [65.106.1.73]

23 93 ms 90 ms 94 ms 65.106.1.78
24 90 ms 89 ms 91 ms p0-0.CHR1.SaltLake-UT.us.xo.net [207.88.83.42]
25 87 ms 86 ms 86 ms 67.110.119.194.ptr.us.xo.net [67.110.119.194]
26 2022 ms 1960 ms 1654 ms box22.bluehost.com [69.89.18.22]

Trace complete.

I have omitted the first few hops, but one can clearly see, once it gets to box 22, BOOM. Anyone have an in here ??

S.D.

[fedor]

Oh, the IP addresses:eek:

[SessionDrummer]

Sorry maybe I wasn't clear in the end of my message, was asking if anyone had an "IN" with Bluehost, so I could get the tech guys, to go and slap box 22 around a bit. You can see by the trace routes, that every hop up untill the last, which is box 22, are normal. Once it hits it, the latency (pings, delays), are extremely high, showing some major issues out there. Any board moderators lurking about, that can give me a hand here ??

Thank you.

S.D.

[Early Out]

Try the tracert again. It looks like it's been fixed - box22 has a different IP address, and the routing has changed significantly.

[SessionDrummer]

Something still seems to be up on 22. I'll keep checking it.


25 87 ms 86 ms 86 ms 67.110.119.194.ptr.us.xo.net [67.110.119.194]
26 * 915 ms 868 ms box22.bluehost.com [69.89.18.22]

Trace complete.



25 87 ms 88 ms 88 ms 67.110.119.194.ptr.us.xo.net [67.110.119.194]
26 421 ms 511 ms 496 ms box22.bluehost.com [69.89.18.22]

Trace complete.





25 87 ms 88 ms 86 ms 67.110.119.194.ptr.us.xo.net [67.110.119.194]
26 259 ms 179 ms 263 ms box22.bluehost.com [69.89.18.22]

Trace complete.



25 90 ms 86 ms 87 ms 67.110.119.194.ptr.us.xo.net [67.110.119.194]
26 272 ms 288 ms 366 ms box22.bluehost.com [69.89.18.22]

Trace complete.

Is there any official word as to what's going on, beyond the CPanel issues ??

S.D.

[Early Out]

Interesting - I get to box22 via a different route, entirely (notice that last hop) and take a look at the IP address for box22!

C:\Documents and Settings\Owner>tracert box22.bluehost.com

Tracing route to box22.bluehost.com [70.96.188.22]
over a maximum of 30 hops:

1 * * * Request timed out.
2 7 ms 7 ms 7 ms ge-2-8-ur01.reston.va.bad.comcast.net [68.87.130.21]
3 8 ms 9 ms 9 ms te-9-3-ur02.arlington.va.bad.comcast.net [68.87.128.161]
4 7 ms 9 ms 9 ms te-9-1-ur01.arlington.va.bad.comcast.net [68.87.128.157]
5 9 ms 9 ms 9 ms te-9-3-ur02.alexandria.va.bad.comcast.net [68.87.128.141]
6 9 ms 10 ms 8 ms te-9-1-ur01.alexandria.va.bad.comcast.net [68.87.128.133]
7 9 ms 9 ms 9 ms te-8-1-ur01.capitolhghts.md.bad.comcast.net [68.87.128.138]
8 13 ms 12 ms 12 ms te-8-1-ur02.capitolhghts.md.bad.comcast.net [68.87.129.150]
9 10 ms 9 ms 9 ms te-9-2-ar01.capitolhghts.md.bad.comcast.net [68.87.129.145]
10 9 ms 9 ms 9 ms pos-2-2-cr01.ritchieroad.md.core.comcast.net [68.87.16.161]
11 9 ms 9 ms 9 ms 12.118.122.5
12 12 ms 11 ms 11 ms tbr2-p011701.wswdc.ip.att.net [12.123.8.30]
13 11 ms 11 ms 9 ms gar1-p390.ascva.ip.att.net [12.123.8.53]
14 11 ms 10 ms 16 ms 12.118.132.34
15 12 ms 12 ms 11 ms so-2-0-0--0.cr02.mcln.va.frontiernet.net [74.40.4.109]
16 * 23 ms 23 ms so-0-0-0--0.ar01.atln.ga.frontiernet.net [74.40.2.46]
17 51 ms 50 ms 50 ms so-0-0-0--0.cr01.dlls.tx.frontiernet.net [74.40.2.41]
18 49 ms 50 ms 50 ms so-7-0-0--0.cr02.dlls.eli.net [207.173.115.118]
19 87 ms 88 ms 88 ms so-0-0-0--0.cr02.phnx.eli.net [207.173.113.113]
20 80 ms 91 ms 90 ms p6-0.cr02.lsvl.eli.net [207.173.114.165]
21 93 ms 93 ms 93 ms so-1-0-0--0.cr02.slkc.eli.net [207.173.114.38]
22 94 ms 92 ms 93 ms ge-7-0-0--0.gw02.slkc.eli.net [207.173.113.26]
23 86 ms 86 ms 86 ms gw0-cust-BLUEHOST-COM.slkc.eli.net [70.97.59.22]
24 87 ms 88 ms 88 ms box22.bluehost.com [70.96.188.22]

Trace complete.

Experiment time: try changing your DNS server, and see if, by resolving to a different IP address, your routing changes. You can use the OpenDNS servers: 208.67.222.222 and 208.67.220.220.

[CausalCrunch]

I'm on box 98. I'm seeing a blank screen whenever I go to a page that's dynamically loaded (PHP,CGI). The source code for the page is:
---
<!-- SHTML Wrapper - 500 Server Error -->

-----
Is this what you're all seeing?

[Early Out]

I'm on box 98. I'm seeing a blank screen whenever I go to a page that's dynamically loaded (PHP,CGI). The source code for the page is:
---
<!-- SHTML Wrapper - 500 Server Error -->

-----
Is this what you're all seeing?
Huh? This thread is about high latency times on box22. How does that relate to a 500 Server Error on box98?

[CausalCrunch]

Was trying to figure out what that symptoms were to see if it was the same problem on a different box. But I think I've figured out that someone was doing a denial of service on one of my scripts.

[SessionDrummer]

Situation Update on Box 22:

Not sure what they did, but the problems, stalling, and unuseable speed have all been fixed. To be fair to BH, had to put that up. I appreciate the hard work they put into it, and I am back to lightning fast Blue Host speeds. :)

S.D.

[damianJ]

I have the same problem expect on box 148

http://www.kdmelectric.com/images/tracert.gif