As I mentioned in a couple of posts last week, and for anyone who visited our site or any of our clients sites during this time, you may have known we had something of an outage. But more and more details are emerging about what really happened.
Our web hosting company (our OLD web hosting company) wanted to migrate us to a new server. We have a dedicated server that we use for our own site and for many client sites. We probably have around 130 or so sites on it. The new server, we were promised, would have better hardware. Our sites would be faster. It all sounded great.
On Monday morning, the migration was complete and we changed the nameservers. The nameservers are what point a domain to the IP where the websites are. The IP is just the location of the computer on the Internet that hosts that site. After the nameservers were changed, everything on our site went down. All our clients went down. Not “went down” actually, went to bad sites. Spam sites. Diabetes sites. Malware sites. Not just us, everyone. And we couldn’t figure out why. Our files were untouched, they were all there where they were supposed to be. Our domains just weren’t pointing to them.
When we would do a DNS lookup on our domains, they would show a weird IP, an IP that we hadn’t configured, that was clearly where all these spam sites were coming from. But that IP was nowhere in our configurations. It was a disaster.
I have a sysadmin who sometimes helps me out with my server, so I asked him to look. We spent most of the day searching for the problem. And then he found it. And it appeared to be just a typo. Instead of sumydesigns.com for the nameservers, they had put what appeared to be a typo version of our domain name. I am not sending any traffic to that site, so you won’t find a link or reference to that domain here. But it was just one letter off. We quickly changed that for every site, and the sites slowly but surely started coming back. This was Monday night, and the hope was that by Tuesday morning, all would be well.
But Tuesday morning, things were messed up again. Upon looking, changes had been made in the server configurations, opening it up for hackers. The IP was changed again. Someone had been in there again messing with us.
So immediately, I knew we needed a new web host, but moving 130+ sites between dedicated servers is no fast feat. I bought a new dedicated server with a big name web hosting company and we started migrating sites after fixing the immediate problems on the server. We spent all day moving, and by late Tuesday night, we hit go on the new server, hoping that by Wednesday morning, everything would finally be all right.
But Wednesday, many clients still couldn’t get to their sites. Email was still being problematic and hardly anyone in Texas could get to our websites. Turns out the people who hijacked my typo-domain name were smart, and set the TTL (time-to-live) on the nameservers to two weeks. Just FYI, a normal TTL is 30 or 60 minutes. So those bogus IPs were going to take up to two weeks to get cleared out of all the DNS servers.
It was a catastrophe of epic proportions. And while I take full responsibility for the problem, it was not at all my fault. I didn’t set up the new server. I didn’t put in the typo. I didn’t open the server up. And while I’ve done everything humanly possible to fix it as fast as possible, I’m at a point where my hands are tied and there is nothing I can do but wait. And for those who know me personally, you know that I am not the kind of person who does well with a “wait and see.” I need to fix things.
My initial instinct was that this was an honest mistake. Why move us to a new server (and we verified, it was a new server, it was in a new datacenter, and the hardware was good) just to screw us up?
But then we discovered something. The typo version of my domain was registered on Monday. On the very same day that we went live on the new server. And as much as I want to believe this was an accident, there’s no possible way that someone would buy that domain, set it up to go to spam/malware sites on the EXACT same day we launch a new server that just happened to have that typo configured in it.
It’s too perfect, and I don’t believe it. We were set up for this. We were hijacked. And I’ve never felt more violated in my life.
Unfortunately, we don’t have any way to prove who the responsible party was. The WHOIS information for that domain is hidden, and there’s likely no way to get it. Our suspicion is that someone at the old web hosting company who was involved in setting up the new server is responsible. But how do you verify that?
I would name the web hosting company here, because I think people should know what happened. But at this point, it would quickly devolve into a he said/she said, and I don’t think that would be productive for anyone involved, especially if we uncover more details and investigate possible litigation.
But suffice it to say, we no longer host with that company. We’ve migrated everything, and terminated all accounts with that company.
I say we were hijacked and not hacked, because nothing was hacked on the websites. My site was never compromised. Our clients sites were never compromised. The domains were just not pointing to the actual sites… they were pointing to spam sites. Not that this is any better – actually, it may have been worse because a hacked site I can fix in a matter of hours, and this has dragged on for a solid week and could be another week before it’s totally resolved.
What’s the lesson to be learned here? First, this could happen to anyone. After digging deeper into it, we’ve found we aren’t the first people to be hijacked in this way, and we probably won’t be the last. The offenders who did this have been around for several years it seems. Second, it’s important to find a reputable and solid web hosting company. The company we left last week was one we’d been with for two years, and had been happy with during most of this time. But clearly, we were mistaken in our choice.
Finally, never, ever migrate to a new server on a Monday morning. That is just begging for trouble!
Leave a Comment
Join our list!
Our blog, delivered to your inbox. Never miss a post!
I hope you can press charges and recover some of your cost.