vBulletin hacked forums: Clean Up Time

With recent vBulletin hacking going on, here are some of my personal suggestions from my personal experience in cleaning up the mess.

I have continuously updated and extended the info provided below, so be sure to check back regularly for more updates. You can use Chrome Page Monitor extension which allows you to be notified via your browser for any changes on specifically monitored pages such as this blog’s page(s). I have now broken the full guide into multiple pages for each step outlined below:

• Background
• Step 0
• Step 1
• Step 2
• Step 3
• Step 4
• Step 5
• Step 6
• Step 7
• Step 8
• Step 9
• Step 10
• Step 11

Background

• Some recommendations will assume you have a VPS or dedicated server and thus have SSH telnet root access to your server. The reason is some of the recommended actions required to clean up the infections require root access to your server or the tools recommended require such.
• The recommendations below do not claim to be a 100% method of cleaning up a hacked vBulletin forum and do not guarantee that you won’t be compromised again if you have other compromised web app scripts on your server or have other server level exploits in play. I provide no support for this other than to outline some of my personal recommendations as a starting point.
• If you are on shared hosting, then your web host provider would be the only one with sufficient privileges and access to take such steps or run such recommended tools.
• Regardless if you’re on VPS, dedicated, cloud or shared web hosting, I would contact and involve your web host provider in this clean up process at the earliest possible moment you find out that you have been hacked. The reason is, hacks could elevate to server level and your web host needs to be informed as they are the ones ultimately responsible for management or handling of your server/server security. Then point your web host to this guide so you can convey additional steps to your web host, if you do not have sufficient privileges or SSH telnet access to follow through with the recommended steps or use the recommended tools.
• Due to the nature of such compromises, it’s impossible for vBulletin tech support side to fully clean up every infection or malware compromise on your site/servers and thus require your web host provider’s assistance as well.
• In some instances there may have been multiple hacker instrusions at different times which I found with my vbplugincheck.sh tool outlined below.
• If any of the below suggested server end tasks are beyond your level of understanding, consult with your web host provider.
• Emphasis on the importance of keeping regular full web site and forum backups for your web files + MySQL database(s) and attachments/avatars if stored in filesystem. And keeping such backups locally as well as offsite on another server. Also don’t forget about the increased disk usage requirements for maintaining copies of your backups. So if full web site and forum backup takes 10GB and you want to backup daily and keep at least 8 copies, you would need an additional 8x10GB = 80GB of disk space. If you want to keep your compromised forum files and MySQL databases online as well as restore a recent clean backup set to compare against both compromised and clean files, then you would need at least 2x times disk space to handle 2x10GB = 20GB of data.

Also read the recent security announcements and vBulletin documentation:

• vBulletin 4.1.x-4.2.x & All versions of vBulletin 5 to remove /install folder and contents from your live forums: http://www.vbulletin.com/forum/forum…-1-vbulletin-5
• vBulletin 5.0.x patch released, for a different security issue: http://www.vbulletin.com/forum/forum…d-all-versions
• vBulletin Manual: Securing your vBulletin installation.

If you do not regularly check your vB forum’s AdminCP control panel for news alerts such as security news, you can use a browser plugin for page monitoring alerts. I use Chrome Page Monitor extension which allows you to be notified via your browser for any changes to any pages for any web site. i.e. vBulletin.com Forum announcement forum or your web hosting provider’s TOS page (when they change TOS). The extension even allows to show diff comparison changes of before and after state of the page you want to monitor. I do this for every online service I use i.e. AU/TOS pages etc. Tip: you can even use Page Monitor extension to monitor this very blog article to be notified for changes and updated info as well ;)

Immediate Actions:

Step 0. Close your forums temporarily. Step 0 stages A to M are also important so you can at a later date diagnosis and investigate the extent of the breach. You may need to temporarily reopen your forums for stages F (scan checks) and stage M (Google Webmaster Tools site verification and scans). Stages include:

A). As per announcement, delete (or if initially unsure rename) your vBulletin installation(s) /install (vB4) or /core/install (vB5) directories completely. If you have more than one vBulletin forum installed (including test copies of your live forums), you need to delete the install directories from those installations as well. If you have SSH and root user access on your VPS or dedicated server, you can use find command to find all directories matching *install* wildcard first. Then only remove the directories that are for your vBulletin forum’s install path for vB4 /install or vB5 /core/install.

find /home -type d -name "*install*" -print

or save to text file

find /home -type d -name "*install*" -print > listinstalldirectories.txt

B). Take a full MySQL database backup of all potentially compromised MySQL databases on server not just vBulletin forum. Quickest way to do this with SSH and root user access on VPS and dedicated servers is outlined in step 2 of my vBulletin 4 upgrade guide.

C). Take a full backup of all potentially compromised files in your web root of your account i.e. /home/username/public_html. Quickest way to do this with SSH and root user access on VPS and dedicated servers is outlined in step 2 of my vBulletin 4 upgrade guide or follow these steps. If your account is at /home/username/public_html quick backup to perserve your entire directory and file permissions and ownerships at /home/username/public_html_backup is to run command below (remembering you’ll use double the disk space as you’re making a 2nd copy):

cp -a /home/username/public_html /home/username/public_html_backup

If you have attachments outside of public_html i.e. /home/username/attachments

cd /home/username
cp -a attachments attachments_backup

D). Download those backups to your local computer for later analysis.

E). Find your Apache access and error log, system, PHP, FTP, SSH/secure log locations and save a copy of those logs for future reference.

F). Use a 3rd party malware online web scanner from Sucuri to scan your forum url(s) – this may not pick up all malware but it’s a start. Maybe interested in an article I wrote on malware infections and Sucuri’s services.

G). Disable vB forum plugin system forum wide by adding the following line to your includes/config.php file (core/includes/config.php for vBulletin 5 users):

define("DISABLE_HOOKS", true);

H). Find and check all files modified in the past X amount of days. So if vBulletin installed at /home/username/public_html/forums and you want to check and find files modified in past 14 days you would type:

find /home/username/public_html/forums -mtime -14

if you want to save output to a text file called modifiedfiles.txt, type:

find /home/username/public_html/forums -mtime -14 > modifiedfiles.txt

It’s quite possible that your vBulletin forums were compromised weeks or months ago and only now had hackers take advantage of it. So you may want to re-run above commands and alter the date from 14 days to 30, 60, 90, 180 and even 365 days.

find /home/username/public_html/forums -mtime -30 > modifiedfiles30.txt

find /home/username/public_html/forums -mtime -60 > modifiedfiles60.txt

find /home/username/public_html/forums -mtime -90 > modifiedfiles90.txt

find /home/username/public_html/forums -mtime -180 > modifiedfiles180.txt

find /home/username/public_html/forums -mtime -365 > modifiedfiles365.txt

So you now have a bunch of saved modifedfiles*.txt files with a list of modified files that span different ranges. This would allow more detailed analysis as well. You can use a file comparison tool to check for differences between each file.

I). Check for malicious .htaccess files on your server including above the site/account’s web root level. i.e. /home/* if your have site at /home/username/public_html. Hackers may have altered your .htaccess files with redirects etc. You need to check, analyse and compare your .htaccess files’ contents to ensure they are what you expect to have in them. The below command in SSH telnet as root user will find all .htaccess named files under /home directory and output their paths.

find /home -name ".htaccess" -print

save output to text file instead

find /home -name ".htaccess" -print > listhtaccessfiles.txt

Some guides explaining how to check .htaccess files here, here and here.

Now if you want to get creative on the command line in SSH, you can find all .htaccess files and spit out the .htaccess files’ respective contents with cat command and for loop like this:

for ht in $(find /home -name ".htaccess" -print); do echo $ht; cat $ht; done

Saving contents to text file

for ht in $(find /home -name ".htaccess" -print); do echo $ht; cat $ht; done > htaccesscontents.txt

If you want to filter each .htacces files contents by ErrorDocument, HTTP_REFERER, HTTP_USER_AGENT and RewriteRule entries you could do something like this (looking for common redirect tactics)

for ht in $(find /home -name ".htaccess" -print); do echo $ht; egrep -i '(ErrorDocument|HTTP_REFERER|HTTP_USER_AGENT|RewriteRule)' $ht; done

Saving contents to text file

for ht in $(find /home -name ".htaccess" -print); do echo $ht; egrep -i '(ErrorDocument|HTTP_REFERER|HTTP_USER_AGENT|RewriteRule)' $ht; done > htaccesscontents_filtered.txt

J). You can also check all files for iframe code too. Some might be legitimately used by vBulletin. So you will have to compare against a fresh set of vBulletin files

find /home -type f -exec grep -irl 'iframe' {} \; -print

save output to text file instead

find /home -type f -exec grep -irl 'iframe' {} \; -print > listiframefiles.txt

K). You can also check all files for base64 too. Some might be legitimately used by vBulletin. So you will have to compare against a fresh set of vBulletin files

find /home -type f -exec grep -irl 'base64' {} \; -print

save output to text file instead

find /home -type f -exec grep -irl 'base64' {} \; -print > listbase64files.txt

L). Checking all cronjobs on your server at server and user level for malicious code. To find all user level cronjobs in SSH telnet as root user type:

for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done

if you want to save output to text file

for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done > listusercronjobs.txt

For server level type:

crontab -l

if you want to save output to text files

crontab -l > listcronjobs.txt

Other server level cronjobs can be specified in the following files and directories.

• /etc/crontab
• /etc/cron.hourly
• /etc/cron.daily
• /etc/cron.weekly
• /etc/cron.monthly
• /etc/cron.d

M). Highly recommended, that you registered a Gmail email account and sign up to use Google Webmaster Tools and register your web site or forum domain name within Google Webmaster Tools as it has a Diagnostic page for regular reporting of any detected Malware infections to your web site. Step 4 overview for Webmasters help for hacked sites video has an overview for this service. Google Web Master Tools help articles has a few pages dedicated to malware clean ups.

• About malware and hacked sites
• Cleaning your site
• Report spam, paid links, malware, and other problems to Google
• Preventing malware infection
• Request a malware review of your site
• Open redirects

Google Security Online blog also have a video informational series for webmasters on how to recover a hacked web site. Direct link to Webmasters help for hacked sites videos and articles here.

Google Help for hacked sites Overview video:

Full list of Google Help for hacked sites video series:

1. Watch the overview Beginner
2. Contact your hoster and build a support network Beginner
3. Quarantine your site Intermediate
4. Touch base with Webmaster Tools Intermediate
5. Assess the damage (hacked with spam) or
6. Assess the damage (hacked with malware) Advanced
7. Identify the vulnerability Advanced
8. Clean and maintain your site Advanced
9. Request a review Intermediate