Introducing CrawlProtect in Securing your WordPress site

Introducing CrawlProtect in Securing your WordPress site

Security is important when managing WordPress websites. It is one of the key components for long term website success. WordPress has some security measures that can be implemented with the use of plugins. It’s not recommended to edit the WordPress core files because of security risks, so it make sense to implement the security features in terms of plugins.

One downside of plugin implementation is that it adds significant strain on the existing WordPress server resources. Some plugins that were badly written can significantly slow down your WordPress website performance.

Thus, one of the feasible recommendations to increase WordPress security while not slowing down the site is using third-party/open source security software like CrawlProtect. In this method, CrawlProtect runs independently from WordPress core and won’t slow down your site while it prevents hackers from getting into your server.

While WordPress is secure (if you continually update it), adding another wall of security using CrawlProtect can really help in hardening your site against hacking.

Recommended Requirements for Running Crawlprotect

The following are the server and software requirements before you can use Crawlprotect:

1.) Apache web server
2.) .htaccess enabled server
3.) mod rewrite enabled server
4.) Latest WordPress version (version 3.4.1 as of July 2012)
5.) MySQL database 5.0 +
6.) PHP version 5.0 +
7.) Linux/Unix operating system used by the server – CrawlProtect utilizes the Unix permissions such as CHMOD in protecting your WordPress site. It is important that your server is running the supported OS.

Other aspects although not a requirement are as follows:

1.) Disable any current security plugins you are using. This will interfere with CrawlProtect implementation and can drastically slow down your WordPress website.

2.) You have not edited any of the WordPress core files. These are the files inside your wp-admin, wp-includes and files in the root directory except the following:

a.) wp-content (your themes and plugins, etc.)
b.) robots.txt
c.) .htaccess
d.) wp-config.php

Before implementing CrawlProtect, it is highly recommended to back up your WordPress site and databases. Although in my own experience, CrawlProtect installation is straightforward, smooth and simple; it is always the best practice to backup your files and databases before implementing some major change in your site.

Download and Upload Crawlprotect to your Server

Follow the steps:

1.) The first step is to download the latest CrawlProtect version. As of July 2012, the latest release is version 3.0.0.

2.) Unzip the contents. When extracted, you should see a folder named as “crawlprotect3-0-0”. Go inside this folder. You might see two folders namely as:

a.) crawlprotect

You will only use crawlprotect folder. If you go inside the crawlprotect folder, you will exactly see the contents as follows:

3.) Before uploading the crawlprotect folder containing the above scripts as shown in the screenshot, it is important to rename this folder to something that is not obvious to your site as well as to the crawlprotect application. Supposing you will rename it as “websiteknight” (don’t use this name though, just an example).

Rename the folder from “crawlprotect” to “websiteknight”.

4.) Upload “websiteknight” to your WordPress root directory, the same path with wp-content, wp-includes and wp-admin, etc. This is how it looks like in your server after being uploaded:

Install CrawlProtect

1.) First create a MySQL database that can be used by crawlprotect. Important data will be stored in this database. Creating the database follows the exact same steps when you are creating your WordPress database.

After creating the crawlprotect MySQL database, take note of the following connection parameters, you will be needing this during installation:

a.) MySQL username
b.) MySQL password
c.) MySQL hostname
d.) MySQL database name

The database can easily created using cPanel provided by your web host.

2.) Clear all of your browser cache and enter this URL in your browser:

Since the CrawlProtect directory is now named as “websiteknight”, then it is accessible in the browser as:

3.) In the installation screen, select the language. In this tutorial, let’s use English and click OK.

4.) Click the “Install” button.

5.) Enter the database username, password, hostname and database name. Please double check the data very carefully before pressing OK.

6.) As of this step, CrawlProtect should be installed in your server without any warnings. However in some server due to permission issue you might see an error “Automatic file creation failed”. Simply follow the Crawlprotect instructions to create two files, namely “connection.php” and “cppf.php” and put that in the “include” folder of CrawlProtect.

7.) Finally the installation is complete if you see “Connection files OK” and “Tables creation OK”. Click “Go” button to start the configuration of CrawlProtect.

CrawlProtect Basic Configuration

1.) Review the website and site url. The site URL is the canonical homepage URL, for example if your site is using non-www version as the official URLs, then it will be:

Otherwise include the www if your website is using www version:

2.) Click OK, and you should be able to see the “The website has been added to the database”. Click Go.

  1. ) You will be required to set up the administrator login. Enter the desired login and password. Assign a very strong password. I recommend at least 16 characters or more, you can create it using Keepass such as shown in this screenshot:

4.) Click OK and login as administrator.
5.) Click “Create or update the .htaccess”.

6.) The default proposed setting is OK, you can tweak it later. Scroll down to the bottom of the page and click “Create my .htaccess file”. The Crawlprotect rules will be integrated to your existing .htaccess, scroll down and click “Put the htaccess in place”. For most web host, the htaccess would be updated without problems. In case of permission errors, simply copy and paste the resulting .htaccess code to your existing WordPress .htaccess (don’t forget to back up your .htaccess first).

CrawlProtect Testing and Fine Tuning

Below are some tests to make sure Crawlprotect is working properly for your site:

1.) Logout from CrawlProtect administration panel.
2.) Try an injection test on the browser to see if CrawlProtect can block it, use this format:

You should be able to see the warning: “This site is protected by CrawlProtect”. If you see it, CrawlProtect is now protecting your WordPress site.

3.) Download your existing WordPress .htaccess and add the following line:

RewriteCond %{REQUEST_URI} !^(.*)wp-login(.*)$ [NC]

Add the above line, next to this:


This line would allow the administrator to access the wp-admin directory normally.

4.) Try to login to your WordPress admin. There should be no errors.

Known limitations and Workaround

There are times when CrawlProtect blocks URLs in your WordPress site if you have certain applications/plugins that uses query string variables in the URL, for example:

Or like this:

The above URL will be blocked by CrawlProtect because of the presence of query string variables (used by hackers when getting into your site). As a result, the application won’t work anymore. The solution to this problem is really simple. You only need to add the file name to the .htaccess so that it will be allowed (not blocked).

In the above two examples, below are the .htaccess lines to be added to allow the application to work normally:

RewriteCond %{REQUEST_URI} !^(.)myplayer(.)$ [NC]
RewriteCond %{REQUEST_URI} !^(.)pusher(.)$ [NC]

Add the above lines next to this:

RewriteCond %{REQUEST_URI} !^(.*)wp-login(.*)$ [NC]

This is the screenshot of the modified .htaccess with the allowed file names to be used with a query string variable:

In case, you don’t remember all of those applications in your site; you can still monitor the valid URLs that are accessed by your normal website visitors but is blocked by CrawlProtect. This is done by logging into the CrawlProtect admin panel. You will see it under “Attacks log”.

One you have CrawlProtect working for around a month, it will now have sufficient data to learn some information about your site hackers and statistics, for example:

If you are not using CrawlProtect, it is highly possible that any of those 4280 SQL injection or 389 shell type hacking attempts can get through your blog. This is a one year data from one of the medium traffic WordPress websites.

It can also summarize the IP address of attackers and the number of times they attempt to hack your site. Highly abusive IP address can be permanently blocked from accessing your site.

What you can do more?

1.) You can easily upgrade CrawlProtect by replacing all files with the new version but make sure you have a backup of connect.php and your .htaccess. You will know if there is a new version by logging as Crawlprotect admin. In the dashboard homepage; click the button “Check if a new release of CrawlProtect is available”.

2.) You can double check the CHMOD of your site folders and files. Vulnerable folders and files for hacking will use CHMOD value of 777 (allowing the public or hackers to read and write the contents). CrawlProtect recommends some value that you can use for different types of folders and files in your WordPress site.

You can browse the current CHMOD settings by clicking “Folders and Files audit” in the CrawlProtect admin dashboard.

About the Author