用IPSet和Dynamic Blocklists加固你的服务器

As a dedicated server or virtual private server (VPS) owner, one important task is to defend against online attacks. IPTables allows a sysadmin to filter traffic by configuring the tables in the Linux kernel firewall. In this tutorial, I will discuss how to use IP sets with dynamic blocklists to better secure your server.

IP sets are a framework inside the Linux kernel, managed by the IPSet utility. It can be used on most servers except OpenVZ VPS. Compared to "vanilla" IPTables, using IP sets could improve efficiency when filtering hundreds of thousands of IPs. Unlike normal IPTables chains which are stored and traversed linearly, IP sets use indexed data structures to make lookups very efficient, even for large sets.

The methods discussed below should be "complementary" to your existing IPTables rules, but not to replace them. To learn more about how to create basic IPTables rules, search for related tutorials on the web, or see my guide on securing the Asterisk VoIP server.

As a side note, check out my tutorial on setting up your own IPsec VPN serverwith both IPsec/L2TP and Cisco IPsec.

To install IPSet (and Wget, will need later), use these commands.

For Ubuntu/Debian:

apt-get update  
apt-get install ipset wget  

For CentOS/RHEL:

# Enable the EPEL repository  
yum install epel-release  
yum install ipset wget  

How can we better secure servers with IPSet? The answer is to take advantage of dynamic blocklists maintained by various security research companies or groups on the Internet, and deny access to these "bad" IPs or IP networks. Here is an example collection of such lists:

Blocklists of Suspected Malicious IPs and URLs

Now, how to actually use these blocklists with IPSet, and update them at set intervals? I came across a very useful script on the Gentoo Forums, created by author "Bones McCracker". Here's a link to his original post, with some good explanations on IPSet. For your convenience, his first script is shown here, which works with the DShield Recommended Block List:

	#! /bin/bash
	# /usr/local/sbin/block
	# BoneKracker
	# Rev. 11 October 2012
	# Tested with ipset 6.13
	# Purpose: Load DShield.org Recommended Block List into an ipset in a running
	# firewall. That list contains the networks from which the most malicious
	# traffic is being reported by DShield participants.
	# Notes: Call this from crontab. Feed updated every 15 minutes.
	# netmask=24: dshield's list is all class C networks
	# hashsize=64: default is 1024 but 64 is more than needed here
	target="http://feeds.dshield.org/block.txt"
	ipset_params="hash:ip --netmask 24 --hashsize 64"
	filename=$(basename ${target})
	firewall_ipset=${filename%.*} # ipset will be filename minus ext
	data_dir="/var/tmp/${firewall_ipset}" # data directory will be same
	data_file="${data_dir}/${filename}"
	# if data directory does not exist, create it
	mkdir -pm 0750 ${data_dir}
	# function to get modification time of the file in log-friendly format
	# stderr redirected in case file is not present
	get_timestamp() {
	date -r $1 +%m/%d' '%R
	}
	# file modification time on server is preserved during wget download
	[ -w $data_file ] && old_timestamp=$(get_timestamp ${data_file})
	# fetch file only if newer than the version we already have
	wget -qNP ${data_dir} ${target}
	if [ "$?" -ne "0" ]; then
	logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
	exit 1
	fi
	timestamp=$(get_timestamp ${data_file})
	# compare timestamps because wget returns success even if no newer file
	if [ "${timestamp}" != "${old_timestamp}" ]; then
	temp_ipset="${firewall_ipset}_temp"
	ipset create ${temp_ipset} ${ipset_params}
	networks=$(sed -rn 's/(^([0-9]{1,3}\.){3}[0-9]{1,3}).*$/\1/p' ${data_file})
	for net in $networks; do
	ipset add ${temp_ipset} ${net}
	done
	# if ipset does not exist, create it
	ipset create -exist ${firewall_ipset} ${ipset_params}
	# swap the temp ipset for the live one
	ipset swap ${temp_ipset} ${firewall_ipset}
	ipset destroy ${temp_ipset}
	# log the file modification time for use in minimizing lag in cron schedule
	logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."
	fi

view rawipset_dshield.sh hosted with ❤ by GitHub

To save the script, click on "view raw" at bottom-right corner, Ctrl-A to select all, Ctrl-C to copy, then paste into your favorite editor. Alternatively, you can download it. In order to enable filtering using the IP set produced by the script, an IPTables rule must be added. One way to do this is to simply append these lines to the end of the script, and save it:

iptables -nL INPUT | grep "block src" &>/dev/null  
if [[ $? -ne 0 ]]; then  
  iptables -I INPUT -m set --match-set block src -j DROP
  iptables -I INPUT -s YOUR_IP_ADDRESS -j ACCEPT
fi  

(Update: Thanks to this Reddit user for suggesting to whitelist one's own IP in case your Internet Service Provider (ISP) appears on a list. I have added a rule to address this. Replace YOUR_IP_ADDRESS above with your public IP.)

Also, for added security you can change the target line to use "https":
(Note: May not work with some Wget versions! Test it before using.)

target="https://www.dshield.org/block.txt"  

After making the changes, save the script as e.g. /root/ipset_dshield.sh, mark it executable, and then test it. If everything works as expected, proceed to add a line to your /etc/rc.local so that the script runs automatically upon reboot. The last step is to create a cron job to fetch the list at set intervals and update your IP set. For example:

# /etc/cron.d/update_block  
# Global variables
SHELL=/bin/bash  
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin  
MAILTO=root  
HOME=/  
# Every 15 minutes, poll for update to dshield
# block list, and update firewall blacklist.
13,28,43,58 * * * * root /root/ipset_dshield.sh &>/dev/null  

To check the contents of the IP set we just created, run the following command. Note that this set was named block:

ipset list block  

Now, we move on to another useful script created by the same author, which works with the IPv4 Fullbogons List by Team Cymru. To help explain what a Fullbogon is, here's a quote from their project page:

"A bogon prefix is a route that should never appear in the Internet routing table. A packet routed over the public Internet (not including over VPNs or other tunnels) should never have a source address in a bogon range. These are commonly found as the source addresses of DDoS attacks.

Bogons are defined as Martians (private and reserved addresses defined by RFC 1918, RFC 5735, and RFC 6598) and netblocks that have not been allocated to a regional internet registry (RIR) by the Internet Assigned Numbers Authority.Fullbogons are a larger set which also includes IP space that has been allocated to an RIR, but not assigned by that RIR to an actual ISP or other end-user ...

It is important to realize that the bogon and fullbogon lists are NOT static lists ... make sure that you have a plan for keeping your filters up-to-date, or within a short space of time you will be filtering legitimate traffic and creating work for network administrators everywhere. This is especially true for the fullbogons list, which has significant changes every day."

Again, for your convenience here is the script:

	#! /bin/bash
	# /usr/local/sbin/fullbogons-ipv4
	# BoneKracker
	# Rev. 11 October 2012
	# Tested with ipset 6.13
	# Purpose: Periodically update an ipset used in a running firewall to block
	# bogons. Bogons are addresses that nobody should be using on the public
	# Internet because they are either private, not to be assigned, or have
	# not yet been assigned.
	#
	# Notes: Call this from crontab. Feed updated every 4 hours.
	target="http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"
	# Alternative source: https://files.pfsense.org/lists/fullbogons-ipv4.txt
	ipset_params="hash:net"
	filename=$(basename ${target})
	firewall_ipset=${filename%.*} # ipset will be filename minus ext
	data_dir="/var/tmp/${firewall_ipset}" # data directory will be same
	data_file="${data_dir}/${filename}"
	# if data directory does not exist, create it
	mkdir -pm 0750 ${data_dir}
	# function to get modification time of the file in log-friendly format
	get_timestamp() {
	date -r $1 +%m/%d' '%R
	}
	# file modification time on server is preserved during wget download
	[ -w ${data_file} ] && old_timestamp=$(get_timestamp ${data_file})
	# fetch file only if newer than the version we already have
	wget -qNP ${data_dir} ${target}
	if [ "$?" -ne "0" ]; then
	logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
	exit 1
	fi
	timestamp=$(get_timestamp ${data_file})
	# compare timestamps because wget returns success even if no newer file
	if [ "${timestamp}" != "${old_timestamp}" ]; then
	temp_ipset="${firewall_ipset}_temp"
	ipset create ${temp_ipset} ${ipset_params}
	#sed -i '/^#/d' ${data_file} # strip comments
	sed -ri '/^[#< \t]|^$/d' ${data_file} # occasionally the file has been xhtml
	while read network; do
	ipset add ${temp_ipset} ${network}
	done < ${data_file}
	# if ipset does not exist, create it
	ipset create -exist ${firewall_ipset} ${ipset_params}
	# swap the temp ipset for the live one
	ipset swap ${temp_ipset} ${firewall_ipset}
	ipset destroy ${temp_ipset}
	# log the file modification time for use in minimizing lag in cron schedule
	logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."
	fi

view rawipset_bogons.sh hosted with ❤ by GitHub

To save the script, click on "view raw" at bottom-right corner, Ctrl-A to select all, Ctrl-C to copy, then paste into your favorite editor. Alternatively, you can download it. Append the lines below to the end of the script to add the required IPTables rules. Note that I put three extra rules with RETURN to exclude certain private IP networks. Please carefully review them and remove those not in use on your server:

iptables -nL INPUT | grep "IP4BOGONS" &>/dev/null  
if [[ $? -ne 0 ]]; then  
  iptables -N IP4BOGONS; iptables -F IP4BOGONS
  iptables -A IP4BOGONS -s 10.0.0.0/8 -j RETURN
  iptables -A IP4BOGONS -s 172.16.0.0/12 -j RETURN
  iptables -A IP4BOGONS -s 192.168.0.0/16 -j RETURN
  iptables -A IP4BOGONS -m set --match-set fullbogons-ipv4 src -j DROP
  iptables -I INPUT ! -i lo -j IP4BOGONS
fi  

As mentioned in the quote above, it is essential that you keep the Fullbogons list up-to-date. Here's an example cron job:

# /etc/cron.d/update_bogons  
# Global variables
SHELL=/bin/bash  
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin  
MAILTO=root  
HOME=/  
# Every four hours, poll for update to ipv4-fullbogons
# block list, and update firewall blacklist.
55 3,7,11,15,19,23  * * * root /root/ipset_bogons.sh &>/dev/null  

So far I have presented two IPSet update scripts, which can be used with the DShield Block List and Team Cymru Fullbogons List, respectively.

Another security research company maintains the ETOpen Ruleset, which is also free (as in beer). View the current rules. To use it with IPSet, see example update scripts. I currently use the second one by Thomas Mueller, but had to fix a few bugs to make it work.

Follow this link to view my modified version of the update script:
(Replace YOUR_IP_ADDRESS at the end with your actual public IP)
https://gist.github.com/hwdsl2/7977b49f7257fe516b60

And here is an example cron job to update this ruleset:

# /etc/cron.d/update_em  
# Global variables
SHELL=/bin/bash  
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin  
MAILTO=root  
HOME=/  
# Every four hours, poll for update to the emerging threats
# ETOpen block list, and update firewall blacklist.
55 3,7,11,15,19,23 * * * root /root/ipset_em.sh &>/dev/null  

To learn more about the detailed usage of IPSet, read the manual page or search for "IPSet tutorial" on Google.

Finally, you can list the IPTables rules we added with these commands. Check the relevant counters after some time and they should increase, which indicates successfully blocking of "bad" traffic using the lists:

iptables -nv -L INPUT; iptables -nv -L IP4BOGONS  

In addition to those discussed above, there are other useful blocklists such as this one by StopForumSpam (example IPSet script). While I have not personally used this or any of the other lists, you can certainly try to adapt the update scripts mentioned in this article to work with them.

This concludes my discussion on using IPSet and dynamic blocklists to protect your server. I hope you will find it useful.

from https://blog.ls20.com/securing-your-server-using-ipset-and-dynamic-blocklists/