Building A High End Router From A Pentium II 266MHz PC

Synopsis
I recently upgraded my router from Debian Sarge (3.1) to Debian Etch (4.0). Debian is a fairly ideal platform for building routers. It includes all the software you need. The packaging is fantastic and most of it integrates fairly well. The Debian infrastructure is unsurpassed and so are its ideals. Here, I will share with you the steps required to build a professional router. I will also point out bugs and problems I had during the installation. Places where Debian could provide better integration and be more seamless.

Objective
My goal is to build a network appliance that provides a compliment of network services; but is also secure, fast (enough), easy to admin, and reliable.

Hardware Description
The system I used was an old Pentium II 266MHz with 128MB of RAM. This is not ideal especially with Gige or 10Gige connections. For one thing, the PCI bus cannot handle those speeds. And for another, the software becomes CPU bound with increasing complexity of rules, connections, etc. I have installed 3 10/100 NICs.

Network Description
My network consists of a DMZ, an intranet with both wired and wireless connections and a WAN (my high speed internet connection). Both the DMZ and intranet are connected to separate switches and I have a basic wireless access point (WAP) connected to the intranet switch for the wireless connections. The WAN connection is DSL.

Software Description
As mentioned before, the operating system used is Debian Etch (4.0). Packages will be described in the following sections.

Functionality Description
I have installed a slew of internet services. As this router is my only point of access to the network and my only primary network appliance, it will be responsible for ALL my network services. I will go into more detail later, but here is a summary of the router's functionality:

• Routing/Packet Inspection/Load Balancing/Masquerading (iptables/ipvsadm)
• Packet Shaping (tc)
• IDS (snort)
• DHCP (dhcp3-server)
• DNS (maradns)
• Dynamic DNS (ddclient)
• Time Synchronization (NTP)
• VPN (undecided)

Install And Basic Configuration
I won't belabor the install. There is a ton of information on the internet regarding Linux and Debian installs. What I wish to cover are the packages I install and some tweaks to make them work better.

Debian Package Manager
First we config the package manager then download and install the packages. As a general principle I do not install non free software or software that requires payment. So my sources list is pretty lean. I do, however, install the volatile archive to ensure transient packages remain up to date:

deb ftp://mirrors.kernel.org/debian/ etch main
deb-src ftp://mirrors.kernel.org/debian/ etch main
deb http://security.debian.org/ etch/updates main
deb http://volatile.debian.org/debian-volatile/ etch/volatile main

Packages
List of packages I generally find useful. You don't need them, but they can help make administration a joy instead of a chore:

pciutils usbutils hdparm sdparm hddtemp wireless-tools ethtool schedutils procps psmisc acct parted mount lslk lsof binutils elfutils
binstats less findutils bzip2 zip unzip grep file time diff colordiff
tshark iptraf tcpdump netcat traceroute ping telnet nmap wget curl vim

BASH
It's a matter of taste, but I find this change to /etc/profile useful:

if [ "$PS1" ]; then
if [ "$BASH" ]; then
PS1='\t [\u@\h \W]$ '
else
if [ "`id -u`" -eq 0 ]; then
PS1='\t [\u@\h \W]# '
else
PS1='$ '
fi
fi
fi

alias grep='grep --color'
alias ls='ls --color'
alias less='less -r'
LL="-MM -J"

It adds color and a better prompt.

File System Security
I install AIDE and made sure the database was installed correctly. This will keep tabs on any unauthorized changes to the system.

VIM
I add the following options to VIM to enable highlighting, etc. Makes for quicker troubleshooting and editing.

syntax on
if has("autocmd")
filetype indent on
endif
set showcmd " Show (partial) command in status line.
set smartcase " Do smart case matching
set incsearch " Incremental search
set hlsearch "
set backupcopy=no "
set ruler "


Logging
Logging is straightforward in Debian Etch. The quality of packaging is such that you don't need to change the logging unless you plan on forwarding to another host or introducing finer grained logs.

Email
I do not like logging into systems every day to check the status. In general I will rely on log checkers which send emails which are forwarded to my email server.

I prefer postfix, so remove exim and install postfix. Postfix needs a few options to set hostname, networks, etc. In general postfix is very easy to set up and administrate. However, due to some limitations of the system (which I will explain in the DNS section), you will have to disable DNS lookups:

myhostname = ix.teramari.us
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = ix.teramari.us, localhost.teramari.us, localhost
relayhost =
mynetworks = 127.0.0.0/8
disable_dns_lookups = yes
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = loopback-only

This setup only allows local deliver, only accepts connections locally, and disables dns lookups. When you disable dns lookups postfix will stop relying on a DNS server and rely exclusively on /etc/hosts. As you will see, I populate the local servers in /etc/hosts because using the dns server as a local cache for the router won't work.

Network And Device Setup
This system has three network cards. The DMZ will get its own subnet as will the intranet. The third NIC, used for the WAN, will not need a subnet. The WAN will be configured to use PPPoE and receives a dynamic IP each time a connection is made.

Install the pppoe and pppoeconf packages. Follow the instructions to setup your connection. When you are done, test the connection and make sure it works. In your /etc/ppp/peers/dsl-provider file, make sure the following options are set:

persist
usepeerdns

This ensures your connection is on 24/7 and forces the router to use external DNS for lookups instead of the local caching server. As it turns out, the dynamic IP will cause a lot of grief for you if you're not careful. Etch is greatly improved over Sarge in this respect, but some work still needs to be done (again, this will be explained later).

Backup
I have chosen backupninja and rdiff-backup for my backup mechanism. Not only will this automate my backups and allow me to backup off server, it will allow me to store diffs over time. I have not setup the backup at this time becaues my other server needs to be upgraded. I will return to this at a later time.

Remote Administration And Management
At this point you should have a secure, reliable and capable machine before you, practically begging you to compute :)

I will describe your remote administration options before configuring the services, as you will need them during the router setup. Unfortunately, there are not a lot of great options in Debian. This is the first major failing of Debian Etch. With the removal of webmin, there is no GUI interface for remote admin. I'm not saying webmin is great (it's okay), but it would be nice if there was something. For the most part you will either be on a console, or using SSH. On a side note, I always install sensord on my machines to monitor hardware. My lowend desktop hardware usually has poor monitoring implementation, but with some tweaks you can usually get some feedback. If anything it might help detect a thermal event or allow you to determine why your machine went up in flames.

Debian Etch Failure #1: No remote GUI administrative tools.

SSH
SSH has a good set of defaults, except that it allows root to connect. I always disable this. It's particularly important to do this because a router connected to the internet will be a huge target for SSH scanners.

Debian Etch Failure #2: Remote login on SSH by default.

I also change the port to 23. This is because I want my web/shell server to be accessible from the internet. Later on, a port scan will reveal 22 and 23 open, both serving SSH on different machines.

Port 23
PermitRootLogin no


Time Management
Install ntp and allow it to synchronize time on the router. Also, make it available to your network as well. Add the following options to /etc/ntp.conf:

broadcast 10.100.255.255

Be sure to change the broadcast to match your network.

DHCP
Installing DHCP will allow the computers on the intranet to obtain IPs. I usually need this right away so I install DHCP first. I use the dhcp3-server package. The debian package has two files you'll need to modify. The first is /etc/default/dhcp3-server. In here you will need to define which interfaces it will listen on. The other file is /etc/dhcp3/dhcpd.conf. Here you tell the DHCP server what IPs to give to whom. dhcp3 is very flexible, but I only need it to service the DMZ and the intranet. Technically I don't need the DMZ, but I include it just in case I'm troubleshooting with a laptop or something. Here are some of the more important options:

option domain-name "teramari.us";
default-lease-time 86400;
authoritative;
subnet 10.100.20.0 netmask 255.255.255.0 {
range 10.100.20.10 10.100.20.200;
option routers 10.100.20.1;
option domain-name-servers 10.100.20.1;
}


dhcp3 needs a domain name. You also need to define how long a lease will last. A high number is good. If authoritative is not set, then the server will not issue IPs, so it's a good idea to do this. The last section is an example subnet. It defines a range of IPs, default gateway and DNS servers for an interface.

IDS
Linux includes a great IDS called snort. Snort scans incoming data for problems and emails a report when it detects a possible problem. Snort uses a rules database so that must be kept up to date. To do this you'll need to install oinkmaster and set up an account on the snort webpage. Snort requires two config files. Here is the content of /etc/snort/snort.debian.conf:

DEBIAN_SNORT_STARTUP="boot"
DEBIAN_SNORT_HOME_NET="10.100.0.0/16"
DEBIAN_SNORT_OPTIONS=""
DEBIAN_SNORT_INTERFACE="any"
DEBIAN_SNORT_SEND_STATS="true"
DEBIAN_SNORT_STATS_RCPT="root"
DEBIAN_SNORT_STATS_THRESHOLD="1"


In /etc/snort/snort.conf, you need to set the HOME_NET and EXTERNAL_NET interfaces so snort knows what's coming and what's going.

var HOME_NET 10.100.0.0/16
var EXTERNAL_NET !$HOME_NET


Setting up oinkmaster is a snap. After you've setup your account you'll receive a URL with your special code in it. Paste that into the url of your /etc/oinkmaster/oinkmaster.conf and you'll be set.

That's 90% of what you'll need, with the exception of the rules database setup. Once you'll setup oinkmaster to download new rules, you'll need to tell snort that in /etc/snort/snort.conf. Here is what I used:

include $RULE_PATH/local.rules
include $RULE_PATH/bad-traffic.rules
include $RULE_PATH/exploit.rules
include $RULE_PATH/scan.rules
include $RULE_PATH/finger.rules
include $RULE_PATH/ftp.rules
include $RULE_PATH/telnet.rules
include $RULE_PATH/rpc.rules
include $RULE_PATH/rservices.rules
include $RULE_PATH/dos.rules
include $RULE_PATH/ddos.rules
include $RULE_PATH/dns.rules
include $RULE_PATH/tftp.rules
include $RULE_PATH/web-cgi.rules
include $RULE_PATH/web-coldfusion.rules
include $RULE_PATH/web-iis.rules
include $RULE_PATH/web-frontpage.rules
include $RULE_PATH/web-misc.rules
include $RULE_PATH/web-client.rules
include $RULE_PATH/web-php.rules
include $RULE_PATH/sql.rules
include $RULE_PATH/x11.rules
include $RULE_PATH/icmp.rules
include $RULE_PATH/netbios.rules
include $RULE_PATH/misc.rules
include $RULE_PATH/attack-responses.rules
include $RULE_PATH/oracle.rules
include $RULE_PATH/mysql.rules
include $RULE_PATH/snmp.rules
include $RULE_PATH/smtp.rules
include $RULE_PATH/imap.rules
include $RULE_PATH/pop2.rules
include $RULE_PATH/pop3.rules
include $RULE_PATH/nntp.rules
include $RULE_PATH/other-ids.rules
include $RULE_PATH/experimental.rules
include sid-msg.map
include $RULE_PATH/specific-threats.rules
include $RULE_PATH/spyware-put.rules
include unicode.map

DNS
On the previous implementation I used djbdns. I absolutely hated it. I know there are a lot of fans but I hated the interface and had a terrible time with failures. Maybe I was using it incorrectly, I don't know. There are only three other major players (Mara, Power and Bind), so I thought I'd give Mara a try. I installed the maradns package and like it quite a bit. It's simple and it works well. Unfortunately it isn't feature rich so I will probably try a different nameserver in the future. The config file is a bit picky, but once you understand that cvs2 is for the new file format and csv1 is for the old, you won't have any problems. Here is a sample of my mararc file:

hide_disclaimer = "YES"
csv1 = {}
# Note: authoritative DNS MUST be loaded from csv2 or it won't work.
csv2 = {}
csv2["teramari.us."] = "db.teramari.us"
bind_address = "10.100.10.1, 127.0.0.1"
recursive_acl = "10.100.0.0/16, 127.0.0.0/8"


The recursive_acl will identify which subnets maradns will perform hostname lookups for. bind_address sets the interfaces it will bind to. The csv2["teramari.us"] option says use the file db.teramari.us for the teramari.us domain.

Once you've set up the domain file, you intranet's DNS needs are complete. Maradns does not handle external DNS, for reasons we will see below. This leaves DNS requests for the router itself. I have three options. I can use external DNS, use mara, or use hosts files. If I use external DNS I will need a hosts file for the local machines. So using mara looks like a good idea. It will handle both the internal and external requests. This seems like the most logical and straight forward thing to do, but it appears that when an interface resets, maradns stops working until it is restarted. As a result the router must use the upstream DNS and the hosts file. Here is an example /etc/hosts file:

127.0.0.1 localhost
10.100.10.1 router router.teramari.us
10.100.10.2 mail mail.teramari.us


Debian Etch Failure #3: DNS needs to be more resilient and continue working even across interface resets.

Dynamic DNS
My ISP provides dynamic IPs. Obviously this is a problem if you are hosting any services because DNS will need to be updated everytime that IP changes. I use the dyndns.org service and debian provides a package called ddclient which does two things. First it will update dyndns.org when you run it. Second, it integrates with ppp and pppoe to automatically update dyndns.org when the interface resets.

This works well for external clients that need to access your publicly exposed network. But there are problems when local services need to use the local IP. There is no method that I know of to look up this information, so I've had to rely on hacks.

The first hack is a script which manually adds a hostname called ppplink to /etc/hosts. This hostname is maintained across connections and is unavailable when the link is down. This allows are local service to use the hostname ppplink instead of an IP which will undoubtedly change.

Debian Etch Failure #4: The new IP is not exposed to the system by ppp or by some other mechanism.

The second hack is a script which manually reloads services that fail when links are reset. Currently this includes ntp, iptables and ipvsadm. So, all three are restarted/reloaded everytime the link is reset.

Debian Etch Failure #5: Services like ntp, iptables and ipvsadm are not resilient across interface resets.

A third problem with dynamic IPs is related to using ddclient when the router uses its own caching nameserver for DNS. When the link is reset ddclient makes an attempt to update dyndns.org. Unfortunately this requires a DNS lookup. For some reason if you are using a local nameserver the lookup will always fail. I would like to look into this sometime. I'm not sure why the DNS fails, but if I tell ppp to use peerdns, then the ddclient update always works.

Firewall/Masquerading
Installation of my firewall rules was relatively straightforward. I have my own script which I place in /etc/init.d and another for ppp. I wish there was a package which allowed for better firewall management/setup, but I am not familiar with one. This is along the lines of no GUI tools.

Port Forwarding/Routing
Most of this is done in the firewall rules by marking packets. Install the ipvsadm package and install rules in /etc/ipvsadm.rules.

Packet Shaping
Same thing as setting up the firewall. Install the scripts and we are done.

VPN
At some point I will install VPN on the server and on my laptop. Because I have not upgraded my laptop to etch, I have elected to defer this project.

Conclusion
Debian Etch is not graceful with dynamic interfaces. It also lacks tools. However, it is a tenfold improvement over Sarge and I'm glad I did the upgrade. The system is far easier to manage than in the past and despite my complaints it is far more robust. For the most part, deploying a service requires little more than installing a package.