mod_bw vs mod_bwshare vs mod_cband

I know there is a lot of confusion over what to use and how to use it. This is what I've been able to gather from lightly looking over the documentation:

mod_bw: Will limit every user to a specific amount of bandwidth or limit the amount fo bandwidth per file type or file size.
Once you've "apt-get install libapache2-mod-bw" and restarted apache2, you can enter one of the configurations below in the custom directives for each subdomain:

Limit every user to a max of 10Kb/s on a vhost:
<IfModule mod_bw.c>
BandwidthModule On
ForceBandWidthModule On
Bandwidth all 10240
MinBandwidth all -1
</IfModule>

Limit al internal users to 1000 kb/s with a minimum of 50kb/s , and
files greater than 500kb to 50kb/s.
<IfModule mod_bw.c>
BandwidthModule On
ForceBandWidthModule On
Bandwidth all 1024000
MinBandwidth all 50000
LargeFileLimit * 500 50000
</IfModule>

Limit avi and mpg extensions to 20kb/s.
<IfModule mod_bw.c>
BandwidthModule On
ForceBandWidthModule On
LargeFileLimit .avi 1 20000
LargeFileLimit .mpg 1 20000
</IfModule>

mod_bwshare: Will limit how much bandwidth can be consumed with in a specified amount of time per IP from a specific directory or the whole site.
Once you've "apt-get install libapache2-mod-bwshare" and restarted apache2, you can enter one of the configuration below in the custom directives for each subdomain:

<IfModule mod_bwshare.c>
BW_tx1debt_max          25
BW_tx1cred_rate         0.095
BW_tx2debt_max          3000000
BW_tx2cred_rate         2500
</IfModule>
* I won't attempt to explain these parameters here, but you can find explanations here: http://www.topology.org/src/bwshare/README.html#config

mod_cband: Will limit users’ and virtualhosts’ bandwidth usage. The current versions can set virtualhosts’ and users’ bandwidth quotas, maximal download speed (like in mod_bandwidth), requests-per-second speed and the maximal number of simultaneous IP connections (like in mod_limitipconn)limit how much bandwidth can be consumed with in a specified amount of time from a specific directory or the whole site.
Once you've "apt-get install libapache2-mod-cband", and restarted apache2, you can enter one of the configurations below in the custom directives for each subdomain:

limit to 256k per user, 3 requests per second, 3 open connections and if they exceed they go down to 128k until they stop clicking on links for a few seconds:
<IfModule mod_cband.c>
CBandRemoteSpeed 256Kbps 3 3
CBandExceededSpeed 128kbps 2 3
</IfModule>
More examples here: http://dembol.org/blog/mod_cband/faq/

How to get all your mail going out of one IP address with POSTFIX

If you have multiple IPs (ETH0,ETH0:0, etc) running on a server with Postfix, chances are you are going to run into the problem of Postfix not using the mail IP that has the matching MX record to send mail.

When this happens most of the mail servers will label your mail as SPAM and/or reject it.

You can force Postfix to use the main servers IP address by adding the following lines to your /etc/postfix/main.cf:
smtp_bind_address = 1.1.1.1

*You would replace 1.1.1.1 with the main IP of your own server.
Restart Postfix with /etc/init.d/postfix restart

This might cause your mail to backup on the server if your have other services  that might need a configuration tweak to continue working correctly, check your mail queue with this:
postqueue -p

If you see mail backing up in that queue for failed connections to 127.0.0.1, then you might have to start making some tweaks. If not, then you should be OK.


If you have AMAVIS running on the server then you might need to make this tweak for AMAVIS to properly work. Create the file /etc/amavis/conf.d/60-custom. Add the following line:
@inet_acl = qw( 127.0.0.1 [::1] 1.1.1.1 );

Restart AMAVIS with /etc/init.d/amavis restart

Now check the Postfix queue with postqueue -p again. If there is any mail still there try flushing it out with postqueue -f. The recheck it after a couple of seconds with postqueue -p again. It should be all gone now.

Keeping the Botters, China and miscreants away.

On a daily basis I troll my FAIL2BAN mail folder to see who is "ding-dong diving", "n*gger knocking" or what-ever the term of the day is.

My Fail2Ban is setup to send me mail every-time it bans an IP. Repeat offenders get the firewall treatment. I run a filter to move the to a separate folder of the same name.

I end up using IP tables to block the repeat offenders. They are just wasting your precious resources.

This is what I use so far:

iptables -I INPUT -m iprange --src-range 107.160.0.0-107.160.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 111.72.0.0-111.79.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 114.96.0.0-114.103.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 120.192.0.0-120.255.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 121.254.128.0-121.254.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 163.177.0.0-163.177.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 183.0.0.0-183.63.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 185.60.229.0-185.60.229.255 -j DROP
iptables -I INPUT -m iprange --src-range 187.1.16.0-187.1.31.255 -j DROP
iptables -I INPUT -m iprange --src-range 189.213.72.1-189.213.79.254 -j DROP
iptables -I INPUT -m iprange --src-range 194.63.142.0-194.63.142.255 -j DROP
iptables -I INPUT -m iprange --src-range 219.234.174.0-219.234.174.191 -j DROP
iptables -I INPUT -m iprange --src-range 222.184.0.0-222.191.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 27.255.64.0-27.255.95.255 -j DROP
iptables -I INPUT -m iprange --src-range 59.99.0.0-59.99.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 60.166.0.0-60.175.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 61.176.0.0-61.176.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 80.82.64.0-80.82.79.255 -j DROP
iptables -I INPUT -m iprange --src-range 81.17.88.0-81.17.95.255 -j DROP
iptables -I INPUT -m iprange --src-range 89.248.174.0-89.248.174.127 -j DROP
iptables -I INPUT -m iprange --src-range 93.174.88.0-93.174.95.255 -j DROP
iptables -I INPUT -m iprange --src-range 192.99.0.0-192.99.255.255 -j DROP
iptables -I INPUT -m iprange --src-range 122.228.228.0-122.228.228.15 -j DROP
This has reduced my Fail2Ban folder from 45+ emails a day to 10 at most.

I still get the occasional flurry, but you firewall the IP range and your done.

I can handle sending out 10 abuse mail forwards a day.

Disabling root ssh is very good Idea

Found a few good sites with too much information: They all boil down to giving SUDO access to one of your other accounts and disabling ssh access for the root account. Your can still login as root through your datacenters provisioned KVM since it emulates a local keyboard and video display. In case you ever get locked out or your IP banned for some reason.

As a failsafe, you might consider putting the public IP of a trusted source in the
/etc/hosts.allow file in case you get locked out accidentally. That why you know that all you need to do is go to that place to restore access to your account. It could be your office's public IP, or campus university public IP, or just another server you have control over.

Once you have SUDO access on another account:
Just edit /etc/ssh/sshd_config and make sure this line PermitRootLogin no , reads as so.

Upgraded most of my boxes to Debian 7 Wheezy and ISPCONFIG

For the most part I'm happy to be on Debian 7 with ISPCONFIG, as DTC was taking forever to update their code to make it Debian 7 compatible; and my clients were installing software requiring the latest PHP and MYSQL. So I really didn't have much of a choice in the matter.

I like and dis-like a few things about ISPCONFIG.

LIKES:

• Offers and Handles multiservers very well.
• easy to install guides
• well documented
• many howto guides for just about every option
• cheap billing module

DISLIKES:

• creating a website or sub-domains does not automatically create a DNS entries/mail domains/etc, you have to expect your customers to know what they are doing. It's not as user-friendly in that aspect. It would be nice if it offered a dummy mode or wizards for users. In this aspect DTC was nice to do everything for you when you added a domain. Then you just had to do any customizations required, but the basics where created for you.

Limiting recursive lookups in Bind

Limiting recursive lookups in Bind is a good idea for several reasons:

1. Why give free DNS service to the entire internet.
2. A source of denial of service flood attack.
3. Increases traffic, which may increase costs.
4. Increases CPU usage.

In /etc/bind/named.conf.options:
 acl "trusted" {
     111.222.333.444/55;
        127.0.0.1;
     localhost;
     localnets;
 };

options {
        directory "/var/cache/bind";
        auth-nxdomain no;    # conform to RFC1035
        additional-from-auth no;
        additional-from-cache no;
        allow-query { any; };
        allow-recursion { trusted; };
        allow-query-cache { trusted; };

};

Binding postfix to use a specific ip address.

Now-a-days with SPF records and such, it's always a good idea to limit postfix to use the ip address specified in the spf record to prevent your mail from being marked as spam. This is only necessary if your server has more that one ip bound to it. 

In /etc/postfix/main.cf add:
smtp_bind_address = 111.222.333.444

This will limit your postfix server to using the ip specified.

Postfix management / Mail administrator primer

I thought I'd write a little primer with some of the most important commands you will use as a mail administrator.

Most servers will use postfix. So I will concentrate on postfix centric commands.

How to check your mail queue:
# postqueue -p

This will all mail that is in the queue waiting to be delivered. This is usually the first place I go to check if someone is sending spam from my server.

Once you find a domain that is sending out spam, you should probably stop your mail server.

Shutdown postfix:
# /etc/init.d/postfix stop

At this point you should probably "plug the hole". What I mean by that is that you should locate the compromised email account and change the password, and notify the user.

If is being generated by the site itself, it will usually say the mail is from "PHPMAILFUNCTION@xzy.com". In that case, you will need to contact the site administrator for that site, and disable the MAIL function in the .htaccess file (if your server supports it).

One you plug the hole, you'll want to delete all the mail in the queue that is coming from that site. Use the following command (replacing example.com with the offending domain):

# find /var/spool/postfix/* -type f -exec grep -irl "example.com" {} \; -exec rm -vf {} \;

Re-run # postqueue -p to double-check you've got it all.

Once you are satisfied your eliminated all the spam in the queues, you can restart the mail server with:
# /etc/init.d/postfix start

That's it, you are good to go. Next I will write an article about accessing the damage done to your IP reputation.

Setting I/O priorities for jobs

Sometimes you need to make a large copy/backup of a couple of hundred gigs. The problem comes in the form that nothing else can get done because that job is using 100% of the I/O. Then your utilization goes though the roof because everything is getting backed-up and nothing can get done.

You can install IOTOP to identify which job is the one using up all the IO. Once you have it identified you can run IONICE to change the priority of the job.

For example, I want a certain process (PID 12404) to only use I/O when no other process requires it, because the task is I/O-heavy, but is not high priority:

# ionice -c3 -p 12404

How limit postfix from sending email certain domains in Debian Squeeze

Well all know those certain users that you know they got that job because they are related to someone. And, most of them tend to send mail to made up domains.

This is how to limit them from sending to domains that don't exist.

Open up:
nano /etc/postfix/maps/header_checks

Add the following line per your requirements:
/^To:.*@hotmail\.com\.mx/ REJECT This domain does not exist.
/^To:.*@gmail\.com\.mx/ REJECT This domain does not exist.

You can easily change To: to From: and block receiving emails from those domains as well.

Here is a guide that might help you with other things you might want to block: http://www.akadia.com/services/postfix_uce.html

Refurbished Supermicro 2U rackmount server bought from Ebay

I originally had a Dell T300 as out in-house server with four 1tb Sata 7.2k drives in a RAID-5 configuration. It's an ageing box with 3.16ghz XEON processor and not a hell of a lot of space to grow.

I saw this ad on Ebay:
Supermicro 2U 12-Bay 2.6Ghz Quad Core 8GB SATA Server, 3Ware 9550SX-12LP Raid

 

I was able to bid the seller down to 500 bucks, and with free shipping, you can't beat it. The seller even said he would throw in the IPMI for free. With 12 drive slots it has plenty of room to expand. Dual-Quad core XEON processors, rails, I can migrate my registered memory from the old server for a total of 24 gigs of ram.

Once it arrives at my collocation center(scheduled to be deliver on the 23rd of this month), I will be migrating my drives (and any memory that will fit) from the old server to the new unit.

Naturally, I will be installing Debian 6.0.7 Squeeze. Why not Debian 7 you say? Because unfortunately, my control panel of choice, DTC, does not work with Debian 7 yet. I guess one day I will get a new fancy control panel with integrated billing and provisioning, but I've yet to feel the need to have a single point of failure for my entire business. My hostbill is doing fine, and my support ticket system is doing pretty good too. If it ain't broke, don't fix it, I say; unless you're getting a significant improvement in some area.

Installing monitoring, cli and configuration software for a PERC5 raid controller on Debian Squeeze

Please add deb http://hwraid.le-vert.net/distrib branch main to /etc/apt/sources.list to access all packages.
distrib can be either debian or ubuntu.
branch can be lenny, squeeze, wheezy and sid for debian, or hardy, intrepid, jaunty and lucid for ubuntu.

In example, for current Debian stable release (Squeeze):
deb http://hwraid.le-vert.net/debian squeeze main

Theses packages are available for amd64 and i386 architectures. Sources packages are available as well (replace deb with deb-src).

Once you add the repo above do:

apt-get install megacli megaclisas-status megactl

To view the original doc: http://hwraid.le-vert.net/wiki/DebianPackages
To get the list of commands: http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS

Installing E-accelerator on Debain Squeeze

Download it

cd /tmp
wget http://www.debiantutorials.com/static/eaccelerator-0.9.6.1.tar.bz2

Unpack it

tar -xvjf eaccelerator-0.9.6.1.tar.bz2

Prepare for compiling it

apt-get install php5-dev make
cd eaccelerator-0.9.6.1
phpize

Compile it

Note: The extra option –without-eaccelerator-use-inode is a fix for problems of basedir-open/basedir-restrictions found with this version of eAccelerator.
See http://tipstricks.itmatrix.eu/?p=1297 for more details.

./configure --enable-eaccelerator=shared --without-eaccelerator-use-inode
make
make install

Add a configuration to it

Add the following configuration in: /etc/php5/conf.d/eaccelerator.ini

extension="eaccelerator.so"
eaccelerator.shm_size="16"
eaccelerator.cache_dir="/var/cache/eaccelerator"
eaccelerator.enable="1"
eaccelerator.optimizer="1"
eaccelerator.check_mtime="1"
eaccelerator.debug="0"
eaccelerator.filter=""
eaccelerator.shm_max="0"
eaccelerator.shm_ttl="0"
eaccelerator.shm_prune_period="0"
eaccelerator.shm_only="0"
eaccelerator.compress="1"
eaccelerator.compress_level="9"

Create the cache directory for it

mkdir -p /var/cache/eaccelerator
chmod 0777 /var/cache/eaccelerator

Restart Apache2
/etc/init.d/apache2 restart

How to fix SSH timeouts

This is such a nuisance. I don't know of anything more annoying than to walk away to the bathroom, kitchen or whatever; and come back to find your terminal session disconnected.

This can also be helpful when running jobs that take a little longer, but I would really advise using SCREEN for that.

We will tackle this problem from both ends, the server and the client.

On the server

Edit /etc/ssh/sshd_config and add the line:


ClientAliveInterval 60

On the client

Edit /etc/ssh/ssh_config and add the line:

ServerAliveInterval 60

Installing Logwatch on Debian Squeeze

Logwatch is a good package that can send you emails every morning that you can peruse over your coffee. Is will run through your log file looking for anomalies.

It doesn't get much easier than this.

apt-get install logwatch

Config file here:

/usr/share/logwatch/default.conf/logwatch.conf

How to turn off IPv6 in Bind9 in Squeeze

In order to stop those nasty messages in your log files about not being able to find the route for IPV6 lookups, just do this:

nano /etc/default/bind9

Change OPTIONS to:

OPTIONS="-4 -u bind"

"-4" will tell bind to use ipv4 only.

How to turn off IPv6 in Debian Squeeze

Disabling IPv6 is not crucial however I like to keep my logs clean and to a minimum. 

Disable ipv6 in kernel :
echo net.ipv6.conf.all.disable_ipv6=1 > /etc/sysctl.d/disableipv6.conf will disable ipv6 at next reboot.

Installing Denyhosts on Debian Squeeze(6.0.7)

While trying to install Denyhosts on Debian(with SYNCING working), I ran around this guide: Preventing_ssh_dictionary_attacks_with_denyhosts on Debian on Howtoforge. It had be written in 2006. Time for an update.

The DenyHosts in the Debian Repos worked fine, if you are not interested in SYNCing. If your not using syncing in Denyhosts, you might as well use Fail2ban, as it covers more services that just SSH.

The sync feature really sets Denyhosts apart from the rest of the pack. Because as soon as an IP is detected and banned on a server half-way around the world, it will will shortly be pre-emptively banned on your server as well. So you are protected by, as well as helping to protect others.

Here is what worked for me:

1 Installation

DenyHosts is written in Python, therefore we must install Python and also the Python development files first:

apt-get install python2.6-dev python2.6

or you can probably get away with doing:

apt-get install python python-dev

I just happened to know that 2.6 was the latest in the repos anyways.

Then we download and install DenyHosts like this:
cd /tmp
wget http://sourceforge.net/projects/denyhosts/files/denyhosts/2.6/DenyHosts-2.6.tar.gz/download
tar xvfz DenyHosts-2.6.tar.gz 
cd DenyHosts-2.6
python setup.py install
This installs DenyHosts to /usr/share/denyhosts.

2 Configuration

Now we have to create the DenyHosts configuration file /usr/share/denyhosts/denyhosts.cfg. We can use the sample configuration file /usr/share/denyhosts/denyhosts.cfg-dist for this:

cd /usr/share/denyhosts
cp denyhosts.cfg-dist denyhosts.cfg

Make sure you set SECURE_LOG and LOCK_FILE to the correct values for your distribution! For Debian, these are:

SECURE_LOG = /var/log/auth.log
LOCK_FILE = /var/run/denyhosts.pid

As we want to run DenyHosts as a daemon, we need the daemon control script /usr/share/denyhosts/daemon-control. Again, we can use the sample script /usr/share/denyhosts/daemon-control-dist to create the needed file:

cp daemon-control-dist daemon-control
Edit /usr/share/denyhosts/daemon-control and make sure you set the correct values for DENYHOSTS_BIN, DENYHOSTS_LOCK, and DENYHOSTS_CFG. For Debian, these are:

DENYHOSTS_BIN   = "/usr/local/bin/denyhosts.py"
DENYHOSTS_LOCK  = "/var/run/denyhosts.pid"
DENYHOSTS_CFG   = "/usr/share/denyhosts/denyhosts.cfg"

Next we have to make that file executable:

chown root daemon-control
chmod 700 daemon-control
Afterwards, we create the system bootup links for DenyHosts do that it is started automatically when the system is booted:

cd /etc/init.d
ln -s /usr/share/denyhosts/daemon-control denyhosts
update-rc.d denyhosts defaults
Finally, we start DenyHosts:
/etc/init.d/denyhosts start
 
DenyHosts logs to /var/log/denyhosts, if you are interested in the logs. The SSH daemon logs to /var/log/auth.log on Debian. You can watch both logs and try to log in with an invalid user or with a valid user and incorrect password, etc. via SSH and see what happens. 

Credits to: Falko Timme <ft [at] falkotimme [dot] com> for writing the original guide I drew most of this text from.