I needed the ability to RSYNC data from the official Raspberry Pi servers on to mine and then in to S3 and for that I used s3fs and FUSE.

FUSE

You can actually do this successfully without requiring FUSE, just by installing the s3fs binary on to your system, but this only allows the user who mounted to access the mounted bucket and also is not possible via /etc/fstab.

FUSE allows you to implement a filesystem within a userspace program, thus allowing us to give other users access and auto-mount using /etc/fstab.

Installation

All commands prefixed with a # need to be run as root or a sudo

Fuse

Installing FUSE is simple

# apt-get install fuse-utils

s3fs

We'll need to get build-essential, pkg-config, libfuse-dev, libcurl4-openssl-dev and libxml2-dev to be able to compile s3fs

# apt-get install build-essential pkg-config libfuse-dev libcurl4-openssl-dev libxml2-dev

Debian 5 & Ubuntu 10.04

If installing either Debian 5 or Ubuntu 10.04, you'll need to install a newer version of fuse than is packaged, I found this info on the s3fs issue tracker here.

First we need to remove the install fuse-utils and libfuse-dev that we install above.

# apt-get purge fuse-utils libfuse-dev

You'll need to export a variable with your arch, i.e

export PLATFORM=amd64

wget http://ftp.us.debian.org/debian/pool/main/f/fuse/libfuse2_2.8.4-1.1_${PLATFORM}.deb
wget http://ftp.us.debian.org/debian/pool/main/f/fuse/libfuse-dev_2.8.4-1.1_${PLATFORM}.deb
wget http://ftp.us.debian.org/debian/pool/main/f/fuse/fuse-utils_2.8.4-1.1_${PLATFORM}.deb
# dpkg -i libfuse2_2.8.4-1.1_${PLATFORM}.deb libfuse-dev_2.8.4-1.1_${PLATFORM}.deb fuse-utils_2.8.4-1.1_${PLATFORM}.deb

Fix missing dependencies

# apt-get -f install

Now run the command below and confirm the output

pkg-config --modversion fuse
2.8.4

s3fs has to be done manually, first off go download the latest revision archive from here.

Once download, gunzip and untar it.

tar -xvzf s3fs-x.xx.tar.gz

Change directory in to your newly extracted archive, and configure.

./configure --exec-prefix=/usr/ --prefix=/ --includedir=/usr/include/ --mandir=/usr/share/man/

This configure command will install the s3fs binary in to /usr/bin and man pages in to /usr/share/man/ which is Debian and Ubuntu correct locations.

Then you'll need to compile and install.

make

# make install

You'll noticed I only run make install as sudo/root, because the other commands do not require it and you should never compile as root.

Configure s3fs

The only configuration you need to do for s3fs is store your S3 credential which you can get here.

Create a file called /etc/passwd-s3fs - MAKE SURE YOU DON'T BREAK /etc/passwd

In it you need to put your access key ID and secret access key, separated with a colon.

ACCESS_KEY_ID:SECRET_ACCESS_KEY

And for security reasons, change the file permissions

# chmod 0600 /etc/passwd-s3fs

Mounting

Manual

Once all the above is done you can mount a bucket using the s3fs binary, I'm going to mount directly to /mnt

# s3fs your-bucket-name /mnt

This will mount it and make it usable for your user.

fstab

Mounting via fstab requires the above FUSE step to be completed.

Your /etc/fstab entry should look like this

s3fs#your-bucket-name  /mnt  fuse allow_other,_netdev,nosuid,nodev,url=https://s3.amazonaws.com 0 0

A brief description of the mount arguments;

• allow_other - allow all users to access the mount point,
• _netdev - The filesystem resides on a device that requires network access,
• nosuid - Do not allow set-user-identifier or set-group-identifier bits to take effect,
• nodev - Do not interpret character or block special devices on the file system and
• url - Use HTTPS instead of HTTP when configure as above

When a country stays lit up for more than 1 tick of the clock in the left hand corner it means that multiple attacks are happening from different IP addresses. An attacker is banned after;

• 1 failed root login,
• 3 failed user logins (including invalid users) and
• 3 failed system logins.

Click here to view the embedded video.

Direct links

• Vimeo
• YouTube - older version of the video with yellow colouring instead of red.

For the app and database servers I do server-side backups storing each website and it's database in it's own folder within /backup in case I require a quick back-up to fix something, rather than the server has died.

This is all well and good but I like having an off-site backup too and for that I use S3...

S3

Amazon's S3 is pretty cheap and very easy to use. Because only data is going in you don't pay a transfer fee and the cost of storage is very affordable, you can see a pricing list here.

To do the backup I use a daily cron job which then uploads the data to S3 using s3cmd.

Installation

All commands prefixed with a # need to be run as root or a sudo

Download the S3 tools package list in to apt

# wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key | sudo apt-key add -
# wget http://s3tools.org/repo/deb-all/stable/s3tools.list -O /etc/apt/sources.list.d/s3tools.list

Update your package list and install s3cmd

# apt-get update && apt-get install s3cmd

Configuration

You'll need to configure the tool to work with your AWS account, so run

# s3cmd --configure

When prompted, fill in your access and secret key which you can find here.

When asked to provide an encryption password, I choose yes but you can say no.

When asking if you want to use HTTPS, I choose yes but again, you can say no, it really depends on how secure you want the data transfer.

I would suggest using an encryption password and enabling HTTPS.

Using s3cmd

Now that s3cmd is installed and configured you can use it.

You can create a bucket using the s3cmd command below, but as far as I know you can't select a location so I create my buckets manually here.

# s3cmd mb s3://your-bucket-name

Once done you can see a list of available buckets with

# s3cmd ls

As shown below

$ s3cmd ls

2012-02-29 20:28 s3://kura-linode-test

Now that this is done we can put some data in there, create a test file

echo "this is a test" > test.file

And put it in S3

# s3cmd put test.file s3://your-bucket-name/

You can see it using

# s3cmd ls s3://your-bucket-name

Download it with

# s3cmd get s3://your-bucket-name/test.file

And delete it with

# s3cmd del s3://your-bucket-name/test.file

Once satisfied with this you can create a shell script to automate some backups for you, I'll provide a simple one below that uploads my home directory

Example

#!/bin/sh

s3cmd sync --recursive --skip-existing /home/kura s3://kura-linode-test/

The unattended-upgrades package used on Debian is based on the one from Ubuntu. It is generally pretty safe in my opinion but I only ever enable it for security upgrades.

Installation

# apt-get install unattended-upgrades apticron

unattended-upgrades handles the actual updates, apticron is used for emailing you of available updates - it is not required but I like it.

Configuring unattended-upgrades

Open up /etc/apt/apt.conf.d/50unattended-upgrades and change it to the content below.

APT::Periodic::Enable "1";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
Unattended-Upgrade::Mail "YOUR_EMAIL_HERE";

// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {

"${distro_id} stable";
"${distro_id} ${distro_codename}-security";

};

// Automatically reboot *WITHOUT CONFIRMATION* if a
// the file /var/run/reboot-required is found after the upgrade
Unattended-Upgrade::Automatic-Reboot "false";

So lets explain the above. As you can see we enable periodic updates, enable update package lists (triggers an apt-get update), enable autoclean to clean out the local package repository every 7 days, enable the actual unattended update and finally you can set your email address so that you will get an email when an update has happened.
Next up we configure the origins to update from, as you can see we've only enabled security and as a very final step we make sure we've disabled automatic reboots - you probably don't want your server randomly rebooting to update the running kernel, this means you will have to reboot when convenient after a kernel update.

Your unattended update will happen every day, triggered by cron.daily. Next time your cron.daily has triggered, look inside /var/log/unattended-upgrades/unattended-upgrades.log, you should see something like this

2012-01-28 06:54:04,730 INFO Initial blacklisted packages:
2012-01-28 06:54:04,730 INFO Starting unattended upgrades script
2012-01-28 06:54:04,731 INFO Allowed origins are: ["('Debian', 'squeeze-security')"]
2012-01-28 06:54:05,952 INFO No packages found that can be upgraded unattended

If you installed apticron in the above step and want to configure it and use it then continue reading, if not then congratulations everything is done.

Configuring apticron

Open up /etc/apticron/apticron.conf, all you need to change is the EMAIL option.

EMAIL="YOUR_EMAIL_HERE"

Now each day you will receive an email when cron.daily runs with all available package updates.

I finally stopped being lazy and decided to create an HTTPS version of this blog. I know it's not at all required but I decided it could/would be a good thing to do anyway.

For now due to the blog software being crappy it's showing an invalid certificate due to including resources that are not secured - I'll work on fixing that.

It's IPv6 address is: 2a01:7e00::f03c:91ff:fe93:505a

Installation

To install we need to run the following command:

# apt-get install -y sks

Now we build the key database:

# sks build

And change the permissions for the sks user:

# chown -R debian-sks:debian-sks /var/lib/sks/DB

Next we need to make sks start from init, open up /etc/default/sks in your favourite editor and initstart to look like below:

initstart=yes

Now we can start the service with:

# /etc/init.d/sks start

Your keyserver will now be up and running on port 11371.

Web interface

We'll need to create a web folder within sks with the following command:

# mkdir -p /var/lib/sks/www/

Change it's permissions so the sks user can access it.

# chown -R debian-sks:debian-sks /var/lib/sks/www

And finally we need create a single HTML file for the interface, I have provided that too.

# wget http://syslog.tv/downloads/sks-index.html -O /var/lib/sks/www/index.html

Now your PGP server should be accessible from a web browser at http://YOUR_SERVER:11371/ and it should look like mine http://syslog.tv:11371/

First up we'll need to install git and some Python tools to get Gitosis installed.

Where # is used it means you need to either run the command as a superuser with sudo or as root.

# apt-get install -y git-core gitweb python-setuptools

Next we have to clone gitosis from it's git repository and install it.

cd /tmp

git clone git://eagain.net/gitosis.git

cd gitosis

# python setup.py install

Adding your git user

# adduser --system --shell /bin/sh --gecos 'git version control' --group --disabled-password --home /home/git git

The above command creates a new system user with /bin/sh as it's shell with no password and a homedir of /home/git/ and also creates a group with the same name.

Initialising gitosis

You'll need an SSH key for this, if you have one simply copy the contents of it to your new git server, if you do not have one then you can generate one on your machine using

ssh-keygen

And then copy the contents to your server.

My file was copied to /tmp/kura.pub so to initialise I used

sudo -H -u git gitosis-init < /tmp/kura.pub

This command MUST be run as sudo.

You need to do the same but replacing kura.pub with your own key, it has to end in .pub

A note on key format

One of my users (@gump) had an issue where Gitosis would complain about his username having invalid characters

gitosis.init.InsecureSSHKeyUsername: Username contains not allowed characters

This is because Gitosis expects your key to have a username and host at the end of the base64 string like below

ssh-rsa AAAAB3NzaC1yc2EA ... NOHgpPwEBzpnw== kura@odin

Configuring and controlling gitosis

Now that git and gitosis are working on your server, from your local machine you now need to clone your gitosis admin and do all your changes locally, pushing them back to the git server where gitosis will automatically pick them up.

So you need to run

git clone git@YOUR_SERVER:gitosis-admin.git

If everything worked correctly you should have a copy on your local machine now, if you run ls you'll see 1 file and a directory.

1. gitosis.conf
2. keydir

Unsurprisingly gitosis.conf is where gitosis is configured and keydir contains public keys for your users. Each user needs their own public key and it must end in .pub.

So open up gitosis.conf in your favourite editor and add the following:

[gitosis]
gitweb = yes

[group admins]
writable = gitosis-admin test1
members = kura

[repo gitosis-admin]
description = Gitosis admin repository
gitweb = yes

So lets separate that in to parts.

Part 1 - we simply tell gitosis to enable gitweb support.

Part 2 - we configure a group called admins, the admins group has write permissions to 2 repositories; gitosis-admin and test. The test repository will automatically become available once we push this configuration back to gitosis later. We also define a user called kura which you should replace with your own username, each user must have a public key in the keydir with the same name as the user with .pub suffix. E.g. the kura user has a key called kura.pub

Part 3 - We create a repository section which is only really used for gitweb to tell it to display that repository publicly via a browser.

If you do not want your repositories to be public then I advice you skip the parts with gitweb = yes above and also uninstall gitweb and skip the gitweb section below. Or you could lock your gitweb via HTAUTH.

Now the changes have been made you need to commit them to git.

git add *

git commit -m "Initial configuration"

And push them back to the server

git push origin master

Now that is done you can test your access to the test repository created earlier.

git clone git@YOUR_SERVER:test.git

cd test

echo "Hello world" > hello

git add hello

git commit -m "Test"

git push origin master

If the above works then congratulations, everything is good.

Adding users and repositories

Users

To add a user to gitosis you need to add them to a group and put a public key with username.pub as the naming format in to keydir.

Repositories

You simply need to name it in a writable section of a group and it'll instantly be accessible. If you want to make it public in gitweb then you'll need to a [repo] section as shown above.

Configure gitweb

Open up /etc/gitweb.conf in your favourite editor and change $projectroot to

$projectroot = "/home/git/repositories/"

You will also need to add the Apache user to the git group

usermod -G www-data,git www-data

By default Debian and Ubuntu will symlink in an Apache2 config to /etc/apache2/conf.d/gitweb which is accessible from a browser on http://YOUR_SERVER/gitweb

It is possible to configure your kernel to panic when OOM is spawned, which in itself is not useful but, coupled with a kernel option for auto-rebooting a system when the kernel panics it can be a very useful tool.

Think before implementing this and use at your own risk, I take zero responsibility for you using this.

sysctl vm.panic_on_oom=1

sysctl kernel.panic=X # X is the amount of seconds to wait before rebooting

DO NOT FORGET TO CHANGE X

This will inject the changes in to a system that is currently running but will be forgotten on reboot so use the lines below to save permanently.

echo "vm.panic_on_oom=1" >> /etc/sysctl.conf

echo "kernel.panic=X" >> /etc/sysctl.conf

X is the amount of seconds to wait before rebooting. DO NOT FORGET TO CHANGE X

Testing

You can test the changes with a simple C program. Please note if you run this you do so at your own risk.

#include
#include
#include

#define MB 10485760

int main(int argc, char *argv[]) {

void *b = NULL;
int c = 0;

while(1) {

b = (void *) malloc(MB);

if (!b) {

break;

}

memset(b, 10, MB);
printf("Allocating %d MB\n", (++c * 10));

}

exit(0);

}

Compilation

You can download the source from here.

To compile run the command below

gcc -O2 oom.c -o oom

Or download a pre-compiled version here.

Usage

And simply run it using

./oom

After a short period of time allocating and using 10MB chunks of memory your system should run out and restart.

First we need to make sure we have all the stuff we need to compile mk livestatus and run it

apt-get install make build-essential xinetd ucspi-unix

MK Livestatus

Grab the mk livestatus source from here, currently it's version 1.1.10p3 but update the commands below to match your version.

wget http://mathias-kettner.de/download/mk-livestatus-1.1.10p3.tar.gz

tar -xvzf mk-livestatus-1.1.10p3.tar.gz

cd mk-livestatus-1.1.10p3

./configure

make && make install

Xinetd

Now that it's compiled we need to write a xinetd config for it, create a new file called /etc/xinetd.d/livestatus and put the following in it

service livestatus
{

type = UNLISTED
port = 6557
socket_type = stream
protocol = tcp
wait = no
cps = 100 3
instances = 500
per_source = 250
flags = NODELAY
user = nagios
server = /usr/bin/unixcat
server_args = /var/lib/nagios3/rw/live
only_from = 127.0.0.1 # modify this to only allow specific hosts to connect, currenly localhost only
disable = no

}

Now we restart xinetd using

/etc/init.d/xinetd restart

Nagios3

Now we need to open up /etc/nagios3/nagios.cfg and add the following 2 lines

event_broker_options=-1
broker_module=/usr/local/lib/mk-livestatus/livestatus.o /var/lib/nagios3/rw/live

Now we need to restart Nagios

/etc/init.d/nagios3 restart

If you take a look in /var/log/nagios3/nagios.log

tail -n 100 /var/log/nagios3/nagios.log

you should see something like below

[1318547328] livestatus: Livestatus 1.1.10p3 by Mathias Kettner. Socket: '/var/lib/nagios3/rw/live'
[1318547328] livestatus: Please visit us at http://mathias-kettner.de/
[1318547328] livestatus: Hint: please try out OMD - the Open Monitoring Distribution
[1318547328] livestatus: Please visit OMD at http://omdistro.org
[1318547328] Event broker module '/usr/local/lib/mk-livestatus/livestatus.o' initialized successfully.

Also, we can ls the newly created socket

ls -lah /var/lib/nagios3/rw/live

srw-rw---- 1 nagios www-data 0 2011-10-14 00:08 /var/lib/nagios3/rw/live

We can test is by creating a test file called host_query with the following content

GET hosts

And run the following command

unixcat < host_query /var/lib/nagios3/rw/live

If all worked you should see output.