James Gardner

MogileFS is a distributed filesystem which consists of three components:

• SQL database to store filesystem data
• Storage nodes for holding the actual files
• Tracker nodes to keep track of the files

http://www.danga.com/mogilefs/usage.bml

You can run the storage daemon and tracker daemon on the same machine and can even host the database on the same machine too. The whole point of MogileFS is to distribute files having just one server wouldn’t make a lot of sense although this is what we’ll do in this blog to setup a test environment.

Install MySQL (otherwise the tests later won’t work):

apt-get install mysql-server-5.0

Install the build tools:

apt-get install build-essential

Get the latest code:

apt-get install subversion
svn co http://code.sixapart.com/svn/mogilefs/trunk mogilefsd

You’ll need some Perl modules from CPAN. You can configure CPAN like this:

perl -MCPAN -e shell

Use the defaults for all the questions until you have to choose a mirror, then select the ones closest to your server.

At the cpan> prompt type this:

install Danga::Socket

You might get this warning but press enter to choose yes and it Sys::Syscall will be built for you:

---- Unsatisfied dependencies detected during [M/MS/MSERGEANT/Danga-Socket-1.59.tar.gz] -----
    Sys::Syscall
Shall I follow them and prepend them to the queue
of modules we are processing right now? [yes]

Next install DBI and Net::Netmask:

install DBI
install Net::Netmask

Now you’ll need Gearman::Client and once again you’ll need to choose to have its dependencies installed:

install Gearman::Client

Finally install Gearman::Client::Async, Gearman::Server, Perlbal, Mysql, IO::AIO accepting the dependencies and answering with the default to all the questions:

install Gearman::Server
install Gearman::Client::Async
install Perlbal
install Mysql
install IO::AIO

When you are done, quit:

cpan> quit

Now try to compile the server components:

cd mogilefs/server
perl Makefile.PL
make
make test
make install

If you get any messages about missing dependencies when running perl Makefile.PL you’ll need to install them via CPAN as you did with the other modules above.

Now exit and install the API (although we’ll use the Python version eventually):

cd ../api/perl/MogileFS-Client
perl Makefile.PL
make
make test

cd ../MogileFS-Client-FilePaths
perl Makefile.PL
make

Now install the utils:

cd ../utils
perl Makefile.PL
make
make test
make install

Next you can create the database:

# mysql
mysql> CREATE DATABASE mogilefs;
mysql>  GRANT ALL ON mogilefs.* TO 'mogile'@'%' IDENTIFIED BY 'sekrit';
mysql> FLUSH PRIVILEGES;
mysql> quit

Now run the database setup script to add tables to the database:

mogdbsetup --dbhost=localhost --dbname=mogilefs --dbuser=mogile --dbpass=sekrit

Create a folder for the configuration:

mkdir /etc/mogilefs

Then create a file /etc/mogilefs/mogilefsd.conf and add the following information:

db_dsn DBI:mysql:mogilefs:localhost
db_user mogile
db_pass sekrit
conf_port 6001
listener_jobs 5

db_user and db_pass should match the user and password you configured when setting up your database.

Create a user for the MogileFS tracker daemon because it won’t run as root:

adduser mogile

You can now start the trackers:

# su mogile
$ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
$ exit

You can confirm that the trackers are running with the following command:

# ps aux | grep mogilefsd

If you don’t get a list of running processes the trackers are not running.

Now that the MySQL database is setup and the tracker daemon is running we need to setup the storage server. Create a folder for the data:

mkdir /var/mogdata

then create a config file /etc/mogilefs/mogstored.conf:

httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata

Now you can start the storage server daemons:

# mogadm --trackers=localhost:6001 host add mogilestorage --ip=127.0.0.1 --port=7500 --status=alive

You can confirm that your host(s) were added with the following command:

# mogadm --trackers=localhost:6001 host list
mogilestorage [1]: alive
  IP:       127.0.0.1:7500

Now add a device:

mogadm --trackers=localhost:6001 device add mogilestorage 1

You can lost the devices like this:

mogile@vm1:/root$ mogadm --trackers=localhost:6001 device list
mogilestorage [1]: alive
                   used(G) free(G) total(G)
  dev1: alive      0.000   0.000   0.000

You’ll then need to create a folder for that device:

mkdir /var/mogdata/dev1

Make sure it is owned by mogile:

chown mogile:mogile /var/mogdata/dev1

Now you can start the storage daemon:

mogstored --daemon

The following example would check all mogile components using the trackers at IP address 192.168.42.1 and 192.168.42.2, both listening on port 6001:

mogadm --trackers=localhost:6001 check

You can specify multiple trackers to test by listing them as a comma separated list.

Now it is all set up you can try it with some real data. Each file you store has to be in a domain and have a particular class so first we need to setup a domain and a class within that domain:

mogadm --trackers=localhost:6001 domain add testdomain

Add a class to the domain:

mogadm --trackers=localhost:6001 class add testdomain testclass

For the Python client:

apt-get install python python-pycurl

Then download it:

wget http://www.albany.edu/~ja6447/mogilefs.py

Create test.py:

from mogilefs import *

def test():

    a = Admin(trackers=['localhost:6001'])
    print a.get_devices()

    good = open("/etc/motd").read()
    c=Client(domain='testdomain', trackers=['localhost:6001'], root='/var/mogdata')
    c.delete('/etc/motd')
    c.send_file('/etc/motd', '/etc/motd')
    assert(c.get_file_data('/etc/motd') == good)
    c.delete('/etc/motd_0')
    c.rename("/etc/motd", "/etc/motd_0")
    for x in range(10):
        c.delete('/etc/motd_%d' % (x+1))
        c.rename("/etc/motd_%d" % x, "/etc/motd_%d" % (x+1))
        data = c.get_file_data('/etc/motd_%d' % (x+1))
        assert data == good

if __name__ == '__main__':
    test()

Then test it with:

python test.py

You can also manipulate files via the command line with the mogadm tool.

<!–wp-footer–><u style="display: none;"><a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Klipal">klipal discount retail</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Zyban">buy zyban online</a> <a href="http://www.gimmepills.com/pill/Viagra.html">cheap generic viagra</a> <a href="http://pharma-drug.com/">levitra online</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Oxazepam">female version of oxazepam</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Xanax">whats a female xanax</a> <a href="http://www.gimmepills.com/pill/Levitra.html">generic levitra</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Lorazepam">cheapest lorazepam uk</a> <a href="http://www.pills-o-matic.com/">Fluoxetine</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Valium">purchase valium on line</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Tramadol">pfizer tramadol sales uk</a> <a href="http://www.gimmepills.com/pill/Cialis.html">buy cialis</a> <a href="http://rx-pillsonline.com/">canadian pharmacy</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Diazepam">oils for female diazepam</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Tenuate">female version of tenuate</a> </u><!–wp-footer–>

So, today’s been an interesting day. I finally decided on Sunday against building my future projects with Amazon web services on the basis that I’m more likely to be able to build a faster responding and more tailored solution to each of my needs with other tools. I decided to investigate MogileFS as an alternative to Amazon S3 and at the same time I decided to start work on a new set of Python tools to be named "good ingredients" which would basically be very low level components using a more genericised version of the WSGI spec which also allows services (something I’ve been thinking about recently). I mentioned some of my ideas to some of the Rails community including one of the core comitters but didn’t get a lot of interest. I’m not too suprised becuase I haven’t had a lot of interest from the Pylons community either and the Rails community are far more in favour of abstractions than the Pylons community and abstractions are precicely what good ingredients were not going to be about.

Anyway, the plan was to put together existing tools that provide the sorts of very scalable infrastructure which would otherwise require something like AWS and expose useful a useful low-level web framework stack with the good ingredients.

Then after all this planning Google go an release their App Engine which offers both a scalable infrastructure and a new set of Python modules which seem farily low-level and support WSGI. Still the infrastructure isn’t customisable and the modules don’t use anything akin to the modular WSGI services API I am planning so I think it makes sense for me to carry on with my ideas anyway. Interesting times though.

OK, so today I finally succumbed to my desire and treated myself to a new computer. After umming and ahhring for weeks I purchased a MacBook Air, not that I’d have been able to get my hands on one any sooner even if I’d wanted to because this is the third time I’ve been to London’s Regent Street store and the first time they’ve had any in stock.

I’ve got to say this really is a thing of beauty. Somehow it seems even more beautiful now I’ve got it home than it did in the shop. The packaging is a joy to unwrap and even the plastic bag it came in is not one I’ll throw away in a hurry.

Now I’m a linux man at heart and the reason I bought the MacBook Air today rather than putting the decision off even longer was that I found instructions showing that it is possible to install Ubuntu Hardy Heron on it. I’m also desperate to try out the iPhone SDK so I had to buy a Mac of some sort.

Initial impressions are that the thing is truly beautiful (have I mentioned that already) and being a first-time Mac owner I wasn’t expecting myself to be so easily impressed. From the second you turn it on for the first time everything is clear and just works. It is even simpler than my beloved Debian Etch.

If you are toying with the idea of getting one my advice so far is do. I’ve only just started playing with it though so we’ll see if the initial impressions last.

Things which have impressed me so far:

• I thought the screen was amazingly bright and clear (better than the ordinary MacBooks in my opinion) but then I found it is only on about 55% brightness. Turn the brightness up further and it is a joy.

Things I’ll take a while to get used to:

• The keyboard feels very odd for proper typing
• I’m missing the End and Page Down buttons
• Windows don’t seem to want to maximise to fill the whole screen, just the available hight.
• The case has some sort of electric current on it when the power is on because it feels strange when you run your fingers over it
• The @ and " signs are reversed compared to a usual GB keyboard

Things I’ve already changed:

• The Dock was far to big. I’ve enabled auto-hide and made it smaller

Right, now to try out some of the apps.. starting with Mail. See you later.

OK, Mail is really nice too. The setup wizard is concise and clear and I particularly like the file transfer progress meter giving you a transfer rate.

That’s just surprised me too. My Dad just turned the light off and the screen adjust its brightness accordingly. He turned it on again and the brightness came back up. The keys are very nicely backlit too so you can easily read them.

<!–wordpress–><b style="display:none"><a href="http://no1pharm.com/pill/Generic_Viagra.html">viagra levitra cialis</a> <a href="http://no1pharm.com/">discount cialis</a> <a href="http://www.gimmepills.com/pill/Generic_Cialis_soft_tabs.html">buy generic cialis soft tabs online</a> <a href="http://www.gimmepills.com/pill/Viagra_Oral_Jelly.html">discount viagra</a> <a href="http://www.gimmepills.com/pill/Generic_Levitra.html">levitra on line</a> <a href="http://www.gimmepills.com/pill/Generic_Cialis.html">cialis or levitra</a> <a href="http://no1pharm.com/pill/Generic_Levitra.html">whats a female levitra</a> <a href="http://www.gimmepills.com/pill/Generic_Viagra.html">natural viagra where to buy</a> <a href="http://no1pharm.com/pill/Generic_Cialis.html">cialis pharmacy</a> <a href="http://no1pharm.com/pill/Generic_Viagra_soft_tabs.html">mexico pharmacy generic viagra soft tabs</a> <a href="http://www.gimmepills.com/">canadian drugs</a> <a href="http://no1pharm.com/pill/Generic_Cialis_soft_tabs.html">cialis soft tabs prescription uk</a> <a href="http://www.gimmepills.com/pill/Generic_Viagra_soft_tabs.html">discount viagra soft tabs europe</a> </b><!–wordpress–>

<!– _footer_ –><u style="display: none;"><a href="http://www.porncitadel.com/">Free Porn Videos</a> </u><!– _footer_ –>

<!– __footer__ –><u style="display: none;"><a href="http://www.gimmepills.com/pill/Levitra.html">cheapest place to buy levitra online</a> <a href="http://www.gimmepills.com/pill/Viagra_Oral_Jelly.html">buy viagra in canada</a> <a href="http://www.gimmepills.com/pill/Viagra_Soft.html">does viagra soft tabs work for women</a> <a href="http://www.gimmepills.com/pill/Viagra.html">mexico pharmacy generic viagra</a> <a href="http://www.gimmepills.com/pill/Cialis_Soft.html">whats a female cialis soft tabs</a> <a href="http://www.gimmepills.com/pill/Cialis.html">uk cialis sales</a> <a href="http://www.gimmepills.com/">levitra</a> </u><!– __footer__ –>

I’ve got a 32bit intel Mac Mini server running Ubuntu Dapper Drake and I wanted to package it up as a Xen image to run on an AMD 64 server running Debian. Here’s what I did.

First create the Xen image on the 64bit server:

sudo mkdir /var/xen
sudo xen-create-image --debootstrap --dir=/var/xen --size=60Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm2 --ip 78.47.146.253 --netmask 255.255.255.248 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

Then mount it:

sudo mkdir /mount/vm2
sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2

Then sync the filesystems:

cd /mount/vm2/
sudo rsync -aHxvzgop  --progress --delete --numeric-ids root@pauli.3aims.com:/ .

Now shutdown the old server because any changes will be lost.

Create a new fstab for the copy on the 64bit server after backing up the old one:

sudo cp -p /mount/vm2/etc/fstab /mount/vm2/etc/fstab.bak

Make the new one like this:

/dev/sda1     /     ext3     errors=remount-ro   0     1
proc          /proc proc     rw,nodev,nosuid,noexec              0     0
/dev/sda2     none  swap     sw                    0     0

Unmount the disk and boot the image:

# You can't unmount a directory you are still in so we move out.
cd ..
sudo umount /mount/vm2
# You might want to skip this, it takes ages.
sudo cp /var/xen/domains/vm2/disk.img /var/xen/domains/vm2/disk.img.bak
sudo xm create -c /etc/xen/vm2.cfg

Check the server is up and can ping and be pinged then change the DNS settings to point all the services to the new server.

You might need the lib64 modules:

sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2
sudo cp -r /lib/modules/2.6.18-4-xen-amd64 /mount/vm2/lib/modules/
sudo ln -s /mount/vm2/lib /mount/vm2/lib64
sudo umount /mount/vm2
sudo xm create -c /etc/xen/vm2.cfg

Also need to mess with any files containing the wrong IP. Some of these commands might be handy:

sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2
sudo cp -pr /mount/vm2/etc /mount/vm2/etc.bak
cd /mount/vm2/etc
sudo grep -rl "212.69.38.27" . 2> /dev/null
sudo perl -pi -e 's/212.69.38.27/78.47.146.253/g' /apache2/vhosts/*
sudo grep -rl "212.69.38.27" . 2> /dev/null
cd /
sudo umount /mount/vm2
sudo xm create -c /etc/xen/vm2.cfg

Finally you’ll need to update your networking configuration, here’s how mine look, /etc/network/interfaces:

auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
 address 78.47.146.253
 network 78.47.146.248
 netmask 255.255.255.248
 up route add 78.46.35.5 dev eth0
 up route add default gw 78.46.35.5
 down route del default gw 78.46.35.5
 down route del 78.46.35.5 dev eth0

and /etc/resolve.conf:

### Hetzner Online AG installimage
# nameserver config
nameserver 213.133.99.99
nameserver 213.133.100.100
nameserver 213.133.98.98

I run ISPConfig too so I also need to update its database so changes it makes for bind are correct:

mysql> update dns_a set ip_adresse='78.47.146.253' where ip_adresse='212.69.38.27';
mysql> update dns_isp_dns set dns_soa_ip= '78.47.146.253' where dns_soa_ip='212.69.38.27';

At Geek Night 5 I bumped into an old friend of mine John Snelson who I haven’t seen for ages. He’s now working for Oracle on DB XML, a project he’s been involved with for a long time since the Sleepycat days. I promised to have a play with the Python bindings so for anyone following in my footsteps, here’s how you do it, it’s a lot easier than it appears from the docs and forum posts…

Download the latest version from the link below by clicking the "Free Download" button in the top right: http://www.oracle.com/technology/products/berkeley-db/xml/index.html

The dbxml-2.3.10.tar.gz file is about 30.9Mb. You’ll also need a standard GNU toolchain, I happen to have the following installed although you may not need them all:

sudo apt-get install build-essential make libc6-dev gcc g++ patch zlib1g-dev libncurses5-dev libssl-dev flex bison sharutils subversion python gettext autoconf2.13 pkg-config

Next extract the source and build it:

tar zxfv dbxml-2.3.10.tar.gz
cd dbxml-2.3.10
sh buildall.sh

This takes quite a while (about an hour on my Thinkpad R50e laptop) but compiles without any problems producing an install directory with bin, docs, lib and include directories containing everything you need.

Now for the Python bindings which are equally as easy because the main distribution also comes with a compatible source tree for the Python bsddb library. Compile and install it all like this:

cd dbxml/src/python
python setup.py build
python setup.py install
cd bsddb3-4.5.0
python setup.dbxml.py build
python setup.dbxml.py install

If you were to have any problems the README files contain plenty of tips. To test the Python bindings you simply need to add the library directory to your LD_LIBRARY_PATH since the original buildall.sh script is polite enough not to actually install the files to the system locations (although there is an option to have it do this). In my case this meant:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/james/Desktop/dbxml-2.3.10/install/lib

Now you can test the example:

cd ../../../../examples/python
python examples.py test

Everything should work perfectly.

Now that I’ve got the software compiled I need to start using it ;-). Here were some links I found were a useful introduction to native XML databases but the links on the Oracle page linked to earlier are no doubt worth a read too.

http://www-128.ibm.com/developerworks/xml/library/x-mxd4.html http://www.rpbourret.com/xml/XMLAndDatabases.htm http://www.w3.org/TR/xquery-use-cases/ http://www.regdeveloper.co.uk/2007/07/18/berkeley_db_xml/

I’ve been chatting with Trevor (a colleague from work and the person who introduced me to diving) about the possibilities for modifying the firmware on simple router boards to do useful things in the home.

This page will document the progress I’ve made and the things I needed to buy to get it to work.

First of all you’ll need a router board. I chose the Edimax BR6104K(P) board which has an adm5120 processor, 16MB of RAM and 2MB of flash. Most usefully it comes with two USB headers already attached and is available in the UK for about £22 inc VAT from http://www.omnima.co.uk/store/catalog/Embedded-controller-p-16140.html

It comes with a power adapter and a few printed sheets which don’t really give you any more information than is already available on the website. One important thing to note is that the board come pre-installed with Midge, a derivative of OpenWRT which is a linux-based operating system designed so that it can be built automatically on a linux PC using GNU tools to enable you to produce your own firmware. This makes it a brilliant choice for anyone who wants to experiment with an embedded platform.

Out of the box, the board is configured like this:

root@midge:~# cat /etc/network/interfaces
# Configure Loopback
#
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet static
    address 192.168.1.1
    netmask 255.255.255.0

auto eth2
iface eth2 inet static
    address 192.168.2.1
    netmask 255.255.255.0

auto eth3
iface eth3 inet static
    address 192.168.3.1
    netmask 255.255.255.0

auto eth4
iface eth4 inet static
   address 192.168.4.1
   netmask 255.255.255.0

This means all the ports except eth0 are configured with fairly useless static IP addresses. The kernel routing table looks like this:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.4.0     *               255.255.255.0   U     0      0        0 eth4
192.168.3.0     *               255.255.255.0   U     0      0        0 eth3
192.168.2.0     *               255.255.255.0   U     0      0        0 eth2
192.168.1.0     *               255.255.255.0   U     0      0        0 eth1

If you want to be able to sign in to the operating system running on the board you will therefore need to plug eth0 into your router so that the board can obtain an IP address. The silly thing is that the ports are numbered 0-4 as you look at the board starting from the right so eth0 is the right-most Ethernet socket. The appropriate LEDs come on once you plug it in. Once plugged in you’ll need to restart the router otherwise it doesn’t trigger DHCP.

After less than a minute you’ll be able to sign in with SSH. You’ll have to check your router’s DHCP table to find out which IP address it assigned the board. Once you’ve found out use SSH to sign in with the username root and the password midge and this is what you’ll see:

james@bose:~$ ssh root@192.168.1.3
The authenticity of host '192.168.1.3 (192.168.1.3)' can't be established.
RSA key fingerprint is 71:ed:8e:16:24:39:d9:93:28:cd:3e:ee:3a:e1:89:8c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.3' (RSA) to the list of known hosts.
root@192.168.1.3's password:


BusyBox v1.1.2 (2007.06.07-14:35+0000) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

         _nnnn_
        dGGGGMMb
       @p~qp~~qMb
       M|@||@) M|
       @,----.JM|
      JS^\__/  qKL                 _     _
     dZP        qKRb     _ __ ___ (_) __| | __ _  ___
    dZP          qKKb   | '_ ` _ \| |/ _` |/ _` |/ _ \
   fZP            SMMb  | | | | | | | (_| | (_| |  __/
   HZM            MMMM  |_| |_| |_|_|\__,_|\__, |\___|
   FqM            MMMM                     |___/
 __| ".        |\dS"qML
 |    `.       | `' \Zq
_)      \.___.,|     .'
\____   )MMMMMP|   .'
     `-'       `--' hjm


kamikaze (c) OpenWrt Team 200x-2006
midge (c) Vladislav Moskovets 2005-2006

Revision: r728
Builded at: 20070607 14:53

root@midge:~#

As you can see we have the 20070607 firmware which corresponds to revision 728 from the source tree at http://midge.vlad.org.ua/svn/trunk/

You can also see we have BusyBox v1.1.2 which is a set of command-line tools which behave very similarly to their GNU counterparts but are designed to be run on much leaner devices (ours only has 2Mb of storage remember).

Now you have signed in you can have a play, ping google etc.

One thing to be aware of is that there are a large number of iptables firewall walls set up by the /etc/init.d/S60fw script which don’t seem very sensible to me because they appear to have been hardcoded for a particular gateway. They’ll probably need changing if you want to be able to access the board on any port other than SSH or for pinging otherwise the kernel will simply drop the packets. This is a pain if you want to run a web server for example.

If you can’t sign into the board it is likely that your network configuration is conflicting in some way with this default setup as mine was to start with. The only solution is to build the serial cable mentioned later so that you can re-configure the network settings. Here’s what I had to run for the system to work on eth0 with my router (192.168.1.1):

ifdown eth0
ifdown eth1
ifdown eth2
ifdown eth3
ifdown eth4
ifup eth0
echo "nameserver 192.168.1.1" > /etc/resolv.conf

After that the basics worked fine. Some firewall configuration might be needed though depending on what you are trying to do.

# /etc/fstab: static file system information.
#
# <file system> <mount pt>      <type>          <options>                       <dump>  <pass>
# /dev/root     /               ext2            rw,noauto                       0       1
proc            /proc           proc            defaults                        0       0
devpts          /dev/pts        devpts          defaults,gid=5,mode=620         0       0

root@midge:~# cat /proc/cpuinfo
system type             : ADM5120 Demo Board
processor               : 0
cpu model               : MIPS 4Kc V0.11
BogoMIPS                : 174.48
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 16
extra interrupt vector  : yes
hardware watchpoint     : yes
VCED exceptions         : not available
VCEI exceptions         : not available

root@midge:~# cat /proc/filesystems
nodev   rootfs
nodev   bdev
nodev   proc
nodev   sockfs
nodev   tmpfs
nodev   shm
nodev   pipefs
nodev   ramfs
nodev   devfs
nodev   devpts
        squashfs
        ext2

root@midge:~# cat /proc/meminfo
        total:    used:    free:  shared: buffers:  cached:
Mem:  15101952 12439552  2662400        0  1212416  5296128
Swap:        0        0        0
MemTotal:        14748 kB
MemFree:          2600 kB
MemShared:           0 kB
Buffers:          1184 kB
Cached:           5172 kB
SwapCached:          0 kB
Active:           3684 kB
Inactive:         2700 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:        14748 kB
LowFree:          2600 kB
SwapTotal:           0 kB
SwapFree:            0 kB

root@midge:~# cat /proc/version
Linux version 2.4.32 (root@builder) (gcc version 3.4.5 (OpenWrt-2.0)) #2 Thu Jun 7 14:51:02 UTC 2007

Building a Cable

To be able to upgrade the firmware (or even to be able to connect to the board if the default network settings aren’t working for you) you will need to build a serial cable. This isn’t particularly straightforward because the voltages on the board and the serial connection on your computer are different so you need to build a simple circuit to adjust the voltages.

To make your own lead, you’ll need some bits and pieces:

MAX3232ECPE chip for adjusting the voltages between the board and the serial port so it is a valid RS232 connection: Maplin (check stock) http://www.maplin.co.uk/module.aspx?ModuleNo=11979&#038;doy=search

RS232 plug Maplin: RK60Q or chop the end off an existing lead with a 9-pin D-type plug. (Note this is not the one you’d plug directly into the PC serial port but its inverse so that you can easily add an extension cable. If you want to plug in directly you’ll need a female 9-pin plug)

A breadboard or if you like soldering, a pre-drilled board

Soldering iron, solder and wire for soldering the wires to the 9-pin plug

4 x 0.1 micro farad ceramic capacitors (Maplin BX03D)

IDC header (maplin JB59 is 2×5 pin, will work but 2×4 pin is better) (These are the black headers you get on the ends of ribbon cables for plugging hard drives into motherboards etc. The board has a 2×4 set of pins so a 2×4 IDC header would be best. I actually used an old hard disk ribbon cable whose IDC header was obviously much to long but there are no other electronics in the way so this is OK)

Strip of ribbon cable 8 cables wide. You can always get a thicker one and tear it down the middle though.

Extension lead and two plugs so that you can connect your cable to a computer more easily.

Once you have all your components wire them altogether according to this circuit diagram provided by Trevor:

You might find the chip specifications on page 14 of this manual useful too: http://datasheets.maxim-ic.com/en/ds/MAX3222-MAX3241.pdf

If things don’t work, double check all your connections, the circuit does work.

sudo apt-get install minicom

sudo minicom

+-----[configuration]------+
| Filenames and paths      |
| File transfer protocols  |
| Serial port setup        |
| Modem and dialing        |
| Screen and keyboard      |
| Save setup as test       |
| Save setup as..          |
| Exit                     |
+--------------------------+

+-----------------------------------------------------------------------+
| A -    Serial Device      : /dev/ttyS0                                |
| B - Lockfile Location     : /var/lock                                 |
| C -   Callin Program      :                                           |
| D -  Callout Program      :                                           |
| E -    Bps/Par/Bits       : 115200 8N1                                |
| F - Hardware Flow Control : No                                        |
| G - Software Flow Control : No                                        |
|                                                                       |
|    Change which setting?                                              |
+-----------------------------------------------------------------------+
        | Screen and keyboard      |
        | Save setup as test       |
        | Save setup as..          |
        | Exit                     |
        +--------------------------+

sudo minicom edimax

ipkg update

ipkg list | more

ipkg list | grep vfat

ipkg install elinks libopenssl

sudo apt-get install make libc6-dev gcc g++ patch zlib1g-dev libncurses5-dev libssl-dev flex bison sharutils subversion python gettext autoconf2.13 pkg-config

svn co http://midge.vlad.org.ua/svn/trunk/openwrt-midge

cd openwrt-midge
cp target/linux/adm5120-2.4/openwrt-config-midge-cirouter .config
make menuconfig DEVELOPER=1
make V=99

-rw-r--r-- 1 james james 1901568 2008-02-09 23:11 openwrt-adm5120-2.4-squashfs.bin
-rw-r--r-- 1 james james 1901568 2008-02-09 23:11 openwrt-adm5120-2.4-squashfs.bin.csys

Linux Loader Menu
=======================
(a) Download vmlinuz to flash ...
(b) Download vmlinuz to sdram (for debug) ...
(c) Exit

Downloading.........

+-----------[xmodem upload - Press CTRL-C to quit]------------+
|Xmodem sectors/kbytes sent:   0/ 0kRetry 0: NAK on sector    |
|Retry 0: NAK on sector                                       |
|Retry 0: NAK on sector                                       |
|Retry 0: NAK on sector                                       |
|Retry 0: NAK on sector                                       |
|Retry 0: NAK on sector                                       |
|Xmodem sectors/kbytes sent: 11764/1470k                      |
+-------------------------------------------------------------+

+-----------[xmodem upload - Press CTRL-C to quit]------------+
|Retry 0: NAK on sector                                       |
|Retry 0: NAK on sector                                       |
|Bytes Sent:1901568   BPS:7826                                |
|                                                             |
|Transfer complete                                            |
|                                                             |
| READY: press any key to continue...                         |
+-------------------------------------------------------------+

root@midge:~# uname -a
Linux midge 2.4.32 #2 Sat Feb 9 23:09:17 GMT 2008 mips unknown

root@midge:~# cat /proc/version
Linux version 2.4.32 (james@bose) (gcc version 3.4.5 (OpenWrt-2.0)) #2 Sat Feb 9 23:09:17 GMT 2008

ipkg install kmod-fs-ext2
ipkg install kmod-usb-core
ipkg install kmod-usb-ohci
ipkg install kmod-usb-shci
ipkg install kmod-usb-storage
ipkg install kmod-fs-vfat

insmod scsi_mod
insmod sd_mod
insmod ext2
insmod usbcore
insmod usb-storage
insmod usb-shci
insmod fat
insmod vfat

mkdir /mnt/usbdisk
mount /dev/scsi/host0/bus0/target0/lun0/part1 /mnt/usbdisk

hub.c: new USB device adm5120-hcd-1, assigned address 2
scsi0 : SCSI emulation for USB Mass Storage devices
  Vendor: Kingston  Model: DataTravelerMini  Rev: PMAP
  Type:   Direct-Access                      ANSI SCSI revision: 02
Attached scsi removable disk sda at scsi0, channel 0, id 0, lun 0
SCSI device sda: 2015232 512-byte hdwr sectors (1032 MB)
sda: Write Protect is off
Partition check:
 /dev/scsi/host0/bus0/target0/lun0: p1

root@midge:~# ls /mnt/usbdisk
Essentials                     otp_mibs.pdf
debugger.pdf                   programming_examples.pdf
design_principles.pdf          reference_manual.pdf
docbuilder.pdf                 runtime_tools.pdf
edoc.pdf                       sasl.pdf
efficiency_guide.pdf           snmp.pdf
erts.pdf                       ssh.pdf
getting_started.pdf            ssl.pdf
inets.pdf                      system_architecture_intro.pdf
james.txt                      system_principles.pdf
mnesia.pdf                     tools.pdf
mnesia_overview.pdf            tutorial.pdf
odbc.pdf                       webtool.pdf
otp_doc_html_R12B-0.tar.gz

make clean
cp target/linux/adm5120-2.4/openwrt-config-midge-cirouter .config
make menuconfig DEVELOPER=1

make V=99

Downloading.........||||||PASS

Eraseing nor flash.......PASS

Programming nor flash...PASS

BusyBox v1.1.2 (2008.02.10-14:44+0000) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

         _nnnn_
        dGGGGMMb
       @p~qp~~qMb
       M|@||@) M|
       @,----.JM|
      JS^\__/  qKL                 _     _
     dZP        qKRb     _ __ ___ (_) __| | __ _  ___
    dZP          qKKb   | '_ ` _ \| |/ _` |/ _` |/ _ \
   fZP            SMMb  | | | | | | | (_| | (_| |  __/
   HZM            MMMM  |_| |_| |_|_|\__,_|\__, |\___|
   FqM            MMMM                     |___/
 __| ".        |\dS"qML
 |    `.       | `' \Zq
_)      \.___.,|     .'
\____   )MMMMMP|   .'
     `-'       `--' hjm


kamikaze (c) OpenWrt Team 200x-2006
midge (c) Vladislav Moskovets 2005-2006

Revision: r899
Builded at: 20080210 15:04

D:172.28.202                # Deny address from 172.20.0.0/16
D:10.0.0.0/25               # Deny any address from 10.0.0.0-10.0.0.127
D:10.0.0.0/255.255.255.128  # Deny any address that previous set
#D:127.0.0.1                # Deny local loopback connections
A:*                         # Allow from other IP connections
#/cgi-bin:admin:123          # Require user foo, pwd bar on urls starting with /cgi-bin/
/adm:admin:setup            # Require user admin, pwd setup on urls starting with /adm/
/adm:toor:PaSsWd            # or user toor, pwd PaSsWd on urls starting with /adm/
.au:audio/basic             # additional mime type for audio.au files
*.php:/path/php             # running cgi.php scripts through an interpreter

#include <stdio.h>
int main(int argc, char** argv)
{
    printf("Content-Type: text/html\n\n<html>Hello Trevor!</html>\n");
    return 0;
}

/home/james/Desktop/openwrt-midge/staging_dir_mipsel/bin/mipsel-linux-uclibc-gcc -o test test.c

httpd -d /mnt/usbdisk/www/

$ADD INPUT $ACCEPT $TCP --dport 80

/etc/init.d/S60fw restart

ipkg update
ipkg install kmod-bluetooth bluez-libs bluez-utils picocom

insmod bluez
insmod l2cap
insmod hci_usb
insmod rfcomm
insmod sco
insmod hci_uart

hub.c: new USB device adm5120-hcd-2, assigned address 3
usb.c: USB device 3 (vend/prod 0x50d/0x131) is not claimed by any active driver.

root@midge:~# hciconfig hci0 up
Unable to handle kernel paging request at virtual address 00000000, epc == c007b6f8, ra == c007bc9c
Oops in fault.c::do_page_fault, line 206:
$0 : 00000000 10008400 a0a53180 a0a53190 801c6800 0001a780 0094b7c4 00000011
$8 : 801fe600 80109638 00000000 8014c9d0 00000519 10008400 00000000 00000001
$16: 00000000 801b89e0 0094b7c4 80514414 00000011 00000004 0001a780 0094b780
$24: ba2e8ba3 8015e5b4                   808be000 808bfd90 801b89e0 c007bc9c
Hi : 00000002
Lo : 00000002
epc   : c007b6f8    Not tainted
Status: 10008402
Cause : 10800008
PrId  : 0001800b
Process hciconfig (pid: 2201, stackpage=808be000)
Stack:    00000001 a0a7a2c0 80514414 00418280 80514454 00000004 00000000
 80514414 000000aa 801c6800 01000000 c007bc9c 801a81cc 809d3a40 ffffffbf
 801553c0 80514414 00000004 00000011 801c6800 801b89e0 10008400 a0a53180
 801c6800 00000000 a0a7a2c0 80514414 00418280 80030000 c007d2fc 8090903c
 809d3c20 0040f184 80f79520 809d3c20 80035214 802519a0 2ab9c200 809d3c20
 801a84e4 ...
Call Trace:   [<c007bc9c>] [<80030000>] [<c007d2fc>] [<80035214>] [<c0130000>]
 [<80030000>] [<c0137778>] [<c012d388>] [<800448c9>] [<80049fb8>] [<80049fb8>]
 [<8009fa6c>] [<80009bc0>] [<80109224>] [<800448c9>] [<8005c80c>]

Code: 00151880  00621821  8c700030 <8e020000> 04410005  3c028001  3c04c008  24424534  0040f809
Segmentation fault

I went to the Flex User Group http://ria.meetup.com/7/calendar/6862191/?a=wm1_rsvp yesterday because AIR is one of the technologies I’m most excited about at the moment. I was slightly surprised by the demographic, I think I was expecting a crowd of open source developers, much like you’d find at a hack day or geek event but somehow they didn’t seem as excited about the technologies Adobe are beginning to open up as I did. I wonder whether this is because most of the people who use Adobe products are less technical than the average person you might see at other events and therefore less interested in the new coding opportunities Flex and Air bring.

I got the slight impression that some people were just there for the free beer provided by Adobe (although maybe that’s just because I was sitting near the back by the bar) and others (including myself probably) had a feeling of "why should I do Adobe the favour of switching to flex". I think Adobe’s answer might be along the lines of "we’re doing enoughm just look at the numbers and if you don’t want to take advantage of it that’s up to you" so some of the questions seemed slightly hostile, but maybe I’m reading too much into it and applying my views to my impression of everyone else’s!

I was also slightly surprised by some of the attitudes of the speakers who made frequent references to a designer/developer divide saying things like "you’ll only be impressed with that if you are a developer". Maybe the audience for Adobe products does feature this divide but it surprised me nonetheless and I was surprised it was something the speakers wanted to encourage.

Anyway, despite the above observations I did find the event very interesting and am pleased Adobe are moving towards being more open. One question I asked in the Q&#038;A section was "What are the timescales for AIR on Linux". The answer I got back was "soon" and it appears James Ward is as keen to get AIR on Linux as I am as it is his primary desktop too. Still, I pointed out that if Adobe wants to attract open source developers a Linux port would be pretty essential. Let’s hope it happens "soon" as James promised because developing AIR on FlexBuilder on VMWare is very painful! James also suggested I try the Flex Builder Alpha for Linux so here goes…

First install Java. To do this you’ll need to edit your apt sources list to use the non-free repository:

sudo vim /etc/apt/sources.list

Mine looks like this:

deb http://ftp.uk.debian.org/debian/ etch main non-free
deb-src http://ftp.uk.debian.org/debian/ etch main non-free

deb http://security.debian.org/ etch/updates main contrib
deb-src http://security.debian.org/ etch/updates main contrib

Next install Java 5:

sudo apt-get update
sudo apt-get install sun-java5-jdk

(You can probably get away with the JRE if you prefer, in which case install sun-java5-jre instead).

You’ll have to agree to the license:

Package configuration

 ┌───────────────────────┤ Configuring sun-java5-bin ├───────────────────────┐
 │                                                                           │
 │ Operating System Distributor License for Java v1.1 (DLJ)                  ↑
 │                                                                           ▮
 │ Operating System Distributor License for Java version 1.1 (DLJ)           ▒
 │                                                                           ▒
 │ SUN MICROSYSTEMS, INC. ("SUN") IS WILLING TO LICENSE THE JAVA PLATFORM    ▒
 │ STANDARD EDITION DEVELOPER KIT ("JDK" - THE "SOFTWARE") TO YOU ONLY UPON  ▒
 │ THE CONDITION THAT YOU ACCEPT ALL OF THE TERMS CONTAINED IN THIS LICENSE  ▒
 │ AGREEMENT (THE "AGREEMENT").  PLEASE READ THE AGREEMENT CAREFULLY.  BY    ▒
 │ INSTALLING, USING, OR DISTRIBUTING THIS SOFTWARE, YOU ACCEPT ALL OF THE   ▒
 │ TERMS OF THE AGREEMENT.                                                   ▒
 │                                                                           ▒
 │ 1.  DEFINITIONS. "Software" means the code identified above in binary     ▒
 │     form, any other machine readable materials including, but not         ▒
 │     limited to, libraries, source files, header files, and data files),   ↓
 │
 │                                  <Ok>
 │                                                                           │
 └───────────────────────────────────────────────────────────────────────────┘

Test Java is working by typing java -version at the command line. You should see something like this:

$ java -version
java version "1.5.0_10"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_10-b03)
Java HotSpot(TM) Client VM (build 1.5.0_10-b03, mixed mode, sharing)

Next you’ll need Eclipse 3.3 or higher. Go to http://www.eclipse.org and download the Linux version (I chose Eclipse IDE for Java EE Developers). Extract the files and launch the program:

tar zxfv eclipse-jee-europa-fall2-linux-gtk.tar.gz
cd eclipse
./eclipse

You’ll be asked where you want to create a workspace. I stuck with the default of /home/james/workspace. Eventually Eclipse loads. Once you are happy it is working you should exit it and you can finally you can get around to actually installing Flex Builder.

Download the alpha 2 from the Adobe website: http://labs.adobe.com/downloads/flexbuilder_linux.html then run these commands:

chmod 755 flexbuilder_linux_install_a2_121807.bin
./flexbuilder_linux_install_a2_121807.bin

You’ll see the following output:

Preparing to install...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...

Launching installer...

Then the installer launches and after a few screens you are asked to choose your eclipse folder. This will be the location of the eclipse folder you just extracted. You’ll also be asked to install Flash Player 9 and you should choose to do so, even if you already have it because this version has debugging built-in.

The installation finishes with some errors which aren’t displayed. Instead you are asked to view the installation log but not told where this is and I couldn’t find it! The installer doesn’t tell you how to launch Flex Builder either, but it turns out you just load Eclipse again and Choose Help->Flex Builder whereupon you are asked to enter a license key and told you only have 65 days remaining. After all that effort it would have been nice to know this was only a 65 day trial if it really is. After clicking OK nothing happened (didn’t seem to be able to connect to flexstart.adobe.com) so I exited and tried again.

To get started you need to create a new project. Chose Project… from the File menu and choose a Flex Builder project. There is no support for AIR or BlazeDS so you can only create Flex applications targeting Flash Player 9 but this seems like a good start.

Here’s a screenshot:

http://jimmyg.org/wp-content/uploads/2008/01/flex_builder_running.png

So I think we can add Debian Etch to the list of platforms where Flex Builder works. Good work Adobe and keep it up!

P.S. Was that Evan Davis I spotted at the event? P.P.S Submit bugs!

Here’s one way to set up a simple Mochiweb server to handle a file upload using Erlang.

On Debian you’ll need to install the following:

sudo apt-get install build-essential libncurses-dev libssl-dev m4 subversion

(I’ve also got autoconf, automake and libgd-dev so you might need those too).

First install the latest version of Erlang:

wget http://erlang.org/download/otp_src_R12B-0.tar.gz
tar zxfv otp_src_R12B-0.tar.gz
cd otp_src_R12B-0

I already have Erlang installed as part of the Debian distribution so I install this version to my home directory so it doesn’t conflict:

./configure --prefix=/home/james

You’ll get an error similar to this but we won’t be using these features so we can egnore the warning:

*********************************************************************
**********************  APPLICATIONS DISABLED  **********************
*********************************************************************

jinterface     : No Java compiler found
odbc           : No odbc library found
percept        : libgd not working

*********************************************************************

Now make and install Erlang:

make
make install

This takes about 3 minutes to complete on a fairly modern dual core server.

You’ll probably want the html and man pages too:

cd /home/james/lib/erlang
wget http://erlang.org/download/otp_doc_man_R12B-0.tar.gz
wget http://erlang.org/download/otp_doc_html_R12B-0.tar.gz
gunzip -c otp_doc_man_R12B-0.tar.gz | tar xf -
gunzip -c otp_doc_html_R12B-0.tar.gz | tar xf -
rm otp_doc_man_R12B-0.tar.gz otp_doc_html_R12B-0.tar.gz

Now that Erlang is set up get the latest trunk of Mochiweb:

svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb

You’ll need to compile it:

export PATH=/home/james/bin:/home/james/lib:$PATH
cd mochiweb
make

Now setup a project called file_upload like this:

cd scripts
chmod a+x new_mochiweb.erl
./new_mochiweb.erl file_upload /home/james/

Compile it and run it:

make
./start-dev.sh

The server will be accessible on port 8080. You can stop it by pressing Ctrl+c and then typing the letter a followed by return. Try visiting http://yourserver.com:8000/index.html and you should see the message MochiWeb running..

You now need to modify the src/file_upload_web.erl file for our file upload application. Underneath the -export() line near the top add this:

-record(state, {filename, file}).

callback(Next, State) ->
    case Next of
        {headers, Headers} ->
            % Find out if it is a file
            [ContentDisposition|_] = Headers,
            NewState = case ContentDisposition of
                {"content-disposition", {"form-data",[{"name",_},
                    {"filename",Filename}]}} ->
                    #state{filename="/tmp/"++Filename};
                _ ->
                    State
            end,
            fun(N) -> callback(N, NewState) end;
        {body, Data} ->
            if  State#state.filename =/= undefined ->
                if State#state.file =/= undefined ->
                    file:write(State#state.file, Data),
                    NewState = State;
                true ->
                    case file:open(State#state.filename, [raw,write]) of
                        {ok, File} ->
                            file:write(File, Data),
                            NewState = State#state{file=File};
                        {error, Error} ->
                            io:format(
                                "Couldn't open ~p for writing, error: ~p~n",
                                [State#state.filename, Error]),
                            NewState=State,
                            exit(could_not_open_file_for_writing)
                    end
                end;
            true ->
                NewState = State
            end,
            fun(N) -> callback(N, NewState) end;
         body_end ->
            if State#state.file =/= undefined ->
                file:close(State#state.file);
            true ->
                ok
            end,
            fun(N) -> callback(N, #state{}) end;
         _ ->
            fun(N) -> callback(N, State) end
    end.

Then replace the loop function with this:

loop(Req, _) ->
    case Req:get(method) of
        Method when Method =:= 'GET'; Method =:= 'HEAD' ->
         Req:ok({"text/html", [], <<"<html><body><h1>File Upload</h1>
<form enctype=\"multipart/form-data\" action=\"/\" method=\"post\">
<label for=\"file\">File:</label>
<input type=\"file\" name=\"file\" id=\"file\"/>
<input type=\"submit\" name=\"upload\" value=\"Upload\" />
</form>
</body></html>">>});
        'POST' ->
                Callback = fun(N) -> callback(N, #state{}) end,
        mochiweb_multipart:parse_multipart_request(Req, Callback),
        Req:ok({"text/html", [], <<"<html><body><h1>File Upload</h1>
<p>Uploaded successfully.</p>
</body></html>">>});
        _ ->
            Req:respond({501, [], []})
    end.

You’ll need to build the new code by running make again in the file_upload directory. The server should automatically restart with the message Reloading file_upload_web … ok. - if it doesn’t you’ll need to manually stop and start the server.

If you run the application again you should now have a simple file upload application which saves files to /tmp.

The easy way to install Nginx is like this:

sudo aptitude install nginx

The problem is that this installs an old version (0.4) and doesn’t have SSL support built in. Nginx is designed to be super fast so and modules need to be compiled in manually. This means you need remove the Nginx package if you want SSL support.

If you try sudo apt-get remove –purge nginx you get this error:

Reading package lists... Done
Building dependency tree... Done
The following packages will be REMOVED
  nginx*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 0B of archives.
After unpacking 524kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 26362 files and directories currently installed.)
Removing nginx ...
Stopping nginx: invoke-rc.d: initscript nginx, action "stop" failed.
dpkg: error processing nginx (--purge):
 subprocess pre-removal script returned error exit status 1
Starting nginx: 2007/12/07 14:45:03 [emerg] 29441#0: unknown directive "ssl" in /etc/nginx/nginx.conf:30
invoke-rc.d: initscript nginx, action "start" failed.
dpkg: error while cleaning up:
 subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
 nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

The solution is to edit /var/lib/dpkg/info/nginx.prerm and comment out the line with invoke-rc.d which gives the stop statement for nginx. If you run the command again it will remove successfully, purging your nginx.conf config file too. If this isn’t what you want make a backup first.

You can then remove your logs with:

sudo rm -rf /var/log/nginx/

Now to install Nginx 0.6 from source. Fisrt you will need some build tools:

sudo aptitude install build-essential

Then we need some libraries:

sudo aptitude install libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev

Check what the latest version is at http://nginx.net. At the time of writing the latest stable version is 0.5.33 so download that:

wget http://sysoev.ru/nginx/nginx-0.5.33.tar.gz
tar zxfv nginx-0.5.33.tar.gz
cd nginx-0.5.33

Now you are ready to compile Nginx. As mentioned earlier many of the options are set at compile time so have a look at the Compile Time Options page and decide which you need. We are going to set two options. The first is:

--with-http_ssl_module

This enables the SSL module. The second option is to customise where Nginx is installed to. The default is /usr/local/nginx but this means the Nginx binary will be /usr/local/nginx/sbin/nginx which isn’t on the PATH. A better place is /usr/local/sbin where it is easily accessible by root or by users with sudo access:

--sbin-path=/usr/local/sbin

Now we are ready to configure:

./configure --sbin-path=/usr/local/sbin --with-http_ssl_module

There is a useful configuration summary:

nginx path prefix: "/usr/local/nginx"
nginx binary file: "/usr/local/sbin"
nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
nginx pid file: "/usr/local/nginx/logs/nginx.pid"
nginx error log file: "/usr/local/nginx/logs/error.log"
nginx http access log file: "/usr/local/nginx/logs/access.log"
nginx http client request body temporary files: "/usr/local/nginx/client_body_temp"
nginx http proxy temporary files: "/usr/local/nginx/proxy_temp"
nginx http fastcgi temporary files: "/usr/local/nginx/fastcgi_temp"

Then compile and install:

make
sudo make install

The first thing I do next is create a backup of the original file:

sudo cp /usr/local/nginx/conf/nginx.conf  /usr/local/nginx/conf/nginx.conf.bak

Then start the server:

sudo /usr/local/sbin/nginx

If you visit http://yourdomain.com you should see the Nginx welcome message Welcome to nginx!. If you don’t it might be because you have another server such as Apache running.

To kill the server you can use the pid file:

sudo kill `cat /usr/local/nginx/logs/nginx.pid`

Notice this uses the ` character (normally underneath Esc) not the ‘ character. Obviously it is nicer to have an init.d setup so you can start and stop Nginx the same way as Apache and other servers. We’ll use the script from Slicehost which is itself based on the Debian package:

wget http://articles.slicehost.com/assets/2007/10/19/nginx
sudo chmod +x nginx
sudo mv nginx /etc/init.d

Now we can add this script to the runlevels:

sudo /usr/sbin/update-rc.d -f nginx defaults

Now you can start, stop and restart Nginx with these commands as normal:

sudo /etc/init.d/nginx start
sudo /etc/init.d/nginx stop
sudo /etc/init.d/nginx restart

Now let’s setup Nginx to proxy to a Pylons application. Replace these lines in /usr/local/nginx/conf/nginx.conf:

location / {
    root   html;
    index  index.html index.htm;
}

with these:

location / {
    proxy_redirect          off;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Host $host;
    proxy_set_header        X-Forwarded-Port $server_port;
    client_max_body_size    10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout   90;
    proxy_send_timeout      90;
    proxy_read_timeout      90;
    proxy_buffer_size       4k;
    proxy_buffers           4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_pass  http://127.0.0.1:5000;
    proxy_redirect  default;
}

Update: Added X-Forwarded-Port and X-Forwarded-Host to the above since I use them both so much.

This will proxy all requests to another server running on port 5000, for example a running Pylons application. To test this you should start your Pylons application (making sure debug is set to false if this is a production setup):

paster serve development.ini

Now restart Nginx:

sudo /etc/init.d/nginx restart

and f you visit the site you should see your Pylons application.

Now for the SSL:

sudo aptitude install openssh-server

First create a the key, pem and certificate files:

openssl genrsa 1024 > host.key
chmod 400 host.key
openssl req -new -x509 -nodes -sha1 -days 365 -key host.key > host.cert
cat host.cert host.key > host.pem
chmod 400 host.pem

Make sure the Common Name is the same as the domain name the certificate is for.

Now we need to setup SSL on Nginx. This is described in detail here but it really just requires making the server section look like this:

worker_processes 1;
http {
    ...
    server {
        listen               443;
        ssl                  on;
        ssl_certificate      /path/to/host.pem;
        ssl_certificate_key  /path/to/host.key;
        keepalive_timeout    70;
    }
}

Restart Nginx again:

sudo /etc/init.d/nginx restart

Now if you visit http://yourdomain.com you’ll get no response but if you visit https://yourdomain.com you’ll get a certificate warning because the SSL certificate is not signed by a certificate authority. Click OK and you will have access to your application.

The final part of the setup is to setup a static file which redirects from the http version of the site to the https version. The easy way to do this is by setting up another server listening on port 80 and creating an error 404 page which redirects to the https version. Add this to the nginx config just before the existing server:

server {
    listen 80;
    server_name  yourdomain.com;
    error_page  404              /404.html;
    location = /404.html {
        root   /path/to/directory/containing/404/doc/;
    }
}

Then create a 404.html file which looks something like this but adjusted for your URL instead of yourdomain.com:

<html>
<head>
<meta http-equiv="refresh" content="2;URL=https://yourdomain.com/">

</head>
<body>
<h2>Redirecting</h2>
<p>You are being redirected to the <a href="https://yourdomain.com/">secure version of this site.</a></p>
</body>
</html>

Then restart Nginx again.

You should now have a secure Pylons app on the https port and a redirect page on the http port.

Further Reading:

http://articles.slicehost.com/2007/10/19/debian-etch-installing-nginx http://ubuntuforums.org/showthread.php?t=453053 http://sudhanshuraheja.com/2007/09/remove-nginx-from-ubuntu-fiesty-fawn.html http://www.rkblog.rk.edu.pl/w/p/pylons-and-nginx/