MogileFS on Debian Etch

Posted:	2008-04-10 00:40
Tags:	XML

Caution!

Although I've got MogileFS working there was a bit of trial and error involved. I believe what I've documented here is the path I have taken but I haven't double checked it.

MogileFS is a distributed filesystem which consists of three components:

SQL database to store filesystem data
Storage nodes for holding the actual files
Tracker nodes to keep track of the files

http://www.danga.com/mogilefs/usage.bml

You can run the storage daemon and tracker daemon on the same machine and can even host the database on the same machine too. The whole point of MogileFS is to distribute files having just one server wouldn't make a lot of sense although this is what we'll do in this blog to setup a test environment.

Install MySQL (otherwise the tests later won't work):

apt-get install mysql-server-5.0

Install the build tools:

apt-get install build-essential

Get the latest code:

apt-get install subversion
svn co http://code.sixapart.com/svn/mogilefs/trunk mogilefsd

You'll need some Perl modules from CPAN. You can configure CPAN like this:

perl -MCPAN -e shell

Use the defaults for all the questions until you have to choose a mirror, then select the ones closest to your server.

At the cpan> prompt type this:

install Danga::Socket

You might get this warning but press enter to choose yes and it Sys::Syscall will be built for you:

---- Unsatisfied dependencies detected during [M/MS/MSERGEANT/Danga-Socket-1.59.tar.gz] -----
    Sys::Syscall
Shall I follow them and prepend them to the queue
of modules we are processing right now? [yes]

Next install DBI and Net::Netmask:

install DBI
install Net::Netmask

Now you'll need Gearman::Client and once again you'll need to choose to have its dependencies installed:

install Gearman::Client

Finally install Gearman::Client::Async, Gearman::Server, Perlbal, Mysql, IO::AIO accepting the dependencies and answering with the default to all the questions:

install Gearman::Server
install Gearman::Client::Async
install Perlbal
install Mysql
install IO::AIO

When you are done, quit:

cpan> quit

Now try to compile the server components:

cd mogilefs/server
perl Makefile.PL
make
make test
make install

If you get any messages about missing dependencies when running perl Makefile.PL you'll need to install them via CPAN as you did with the other modules above.

Now exit and install the API (although we'll use the Python version eventually):

cd ../api/perl/MogileFS-Client
perl Makefile.PL
make
make test

cd ../MogileFS-Client-FilePaths
perl Makefile.PL
make

Now install the utils:

cd ../utils
perl Makefile.PL
make
make test
make install

Next you can create the database:

# mysql
mysql> CREATE DATABASE mogilefs;
mysql>  GRANT ALL ON mogilefs.* TO 'mogile'@'%' IDENTIFIED BY 'sekrit';
mysql> FLUSH PRIVILEGES;
mysql> quit

Now run the database setup script to add tables to the database:

mogdbsetup --dbhost=localhost --dbname=mogilefs --dbuser=mogile --dbpass=sekrit

Create a folder for the configuration:

mkdir /etc/mogilefs

Then create a file /etc/mogilefs/mogilefsd.conf and add the following information:

db_dsn DBI:mysql:mogilefs:localhost
db_user mogile
db_pass sekrit
conf_port 6001
listener_jobs 5

db_user and db_pass should match the user and password you configured when setting up your database.

Create a user for the MogileFS tracker daemon because it won't run as root:

adduser mogile

You can now start the trackers:

# su mogile
$ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
$ exit

You can confirm that the trackers are running with the following command:

# ps aux | grep mogilefsd

If you don't get a list of running processes the trackers are not running.

Now that the MySQL database is setup and the tracker daemon is running we need to setup the storage server. Create a folder for the data:

mkdir /var/mogdata

then create a config file /etc/mogilefs/mogstored.conf:

httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata

Now you can start the storage server daemons:

# mogadm --trackers=localhost:6001 host add mogilestorage --ip=127.0.0.1 --port=7500 --status=alive

You can confirm that your host(s) were added with the following command:

# mogadm --trackers=localhost:6001 host list
mogilestorage [1]: alive
  IP:       127.0.0.1:7500

Now add a device:

mogadm --trackers=localhost:6001 device add mogilestorage 1

You can lost the devices like this:

mogile@vm1:/root$ mogadm --trackers=localhost:6001 device list
mogilestorage [1]: alive
                   used(G) free(G) total(G)
  dev1: alive      0.000   0.000   0.000

You'll then need to create a folder for that device:

mkdir /var/mogdata/dev1

Make sure it is owned by mogile:

chown mogile:mogile /var/mogdata/dev1

Now you can start the storage daemon:

mogstored --daemon

The following example would check all mogile components using the trackers at IP address 192.168.42.1 and 192.168.42.2, both listening on port 6001:

mogadm --trackers=localhost:6001 check

You can specify multiple trackers to test by listing them as a comma separated list.

Now it is all set up you can try it with some real data. Each file you store has to be in a domain and have a particular class so first we need to setup a domain and a class within that domain:

mogadm --trackers=localhost:6001 domain add testdomain

Add a class to the domain:

mogadm --trackers=localhost:6001 class add testdomain testclass

For the Python client:

apt-get install python python-pycurl

Then download it:

wget http://www.albany.edu/~ja6447/mogilefs.py

Create test.py:

from mogilefs import *

def test():

    a = Admin(trackers=['localhost:6001'])
    print a.get_devices()

    good = open("/etc/motd").read()
    c=Client(domain='testdomain', trackers=['localhost:6001'], root='/var/mogdata')
    c.delete('/etc/motd')
    c.send_file('/etc/motd', '/etc/motd')
    assert(c.get_file_data('/etc/motd') == good)
    c.delete('/etc/motd_0')
    c.rename("/etc/motd", "/etc/motd_0")
    for x in range(10):
        c.delete('/etc/motd_%d' % (x+1))
        c.rename("/etc/motd_%d" % x, "/etc/motd_%d" % (x+1))
        data = c.get_file_data('/etc/motd_%d' % (x+1))
        assert data == good

if __name__ == '__main__':
    test()

Then test it with:

python test.py

You can also manipulate files via the command line with the mogadm tool.

(view source)