James Gardner: Home > Blog > 2008 > SMS Gateway with Orange SMS API,...

SMS Gateway with Orange SMS API, Python, Nginx, mod_wsgi and Postfix

Posted:2008-05-19 00:21
Tags:Python, Web, Hosting, SMS, Email

Orange are beginning to open up some of their APIs. One API I'm particularly interested in is the Orange SMS API because during the Alpha Orange are making both sending and receiving SMS messages free of charge. When you sign up at http://orangepartner.com you are given 1000 credits with each text sent costing 10 credits. Each SMS received gains you 2 credits. You can request more messages for testing.

Update: You can only send/receive 10 SMS messages per day though. This is completely impractical for any sort of serious debugging so I'm mentioned this to Orange.

Here's how the SMS API works. When you send an SMS fro your phone to 967482 Orange looks at the first word to find out which partner account the SMS is for. I've chosen the word FOR so SMS messages starting FOR are sent to me. In the control panel I can choose to have such messages sent on to an email address, passed as query parameters to a URL or both. These can then be processed.

In this article we are going to do three things:

Register with Orange

First register as an Orange partner and choose a keyword in the Administrator Web Interface. I've chosen FOR. You'll be given an access key which you'll need to make a note of and keep secret. Choose for received messages to be sent to a URL. I'm going to choose http://sms.3aims.com/receive

Setting up The Server

I created a new Xen virtual machine and pointed sms.3aims.com to that machine. Next I create a new user called sms and install some software on the virtual machine:

adduser sms
apt-get install python2.5 python2.5-dev build-essential mysql-server-5.0 screen mercurial subversion python2.4 python2.4-dev libpcre3-dev libssl-dev

Then as sms let's set up a virtual Python install:

cd /home/sms
mkdir download
mkdir lib
cd download
wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.0.tar.gz
tar zxfv virtualenv-1.0.tar.gz
cp virtualenv-1.0/virtualenv.py ./
python2.5 virtualenv.py --no-site-packages ~/env

We are going to serve the project using Nginx and the mod_wsgi plugin for no other reason than I haven't tried it before:

cd ~/download
wget http://sysoev.ru/nginx/nginx-0.5.34.tar.gz
cd ../lib
tar zxfv ../download/nginx-0.5.34.tar.gz
hg clone http://hg.mperillo.ath.cx/nginx/mod_wsgi

You'll need to patch mod_wsgi/config to set PYTHON to python2.5.

Now compile and install nginx:

cd nginx-0.5.34
./configure --add-module=/home/sms/lib/mod_wsgi --with-debug --with-http_ssl_module --sbin-path=/usr/local/sbin
make

We’ll use the script from Slicehost which is itself based on the Debian package:

cd ~/download
wget http://articles.slicehost.com/assets/2007/10/19/nginx
chmod +x nginx

then as root:

cd /home/sms/lib/nginx-0.5.34
make install
cd /home/sms/download
mv nginx /etc/init.d
/usr/sbin/update-rc.d -f nginx defaults
ln -s /usr/local/nginx/conf/nginx.conf /etc/nginx.conf

Edit /etc/nginx.conf to replace the http section with this:

worker_processes 5;
http {
    include       conf/mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  60;
    wsgi_enable_subinterpreters on;

    env PYTHON_EGG_CACHE=/home/sms/cache;

    server {
        listen       80;
        server_name  sms.3aims.com;
        location / {
            wsgi_pass /home/sms/sms/handle.py;
            wsgi_var PATH_INFO $fastcgi_script_name;
            wsgi_var REQUEST_METHOD $request_method;
            wsgi_var QUERY_STRING $query_string;
            wsgi_var SERVER_NAME $server_name;
            wsgi_var SERVER_PORT $server_port;
            wsgi_var SERVER_PROTOCOL $server_protocol;
            wsgi_var CONTENT_TYPE $content_type;
            wsgi_var CONTENT_LENGTH $content_length;

            wsgi_pass_authorization off;
            wsgi_script_reloading on;
            wsgi_use_main_interpreter on;
            # If you were using Pylons you'd add this, but we don't need to
            # wsgi_python_optimize 0; # pylons uses doc strings...
        }
    }
}

Nginx has an asynchronous architecture and by default only has one worker process. This means that if your Python WSGI application served from Nginx blocks for any reason (perhaps database access for example) then the whole server will block. Two solutions are:

We've already increased the number of worker threads to 5 but in order to program asynchronously we have to use a slightly different architecture to something like Pylons.

By the way, notice the line to set the value of PYTHON_EGG_CACHE. This is essential so that setuptools doesn't complain. We'll create this directory and set the permissions in a bit.

Test Application

Now that the server is set up, let's write a test application. We want this application to be able to use the virtual environment we set up for Python so there are a few lines at the front to set up the correct paths:

import site
site.addsitedir('/home/sms/env/lib/python2.5')
site.addsitedir('/home/sms/env/lib/python2.5/site-packages')

def application(environ, start_response):
    start_response('200 OK', [('Content-type','text/html')])
    return ['Hello World!']

Save this as /home/sms/sms/handle.py then restart the Nginx server. You'll also need to create /home/sms/cache then change its permissions with chmod 777 /home/sms/cache. Visit http://sms.3aims.com and you'll see the Hello World! message.

Setting up Email on the Server

We're going to use Postfix 2.5 (just because I like the milter support of Postfix 2.4 which Postfix 2.3 which comes with etch doesn't support) but otherwise the installation will be compatible with the excellent tutorial by Chris Haas at http://workaround.org/articles/ispmail-etch/

Setting up the hostname

Become root on your server and make sure that /etc/hostname contains the host name without the domain part. The file /etc/mailname should contain the fully-qualified host name with the domain part.

You'll probably need to fix /etc/hosts too. Run hostname --fqdn and see if you get the fully-qualified hostname. If you just get the hostname without the domain please check that your /etc/hosts file has the fully-qualified hostname first in the list.

Wrong:

78.47.146.250   sms sms.3aims.com

Right:

78.47.146.250   sms.3aims.com sms

Setting the Locale

As root:

apt-get install locales

Then if needed:

apt-get install debconf
dpkg-reconfigure locales

Here's what you get:

Package configuration



 ┌──────────────────────────┤ Configuring locales ├──────────────────────────┐
 │                                                                           │
 │ Locale is a framework to switch between multiple languages for users who  │
 │ can select to use their language, country, characters, collation order,   │
 │ etc.                                                                      │
 │                                                                           │
 │ Choose which locales to generate.  The selection will be saved to         │
 │ `/etc/locale.gen', which you can also edit manually (you need to run      │
 │ `locale-gen' afterwards).                                                 │
 │                                                                           │
 │ When `All locales' is selected, /etc/locale.gen will be set as a symlink  │
 │ to /usr/share/i18n/SUPPORTED.                                             │
 │                                                                           │
 │                                  <Ok>                                     │
 │                                                                           │
 └───────────────────────────────────────────────────────────────────────────┘

Select the locale(s) you want:

Package configuration

                     ┌──────┤ Configuring locales ├───────┐
                     │ Locales to be generated:
                     │
                     │    [ ] en_BW ISO-8859-1
                     │    [ ] en_BW.UTF-8 UTF-8
                     │    [ ] en_CA ISO-8859-1
                     │    [ ] en_CA.UTF-8 UTF-8
                     │    [ ] en_DK ISO-8859-1
                     │    [ ] en_DK.ISO-8859-15 ISO-8859-15
                     │    [ ] en_DK.UTF-8 UTF-8
                     │    [ ] en_GB ISO-8859-1
                     │    [ ] en_GB.ISO-8859-15 ISO-8859-15
                     │    [*] en_GB.UTF-8 UTF-8
                     │    [ ] en_HK ISO-8859-1
                     │    [ ] en_HK.UTF-8 UTF-8
                     │
                     │
                     │       <Ok>           <Cancel>
                     │                                    │
                     └────────────────────────────────────┘

I choose to make the new locale the default:

Package configuration

  ┌─────────────────────────┤ Configuring locales ├─────────────────────────┐
  │ Many packages in Debian use locales to display text in the correct      │
  │ language for users. You can change the default locale if you're not a   │
  │ native English speaker. These choices are based on which locales you    │
  │ have chosen to generate.                                                │
  │                                                                         │
  │ Note: This will select the language for your whole system. If you're    │
  │ running a multi-user system where not all of your users speak the       │
  │ language of your choice, then they will run into difficulties and you   │
  │ might want not to set a default locale.                                 │
  │                                                                         │
  │ Default locale for the system environment:                              │
  │                                                                         │
  │                               None                                      │
  │                               en_GB.UTF-8                               │
  │                                                                         │
  │                                                                         │
  │                   <Ok>                       <Cancel>                   │
  │                                                                         │
  └─────────────────────────────────────────────────────────────────────────┘

Setting Up Backports

Read this for background information: http://www.backports.org/dokuwiki/doku.php?id=instructions

The only change is we add the etch backports to the /etc/apt/sources.list:

deb http://www.backports.org/debian etch-backports main

Then run:

wget -O - http://backports.org/debian/archive.key | apt-key add -
apt-get update

Installing Postfix

Install it with this command to explicitly use the backports version:

apt-get -t etch-backports install postfix-mysql

Now you will see a lot of screens:

Package configuration

  ┌────────────────────────┤ Postfix Configuration ├────────────────────────┐
  │                                                                         │
  │ Please select the mail server configuration type that best meets your
  │ needs.
  │                                                                         ▒
  │  No configuration:                                                      ▒
  │   Should be chosen to leave the current configuration unchanged.        ▒
  │  Internet site:                                                         ▒
  │   Mail is sent and received directly using SMTP.                        ▒
  │  Internet with smarthost:                                               ▒
  │   Mail is received directly using SMTP or by running a utility such     ▒
  │   as fetchmail. Outgoing mail is sent using a smarthost.                ▒
  │  Satellite system:                                                      ▒
  │   All mail is sent to another machine, called a 'smarthost', for        ▒
  │ delivery.                                                               ▒
  │  Local only:
  │
  │                                 <Ok>
  │                                                                         │
  └─────────────────────────────────────────────────────────────────────────┘

Click OK then choose Internet Site. You'll see this, enter the domain which will appear after the @ in email addresses:

Package configuration


 ┌─────────────────────────┤ Postfix Configuration ├─────────────────────────┐
 │ The "mail name" is the domain name used to "qualify" mail addresses       │
 │ without a domain name.                                                    │
 │                                                                           │
 │ This name will also be used by other programs. It should be the single,   │
 │ fully qualified domain name (FQDN).                                       │
 │                                                                           │
 │ Thus, if a mail address on the local host is foo@example.org, the         │
 │ correct value for this option would be example.org.                       │
 │                                                                           │
 │ System mail name:                                                         │
 │                                                                           │
 │ sms.3aims.com____________________________________________________________ │
 │                                                                           │
 │                    <Ok>                        <Cancel>                   │
 │                                                                           │
 └───────────────────────────────────────────────────────────────────────────┘

Implementing the Code

The code is going to be implemented in three parts:

Setting up Logging

Since the scripts we are going to use are executed by either postfix or as a result of an HTTP request from Orange, we won't have any particularly easy way to see what's going on so we'll set up a logging configuration to log data to /var/log/syslog. To do this create a logging configuration which looks like this in /home/sms/sms/logging_basic.ini:

[formatters]
keys: detailed

[handlers]
keys: syslog

[loggers]
keys: root,sms

[logger_root]
level: INFO
handlers: syslog

[logger_sms]
level: DEBUG
handlers: syslog
qualname: sms

[handler_syslog]
class: handlers.SysLogHandler
args: ['/dev/log']
propagate: 0
formatter: detailed

[formatter_detailed]
%(name)s:%(levelname)s %(message)s

This will log all child loggers of sms which have levels of DEBUG or above to the syslog.

To use it in a Python application use this code:

import logging
import logging.config
logging.config.fileConfig('/home/sms/sms/logging_basic.ini')
log = logging.getLogger('sms.some_child')

You can then log results like this:

log.debug(result)

Implementing the Send SMS API

Let's start with the send SMS library. Create an sms.py file within the /home/sms/sms directory:

"""\
Simple module for sending SMS messages
"""
import logging
import logging.config
logging.config.fileConfig('/home/sms/sms/logging_basic.ini')
log = logging.getLogger('sms.sms')

import urllib

API_KEY = 'XXXXXXXXXXX'

def send_sms(to, message):
    log.debug("Sending SMS to %r with message %r", to, message)
    url = 'http://sms.alpha.orange-api.net/sms/sendSMS.xml'
    data = {'id': API_KEY, 'to':to, 'content':message}
    fp = urllib.urlopen(url, urllib.urlencode(data))
    result = fp.read()
    fp.close()
    log.debug("Sending SMS result was %r", result)
    return result

You'll need to replace all the XXX characters with the correct API key you get from the Orange control panel.

Implement the Web Server Part

Now we are ready to implement the webserver part. This is going to When a txt is sent to 967482 and starts with the keyword you've specified, Orange will call the URL you entered. For example. This text:

For 447980233595 Test

would result in this URL being called:

http://sms.3aims.com/retrieve?api=receivesms&from=447980233595&dateCreated=2008-05-18+22%3A20%3A10.863&content=For+07964763169+Test

Our code needs to handle this and then make a call to the API to send the txt on:

#! -*- coding: utf-8 -*-

import site
site.addsitedir('/home/sms/env/lib/python2.5')
site.addsitedir('/home/sms/sms')
site.addsitedir('/home/sms/env/lib/python2.5/site-packages')
from webob import Request, Response

# Set up logging
import logging
import logging.config
logging.config.fileConfig('/home/sms/sms/logging_basic.ini')
log = logging.getLogger('sms.handle')

from sms import send_sms
import webhelpers.mail as mail

def application(environ, start_response):
    log.debug('Message receive begun')
    request = Request(environ)
    response = Response()
    response.content_type = 'text/html'
    if request.path_info == '/retrieve':
        if request.params.get('api') != 'receivesms':
            log.error('Invalid API')
            response.body = 'Invalid API'
        elif not request.params.get('content'):
            log.error('No content')
            response.body = 'No content'
        elif not request.params.get('from'):
            log.error('No from')
            response.body = 'No from'
        else:
            content = request.params['content'].strip(' ')
            parts = content.split(' ')
            if len(parts) < 3:
                log.error('Unknown SMS structure')
                response.body = 'Unknown SMS structure'
            else:
                code = parts[0].lower()
                to = parts[1].lower()
                content = ' '.join(parts[2:])
                log.debug('Code: %r, To: %r, Content: %r', code, to, content)
                if code not in ['for']:
                    log.error('Received SMS with an unknown keyword %r', code)
                    response.body = 'Unknown SMS keyword'
                else:
                    if '@' in to:
                        # Treat as an email address
                        log.debug('Sending email to %r', to)
                        mail.send(
                            mail._prepare(
                                mail.plain(content),
                                from_name = '3aims SMS Gateway',
                                from_email = 'noreply@sms.3aims.com',
                                to=[to],
                                subject = 'SMS from %s'%request.params['from']
                            ),
                            sendmail = '/usr/sbin/sendmail'
                        )
                        log.debug('Sent email')
                        response.body = 'Sent email'
                    else:
                        if to.startswith('0'):
                            # Assume a UK number
                            to = '44'+to[1:]
                            log.debug('No country code, using %r', to)
                        from_ = 'From '+request.params['from']+'. '+content
                        log.debug('Sending SMS to %r from %r', to, from_)
                        response.body = send_sms(to, from_)
                        log.debug('Result: %r', response.body)
    else:
        response.status = '200 Not Found'
        response.body = 'Not Found'
    return response(environ, start_response)

Set up the Forwarding

When messages are sent to sms att sms.3aims.com we want them to piped to a program which sends them as an SMS message. This requires us to create a .forward file for the sms user to forward the email to the program.

As the sms user:

touch ~/.forward
chmod 0640 ~/.forward

Add the following content to the .forward file:

"|/home/sms/sms/receive.py"

We'll need the latest WebHelpers package:

cd ~/lib
hg clone http://pylonshq.com/hg/webhelpers
cd webhelpers
~/env/bin/python2.5 setup.py develop

Then create /home/sms/sms/receive.py with this content:

#!/home/sms/env/bin/python2.5

import logging
import logging.config
import sys
import webhelpers.mail as mail
from email.feedparser import FeedParser
from sms import send_sms

# Set up logging
logging.config.fileConfig('/home/sms/sms/logging_basic.ini')
log = logging.getLogger('sms.receive')

# Parse the message
f = FeedParser()
f.feed(sys.stdin.read())
message = f.close()

# Process the message
to = message['subject']
if not message.is_multipart():
    msg = message.get_payload()
else:
    raise Exception('Multipart messages not supported')

result = send_sms(to, msg)
log.debug(result)
log.debug(to)
log.debug(msg)

That's it. You can now send SMS's to email addresses, have emails forwarded to SMS and also have SMS's send to other mobile numbers all via your server.

With these basics in place the possibilities are fairly limitless.

(view source)

James Gardner: Home > Blog > 2008 > SMS Gateway with Orange SMS API,...