For whatever reason, Amazon’s S3 has captured my attention since the day it launched.

I want to use S3 for an online backup of my most important files. (I already do nightly local backups of everything, but that’s not enough. I’m paranoid.)

I’m finally close to reaching this goal, but it’s not been terribly easy.

Here’s a brief recount of my trials with S3 so far:

1. s3DAV. It didn’t work so well on OS X. The developer was really helpful, so we traded a few emails while he attempted to get it working on OS X from across the Atlantic. No luck.

2. JungleDisk (via Steven Frank). Much, much better. Of course I jumped in head-first and ran my backup script immediately, which attempted to copy 10GB of data to my S3 account. That took all night and then some; I finally had to cancel the operation. But somehow I managed to bork (technically speaking, of course) my S3 bucket so JungleDisk beach-balled every time I tried to connect.

3. S3 Browser. I found this while looking for a way to get into my account and delete the screwed-up bucket. Too bad there was no way to delete more than one file at a time — I would’ve had to press “Delete” 15,000 times. Not fun.

4. jSh3ll. Like most things of this nature, I had to turn to the command-line to get anything done. In a rare moment of competency, I downloaded and built jSh3ll (using Ant, no less) on my Power Mac and connected to my S3 account. I deleted the troublesome bucket from the command-line and now I’m back to square one (mostly).

I’m running a much smaller backup to S3 right now. Wish me luck.

(Oh, and have a good weekend! Michelle and I are headed to Green Lake for the holiday. How about you?)

1. FOAF to hCard via SPARQL "So I used the scutterplan from the FOAFBulletinBoard, going 2 steps, producing 17382 statements, containing I think it was 2035 foaf:Person nodes...To the results I applied sparql2hcard.xsl using xsltproc, script was sparql-hcard.sh (I do hope this was the latest SPARQL XML results format...), producing these hCards." XLST, querying - sounds like descriptors.


2. Timeline "...a DHTML-based AJAXy widget for visualizing time-based events. It is like Google Maps for time-based information."


3. iris: open your eyes to the future of the desktop "Iris is a java open source source integrated desktop environment including email, calendar, file browser, web browser, chat, and a data mining toolkit (clustering, indexing, and lots more)." Uses Jena.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) unlimited
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

core file size        (blocks, -c) 0
data seg size         (kbytes, -d) 6144
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 256
pipe size          (512 bytes, -p) 1
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 100
virtual memory        (kbytes, -v) unlimited

Neil lived in a purple palace with a purple castle to sleep in. He raced around his cage at top speed, sending bits of litter out the corners as he peeled out. Neil loved cantaloupe, blueberries, cheerios, broccoli, and peppers. When he ate watermelon the juice ran down the corners of his mouth and stained his white fur red and we called him a vampire piggie. When Neil was a baby he climbed his castle and out of his aquarium and scaled a book case (a height of a foot above the aquarium) we don’t know how he did it. Neil and I used to play a game called “reverse lion tamer” where I would open my mouth and he would insert his entire head. He was not scared of the cat and would blithely ignore the paw fishing around for him until we screamed at Riley. When Neil was excited he would bounce around on your chest like a piece of popcorn popping while chittering. When he was scared he would whistle softly. When I held him in my arms with his head on my chest he would butt his head up under my chin.

When Neil’s eye was bad they said he might not even live and he got his sight back.

When he broke his teeth off he grew them back.

It turns out that Neil probably had a very inner ear infection that gave no symptoms. This ear infection went on so long it seeped into his bones, his nervous system, and his brain. Neil could not fight hard enough to win this battle.

And neither could we.

Neil died today, at 6:30 am at the specialist vet in Houston we took him to. We were supposed to plan a course of action today for the newly developed nervous system problems but it was too late to plan. We had promised him that if he lived until Monday the special vet would make it all better. He lived until Monday but she didn’t make it better.

Neil was supposed to live at least five more years.

I loved him very very much and I miss him very very much.

As part of the same preparation I really enjoyed to program a classic:

Which are the 92 solutions to the 8 queen problem?

First, the usage intentions and conditions of satisfaction:

    [TestMethod]    public void validSolutions()    {      List<ChessBoard> boards = ChessBoard.AllQueens();      foreach (ChessBoard b in boards)      {        IList all=b.Pieces;        Assert.IsTrue(all.Count == 8);        for (int k = 0; k < 8; k++)        {          IList l = b.Rank[k];          Assert.IsTrue(l.Count == 1);        }        foreach (char c in 'abcdefgh')        {          IList l = b.File[c];          Assert.IsTrue(l.Count == 1);        }      }    }    [TestMethod]    public void allSolutions()    {      List<ChessBoard> boards = ChessBoard.AllQueens();      Assert.AreEqual(92, boards.Count);    }

    public static List<ChessBoard> AllQueens()    {      ResultKeeper keeper = new ResultKeeper();      ChessBoard board = new ChessBoard();      board.OnNewSolution += new ChessBoard.OnNewSolutionHandler(keeper.OnNewSolution);      board.GetQueens();      return keeper.Results;    }

  class ResultKeeper  {    List<ChessBoard> results;    public ResultKeeper()    {      results = new List<ChessBoard>();    }    public void OnNewSolution(int count, ChessBoard board)    {      results.Add(board);    }    public List<ChessBoard> Results    {      get      {        return results;      }    }  }

  public class ChessBoard  {    private char[,] board;    private int hit_count;    public ChessBoard()    {      board  = new char[8, 8];      reset();    }    private ChessBoard(char[,] b)    {      board = b;    }    public delegate void OnNewSolutionHandler(int count,ChessBoard board);    public event OnNewSolutionHandler OnNewSolution;    public IList Pieces    {      get       {        ArrayList result = new ArrayList();        for (int k = 0; k < 8; ++k)          for(int j=0;j<8;++j)            if (board[k,j] == 'X')              result.Add(board[k,j]);        return result;      }    }    public FileLine File { get { return new FileLine(board); } }    public RankLine Rank { get { return new RankLine(board); } }    public char[,] InternalState    {      get { return board; }    }    public static List<ChessBoard> AllQueens()    {      ResultKeeper keeper = new ResultKeeper();      ChessBoard b = new ChessBoard();      b.OnNewSolution += new ChessBoard.OnNewSolutionHandler(keeper.OnNewSolution);      b.GetQueens();      return keeper.Results;    }    public void GetQueens()    {      find_safe_positions();    }    void reset()    {      for (int k = 0; k < 8; ++k)        for (int j = 0; j < 8; ++j)          board[k, j] = '-';    }    private void find_safe_positions()    {      int rank = 0;      find_safe_position(rank);    }    private bool find_safe_position(int rank)    {      for (int file = 0; file < 8; ++file)      {        if (is_safe(rank, file))        {          board[rank, file] = 'X';          if (rank == 7)          {            NewSolutionFound();            board[rank, file] = '-';            continue;          }          if (find_safe_position(rank + 1))            return true;          else            board[rank, file] = '-';        }      }      return false;    }    bool is_safe(int rank, int file)    {      bool safe = empty_rank(rank) && empty_file(file) && empty_cross(rank, file);      return safe;    }    bool empty_rank(int rank)    {      for (int k = 0; k < 8; ++k)        if (board[rank, k] == 'X')          return false;      return true;    }    bool empty_file(int file)    {      for (int k = 0; k < 8; ++k)        if (board[k, file] == 'X')          return false;      return true;    }    bool empty_cross(int rank, int file)    {      for (int k = rank, j = file; k >= 0 && j >= 0; --k, --j)        if (board[k, j] == 'X')          return false;      for (int k = rank, j = file; k < 8 && j < 8; ++k, ++j)        if (board[k, j] == 'X')          return false;      for (int k = rank, j = file; k < 8 && j >= 0; ++k, --j)        if (board[k, j] == 'X')          return false;      for (int k = rank, j = file; k >= 0 && j < 8; --k, ++j)        if (board[k, j] == 'X')          return false;      return true;    }    void NewSolutionFound()    {      ChessBoard b = new ChessBoard(board.Clone() as char[,]);      ++hit_count;      if (OnNewSolution != null)        OnNewSolution(hit_count, b);    }    public class FileLine    {      private char[,] board;      internal FileLine(char[,] b) { board = b; }      public IList this[char c]      {        get        {          ArrayList result = new ArrayList();          int j = (int)c - 97;          for (int k = 0; k < 8; ++k)            if (board[k, j] == 'X')              result.Add(board[k, j]);          return result;        }      }    }    public class RankLine    {      private char[,] board;      internal RankLine(char[,] b) { board = b; }      public IList this[int n]      {        get        {          ArrayList result = new ArrayList();          for (int k = 0; k < 8; ++k)            if (board[n, k] == 'X')              result.Add(board[n, k]);          return result;        }      }    }  }

In preparation to write some papers about software design techniques, I am planning to illustrate those techniques with a variety of examples. As part of the effort I will document my thoughts while doing reflective practicum, identifying and documenting patterns of thought while programming solutions to several kinds of problems; including implementation of simple algorithms like insertion sort, selection sort, bubble sort and more complex ones. By now, let?s see the code so far.

Consider the following main function:

#include <iostream>#include <vector>#include <iterator>using namespace std;void main(){  sort( sort_by_selection<vector<int> >() );  sort( sort_by_insertion<vector<int> >() );  sort( sort_by_bubble<vector<int> >()    );}

template<typename T>void sort(T sort_algorithm){  int v[]={100,80,70,70,60,50,40,40,20,10,-1,-2};  vector<int> V(v,v+sizeof(v)/sizeof(int));  ostream_iterator<int> out(cout,' ');  copy(V.begin(),V.end(),out);  sort_algorithm(V);  cout << endl;  copy(V.begin(),V.end(),out);  cout << endl;}

template<typename T>class sort_by_selection{public:  void operator()(T& v)  {    T::size_type length=v.size();    if(length<2) return;    for(int k=0;k<length;++k)    {      int min=k;      for(int j=k+1;j<length;++j)        if(v[j] < v[min])          min=j;      T::value_type t=v[k];      v[k]=v[min];      v[min]=t;    }  }};template<typename T>class sort_by_insertion{public:  void operator()(T& v)  {    T::size_type length=v.size();    for(int k=1;k<length;++k)    {      T::value_type key=v[k];      int j=k-1;      while(j>=0 && key < v[j])      {        v[j+1]=v[j];        --j;      }      v[j+1]=key;    }  }};template<typename T>class sort_by_bubble{public:  void operator()(T& v)  {    T::size_type length=v.size();    for(int k=0; k<length-1; ++k)    {      for(int j=length-1; j>k; --j)      {        if( v[j] < v[j-1] )        {          T::value_type t=v[j-1];          v[j-1]=v[j];          v[j]=t;        }      }    }  }};

Jason Levitt has been teasing me in our discussions on cross-domain requests about Yahoo's upcoming authentication API.
The recurring problem: how to offer web APIs that can be mashed up but involve personal data?
You want to allow for a large number of third parties to integrate with your services, but don't want phishing sites to abuse them.

Let me do a quick re-cap of the problem space before analyzing the pieces of Yahoo's solution.

Communication techniques:

• cross-domain web APIs using script includes,
• accessing web APIs accross domains using a web proxy,
• using a client-based cross-domain solution (see FlashXMLHttpRequest),
• neutered XMLHttpRequest with cross-domain support (see Chris Holland's ContextAgnosticXMLHttpRequest and Douglas Crockford's JSONRequest),
• cross-document messaging across domains (see CrossDocumentMessageEvent from the WHATWG).

In all these cases, there is no good authorization story, that would allow for working with personal data stored in the service in a secured way.

Authorization techniques:

The problem is that API keys and crossdomain policy files are too restrictive because the service needs to decide which third-parties to let in.

On the other end, access control based on the user authentication cookies are very open to un-planned integration, but also create a huge phishing risk.
This is a classic example of the confused deputy problems that appear in principal-based security models.

As a result, most web APIs today don't involve any user data (search, maps, ...) or non sensitive user data.

Yahoo APIs:

Let me re-iterate that I don't think this protocol is about Identity, unlike Passport, TypeKey or CardSpace (aka. InfoCard), but rather simply authority and access. This characteristic is important: we want services to cooperate without being tighly coupled at the identity level. Drew Dean's slides frames the issue as allowing 'Pseudonymous delegation of partial rights', which means the names of a user in different services don't have to match and the authority that is granted is granular.

What's great about this model is that the authority carried by a capability can be as granular as the design and scenario require, and are only be given out to third-parties under certain conditions, which again are chosen to fit the desired requirements and user experience.

For example, the authority granted could vary in range in action and scope: a handle could give access to the user's entire data, or maybe only partial access to part of the user's data.
The design of the capabilities could also comprise additional dimensions, such as a time restriction. For example, a capability could be only valid for 24 hours.
One of the myths of capability systems is that capabilities cannot be revoked. It is actually possible and in Yahoo's design, any granted authority can be revoked by the user at any time.

One common policy for giving out capabilities is to get consent from the user. The screenshots of the F-Spot integration with Flickr (found on this thread) show the Yahoo consent UI.
Although I don't like the desktop/web integration in this scenario and I have some concerns about repeatedly prompting the user for consent, I believe that this approach has a lot of potential for cross-domain service integrations on the web.
Cross-domain support in browsers will be the main remaining link missing to unleash some really cool web apps. In the meanwhile, you can use FlashXMLHttpRequest or some other cross-domain workaround.

I look forward to reading the documentation when the protocol is released and trying out the resulting user experience in practical scenarios. Let me know if you find any other information.
Jason mentioned that the protocol is open and can be simply implemented, which means that it could be supported by other services and hopefully used in a wide variety of mashups.

The 'FuckedGoogle' blog brings up a good question: how do you prove whether a company like Google is identifying and accounting for fraudulent clicks honestly?
This is important because, as that blog pointed out many times, Google has a conflict of interest in being transparent and solving this problem, as it benefits from the fraud.

The post brings up two possible scenarios: an insider releases some damning evidence or a click fraud artist reveals himself and demonstrates how to circumvent any protection that might be in place.
The problem is that the second scenario is unlikely: click fraud scammers want to keep their activities secret.

How to break this secrecy? Here's a provocative approach.
It only requires 1 million dollars of investement ;-)
You would build a contest: '1 million dollars to whomever can provide unrefutable proof of clickfraud of a certain amount over a certain period of time'.
Then you use that proof in a class action lawsuit against Google where you make your million back and more...

A variant of this would be for the DOJ and the de-frauded parties to make a deal with a click frauder, insuring that he would not be pursued and that he can get a portition of any settlement money from the class action lawsuit supported by the evidence he provided.
If the deal takes place because the click frauder was himself sued for click fraud, then the deal wouldn't need to include any financial compensation, only the 'get out of jail free' incentive.

Update (2006/07/16): FuckedGoogle now returns a 404. Not sure what's going on.

..or at least that's what 10 months of spam is telling me.

There is this unanswered entry.

I wonder if none of the readers noticed as SpamBayes filtered it all out. :-P  Let OSCON come and go with nary a signal and let's mothball it and move the discussions to the wiki.

The Observer Pattern in Python

I find that the best way to get a deep understanding of a programming concept is to code it up yourself.  Here's my take on the standard Observer Pattern.  I wanted to fully understand it, because I expect to need it shortly.  This also gave me the opportunity to learn about the handy-dandy weakref module.

This implementation is a bit different from the standard GoF one.  For one thing, I wanted a clean way for an Observer to be able to observe multiple Observables, while being able to easily know which Observable was sending it an update notification. That's why, when registering an Observer with an Observable, the Observer tells the Observable the name of the Observer's function to call for the update notification.

Another difference has to do with the methods that the Observer base class provides.  Now, strictly speaking from a Python perspective, the Observer base class is unnecessary.  Any class that provides a method with the right signature can be used as an Observer.  But when I was talking out this design in my head, I kept using phrases like 'The observer tells the observable that it is interested in it.'  This really seemed like an action performed by the Observer, so I wanted that behaviour in an Observer base class.

The last big difference has to do with the way an Observer subscribes and unsubscribes to update notification by an Observable.  The standard GoF implementation has the Observable class having 'attach' and 'detach' methods.  I have an 'attach' method (named addObserver), and a 'detach' method (named removeObserver).  But I'm also using weakref.WeakKeyDictionary as the way that an Observable keeps a list of its Observers.  This means that the fact that there's a reference to the Observer in the Observable does not keep the Observable alive if it goes out of scope or is deleted.  The weakref to the Observer will be automatically deleted from the Observable's list (I know, a dictionary is not a list.  I'm using the word 'list' here loosely.)  So, the 'removeObserver' method is not strictly needed; just delete an Observer, and it gets removed from the list kept by any Observable it subscribed to.

Here's the code.  It should go in a file named observer.py:

import weakref
import types

##
# The Observer Pattern in Python
#
# Design goals:
# 1. An Observer should be able to observe multiple Observables.
# 2. An Observer should be able to tell an Observable what kinds of
# events it is interested in observing.
# 3. When an Observer is deleted, all Observables that it is observing
# should be notified to remove that Observer.
# 4. When an Observer is notified of an event,
# it should be able to tell which Observable is calling it,
# and which event occured.
##

##
# The abstract Observable Class.
#

class Observable(object):
    
    def __init__(self):
        # A WeakKeyDictionary is one where, if the object used as the key
        # gets deleted, automatically removes that key from the
        # dictionary.  Thus, any Observers which get deleted will be
        # automatically removed from the observers dictionary, thus having
        # two effects:
        # We won't have references to zombie objects in the dictionary, and
        # We won't have zombie objects, because the reference in this
        # dictionary won't stay around, and so won't keep the deleted object
        # alive.
        self._observers = weakref.WeakKeyDictionary()

    ##
    # Add an observer to this Observable's notification list.
    # @param observer The Observer to add.
    # @param cbname The name (as a string) of the Observer's
    # method to call for an event notification, or None for the default
    # 'update' method.
    # @param events The events the Observer is interested in being
    # notified about.  None means all events.

    def addObserver(self, observer, cbname=None, events=None):
        if cbname is None:
            cbname = 'update'
        if events is not None and type(events) not in (types.TupleType, 
                types.ListType):
            events = (events,)
        self._observers[observer] = (cbname, events)

    ##
    # Remove an observer from this Observable's list of observers.
    # Note that this function is not strictly required, because when a
    # registered Observer is deleted, the weakref mechanism will cause
    # it to be removed from the notification list.
    # @param observer the Observer to remove.

    def removeObserver(self, observer):
        if observer in self._observers:
            del self._observers[observer]

    ##
    # Notify all currently-registered Observers.
    # Each observer must have an 'update' method, which should take
    # three parameters (in addition to self): the Observable, an event,
    # and a message.
    # This method will be
    # called if the event is one that the Observer is interested in,
    # or if event is 'None', or if the Observer is interested in all
    # events (it was registered with an event list of 'None').
    # @param event The event to notify the Observers about.  None
    # means no specific event.
    # @param msg A reference to any data object that should be passed
    # to the Observers, or None.

    def notifyObservers(self, event=None, msg=None):
        for observer, data in self._observers.items():
            #print 'data is', data
            cbname, events = data
            #print 'cbname is', cbname
            #print 'events is', events
            if events is None or event is None or event in events:
                cb = getattr(observer, cbname, None)
                if cb is None:
                    raise NotImplementedError, 'Observer has no %s method.' % 
                        cbname
                cb(self, event, msg)

##
# The abstract Observer Class
# This class is a mix-in to add Observable registration methods to
# a concrete Observer class.  It is not strictly required.
#

class Observer(object):

    ##
    # @param observable The Observable to observe.
    # @param The events this Observer is interested in being
    # notified about.  This should be a tuple or list of events.

    def __init__(self, observable=None, cbname=None, events=None):
        if (observable is not None):
            observable.addObserver(self, cbname, events)

    ##
    # Inform an Observable that you would like to be notified when
    # an interesting event occurs in the Observable.
    # @param observable The Observable this Observer would like
    # to observe.
    # @param cbname The name (as a string) of the Observer's
    # method to call for an event notification, or None for the default
    # 'update' method.
    # @param events A tuple or list of events this Observer would like
    # to be notified of by the Observer, or None if it would like to
    # be notified of all events.

    def subscribeToObservable(self, observable, cbname=None, events=None):
        assert observable is not None, 'Observable is None'
        observable.addObserver(self, cbname, events)

    ##
    # Inform an observable that this Observer is no longer interested in it.
    # Note that this function is not strictly required, because when a
    # registered Observer is deleted, the weakref mechanism will cause
    # it to be removed from the Observable's notification list.
    # Use this function when you want to unsubscribe an Observer
    # without deleting it.
    # @param observable The Observable that this Observer no longer wants
    # to observe.

    def unsubscribeToObservable(self, observable):
        assert observable is not None, 'Observable is None'
        observable.removeObserver(self)

Here's the code for the unit tests, which also serves as usage examples.  It should go in a file named test_observer.py:

#! /usr/bin/env python

import unittest
from observer import Observable, Observer

EVENT_FOO, EVENT_UPDATE = range(2)

class Stuff(Observable):

    def __init__(self):
        Observable.__init__(self)
        self._data = None

    def setData(self, data):
        self._data = data
        self.notifyObservers(None, data)

    def setDataWithUpdateEvent(self, data):
        self._data = data
        self.notifyObservers(EVENT_UPDATE, data)

    def setDataWithFooEvent(self, data):
        self._data = data
        self.notifyObservers(EVENT_FOO, data)

class StuffWatcher(Observer):

    def __init__(self, observable=None, cbname=None, events=None):
        Observer.__init__(self, observable, cbname, events)
        self._data = None
        self._updateData = None
        self._reportData = None

    def update(self, observable, event, msg):
        self._data = msg
        self._updateData = msg

    def report(self, observable, event, msg):
        self._data = msg
        self._reportData = msg

class TestCase_01_Observer(unittest.TestCase):

    def test_01_noMethod(self):
        observable = Stuff()
        observer1 = StuffWatcher(observable, 'foo')
        self.assertRaises(NotImplementedError, observable.setData, 10)

    def test_02_simple(self):

        observable = Stuff()
        observer1 = StuffWatcher(observable)
        observer2 = StuffWatcher(observable)
        self.assertEqual(len(observable._observers), 2)

        observable.setData(10)
        self.assertEqual(observer1._data, 10)
        self.assertEqual(observer2._data, 10)

        observable.setData(20)
        self.assertEqual(observer1._data, 20)
        self.assertEqual(observer2._data, 20)

        del observer1
        self.assertEqual(len(observable._observers), 1)

        observable.setData(30)
        self.assertEqual(observer2._data, 30)

        observer3 = StuffWatcher()
        self.assertEqual(len(observable._observers), 1)
        observer3.subscribeToObservable(observable)
        self.assertEqual(len(observable._observers), 2)
 
        observable.setData(40)
        self.assertEqual(observer2._data, 40)
        self.assertEqual(observer3._data, 40)

        del observer2
        self.assertEqual(len(observable._observers), 1)

        observable.setData(50)
        self.assertEqual(observer3._data, 50)

    def test_03_specificEvents(self):

        observable = Stuff()
        observer1 = StuffWatcher(observable, events=EVENT_UPDATE)
        observer2 = StuffWatcher(observable, events=EVENT_FOO)

        observable.setDataWithUpdateEvent(10)
        self.assertEqual(observer1._data, 10)
        self.assertEqual(observer2._data, None)

        observable = Stuff()
        observer1 = StuffWatcher(observable, events=(EVENT_UPDATE, EVENT_FOO))

        observable.setDataWithUpdateEvent(10)
        self.assertEqual(observer1._data, 10)

        observable.setDataWithFooEvent(20)
        self.assertEqual(observer1._data, 20)

    def test_03_multipleObservables(self):
        observable1 = Stuff()
        observable2 = Stuff()
        observer = StuffWatcher()

        observer.subscribeToObservable(observable1, 'update', EVENT_UPDATE)
        observer.subscribeToObservable(observable2, 'report', EVENT_FOO)
        self.assertEqual(len(observable1._observers), 1)
        self.assertEqual(len(observable2._observers), 1)

        observable1.setDataWithUpdateEvent(10)
        self.assertEqual(observer._data, 10)
        self.assertEqual(observer._updateData, 10)
        self.assertEqual(observer._reportData, None)

        observable1.setDataWithFooEvent(20)
        self.assertEqual(observer._data, 10)
        self.assertEqual(observer._updateData, 10)
        self.assertEqual(observer._reportData, None)

        observable2.setDataWithUpdateEvent(30)
        self.assertEqual(observer._data, 10)
        self.assertEqual(observer._updateData, 10)
        self.assertEqual(observer._reportData, None)

        observable2.setDataWithFooEvent(40)
        self.assertEqual(observer._data, 40)
        self.assertEqual(observer._updateData, 10)
        self.assertEqual(observer._reportData, 40)

if __name__ == '__main__':
    unittest.main()

Python and X10 Home Automation, Part 1.1

I wrote previously about Project WiSH, which gives you access to X10 computer interface controllers via device drivers.  I also wrote about uprading my home system to Mandrake 10.  Mandrake 10 has the new 2.6 kernel.  Sadly, WiSH does not (yet) work with that kernel.  So, I had to find an alternative.

From googling around, and comments from readers, I found the 'bottlerocket' software, for accessing the CM-17a 'Firecracker' X10 computer interface.  This gives you a simple command-line program for issuing X10 commands via the CM-17a.  It lacks the ability to watch for an X10 command coming in from a sensor, or querying the status of a status-capable X10 controller, so it's not a complete replacement for WiSH (but that functionality is not supported by the CM-17a anyway, so you don't get it with WiSH either), but it will do me for now.

When you are doing time-based automation under Linux, it makes sense to leverage Linux's built-in abilities.  The 'cron' system lets you schedule tasks for execution based on date/time, either periodically on a period of your choosing, or at one particular data/time.  The 'at' command lets you schedule a command for execution at one particular time in the furture.  For example, typing the following command at the Linux command-line prompt will schedule the running of the command '/usr/local/bin/br a2 on' for 8:00pm today ('br' is the bottlerocket command):

at 8:00pm today <<EOF
/usr/local/bin/br a2 on
EOF

To test the usability of 'at' for X10 home automation, I wrote a simple shell script that issues two 'at' commands -- one to turn on my driveway lights at dusk, and one to turn them off at 10:30pm.  I then set up a 'cron' job to run this shell script ever day at 00:05 in the morning.  Thus, each day the two 'at' jobs are reissued for the current day.  It's necessary to reissue the commands each day because an 'at' job is a one-shot deal, while 'cron' is what you use to run commands at regular intervals.

This system of 'at', shell scripts, and 'cron' works fine, and demonstrates the simplest Linux/X10 home automation setup.  For your needs, this may be all that you require.  For me, this was just baby's first steps.

What I want is a more capable solution to the X10 home automation problem.  I want to be able to schedule both periodic and one-time events.  I want a system that understands how to deal with times like 'sunset' and 'sunrise'.  I want to be able to alias the X10 house/unit code that controls my driveway lights as 'driveway lights'.  I want to be able to create macros that trigger multiple commands, so that I can execute macro 'wakeup', and have the commands sent to turn on the TV, the coffee maker, and the window blinds opener(insert images of George Jetson being ejected from bed like toast from a toaster).  I want to have a web interface to all of this.  And, of course, I want to do it all with Python.

Sure there are several other pre-existing Linux solutions for X10 home automation.  For fun and learning, stay tuned as I put together my own solution using Python.

Python and X10 Home Automation, Part 1

I recently saw an ad from x10.com for a free (you pay shipping) X10 starter kit, including a 'Firecracker' computer interface.  That was a deal I couldn't pass up, so I ordered it through their web site, and 3 days later, the kit arrived.

The kit consists of the CM-17a 'Firecracker' serial computer interface, which transmits via radio, a transceiver module which receives the radio commands from the Firecracker and retransmits them via the X10 protocol over your house wiring, a lamp modules for controlling... lamps, and a PalmPilot-sized hand-held remote control that lets you manually do what the computer interface does.  Oh, and the transceiver module also double as an appliance module, allowing you to control appliances of up to 500 watts.

With the hand-held controller, you can control any X10 modules you have, either the ones that come with the kit, or any add-on modules you may want to buy.  You could go wild, like many do, and completely automate your home -- lights, appliances, garage door, pool heater, ferret feeder, whatever.

But with the computer interface, things get much more interesting.  You can, for example, download from x10.com a free application that duplicates the appearance and functionality of the hand-held controller on your computer screen.  Or, you can download, for $20, an application that fully utilizes your computer and the x10 interface to do full automation.  Want your hot-tub to turn on at a certain time every day?  No problem.  Want your lights to simulate an occupied house while you are on vacation?  Easy.

Naturally, hand an X10 computer interface to a Python programmer, and he'll immediately start writing code for it.  Or that was my intent, anyway.  The first thing I did was google around for any existing Python projects for X10. I found two, Pyxal and Pyx10.  Both projects seem to be unmaintaned.  Pyxal is pure Python, and does not support the recent X10 controllers, like the Firecracker.  Pyx10 uses a wrapper to turn the XAL library into a Python extention module.  It supports recent X10 controllers, including the Firecracker.

I downloaded and examined both.  Pyxal was right out, as it has no Firecracker support (why not add it yourself, you ask?  I'll get to that in a moment...).  Pyx10 and XAL looked good.  After compiling and installing XAL (a snap), I tried compiling Pyx10.  Nope.  The wrapper code for XAL would not compile.  From a quick exam, it looked like it was out-of-sync with XAL.

I could have continued hacking at it to get it to work, but further googling (the trademark police are gonna get me), I found Project WiSH, a project for turning X10 device drivers into... well, Linux device drivers.  Super!  Instead of having to do low-level device handling from my code, I can simply open a linux device driver and write commands to it, just like I was writing text to a file.  And WiSH was a snap to compile and install.  Just make sure you have your kernel source loaded on your machine.  (For the CM-17a 'Firecracker', be sure to download the 1.6.10 version of WiSH.  The later 2.0.1 version does not yet support it.  But both versions support the CM-11a, which is the other modern popular X10 computer interface controller.)

Now, I do my work under Linux, so this is just what the code doctor ordered.  Actually, it's even better than it sounds.  You see, there's this little bit of info about that Firecracker X10 controller...

If you look at one of the other X10 computer interfaces, say the CM-11a that comes with another of the home automation intro packages that x10.com sells, you will see that it is controlled via the computer in a manner rather like an external serial modem.  Connect it to your serial port, and send it strings of ASCII characters.  Not so with the CM-17a 'Firecracker'.  This little guy is a serial pass-thru 'dongle', very small.  From what I can tell from my Google research, you must directly control the radio transmitter in it via bit-tiddling the RTS and DTR lines of the serial port.  You must assemble a 5-byte command via bit masking, then bit-shift it out to the CM-17a by directly controlling the states of the RTS and DTR lines, doing the timing yourself.  There are no smarts.  Ouch.  No wonder this is the bargain-basement controller.

The CM-11a controller has another advantage, too.  It's smart, it has its own processor.  So you don't even need to leave your computer on to do real-time home automation.  Use the scheduling software to send it commands, like 'turn on my security light at local-time dusk, and turn it off at dawn', and the CM-11a will do it, all by itself.

But I don't have the CM-11a.  I have a CM-17a and a Linux box.  Add in the device drivers from Project WiSH, and from a Linux command line, I can execute 'echo 'on' >>/dev/x10/a1', and send the 'on' command to the X10 device at house code 'A', unit code '1'.  How cool is that?

OK, how can we combine equal portions of X10, Project WiSH, Linux, Python, and fun?  (OK, fun gets a bigger portion.)

Here's the deal.  I work for a major software house.  We do automated nightly compiles of our code on all of the platforms we support (Linux, various flavors of UNIX, Windoze).  The last thing you want is for some code change you made that day to 'break the build'.  The automated process sends out email giving that night's build status.  If you broke the build, it's supposed to be your first priority to fix it.

I keep forgetting to check my email.  I have many projects, they grab my attention, and it may be hours before I check my mail.  Yes, I have a little task bar thingie that tells me if I get new mail.  I don't look at it if I'm concentrating on a problem.

Python and X10 to the rescue!  (This is a fun solution looking for a problem.)  I now have a Python script that is run via cron every 10 minutes.  It uses the poplib and email modules to grab and parse my email, looking for the specific patterns that a 'you broke the build' message will contain.  If it finds such a message, it opens and writes an 'on' command to the proper X10 device driver, which then turns on the BIG RED ROTATING LIGHT.  I kid you not.

This is so much fun!