Chui’s counterpoint

I was playing around with code generation of some scaffolding-style code for CodeIgniter. Initially, python’s string interpolation operator % sufficed. As time progressed, it became clearer that resorting to a templating library buys me cleaner code.

Problem #1
The $ used by Genshi for string interpolation conflicts with PHP’s $variables.

Solution #1
After 10 minutes of source diving (by the way, Genshi source is a work of art - wish I could write like that).

import genshi.template.interpolation
genshi.template.interpolation.PREFIX = '^'

from genshi.template import TextTemplate
template = TextTemplate("$this->^{foo} = 'home';")
stream = template.generate(foo="customer")
print stream

Problem #2
The generated output quotes the > into >

Solution #2
The Stream render() method takes a parameter specifying the kind of escaping.

print stream.render('text')

Overall Solution

from genshi.template import TextTemplate
import genshi.template.interpolation
genshi.template.interpolation.PREFIX = '^'
for tablename, table in specs.items():
    fd = open('controllers/%s.php' % tablename, 'w')
    template = TextTemplate(open('controllers.php', 'r'), filename='controller.php')
    print template.generate(tablename=tablename, columns=table).render('text')

I was hoping to write a simple tee class by subclassing file, but there must be further type magic getting in the way when print >> is used.

import sys

class tee(file):
   def write(self, text):
     sys.stdout.write(text)
     file.write(self, text)

fd = tee("output.txt", "w")
print >> fd, "Test 1"
fd.write("Test 2\n")

$ python test.py
Test 2

$ cat output.txt
Test 1
Test 2

Using the dis module, we can tell that print >> is compiled into PRINT_ITEM_TO.

>>> def t(): print >> fd, "foo"
...
>>> import dis; dis.dis(t)
  1           0 LOAD_GLOBAL              0 (fd)
              3 DUP_TOP
              4 LOAD_CONST               1 ('foo')
              7 ROT_TWO
              8 PRINT_ITEM_TO
              9 PRINT_NEWLINE_TO
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE

and looking through ceval.c, searching for PRINT_ITEM_TO yields

 case PRINT_ITEM_TO:
    w = stream = POP();
    /* fall through to PRINT_ITEM */

case PRINT_ITEM:
    v = POP();
    if (stream == NULL || stream == Py_None) {
        w = PySys_GetObject("stdout");
        if (w == NULL) {
            PyErr_SetString(PyExc_RuntimeError,
                    "lost sys.stdout");
            err = -1;
        }
    }
    /* PyFile_SoftSpace() can exececute arbitrary code
       if sys.stdout is an instance with a __getattr__.
       If __getattr__ raises an exception, w will
       be freed, so we need to prevent that temporarily. */
    Py_XINCREF(w);
    if (w != NULL && PyFile_SoftSpace(w, 0))
        err = PyFile_WriteString(" ", w);
    if (err == 0)
        err = PyFile_WriteObject(v, w, Py_PRINT_RAW);

Any ideas why?

What would you automate into your boilerplate code after having 10 years of writing database applications? Here are some links to promising projects/essays:

• Andromeda
• Django Admin
• Dataphor

  Promises: User interface “hints” integrated with the data model

• Naked Objects
• MS Access and Query By Form
• I like how CodeIgniter makes it possible to compose declarative data validations e.g. “valid_email|matches[email_confirm]|min_length[6]“
• We need standard ValueConverters for currency, URL, dates and email addresses
• Someone should write an essay on why database synchronization for off-line applications is a bad idea, and why journaling works better
• More ideas here - There are no attributes - CQL sounds cool

What are your thoughts here? Are there any other notable projects in a similar vein?

Sometimes, your Zope application may be unnecessarily writing too many times to the database. An easy way to track what objects are being written to is to use the analyze.py utility. Look at the Count and Pct, and make sure that relatively static objects do not get too many writes.
$ ../../bin/python analyze.py Data.fs
Processed 13382 records in 5734 transactions
Average record size is 1742.56 bytes
Average transaction size is 4066.78 bytes
Types used:
Class Name Count TBytes Pct AvgSize
---------------------------------------------- ------- --------- ----- -------
AccessControl.User.User 1 138 0.0% 138.00
AccessControl.User.UserFolder 1 110 0.0% 110.00
App.ApplicationManager.ApplicationManager 1 122 0.0% 122.00
App.Product.Product 33 37807 0.2% 1145.67
App.Product.ProductFolder 1 2543 0.0% 2543.00
App.special_dtml.HTML 15 46973 0.2% 3131.53
BTrees.IIBTree.IIBTree 1190 137981 0.6% 115.95
BTrees.IIBTree.IIBucket 231 158454 0.7% 685.95

Python developers are probably already familiar with List Comprehensions, and lambda expressions. The interesting twist with LINQ is these expressions are translated into SQL and executed on the RDBMS instead of being done on the client side. There was a pretty clever Python project that achieved this in Python through dissassembly of python bytecodes, but I can’t recall it’s name [Update 23 Apr 2008: see http://svn.aminus.net/dejavu/branches/ldap/logic.py ] . There are some interesting ideas raised in LINQ that even Python developers ought to explore and consider adopting in a future Python. [Edit: 23 Apr 2008: Meaning I'd prefer to see this in Python core through AST transformation rather than bytecode hacks, C# has just got macros.]

1. Setting up a Linq DataContext

   MyDatabaseContext db = new MyDatabaseContext(MyConnectionString); // or MyDatabaseContext db = new MyDatabaseContext();
 

2. Basic LINQ Expression

   var AnIQueryable = from Customer in db.Customers
where Customer.FirstName.StartsWith("m")
select Customer; 


3. Logging the generated queries

   db.Log = Console.out

   The previous example translates into a LIKE operator:

   SELECT [t0].[CustomerID], [t0].[FirstName] AS [FirstName]
FROM [dbo].[Customer] AS [t0]
WHERE [t0].[FirstName] LIKE @p0

4. LINQ with Aggregates

   var AverageRuns =(from Master in this.db.Masters select Master.Runs).Average()
   

   The SQL generated isn’t too shabby either:

   SELECT AVG([t0].[Runs]) AS [value]
   FROM [dbo].[Master] AS [t0]

5. LINQ grouped aggregates

   
      var categories =
         from p in products
         group p by p.Category into g
         select new
                 {Category = g.Key,
                  AveragePrice = g.Group.Average(p => p.UnitPrice)
                  };
   

6. Lambda Expressions

   • Map

     db.Customers.Select(customer => master.Customer.LastName.length)

   • Filter

     db.Customers.Where(customer => customer.LastName.StartsWith("M")

   • Reduce / Fold

     
double[] doubles = { 1.7, 2.4, 3.5, 8.9 };
double result = doubles.Aggregate(0.0, (d1, d2) => d1+d2);


     Note: You can’t use Aggregate directly on IQueryable from database contexts. Instead,
     
var arr_masters = this.db.Masters.ToArray();
var totalRuns = arr_masters.Aggregate
(0, // seed
(int accum, MyDB.Master master) => accum + master.Runs
);


7. Updating Database

   
   this.Validate();
   this.masterBindingSource.EndEdit();
   db.SubmitChanges();
   

8. How to cheat

   Use Microsoft’s 101 LINQ samples

Here’s a Compact Inverted Calendar for Business Cards, good for 2008 and 2009.

              1  2  3  4  5  6  7
              8  9 10 11 12 13 14
             15 16 17 18 19 20 21
             22 23 24 25 26 27 28
             29 30 31
        2008                      2009
         Jun Su Mo Tu We Th Fr Sa Feb Mar Nov
     Sep Dec Mo Tu We Th Fr Sa Su Jun
 Jan Apr Jul Tu We Th Fr Sa Su Mo Sep Dec
         Oct We Th Fr Sa Su Mo Tu Apr Jul
         May Th Fr Sa Su Mo Tu We Jan Oct
     Feb Aug Fr Sa Su Mo Tu We Th May
     Mar Nov Sa Su Mo Tu We Th Fr Aug

Sources: invertedcalendar_py.txt

Tags: calendar

I don’t know what got into the head of the designers of XML, that CDATA escape sections have their own escaping syntax. Lshift has a low down on the details: Escaping XML and CDATA. Good to know that xml.dom.minidom gets this right.

I had to deal with an XMLRPC server that returned data in ‘iso-8859-1′ encoded, but didn’t have the encoding wired into the XML declaration. After some digging around, this hardcoded changes fixed the problem.

    class ExpatParser:
        # fast expat parser for Python 2.0 and later.  this is about
        # 50% slower than sgmlop, on roundtrip testing
        def __init__(self, target):
            src_encoding = 'iso-8859-1'
            self._parser = parser = expat.ParserCreate(src_encoding, None)

Hope it’s useful to you.

Differences in the Interpreter

One gotcha is that the Prolog interpreter runs in a query mode, while code in file runs as assertions. Python, on the other hand, makes no such distinction. Code that you can compile in a file can equally be typed into the interpreter.

i.e. You have to write the rules in a text file, and then load it in the interpreter before querying.

% In a file-------------
% test.pl
% A Person has age, height and weight
person(john,  25, 165, 75).
person(mary, 13, 125, 30).
person(gary,  66, 180, 90).
person(gale,  65, 169, 70).
person(mark, 26, 182, 95).

% ------- Run in the interpreter -----------
% Loads file. equivalent to consult('test.pl').
['test.pl'].

Variables

A rather crude way of doing variables in Prolog is via assertions and retraction.

assert(num_flowers(25)).
num_flowers(NumFlowers).
NumFlowers = 25

List comprehensions, filter

% older = [person for person in persons if age > 20]
findall(
  person(Name, Age, Height, Weight),
  (person(Name, Age, Height, Weight), Age > 20),
  Older).

Older = [person(john, 25, 165, 75),
 person(gary, 66, 180, 90),
 person(gale, 65, 169, 70),
 person(mark, 26, 182, 95)]

Access to members of a tuple (Term) and map

% total_height_older = [person[2] for person in older]

maplist(arg(2), $Older, Heights), sumlist(Heights, TotalHeight).
Heights = [25, 66, 65, 26],
TotalHeight = 182

Mind Reading

Incidentally, SWI-Prolog is astonishingly clever at reading minds. Here’s an interactive session where I misspelt person with erson.

7 ?- erson(john, Age, Height, Weight).
Correct to: person(john, Age, Height, Weight)? yes

Age = 25,
Height = 165,
Weight = 75

Reference

[1] comp.lang.prolog

Actually, the title should be “you shouldn’t bother trying to build python extensions with Visual Studio 2005″. All sorts of complications arise and I haven’t got it working at all. Here’s some references if you enjoy a little self-flaggelation.

1) Make sure you understand manifests. Building DLLs will never be the same again! Richard Grimes has an unflattering write up on MSVCR start up errors.

2) Try and see if DISTUTILS_USE_SDK works or not (It doesn’t with Visual Studio 2005)

3) Checkout this mailing thread Python and MSVC 8.

Here’s the easy way out:

1) Install ActivePython
2) Install Cygwin, then select GCC and MINGW
3) Grab pexports

pexports C:\\Windows\\System32\\python25.dll > python25.def
dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a
cp libpython25.a /usr/lib/.

Here’s the trickiest bit:
When you call python setup.py build -c mingw32, the linker will fail.

Here’s an example:

C:\cygwin\bin\gcc.exe -mno-cygwin -shared
-s build\temp.win32-2.5\Release\mercurial\base85.o build\temp.win32-2.5\Release\mercurial\base85.def
-LC:\python25\libs
-LC:\python25\PCbuild
-lpython25 -lmsvcr71
-o build\lib.win32-2.5\mercurial\base85.pyd
build\temp.win32-2.5\Release\mercurial\base85.o:base85.c:(.text+0x22e): undefined reference to `__imp__PyExc_ValueError'


Manually run the following instead:

C:\cygwin\bin\gcc.exe -mno-cygwin -shared
-s build\temp.win32-2.5\Release\mercurial\base85.o build\temp.win32-2.5\Release\mercurial\base85.def
-LC:\python25\libs
-LC:\python25\PCbuild
-lpython25 -lmsvcr71
-o build\lib.win32-2.5\mercurial\base85.pyd
build\temp.win32-2.5\Release\mercurial\base85.o:base85.c:(.text+0x22e): undefined reference to `__imp__PyExc_ValueError'


Manually build the DLL this way:

$ gcc -mno-cygwin -mdll build/temp.win32-2.5/Release/mercurial/base85.o
build/temp.win32-2.5/Release/mercurial/base85.def
-lpython25 -lmsvcr71 -o build/lib.win32-2.5/mercurial/base85.pyd


To be honest, I don’t know why -shared is broken in my version of ld

$ ld -v
GNU ld version 2.17.50 20060817


Hope this helps you out.

ps. If you find this article useful, please provide some link love. (i.e. link to it)

I have been playing with Mercurial for a little web project. It works fine locally, but when I tried to publish a static copy on my web host, it started playing up.

I was unable to clone the remote repository using:
hg clone static-http://www.redmountainsw.com/phpar/ new_local_dirname

I kept getting this message:

requesting all changes
adding changesets
transaction abort!
rollback completed
abort: integrity check failed on data/install.php.d:0!

For a while I thought the problem had to do with line-endings, as the development was done on a Windows box, and the repository was uploaded to a Unix shared host. This turned out not to be the case after all.

Inspecting the http result with curl,
curl -i http://www.redmountainsw.com/phpar/.hg/data/install.php.i

it turns out that Apache’s php handler was processing the contents of the php file.

Resolution
Create a .htaccess file in the .hg directory like this:

php_flag engine off

Now everything works!

I’ve exceedingly poor memory, so here’s my shortcuts.

python manage.py dumpdata > data.json
python manage.py reset registration
python manage.py loaddata data.json

Readers are probably aware how fond I am of spreadsheets as a tool for rapidly prototyping applications. Here’s another implementation of spreadsheets based around IronPython, but addresses issues like shared updates.

It happens that I often have to dial into users sites to troubleshoot problems, and they usually run our Python software on Windows. Since I often don’t have access to tools like curl, I’d like to present the next best thing. A command-line interface to urllib2 which uses the same options as curl. This script doesn’t require curl binaries.

It doesn’t do everything curl does, but I’ll update it as soon as I need the feature.

Update: Patch from Michael Watkins

#!env python
'''python curl.py [options] url
Options
  -I/--head  headers only
  -i        include headers
  -h/--help
  -u user:pass
  --user=user:pass
'''

import getopt
import sys

options = getopt.getopt(sys.argv[1:],
    'hu:iI',
    ['help', 'head', 'user='])
include_header = 0
header_only = 0
username, password = None, ''
for option, value in options[0]:
    if option in ('-h', '--help'):
        print __doc__
        sys.exit(0)
    elif option == '-i':
        include_header = 1
    elif option in ('-I', '--head'):
        header_only = 1
    elif option in ('-u', '--user'):
        username, password = value.split(':', 2)

import urllib2 as u
urls = options[-1]
for url in urls:
    req = u.Request(url)

    if username != None:
        import base64
        header = 'Basic %s' % \\
            base64.encodestring(
            '%s:%s' % (username,password))[:-1]
        req.add_header('Authorization', header)

    try:
      f = u.urlopen(req)
      if include_header or header_only: print f.headers
      if not header_only: print f.read()
    except Exception, e:
      print e.read()
else:
    print 'try curl --help'

Yang Xing reports:

Developer [sic] should avoid encoding Space into “+” or double encoded into “%2b”. It is recommended that when encode [sic] URL use “System.Uri.EscapeDataString”, when decode URL use “HttpUtility.UrlDecode”

Sigh. There are days when one just wishes one’s back in Python-land.

Contrast with the following Python-equivalent:

>>> urllib.quote('255 m')
'255%20m'
>>> urllib.quote_plus('255 m')
'255+m'
>>> urllib.unquote('255%20m')
'255 m'
>>> urllib.unquote_plus('255%20m')
'255 m'
>>> urllib.unquote_plus('255+m')
'255 m'

This is nothing special, but I posted it for personal reference.

#!/usr/bin/python
import os
hostname = raw_input("Enter hostname: e.g. example.com ")
os.system("/usr/sbin/useradd -s /bin/false %s" % hostname)
print "User added .."

vhost = """

ServerName name.com
ServerAlias www.name.com
DocumentRoot /home/name.com/public_html
CustomLog /home/name.com/web.log combined
CustomLog /home/name.com/main.referer.log referer

"""

open('/etc/httpd/conf/%s' % hostname, 'w' ).write(vhost.replace('name.com', hostname))
print "%s was added" % hostname
os.chmod('/home/%s' % hostname, 755)

>>> os.path.join("/a/b/c", "/d/e/f")
"/d/e/f"

This can be a problem if “/d/e/f” comes from an untrusted source.

As a safety measure, avoid using os.path.join in your web applications, roll your own and call it “safe_join”. You will sleep better.

A value is a value
Verily it is a value
A reference is a reference
Verily it is a reference

道可道，非常道。名可名，非常名。

A value’s existence is eternal¹
But we can’t handle it until a reference is created²

无名天地之始。有名万物之母。

Break out of the python interpreter,
one will grok values actually are.
But running in the interpreter,
one can never know what the true value is³

故常无欲以观其妙。常有欲以观其徼。

But a value and a reference are of the same essense⁴
of the strange machine we exist in⁵

此两者同出而异名，同谓之玄。

As mysterious a language like C is,
it’s the gateway to understanding how python works.

玄之又玄，众妙之门。

Reference: Here’s a good English interpretation of Tao Te Ching

¹Particular for some integers, which are already initialized in the interpreter, or until garbage collected (but only the interpreter knows that)
²Not strictly true:

>>> import dis; dis.dis(lambda: 1+1)
          0 LOAD_CONST               1 (1)
          3 LOAD_CONST               1 (1)
          6 BINARY_ADD
          7 RETURN_VALUE

³The is operator and the id() function come pretty close, but you aren’t inspecting memory here.
⁴both exists in the heap of the Python VM
⁵Python virtual machine

Embedding IronPython is quite trivial and is documented here.

You can call functions in the embedded python module by
(1) retrieving the function using Evaluate()
(2) then apply the function using the Call() method

using (IronPython.Hosting.PythonEngine engine
  = new IronPython.Hosting.PythonEngine())
{
  engine.Execute(@"
  def foo(a, b):
    return a+b*2");

  // (1) Retrieve the function
  IronPython.Runtime.Calls.ICallable foo
   = (IronPython.Runtime.Calls.ICallable) engine.Evaluate("foo");

  // (2) Apply function
  object result = foo.Call(3, 25);
}

The ICallable.Call method can only be called varargs-style, and can’t be called using keyword arguments.
(This is probably a bug.)

Call(params object[] args)

To use keyword arguments, you’ll need to cast the function into PythonFunction.

Note:
Unfortunately, Python classes aren’t CLR classes, and couldn’t be accessed directly or subclassed by C# by referencing a built assembly. You’ll have to go through the hosting interfaces to get an instance of the class.

Gleaned from Usenet

• Varargs in Python functions
• Encoding HTML Entities in Python
• True And False
• Ten sample simple introductory python programs to get you started