A Python curl

Thursday, 12 July 2007

It happens that I often have to dial into users sites to troubleshoot problems, and they usually run our Python software on Windows. Since I often don’t have access to tools like curl, I’d like to present the next best thing. A command-line interface to urllib2 which uses the same options as curl. This script doesn’t require curl binaries.

It doesn’t do everything curl does, but I’ll update it as soon as I need the feature.

Update: Patch from Michael Watkins

#!env python
'''python curl.py [options] url
Options
  -I/--head  headers only
  -i        include headers
  -h/--help
  -u user:pass
  --user=user:pass
'''

import getopt
import sys

options = getopt.getopt(sys.argv[1:],
    'hu:iI',
    ['help', 'head', 'user='])
include_header = 0
header_only = 0
username, password = None, ''
for option, value in options[0]:
    if option in ('-h', '--help'):
        print __doc__
        sys.exit(0)
    elif option == '-i':
        include_header = 1
    elif option in ('-I', '--head'):
        header_only = 1
    elif option in ('-u', '--user'):
        username, password = value.split(':', 2)

import urllib2 as u
urls = options[-1]
for url in urls:
    req = u.Request(url)

    if username != None:
        import base64
        header = 'Basic %s' % \\
            base64.encodestring(
            '%s:%s' % (username,password))[:-1]
        req.add_header('Authorization', header)

    try:
      f = u.urlopen(req)
      if include_header or header_only: print f.headers
      if not header_only: print f.read()
    except Exception, e:
      print e.read()
else:
    print 'try curl --help'
Bookmark and Share

You should follow me on twitter here

The entry 'A Python curl' was posted on July 12th, 2007 at 4:33 pm and last modified on May 12th, 2008 at 7:15 am, and is filed under Python. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

4 Responses to “A Python curl”

  1. Michael Watkins writes:
    No. 1 — July 13th, 2007 at 2:26 am

    A good idea, I run into the same need as well. Other than dumping out the page, I find I most often use curl with -I (headers only) parameter. Here’s a patch…

    % diff curl_orig.py curl.py
    1,2c1,2
    #!/usr/bin/env python
    > ‘python curl.py [-i] [-I] [-h|--help] [-u user:pass] [--user=user:pass] url’
    8c8
    ‘hu:iI’,
    10a11
    > header_only = 0
    19a21,22
    > elif option == ‘-I’:
    > header_only = 1
    28,29c31
    header = ‘Basic %s’ % base64.encodestring(‘%s:%s’ % (username,password))[:-1]
    34,35c36,37
    if include_header or header_only: print f.headers
    > if not header_only: print f.read()

  2. KH writes:
    No. 2 — May 6th, 2008 at 7:43 pm

    That code is broken.

  3. lijan writes:
    No. 3 — June 26th, 2009 at 9:21 pm

    Works for me in OpenSUSE 11.1 with Python 2.6:
    ##############################################
    #!/usr/bin/python
    ””’python curl.py [options] url
    Options
    -I/–head headers only
    -i include headers
    -h/–help
    -u user:pass
    –user=user:pass
    ”’

    import getopt
    import sys

    options = getopt.getopt(sys.argv[1:],
    ‘hu:iI’,
    ['help', 'head', 'user='])
    include_header = 0
    header_only = 0
    username, password = None, ”

    for option, value in options[0]:
    if option in (‘-h’, ‘–help’):
    print __doc__
    sys.exit(0)
    elif option == ‘-i’:
    include_header = 1
    elif option in (‘-I’, ‘–head’):
    header_only = 1
    elif option in (‘-u’, ‘–user’):
    username, password = value.split(‘:’, 2)

    import urllib2 as u
    urls = options[-1]
    if `urls` != ‘[]‘:
    for url in urls:
    req = u.Request(url)

    if username != None:
    import base64
    header = ‘Basic %s’ % base64.encodestring(‘%s:%s’ % (username,password))[:-1]
    req.add_header(‘Authorization’, header)

    try:
    f = u.urlopen(req)
    if ( include_header or header_only ): print f.headers
    if not header_only: print f.read()
    except Exception, e:
    print e
    else:
    print ‘try curl –help’

  4. lijan writes:
    No. 4 — June 26th, 2009 at 9:57 pm

    spacing disappeared in my previous post, so here is patch for the original file(already patched by M.W.) :
    $ diff curl-orig.py curl.py

    1c1
    #!/usr/bin/python
    19a20
    >
    30c31

    32,34c33,36
    < urls = options[-1]
    < for url in urls:
    urls = options[-1]
    > if `urls` != ‘[]‘:
    > for url in urls:
    > req = u.Request(url)
    36,41c38,41
    < if username != None:
    < import base64
    < header = ‘Basic %s’ % \\
    < base64.encodestring(
    < ‘%s:%s’ % (username,password))[:-1]
    if username != None:
    > import base64
    > header = ‘Basic %s’ % base64.encodestring(‘%s:%s’ % (username,password))[:-1]
    > req.add_header(‘Authorization’, header)
    43,48c43,48
    < try:
    < f = u.urlopen(req)
    < if include_header or header_only: print f.headers
    < if not header_only: print f.read()
    < except Exception, e:
    try:
    > f = u.urlopen(req)
    > if ( include_header or header_only ): print f.headers
    > if not header_only: print f.read()
    > except Exception, e:
    > print e
    50c50
    print ‘try curl –help’

Leave a Reply

«
»