Serving Large Files with Zope

Here are some timings using asyn_http_bench before and after a small change was applied to ZServer/HTTPServer.py:

asyn_http_bench myhost 80 /myurl 3 3

/myurl returns a string of varying length (10Mb, 20Mb, etc)

Size	Before	After
10 Mb	26.5 sec	9.8 sec
20 Mb	88.4 sec	20.47 sec
30 Mb	194.6 sec	30.3 sec
40 Mb	connection errors	40.2 sec


$ diff -c HTTPServer.py.ZopeVendor   HTTPServer.py.ZopePatch27
*** HTTPServer.py.ZopeVendor    Thu Sep  7 16:49:08 2006
--- HTTPServer.py.ZopePatch27   Thu Sep  7 16:48:50 2006
***************
*** 74,79 ****
--- 74,81 ----

  class zhttp_collector:
+     ac_in_buffer_size = 1< <20
+
      def __init__(self, handler, request, size):
          from ZServer import LARGE_FILE_THRESHOLD
          self.handler = handler
***************
*** 303,309 ****
          # producers by default
          if self.closed:
              return
!         self.producer_fifo.push(producer)
          if send: self.initiate_send()

      push_with_producer=push
--- 306,315 ----
          # producers by default
          if self.closed:
              return
!         if type(producer) == type(''):
!             self.producer_fifo.push (producers.scanning_producer (producer, self.ac_out_buffer_size))
!         else:
!             self.producer_fifo.push (producer)
          if send: self.initiate_send()

Update: 27/04/2007 – There are some simple optimizations available in asyncore for Zope to serve large files

technorati tags:zope, plone, python

About this entry

You’re currently reading “ Serving Large Files with Zope ,” an entry on Chui's Counterpoint

Published:: 9.7.06 / 7pm

Category:: Python

Lennart Regebro 9.15.06 / 10pm

How about some performace tests for the separate changes? Like with and without the buffer size?

Chui 9.16.06 / 9am

Changing the buffer size clearly improves the performance, I’ve lost the numbers, so I can’t provide exact figures here. There is a tradeoff involved, since you are allocating more memory for each client. This has limits on scalability.

If you are serving large files that are not stored in the ZODB, but in the filesystem, my recommendations are:

1. do not do this “return open(filename).read()”, instead, use the filestream_iterator. I’ve timed clear performance improvements. e.g. 20Mb file with 9 concurrent connections, 88.4 seconds vs 18.7 seconds.

2. when you operate in a LAN, the default buffer size is way too small. This results in asyncore sending too little data at one time. As a comparison, to match IIS in a LAN, at one site, I had to set the output buffer at 4 Mb.

I have a patch which adaptively increments the buffer size so that fast connections get sent more data at one time compared to slow clients. I’ll post it shortly.

I have another patch which modifies asyncore to minimize string creation, but it only gives 5% gain, at the expense of more complicated code.

Chui's Counterpoint

Serving Large Files with Zope

About this entry

2 Comments

Have your say

About

Recently

Categories