betabug... Sascha Welter

home english | home deutsch | Site Map | Sascha | Kontakt | Pro | Weblog | Wiki

Entries : Category [ zope ]
All around the Zope application server
[digital]  [language]  [life]  [security]  [media]  [zope]  [tourism]  [limnos]  [mac]  [athens]  [travel]  [montage]  [food]  [fire]  [zwiki]  [schnipsel]  [music]  [culture]  [shellfun]  [photography]  [hiking]  [pyramid]  [politics]  [bicycle]  [naxos]  [swim] 

13 June 2008

Patch fun: have a look

Wu's learning resource online
 

A few days ago I reported about Wu's "learning Zope product", where I helped out with patches and advice. In the meantime Wu has put some resources online: There's svn public access and a little project description on his trac (with instructions to get the code via svn).

Note that this isn't a full fledged, "production quality", working product, it's more something to look at and learn (and if you know better, send patches and improvements). If you want to learn, I suggest you follow some of the patches in the timeline - they're each like a small mini-lesson in Zope Product programming. There's one where we are putting in a Catalog, one where we start batching results, ... and whenever we find time we will add more.


Posted by betabug at 09:01 | Comments (2) | Trackbacks (0)
02 July 2008

PNG, Transparency, IE6, AlphaImageLoader, and SSL

Well, you did not expect to get away without problems here, did you?

In case you try to run one of the AlphaImageLoader fixes out there in order to teach Internet Explorer 6 to handle PNG images with transparent alpha channel information somehow gracefully... and in case you try to run this over SSL (https), you might or might not have run into problems where your images all disappeared. One moment, while you where still testing over unencrypted http, everything was there, then you go the the HTTPS site and all you see is your images blinking up before disappearing.

Been there, done that. It's the same problem with IE6 and SSL over and over. What you have to do is give those images a header that allows them to be cached. In Zope it will be enough to associate them with an "Accelerated HTTP Cache Manager". It's still not very nice and there are lots of caveats. We didn't expect better from IE though, did we?

Forgot to mention what got me to the solution:

This problem is happening because Internet Explorer requires that any content retrieved by the browser that is to be opened by a plugin must be cached. In this case, the ImageBundle .png file is opened by AlphaImageLoader, which qualifies as a plugin.

(From this google-web-toolkit issue page.)


Posted by betabug at 16:19 | Comments (1) | Trackbacks (0)
09 July 2008

Zope Incremental Backup Script

What I use now
 

With your Zope installation comes the repozo.py script, that allows you to do incremental backups of the ZODB (the Data.fs file). Around that script most people build a shell script and run it through a cron job. Since I just did a new installation (yeah, that's rare for me), I brushed up what I use so far. I'm now automatically keeping 2 generations of backups, whenever a "generation switch" is done, all the moving, deleting etc. is done by the script. The resulting main folder I rsync to a remote machine. Here comes the script...


In fact there are some parts of the shell script that could be done cleaner (e.g. my argument parsing suxx). But the script works and requires little to no work. Setup is by manually adjusting some paths in the script, for a solution that you want to run unchanged on many hosts, you'd likely want to do that with setup files or command line parameters.

Anyway here is what I have now:

#!/bin/sh

# settings:
# (directories without ending slash "/")
ZOPE_HOME="/usr/local/lib/zope29"
INSTANCE_HOME="/var/zope/instancename"
BACKUP_TO="/otherdisk/path/to/zopebackup"

SOFTWARE_HOME="$ZOPE_HOME/lib/python"
DATA="$INSTANCE_HOME/var/Data.fs"
export PYTHONPATH="$PYTHONPATH:$SOFTWARE_HOME"

# determine if full or incremental from command line argument
TYPE="incremental"
if [ $# -eq 1 ]; then
    if [ $1 == "-f" ]; then
        TYPE="full"
    fi
fi

if [ $TYPE == "full" ]; then
    echo "*** Full backup ..."
    mkdir -p $BACKUP_TO
    rm -r $BACKUP_TO/old
    mv $BACKUP_TO/current $BACKUP_TO/old
    mkdir $BACKUP_TO/current
    $ZOPE_HOME/bin/repozo.py -B -F -v -r $BACKUP_TO/current -f $DATA
else
    echo "*** Incremental backup ..."
    mkdir -p $BACKUP_TO/current
    $ZOPE_HOME/bin/repozo.py -B -v -r $BACKUP_TO/current -f $DATA
fi

As the script is based on various pieces floating around on the net, feel free to copy, improve and share again!

My setup:

When you "pack" your Data.fs, repozo.py will start with a full backup again, but the "remaining" files from the previous incremental backups won't be cleaned up. They will be taken care of in the next "generation switch", so I don't really care.

Posted by betabug at 14:13 | Comments (3) | Trackbacks (0)
25 October 2008

Zwiki 0.61 Released - Cleanup and Catchup

Moving ahead

Woke up this lazy Weekend noon to find Simon has released 0.61 of the stable branch of Zwiki. This is not some bombastic feature release, rather a version that's catching up and cleaning up some stuff. I'm happy that Simon did this, since I've been slacking too much on this project, maybe it will get me going a bit again now.


Posted by betabug at 12:01 | Comments (0) | Trackbacks (0)
18 February 2009

Move a Python module from one Zope Product to another

From the 'Movers and Shakers' department
 

Last December I started on a another refactoring of our company's Zope Application. This time I was going to split out some of the stuff into separate Products, so I could install them only when needed. All nice and clean, but the instances of those classes / products in the ZODB didn't like their code to be moved out from under their feet. I asked around on #zope and TresEquis showed me what to do. Since this stuff is kind of hard to find, here is a little writeup.


Basically what we are doing is to place a snippet of code into the __init__.py of the new location, then remove the registerClass() calls from the __init__.py of the old location. That snippet says something like "I'm now responsible for these modules formerly found there." Lot's of talk, here is what it looks like:

__module_aliases__ = (
                        ( 'Products.OldProd.mod1',
                          'Products.NewProd.mod1' ),
                        ( 'Products.OldProd.mod2',
                          'Products.NewProd.mod2' ),
                     )

In our example this would be placed in the __init__.py of NewProd.

As Simon noticed, you can use a list of tuples too. If you want to move the modules around in the same Zope Product, that will work too: Fix the import lines, declare __module_aliases__, all in the same __init__.py.

Tres offered some more hints on this thing, basically he wrote:

[This] creates aliases to the old names in sys.modules. You probably want to use them to support a content migration or at least touch and store all content.

So far in my experience it has worked fine, but I don't know about the longevity and trustworthyness. of this solution. Myself I will follow Tres advice and have every instance touched and re-stored in the ZODB at some point (lots of disk thrashing involved).

Posted by betabug at 09:34 | Comments (0) | Trackbacks (1)
29 April 2009

Connect Zope to ZEO across Europe

Long Distance...
 

Just did an interesting test setup here... installed a ZEO server on a machine I have in a housing with good bandwidth in Austria, then connected a Zope instance as a ZEO client to it, from here in Greece. Took me a couple of minutes to setup. I can say it works at least for a first test setup. What this means is that the local machine renders web pages with data from a database that is 1288 km away...


Red tourist train in Athens (Thision area)

More about the setup: In Athens I'm on a quite lame ADSL connection (theoretically 4Mbps/1Mbps). The machine in Vienna has nice, phat pipes. Both have about the same CPU power and RAM setup (dual core 2.x Ghz, 4Gig RAM, so nothing special these days). I'm tunneling the Zope-ZEO connection over SSH port forwarding. This is just a first test setup, so I'd have to think about keeping the ssh connection open despite frequent line drops and sometimes high packet loss on the Greek ADSL connection. I have the ZEO mount point "cache-size" parameter at 30000 objects.

Some observation: Rendering pages is quite fast. Startup of Zope is fast too, but so far I have got almost no data in there, so that means nothing. After startup or restart the first requests take quite some time, probably the cache has to be primed. Have to think about persistent ZEO caches here maybe. The first write to the DB took quite long too, but that was the "main" object of my application. Further writes to the db seem to be done with an acceptable speed, but then I'm right now a single user doing almost nothing here.

There are a lot of "ifs" and "buts". My idea of a setup is a ZEO+Zope setup for serving pages on the fast lines in Austria. For updating content, there would be a second connection from a local Zope here in Greece. That way, customers would benefit from the fast lines, while we could work locally doing data insertions that are computing and local disk access intensive.

Posted by betabug at 15:34 | Comments (0) | Trackbacks (0)
24 July 2009

Authenticating Multiple Subdomains with CookieCrumbler

Patch a litte bit here and there
 

When Zope 2 devs want to provide users with an HTML login form, the tool to use is Shane Hathaway's CookieCrumbler product. On one of our sites, I wanted users to be able to authenticate with one login form to multiple subdomains: www.example.com, download.example.com, mail.example.com. Which, by the HTTP cookie specification is possible (it's not possible to have a cookie work on example.org and www.example.org - there have to be 2 dots in the domain). The CookieCrumbler code didn't allow for this, so I hacked a little patch together...


Basically the patch adds a "cookie_domain" property and the associated code to set the "domain" setting on the auth cookie. The property you set to ".example.com" (for our example). The web server will then deliver the same auth cookie to all *.example.com domains. Remember to use SSL for all logged in pages.

Download the patch and apply with the usual patch -p0 < cookiecrumbler_domain.patch in the CookieCrumbler product folder. Since I wrote this patch against a CC that was already patched with the "log auth names to the access log" patch, it might not apply clean to a stock CC. In that case you'll need to manually clean things up a bit.

Posted by betabug at 09:48 | Comments (1) | Trackbacks (0)
18 November 2009

Catching Python xmlrpclib Exceptions

Don't you just like these?
 

When using Zope, you don't want to have bare "except:" clauses lying around [1] and when using xmlrpclib in Zope, you will have to guard against whatever connection trouble there may be. Problem is, xmlrpclib seems to sometimes report errors by raising some nonstandard looking exceptions that show up in a traceback like this:

error: (61, 'Connection refused')

Uhm, yeah. In order to catch "error" with a try-except, I should know where it comes from. Lucky me, the traceback mentions that this one is from:

Module httplib, line 630, in connect

Looking there, I see that the code is raising socket.error.

xmlrpclib also has a few errors of its own. I should decide which ones I want to catch, but in general I can do something like this:

from socket import error as socket_error
#... lots of stuff...
try:
    result = server.xmlrpc_update_address_company(id, data)
except (socket_error, xmlrpclib.Fault, \
    xmlrpclib.ProtocolError, xmlrpclib.ResponseError), error_code:
    handle_error(error_code) # whatever you do...

This will get most tracebacks off the back of my users, but of course now the burden of handling the errors is on me. That's the life of the programmer.

[1]because those broad "except:" clauses will also catch ConflictErrors, which Zope should be let to handle by itself. It's also bad style even in general Python.

Posted by betabug at 11:35 | Comments (0) | Trackbacks (0)
14 December 2009

Compile Python 2.4 on Mac OS X 10.6 Snow Leopard

Old stuff on newish stuff - "reminder to self" post
 

For older Zope versions it is still necessary to use Python 2.4 versions. To compile Python 2.4 on Mac OS X Snow Leopard, I had to jump through a few little hoops:

First I downloaded and unpacked the python tarball, then I applied the following patches:

After this little process the python2.4 binary works just fine.


Posted by betabug at 12:46 | Comments (9) | Trackbacks (0)
13 January 2010

DateTime SyntaxError on PDT Timezone

For example in the MiniPlanet
 

My MiniPlanet gave me SyntaxErrors on one feed, due to the feed in question being in the timezone "PDT". Problem is that Zope's DateTime module does not know that timezone. A quick search and some reading up suggested the following patch to $SOFTWARE_HOME/DateTime/DateTime.py:

--- old_DateTime.py     Wed Jan 13 10:21:24 2010
+++ DateTime.py Wed Jan 13 10:20:46 2010
@@ -246,6 +246,7 @@
            'us/samoa':'US/Samoa',

            'ut':'Universal',
+           'PDT':'GMT-7', 'pdt':'GMT-7',
            'bst':'GMT+1', 'mest':'GMT+2', 'sst':'GMT+2',
            'fst':'GMT+2', 'wadt':'GMT+8', 'eadt':'GMT+11', 'nzdt':'GMT+13',
            'wet':'GMT', 'wat':'GMT-1', 'at':'GMT-2', 'ast':'GMT-4',

(Patchfile, apply as usual, $STANDARD_DISCLAIMER applies, watch your steps!) Patching this directly in the Zope source code isn't the most elegant of solutions, but it works. (At least till the next Zope upgrade.) Well, it could be fixed in newer versions of Zope, what do I know with the old cruft that I'm running there.


Posted by betabug at 10:46 | Comments (1) | Trackbacks (0)
03 February 2010

Having Fun with ZEO

Pull out the tablecloth
 
It's snowy on the higher hills around Athens today

While it's cold (for our standards) with even a bit of snow on the higher hills around town, I'm having fun with ZEO. The thing is, I've got to run a one-off script in one of our instance, doing a catalog query and changing something on every found object. This is ideal to do with a plain old Python Script in the ZMI... except of course I have blocked any attempts to change those objects except through properly secured filesystem code.

Now I could change the permission and restart Zope and edit the objects and go back... and in the process kick out all users twice for the downtime. Don't like. Our ZEO setup offers a fun way of doing this without bothering users:


Because our setup is sporting one ZEO server hosting the same ZODB to multiple Zope instance, I did the code change / restart / edit / code change back / restart cycle only on that instance. The other instance, the one where most of the users are working on remained unchanged. It continued to run with the unchanged code, it wasn't restarted. The result: I got my job done, nobody noticed.

This is no big news for people who use ZEO intensely, some people even keep one Zope instance set up at hand for just those cases. That Zope instance doesn't even need to be constantly running. Another often seen trick is to update instances in round robin fashion, moving users without anybody noticing to different instances. Can't do that when the objects or their APIs have changed too much, but for most of the cases it works just fine.

Posted by betabug at 15:35 | Comments (0) | Trackbacks (0)
02 March 2010

Zope 2.12 eggified Installation Stuff Learned

Learn something every day on #zope
 

There was a short zope-dev meeting on #zope today. I'm not a zope-dev, but after they were done, I asked a question^W^W^Wgriped a bit about the new "eggified" Zope 2.12 install procedure. A big discussion ensued. I learned / noticed a couple of things:

As for the offline-stuff (or the "why do I have to re-download all this each time"), apparently easy_install can do some kind of storage of the eggs it downloads. But this seems to work only on the same platform with the same libraries. I think it's a good idea to ensure that you get the exact same stuff in a reinstall on the same machine, but it won't help when for example the developer machines and the production boxes differ slightly. I haven't looked after it really, but there seems to be another way, setting up a local proxy of PyPi or something... but people sounded a bit like if that is a big, scary thing. Buildout seems to have a better solution there, but after my previous experience and Tres recommendation, I'm not going to go there right now.

Update: just noticed that Tres also mentioned using compoze to make a local "egg store".


Posted by betabug at 22:12 | Comments (2) | Trackbacks (0)
06 April 2010

ValueError: The permission XYZ is invalid

It ain't used really
 

Oh, and while I'm at the topic of "stupid little things I've left off half finished", let me add a note to self: When you get an error "ValueError: The permission XYZ is invalid" in Zope 2, it's because you've set up a permission (e.g. with manage_permission()), but you haven't actually used it to declare security on a method somewhere.

This will probably bite you only if you prepare for using that new permission, then leave things as they are, only to later return and innocently expect unit tests to run through. When in doubt, just add a bogus method that is declareProtected() with your new permission and the Tracebacks will stop blowing up in your face.


Posted by betabug at 16:12 | Comments (0) | Trackbacks (0)
23 June 2010

Running Python's subprocess.Popen with a timeout from within Zope

Yupp, sometimes enough is not enough
 

There are moments when you go outside the world of Python to run something on the command line in a shell. Python's subprocess module makes this doable. Run enough processes on the shell and sure enough, some of them will get stuck. This can spoil your and your users day and should be caught by robust programming practices. A timeout is one such robustness solution. But subprocess.Popen doesn't have an option for a that, even though Guido van Rossum suggested to add a timeout option to it in 2005. There are a couple of suggestions for crafting such a thing in, some more complicated, some less, some extra complicated for working in Windows, some not. I did not have the Windows requirement here, but I ran into another stumbling block.

One of the simplest solutions seems to have been to use the signal module to send a timed SIGALRM to itself. Unfortunately signals in python can be only received in the main thread. That makes the signal solution solution not work in Zope. With some help from Marius Gedminas on #zope, I came up with another solution. I've used threading to spin of a "watchdog" thread that will kill my potentially stuck subprocess after a timeout. Meanwhile, if the main thread finds that the subprocess finished normally, it can cancel out the watchdog. There's even a flag that will be set so the main thread knows if there's been success or a kill...


So, here is the little method I'm using:

import time
import threading
import signal

# ....

def run_popen_with_timeout(command_string, timeout, input_data):
    """
    Run a sub-program in subprocess.Popen, pass it the input_data,
    kill it if the specified timeout has passed.
    returns a tuple of success, stdout, stderr
    """
    kill_check = threading.Event()
    def _kill_process_after_a_timeout(pid):
        os.kill(pid, signal.SIGTERM)
        kill_check.set() # tell the main routine that we had to kill
        # use SIGKILL if hard to kill...
        return
    p = Popen(command_string, bufsize=1, shell=True,
              stdin=PIPE, stdout=PIPE, stderr=PIPE)
    pid = p.pid
    watchdog = threading.Timer(timeout, _kill_process_after_a_timeout, args=(pid, ))
    watchdog.start()
    (stdout, stderr) = p.communicate(input_data)
    watchdog.cancel() # if it's still waiting to run
    success = not kill_check.isSet()
    kill_check.clear()
    return (success, stdout, stderr)

Here is simple a test, which I run with ZopeTestCase and which assumes that run_popen_with_timeout lives in a file called utilities.py:

def test_run_popen_with_timeout(self):
    '''run_popen_with_timeout - check for running/killing a subprocess on timeout'''
    from utilities import run_popen_with_timeout
    input_data = 'bla' # not used really
    command_string = os.path.join(os.path.dirname(__file__), 'takestime.py')
    timeout = 2.0
    success, stdout, stderr = run_popen_with_timeout(command_string, timeout, input_data)
    self.failIf(success)
    self.failIf('done' in stdout)
    timeout = 10.0
    success, stdout, stderr = run_popen_with_timeout(command_string, timeout, input_data)
    self.failUnless(success)
    self.failUnless('done' in stdout, 'no "done" in stdout: ' + stdout + ' stderr: ' + stderr)

The test uses this helper script to simulate a sometimes long running process:

#!/usr/local/bin/python
# a simple test script
# to test run_popen_with_timeout
# save as "takestime.py" in your tests directory
import time
print 'starting'
for i in range(3):
    time.sleep(2)
    print 'waiting'
print 'done'

Posted by betabug at 15:37 | Comments (4) | Trackbacks (0)
23 November 2010

Another Film and a Catalog Reindexing Bug

It's getting better every day
Sign saying "Shock Prices" in Athens' center

Today I got back three color films from development. One of them was a Fuji Pro 400 H, a film I've bought for the first time. So far it looks like the colors are in line with the Fuji 160S that we all love and worship (haven't tried the "replacement" 160NS yet). having about 2 stops more of wiggle space is nice, but the weather was so good I didn't really need it. This being 120 film for exposures in 6x6cm, I didn't see any grain. I need to re-scan most of those pictures though, the lab scans are just not cutting it.

BTW, the sign says: "The crisis brings the solution: Shock prices - everything at cost prices". As the Beatles song says, it's getting better every day :-)

All the while in a world filled with more technology, I'm still battling a bug in one of my Zope products: I've got this thing that is something like a circular reference of indexes. This is of course the point in my blog post, where I have lost even the last of my readers - after film photo geeking it out and then posting about some waaaay off esoteric tech stuff. Well, I'm working on this bug on and off for days now. Which means I have been working on it and doing other stuff in the meantime, since a really good solution hasn't come up yet. Like every other bug, at some point in time this one will be eradicated and forgotten too.


Posted by betabug at 22:05 | Comments (0) | Trackbacks (0)
11 March 2011

Test Coverage on Zope Tests

Hey, I discovered it's already there!
 

Sometimes I'm a bit late to the party: This morning I was thinking about checking how much of my code is really covered by my tests. So I searched the f* web for how to run python coverage.py on Zope. Turns out it's already in there, ever since something like 2005. This is really one of the advantages of running a stable, mature platform like Zope.

What you need to do is to run zopectl test with the additional argument --coverage test_dir, e.g. I did:

./bin/zopectl test --coverage cov_test -k -m FinancialService

Then I found reports for every .py file in my FinancialService Product in the cov_test directory. The reports list how many times each line was run -- and they mark lines that weren't run at all. So far the test output is complaining about not giving coverage for functional.txt which is a file with functional doctest - this file consists of tests itself, I guess not testing it is ok. Since running the tests like this also takes much longer, I also had one test failing where a test on a timestamp failed.

Now I can fill the glaring holes in my test coverage. Sometimes I've missed a complete method, sometimes it's just a twisting turn in an if-clause. Testing fun, here I come!


Posted by betabug at 10:41 | Comments (0) | Trackbacks (0)
Prev  1   2   3   4   5   6   [7]   8   Next