betabug... Sascha Welter

home english | home deutsch | Site Map | Sascha | Kontakt | Pro | Weblog | Wiki

Entries : Category [ zwiki ]
Posts about Zwiki - the Zope wiki engine and the easiest CMS on Zope!
[digital]  [language]  [life]  [security]  [media]  [zope]  [tourism]  [limnos]  [mac]  [athens]  [travel]  [montage]  [food]  [fire]  [zwiki]  [schnipsel]  [music]  [culture]  [shellfun]  [photography]  [hiking]  [pyramid]  [politics]  [bicycle]  [naxos]  [swim] 

05 October 2007

Copy a Folder's Contents

One of those small things

Zwiki's new revision system uses a BTreeFolder2 to permanently store revisions. That's a wise decision, because those revisions can really pile up. Some wikis ran with an interim version of the code, where it created a normal folder to hold revisions. What I did there is to rename the folder to "revisions_orig", create a new "revisions" folder and copy the contents over. Here is a little python script that does the copying over, when the cookie used by ZMI copy & paste can't seem to hold them all. Nothing special, just one of those posts to remind myself...

Create a new "Script (Python)" object in the ZMI, fill it in with this code:

### Parameter List: from_folder, to_folder

from_folder = getattr(context.aq_explicit, from_folder)
to_folder = getattr(context.aq_explicit, to_folder)

from_folder_ids = from_folder.objectIds()

from_copied = from_folder.manage_copyObjects(from_folder_ids)

print 'done'
return printed

Running it should be self-explaining.

Posted by betabug at 11:22 | Comments (0) | Trackbacks (0)
14 October 2007

HelMUG, Wetter, programmieren und Die Katze auf dem heissen Stoffdach!

Vom einem zum andern

Das HelMUG-Treffen heute war ganz nett. Zwar sind wiedermal nicht allzu viele Leute dagewesen, aber wir konnten nett über die Befindlichkeit des Vereins reden. Im Grunde läuft alles ganz gut, kein Krach, kein Streit, keine Katastrophen. Aber eigentlich könnte etwas mehr laufen (nein nein, nicht von Krach & Katastrophen). Nur, wenn die Leute nicht wollen, dann wollen sie nicht. (Weiter geht's übers Wetter, Programmieren... und die Katze...)

Katze auf 2CV

Seit dem (zumindest von mir) lang erwartetem Regen ist es einiges kühler geworden, 17° ist ganz schön kalt. Bins halt nicht mehr gewohnt. Obwohl gestern war's noch wärmer, fand auch diese Katze hier. In unserer Strasse war ein fetter, roter 2CV [1] geparkt. Katzen lieben 2CV-Dächer, da ists schön weich zu liegen:

Nach dem HelMUG-Treffen [2] gings dafür zum Barba Yannis nach Exarcheia, was feines essen. Von da nach Hause, wo ich an Zwiki weitergewerkelt habe.

Dabei ist mir wiedermal aufgefallen, dass die bekannte 80-20 Regel auch bei Open Source Software gilt - und sie dort auch eine ganz banale Grundlage hat: Die Leute (laut Faustregel 20%) die den grössten Teil der Arbeit (laut Faustregel 80%) machen [3], bringen das ganz einfach deswegen zu Stande, weil man im programmieren "drin sein" muss. Wer viel Programmcode schreibt, dem fällt es leicht viel Programmcode zu schreiben, er/sie muss nicht jedes irgendwas nachschauen, jede Verzweigung nachvollziehen, weil alles "erst grad schon gesehen" noch frisch im Gedächtnis ist.

Mich selbst seh ich an der Grenze von der grossen Gruppe die wenig macht, aber nach oben strebend. Um da hin zu kommen nehme ich mir vor immer etwas mehr zu machen, irgendwann werd ich die kritische Masse schon erreichen :-)

[1]der 2CV wohnt hier in der Nähe und aufgrund der Parkplatz-Situation steht er mal hier mal da im Quartier.
[2]HelMUG ist die Griechische Mac User Group.
[3]Code schreiben, Bugs fixen, Dokumentation schreiben... also "Arbeit machen" nicht im Sinn von "Arbeit verursachen".

Posted by betabug at 20:39 | Comments (0) | Trackbacks (0)
30 October 2007

Quasi-Normal in Numbers

How much is that?

Zope programmers learn sooner or later that persistent objects don't like dictionaries and lists as attributes. Why? Because to keep their values around you have to assign them back to the object - and that will write a new revision compromising all of the object to the ZODB. Which wastes space and can lead to more ConflictErrors. But how much space? Yesterday evening I found one such case in Zwiki and in moving the dictionary in question to a BTree, I wrote down some numbers, in the process also rediscovering the ZCatalog in there...

Zwiki visitors can "rate" pages with 1-5 stars. The code in question stored the personal rating in a dictionary on the page. At first there's nothing bad about that, there are not so many ratings by so many people anyway. But the mentioned unpersistence on dictionary attributes led me to rewrite the code. The diff will soon be in the Zwiki darcs repository, so I'm not pasting it here. It's not the insteresting part. Let's look instead at the Data.fs growth, measured by a simple ls -l.

All numbers are in bytes. For reference I've rated pages from two browsers, one with a logged in user, on with a quasi-anonymous user with a user name set in a cookie. I've always double checked that simple page loads won't trigger stuff that grows the Data.fs, but otherwise things might not be very scientific.


For comparison, I've looked at some simple actions on a Zwiki page: Adding a comment and a visitor setting their name in the "options" page. Obviously the Data.fs growth of saving a comment is highly dependant of the size of the page and the comment.

comment add: ~ 26651
saving name in options: 970

As I was checking the numbers for the old code, quickly I discovered that some of the growth is due to the ZCatalog getting fed too. We can measure that effect.

old code

voting: 8755
voting from other account: 8568
voting on big page: 20264

old code without catalog reindex

voting: 5899
voting from other account: 5896
voting on big page: 17396
another vote on big page: 17414

Some of the growth clearly is coming from reindexing the object in the catalog. But the main observation here is that voting on a big page results in more bytes being written to Data.fs, since we are still writing down all of the object for each vote.

new code

voting: 2556
voting from other account: 2553
voting on big page: 2949
another vote on big page: 2528

Here I have rewritten the code to store ratings in an OOBTree. The migration of dictionary to BTree is not reflected in the numbers - that has to be done just once anyway, not for each rating being registered. We already see some reduction here, but the main observation is again on the difference of the "normal" page (more or less a default Zwiki FrontPage) to a slightly bigger page: There are in fact less bytes written when voting on the big page now, obviously we do not write all of the object to the ZODB any more. But then we still write a lot of bytes for such a small vote. Where does it all go?

new code without catalog reindex

voting: 169
voting from other account: 166
voting on big page: 169
another vote on big page: 166

Getting rid of reindexing for a moment and... we get very reasonable numbers all of a sudden. I can imagine a bit of overhead for writing objects to an OODB, so writing 169 bytes for a vote looks reasonable. You can even see that the 2nd accounts username is a bit shorter than the 1st. Lesson learned here: If you don't need to index your object in the ZCatalog, don't.

Reindexing all of the indexes in the ZCatalog is some overhead, but we do not change so much on the object. So why not just reindex only those indexes which we actually changed on our object?

new code reindexing only 2 indexes

voting: 2365
voting from other account: 2362
voting on big page: 2340
another vote on big page: 2337

The code for this looks basically like this:

catalog.catalog_object(object_that_changed, idxs=['rating', 'voteCount'])

We are getting down a bit again, but only slightly so. What might be the reason?

new code reindexing only 2 indexes, without metadata

voting: 608
voting from other account: 614
voting on big page: 591
another vote on big page: 514

Where we have used an optional parameter on the same line:

catalog.catalog_object(self, idxs=['rating', 'voteCount'], \

Here we told the catalog update to not update metadata. Obviously in practice this might not be a good choice for our code, as index and metadata diverge now, but it can show us something in the numbers: They are very reasonable now, we're writing a few hundred bytes, but we've got updated index and an updated object. Lesson learned here No. 1: If you don't need metadata in your ZCatalog, don't put it there.

Oh, and Lesson learned here No. 2: Even though the idxs parameter on the catalog_object() will update only the specified indexes, the call still will update all the metadata. There is a comment in the Zope code (in Products/ZCatalog/ on updateMetadata()) "Given an object and a uid, update the column data for the uid with the object data if the object has changed" which could be mistaken that only the changed metadata is updated, but indeed all the metadata seems to be rewritten (will need to grok that particular piece of code more).


For the moment we will go for the "new code reindexing only 2 indexes" version in Zwiki. BTrees are nice. Cataloging stuff is a tradeoff.

Back to our quasi-normal state of hacking, or as Saad reports about the strike in Paris:

Le trafic aérien à Air France devrait reprendre de façon quasi-normale...

Posted by betabug at 11:34 | Comments (0) | Trackbacks (0)
01 November 2007

Zwiki Bunker

Hack the planet... what else?

My spanish friends r0sk and Wu coined the term "bunker" for a couple of hackers getting together and working/teaching/learning together on some topics till late in the night. Had the first Zwiki bunker here now... still tired today. What happened? Yesterday evening tralala (George Tellalov) came over to my place so we could do some work on a new plugin for Zwiki. What we were trying to do is build a proper plugin, following the functionality of the TaggingPrototype from John Maxwell.

It's been a long time that I had the chance to do some pair programming. It was great. tralala is an extremely capable python programmer. I hope I was able to give him some pointers on the Zope and Zwiki related stuff (and some links into using darcs). Looks like I have some reading up to do in the python department :-) e.g. this unifying types and classes thing. I'm so behind the times.

Unfortunately the solution evaded us in the last moment. It was all set up nice and cool, but something got stuck in the Zope innards. The tagging stuff itself is no big deal, straightforward. But we were also aiming at providing a proper hook system for plugins to register snippets of html to display in the wiki interface. tralala found an elegant solution, it's just not working with Zope yet. Stay tuned!

Posted by betabug at 09:30 | Comments (2) | Trackbacks (0)
13 November 2007

Stupid apache adds Content-type to 304 Replies

Now that we fixed it in Zope...

Last weekend I upgraded my Zope to profit from the bugfix for the 304 responses should not have Content-Length header issue. Funny enough, my pages were still setting Content-type headers for those empty 304 responses. I was ready to blame Zope, until I noticed that my local test instance didn't do it. In fact Zope doesn't do this at all, but as soon as you place it behind apache, apache 1.3 will happily add the DefaultType content-type to empty replies. Don't believe me? It's easy to try it yourself. Also these tomcat people noticed the same thing - they seem to disagree on the reading of the RFC though.

So what do I do now with the Zwiki code that handles "If-modified-since" headers and 304 replies? Add the header back in, f* the RFC? What a mess. It wouldn't be so bad if some proxy servers and Safari didn't mess it up when they get a 304 with text/plain all of a sudden on a locally cached page.

Posted by betabug at 21:27 | Comments (1) | Trackbacks (0)
23 December 2007

Fix Functional Tests in Zope 2.10

Copy + Paste, my friend

A few weeks ago I introduced a new bug into the Zwiki code. A bug which made it past our tests. So I made up my mind to finally add some high level functional tests to our arsenal. Been there, done that, it's not so difficult. Then I hit the error TraversalError('No traversable adapter found', obj). It seems that the Zope 3 Page Template implementation in Zope 2.10 made it impossible to run the same functional tests on Zope 2.9 and 2.10. Which is kind of unfortunate if you happen to work on a Product that should run under multiple Zope versions. The solution is to copy+paste about 10 lines of code...

Searching the web I discovered a couple of hints at what I had to do:

In the end I had to combine philiKON's basic line, with the imports stolen from Łukasz, then I brewed it up into the form of Martin's setup (but changed to a more "normal" test setup).

Before I go through the solution in detail, I want to deliver a little rant to the Zope developers: I'm all for building modern stuff, modern programming paradigms, less boilerplate code, code reuse, etc. But (you knew there was a "but" coming) if the new stuff breaks existing code and the solution is just to copy and paste some boilerplate code that I don't even have to understand and that has not a snowball-in-hell chance to ever need to be changed / adjusted / customized, then... this new code stuff of yours just [choose your prefered expletive here] and should be considered broken.

The copy+pasted fix I'm applying here should probably be put into 2.10's ZopeTestCase itself, because then I wouldn't have the situation that my code works with only either 2.9 or 2.10 - I could leave my code alone and it would work on both Zopes (well, if I hadn't chosen to go with zope.testbrowser which isn't in 2.9's Five). But is this really the result of shiny new Interfaces, and IThis and IThat, that I have to copy+paste some blabla-code somewhere? End of rant.

So, let's go through the setup:

import unittest
import doctest

from Testing import ZopeTestCase

# ================== add these ========================
# imports copied from Lukasz mostly
    from zope import traversing, component, interface
except ImportError:
    print '--------------------------------------------'
    print 'Functional tests will only run in Zope 2.10+'
    print '--------------------------------------------'
from zope.traversing.adapters import DefaultTraversable
from zope.traversing.interfaces import ITraversable
from zope.component import provideAdapter
from zope import interface
from zope.interface import implements
# ================== back to normal setup =============

class TestZWikiFunctional(ZopeTestCase.FunctionalTestCase):
    Testing browser paths through ZWiki.
    # ================== add these ========================

    def beforeSetUp(self):
        super(ZopeTestCase.FunctionalTestCase, self).beforeSetUp()
        component.provideAdapter( \
            traversing.adapters.DefaultTraversable, \
    # ================== till here ========================

    def testSomething(self):
        # This is here, because otherwise beforeSetUp wouldn't be run
        # it's only necessary because all my tests are in the
        # FunctionalDocFileSuite - which is added afterwards

    # old-style functional tests would go here

def test_suite():
    suite = unittest.makeSuite(TestZWikiFunctional)
    # the following is only when you go with zope.testbrowser tests:
            'functional.txt', package='Products.ZWiki',
            optionflags=doctest.REPORT_ONLY_FIRST_FAILURE |
                        doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS))
    return suite

if __name__ == "__main__":

So, if I count this right, I have to add about 10 lines of boilerplate, never to be changed code to get Functional tests running on Zope 2.10 again. This seems to be only if your Functional tests depend on Page Templates - but of course which Functional tests wouldn't depend on Page Templates?

The end result (the complete tests with the files functional.txt and will hopefully soon turn up in ZWiki source code, for the moment these links point to my private repo.

As a final note I want to say that I really enjoyed working with zope.testbrowser. It's fun to write tests like that, even though mine aren't perfect yet.

Posted by betabug at 13:01 | Comments (0) | Trackbacks (0)
16 April 2008

Missed Episodes: ZopeEditManager

Nice for ZWiki editing

The few times in many years that I tried out the ExternalEditor product I a.) never got it to work on Mac OS X and then b.) had already moved away from Through-The-Web (TTW) editing and therefore felt no more need to follow up on it. Today, following a problem someone had on #zwiki, I looked at ExternalEditor again, only to see that I had "missed episodes" of the show (as a Greek expression goes), since there is now a nice GUI application to handle the client side on Mac OS X: ZopeEditManager.

Since I'm not going to move back to editing Zope code in the ZMI, instead preferring to write my code on the file system, what could be the use for this thing for me now? Simple: ZWiki integrates very well with ExternalEditor, so having this thing installed, would allow me to edit wiki pages in a proper text editor. In fact, it works really well, took only a minute to configure for FireFox (after reading the Readme.html).

Next step would of course be to go back to a unixish setup, since my editor of choice is vi now... :-)

Posted by betabug at 15:23 | Comments (5) | Trackbacks (0)
03 May 2008

Like Riding a Build Cycle... You Never Forget

Picked up on Zwiki again a bit

After some months of above average slacking, I picked up on Zwiki hacking again a bit the last few days. I'm enjoying it, but it took some time to get started again. Where is the stuff again? How did we do this, that, the other thing? We also changed some things in the way we worked, with checkin messages now going to GeneralDiscussion (unfortunately they have to be cleaned up manually).

Speaking of such stuff, I really enjoyed this article called Software Builders by Diomidis Spinellis, quote:

"As much as I nostalgically remember the days when I could cook and eat dinner while compiling an application, a quick build cycle can keep developers focused..."

Wonderful! Back to Zwiki, we're (actually mostly Simon is - as mentioned I've been slacking a lot) moving the code base to Unicode for the data storage. Even though it's complicated, I've got a good feeling about it. We also have a Roadmap 2008 now. We've got an -unstable branch, which currently is at the point where it powers without much trouble. At first the Roadmap said that Plone support is to be terminated, but lately it's back, on minimal burner (no, I don't care, as long as I don't have to look at that stuff).

What I really would like next, is to test for unicode/utf8 transitions in our functional tests (built upon zope.testbrowser), but so far I haven't found out how I could do that. I'm talking about going through a cycle of page creation / edit / display page / search page with mixed Greek and German text and properly checking the displayed content at each stage. Problem is, it's all in a text file and my first tries resulted in UnicodeErrors even for stuff that worked in a browser. Of course I'm doing something wrong there.

Posted by betabug at 14:42 | Comments (1) | Trackbacks (0)
14 May 2008

Got darcs 2 For PowerPC

These machines are still valuable...

The new ZWiki -unstable repository is in darcs 2 format. This resolves a couple of problems sm had with bringing the stable and unstable branches together. On my MacBook it was pretty easy to get a new binary package for darcs 2. But new OS X packages are done for Intel Macs only. At work my dev machine is a trusty old dual-G4 (fast enough for most anything still). Darcs is written in Haskell, so you need the GHC compiler to build it. First I tried to build ghc from macports... after about 4 hours it failed with a compiler error. I gave up for the moment.

Now I had the brilliant (or rather "obvious") idea to get a binary package for GHC. Indeed there is a binary installer for GHC on Mac OS X PowerPC too! Installing that was a snap, and afterwards building darcs 2 with it was a simple ./configure && make && make install game. The result is that I can hack on Zwiki at work again now.

Posted by betabug at 13:55 | Comments (0) | Trackbacks (0)
26 May 2008

How to add a Zwiki programmatically

Up to your own choices

Zwiki of course has a method manage_addWiki to add a wiki from a script. But this will add the "basic" standard wiki to your Zope ZODB. We had a plan here where we would add a special wiki for each user account... wasn't done (yet), but out of it came a simple code snippet that shows how to do it...

This is a method in one of our "container" products. It's not part of the class, as the plan was to run it from manage_afterAdd, but of course you could change it a bit and run it from the class too (in which case you'd probably change "add_point" to "self").

def add_user_wiki(add_point):
    # we're using BTreeFolder2
    # could be that someone wants a real big wiki
    # but also it's showing another difference to manage_addWiki
        'notebook', 'My Notebook Wiki')
    nb = add_point.notebook
    # add a wiki page with minimal welcome text
    # set properties
    # set permissions
    nb.manage_permission('Zwiki: Add pages',
        ('staff','account','user'), acquire=1)
    nb.manage_permission('Zwiki: Add comments',
        ('staff','account','user'), acquire=1)
    nb.manage_permission('Zwiki: Edit pages',
        ('staff','account','user'), acquire=1)
    nb.manage_permission('Zwiki: Delete pages',
        ('staff','account','user'), acquire=1)
    nb.manage_permission('Delete objects',
        ('staff','account','user'), acquire=1)
    nb.manage_permission('Copy or Move',
        ('staff','account','user'), acquire=1)
    # shortcut to create catalog, create index_html:
    hello_text = """Hello!

This is your **notebook**, where you can leave notes,
instructions, thoughts,... anything for you and your
friends to work together!

You can edit the text of this page and add new pages
as you please."""
    nb.Hello.edit(text=hello_text, log='initial content', type='rst')

The result is a properly set up wiki, but with just this one page and the permissions set to allow actions for a certain set of roles only. It wouldn't make sense to have a method like this in the Zwiki code, as the code is mainly setup and choices of configuration - everybody would likely want to run something different here.

Posted by betabug at 11:26 | Comments (0) | Trackbacks (0)
25 October 2008

Zwiki 0.61 Released - Cleanup and Catchup

Moving ahead

Woke up this lazy Weekend noon to find Simon has released 0.61 of the stable branch of Zwiki. This is not some bombastic feature release, rather a version that's catching up and cleaning up some stuff. I'm happy that Simon did this, since I've been slacking too much on this project, maybe it will get me going a bit again now.

Posted by betabug at 12:01 | Comments (0) | Trackbacks (0)
22 January 2010

Want: Memory Optimization for ZWiki

Back to ZWiki hacking: An evening at the code
Mist on the edge of the Parnasos mountains in Greece

Yesterday evening for the first time in a looong time I got busy with ZWiki source code. I had checked out the latest darcs code and was basically looking to improve something in a general way, not just fix bugs. One such point is memory use optimization and a field I don't know that much about yet. A chance to learn combined with a chance to do a good deed... here I come!

The concept is basically to find cases where we store (large) data in basic attributes on one of our objects. Zope loads these "secondary objects" into memory, as soon as it loads the main object. Once you "move them out" into objects that are derived from "persistent", Zope will load them only when they are directly accessed. kosh (of #zope fame) explained the concept famously some time ago.

My first bet was the actual page content: It can be big and it will only be really needed when we actually display the page, not when we access the page object to get information e.g. for the contents page or for showing the page title in some page list. Unfortunately the content is "hidden" in the DTMLmethod object ZWikiPage is based on, so I can't easily mess with it. But there is another candidate: The "prerender cache" we use to store the "halfbaked" html to get page content rendered faster.

So, yesterday I managed in a few lines to create a simple object to store that cache (basically a big string) and attach it to the ZWiki page object whenever needed. It all works and passes all the tests, but I have yet to prove that it will save memory use in the long run and on large wikis. The current very experimental code is in my ZWiki darks repository - not meant for production use in any way! (... and of course if you find this post a few months from now, it might well not be in there any more.)

Posted by betabug at 09:39 | Comments (0) | Trackbacks (0)
12 April 2011

I'm a Freelancer now/soon

Python, Zope, Pyramid, whatever comes along

After 6 years at the Graphics Garage, my big projects there seems to be done. I've written three big systems there: Two of them to administrate every bit of the company, including organizing all the communication with the clients into an Extranet solution. It's time to move on... I'll start working on my own as a freelancer now.

I have on and off worked with Python and Zope since 2002. I've also worked with other web based systems of all kinds (and some not-web-based ones), so I guess I'll stay in the web based programming area for now. I've worked on big and small projects, in teams and alone. Some of my stuff is open source and can be admired from my pro page (with links to my resume too), some is proprietary client stuff and will stay hidden in their code vaults forever.

With all that Zope experience, I'm obviously there for any ol' Zope site in need of an overhaul or extension. For the "new" stuff that's not Zope, right now I've started a little fun project with Pyramid and I've got some Django and Ruby scheduled too. Whatever comes along though, if the tool does the job, it's fine for me.

For starters I continue supporting the projects of the Garage of course, putting my projects there into "maintenance mode". I also have the first project coming along from a customer in Switzerland. I've got some leads from France, so I might be having fun soon as an "international enterprise".

I don't want to hurry it in the beginning (starting too many things at once is as bad as starting nothing), but I've still got some capacity free. So, service announcement: If you need some good programming done, drop me a line!

Posted by betabug at 10:09 | Comments (1) | Trackbacks (0)
07 November 2011

Welcome to Betabug Sirius

My new Company Site

It has been quite some time that I announced that I'd be working as a freelancer. Lots of stuff had to be done in that time, but finally things are ready. I've founded my own little company and set up a small website: Welcome to Betabug Sirius!

For once this isn't a "ZWiki-As-A-CMS" site (which is what I usually do when I want a site to go up fast), but a plain, static html site, done in vi. I know it won't scale when I'll want to expand it, but I had fun coding it up in the old style.

So far I already have a few customers, working with Zope (mostly in bugfixing and maintenance of existing / legacy sites) and with Pyramid (building a brand new web application). There is also a project to build something unique and "our own" on a longer horizon, involving technology and art.

There has been and still is a lot of bureaucracy, but so far the ride has been smooth and sometimes even fun. Part of the strategy is to work together with other companies to form flexible teams for each project. That's something that has worked real well so far, giving me fun and inspiration to work with others.

Posted by betabug at 10:26 | Comments (7) | Trackbacks (0)
Prev  1   [2]