betabug... Sascha Welter

home english | home deutsch | Site Map | Sascha | Kontakt | Pro | Weblog | Wiki

04 July 2007

RAM Cache a Method from a Python Product

So you want to keep it for a while?

Say this method of yours takes some time to run. You've learned to use a RAM Cache Manager to cache DTML methods in the ZMI, but now you have made your own filesystem based Python product, and a method of that should get the speed boost. It's nearly as easy as clicking it together with a "DTML Method" in the ZMI, so lets have a look at how to cache-enable a python method in a Zope product...

Our setup: Just this one method

We made our own Python product, which we based on CatalogPathAware to make it interact with Zope's various caches. There we have this one method, lets call it "bigman", where all the heavy lifting is happening, maybe it looks something like this:

def bigman(self, someparameter):
    Do some heavy lifting.
    for item in self.objectValues():
        # ... and so on ...
Appart from this method we also have set up a "RAM Cache Manager" in the ZMI, so it can be "seen" from the instance of our final product.

The principle: Move the method behind a strawman

To make our "bigman" method cached and control when to return cached data, when to run the method and cache the data, we need a new method that controls the behaviour. To get there, we first rename our "bigman" method and declare it private:

def _bigman(self, someparameter):

Now we write a new method to control access to our renamed "_bigman":

def bigman(self, someparameter):
    Get bigman data either from cache or from the real method.
    view_name = 'bigman' # choose your own here!
    keywords = {'someparameter':someparameter} # see note
    cached_data = self.ZCacheable_get(view_name=view_name,
                            keywords=keywords, default=None)
    if cached_data is not None:
        return cached_data
        # no data in the cache, so cache it
        data_to_cache = self._bigman(someparameter)
                    keywords=keywords, view_name=view_name)
        return data_to_cache

Control Flow and "keywords"

Some things are happening here: First of all we set up a "view". This is some kind of identifier that tells the RAM Cache Manager which method is being fingerprinted. The term "view" has had a bit of an inflation lately, a "view" here has nothing to do with "Five" views, Z3 views, or any other views you may or may not know in Zope. Just think of it as "identifier". The Cache Manager uses it to keep cached data appart.

The next line declares "keywords", which is another way to keep the cached data appart. Where "view" was used to keep the methods appart, the keywords can be used to keep parameters to methods appart. Methods obviously should return different data when parameters change - even when cached. Without changing keywords, the Cache Manager will keep just 1 copy of our methods results. If "bigman" normally returns different results depending on "someparameter" being "blue", "red", "green", or "yellow", then with "keywords" set up like this, the Cache Manager will keep 4 results. It's very similar to the "Names from the DTML namespace to use as cache keys" field in the ZMI "Cache" tab of DTML methods. If you don't need varying parameters, my guess is you can leave the dictionary empty or use something like keywords = {'a':'b'}

From there on we call the ZCacheable_get method (we inherited that through CatalogPathAware) with our parameters. If no data is in the cache for our combination of "view" and "keywords", then we get back None. In that case we want to run our "expensive" method and feed the output to the cache (and to the user). The timing setup, when to throw away "old" cache entries, how long to keep things, how many entries to keep, all that is set up in the RAM Cache Manager.

All set up... but nothing in the cache!?

At this point we're almost done: Our setup works, but it never gets anything into or from the cache. The reason is that we still have to associate our products instance with our RAM Cache Manager in the ZMI. Just "associate" your instance in the ZMI, like you have learned to do with DTML Methods. this will enable all cacheable methods of a product, and it's the "views" that lets the cache keep them apart.

Invalidating the cache with a "view"

One last thing remains: It might happen that we know that our data in the cache has become stale, for example if we process some input that changes the underlying data. In that case we will invalidate the cache with something like this:


Posted by betabug at 13:47 | Comments (0) | Trackbacks (0)
ch athens
Life in Athens (Greece) for a foreigner from the other side of the mountains. And with an interest in digital life and the feeling of change in a big city. Multilingual English - German - Greek.
Main blog page
Recent Entries
Best of
Some of the most sought after posts, judging from access logs and search engine queries.

Apple & Macintosh:
Security & Privacy:
Misc technical:
Athens for tourists and visitors:
Life in general:
<< Relaxing Vegetable Soup | Main | Spammers, Scrapers, Browserstats >>
There are no comments.
You can trackback to:
There are no trackbacks.