feedparser media:title problems

For some days, some of my "other weblogs" feeds had displayed something funny in some of the entries. For example the titles of Adamo's posts were shown just as "adamo". Turns out there is a little bug in the python feedparser module... it's already a submitted issue, that the media:title tag is misparsed as title. For me this happened with two feeds from Wordpress blogs. The workaround was to use the atom feed instead of the rss feed, as described on the MiniPlanet wiki page.

Update: There's even an unofficial patch for the issue. I haven't tested this one yet.

Old Code

Just now I went to fix a little bug I had in one of the oldest pieces of code I wrote in this company. It's like seeing a family member I haven't visited for a long time. It made me think of "back then" when I started here. How long ago that was, how nervous, unsure of myself I was, but also how happy I am here. The code isn't really that great, I was out of touch with hacking for some time back then.

But then, that piece of code does a good job if you take the crufty bits aside, there is some elegance in there. I will give it a cleanup soon, and it will be with loving care, none of that "I hate this old code I wrote, lets just rip it all out and write from scratch". The idea of that code has proven to be very successful, let's go for something of a family reunion.

Oh yes, that bug. It's gone. Was gone in no time actually, I still know my way around in that code, and it was kind of obvious were the bug would be sitting.

Cables cut: OTEnet (and HelMUG) Offline

Since this morning 11:00 the HelMUG server had been unreachable. Net traffic in all of Greece was sucking real bad, at times I had packet loss of 90% in getting abroad. Apparently optical cables were cut in two places at once: in Κάστρο Βοιωτίας (Kastro Viotias) and Χιλιομόδι Κορινθίας (Hiliomodi, Corinth). Don't ask me where that is. See the official announcement from OTEnet (in Greek). I don't know how much of OTEnet is out, but the OTEnet hosting in Thessaloniki is definitely not reachable.

Update: about at 17:00 connectivity was back to the HelMUG server.

Working Away

The first day of "working but not being in the office" turned out to be productive. Setting myself up took some time at first. I also found out, that our office with a proper office chair and desk is much more ergonomic than a corner of the gf's desk or a spot on the sofa. That was counteracted by a much more than average work motivation :-) It's cool to be able to be here and work!

At first I had to bring my laptop to work level again. A secure tunnel to my company mail, ssh-agent for company cvs access, bringing my local repo up-to-date, installing some packages I'd been missing, configuring my local testing setup (where the biggest problem was that I didn't have access to my bookmarks from work). It seems that it's been some while since I've been hacking company stuff from the macbook. A VPN would have been useful, if I get to do more "away working", I'll look into setting one up.

As for the actual work, for the moment I'm mostly cleaning up older code. Now that we've moved to Zope 2.9 and therefore Python 2.4, I'm replacing popen calls with the new subprocess module's Popen class. That stuff is much more flexible and it allows me to keep STDERR output out of the console. I'm also examining some of my older datastructures for stuff like I'd described in Quasi-Normal in Numbers - where an approach using BTrees saves a lot of memory and disk writes. It seems I'm getting more and more strict in these matters.

Cereal-Killer!

It's not good to talk to a dead server. But when your server is only almost dead, it might be possible to talk to it through a serial console. Apple's G4 computers never had serial ports built in, but there are products that let you exchange the modem with a "real" serial port. Wildweasel (of #bsdcow fame) got me one, and he also soldered me a suitable cable with his own bare hands (thanks wiwi!). Yesterday I finally set out to install it on my server. It was quite an adventure, but my server is still running (hey, you're reading this on it)...

We arrived in the server room, I switched off the server, took it out of the rack, opened it and inserted the serial card. Wait. I tried to insert the serial card. It wouldn't fit, for something like a few millimeters. Tried this way, tried that way, looked at it from this side, looked at it from the other side. No go.

Since there was no way, I just went over to the club room, ready to eat some pizza and then go back home. When I told the members of the IN-Ulm club there about the card, one of them (raimund) spoke up and after checking the card declared that it should be possible to cut a few mm of plastic and board. I pulled out my "cybertool" swiss army knife and he started to cut and file.

Now, if this sounds cheesy, dangerous, or crazy to you, I must tell you... you are right. Kids, don't try this at home. Or at work. Some people are crazy enough it seems. Since cutting on the "cereal" board wasn't enough, he cut from the power connector (the one that was in the way) too. That was when it got really scary. In the end the board went in, a little bit wacky and I wouldn't want to shake the box too much. But it works now and hopefully will for a while. Many thanks to raimund, Taxman and the other IN-Ulm guys! Thanks to the_eye for the pictures, there are some more pictures here.

Why did the board not get in cleanly? Apparently (from the leaflet) it was meant for older versions of the G4. Apple must have moved that silly connector around a few mm, and the guy who designed the board made some decisions of placing stuff that he wouldn't have done in hindsight.

I probably would not have gone with this approach, but I had traveled from Athens, over Munich, to Ulm to put in this board (well, not exclusively). Having to go home without the card in would have been ugly. Lucky me it worked.

Είναι έτοιμη η Ελλάδα για το Internet;

<rant>Χτες και σήμερα κοίταξα λίγο τα moderation requests των mailing lists του HelMUG (Hellenic Mac User Group). Οι λίστες αυτές δεν είναι moderated, μόνο αν το λογισμικό υποθέτει να είναι spam το μήνιμα μας το κρατάει. Πολλά από τα συνηθισμένα λοιπόν, χαπάκια κτλ. - αλλά και μερικά που αλλού έχουν γίνει πολύ σπάνια πια: Γιατί πιστεύουν τόσοι μικρή επιχειρηματίες ότι μπορούν να στείλουν "διαφημιστικά email" (δλδ. spam) οπού θέλουν; "Προσφορές" για έντυπα, σεμινάρια, web sites, κτλ. σε έναν σύλλογο; Θα μπορούσα να πω ότι ηλίθιοι και άνθρωποι χωρίς ηθική υπάρχουν παντού - και μάλλον θα το πω αυτό για να κρατηθώ...

Εννοείται ότι δεν βγάζεις άκρη να γράφεις στο abuse@ των providers. Π.Χ. η μαλ... "globalgreece.gr" που εδώ και πολύ καιρό στέλνει spam στο ΔΣ του συλλόγου μέσον OTEnet: Προσπαθήσαμε μερικές φορές να τους γράφουμε ότι δεν μας απασχολούν αυτά που γράφουν, τίποτα. Έγραψα τελικά στο abuse@otenet.gr:

This customer of yours has been and is repeatedly spamming our mailing lists. We have repeatedly told them to remove our addresses.

Please inform your customers about your Acceptable Use Policy.

Απάντηση:

Thank you for bringing this matter to our attention. We inform you that we have tracked down the compromised machine and taken all the necessary actions.

Τι λές "compromised machine"; Μήπως θέλατε να πείτε "compromised minds";

Ωραίο αυτό με τα spammers, η επόμενη μ... είναι τα chain mails. Μου ήρθε σήμερα ένα από τα τύπου "GET RICH QUICK" mails, από κάποιον "Evagelos Barlas" (δεν έχω ιδέα ποιος είναι, μέσων Χελμούγκ πρέπει να βρήκε το mail μου). Ο κύριος αυτός το έκανε forward σε ~ 100 διευθύνσεις - όλα αυτά μέσα στο mail εννοείται. Όσοι του το είχαν κάνει forward αυτουνού επίσης είχαν όλο το address book μέσα στο mail. Περίπου 20 επίπεδα forward. Μεταξύ άλλων από ανθρώπους που έχουν τον τίτλο "Υπεύθυνη Διαφήμισης" σαν τίτλος στην signature τους.

Όχι ότι είναι και τίποτα το εξαιρετικό. Μερικές φορές μου έρχεται να βγάλω όλα αυτά τα ονόματα (τον εταιρειών και τον chain mail ηλίθιων) στην φόρα. Δεν νομίζω ότι θα αλλάζει κάτι. Θα ήτανε ωραία να μπορούσα να τους πω: Παιδιά αφήστε αυτό με το Ιντερνέτη λίγο ακόμα, πηγαίνετε να παίξτε λίγο ακόμα tetris στα PCιά σας.

The Software Equivalent of Duct Tape + Swiss Army Knife

A few days ago, while waiting for the Metro to get me and some friends to a party, I was thinking about some of the crazy stuff I have done with crappy stuff in the past. I'm not a hardware guy, so I have to do the equivalent of building a rocket launcher with duct tape and a Swiss army knife in software. Here are two things that immediately come to my mind:

First was that crazy sales and contact management database I had built way back then, with FileMaker 3 (then moved to 4). Not crazy? No, so far that's just lame, as lame as FileMaker. But the crazy part was that this was a distributed, replicated database. Keeping the distributed databases from multiple locations in sync over email. It was possible for people to work on it offline on their laptops. The result was slightly crappy in hindsight, but it worked.

Next was of course the first SMS chat on TV (as far as I know ever), which was built in the time frame of 2 weeks. When the people who should originally do it gave up (without even trying), I stepped up and said: "I can do that." Well, I could, but all I had was a handful of crap hardware that nobody else wanted. Don't believe me? Would you say that a couple of Macintosh Performa 630 boxes with Mac System 7.6 are indeed crappy? One of them with 32MB RAM, the other two with 8MB. They were even crappy back in 2001 when this was taking place. Ups. I did this hack in AppleScript and using Claris Emailer. It worked quite well I'd say, even though the Claris Emailer mail database needed emptying quite often :-)

So, what are your software duct tape stories?

Shut Up and Listen to the Music

My work environment has many acoustic disturbances: phones ringing, people talking, a radio station shuffling through the same 100 songs, sometimes even machines rattling. To listen to music I'm using good old "over the ears" headphones. They passively cancel out some noise, I had considered to get active noise canceling, but from what I hear, they won't do much good with things like the radio whining on. So instead I listened to my friend saad and got Shure e2c in-ear headphones. Haven't tried them at work yet, but they worked wonders on the plane back from Munich...

The problem with plane's in general is that they are noisy in itself, they are filled with noisy people, and there is a law that every plane gets issued at least one crying infant. Oh, and you are forbidden to complain about the infant, else the "mothers of the earth united league" will threaten to kill you. The problem with this particular flight was that it was taking off at 7 AM, which means I got up at 3 AM. I believe this flight time is proof that Lufthansa is very lax on drug testing their employees. I was in the land between being awake and asleep during all of the flight.

I'm not going to complain any more about this flight (or infants) though. I slumbered almost through all the flight. I inserted the headphones the moment electronic devices were allowed after take off and took them out when I had to stop the iPod before landing. I listened to 2 1/2 albums and really heard the music. In the process I even heard new stuff in the music, details that I hadn't heard before, even though I know some of the stuff I listened to since a long time. I was able to hear the music at low volume too, which is what allowed me to slumber in the first place.

I hear that the problem with in-ear headphones is finding the right plugs for your ears. The e2c box contained 3 kinds of plugs, in 3 sizes each. I found some that seem to work for me (and it took me some time to figure out how the earphones go into the ear, I mean wouldn't marking "right" and "left" be kind of have been obvious?) Whatever, I have my system now.

In case you try out these headphones now and they suxx for you, please complain to saad - he's the guy that made me do this! In fact it's strange how lately I'm doing what people tell me to do... I got a camera so wildweasel stops whining, saad tells me to get earphones so I go out and buy some... Also Shure doesn't sell the E2C model any more, they've been replaced with something more expensive, I've got mine from Amazon where they still had some old stock apparently.

There Should Be a Rootkit Detector On It Too...

Somehow I think this combination of "matohandra" (ματόχαντρα) and USB memory stick on a detachable part of my keychain is extremely funny. If you didn't know, a "matohandra" is a charm that helps against the evil eye. Actually there should be a rookit detector on the USB memory stick too :-) The new and the old.

Of course if I had such bad taste as to use Windoze, I could put an Antivirus package on the USB stick, that would fit in with the "metaphysical" aspect too.

How Not To Buy a Camera - and get away with it!

In case you are planning to buy a digital camera, the prudent way is to check on the web what's available, find out which cameras have positive reviews, and then decide which ones offer the best features and quality for the best price. Don't just buy a cheap camera on the spur of the moment... except that's exactly what I did. Lucky bastard that I am, I got a good value for my money, but for a moment I was scared clean and nice. Read on for my review of this camera...

I had been at a department store back in Switzerland (never buy cameras at department stores!) and while waiting for the gf, I strolled through the electronics corner. Usually I just look at the computers, but since wildweasel has burned a hole in my ears whining about my lousy phone camera pix, I had a look at the digicams. Shouldn't have done that, because I saw a camera at a reduced price, at the bottom of one of the locked glass vitrines, down from ~ 100 Euro to ~ 60 Euro. As my friend saad always says, I'm the master of cheap jokes, so this hit my cheapness detonator switch.

Once the gf came back, I showed her my find and we got a sales droid to open the lock and hand me a "Samsung Digimax Cyber 530" - whatever that is. (This camera is sold also under a couple of other names, like Kenox S500, Digimax S500.) To tell you the truth, I have heard the name Samsung before, and I've read a general good review about Samsung stuff on gizmodo or so, that's about as much as I know. The sales droid tried to scare me that this camera takes two AA cells, and tried to tempt me with a Sony camera that cost ~ 120 Euro - only a little more. Having to buy batteries wasn't to my liking, but the poor guy couldn't guess that I have a strong dislike of Sony stuff (because whatever Sony I have owned in my life has died of buttons not working any more). I got the Samsung.

Auto sensitivity is low sensitivity

Next thing I'm back at my fathers place, playing around with the camera. I'm indoors, in a room with a few windows, but every picture I take is violently shaken with smears due to long exposure and me shaking the camera. I felt like Mr. Stupid. To give you some background information: I used to be a photographer. As such I never liked the light emitted from the built in flashes in cameras. They are the cause of very sucky pictures. I'm also willing and able to take long exposure shots holding a camera steady in my hands. I've been known to hold a steady camera through 1 second exposures. So I was mighty confused.

The solution was to read through the flimsy little manual. Apparently, in automatic mode the camera sets the sensitivity of the chip very low, which is fine in bright light and when using the ugly built in flash. What I had to do is keep the camera on "program" mode, and manually set the sensitivity to 400 ISO. That gives acceptable times even in indoor situations for me (which means exposures like 1/3 second - not acceptable for most people I guess). The good thing is that with "program" mode, the camera even remembers that I have set the flash to "off".

Image quality: fine

As most of the reviews I had seen state, the image quality is good. In my totally unscientific tests, I see nice sharp pictures, even though I don't even run the camera on the highest resolution (which is 5.1 MegaPixel, but I want my pix for this blog, where stuff gets massively resized anyway). I don't really see a lot of optical aberration (neither with the zoom on wide angle nor on tele). Of course I'd have to take real measurements to get these really examined, but if the lens isn't aberrating to the point that I see it right away, I'm not scared.

The colors look good too, but I've never been very good at color correction anyway. In my totally unscientific experiments I did not have to correct the colors to get acceptable shots. In fact when I attempted to correct colors (e.g. for a blue hint of the sky on a white ice rink), I couldn't find a setting that improved much. I found out how to correct the white balance too (again only in "program" mode), using the on screen menu. Not very often needed, but helped a bit with taking indoor lamplight shots in front of window light.

Handling / User interface

The camera's handling is simple. No surprises and most things work as expected. That's an important thing, because the manual suxx, and if I had to rely only on the little information given in the manual, I'd be lost. (Yes, I'm the kind of guy who read the manual, and indeed have taken useful information out of this one too.) But most things are just where I'd expect them to be, and work just how I'd expect them to work. There is cool stuff like thumbnails of the stored pictures and a zoom view that shows you "where you are" - I don't know how standard they are for cameras like these.

Some of the on screen stuff is never explained in the manual. Most of it I can figure out. There are some "advanced" flash modes (flash-A, flash-S icons), which I'll have to experiment to find out what they do. As mentioned, I'm not much of a built in flash user. In the meantime I've downloaded a real manual off the Web. The flash modes are nothing fancy, but I'm unlikely to use the flash much anyway.

To connect to my Mac I used the USB cable that came with the camera. It has a special connector on the camera side, so I'll have to take care never to lose it. The camera shows up as a removable drive on the Mac then, with access to pictures and movies through the Finder. All I need. There's a CD with Windows software, which I guess makes a nice coaster for your teapot. The camera takes SD memory cards, I got a cheap 1Gig with space for a lot of pix.

Fast is nice: Something that always was annoying with my phone's camera was the long time it took switching from picture taking mode to viewing mode. If there were a couple hundred photos in there, it could sometimes take almost a minute. On the new camera it's instantaneous. I've set the time that a pic will be auto-displayed after snapping it to minimum and just fast switch to the "playback" mode when necessary.

Testing a new moblog setup for COREBlog

I'm just testing a new moblog setup for my weblog. "moblog" means "mobile blogging", which means I can post by mail or with my phone. As this is just a test, the real content will come later likely...

Δεν έχεις head set; Καινούρια πατέντα!

Δουλεύεις στα ορυχεία του κώδικα και σε παίρνουν τηλέφωνο; Πρέπει να μιλάς και να έχεις τα χέρια στο πληκτρολόγιο; Σήμερα ο μπεταμπαγκτσής βρήκε καινούρια πατέντα για να μιλάς ελεύθερα και χωρίς χεντ σετ... use a standard telephone and big phat earphones!

(You can provide "on hold" muzak too this way :-)

A Printer To Plug In

This evening I bought a printer. I believe it's the third printer I bought in all my life. I'm not the kind of guy who prints out a lot of stuff, I value my trees. But lately I thought very much about reviving an old project of mine, which will involve going through some texts, making revisions. A job that is done much better on paper than on screen. When I saw how cheap laser printers have become, I nearly fell off my chair. 100 Euros for a laser printer? wtf?

Well, what can I say now? I've got a brand new Brother HL-2030 at home. It has a USB connection (I'm used to networked printers, but this is fine for my single computer). It prints quite nice, has space for 250 pages, doesn't make a lot of noise. It also gets disconnected from the electricity plug when not in use, no standby power for me please.

FF nearly fell over when he watched me switch it on for the very first time, then connect the MacBook, and without installing anything, just do the first printout. The Mac picked up the printer model right away and apparently the printer description came with the OS. I had to change the default paper size to be A4, that's all. FF's laptop has "Vista", so there would be a special CD for him, together with an extra booklet (apart from the normal Windows setup booklet). To imagine that when he shopped for a computer they told him not to buy a Mac, because it would be difficult to find a matching printer.

Comments for IE6 disabled

The last few days I had a huge spamrun targetting my weblog. The attack is pretty stupid, no comments get past coreblog's defenses, but they generate a lot of traffic and confusion in the logs/stats. The botnets the spammers are using seems to be "fresh", so it's mostly not yet in the HoneyPotBL. I had started to block the IP addresses whenever I spotted them, but playing "whack a mole" with spammers gets boring real fast.

Stupid enough, the spammers are abusing the user agent string of IE6. I can't just block that, as still many people use this. But I can block this particular browser from doing POST requests on my server, returning a 403 error code. The requests are still there, but they eat up a lot less resources.

For how it's done on the technical side, read on...

Basically what I do is to change some settings in apache's httpd.conf:

    SetEnvIfNoCase User-Agent "^Mo...(user-agent-string-here)$" block_bad_bots_post

    ... snip ....

    <VirtualHost _default_:80>
    ... snip ....
    <Location "/">
    Order Allow,Deny
    Allow from all
    Deny from env=block_bad_bots
    <Limit POST>
    Deny from env=block_bad_bots_post
    </Limit>
    </Location>

This allows IE6 users to browse the site (GET requests are succeeding), while they can't POST to forms. They get a 403 status code, with a helpful text (for those cases when a real human being attempts to post a comment). I have another Deny line, for all those user agent strings I totally disallow.

It's a mean world out there.

Google Mailservers used for Dictionary Attack

The last few days I noticed a nice, big dictionary attack targeting my lowly mail server. It's not been the first time and I'm not the only target of this, looks like Tor has seen it too. What is interesting about the attack is that it's abusing a lot of google servers...

Here are some sending mail servers spotted in my logs::

relay=wx-out-0506.google.com [66.249.82.227]
relay=fg-out-1718.google.com [72.14.220.158]
relay=wf-out-1314.google.com [209.85.200.174]
relay=py-out-1112.google.com [64.233.166.177]
...

Of course there aren't only google servers abused in the attack, but there are so many of them, that I spotted them immediately in the logs.

Makes me wonder if nobody at super-hightech google noticed this. Searching around on the web I didn't find anything regarding the attacks at first, until I noticed in a comment in this (only slightly related) post about google mailservers being blacklisted at the New York Times that apparently the captcha test for creating gmail accounts has been broken. According to that comment, it might well be that those accounts might be put to use now by the spammers.

The dictionary attack didn't "get through" on my server of course - I didn't notice a single mail being delivered from it. But that is not such a big relief, the ugly things with dictionary attacks is that even if they don't get through, we loose. Every mail that doesn't get through allows them to strike one match off their space of possible addresses. They are also abusing our server resources, but with the mail server set up to deny mails right at the SMTP prompt, without generating bounces, that's not hurting so much.

Notes About CSS Editing

I'm doing a lot of CSS work lately. Even though I believe CSS is vastly superior to table based layout and font tags and center tags (and all that garbage people used to use), I'm hating it. The problem with CSS is that it's much too clever and meant to be much too clever. There is just no way a machine seems to be able to output meaningful CSS... so in the end it's all handcrafted. And you have to think about it. Tired of it all (I've been that for a while now), yesterday I looked at some of the options for making my life easier and thought about some of the things I could have done better...

Bad news first: There really is no CSS editor. No matter what people tell you. None of the products does much more than you can do with a simple text editor. Sure, you won't have to look up how to make something bold (font-weight: bold;), but you pay for that with endless forms of options for every aspect of styling. Clicking it all together doesn't really help.

What we would really need is a tool to help with the planning stage of CSS design: Something to help you rough out the code structure, to keep track of what elements exist and all that.

At the current project, I've made a mess of it all. I'm paying for it now, with every edit going through a minefield, likely changing something on another page I've never suspected. Also searching like crazy for every edit where I do have to make changes and which parts of the CSS file affect my current element. I've looked at some of the tools. I liked cssedit, which tells me almost everything I want to know about the current element.

But even better was the FireFox extension Firebug: This one lets me "inspect" an element, then shows me all of the CSS that affects this element. Which rules are applied directly, which are inherited, which are inherited and overridden. This helps a lot.

Apart from that it comes down too this CSS design strategy:

• learn CSS well, there really isn't a tool that will help you with the thought process; if you have to slap a class on everything, then you're not there yet
• design top-down, main layout elements first, small stuff later
• try to go through your layout sketches, identify stuff that is similar and try to find suitable class names for these - helps a lot in making the structure obvious
• work with standards compliant browsers first, put IE "fixes" into a separate CSS file (included with conditional IE comments) later
• avoid all the "IE hacks", hacks in general, floats, clearfixes, etc. like the plague - you will cry yourself to sleep for nights on end for every time you use these, so better keep it as low as possible
• while we're at it: everybody is shouting at tables, but keep in mind that if your data is really tabular, then use a table... that's what they were meant for!
• if you're stuck with stuff "overlapping" (different kind of elements being lumped together under the same styles), eat the bitter pill and clean up your CSS bringing related stuff together, eliminate special cases as much as possible, use special classes (and sometimes IDs) on outer divs if you have to