betabug... Sascha Welter

home english | home deutsch | Site Map | Sascha | Kontakt | Pro | Weblog | Wiki

Entries : Category [ shellfun ]
Have fun in the Unix shell, with some more or less useful commands
[digital]  [language]  [life]  [security]  [media]  [zope]  [tourism]  [limnos]  [mac]  [athens]  [travel]  [montage]  [food]  [fire]  [zwiki]  [schnipsel]  [music]  [culture]  [shellfun]  [photography]  [hiking]  [pyramid]  [politics]  [bicycle]  [naxos]  [swim] 

17 June 2005

My Connection Sucks^W Drops Too

But I'm taking control of the command line through screen(1)

Yesterday on irc (...sounds like one of those stories...) on #zope I talked to someone who goes by the nick "swissmade" and who was connected through a satellite link. He would come in again and again and ask the same question. When we finally got the answer through to him, he apologized by mentioning that output was often garbled due to the sat link. I recommended a shell account on a well connected server ( shell accounts come to mind, though not with a "free" account, rather one of the paying kind which lets you use background processes after you log out). Together with that, take the "screen" utility and irssi with bitlbee for irc and AIM/ICQ/MSN etc.

I had discovered screen some years ago in an article on daemonnews, the bsd news zine. The article stuck in my mind, as if it was yesterday. It was about a guy who lived in some rural area and through the crappy phone lines he often could get only about a 26kbps dial up connection. And lots of disconnects. Sounds like Greece to me. He solved his disconnect problems using screen - he would still get disconnected, but when he dialled in again, he could resume his session right where he left off. All it needs is a quick

screen -dr
and you get your session exactly where you left it, down to the key you pressed in the middle of a word when the line went down. I relied on this, last summer on Limnos island, when I was getting on the internet through GPRS. Most of the time I just picked up my mail with IMAPS (hey, bandwidth is expensive with GPRS), but sometimes I would take control of my server through my neverending screen session. There are quite a few tutorials about screen(1) around, so when I was hunting for that old article I found a lot of them.

Posted by betabug at 09:10 | Comments (0) | Trackbacks (0)
02 June 2006

Burning Bytes Faster with sed

Logfile Unix Trivia Quiz

This morning's Unix trivia quiz from my friend Saad: What tool to use to chop down a logfile to list only entries after a certain date? Modern admins would likely pull out their Perl, Python, TCL, or other scripting language tool chest. My answer: There is a much, much faster tool for such a simple job. And it's a one liner too. The solution:

sed -n '/02\/Jun\/2006:08:47:14 +0200/,$ p' < infile > outfile
This will find the first line with that date/time in the logfile, then print this and any following lines. A bit of shell redirecting will push it all neatly into a new file.

Reply to the solution: "f* fast sed!" Tools that were written on 1Mhz class machines tend to be fast. It's interesting how e.g. Perl advocates say that using a tool like Perl is better than shell toolkit based solutions, because Perl doesn't need to spawn so many processes. But sed is so fast, you can spawn and run a couple of sed jobs while Perl is still waking up and scratching its camel ballz. Sometimes there is just the right tool for the job. Even though I use sed only very infrequently I managed with a little sed web help to come up with my oneliner while saad was still pushing {/} braces into Perl code... pwned!

Posted by betabug at 10:49 | Comments (0) | Trackbacks (0)
19 March 2007

Generating Tags for Python Code (including Subfolders)

Another of those Unix / Shell "reminder to self" posts

If you want to create a "tags" file for use with vi / vim (or even emacs [makes sign of cross]), there are a couple of tools out there. ctags(1) comes with most modern OSs, but it doesn't do python. So hint 1: Use the script that came with your python distribution [1]. To build tags for all python files in a directory the usage is something like: *.py

Hint 2: If you want to build tags for all python scripts in a folder hierarchy (say for some project) you can use:

find -X . -name \*.py -print | xargs
These of course assume that you moved to somewhere in your shell's $PATH. Hint 3 would be the -X argument to find(1), it's a safety switch that tells find(1) to ignore filenames xargs would choke on.

1: If your system has python but doesn't have, download the matching tarball from, just untar it and find it in the contents (Tools/scripts/

Posted by betabug at 08:53 | Comments (1) | Trackbacks (0)
18 September 2007

Daily Shell Fun

Know your tools

One thing I dislike a lot about the GUI admin tools from Mac OS X Server is that they are quick to turn around and make changes to config files you edited by hand. Sometimes those changes are a real pain, like when on one server I administrate it switched on apache's proxy - despite the "enable proxy" check box being distinctly off. The result was that spammers hit the server pretty hard. On Monday morning when the mess was noticed I had some cleaning up to do. Amongst others the (external) firewall had sent the previous administrator some "attack notification" mails. Some 33000+ such mails to be exact...

Monday morning (I had already plugged the hole) the previous admin called me, telling me about the notifications and asking me if it wouldn't be better to change them to go to my address. Sure, no problem, changed in a minute. An hour later he called again: "Did you make the change? You didn't, the mails are still coming in my mailbox." I suspected some mails to still be in some mail server queue.

Jump into the shell on the server, a quick mailq and... output. A lot of output. It doesn't stop pouring out. All of those mails are destined for him. Time to hit Control-C and clean up.

My first step was to redirect the output from mailq into a file, passing it through grep, searching for his email address. That took forever. I took a line count with wc -l from time to time, just to find out where we were heading. We passed 16000, 20000, 40000, ... It took a lot of time too. Each mail was listed with two lines. Something like this:

B3A1E2002948*     4563 Fri Sep 14 17:10:35
Should have been more careful with that grep line there. Instead of running the command again, I decided to use sed. sed is fast. My command was something like this:
sed -n '/^[1-9]/p' in.txt > out.txt
which resulted in a file that contained only the lines with the queue ID at the start. This sed command ran so fast that I would have been tempted to believe it didn't work - had I not had some previous experience with sed.

I could have isolated that with sed too, but again I was in a hurry (all those mails were on their way to his mailbox, which likely wouldn't be happy to get filled with thousands of alerts). I started vim and hit it with the following command:

This searches in all of the file and puts the distinct hexadecimal ID into a regex group, then replaces only that group. Quick and fast. After saving that file came the last step, running this command as root (Mac OS X uses postfix by default.):
# postsuper -d - < out.txt
postsuper: Deleted: 33222 messages
Say what? There were 33222 alert messages in the queue and some more would already have been saved. Deleting all those messages took a long time by the way. But if the recipient would have had to delete them one-by-one (or a screenful at a time) on his GUI client, it would have taken much longer. All in all it was a nice exercise in the usage of simple Unix shell tools. I could have done it much more elegant and direct, but to my defense I tried to get running before I was thinking.

Posted by betabug at 21:39 | Comments (1) | Trackbacks (0)
22 April 2008

Even Printers Need Love

Look after their needs and feelings!
LOVE LOW... PET ME! on printer's status display

Yes, nowadays printers too need attention, care, ... and sometimes even love. Fortunately advanced electronic devices can call out for help if one of their basic needs is running low. Unfortunately for the machine in question, people just come to grab up their printouts, without ever glancing at the display or paying attention to the little machine's feelings.

Just in case you wonder why this is in the "digital" category: It's done with a little perl (ugh!) script I found in a blog entry titled INSERT COIN. All you need is a networked HP laser printer and its IP address.

Posted by betabug at 15:05 | Comments (1) | Trackbacks (0)
21 May 2008

find, sed, and xargs in the same command line

How geeky can you get?

A simple problem for some shell fun: I get handed a folder full of folders, each of the child folders containing one font file (.otf file). The child folders names contain spaces. I want all those .otf files in one single folder, but I don't want to go through all the folders with the mouse and drag and drop them over (that's so 1980s).

Instead I spent some time (probably more than drag and dropping would have taken) to try and assemble this puppy of a command line:

find . -name *.otf -print | sed -e 's/.*/"&"/' | xargs -J % mv % ../mpr/

With find, sed, and xargs all in the same one liner, total command line geek fun ensues!

Posted by betabug at 13:38 | Comments (1) | Trackbacks (0)
09 June 2008

Shell fun: at(1) and leave(1)

Time is on your side!

Want to send a mail message at a pre-specified time? Want to be reminded when to leave from work? People have worked with the Unix shell for many years now and they have developed the tools to make things like that easy. Here is how to do these things with at and leave...

First our mail example. There is a nice command called at, which allows things to be run "at a specified time". You can of course view it's man page with man at, but here is a quick rundown: First, you specify the time you want a command to be run as the commands argument, in 24 hour time (without the ":"), like at 2330. You can also give it a relative time as in at + 6 hours or at 0130 + 3 days. The actual command that will be run is expected on the "standard input", so after running the at command, you can type it in (or when in a script, you can pipe it in) and type ^D (control-D) when you're done:

at + 6 hours
mail -s 'Mail from the past'
This mail...

        ... comes from the past.
          (6 hours past, to be exact.)
commands will be executed using /bin/ksh
job 1213010520.c at Mon Jun  9 18:22:00 2008

Of course this isn't very comfortable, so you might want to prepare the message in advance in a text file and pipe it in:

at + 1 hours
cat todo_while_im_away.txt | mail -s "I'm on my way"
commands will be executed using /bin/ksh
job 1213016478.c at Mon Jun  9 13:34:00 2008

... and now leave!

Which maybe reminds you that you need a reminder to stop that concentrated hacking session, zoom out of "the zone" and get going, because you gonna miss your train. No problem, just tell your shell at what time you will have to leave.

The usage of leave (read the man page with man leave again) is even simpler:

leave 1800
# or even:
leave +0030

In the first case you want to leave at 6pm, in the second case you want "just 30 minutes more" of sweet hacking time.

How does leave remind you? It will "beep" your terminal and print "You have to leave in 5 minutes." and then "Just one more minute!", followed by "Time to leave!" on your terminal. It doesn't wait until you enter a command to get the next terminal prompt, it writes over your input right now. Leave will continue prompting, until you kill its process or log out (thus killing its process too).

Do I see that?

leave was obviously written in a time when people worked in one shell window alone. For a long time I didn't gt it to be much useful any more. But nowadays I use screen(1) a lot and have an irssi session open for IRC (and AIM through bitlbee). Inside screen, I will be visibly and audibly reminded, that there is a "bell in window 3"... and I can go and check and see that I should leave in 5 minutes. So "screen" makes "leave" useful again.

Posted by betabug at 10:41 | Comments (1) | Trackbacks (0)
12 June 2008

Shell fun: get reminders with calendar(1)

This is so obvious

Quite similar to my last, lame shell fun post (which mentioned at(1)), is the calendar utility. This thing (when your system is set up properly) will send you reminder mails for calendar events you write into a special plain text file. It can also remind you of upcoming stuff when you log into your system. calendar is ages old, but still very useful when done right, especially for us coder types, who don't have 25 appointments each day...

The calendar file is usuall ~.calendar or (in order to be able to use sub-files at some point) ~/.calendar/calendar. Let's look at some typical entries there:

06/15    Write next shell fun post

Where the whitespace between the date and the text should consist of a Tab character. In theory different date formats should be recognized, but I hadn't much success with it, so I'm sticking to the weird US style of mm/dd. There's no year, if you don't want this event to stick around next year, go and clean up some time.

But wait! Some stuff that comes back each year makes very much sense, like... birthdays!

08/01    w... birthday
04/17    s.... birthday

There are some more fancy things you can do, like:

*/01    Rent!
/* simple: the first day of each month */

*/Mon-1    Monday - Notify about HelMUG Meeting next Sunday
/* which will be shown on the last Monday of each month */

*/Tue+1    OpenCoffee Athens First Tuesday Each Month

The last one will remind me to go to the OpenCoffee Athens event, to meet Stavros Messinis. If OpenCoffee is at all interesting (I don't know yet, haven't been there), I can leave this in and will be reminded each month.

To be reminded

To actually see the reminders, I have two ways:

  • My OpenBSD system is set up in /etc/daily to run calendar as root and mail reminders to all users. (We're assuming that mail is actually set up properly to deliver those mails.):

    echo "Running calendar in the background."
    calendar -a &amp;
  • I've got this here in my .kshrc, which prints out the events for the next 10 days when I log in to an interactive terminal:

    if [ -o interactive ]; then
        echo coming up:
        /usr/bin/calendar -A 10

Also I can always run calendar -A 30 (or similar) to see the upcoming events for the next 30 days. As always, man calendar knows more!

Posted by betabug at 14:06 | Comments (0) | Trackbacks (0)
17 June 2008

Shell fun: escape ssh(1)

Hidden channels of communication

This shell fun post really needs a hat tip to kargig for his SSH Escape Characters post. I had read about ssh escape characters ages ago and never considered them very useful. kargig's post inspired me to experiment a bit more and finally (with a default setup that works for me and 2 nice use cases for me) I start to like those hidden beasts...

First of all, if you jack in to other servers using the default OpenSSH configuration, ~ is already set up as the default ssh escape character. But there is a trick to it: If you just press ~ while ssh-ing to another system, nothing happens. That is a Good Thing(TM), because otherwise I would get confused whenever I wanted to spy on Wu by typing cd ~wu. If you want to get into the "hidden ssh menu", you have to press "Return" (aka "Enter" for some people) first, then the tilde ~.

And voilà! So press Enter, followed by ~? and you will see this nice help menu:

kronos /home/betabug$ ~?
Supported escape sequences:
~.  - terminate connection
~B  - send a BREAK to the remote system
~C  - open a command line
~R  - Request rekey (SSH protocol 2 only)
~^Z - suspend ssh
~#  - list forwarded connections
~&amp;  - background ssh (when waiting for connections to terminate)
~?  - this message
~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

The characters that form this output actually come from your local system. As kargig mentions, one he gets out of it is to terminate hanging sessions - I don't much have that problem (maybe I should start to use Linux too...), but I have two other nice uses for this stuff:

Which sometimes is a nice little time saver!

Posted by betabug at 18:47 | Comments (0) | Trackbacks (0)
16 July 2008

Shell fun: take a vacation(1)

Nobody likes mail auto-responders, but...

Nobody likes auto-reply messages, but sometimes - and when used correctly - the Unix vacation(1) utility can be useful. There are a few steps to set it up. I'm taking this on my OpenBSD machine, other flavors of Unix might vary...

The first step in our "used correctly" plan is to use the official vacation program for our "auto-away" messages, not just some dumb auto-reply script. vacation is clever enough to keep a log of who it has informed of you being away (thus avoiding mail loops) and it know to avoid replying to mailing lists and bulk mails. All this is in contrast to some other such solutions, it's incredible what kind of dumb stuff appears till this day, when the well behaving Unix programs have been around so long.

Not only is vacation much safer and cleverer than other auto-reply scripts, it's also ultimately more geeky - and thus rewarding!

Initialize vacation

You need to initialize the vacation program for your account. Simply type:

vacation -i

while being logged in as your normal user (the account you want to use vacation for. This will generate the database ~/.vacation.db for your account.

sendmail restricted shell setup

If - like on OpenBSD - your sendmail is using the "sendmail restricted shell", you will have to enable the use of the vacation program with sendmail. This happens by adding a soft link to the vacation executable to a special directory. Sendmail will only execute programs that have links there. This setup has to be done only once, then it will work for all users. As root do:

# ln -s `which vacation` /usr/libexec/sm.bin/vacation

(Of course with the correct location of that directory, consult your docs if in doubt.)


The file ~/.vacation.msg contains the full message that will be delivered to recipients. man vacation gives a nice example. If you're not used to type mail messages with full headers, take a good look and mind to include the blank line between the headers and the message body. It could look something like this:

From: (Fred Unixuser)
Subject: I am on vacation
Delivered-By-The-Graces-Of: The Vacation program
Precedence: bulk

I am on vacation from last year till next year.
I'll get your mail with subject
but I will be *very* slow to react.

Fred Unixuser

vacation will include the senders subject wherever you type $SUBJECT.


To enable all of this vacation mail reply sending, you could be using a simple .forward file in your home directory. It would contain something like this:

\loginname, "|/usr/libexec/sm.bin/vacation -a alias1 -a aka1 loginname"

Here you have:

  1. the login name of your account, with a backslash in front, so you actually get a copy of the incoming mail message
  2. in quotes the command to pipe the message to vacation
  3. with one or multiple -a parameters, you can tell vacation to also reply to mails with those "aliases" somewhere in the recipient headers.
  4. the last argument to vacation is again the login name of your account

Give it a try!

With all this set up, you can try sending mail to your account. For the first mail you send there, you should get an auto-reply. Further mails shouldn't get auto-replies (until - in the default setting - one week has passed). Don't forget to switch it off when you're back... wait... you're coming back from that vacation, right? I can't do all the job myself!

Posted by betabug at 15:28 | Comments (0) | Trackbacks (0)
16 October 2008

Shellfun: Let the past come to haunt you!

Using history-search in ksh

OpenBSD's standard ksh shell has a feature I didn't learn for a long time: It allows you to search through past used commands (the "history" - Other modern shells have similar mechanisms, I'll follow OpenBSD's ksh here anyway.). Most people are familiar with using the arrow keys to go step by step back through the history to re-use a command you used a while ago, instead of typing it in again and again.

But it could get tiresome when you would have to step through a lot of commands to find that clever assortment of pipes and redirections with just the right set of parameters on that command you used 4 hours ago. What you can do there is what man ksh describes as "search-history". You pull it up with ^R (control-r) and start typing a few characters that you remember from your command line. "search-history" will then start to autocomplete with what it finds from your command history. Typing ^R again will get you the "next find" in the list, ^C will cancel the search and of course you can edit the line that you got. As usual, man ksh has a bit more information (grep for "search-history" in there).

Posted by betabug at 09:51 | Comments (0) | Trackbacks (0)
17 March 2009

Shell fun: limit bandwidth on scp

... but is that really fun?

I wasn't really sure if this post should go into the shell fun category.. but anyway here goes a little hint about one sometimes overlooked feature of everybody's beloved OpenSSH scp command: You can limit the bandwidth it uses with the -l parameter. Just give it the maximum number of kilobits/second that it should use.

This parameter can be fun, because it can allow you to surf the web. How so? Imagine you have a limited amount of uplink bandwidth - like on your average ADSL line - ok, no need to imagine much there, you probably are there already. Now you fill up all that uplink bandwidth by beaming a big file to some remote server with scp. What happens is that your asymmetrical pipe is congested and surfing (or irc chatting or whatever) becomes painfully slow. You could go all the way and implement some traffic shaping. Or you could switch to using

scp -l 200 hugefile.tgz

and thus use only 25kByte/second (200kbit/sec = ~25kByte/sec) of your wimpy 1024kbit/second uplink bandwidth for this. The uplink will take longer, but at least you can get back to read some webcomics - or whatever urgent business you had at hand.

Posted by betabug at 14:09 | Comments (4) | Trackbacks (0)
28 July 2009

How to enable identd / auth on Mac OS X 10.4

Fast and simple... no installs needed

IRC on the shell is fun... but some IRC servers require to get an auth response from your machine. Since I'm on a 3G connection right now and I wanted to IRC on Mac OS X... which doesn't run identd by default, I had a non-fun problem.

Now there are people saying that you need to install some software on Mac OS X to get identd (the "auth" service) running. At least for 10.4 this is not true, everything is still there in the system. Of course Apple is progressively trying to cripple the Unix base of the system, e.g. by replacing simple config files with weird xml setups. But still it's possible to get things going with built in tools.

So what did I do (apart from searching the f* web)?

Step 1: Enable the auth service in /etc/inetd.conf

Yes, indeed, the very good old inetd.conf, not xinetd, not launchd, not xml something. There is just one line that needs to be uncommented. Change this line:

# auth    stream  tcp     wait    root    /usr/libexec/identd     identd -w -t120

to this line:

auth    stream  tcp     wait    root    /usr/libexec/identd     identd -w -t120

Step 2: Start inetd

Yes, indeed again, inetd is still there. It's probably working through 3 layers or more of Apple bogon fields (aka inetd over xinetd over launchd), but it works. Run this little command as root:

# /System/Library/StartupItems/IPServices/IPServices start

As mentioned, this works for me in 10.4 (Tiger), where I needed it to connect to IRC. Dunno if this still works for 10.5 (Leopard), but I guess it might as well work - give it a try and leave a comment.

Also this doesn't auto-start when you start up the machine. You'll either have to repeat step 2 after each restart ... or find out how to auto-start things on boot. Have fun!

Posted by betabug at 10:43 | Comments (2) | Trackbacks (0)
06 January 2010

Mistypes and Other Fun

Smelling Pistakes for everyone!

While my server install moves forward (having passed the most crucial stages and avoided the whole Open Firmware question by booting from bsd.rd and using the upgrade mechanism), I was browsing through the OpenBSD packages directory for my platform. I was trawling for stuff I had forgotten and for stuff that might be fun. Here is some:

sl - Steam Locomotive

The Steam Locomotive "utility" explores one very common typo that lots of people make in the shell: instead of typing ls, we are prone to type sl - which now lets a huge, ascii Steam Locomotive roll over your terminal. Kind of reminds me of the "insults" setting of sudo, but in more of a "clown hitting you with an anvil on the head" mode.

globe - display the currently lighted face of Earth in ASCII

Let's face it: ASCII roxx. All those funny tools that convert pictures to ASCII art, or even videos to ASCII movies leave me with a chuckle. globe is not that special, but looking at the surface of the earth as currently seen from the sun is kind of neat. Also lets you guess which of your worldwide friends might be awake right now.

figlet - generates ASCII banner art

Apart from those two new additions, I got the ever fun figlet, which you all knew and love, which generates ASCII banner art.

 .::::::..,:::::: .,:::::: .-::::-.
;;;`    `;;;;'''' ;;;;'''';;'```;;;
'[==/[[[[,[[cccc   [[cccc    ,n[['
  '''    $$$""""   $$""""   d$P"
 88b    dP888oo,__ 888oo,__ ""

Figlet offers a lot of different "fonts" to display what you give it in huge letters combined out of ASCII letters. Indispensable, what else can I say?

(Get all these from the links given or in OpenBSD directly through the package system.)

Posted by betabug at 18:53 | Comments (0) | Trackbacks (0)
16 March 2012

Where to find all the setxkbmap options

They're not in the man page, but you've got them already

When configuring a keyboard on a unixish machine running X (aka X11, the X Windows System), nowadays the way to configure things is using setxkbmap(1). There's some good documentation in the manpage, but at some point the man page tells us only that there are options one can configure... but not what those possible options are. At first I went on a broad search through the web, until I figured that the directory path given at the top of the man page is a strong hint. In fact the list of predefined options can be found right on the system, in:


In that file we first find a list of keyboard "model" names, to be used like:

setxkbmap -model thinkpad

Next come the keyboard layouts, for various languages and configurations. These can be combined, like here where I'm using the Swiss (German) and the US keyboard alternatively - the stuff in the parenthesis is the "variant" - Swiss keyboards come in German or French:

setxkbmap 'ch(de),us'

The last big part of the list are all the "options" for setting up various keys and key combinations. So in my Swiss+US combo, I'm switching languages using Alt+Space (which is pretty close to how it's done on a Mac, given that the left Alt key is right next to the space bar). There are tons of options to switch keyboard layouts. I also have the Caps Lock key configured as a Control key - since in an setup, lots of stuff are done with the Ctrl key and I'm never using CapsLock anyway:

setxkbmap -option 'grp:alt_space_toggle,ctrl:nocaps' 'ch(de),us'

Multiple options are separated by comma. I'm single-quoting them, so no shell confusion - don't think it's strictly necessary.

There's also tons of options to set up various of the extra keys to be found on any average PC keyboard (5th level key anyone?) and for stuff like how to use the numerical keypad. Just have a look, it's right in your system!

Posted by betabug at 23:01 | Comments (5) | Trackbacks (0)
03 April 2012

A Glimpse into the Matrix

In the shell, where else
A glimpse into the structure of the matrix, right in your shell

The Unix shell is the closest thing we have to working with the very structure of the Matrix surrounding us. It's as close to the bare metal as we can get. (Sure, there are people trying to sell you a debugger for that, but what does their debugger run in? Right, thought so.)

Now the OpenBSD ports tree contains a tool that lets you see the Matrix right in your shell window. It's called cmatrix and you can grab it from packages right away. (If your OS doesn't have it in their package manager, you might get it from the one who made it.)

Obviously it shows the Matrix in its encrypted version. It's up to you to see the world in there or whatever you want. Figure it out and you might become someone important... like an actor.

Posted by betabug at 14:45 | Comments (0) | Trackbacks (0)
[1]   2   Next