- Entries : Category [ shellfun ]
- Have fun in the Unix shell, with some more or less useful commands
17 June 2005
My Connection Sucks^W Drops Too
But I'm taking control of the command line through screen(1)
Yesterday on irc (...sounds like one of those stories...) on #zope I
talked to someone who goes by the nick "swissmade" and who was connected
through a satellite link. He would come in again and again and ask
the same question. When we finally got the answer through to him, he
apologized by mentioning that output was often garbled due to the sat
link. I recommended a shell account on a well connected server
(freeshell.org shell accounts come
to mind, though not with a "free" account,
rather one of the paying kind which lets you use background processes
after you log out). Together with that, take the "screen" utility and
irssi with bitlbee for irc and AIM/ICQ/MSN etc.
I had discovered screen some years ago in an article on
daemonnews, the bsd news zine.
The article stuck in my mind, as if it was yesterday. It was about a guy
who lived in some rural area and through the crappy phone lines he often
could get only about a 26kbps dial up connection. And lots of
disconnects. Sounds like Greece to me. He
solved his disconnect
problems using screen - he would still get disconnected, but when he
dialled in again, he could resume his session right where he left off.
All it needs is a quick
screen -dr
and you get your session exactly where you left it, down to the key you
pressed in the middle of a word when the line went down. I relied on
this,
last summer on Limnos island, when I was getting on the internet through
GPRS. Most of the time I just picked up my mail with IMAPS (hey,
bandwidth is expensive with GPRS), but sometimes I would take control of
my server through my neverending screen session. There are quite a few
tutorials about screen(1) around, so when I was hunting for that old
article I found a lot of them.
02 June 2006
Burning Bytes Faster with sed
Logfile Unix Trivia Quiz
This morning's Unix trivia quiz from my friend Saad:
What tool to use to chop down a logfile
to list only entries after a certain date?
Modern admins would likely
pull out their Perl, Python, TCL, or other scripting language tool chest. My
answer: There is a much, much faster tool for such a simple job. And
it's a one liner too. The solution:
sed -n '/02\/Jun\/2006:08:47:14 +0200/,$ p' < infile > outfile
This will find the first line with that date/time in the logfile, then
print this and any following lines. A bit of shell redirecting will push
it all neatly into a new file.
Reply to the solution: "f* fast sed!" Tools that were written on 1Mhz
class machines tend to be fast. It's interesting how e.g. Perl
advocates say that using a tool like Perl is better than shell toolkit
based solutions, because Perl doesn't need to spawn so many processes.
But sed is so fast, you can spawn and run a couple of sed jobs while Perl is still
waking up and scratching its camel ballz. Sometimes there is just the
right tool for the job. Even though I use sed only very infrequently I
managed with a little sed web help to come up with my oneliner while saad
was still pushing {/} braces into Perl code... pwned!
19 March 2007
Generating Tags for Python Code (including Subfolders)
Another of those Unix / Shell "reminder to self" posts
If you want to create a "tags" file for use with vi / vim (or even emacs [makes sign of cross]), there are a couple of tools out there. ctags(1) comes with most modern OSs, but it doesn't do python. So hint 1: Use the script ptags.py that came with your python distribution [1]. To build tags for all python files in a directory the usage is something like:
ptags.py *.py
Hint 2: If you want to build tags for all python scripts in a folder
hierarchy (say for some project) you can use:
find -X . -name \*.py -print | xargs ptags.py
These of course assume that you moved ptags.py to somewhere in your
shell's $PATH. Hint 3 would be the
-X argument to find(1), it's a safety switch that tells find(1) to ignore filenames xargs would choke on.
1: If your system has python but doesn't have ptags.py, download the
matching tarball from python.org, just untar it and find it in the
contents (Tools/scripts/ptags.py).
18 September 2007
Daily Shell Fun
Know your tools
One thing I dislike a lot about the GUI admin tools from Mac OS X Server is that they are quick to turn around and make changes to config files you edited by hand. Sometimes those changes are a real pain, like when on one server I administrate it switched on apache's proxy - despite the "enable proxy" check box being distinctly off. The result was that spammers hit the server pretty hard. On Monday morning when the mess was noticed I had some cleaning up to do. Amongst others the (external) firewall had sent the previous administrator some "attack notification" mails. Some 33000+ such mails to be exact...
Monday morning (I had already plugged the hole) the previous admin called me, telling me about the notifications and asking me if it wouldn't be better to change them to go to my address. Sure, no problem, changed in a minute. An hour later he called again: "Did you make the change? You didn't, the mails are still coming in my mailbox." I suspected some mails to still be in some mail server queue.
Jump into the shell on the server, a quick mailq and... output. A lot of output. It doesn't stop pouring out. All of those mails are destined for him. Time to hit Control-C and clean up.
My first step was to redirect the output from mailq into a file, passing it through grep, searching for his email address. That took forever. I took a line count with wc -l from time to time, just to find out where we were heading. We passed 16000, 20000, 40000, ... It took a lot of time too. Each mail was listed with two lines. Something like this:
B3A1E2002948* 4563 Fri Sep 14 17:10:35 xy@example.org
xy@example.org
Should have been more careful with that grep line there. Instead of running the command again, I decided to use
sed.
sed is fast. My command was something like this:
sed -n '/^[1-9]/p' in.txt > out.txt
which resulted in a file that contained only the lines with the queue ID at the start. This sed command ran so fast that I would have been tempted to believe it didn't work - had I not had some previous experience with sed.
I could have isolated that with sed too, but again I was in a hurry (all those mails were on their way to his mailbox, which likely wouldn't be happy to get filled with thousands of alerts). I started vim and hit it with the following command:
:%s/^\([0-9A-F]*\).*/\1/
This searches in all of the file and puts the distinct hexadecimal ID into a regex group, then replaces only that group. Quick and fast. After saving that file came the last step, running this command as root (Mac OS X uses postfix by default.):
# postsuper -d - < out.txt
postsuper: Deleted: 33222 messages
Say what? There were
33222 alert messages in the queue and some more would already have been saved. Deleting all those messages took a long time by the way. But if the recipient would have had to delete them one-by-one (or a screenful at a time) on his GUI client, it would have taken much longer. All in all it was a nice exercise in the usage of simple Unix shell tools. I could have done it much more elegant and direct, but to my defense I tried to get running before I was thinking.
22 April 2008
Even Printers Need Love
Look after their needs and feelings!
Yes, nowadays printers too need attention, care, ... and sometimes even love. Fortunately advanced electronic devices can call out for help if one of their basic needs is running low. Unfortunately for the machine in question, people just come to grab up their printouts, without ever glancing at the display or paying attention to the little machine's feelings.
Just in case you wonder why this is in the "digital" category: It's done with a little perl (ugh!) script I found in a blog entry titled INSERT COIN. All you need is a networked HP laser printer and its IP address.
21 May 2008
find, sed, and xargs in the same command line
How geeky can you get?
A simple problem for some shell fun: I get handed a folder full of folders, each of the child folders containing one font file (.otf file). The child folders names contain spaces. I want all those .otf files in one single folder, but I don't want to go through all the folders with the mouse and drag and drop them over (that's so 1980s).
Instead I spent some time (probably more than drag and dropping would have taken) to try and assemble this puppy of a command line:
find . -name *.otf -print | sed -e 's/.*/"&"/' | xargs -J % mv % ../mpr/
With find, sed, and xargs all in the same one liner, total command line geek fun ensues!
09 June 2008
Shell fun: at(1) and leave(1)
Time is on your side!
Want to send a mail message at a pre-specified time? Want to be reminded when to leave from work? People have worked with the Unix shell for many years now and they have developed the tools to make things like that easy. Here is how to do these things with at and leave...
First our mail example. There is a nice command called at, which allows things to be run "at a specified time". You can of course view it's man page with man at, but here is a quick rundown: First, you specify the time you want a command to be run as the commands argument, in 24 hour time (without the ":"), like at 2330. You can also give it a relative time as in at + 6 hours or at 0130 + 3 days. The actual command that will be run is expected on the "standard input", so after running the at command, you can type it in (or when in a script, you can pipe it in) and type ^D (control-D) when you're done:
at + 6 hours
mail -s 'Mail from the past' betabug@example.org
This mail...
... comes from the past.
(6 hours past, to be exact.)
^D
commands will be executed using /bin/ksh
job 1213010520.c at Mon Jun 9 18:22:00 2008
Of course this isn't very comfortable, so you might want to prepare the message in advance in a text file and pipe it in:
at + 1 hours
cat todo_while_im_away.txt | mail -s "I'm on my way" betabug@example.org
^D
commands will be executed using /bin/ksh
job 1213016478.c at Mon Jun 9 13:34:00 2008
Which maybe reminds you that you need a reminder to stop that concentrated hacking session, zoom out of "the zone" and get going, because you gonna miss your train. No problem, just tell your shell at what time you will have to leave.
The usage of leave (read the man page with man leave again) is even simpler:
leave 1800
# or even:
leave +0030
In the first case you want to leave at 6pm, in the second case you want "just 30 minutes more" of sweet hacking time.
How does leave remind you? It will "beep" your terminal and print "You have to leave in 5 minutes." and then "Just one more minute!", followed by "Time to leave!" on your terminal. It doesn't wait until you enter a command to get the next terminal prompt, it writes over your input right now. Leave will continue prompting, until you kill its process or log out (thus killing its process too).
leave was obviously written in a time when people worked in one shell window alone. For a long time I didn't gt it to be much useful any more. But nowadays I use screen(1) a lot and have an irssi session open for IRC (and AIM through bitlbee). Inside screen, I will be visibly and audibly reminded, that there is a "bell in window 3"... and I can go and check and see that I should leave in 5 minutes. So "screen" makes "leave" useful again.
12 June 2008
Shell fun: get reminders with calendar(1)
This is so obvious
Quite similar to my last, lame shell fun post (which mentioned at(1)), is the calendar utility. This thing (when your system is set up properly) will send you reminder mails for calendar events you write into a special plain text file. It can also remind you of upcoming stuff when you log into your system. calendar is ages old, but still very useful when done right, especially for us coder types, who don't have 25 appointments each day...
The calendar file is usuall ~.calendar or (in order to be able to use sub-files at some point) ~/.calendar/calendar. Let's look at some typical entries there:
06/15 Write next shell fun post
Where the whitespace between the date and the text should consist of a Tab character. In theory different date formats should be recognized, but I hadn't much success with it, so I'm sticking to the weird US style of mm/dd. There's no year, if you don't want this event to stick around next year, go and clean up some time.
But wait! Some stuff that comes back each year makes very much sense, like... birthdays!
08/01 w... birthday
04/17 s.... birthday
There are some more fancy things you can do, like:
*/01 Rent!
/* simple: the first day of each month */
*/Mon-1 Monday - Notify about HelMUG Meeting next Sunday
/* which will be shown on the last Monday of each month */
*/Tue+1 OpenCoffee Athens First Tuesday Each Month
The last one will remind me to go to the OpenCoffee Athens event, to meet Stavros Messinis. If OpenCoffee is at all interesting (I don't know yet, haven't been there), I can leave this in and will be reminded each month.
To actually see the reminders, I have two ways:
My OpenBSD system is set up in /etc/daily to run calendar as root and mail reminders to all users. (We're assuming that mail is actually set up properly to deliver those mails.):
echo "Running calendar in the background."
calendar -a &
I've got this here in my .kshrc, which prints out the events for the next 10 days when I log in to an interactive terminal:
if [ -o interactive ]; then
echo coming up:
/usr/bin/calendar -A 10
fi
Also I can always run calendar -A 30 (or similar) to see the upcoming events for the next 30 days. As always, man calendar knows more!
17 June 2008
Shell fun: escape ssh(1)
Hidden channels of communication
This shell fun post really needs a hat tip to kargig for his SSH Escape
Characters post. I had read about ssh escape characters ages ago
and never considered them very useful. kargig's post inspired me to
experiment a bit more and finally (with a default setup that works for me
and 2 nice use cases for me) I start to like those hidden beasts...
First of all, if you jack in to other servers using the default OpenSSH
configuration, ~ is already set up as the default ssh escape
character. But there is a trick to it: If you just press ~ while
ssh-ing to another system, nothing happens. That is a Good Thing(TM),
because otherwise I would get confused whenever I wanted to spy on Wu by
typing cd ~wu. If you want to get into the "hidden ssh menu", you
have to press "Return" (aka "Enter" for some people) first, then the
tilde ~.
And voilà! So press Enter, followed by ~? and you will see this nice help menu:
kronos /home/betabug$ ~?
Supported escape sequences:
~. - terminate connection
~B - send a BREAK to the remote system
~C - open a command line
~R - Request rekey (SSH protocol 2 only)
~^Z - suspend ssh
~# - list forwarded connections
~& - background ssh (when waiting for connections to terminate)
~? - this message
~~ - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)
The characters that form this output actually come from your local
system. As kargig mentions, one he gets out of it is to terminate
hanging sessions - I don't much have that problem (maybe I should start
to use Linux too...), but I have two other nice uses for this stuff:
Adding a port forwarding to an existing connection. I.e. to
add a forwarding for VNC to an existing connection, do ~C and
on the prompt do:
ssh> ?
Commands:
-L[bind_address:]port:host:hostport Request local forward
-R[bind_address:]port:host:hostport Request remote forward
-KR[bind_address:]port Cancel remote forward
ssh> -L 5900:127.0.0.1:5900
My other use is coupled with ssh-agent(1): When I have ssh-agent set up
to "cache" my ssh private key and I use a terminal window to log in to
a server, I often want to do a short job with the ssh-key too (like
scp'ing a file or doing a quick login to another machine). Now I can
temporarily background my main ssh connection and do my job:
kronos /home/betabug$ ~^Z [suspend ssh]
Suspended
[briareus:~] betabug% scp important_file.tgz lamesrv:
[briareus:~] betabug% fg
kronos /home/betabug$
Which sometimes is a nice little time saver!
16 July 2008
Shell fun: take a vacation(1)
Nobody likes mail auto-responders, but...
Nobody likes auto-reply messages, but sometimes - and when used correctly - the Unix vacation(1) utility can be useful. There are a few steps to set it up. I'm taking this on my OpenBSD machine, other flavors of Unix might vary...
The first step in our "used correctly" plan is to use the official vacation program for our "auto-away" messages, not just some dumb auto-reply script. vacation is clever enough to keep a log of who it has informed of you being away (thus avoiding mail loops) and it know to avoid replying to mailing lists and bulk mails. All this is in contrast to some other such solutions, it's incredible what kind of dumb stuff appears till this day, when the well behaving Unix programs have been around so long.
Not only is vacation much safer and cleverer than other auto-reply scripts, it's also ultimately more geeky - and thus rewarding!
You need to initialize the vacation program for your account. Simply type:
vacation -i
while being logged in as your normal user (the account you want to use vacation for. This will generate the database ~/.vacation.db for your account.
If - like on OpenBSD - your sendmail is using the "sendmail restricted shell", you will have to enable the use of the vacation program with sendmail. This happens by adding a soft link to the vacation executable to a special directory. Sendmail will only execute programs that have links there. This setup has to be done only once, then it will work for all users. As root do:
# ln -s `which vacation` /usr/libexec/sm.bin/vacation
(Of course with the correct location of that directory, consult your docs if in doubt.)
The file ~/.vacation.msg contains the full message that will be delivered to recipients. man vacation gives a nice example. If you're not used to type mail messages with full headers, take a good look and mind to include the blank line between the headers and the message body. It could look something like this:
From: me@example.org (Fred Unixuser)
Subject: I am on vacation
Delivered-By-The-Graces-Of: The Vacation program
Precedence: bulk
I am on vacation from last year till next year.
I'll get your mail with subject
``$SUBJECT``
but I will be *very* slow to react.
Fred Unixuser
vacation will include the senders subject wherever you type $SUBJECT.
To enable all of this vacation mail reply sending, you could be using a simple .forward file in your home directory. It would contain something like this:
\loginname, "|/usr/libexec/sm.bin/vacation -a alias1 -a aka1 loginname"
Here you have:
- the login name of your account, with a backslash in front, so you actually get a copy of the incoming mail message
- in quotes the command to pipe the message to vacation
- with one or multiple -a parameters, you can tell vacation to also reply to mails with those "aliases" somewhere in the recipient headers.
- the last argument to vacation is again the login name of your account
With all this set up, you can try sending mail to your account. For the first mail you send there, you should get an auto-reply. Further mails shouldn't get auto-replies (until - in the default setting - one week has passed). Don't forget to switch it off when you're back... wait... you're coming back from that vacation, right? I can't do all the job myself!
16 October 2008
Shellfun: Let the past come to haunt you!
Using history-search in ksh
OpenBSD's standard ksh shell has a feature I didn't learn for a
long time: It allows you to search through past used commands (the
"history" - Other modern shells have similar mechanisms, I'll follow
OpenBSD's ksh here anyway.). Most people are familiar with using
the arrow keys to go step by step back through the history to re-use
a command you used a while ago, instead of typing it in again and
again.
But it could get tiresome when you would have to step through a lot
of commands to find that clever assortment of pipes and redirections
with just the right set of parameters on that command you used 4
hours ago. What you can do there is what man ksh describes as
"search-history". You pull it up with ^R (control-r) and start
typing a few characters that you remember from your command line.
"search-history" will then start to autocomplete with what it finds
from your command history. Typing ^R again will get you the "next
find" in the list, ^C will cancel the search and of course you can
edit the line that you got. As usual, man ksh has a bit more
information (grep for "search-history" in there).
17 March 2009
Shell fun: limit bandwidth on scp
... but is that really fun?
I wasn't really sure if this post should go into the shell fun category.. but anyway here goes a little hint about one sometimes overlooked feature of everybody's beloved OpenSSH scp command: You can limit the bandwidth it uses with the -l parameter. Just give it the maximum number of kilobits/second that it should use.
This parameter can be fun, because it can allow you to surf the web. How so? Imagine you have a limited amount of uplink bandwidth - like on your average ADSL line - ok, no need to imagine much there, you probably are there already. Now you fill up all that uplink bandwidth by beaming a big file to some remote server with scp. What happens is that your asymmetrical pipe is congested and surfing (or irc chatting or whatever) becomes painfully slow. You could go all the way and implement some traffic shaping. Or you could switch to using
scp -l 200 hugefile.tgz example.org:bigstuff/
and thus use only 25kByte/second (200kbit/sec = ~25kByte/sec) of your wimpy 1024kbit/second uplink bandwidth for this. The uplink will take longer, but at least you can get back to read some webcomics - or whatever urgent business you had at hand.
28 July 2009
How to enable identd / auth on Mac OS X 10.4
Fast and simple... no installs needed
IRC on the shell is fun... but some IRC servers require to get an auth response from your machine. Since I'm on a 3G connection right now and I wanted to IRC on Mac OS X... which doesn't run identd by default, I had a non-fun problem.
Now there are people saying that you need to install some software on Mac OS X to get identd (the "auth" service) running. At least for 10.4 this is not true, everything is still there in the system. Of course Apple is progressively trying to cripple the Unix base of the system, e.g. by replacing simple config files with weird xml setups. But still it's possible to get things going with built in tools.
So what did I do (apart from searching the f* web)?
Yes, indeed, the very good old inetd.conf, not xinetd, not launchd, not xml something. There is just one line that needs to be uncommented. Change this line:
# auth stream tcp wait root /usr/libexec/identd identd -w -t120
to this line:
auth stream tcp wait root /usr/libexec/identd identd -w -t120
Yes, indeed again, inetd is still there. It's probably working through 3 layers or more of Apple bogon fields (aka inetd over xinetd over launchd), but it works. Run this little command as root:
# /System/Library/StartupItems/IPServices/IPServices start
As mentioned, this works for me in 10.4 (Tiger), where I needed it to connect to IRC. Dunno if this still works for 10.5 (Leopard), but I guess it might as well work - give it a try and leave a comment.
Also this doesn't auto-start when you start up the machine. You'll either have to repeat step 2 after each restart ... or find out how to auto-start things on boot. Have fun!
06 January 2010
Mistypes and Other Fun
Smelling Pistakes for everyone!
While my server install moves forward (having passed the most crucial
stages and avoided the whole Open Firmware question by booting from bsd.rd
and using the upgrade mechanism), I was browsing through the OpenBSD
packages directory for my platform. I was trawling for stuff I had forgotten
and for stuff that might be fun. Here is some:
The Steam Locomotive "utility" explores one very common typo that
lots of people make in the shell: instead of typing ls, we are prone
to type sl - which now lets a huge, ascii Steam Locomotive roll
over your terminal. Kind of reminds me of the "insults" setting of
sudo, but in more of a "clown hitting you with an anvil on the head"
mode.
Let's face it: ASCII roxx. All those funny tools that convert pictures
to ASCII art, or even videos to ASCII movies leave me with a chuckle.
globe is not that special, but looking at the surface of the earth
as currently seen from the sun is kind of neat. Also lets you guess
which of your worldwide friends might be awake right now.
Apart from those two new additions, I got the ever fun figlet, which
you all knew and love, which generates ASCII banner art.
.::::::..,:::::: .,:::::: .-::::-.
;;;` `;;;;'''' ;;;;'''';;'```;;;
'[==/[[[[,[[cccc [[cccc ,n[['
''' $$$"""" $$"""" d$P"
88b dP888oo,__ 888oo,__ ""
"YMmMY" """"YUMMM""""YUMMMMM
Figlet offers a lot of different "fonts" to display what you give
it in huge letters combined out of ASCII letters. Indispensable,
what else can I say?
(Get all these from the links given or in OpenBSD directly through the package system.)
16 March 2012
Where to find all the setxkbmap options
They're not in the man page, but you've got them already
When configuring a keyboard on a unixish machine running X (aka X11,
the X Windows System), nowadays the way to configure things is using
setxkbmap(1). There's some good documentation in the manpage, but at
some point the man page tells us only that there are options one can
configure... but not what those possible options are. At first I went on
a broad search through the web, until I figured that the directory path
given at the top of the man page is a strong hint. In fact the list of
predefined options can be found right on the system, in:
/usr/X11R6/share/X11/xkb/rules/base.lst
In that file we first find a list of keyboard "model" names, to be used
like:
setxkbmap -model thinkpad
Next come the keyboard layouts, for various languages and
configurations. These can be combined, like here where I'm using the
Swiss (German) and the US keyboard alternatively - the stuff in the
parenthesis is the "variant" - Swiss keyboards come in German or
French:
setxkbmap 'ch(de),us'
The last big part of the list are all the "options" for setting up
various keys and key combinations. So in my Swiss+US combo, I'm
switching languages using Alt+Space (which is pretty close to how it's
done on a Mac, given that the left Alt key is right next to the space
bar). There are tons of options to switch keyboard layouts. I also have
the Caps Lock key configured as a Control key - since in an X.org setup,
lots of stuff are done with the Ctrl key and I'm never using CapsLock
anyway:
setxkbmap -option 'grp:alt_space_toggle,ctrl:nocaps' 'ch(de),us'
Multiple options are separated by comma. I'm single-quoting them, so no
shell confusion - don't think it's strictly necessary.
There's also tons of options to set up various of the extra keys to be
found on any average PC keyboard (5th level key anyone?) and for stuff
like how to use the numerical keypad. Just have a look, it's right in
your system!
03 April 2012
A Glimpse into the Matrix
In the shell, where else
The Unix shell is the closest thing we have to working with the very structure of the Matrix surrounding us. It's as close to the bare metal as we can get. (Sure, there are people trying to sell you a debugger for that, but what does their debugger run in? Right, thought so.)
Now the OpenBSD ports tree contains a tool that lets you see the Matrix right in your shell window. It's called cmatrix and you can grab it from packages right away. (If your OS doesn't have it in their package manager, you might get it from the one who made it.)
Obviously it shows the Matrix in its encrypted version. It's up to you to see the world in there or whatever you want. Figure it out and you might become someone important... like an actor.
