betabug... Sascha Welter

home english | home deutsch | Site Map | Sascha | Kontakt | Pro | Weblog | Wiki

15 January 2007

Date Calculations in Shell Scripts

Keeping 3 days of backups...

Let's say we are dumping a database into a file for backup purposes. We want to keep 3 days of backups around. We could of course rotate the filenames (similar to what newsyslog does: basename.gz, basename.0.gz, basename.1.gz). But we want to backup the directory to a remote host using rsync - and renaming the files would mean to transmit all files with each remote backup. A better naming scheme is to use names with dates, like basename_20070115.gz. That way names don't change and rsync doesn't retransmit yesterday's file. The problem then is that we have to delete backup files older than 3 days. We need a way to calculate dates in a shell script...


The secret sauce in the (partly pseudo) code is that we use the "unix epoch" - seconds since 1970-1-1 - for our calculations. Let's go through the code:

#!/bin/sh
cd /backup/directory

budate=`date '+%Y%m%d'`
                        # today, normal date format, 
                        # backticks to get the at the string
bufilename=basename_`date '+%Y%m%d'`.gz
                        # same thing rolled into filename

# now delete the file from 3 days back, 
# so we have always 3 days of backups
let deletedate=`date '+%s'`-259200
                        # todays date in seconds - 3 days in seconds
                        # key points are using `let' for sh calculations
                        # and using `date' with the %s format string
delfile=basename_`date -r $deletedate '+%Y%m%d'`.gz
rm $delfile

dosomebackupscript -o $bufilename
                        # create the new file

This works by calculating the filename for the file 3 days ago, and deleting that file. After that is done, a different backup script creates the latest file with the same naming scheme. The script runs in a cron job, when it doesn't run on one day, then a stale backup copy is left over. We're not attempting to solve that here.

Posted by betabug at 23:26 | Comments (13) | Trackbacks (0)
ch athens
Life in Athens (Greece) for a foreigner from the other side of the mountains. And with an interest in digital life and the feeling of change in a big city. Multilingual English - German - Greek.
Main blog page
Recent Entries
Splish splash I was taking a vacation (08/18 11:40)
Ferienzeit (07/29 17:09)
Backyard Adventure (07/08 08:18)
Danakos Climbing Gym (06/26 18:40)
Best of
Some of the most sought after posts, judging from access logs and search engine queries.

Apple & Macintosh:
Security & Privacy:
Misc technical:
Athens for tourists and visitors:
Life in general:
<< Longboarding the Coast Line | Main | Literarisches Mischmasch, sogar in Indien >>
Comments
Re: Date Calculations in Shell Scripts

Great tip, it's a common problem on remote backups.

Posted by: r0sk at January 16,2007 14:02
Re: Date Calculations in Shell Scripts

I am trying to implement this strategy and am getting the following syntax error. Please advise:

date: illegal option -- r
usage: date [-u] mmddHHMM[[cc]yy][.SS]
date [-u] [+format]
date -a [-]sss[.fff]

Here is my code:
echo "Deleting Backup from Last night" | tee -a ${spool_file}
let deletedate=`date '+%s'`-86400
delfile=${ORACLE_SID}_`date -r $deletedate '+(%A) %B %e, %Y %H:%M'`
deldir="${HOT_BACKUP}/${delfile}"
rm $deldir
echo "\nEnd Delete Backup Of ${ORACLE_SID}: `date '+(%A) %B %e, %Y %H:%M'` " | \
tee -a ${spool_file}

Posted by: Beverly at March 14,2007 23:01
Re: Date Calculations in Shell Scripts

Beverly, your `date' command doesn't support the -r option. I've heard that older versions of `date' don't. Maybe you can discuss that with your system administrator and see if you can get a newer `date'.

Posted by: betabug at March 15,2007 09:02
Re: Date Calculations in Shell Scripts

Beverly, you date program is fine, but you need to use the -d function like `date -d '3 days ago'` or `date --date='3 days ago'`

Good luck with it!

Posted by: Johan at September 15,2007 15:41
Re: Date Calculations in Shell Scripts

Apparently the available parameters to the `date` program differ quite a bit on various Unix-derived systems. Check the man page of yours what options you have.

Posted by: betabug at November 16,2007 21:50
Re: Date Calculations in Shell Scripts

Very nice and quick example , thanks a lot

Posted by: xmariachi at November 13,2009 12:37
Re: Date Calculations in Shell Scripts

it is pretty wrong to use this.
today is 2009 12 16, and with this code:
saradoc:/$ let deletedate=`date '+%Y%m%d'`-17
saradoc:/$ echo $deletedate
20091199

Posted by: Nick at December 16,2009 15:28
Re: Date Calculations in Shell Scripts

Nick, no where did I tell people to use something wrong like the code you gave there. That would be completely stupid.

Use -d or -r, depending on what your `date` command supports.

Posted by: betabug at December 16,2009 16:11
Re: Date Calculations in Shell Scripts

For gnu's date it seems you need to specify @ infront of those epoch seconds, like so:

let deletedate=`date '+%s'`-259200
echo `date -d @$deletedate '+%Y%m%d'`

Posted by: PhatR at February 24,2010 13:31
Re: Date Calculations in Shell Scripts

Thanks PhatR, your post solved it :-)

Thanks also go to the original poster.

Regards.

Posted by: Top4toP at April 10,2010 19:38
Re: Date Calculations in Shell Scripts

When I execute my shell script I want my script to either pick a date provided by user or automatically pick the current date and apply it to my test results file in the format: "Test_ABC_DDMMYYY". Can anyone provide me any suggestion?

Thanks
Nick

Posted by: Nick at June 22,2012 13:46
Re: Date Calculations in Shell Scripts

Nothing is working for me:
> deletedate=`date '+%s'`-259200

>echo $deletedate
1370758407-259200

>echo `date -d @$deletedate '+%Y%m%d'`
date: illegal option -- d
Usage: date [-u] [+Field Descriptors]


Can someone help?

Posted by: Sumit at June 09,2013 08:22
Re: Date Calculations in Shell Scripts

Sumit, yes, in fact someone can help: you.
Check the man page of your date command on your system. As mentioned before, the options may differ quite a bit. You will have to adapt to what your date command does.

Posted by: betabug at June 09,2013 12:00
Trackbacks
You can trackback to: http://betabug.ch/blogs/ch-athens/524/tbping
There are no trackbacks.
Leave a comment