Archive for the 'Command Line' Category

exec

  • english
  • spanish

exec is a built-in shell command that forces a binary to be executed by the currently running shell process instead of forking the process and running the binary on that child process.

When you run a command on a shell-script, it forks a child process and runs the command there. On a syscall level this is the classic:

if( (pid=fork()) == 0) { exec(command); exit(); } wait();

And this is usually what we want, because we will keep running commands after that one. Nevertheless, sometimes this is a problem, like when:

  • we have a program that’s going to monitor a given process, and it doesn’t run properly if there’s an intermediate shell process but we need to run this second process via a shel-script for whatever reasons (to initialize some variables, run the program with nice, whatever)
  • on MacOS we’re running a program via a shell-script and get two icons on the dock, one for the shell and another one for the program

Running a command with exec forces the shell not to fork, but to run the command directly over the shell process. An important thing to note here is that the shell-script will end there, no further commands of the shell script will be executed as the shell process will be substituted by the command process, so to speak.

#!/bin/sh # initialize variables, parse command-line parameters, etc. export IP=$1 exec nice command $*
Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

tac

  • english
  • spanish

A reverse cat!

# cat file 1st line 2nd line 3rd line # tac file 3rd line 2nd line 1st line

Sometimes you need to process a file starting at the end, case in point a log file and you want to go from the last lines up. I remember programming something like this in C some years ago because I didn’t knew this command existed…

Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

rsync –partial

  • english
  • spanish

Everybody knows the rsync command. It’s very useful when you need to keep two machines in sync over the network, minimizing the bandwidth used.

The only problem lies when running it for the first time, or when a new file appears and it’s newly uploaded to the other end: if the network connection is unstable and goes down, rsync deletes the partially uploaded file and it needs to be uploaded again. In other words, when copying new files instead of syncing existing ones, rsync only copies full files.

Luckily there’s a command line parameter to avoid this behaviour and, in the event of a network failure, force rsync to leave the partially-uploaded file as-is and continue syncing it from there on the next run:

# rsync --help ...   --partial keep partially transferred files
Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

cut | sort | uniq: Apache logs

  • english
  • spanish

Many times the reason because a web server is slow and unresponsive is that it’s under “attack”, on purpose or not, by a bot. I’ve seen cases where Google Bot, bots from research engines from universities or some other kind of indexer were responsible for more than half the traffic of a site. These cases are not real DoS attacks, this traffic can be considered legitimate, but the result is that it brings the service down. You can instruct some of these bots not to visit your site so often, like Google Bot using the Google Webmaster Tools and the sitemaps and/or robots.txt files, but usually you can’t and have to consider filtering all this traffic at the firewall. But in any case, the first step is realizing that a single IP (or a couple of them) is responsible for most of your traffic, identifying this IP and using whois learn who it belongs to.

You can run something like this to list the top five IP addresses on your Apache’s access.log:

cut -d" " -f 1 access.log | sort | uniq -c | sort -nr | head -n 5
Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

tar stdin/stdout

  • english
  • spanish

The tar command, like many other UNIX commands, can use stdin/stdout as its input/output target by using “-” instead of a real file.

What’s this useful for? For example, copying a full directory tree while keeping dates, ownerships and permissions, locally or remotely with ssh.

$ tar cf - . | ( cd /some/other/dir ; tar xfv - ) $ tar cf - dir | ssh usr@srv "tar xfv -"
Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

Shell-script: variable default vaule

  • english
  • spanish

It’s possible to assign a default value to a variable in case it isn’t already defined. The syntax is:

FOO=${FOO:=bar}

Example:

$ FOO=${FOO:=bar} $ echo $FOO bar $ BAR=test $ BAR=${BAR:=foo} $ echo $BAR test
Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

watch

  • english
  • spanish

watch runs a command and keeps its output on screen updating it ever X seconds and, optionally, highlighting the differences on the output between executions. It’s an easy way to monitor the output of some command without having to spend several minutes pressing “cursor up - enter”. :-)

watch -n 1 "ps aux | grep apache"

Si te ha interesado, ¿me invitas a una cerveza? / If you liked the post, would you buy me a beer?

Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

Shell-script: substrings

  • english
  • spanish

You can work with substrings in bash using the following syntax:

${VARIABLE:START:LENGTH}

Like in:

$ A=abcdefghi $ echo ${A:3:2} de
Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

cssh: Cluster SSH

  • english
  • spanish

cssh.jpg

Cluster SSH (cssh) is one of mankind’s greatest achievements. Really. Or at least, from a sysadmin point of view. :-) It’s one of those programs that, once you know it and start using it, you wonder how were you able to survive all those dark years without it.

cssh takes a username and several IP addresses on the command line, and opens a SSH session against each of those servers on an independent xterm window. Then, you can click on any of these windows and work independently with that particular server, or go back to the cssh console and write there, and ssh will retransmit each keypress to every SSH session.

This is a very useful tool when working with server farms, clusters, etc. For example lately I’ve been doing some monitoring and maintenance tasks on a 32-server farm on a major ISP: one by one each config modification would have take hours, with cssh it is a matter of minutes. :) Besides, as it replicates every keypress, it even works with text editors like vi: you can edit a file on every server at the same time, navigate through it, modify it… in parallel on every server. You have to be very careful when doing this, though. ;)

Si te ha interesado, ¿me invitas a una cerveza? / If you liked the post, would you buy me a beer?

Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb

Shell-script: timestamp

  • english
  • spanish

A one-liner function that, when piped with the output of another command, prepends each line of that command’s output with a timestamp.

Very useful with commands that output a series of lines periodically but without a timestamp (like vmstat), so that you can’t just send their output to a file and go back to it later without a timeframe.

$ function timestamp { while read l; do d=`date +%H:%M:%S`; echo -e "$d $l"; done; }
$ vmstat 1 | timestamp 12:17:03 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 12:17:03 r b swpd free buff cache si so bi bo in cs us sy id wa 12:17:03 1 0 4 16740 42816 395164 0 0 6 5 174 11 3 1 96 0 12:17:04 0 0 4 16656 42816 395184 0 0 0 0 393 510 1 1 98 0 12:17:05 0 0 4 16656 42816 395184 0 0 0 0 391 781 2 1 98 0 12:17:06 1 0 4 16656 42824 395176 0 0 0 84 462 976 3 1 95 0 12:17:07 0 0 4 16656 42824 395184 0 0 0 0 433 1545 11 3 86 0 12:17:08 0 0 4 16656 42824 395184 0 0 0 0 356 807 1 2 97 0
Enviar a:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • BarraPunto
  • Meneame
  • Slashdot
  • Technorati
  • YahooMyWeb



Creative Commons Attribution-NonCommercial 2.5 Spain
Creative Commons Attribution-NonCommercial 2.5 Spain