Command Line Interface Ninja Howto
This guide is intended to help get the most out of the command-line interface aka CLI. The shell is a very powerful tool regardless if your using it on Linux or Cygwin on a Windows box. It’s interesting to me when I see people using an interface like Windows Explorer to a multitude of file renames when it could be done in an instant on the command line. I hope that after reading this you’ll be increasingly empowered to instruct the computer do more of the work for you.
Shells
I’m most comfortable with the bash shell. In my career, I’ve debugged some pretty hairy ksh scripts, and a lot of the engineers that I work with seem to prefer ksh. So, ksh may be a more portable scripting language, however, I hold the opinion that bash is a better CLI than ksh. Bash maintains the original history function of the bourne shell (/bin/sh) while providing forward and backwards searches through the history by using <ctrl>-r and <ctrl>-s (eg ^r and ^s). So, it’s very easy to step forward and backwards through your command history which I find to be infinitely useful.
It’s actually pretty irrelevant what shell you use to issue the commands. Most of the utilities that you use on the command line work with any shell. The only differences are conditional and loop constructs which are shell-specific. You can get around this by using find and xargs, perl, or awk and sed instead of a shell construct. Usually it takes an excessively complicated task for me to break-out a loop on the command line (usually 3 or 4 operations per loop iteration). All of the examples in this document are done using bash.
echo $SHELL /bin/bash
By default, most distros have bash and tcsh. So, if you’re running Linux, especially for the first time, I’d suggest staying with bash. These are the shells that are installed by default on Redhat 7.1:
find /bin -name "*sh" -exec ls -l \{\} \;
-rwxr-xr-x 1 root root 512668 Feb 28 2001 /bin/bash
lrwxrwxrwx 1 root root 4 Jul 31 2001 /bin/sh -> bash
-rwxr-xr-x 1 root root 94748 Jan 8 2001 /bin/ash
lrwxrwxrwx 1 root root 3 Jul 31 2001 /bin/bsh -> ash
lrwxrwxrwx 1 root root 4 Jul 31 2001 /bin/csh -> tcsh
-rwxr-xr-x 1 root root 289916 Mar 28 2001 /bin/tcsh
It’s interesting that the Bourne shell /bin/sh is symbolically linked to bash. So, when shell scripts start-up with #!/bin/sh, bash, not a true Bourne shell, is getting invoked. Bash is back-compatible with Bourne, and stands for Bourne Again SHell. So, it’s understandable that it has reached the point where it is completely compatible with Bourne.
All of the subsequent examples were executed under bash.
Using Loops and find with xargs
OK, you want to convert a whole directory full of wav files to 128 kbps mp3 files. How do you do it? Here’s one way:
Example 1:
ls
1.wav 2.wav
for file in *.wav; do lame $file `basename $file .wav`.mp3; done;
Assuming raw pcm input file
LAME version 3.91 MMX (http://www.mp3dev.org/)
CPU features: i387, MMX (ASM used), 3DNow!
Using polyphase lowpass filter, transition band: 15115 Hz - 15648 Hz
Encoding 1.wav to 1.mp3
Encoding as 44.1 kHz 128 kbps j-stereo MPEG-1 Layer III (11x) qval=5
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
0/2 ( 0%)| 0:00/ 0:00| 0:00/ 0:00| 0.0000x| 0:00
average: 128.0 kbps LR: 1 (50.00%) MS: 1 (50.00%)
Writing LAME Tag...done
Assuming raw pcm input file
LAME version 3.91 MMX (http://www.mp3dev.org/)
CPU features: i387, MMX (ASM used), 3DNow!
Using polyphase lowpass filter, transition band: 15115 Hz - 15648 Hz
Encoding 2.wav to 2.mp3
Encoding as 44.1 kHz 128 kbps j-stereo MPEG-1 Layer III (11x) qval=5
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
0/2 ( 0%)| 0:00/ 0:00| 0:00/ 0:00| 0.0000x| 0:00
average: 128.0 kbps LR: 1 (50.00%) MS: 1 (50.00%)
ls
1.wav 1.mp3 2.wav 2.mp3
This single command line will compress a whole directory full of wav files to 128 kbps mp3 files using the lame encoder. Most CLI users don’t realize that it’s possible to use a for loop on a single command line, but it is fairly straight-forward exercise. The basename command removes any leading directory information from the file, and in this case remove the wav suffix from the file. So, the input file is the original file name while the output filename is the original name with a mp3 extension. The example demonstrated here works only with bash because we explicitly used it’s for loop syntax. Consider this command, which accomplishes exactly the same results as the first, but will execute properly independent what shell you are using:
Example 2:
find . -name "*.wav" -exec basename \{\} .wav \; | xargs -i lame \{\}.wav \{\}.mp3
...
This looks intimidating, however, it’s really quite trivial. My problem is that I never remember the loop syntax. So, most of the time I’ll think about how to solve problems using find and xargs. The find command is very intimidating as there are a myriad of options that aren’t very intuitive, but the most common options “-name”, “-type”, and “-exec” are sufficient 95% of the time. So, you can probably understand that the find command is finding all of the wav files in the directory and executing basename on each one of them which circumvents the need for a loop. All of the funny \{\} just indicate where the filename is being inserted into the command. xargs really accomplishes the same thing as an exec, but it needs to be executed after find because the filenames need to be converted to the basename before you start to encode them. So, find passes the basenames of the files into xargs which runs lame on all of them and outputs them to mp3. Lets break the command down into parts here to understand how it works.
Example 3:
find . -name "*.wav" ./1.wav ./2.wav
OK, so find just found the two wav files here. Now lets add the -exec;.
Example 4:
find . -name "*.wav" -exec basename \{\} .wav \;
1
2
The “-exec” is the most non-intuitive option in the find command. You have to remember the \; or find errors out with “find: missing argument to `-exec’”. So *REMEMBER* the “\;” at the end of an -exec statement.
OK, now that I’m off the soapbox, well add the xargs command in, but replace the lame encoder with an echo to illustrate what xargs is actually doing.
Example 5:
find . -name "*.wav" -exec basename \{\} .wav \; | xargs -i echo lame \{\}.wav \{\}.mp3
lame 1.wav 1.mp3
lame 2.wav 2.mp3
The text output to the console was the actual commands that xargs was executing. I encourage you to always
do an echo before piping lots of arguments into xargs, *ESPECIALLY* when you’re removing files. It’s
very easy to send unintended output from a find command. So, be careful.
Advanced find / xargs
The find and xargs commands are basically like their own obscure programming language. The biggest hurdle
in using these commands is remembering the nuances of both find and xargs. Typically you use xargs with no options, or “-i” which basically tells xargs to replace \{\} with current argument. With find, normally you use “-names”, and occasionally “-exec”, “-perm”, and “-type”.
Here’s an example of using find to locate non-executable perl scripts in your home directory and make them executable.
Example 6:
find ~ -name "*.pl" -a ! -perm -go=x -exec chmod go+x \{\} \;
After you make all of your perl scripts executable, you may recall that they need to start with a #!/usr/bin/perl (or #!wherever/perl at your site) for the script to execute properly. When using find “-a” is and, “-o” is or, and ! is not. You can string out ands and ors with find primitives to your hearts content. This command basically says find all the pl files where the group and other do not have their execute bit set.
So, perhaps it’s possible to used a -a in conjunction with an exec to insure that all of the executable scripts contain the proper header. So, we’ll need to take the ! out of the command, and find all of the executable perl scripts. Technically, we know all of the perl scripts are executable now, but lets leave the permission checking in just to be safe.
Example 7:
find . -name "*.pl" -a -perm -go=x -a ! -exec egrep -q '#!/usr/bin/perl' \{\} \; -print
./nipper.pl
./server.pl
There’s a lot to digest in that find command. Recall that -a means logical and, and the ! means not. I pulled a fast one and used exec as a condition. I remembered the egrep returned 0 (good status) if it found something and 1 if it didn’t. So, you can use exec as any other statement. The caveat to that is, find no longer prints the file name when you use an exec. Hence, I tacked on a -print at the end to force the print.
It is evident now that attempting to execute nipper.pl or server.pl will result in an error because neither contain the proper headers. Hmmm, can we use xargs to automagically add the line in??
Example 8:
find . -name "*.pl" -a -perm -go=x -a ! -exec egrep -q '#!/usr/bin/perl' \{\} \; -print | xargs perl -e ' open IF, $ARGV[0]; open OF, ">/tmp/$ARGV[0]"; print OF "#!/usr/bin/perl"; while(<IF>) { print OF $_; } system "mv /tmp/$ARGV[0] $ARGV[0]"; '
Using the xargs command is basically passing the argument into command line perl script here. Notice that I didn’t use the -i option because I only to pass the argument once. I had to resort to a perl script to add the line as xargs can only execute one command at a time. That example got a little complicated, but this is the guide to becoming a CLI Ninja…
This next example shows some of the advanced features of xargs.
Example 9:
for i in `seq 1 10`; do echo line $i; done; line 1 line 2 line 3 line 4 line 5 line 6 line 7 line 8 line 9 line 10 for i in `seq 1 10`; do echo $i; done | xargs -l2 line 1 line 2 line 3 line 4 line 5 line 6 line 7 line 8 line 9 line 10
for i in `seq 1 10`; do echo $i; done | xargs -n3 line 1 line 2 line 3 line 4 line 5 line 6 line 7 line 8 line 9 line 10
The default action of xargs is to echo -l2 basically says echo two lines per line. Using “-n3″ says to display two arguments per line.
Here’s a useful example of grepping on all files in the entire current directory tree for a certain search string. You may wonder about the -n2 passed to xargs. That is there to force grep to print out the filenames. When you give two files as input to grep, it will precede every line with the filename.
Example 10:
find . -name "*.[Cch]" | xargs -n2 egrep "stdlib\.h" /home/hoyhoy/project/nipper/file1.h: #include <stdlib.h> /home/hoyhoy/project/nipper/file2.h: #include <stdlib.h>
Here’s another example of how to use find to see home many music files you have on your hard drive.
Example 11:
find . -iname "*.mp3" | wc -l 8729
wc -l counts the lines. -iname is a case insensitive search. For some reason, I think a smb share or something caused some of the file extensions to be MP3. I think I can find the capitalized MP3 files and rename them.
Example 12:
find . -name "*.MP3" | xargs basename -i \{\} MP3 | xargs -i mv \{\}.MP3 \{\}.mp3
The problem with Example 11 is that it only works in the current directory because basename trucates the leading path and trailing suffix. In Example xx a more robust solution is explored.
Example 13:
find ~/irclogs/ -type f | xargs cat | egrep "http://" | sed "s/.*\(http:[^ ]*\)/\1/" | more
This command will parse all of your irc logs and retrieve all the hyperlinks from them. It’s fun to look at all of the interesting links that are posted when you were asleep or at work. The “-type f” parameter is necessary so as to not send directory names into egrep.
Example 14:
perl -e ' for ($i=1; $i<=7; $i++) { printf "http://etree07.archive.org/etree/lf2002-04-12.shnf/lf2002-04-12d3/lf2002-04-12d3t0%d.shn\n", $i; } ' | xargs -n 1 wget
Retrieving files from the web can sometimes be an arduous chore. Retrieving a large list of files by hand is rather painful and unnecessary, especially if the file names follow some kind of convention. Example 14 downloads a full disc of a live Little Feat show at the etree archives at archive.org. This is a very useful way to go about downloading files that are consecutively numbered. Basically, perl is being used to generate the parameters and then pass them in to the web grabber, wget. Since all of the files at etree are consecutively numbered, it was trivial to generate the urls to send to wget. Wget is an awesomely useful tool. Later in this document, I’ll explore some wget wizardry.
Example 15:
find . \( -perm -u=w -a -name '*.[Chyl]' \) -a ! \( -name '*.tab*' \) -print -exec ci -q -f -u -m"u" \{\} \;
Back in the old days, I used to use RCS quite a bit. I use CVS almost exclusively now, however, if you have a small project it’s rather overkill. The find command here used to be an alias that I used quite frequently to check all of the files in that I had checked out.
Example 16:
sudo find /var/log -mtime 0 -type f -exec ls -l \{\} \;\
| awk ' { print $4" "$9 } '\;
Regularly investigating your log files is typically a good idea. This command shows you any files that have been updated in the past day and what user updated them.
Example 17:
sed "s/[^0-9]*\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1 \2 \3 \1.\2.\3/" data.txt\
| awk ' { if (length($4)>1) print int($1)*256*256+int($2)*256\
+int($3)" "$4; else print int($1)*256*256+int($2)*256" "$3\
} ' > sorted.txt
The previous command takes a file with a list of IP Addresses, and
numerically sorts it (ie 192.168.0.0, 192.168.0.1, …). This command
was actually slightly more complicated than it needed to be. Walter Baeck
from Alcatel, knew some awk trickery to reduce it to.
Example 18:
awk -F. '{print $1*65536+$2*256+$3 " " $0}'&92;
data.txt | sort -n | cut -f 2- -d ' ' >sorted.txt
Someone at work asked me the easiest way to do a global file
replacement across all the files in a directory. I came up with some
crazy awk script that piped the results into sh. That’s way overkill,
perl one-liner. This command replaces foo with bar in every file in every
directory in the current tree and beneath. It also creates backups of
everything due to the risk of screwing up a whole lot of files in one
fell swoop.
Example 19:
find . -type f -exec perl -i -pe 's/foo/bar/' \{\} \;
After you’ve verified you didn’t bork all the files in the current
tree, you can remove all of the .bak files with the following.
Example 20:
find . -name "*.bak" -exec rm -f \{\} \;
