GREP SED AWK VI and FIND Usage Notes for Webmasters

by troy · Published October 11, 2014 · Updated October 11, 2014

When it comes to manipulating files, you can do almost anything with GREP SED AWK VI and FIND (and I’m a novice). My notes (taken from far more capable individuals online than I. Thank you all for the education.)

File manipulation using AWK, GREP, SED, FIND, VI

AWK

AWK to extract IPs from logs

Lists of IPs from log file with count

sudo awk '$13 = /spinn/ {print $2}' /var/log/apache2/other_vhosts_access.log | sort | uniq

creates a list of IPs from log file

sudo awk '($(NF-1) = /Ban/){print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort -n

GREP

GERP to see only the lines in a config file that aren’t blanks or don’t start with a #

sudo grep "^[^#]" /etc/squid/squid.conf

GREP to find using OR logic – multiple strings

sudo grep -RPn '(passthru|shell_exec|system|phpinfo|base64_decode|chmod|mkdir|fopen|fclose|readfile) *\(' *

GREP to find a strings – list file names only -l

sudo grep -ril 'eva1fY' *

GREP to find a strings – recursively -R

sudo grep -R 'error_reporting(0)' *

GREP to list files with the current hostname in them

sudo grep -r -l $( hostname) .

How to use OR logic

sudo grep -r "drupal\|joomla\|wordpress" .

-l for file names only

sudo grep -lr "modules" .

GREP to find files with ” IN THE FILE and COPY to folder

sudo cp `grep -l "jose" *` ~/folder/

GERP to find your httpd.conf file – find where Apache is getting its configuration from:

sudo apache2ctl -V | grep SERVER_CONFIG_FILE

SED

find a line in a log file

sudo sed -n '/127.0.0.1/{H;g;p};H' error.log.1|tail -n3
sudo sed -n '1h;2,4 {; H; g; };/127.0.0.1$/p;1,3d;N;D' error.log

GREP in script find a file and delete something in those files

#!/bin/bash
for file in `grep -ril "eva1fYlbakBcVSir" *`
do
	sed -i '$ d' "$file"
done

find (SED) lines matching pattern (IP address) in log files

REPLACE a string AND create a new file

sudo sed 's/string_old/string_new/' filename > newfile

print lines containing a string (IP address in this case)

cat filename | sed -n '/127\.0/p'

delete lines matching pattern

sudo sed '/pattern/d'

FIND and SED

FIND and DELETE (SED) lines matching pattern (IP address) in log files

sudo find *.log -type f -exec sed -i -e '/88\.66\.4\.92/d' {} \;

FIND and edit (SED) a file (and not create a new file)

sudo find . -type f -exec sed -i -e 's/pattern/replace/g' {} \;

FIND

find a file by name #

sudo find ~/ -name 'banner3.jpg' 2>/dev/null

find and copy # combined to make a script to double filter

sudo find /media/backup01/* -name '*.txt' -exec cp {} ~/dump \;

+ increases speed

sudo find . -name '*.py' -exec grep --color 'xrange' {} +

FIND and EXEC (execute another command)

sudo find /var/www/www.example.com/media/ -type f -name "*.jpg" -exec chmod 775 {} \;

FIND and DELETE lines matching pattern in log files

sudo find *.log -type f -exec sed -i -e '/88\.66\.4\.92/d' {} \;

FIND and edit file (not create a new file)

sudo find . -type f -exec sed -i -e 's/pattern/replace/g' {} \;

find all files in /directoryname and copied to another directory /home/username/directory

sudo find /directoryname -iname "*file*.txt" -exec cp {} /home/username/directory \;

useful for upgrading software – list all directories in /var/www THEN copy all files in directory-name to each

sudo find /var/www -maxdepth 1 -type d | cp -Rf ~/directory-name/* '*'

FIND directories 1 level deep and execute a script on them

sudo find -maxdepth 1 -type d | ~/close-wp.sh '*'

FIND directories 1 level deep and move them to another directory

sudo find -maxdepth 1 -type d | mv '*' /home/username/directory-name

FIND and DELETE directories

sudo find /media/ -type d -name 'directory-name' -exec rm -r {} \;

FIND and DELETE images larger than x

sudo find /var/www/www.example.com/wp-content/uploads/ -type f -size +100k -exec rm -r {} \;

FIND and DELETE files older than 5 days | -mtime = days

sudo find /path/to/files* -mtime +5 -exec rm {} \;

FIND and DELETE files with confirmation

sudo find . -type f -name "*.jpg" -exec rm -i {} \;

Misc VI commands

correct replace command in vi

:.,$s/up/right/

now edit that same set of filenames

sudo vi $( grep -r -l $(hostname) . )

delete all matching lines starting with foo

:g/.*foo.*/d