Cancel

AWK command in linux

awk is a text processing command like search, replace, and manipulate text. It allows you to use variables, functions, and operators. Some Engineers said it is another programming language.

Options are the real power of the awk command. Here I’ll cover the most common options.

  • -F <delimiter> input field separator or delimiter.
  • -v name=key assign multiple variables.
  • ; separate two or more actions.
  • print returns all the lines.
  • $0 return first line.
  • $1 return first field.
  • $NF return last field.
  • $FS return field separator, default is whitespace.
  • NR returns a total number of lines.

I have created dummy content for you to test all the above options.

1
2
3
4
5
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync

Get options listed above and put them one by one to the following structure.

You have 3 options to run the awk command. In this article, I will be using option 1 most of the time but you are free to choose other options as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# option 1
awk [options] 'selection {action_or_function_or_condition}' filename.txt

# option 2
cat filename.txt | awk [options] 'selection {action_or_function_or_condition}'

# option 3
<ANY_LINUX_COMMAND> | awk [options] # so on...

# to execute BEGIN code first.
# we use this for creating functions or setting conditions or creating variables or etc.
awk 'BEGIN { YOUR_CODE_WILL_BE_HERE }' filename.txt

# to execute between BEGIN and END code, this option we normally use every time.
awk '{}' filename.txt

# to execute END code at the end of the file.
awk 'END {}' filename.txt

To return the first field of all lines using a colon as a field separator.

1
awk -F: '{print $1}' filename.txt

To return the last field of all lines.

1
awk -F: '{print $NF}' filename.txt

To return the total number of lines.

1
awk 'END {print NR}' filename.txt

To return 3 to 5 records.

1
awk -F: 'NR==3, NR==5' filename.txt

To return even records.

1
awk -F: 'NR % 2 == 0' filename.txt

To return the line of matched pattern.

1
awk '/root/ {print $1}' filename.txt

To return the line of matched pattern, find only between 1 and 3 lines.

1
awk '/root/ {print $1,$3}' filename.txt

awk and bash scripting both have similar operators, array, dictionary, if-else statement, case statement, loops and etc. read this article.

Once you read the article mentioned above, You can add any features inside the BEGIN section as mentioned in the following example.

1
2
3
4
5
6
7
# dictionary
awk 'BEGIN {
  fruits["mango"] = "yellow";
  fruits["orange"] = "orange"
  print fruits["orange"]
  print fruits["mango"]
}'

awk provides built-in functions too.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# length of string
awk '{print length("hello")}' filename.txt # ==>> 5
# to uppercase
awk '{print toupper("hello")}' filename.txt # ==>> HELLO
# to lowercase
awk '{print tolower("HELLO")}' filename.txt # ==>> hello
# to return x number of characters
awk '{print substr("hello", 1, 3)}' filename.txt # ==>> hel
# to convert x to int
awk '{print int(12.350)}' filename.txt # ==>> 12
# to find x and replace all
awk 'BEGIN {
  str = "One Two Three Four Five"
  print "before = " str

  gsub("Four", "4", str)
  print "after = " str
}' filename.txt

To create a custom function. It must be created outside the BEGIN section otherwise you will end up with a lot of errors.

1
2
3
4
5
6
7
8
9
10
11
12
awk '
  function min(n1, n2) {
    if (n1 < n2)
      return "smaller than " n2;
    return "larger than " n2;
  }

  # now you can call the function
  BEGIN {
    print min(10, 60)
  }
'

To create a custom header using formatted string.

1
2
3
4
5
6
7
8
9
awk -F: 'BEGIN {
    # add header,
    # if you dont want to keep,
    # then remove only BEGIN section.
    printf "%-10s %s\n", "User", "Home"
    printf "%-10s %s\n", "----", "----"
  }
  { printf "%-10s %s\n", $1, $(NF-1) }
' filename.txt