Awk ‘ visited[$0]++’ or removing duplicate lines from files explained

Awk ‘ visited[$0]++’ or removing duplicate lines from files explained

To remove the duplicate lines preserving their order in the file use:

The script keeps an associative array with indices equal to the unique lines of the file and values equal to their occurrences. For each line of the file, if the line occurrences are zero then it increases them by one and prints the line, otherwise it just increases the occurrences without printing the line. Piping a bunch of commands we can overcome this issue:

Suppose we have the following file:

prepends the order number in each line.

Source: iridakos.com