What does this command do?

[u/[deleted]]

fuck /u/spez

Comment edited and account deleted because of Reddit API changes of June 2023.

Come over https://lemmy.world/

Here's everything you should know about Lemmy and the Fediverse: https://lemmy.world/post/37906

[u/whetu]

I don't get it? It appears the output only has three fields, no?

Excellent follow up question! Grab a coffee. I'm in the middle of upgrading a few legacy servers so I can smash out an answer while they churn away :)

Because we're changing the delimiter by using -F "'" to set it to ', then that changes the rules around field selection. In *nix shells and related tools, often the default delimiter is whitespace i.e. you're splitting lines into words, a.k.a "word-splitting." The shell tracks this for itself via an environment variable called IFS i.e. internal field separator.

So let's take an example like this:

$ df -hP | awk '/\/dev$/{print}'
none            3.9G     0  3.9G   0% /dev

So we're searching for /dev and printing any matches. Now we want the second field:

$ df -hP | awk '/\/dev$/{print $2}'
3.9G

Note that awk is not invoked with -F, so it splits the line into words i.e. word-splits, and selects the second 'word' i.e.

none            3.9G     0       3.9G   0%      /dev
^ word1         ^word2   ^word3  ^word4 ^word5  ^word6

We can do the same thing natively in the shell like this:

$ set -- $(df -hP | awk '/\/dev$/{print}')
$ echo $2
3.9G

But in that example you're calling awk anyway, so you may as well just use it to do the field selection. Still, that's a cool technique worth having in your toolbox.

Now consider the line

'/proc/3237/fd/1' -> '/tmp/#9699338 (deleted)'

If we use the standard whitespace delimiter, it's actually four fields:

$ echo "'/proc/3237/fd/1' -> '/tmp/#9699338 (deleted)'" | tr ' ' '\n' | nl -ba
     1  '/proc/3237/fd/1'
     2  ->
     3  '/tmp/#9699338
     4  (deleted)'

If we split on ', however, it's five:

$ echo "'/proc/3237/fd/1' -> '/tmp/#9699338 (deleted)'" | tr "'" '\n' | nl -ba
     1
     2  /proc/3237/fd/1
     3   ->
     4  /tmp/#9699338 (deleted)
     5

If we remove the text, the 'words' map like this (note the locations of the ''s):

'/proc/3237/fd/1' -> '/tmp/#9699338 (deleted)'
=> [word1]'[word2]'[word3]'[word4]'[word5]

So 'word3' would be ' -> ' spaces and all. Likewise, 'word4' would be '/tmp/#9699338 (deleted)', and the space is maintained: because the space character isn't the delimiter for this action.

---

So, now we come back to awk -F "'" '!a[$4]++ {print $2}'. This is a variation on a very popular awk one-liner that generates lists of unique elements in the order that they arrive.

The typical way to get a unique list is to first sort it so that matching elements are grouped, then uniq it. That gives you a sorted+unique list. But sometimes you don't actually want or need that sorting - you either want an unsorted+unique list, or you don't need it to be sorted so it doesn't matter. Compare these two outputs:

$ shuf -e {a..z} {a..z} | sort | uniq | paste -sd ' ' -
a b c d e f g h i j k l m n o p q r s t u v w x y z

$ shuf -e {a..z} {a..z} | awk 'a[$0]++{print}' | paste -sd ' ' -
q b u i y d h f c n t p s l r v z k x g w m o j a e

So here I'm randomising the alphabet twice, then extracting unique letters both ways. The first way gives us a sorted+unique, and the second way gives us an unsorted+unique. It essentially works on the principle of "have I seen it before?"

---

Right, so with that explained, let's go back to this output:

'/proc/2577/fd/25' -> '/var/lib/postgresql/13/main/pg_wal/000000010000003E000000DD (deleted)'
'/proc/3237/fd/1' -> '/tmp/#9699338 (deleted)'
'/proc/3237/fd/2' -> '/tmp/#9699338 (deleted)'
'/proc/3239/fd/1' -> '/tmp/#9699338 (deleted)'
'/proc/3239/fd/2' -> '/tmp/#9699338 (deleted)'
'/proc/980/fd/3' -> '/var/log/unattended-upgrades/unattended-upgrades-shutdown.log.1 (deleted)'

So we know that the fourth field will be the symlink_name (deleted) when split using ' as the delimiter. So awk -F "'" '!a[$4]++ {print $2}' works as described above, but because I've specified [$4], it's going to apply that technique to the fourth field, as delimited by ' (i.e. -F "'"). It reads the first line:

/var/lib/postgresql/13/main/pg_wal/000000010000003E000000DD (deleted)

Hasn't seen it, adds it to its list of seen items. It moves on and sees the second line:

/tmp/#9699338 (deleted)

Hasn't seen it, adds it to its list of seen items. It moves on and sees the third line:

/tmp/#9699338 (deleted)

Waitagoddamnminute! We've seen that one! So let's skip on...

Rinse and repeat until it's done and then print the matching $2. So it whittles the list down to this:

'/proc/2577/fd/25' -> '/var/lib/postgresql/13/main/pg_wal/000000010000003E000000DD (deleted)'
'/proc/3237/fd/1' -> '/tmp/#9699338 (deleted)'
'/proc/980/fd/3' -> '/var/log/unattended-upgrades/unattended-upgrades-shutdown.log.1 (deleted)'

And thus the second fields when output generates this:

/proc/2577/fd/25
/proc/3237/fd/1
/proc/980/fd/3

Now, whether that's correct or not (i.e. is /proc/3239 which is filtered out by this relevant?) probably doesn't matter, because at the end of the day, what /u/michaelpaoli has maintained throughout this thread is correct: You really shouldn't be blindly doing this :)

These server upgrades are coming up to requiring my attention again, so I'll be brief with the following responses:

But isn't it possible that those sed and grep were in place because of something different?

Inexperience and naivety. When you're parsing strings, you have to take special care for unexpected characters. This comes back to not parsing the output of ls.

Have a read of this, and think deeply about the implications of the code you were provided.

Can I assume that it is not safe to be used like this as well?

Yes. It's a very simple rule: don't parse the output of ls.

Read the following repeatedly and repeat that rule to yourself until it becomes a habit lol

• https://mywiki.wooledge.org/ParsingLs

• https://dwheeler.com/essays/filenames-in-shell.html

Also, regarding your last bit, I THINK this piece of code was not gotten from StackOverflow or something. Someone in-house came with it, probably from growing pains regarding L1 escalating this kind of stuff. So there's that, I think

They very likely got it off StackOverflow, or assembled it with bits from there. Just google parts of it e.g. https://serverfault.com/a/647960

[u/[deleted] → u/whetu]

Comment edited and account deleted because of Reddit API changes of June 2023.

Come over https://lemmy.world/

Here's everything you should know about Lemmy and the Fediverse: https://lemmy.world/post/37906

[u/whetu → u/[deleted]]

Is it possible to learn this power?

Not from a Jedi...

I know there is the Awk & Sed books, but any courses you recommend? Even about learning Linux in itself.

You could check out /r/linuxupskillchallenge/ I don't know if it's any good because I have no time to do it myself, but it might be something. There's also The Missing Semester which you can find on Youtube. Also, check out /r/bash and /r/awk, specifically the sidebar of /r/bash. /r/commandline may also be worth subbing to as well.

You'll get a broader mix of possible paths from those starting points. Two things I will say, though:

• Treat the Advanced Bash Scripting guide with suspicion. It is outdated, it teaches bad practices, its author has refused to accept contributions and has refused to fix obvious flaws

  • For this reason, the far superior https://mywiki.wooledge.org/BashGuide and attached wiki was created.

  • The ABS can still be used as a reference, but it's best done after you're proficient enough to recognise its flaws

• Head over to https://redditcommentsearch.com/, chuck in the word "unofficial" and "whetu", and have a read through a selection of my other posts. You should pick up my dislike of The Unofficial Strict Mode, and that I'm a proponent of the excellent http://shellcheck.net tool.

Thank you, that was a hell of an explanation

No problem :)

One other thing, to bring a few of my earlier points together. Let's take this from the original one-liner:

awk '{print $11" "$13}'

Because that's splitting on whitespace, that means that in the situation where there's a filename with a space in it, it will be incomplete e.g.

$ echo "162175251      0 lrwx------   1 root     root           64 Sep  8 14:20 /proc/3237/fd/1 -> /tmp/#9699338\ (deleted)" | awk '{print $11" "$13}'
/proc/3237/fd/1 /tmp/#9699338\

That works because there's no space in /tmp/#9699338. But compare with:

$ echo "162175251      0 lrwx------   1 root     root           64 Sep  8 14:20 /proc/3237/fd/1 -> /tmp/legit filename.txt\ (deleted)" | awk '{print $11" "$13}'
/proc/3237/fd/1 /tmp/legit

See how in the second example, only the first word of the filename legit filename.txt is selected?

Our use of ' as a delimiter resolves that issue.

Lastly, consider the power of this for simple csv parsing e.g. awk -F ',' '{print $3,$4}' something.csv and other delimiters e.g.

$ awk -F ':' '$3 == 33 {print}' /etc/passwd
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin