On 20Jun2017 10:19, bruce badouglas@gmail.com wrote:

Curious as to how to get the following to work remotely over SSH. The cmd is used to remove redundant lines, while maintaining order in the output file.

The following works if I insert it in the remote term. cat /dog/aaa.dat | awk '!a[$0]++' > /dog/aaa.dat_tmp

However I'm unable to get the cmd to work via SSH from the local to the remote. I've tried the following...

ssh crawl_user@67.205.11.111 cat /dog/aaa.dat | awk '!a[$0]++' /dog/aaa.dat_tmp

This runs the cat remotely, fetching the remote file aaa.dat, then awks locally, saving to the local file aaa.dat_tmp.

ssh crawl_user@67.205.11.111 ' cat /dog/aaa.dat | awk '!a[$0]++' > /dog/aaa.dat_tmp '

This runs remotely, but the single quotes do not nest. You're opening some quotes with the cat and closing just after the awk. The awk expression is not quoted, not what you want.

ssh crawl_user@67.205.11.111 " cat /dog/aaa.dat | awk '!a[$0]++' > /dog/aaa.dat_tmp "

This double quotes the whole thing: it all runs remotely, with single quotes getting to the remote shell to quote the awk argument.

ssh crawl_user@67.205.11.111 ' cat /dog/aaa.dat | awk '!a[$0]++' > /dog/aaa.dat_tmp '

Like #2, but with a _local_ backslash to prevent the exclaimation mark being used for local history expansion. So, incorrectly quoted.

ssh crawl_user@67.205.11.111 " cat /dog/aaa.dat | awk '!a[$0]++' > /dog/aaa.dat_tmp "

This doubnle quotes the remote command line as for #3, but in doubnle quotes the slosh gets used up locally.

ssh crawl_user@67.205.11.111 ' cat /dog/aaa.dat | awk '"!"a[$0]++' > /dog/aaa.dat_tmp '

Kind of similar.

ssh crawl_user@67.205.11.111 " cat /dog/aaa.dat | awk '"!"a[$0]++' > /dog/aaa.dat_tmp "

This is in some ways like #2: you're (double) quoting from cat to awk; ah, I see you're quoting the whole thing but exiting the quotes for the exclaimation mark so that you can backslash it. That's actually pretty good.

In some cases, the cmd returns -- bash: !a[$0]++: event not found

Yes. Bash history expansion isn't really part of the shell command language, it of more like an extra macro expansion that happens in the interactive command line _before_ the shell's normal syntax is used.

One easy way to bypass the history expansion is to put your command in a script; histpry expansion is an interactive thing, not active inside a script. So a whole layer of annoyance is removed.

So your tasks are twofold: prevent bash wanting to do a history expansion when it sees the exclaimation mark, and to get the command line intact to the far end.

#7 above is the best of your choices, but you need the awk expression to get into awk cleanly at the far end, so you need quotes at the far end.

I would be separating the construction of the remote command from the ssh call. What you want remotely is what you pasted into the terminal:

cat /dog/aaa.dat | awk '!a[$0]++' >/dog/aaa.dat_tmp

It is generally best to use single quotes if you do not need to embed values in the string, so cut #1 (not yet complete) would be:

shcmd='cat /dog/aaa.dat | awk '!a[$0]++' >/dog/aaa.dat_tmp'

However, you need the single quotes at the far end to prevent the remote shell expanding the $0. I usually _keep_ the outer single quotes and exit the quote just long enough to insert a literal single quote:

shcmd='cat /dog/aaa.dat | awk '''!a[$0]++''' >/dog/aaa.dat_tmp'

That sequence ''' is: - ' exiting the first single quoted stretch - ' putting in a literal single quote character - ' entering single quotes again

Then I'd just use ssh:

ssh crawl_user@67.205.11.111 "$shcmd"

which keeps the logic much clearer.

Cheers, Cameron Simpson cs@zip.com.au