Mixing shell variables and python variables in IPython '!command'

Question

Trying to figure out whether this behaviour on IPython (v7.12.0, on Amazon SageMaker) is a bug or I'm missing some proper way / documented constraint...

Say I have some Python variables like:

NODE_VER = "v16.14.2"
NODE_DISTRO = "linux-x64"

These commands both work fine in a notebook:

!echo $PATH
# Shows **contents of system path**
!echo /usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:
# Shows /usr/local/lib/nodejs/node-v16.14.2-linux-x64/bin

...But this does not:

!echo /usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:$PATH
# Shows:
# /usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:**contents of system path**

I've tried a couple of combinations of e.g. using $NODE_VER syntax instead (which produces node--/ instead of node-{NODE_VER}-{NODE_DISTRO}/, but seems like any combination using both shell variables (PATH) and Python variables (NODE_VER/NODE_DISTRO) fails.

Can anybody help me understand why and how to work around it?

My end goal, as you might have guessed already, is to actually add this folder to the PATH rather than just echoing it - something like:

!export PATH=/usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:$PATH

Looks as though any $name that it can't find in its own namespace, disables the substitution. eg !echo {NODE_VER} $foo. I haven't been able to find the code that performs the $ and {} substitutions - but I expect this behavior will be obvious from that. I suspect it's intentional, trying to avoid ambiguities. But I haven't seen it documented. — hpaulj
– hpaulj, Commented Mar 25, 2022 at 4:43

hpaulj · Accepted Answer · 2022-03-25 07:08:28Z

6

How to reference both a python and environment variable in jupyter bash magic?

Try

!echo /usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:$$PATH

$$PATH forces it to use the system variable rather than try to find a Python/local one.

Various examples:

In [130]: foo = 'foo*.txt'
In [131]: HOME = 'myvar'
In [132]: !echo $foo
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt
In [133]: !echo $foo $HOME
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt myvar
In [134]: !echo $foo $$HOME
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt /home/paul
In [135]: !echo $foo $PWD
/home/paul/mypy
In [136]: !echo $foo $$PWD
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt /home/paul/mypy
In [137]: !echo {foo} $PWD
{foo} /home/paul/mypy
In [138]: !echo {foo} $$PWD
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt /home/paul/mypy

Any variable not locally defined forces the behavior you see:

In [139]: !echo $abc

In [140]: !echo {foo} $abc
{foo}

It may put the substitution in a try/except block, and "give up" if there's any NameError.

This substitution can occur in most of the magics, not just !.

edited Mar 25, 2022 at 7:08

answered Mar 25, 2022 at 4:55

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

dingus Over a year ago

Thanks @hpaulj, this works! Just to add for anybody else that comes across the particular use-case mentioned in the question, using this to edit $PATH won't actually persist because each ! command runs in a different subshell so seems to see whatever the original $PATH was.

Collectives™ on Stack Overflow

Mixing shell variables and python variables in IPython '!command'

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related