Source code for an eval'ed/dynamically generated function in Python

Question

I am looking for a way to create a Python function from a string containing the source code at runtime, in such a way that the source code is available by inspection.

My current method looks like:

src = 'def foo(x, y):' + '\n\t' + 'return x / y'
g = {numpy: numpy, ...}  # Modules and such required for function
l = {}
exec(src, g, l)
func = l['foo']

Which works perfectly fine, but the function has no source code/file associated with it. This makes debugging difficult (note the line the error occurs on in foo() isn't shown):

>>> foo(1, 0)
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-85-9df128c5d862> in <module>()
----> 1 myfunc(3, 0)

<string> in foo(x, y)

ZeroDivisionError: division by zero

If I define a function in the IPython interpreter, I can get the source using inspect.getsource and it will be printed in tracebacks. inspect.getsourcefile returns something like '<ipython-input-19-8efed6025c6f>' for these types of functions, which of course isn't a real file. Is there a way to do something similar in a non-interactive environment?

inspect.getsource does not work for functions defined in a standard interactive session. If it seemed to work when you tried it, you may have been running IPython without realizing it. — user2357112
– user2357112, Commented Jun 12, 2018 at 17:29
shouldn't You use compile ? . With compile You can first store the compiled code and then exec it when needed. where exec and eval makes evaluation in place without returning objects — Take_Care_
– Take_Care_, Commented Jun 12, 2018 at 20:50
@user2357112 you're correct, I did it in IPython and incorrectly assumed it also worked in the standard interactive interpreter. — JaredL
– JaredL, Commented Jun 13, 2018 at 21:29

JaredL · Accepted Answer · 2018-06-18 19:23:59Z

4

So I was able to partially figure this out by digging into the IPython source code. It makes use of the builtin module linecache, which contains functions for reading source code from files and caching the results. The inspect and traceback modules both use this module to get the source of a function.

The solution is to create the function the same way as in the question, but using compile with a made-up and unique file name:

source = 'def foo(x, y):' + '\n\t' + 'return x / y'

filename = '<dynamic-123456>'  # Angle brackets may be required?
code = compile(source, filename, 'exec')

g = {numpy: numpy, ...}  # Modules and such required for function
l = {}
exec(src, g, l)
func = l['foo']

linecache contains a variable cache which is a dictionary mapping file names to (size, mtime, lines, fullname) tuples. You can simply add an entry for the fake file name:

lines = [line + '\n' for line in source.splitlines()]

import linecache
linecache.cache[filename] = (len(source), None, lines, filename)

The function will then work with inspect.getsource(), IPython's ?/?? syntax, and in IPython tracebacks. However, it still doesn't seem to work in built-in tracebacks. This is mostly sufficient for me because I almost always work in IPython.

EDIT: see user2357112's comment below for how to get this working with traceback printing in the builtin interpreter.

edited Jun 18, 2018 at 19:23

answered Jun 16, 2018 at 7:37

JaredL

1,4732 gold badges13 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2357112 Over a year ago

If you want this to take effect in a regular (non-IPython) interactive session, installing traceback.print_exception as the traceback printer in place of the default sys.excepthook should do it.

Collectives™ on Stack Overflow

Source code for an eval'ed/dynamically generated function in Python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related