0

I am using Python 3.9.12. I have an application where a certain function call will run many times and also has many necessary input arguments. I first coded the function with named inputs (with default values), this was fastest. Then I wanted to "clean up" how the arguments were passed to the function for better organization. For this I tried to use a Python dictionary and a Numpy array. Both of these methods were significantly slower.

Below is a copy of the Python code I am working from:

===============UPDATE========================

It was pointed out by two below that I am calling the helper functions get_par_default() every time unnecessarily so I updated the code and results, same general trend, but times are closer to each other. Seems at the moment that best option is essentially named input arguments and disallow positional argument calls for most of the parameters.

import numpy as np
import time as time

def get_par_default():
    return {'a':0.16, 'b':0.18, 'F0':0.0, 's':0.0, 'eps':3.0e-3, 'V':3.0, 'p0':0.30575896, 'p10':0.48998486,'q0':0.06468597,'q10':0.27093151}

def get_par_arr_default():
    return np.asarray([0.16,0.18,0.0,0.0,3.0e-3,3.0,0.30575896,0.48998486,0.06468597,0.27093151])

def namedargs(Ep,Eq,a=0.16,b=0.18,F0=0.12,s=0.0,eps=3.0e-3,V=3.0,p0=0.5,p10=0.956,q0=0.1,q10=0.306):
    #do some dummy calculation
    ans = Ep*Eq*a*b*F0*s*eps*V*p0*p10*q0*q10
    return ans

def dictargs(Ep,Eq,pars=get_par_default()):
    #do some dummy calculation
    ans = Ep*Eq*pars['a']*pars['b']*pars['F0']*pars['s']*pars['eps']*pars['V']*pars['p0']*pars['p10']*pars['q0']*pars['q10']
    return ans

def nparrargs(Ep,Eq,pars=get_par_arr_default()):
    #do some dummy calculation
    ans = Ep*Eq*pars[0]*pars[1]*pars[2]*pars[3]*pars[4]*pars[5]*pars[6]*pars[7]*pars[8]*pars[9]
    return ans

#the stuff below is so this functionality can be used as a script
########################################################################
if __name__ == "__main__":

    Ep=20.0
    Eq=10.0

    start = time.time()
    for i in range(10000): namedargs(Ep,Eq)
    end = time.time()
    print('Evaluation Time: {:1.5f} sec.'.format(end-start))

    start = time.time()
    args=get_par_default()
    for i in range(10000): dictargs(Ep,Eq,pars=args)
    end = time.time()
    print('Evaluation Time: {:1.5f} sec.'.format(end-start))

    start = time.time()
    args=get_par_arr_default()
    for i in range(10000): nparrargs(Ep,Eq,pars=args)
    end = time.time()
    print('Evaluation Time: {:1.5f} sec.'.format(end-start))

The resulting output when this is run with python argtest.py is:

  Evaluation Time: 0.00251 sec.
  Evaluation Time: 0.00394 sec.
  Evaluation Time: 0.01101 sec.

Why is the version where I pass in arguments as named parameters with default values the fastest by at least 2.6 times faster? Is there a way to retain the excellent organization of the "dictionary" method yet not sacrifice the execution time?

2 Answers 2

3

The timing discrepancy you're observing is because:

  1. You made a simple mistake: You defined a default dict/array for these functions, but then recreated it again for every call rather than letting the default get used, so the defaults didn't save you anything, and recreating the dict/arrays for each call involved extra pointless work.
  2. Even if you weren't recreating the arguments for no reason, accessing function local variables (including parameters), under CPython is, under-the-hood, roughly equivalent to a simple C array access (just loading a value at an offset from a low-level pointer). Accessing your dict or numpy.array has the same fairly trivial local variable lookup overhead (to load pars each time) plus additional overhead to perform a key or index lookup (which is meaningfully more costly than local variable lookup, and you do it many times per call).

The only way to get the dict version to work at the same speed would be to use said dict to dynamically generate the named parameters version of the function (e.g. create the function definition as a string from the dict, then compile+eval or exec it to compile the function up front†). dict lookups are strictly slower than local variable lookups. There are some optimizations in more modern CPython (3.9 is quite old at this point, not sure why you haven't upgraded) that reduce the overhead of read-only access to dicts in specific cases by using a per-op-code cache, but the benefits of that are largely limited to the hidden dicts involved in attribute, global, and built-in lookup, not explicit use of dicts.

I'm not sure why you think it's cleaning up the arguments to have a separate function that makes a dict with the same values, and prevents callers from replacing the defaults individually at call time, instead requiring them to construct and modify a dict or array to pass in. I might suggest making your defaulted arguments keyword-only (there are enough of them that attempts to pass them positionally are likely to make mistakes, and make the calling code harder to maintain when it's not clear what arguments are being passed). To do this, just add a * in the parameter listing, and perhaps break up the arguments visually into related groups per line (a single argument per-line, regardless of grouping, is also a reasonable option), e.g.:

def namedargs(Ep, Eq,
              *,
              a=0.16, b=0.18,
              F0=0.12,
              s=0.0,
              eps=3.0e-3,
              V=3.0,
              p0=0.5, p10=0.956,
              q0=0.1, q10=0.306):

Now callers will have to be explicit about passing a vs. b vs. s, making maintenance easier, and the parameters, while verbose, are fairly readable in the source code.


†Per your request, dynamically generating a function using get_par_default and a template string:

def get_par_default():
    return {'a':0.16, 'b':0.18, 'F0':0.0, 's':0.0, 'eps':3.0e-3, 'V':3.0,
            'p0':0.30575896, 'p10':0.48998486, 'q0':0.06468597, 'q10':0.27093151}

eval(compile(f'''
def dynnamedargs(Ep, Eq, *, {', '.join(f"{k}={v!r}" for k, v in get_par_default().items())}):
    """I only exist thanks to dynamic code-generation"""
    ans = Ep*Eq*a*b*F0*s*eps*V*p0*p10*q0*q10
    return ans
''', __file__, 'exec'))

help(dynnamedargs)

Try it online!

which, by using help to display information about the compiled function, outputs:

Help on function dynnamedargs in module __main__:

dynnamedargs(Ep, Eq, *, a=0.16, b=0.18, F0=0.0, s=0.0, eps=0.003, V=3.0, p0=0.30575896, p10=0.48998486, q0=0.06468597, q10=0.27093151)
    I only exist thanks to dynamic code-generation

demonstrating that the generated function has defaults derived from the dict produced by get_par_default, but is not runtime reliant on such a dict. If you call it, given it's a garbage implementation that multiplies all its arguments, it will always return zero or NaN I believe (and I'm not sure about the latter), but you could pass it non-default values for individual arguments as normal for keyword arguments, and demonstrate it's working normally, e.g.:

print(dynnamedargs(12, 10, F0=1, s=12.4, eps=5))

outputs:

1.6877889687359113

To explain what's happening, I just used a trivial triple-quoted f-string as the template (with the first line blank so I could align the def visually with the rest of template code; if you want the whole string indented because it's being defined in a nested context, inspect.cleandoc is useful for stripping off common leading whitespace to avoid syntax errors), and generated the actual string of arguments and their defaults in it single placeholder. Note that I used !r to coerce the values to their repr form; it's not necessary here, since float and int have identical str and repr forms, but you want to be in the habit to avoid mistakes if one of the arguments has a str form that is "human-friendly" and doesn't constitute valid Python code, e.g. because the default is in fact a str, which wouldn't be quoted or escaped unless converted to repr form; reprs are usually valid Python code by convention.

compile is just converting from string to compiled code, and by passing __file__ it will act as if it were defined in this file. exec mode means it can be composed of complex and/or multiple statements. Once compiled, eval then actually executes the compiled code, defining the function (since I passed no other arguments, it's defined in the current scope, which is global).

Sign up to request clarification or add additional context in comments.

6 Comments

thanks, point taken, question updated. part of the reason (hidden to you) for using the dict is that the parameters will likely have to be updated in the future and there are many functions that use some subset of these parameters. If I use the dict option I can simply add a new parameter and address it only in the function that needs it. I guess I could still do that with your suggestion, so maybe you're just right
@villaa I would also add, it never makes sense to use a numpy.ndarray object like this instead of just using a list, it will be slower always because you are simply retrieving the values and doing math with those value on the python level, not taking advantage of vectorized operations. When you do that, numpy has to box the values into python objects, which creates even more overhead than simply retrieving an object from a list, this is on top of numpy just having more overhead for indexing in general because it have to figure out what to do (numpy supports complicated indexing)
@villaa: If the performance of all these various function is that critical, then dynamically generating the functions is the only way to get that sort of performance. You could get close to it by having a tuple (or named tuple of some form) that is used as the default, unpacking it in the first line of the function (accessing it, either by index or by attribute name, would be slower), but the tuple form would be less self-documenting, and the named tuple approach seems kind of weird when you only ever unpack it for performance reasons (that said, callers could set it up more easily).
@villaa: Something like this, but using your dict of defaults to fill-in the arguments dynamically, rather than it being a static string (if it were a static string, there'd be no purpose). I'll work up a quick example for the answer.
@villaa: Example added to answer.
|
-1

The main reason is that dict lookups involve hashing and potential dynamic checks, which introduce overhead compared to direct mapping.So, the parameters in namedargs have specific types (e.g., a=0.16 implies a is a float), and there is no implicit type conversion happening during the function call.

For nparrargs, using NumPy arrays may involve additional steps compared to directly working with variables, and there might be performance differences due to potential type conversions during array operations.

3 Comments

thanks, is there a workaround which keeps the clean organization?
@villaa I have tested with using dataclass however still dict is better option so far. Maybe u can use hybrid approach
This answer has a kernel of truth to it (the dict lookups are relevant), but it's also got serious mistakes. Python parameters do not have specific types (you can type annotate them, but that's not checked at runtime), only the types that are actually passed, and that's exactly the same for the keyword-argument and dict cases (it's different for numpy, but that's cause numpy is weird in many ways).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.