`AttributeError: 'Tensor' object has no attribute 'numpy'` tracking intermediate values in custom call method

Question

Basically I have a custom call method that computes some interesting intermediate values that are important enough to track when training, but not important enough to provide as an additional return value. My model looks something like

class MyModel(tf.keras.Model):
    def __init__(self, *args, **kwargs):
        self.intermediate_values: List[numpy.ndarray] = []
        super().__init__(*args, **kwargs)


    def call(self, inputs, training=False):

        # Do some linear algebra with inputs and different layers etc.

        intermediate_tensor = some_intermediate_result

        if training and some_other_condition:
            self.intermediate_values.append(intermediate_tensor.numpy())

        # Do some more stuff with intermediate_tensor and different layers etc.

        return final_result

This of course throws AttributeError: 'Tensor' object has no attribute 'numpy' since, as I understand it, evaluating to numpy is only applicable to "eager tensors" and here intermediate_tensor is technically a node in the model call graph(?or whatever the correct nomenclature is).

Directly storing a copy of intermediate_tensor also of course doesn't solve the problem since its still not an eager tensor.

A callback won't work since this in right in the middle of the call method.

Maybe a custom metric class that gets set/managed entirely internally in the MyModel class?

I just can't seem to figure out how to actually extract the values for later analysis when the graph is executed.

stackoverflow.com/questions/63869134/… this could be one solution if the drawbacks are ok for you. — mhenning
– mhenning, Commented Jul 22, 2023 at 22:42
@mhenning thanks for the link, but yeah just executing the whole thing in eager mode is kind of a non-starter. Again its not necessarily required to be numpy, just some way--any way-- of getting those values out mid-call is what I need — charlestoncrabb
– charlestoncrabb, Commented Jul 22, 2023 at 23:04
I've never dealt with this challenge before, it seems very non-trivial. github.com/tensorflow/tensorflow/issues/… was an interesting read about non-eager Tensor behavior. If TensorFlow can't give you the functionality you need, there is also always PyTorch, where this would be really easy to implement, I think. — mhenning
– mhenning, Commented Jul 23, 2023 at 0:16
You always have the option of breaking away from model.fit and writing your own training loop, which gives you a lot more control over the details. — xdurch0
– xdurch0, Commented Jul 23, 2023 at 6:18
@xdurch0 yeah one thing I didn't mention is that the train_step method is also overridden, I'm not sure how moving the problem to the train step will help though — charlestoncrabb
– charlestoncrabb, Commented Jul 23, 2023 at 22:20

Faisal Shahbaz · Accepted Answer · 2023-07-28 13:04:07Z

Try to resolve your mention error. AttributeError: 'Tensor' object has no attribute 'numpy',

Create the convert_np_function function and change tensor to a numpy arr and adds into the self.intermediate_values list. tf.py_function is used to build a tensorflow operation that call for custom function during graph execution. The tf.numpy_function handles the conversion between tensorflow tensors and numpy arr inside the custom function.

import tensorflow as tf
import numpy as np

class MyModel(tf.keras.Model):
    def __init__(self, *args, **kwargs):
        self.intermediate_values = []
        super().__init__(*args, **kwargs)

    def call(self, inputs, training=False):

        intermediate_tensor = some_intermediate_result

        if training and some_other_condition:
            py_func = lambda t: tf.py_function(self.convert_np_function, [t], [])
            tf.numpy_function(py_func, [intermediate_tensor], [])

        return final_result

    def convert_np_function(self, intermediate_tensor):
        intermediate_value = intermediate_tensor.numpy()
        self.intermediate_values.append(intermediate_value)

Collectives™ on Stack Overflow

`AttributeError: 'Tensor' object has no attribute 'numpy'` tracking intermediate values in custom call method

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related