How to read custom binary files in Tensorflow?

Question

I have a piece of code to read some binary files which are written in a specific format. I want to write a tensorflow version of my read_file() function. Is there any way to do so? I don't want to wrap my function using tf.py_function().

def read_file(filename):
    f = open(filename, 'rb')
    w = int(np.fromfile(f, np.int32, count=1))
    h = int(np.fromfile(f, np.int32, count=1))
    data = np.fromfile(f, np.float32, count=2 * w * h)
    data = np.resize(data, (h, w, 2))
    f.close()
    return data

Scholar · Accepted Answer · 2020-05-25 17:57:53Z

1

You could potentially use tf.data.FixedLengthRecordDataset to read samples encoded with a fixed length of bytes from a binary file.

answered May 25, 2020 at 17:57

Scholar

5125 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Timbus Calin · Accepted Answer · 2020-01-08 08:59:04Z

0

Unfortunately, every python code that you want to use beside the tf* functions must be wrapped inside the tf.py_function().

There is no other way to integrate custom Python code with the TensorFlow specialised functions.

You can find even more details about this at this question: Is there an alternative to tf.py_function() for custom Python code?

answered Jan 8, 2020 at 8:59

Timbus Calin

15.2k6 gold badges49 silver badges69 bronze badges

6 Comments

Ali Salehi Over a year ago

The key function here is np.fromfile(), I couldn't find any Tensorflow alternative for that, are you aware of any? With that, it is easy to convert the whole function.

Timbus Calin Over a year ago

What if use use tf.constant(np.fromfile()) to convert it to Tensor?

Timbus Calin Over a year ago

Curious about the result, never tried it, tell me :D

Ali Salehi Over a year ago

Thanks, but that doesn't work here. I'm using a wrapped version of read_file() to make my tf.data.dataset, and trying to speed up the process wanted to avoid using tf.py_function() by converting the entire code to tensorflow functions. I guess I need to write my own custom tf op.

Timbus Calin Over a year ago

Yes, I know. That's why I put that link to my question, because there was an answer stating there that you cannot do that.

|

Collectives™ on Stack Overflow

How to read custom binary files in Tensorflow?

2 Answers 2

Comments

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related