17

Good day,

I've searched for this but haven't come up with any responses. I wish to send a multi dimensional numpy array over a socket. Hence, I decided to convert it to a string:

However, it destroys the representation of the array:

>>> import numpy as np
>>> x = np.array([[0, 1], [2, 3]])
>>> xstring = x.tostring()
>>> print xstring

>>> print x
[[0 1]
 [2 3]]
>>> print xstring

>>> nparr = np.fromstring(xstring, dtype=np.uint8)
>>> print nparr
[0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0]

Is there anyway I can get the conversion to string to somehow, save the dimension of it?

Improve this question
1
  • 1
    You're almost doing it right - first - your dtype is not np.uint8 - this loads the buffer as bytes (and it is serialized from int64). You need to use the correct dtype in fromstring, and then .reshape() the result to the original shape - in your case (2, 2). If you're sending it e.g. over a socket, the shape needs to be provided separately. Commented Apr 12, 2018 at 12:25

4 Answers 4

Reset to default
11

Try this example:- 

import socket
import numpy as np
from cStringIO import StringIO

class numpysocket():
    def __init__(self):
        pass

    @staticmethod
    def startServer():
        port=7555
        server_socket=socket.socket() 
        server_socket.bind(('',port))
        server_socket.listen(1)
        print 'waiting for a connection...'
        client_connection,client_address=server_socket.accept()
        print 'connected to ',client_address[0]
        ultimate_buffer=''
        while True:
            receiving_buffer = client_connection.recv(1024)
            if not receiving_buffer: break
            ultimate_buffer+= receiving_buffer
            print '-',
        final_image=np.load(StringIO(ultimate_buffer))['frame']
        client_connection.close()
        server_socket.close()
        print '\nframe received'
        return final_image

    @staticmethod
    def startClient(server_address,image):
        if not isinstance(image,np.ndarray):
            print 'not a valid numpy image'
            return
        client_socket=socket.socket()
        port=7555
        try:
            client_socket.connect((server_address, port))
            print 'Connected to %s on port %s' % (server_address, port)
        except socket.error,e:
            print 'Connection to %s on port %s failed: %s' % (server_address, port, e)
            return
        f = StringIO()
        np.savez_compressed(f,frame=image)
        f.seek(0)
        out = f.read()
        client_socket.sendall(out)
        client_socket.shutdown(1)
        client_socket.close()
        print 'image sent'
        pass

In this model client sends multidimensional ndarray to server. There are two functions startServer() and startClient(). startServer takes no arguments but startClient needs server address as well as the ndarray as arguments. First start Server and then start client. Server starts reading from buffer only after receiving the shutdown message from client.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

4

Indeed, .tostring only returns the raw data. This means you additionally need to send the shape and dtype of the array if these aren't known on the other side.

Maybe it's easier to serialize the array using Pickle:

import numpy as np
from cPickle import dumps, loads

x = np.array([[1, 2],[3, 4]], np.uint8)
print loads(dumps(x))
# [[1 2]
#  [3 4]]

Though for very small arrays the size overhead could be significant:

print len(x.tostring()), len(dumps(x))
# 4 171

For more on using Pickle, see here.

Improve this answer

3 Comments

Pickle was extremely slow, and made my 2d array massive. It also occasionally crashed my program with some keyerror exceptions
@ovfstack: You're right. On Python 3 the overhead is negligible for large arrays and it runs only twice as slow as .tostring(). For Python 2 you should see a great improvement if you specify protocol=2, though it still seems to be 4 times slower than .tostring() in this case.
I just used the imencode function that opencv provides. It was also able to "compress" my array and imdecode was able to recover it with decent accuracy
0

This is a slightly improvised answer for ajsp answer using XML-RPC.

On the server-side when you convert the data, convert the numpy data to a string using the '.tostring()' method. This encodes the numpy ndarray as bytes string. On the client-side when you receive the data decode it using '.fromstring()' method. I wrote two simple functions for this. Hope this is helpful.

  1. ndarray2str -- Converts numpy ndarray to bytes string.
  2. str2ndarray -- Converts binary str back to numpy ndarray.
    def ndarray2str(a):
        # Convert the numpy array to string 
        a = a.tostring()

        return a

On the receiver side, the data is received as a 'xmlrpc.client.Binary' object. You need to access the data using '.data'.

    def str2ndarray(a):
        # Specify your data type, mine is numpy float64 type, so I am specifying it as np.float64
        a = np.fromstring(a.data, dtype=np.float64)
        a = np.reshape(a, new_shape)

        return a

Note: Only problem with this approach is that XML-RPC is very slow while sending large numpy arrays. It took me around 4 secs to send and receive a (10, 500, 500, 3) size numpy array for me.

I am using python 3.7.4.

Improve this answer

Comments

0

I recently wanted to do a similar thing and tried to do everything on my own which takes up a lot of time and efforts. Turns out, a library already exists in python that does everything for you (maintains shapes and abstracts the tedious socket code)!

Check out - https://pypi.org/project/mlsocket/

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.