0

I'm trying to send a stream of data via socket in Python. So far I manage to create a dummy_data_gen.py which sends a line containing 4 floats to the server.py. However, I'm still having issues in the stability of the all setup.

server.py:

import sys
import time
import socket
import numpy as np

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Bind the socket to the port
server_address = ('localhost', 5002)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)

# Listen for incoming connections
sock.listen(1)

# Create a list to store the incoming data
data = []

while True:

    # Wait for a connection
    print >>sys.stderr, 'waiting for a connection'
    connection, client_address = sock.accept()

    try:
        print >>sys.stderr, 'connection from', client_address

        while True:

            incoming_data = connection.recv(48).split(',')
            print incoming_data

            event = float(incoming_data[0]), float(incoming_data[1]), float(incoming_data[2]), float(incoming_data[3])

            data += [event]

            time.sleep(0.01)

    finally:
        # Clean up the connection
        connection.close()

dummy_data_gen.py

import sys
import time
import socket

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Connect the socket to the port where the server is listening
server_address = ('localhost', 5002)
sock.connect(server_address)

file = '../data/myfile.txt'

# Simulating a real-time data stream at 100 Hz
try:
    with open(file) as f:
        for line in f:
            sock.sendall(line)
            time.sleep(0.01)

finally:
    print >>sys.stderr, 'closing socket'
    sock.close()

My problem is that sometimes the communication is working properly, however, I have situations where I receive more data per line than I should. In the following output example the first 7 lines are correct, however the following lines are incorrect and therefore problematic:

['391910745379', '24.134277', '-1.9487305', '-117.373535', '\n']
['391920745379', '24.434082', '-1.3491211', '-117.373535', '\n']
['391930745379', '23.68457', '-0.5996094', '-116.62402', '\n']
['391940745379', '24.434082', '-1.0493164', '-115.57471', '\n']
['391950745379', '24.134277', '-1.0493164', '-116.47412', '\n']
['391960745379', '23.234863', '-1.0493164', '-116.47412', '\n']
['391970745379', '24.583984', '-0.89941406', '-116.92383', '\n']
['391980745379', '23.384766', '-0.2998047', '-116.62402', '\n39']
['1990745379', '23.68457', '-0.5996094', '-115.72461', '\n39200']
['0745379', '23.834473', '-0.44970703', '-115.87451', '\n392010']
['745379', '23.534668', '-1.0493164', '-114.9751', '\n392020745']
['379', '23.384766', '-1.7988281', '-115.72461', '\n39203074537']
['9', '22.935059', '-0.44970703', '-114.9751', '\n392040745379', '']

I tried to play with the connection.recv bytes but I'm still facing this issue.

EDIT1: Following some suggestions I modified the server.py as follows:

del_message = '\n'
del_stream = ','

while True:
        _buffer += connection.recv(1)
        if del_message in _buffer:
            incoming_data = _buffer.split(del_stream)
            event = float(incoming_data[0]), \
                        float(incoming_data[1]), \
                            float(incoming_data[2]), \
                                float(incoming_data[3])

This approach seems to solve my issue, however the performance is extremely slow. My files contains approximately 6300 lines that were actually sent in 70 seconds (time interval at which the socket was closed on my dummy data generator). However, I took almost 10 minutes to receive all of the 6300 lines. It seems also that I receive more samples per second on the beginning rather that on the end of the stream.

3
  • Can you specify what's incorrect about the lines /problematic? Commented Oct 27, 2015 at 21:45
  • 1
    @inbinder The problem is that the stream of data being sent is composed by 4 elements. You can see on the last line for instance that we have 5 elements for instance. Furthermore, on line 9 it seems that some elements in from future data stream sending is present on the last item of the list Commented Oct 27, 2015 at 21:53
  • @dudas The stream of data being sent is composed of a stream of elements. Commented Oct 28, 2015 at 7:49

1 Answer 1

1

If you have a message protocol that terminates messages with a newline, you need to write some code to implement that protocol. It won't work by magic.

You need a "receive a message" function, where "message" is defined as "a sequence of bytes delimited by a newline". You've never written any such function, so you're not receiving messages but just the chunks of bytes you're sending.

Sign up to request clarification or add additional context in comments.

9 Comments

What do you mean by "implementing that protocol." stripping off the "\n"?
No. He needs a "receive a message" function, where "message" is defined as "a sequence of bytes delimited by a newline". He never wrote any such function. Yet he's expecting to receive messages.
@DavidSchwartz Thanks for your help. However I think I didn't fully understand of how that could help in situations where I have some information after the newline. Additionally, can you give me an example of how to establish a protocol?
@dudas There is no "information after the newline". By definition, the message ends with a newline. You're seeing information after the newline because you never wrote a function to receive a message. Write one. You're calling connection.recv expecting it to understand what you mean by "message". How could it know?
@DavidSchwatrz Sorry, I dont't want to pass as a lazy person, but I'm honestly still get confused of what function should I write. Can you elaborate on that please?
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.