Send and receive Buffers via nodejs net Socket is very slow

Question

I try so send nodejs Buffers via node:net Server and Socket. That works fine in principle.

But when I re-create the Buffer on the receiver side using Buffer.concat(), this is really slow. Expected: when doubling the Buffer size, I would expect the process to take roughly twice as long. The "overall speed" (bytes per second) would be roughly constant. Observed: for a 5MB Buffer, it takes 0.1 seconds; for a 50MB, it takes 10 seconds.

It there anything I could do better?

The test code is: (socketSpeed.js)

const { Socket, Server } = require('node:net');

const server = process.argv[2] === 'server'
const client = process.argv[2] === 'client';
const address = process.argv[3];
const port = process.argv[4];
const mbToSend = process.argv[5];

if (server) {
    const serverOpts = {
        keepAlive: true,
        keepAliveInitialDelay: 3000,
    };
    socketServer = new Server(serverOpts);
    socketServer.on('connection', (socket) => {
        console.log('Server connected'); 
        socket.on('data', async (data) => {
            console.log('Server input: ' + data.toString());
        });
        socket.on('error', async (error) => {
            console.log('Server error: ' + error);
        });
        socket.on('end', async () => {
            console.log('Server end reached: ');
        });

        // here: send some data...
        console.log(`Sending ${mbToSend} MB.`);
        socket.write(Buffer.from('x'.repeat( mbToSend * 1024 * 1024))); // 10 MB
        socket.end();
        socketServer.close();
    });
    const options = {
        port: port,
        host: address,
    };
    socketServer.listen(options);


} else if (client) {
    const socketClient = new Socket();
    const options = {
        port: port,
        host: address,
    };

    socketClient.on('connection', (socket) => {
        console.log('Client connected');        
    });
    socketClient.connect(options);

    let dataReceived = 0;
    let allData = undefined;

    const start = Date.now();
    socketClient.on('data', async (data) => {
        dataReceived += data.length;
        // ----------------------------
        // This data collection below using Buffer.concat causes the slow-down!
        // ----------------------------
        if (allData === undefined) allData = data;
        else allData = Buffer.concat([allData, data]);
    });
    socketClient.on('error', async (error) => {
        console.log('Client error: ' + error);
    });
    socketClient.on('end', async () => {
        const end = Date.now();
        const time = (end - start) / 1000;
        
        console.log(`Client end reached, got ${dataReceived / 1024/1024} MB in ${time} seconds.`);
        console.log(` speed: ${dataReceived/1024/1024 / time} MB/s`);
    });

} else {
    console.log('run thes in 2 different terminals:');
    console.log('    node socketSpeed.js server localhost 34567 10)');
    console.log('    node socketSpeed.js client localhost 34567');
}

Just run in 2 shells node socketSpeed.js server localhost 34567 10 (the 10 means, that one 10 MB Buffer is sent from the server to the client) node socketSpeed.js client localhost 34567

Why dont you store all chunks received in a Array, and do a Buffer.concat at the end of the connection, instead of concat every chunk and override "allDate" every time? — Marc
– Marc, Commented Mar 27 at 15:10
Thank you for that suggestion! Nice! (Why did I not think of that?) That should also work, I did not try it. And it would have the real big advantage, that I do not need the final buffer size from the beginning. So this is probably better than my solution. — GerritP
– GerritP, Commented Mar 28 at 12:01

GerritP · Accepted Answer · 2025-03-27 08:51:35Z

0

Found the answer - or a possible answer:

Instead of the expensive

Buffer.concat([allData, data]);

This is much faster:

1.) Pre-alloc a Buffer:

const allData = Buffer.alloc(mbExpected * 1024 * 1024); 
// if the size is too small, I get errors (I assumed auto-expand, but...)

// store the current position in the buffer
let currentPosition = 0;

2.) Fill the data into the buffer:

allData.fill(data, currentPosition, currentPosition + data.length);
currentPosition += data.length;

With this, the code runs a lot faster!

edited Mar 27 at 8:51

answered Mar 27 at 8:01

GerritP

335 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Send and receive Buffers via nodejs net Socket is very slow

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related