2

How can I efficiently write a mysql result set to a file in CSV format without loading all of the rows into memory? NodeJS code, please.

0

1 Answer 1

3

This example works with npm packages csv-stringify and either mysql or mysql2 and any Node version that supports the async keyword.

const csvstringify = require('csv-stringify');
const fs = require('fs');

const outputStream = fs.createWriteStream('output.csv', {encoding: 'utf8'});

// Capture events on the outputStream before writing begins
const finishedWriting = new Promise((resolve, reject)=>
  outputStream.on('finished', resolve).on('error', reject));

const connection = __Create a mysql2 Connection object here__
const generator = connection.connection.query('SELECT...');
let recordsProcessed = 0;

try {
  await new Promise((resolve, reject) => {
    // When using a connection pool, the 'error' connection event is called only
    // when enableKeepAlive is true. See:
    // https://github.com/sidorares/node-mysql2/issues/677#issuecomment-588530194
    // Without this handler, this code will hang if the database connection is
    // lost while reading the result set.
    connection.on('error', reject);

    generator
      .on('result', row => ++recordsProcessed) // Counting rows as an example
      .stream({highWaterMark: 10})
      .on('error', reject)
      .on('end', resolve)
      .pipe(csvstringify({header: true}))
      .pipe(outputStream)
      .on('error', error => {
        // Handle stream write error
        // See also https://github.com/sidorares/node-mysql2/issues/664
        // Data is being sent from server to client; without calling destroy, the
        // connection will go back to the pool and will be unusable. The
        // callback provided to destroy() is never called.
        connection.destroy(); // This appears to cause file descriptor leaks
        reject(error);
      });
  });
}
finally {
  connection.on('error', error => console.log(error)); // Remove the handler.
                                                       // Is there a better way?
}

await finishedWriting;
Sign up to request clarification or add additional context in comments.

1 Comment

I can't seem to get this example to work in Node 8.5.1 using csv-stringify 6.1.0. The error I get is "TypeError: stringify is not a function". Anyone else have this issue?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.