Correct way to receive json data and write to file using Node.js

Question

I am new to Node and trying to request some json data from a server and save to a file. I can receive the data no problem but can't work out how to write to a file once it has received all the data. Do I need a callback or do I need to use http.CreateServer()? Any help with this would be appreciated.

This is what I have so far:

"use strict";

const request = require('request');
const fs = require('fs');

var options = {
    url: 'url-to-request-data',
    method: 'GET',
    accept: 'application/json',
    json: true,} 
};

// Start the request
request(options, function(error, response, body) {
  if (error) { 
      return console.log(error); 
  } else {
        var path = "~/jsonfile.json";
        fs.writeFile(path, body);
  }
  });

Marcos Casagrande · Accepted Answer · 2018-06-13 23:33:00Z

2

You have several issues.

fs.writeFile takes a third argument, a callback function, where it will notify you of any error, which you're getting.

fs.writeFile(path, body, err => console.error(err));

On *nix systems with that file path, you will get Error: ENOENT: no such file or directory

~ is a bash expansion, node does not know what to do with it.

use:

const os = require('os');
const path = require('path');

const filePath = path.join(os.homedir(), 'jsonfile.json');
fs.writeFile(path, body, err => console.error(err));

Then you will get [Object object] written to ~/jsonfile.json if that URL returns a json as you're clearly requesting one.

You have to solutions:

Remove json: true
Or fs.writeFile(path, JSON.stringify(body), err => /* ... */)

If you are only writing to a file, the best way to go is using streams

const filePath = path.join(os.homedir(), 'jsonfile.json');
request(options).pipe(fs.createWriteStream(filePath));
// See link for error handling

You can read more about request and streams, and how to handle errors in here

edited Jun 13, 2018 at 23:33

answered Jun 13, 2018 at 23:18

Marcos Casagrande

40.7k11 gold badges93 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Patrick Roberts Over a year ago

Thanks for catching the json: true, I didn't notice that until I saw that mentioned in your answer. I changed it in mine accordingly.

ozzyzig Over a year ago

Thanks, I just used ~/ as I didn't want to write the full file path

Patrick Roberts Over a year ago

You could also do path.replace('~', os.homedir())

Patrick Roberts · Accepted Answer · 2018-06-13 23:22:47Z

1

The json data is quite large so is fs.writeFileSync the best way to go? – ozzyzig

No. You should create a Writable Stream for large files, that way you're only buffering chunks of data at a time in memory, rather than the entire file at once. request() returns an object that implements the Stream interface, so you can simply .pipe() it to fs.createWriteStream():

'use strict';

const request = require('request');
const fs = require('fs');

var options = {
  url: 'url-to-request-data',
  method: 'GET',
  accept: 'application/json',
  // json: true,
};
var path = '~/jsonfile.json';
var ws = fs.createWriteStream(path);

// Start the request
request(options).on('error', function (error) {
  console.log(error);
}).on('close', function () {
  console.log('Done');
}).pipe(ws);

edited Jun 13, 2018 at 23:22

answered Jun 13, 2018 at 23:17

Patrick Roberts

52.4k10 gold badges120 silver badges166 bronze badges

Collectives™ on Stack Overflow

Correct way to receive json data and write to file using Node.js

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related