javascript - Node: fs write() doesn't write inside loop. Why not?

Question

I want to create a write stream and write to it as my data comes in. However, I am able to create the file but nothing is written to it. Eventually, the process runs out of memory.

The problem, I've discovered is that I'm calling write() whilst inside a loop.

Here's a simple example:

'use strict'

var fs = require('fs');
var wstream = fs.createWriteStream('myOutput.txt');

for (var i = 0; i < 10000000000; i++) {
    wstream.write(i+'
');
}

console.log('End!')
wstream.end();

Nothing ever gets written, not even hello. But why? How can I write to the file within a loop?

Answer 1

The problem is that you aren't ever giving it a chance to drain the buffer. Eventually this buffer gets full and you run out of memory.

WriteStream.write returns a boolean value indicating if the data was successfully written to disk. If the data was not successfully written, you should wait for the drain event, which indicates the buffer has been drained.

Here's one way of writing your code which utilizes the return value of write and the drain event:

'use strict'

var fs = require('fs');
var wstream = fs.createWriteStream('myOutput.txt');

function writeToStream(i) {
  for (; i < 10000000000; i++) {
    if (!wstream.write(i + '
')) {
      // Wait for it to drain then start writing data from where we left off
      wstream.once('drain', function() {
        writeToStream(i + 1);
      });
      return;
    }
  }
  console.log('End!')
  wstream.end();
}

writeToStream(0);