Executing asynchronous calls in a synchronous manner

I've been trying to wrap my head around this issue for the last hours but can't figure it out. I guess I still have to get used to the functional programming style ;)

I wrote a recursive function that traverses through a directory structure and does things to certain files. This functions uses the asynchronous IO methods. Now I want to perform some action when this whole traversing is done.

How would I make sure that this action is performed after all parse calls have been performed but still use the asynchronous IO functions?

var fs = require('fs'),
    path = require('path');

function parse(dir) {
    fs.readdir(dir, function (err, files) {
        if (err) {
            console.error(err);
        } else {                
            // f = filename, p = path
            var each = function (f, p) {
                return function (err, stats) {
                    if (err) {
                        console.error(err);
                    } else {
                        if (stats.isDirectory()) {
                            parse(p)开发者_开发知识库;
                        } else if (stats.isFile()) {
                            // do some stuff
                        }
                    }
                };
            };

            var i;
            for (i = 0; i < files.length; i++) {
                var f = files[i];
                var p = path.join(dir, f);
                fs.stat(p, each(f, p));
            }
        }
    });
}

parse('.');

// do some stuff here when async parse completely finished

Look for Step module. It can chain asynchronous functions calls and pass results from one to another.

You could use async module . Its auto function is awesome . If you have function A() and function B() and function C() . Both function B() and C() depend of function A() that is using value return from function A() . using async module function you could make sure that function B and C will execute only when function A execution is completed .

Ref : https://github.com/caolan/async

async.auto({
            A: functionA(){//code here },
            B: ['A',functionB(){//code here }],
            C: ['A',functionC(){//code here }],
            D: [ 'B','C',functionD(){//code here }]
        }, function (err, results) {
              //results is an array that contains the results of all the function defined and executed by async module
              // if there is an error executing any of the function defined in the async then error will be sent to err  and as soon as err will be produced execution of other function will be terminated
            }
        })
    });

In above example functionB and functionC will execute together once function A execution will be completed . Thus functionB and functionC will be executed simultaneously

functionB: ['A',functionB(){//code here }]

In above line we are passing value return by functionA using 'A'

and functionD will be executed only when functionB and functionC execution will be completed .

if there will be error in any function , then execution of other function will be terminated and below function will be executed .where you could write your logic of success and failure .

function (err, results) {}

On succesfull execution of all function "results" will contain the result of all the functions defined in async.auto

function (err, results) {}

Take a look at modification of your original code which does what you want without async helper libs.

var fs = require('fs'),
    path = require('path');

function do_stuff(name, cb)
{
    console.log(name);
    cb();
}

function parse(dir, cb) {
    fs.readdir(dir, function (err, files) {
        if (err) {
            cb(err);
        } else {             

            // cb_n creates a closure
            // which counts its invocations and calls callback on nth
            var n = files.length;
            var cb_n = function(callback)
            {
                return function() {
                    --n || callback();
                }
            }

            // inside 'each' we have exactly n cb_n(cb) calls
            // when all files and dirs on current level are proccessed, 
            // parent cb is called

            // f = filename, p = path
            var each = function (f, p) {
                return function (err, stats) {
                    if (err) {
                        cb(err);
                    } else {
                        if (stats.isDirectory()) {
                            parse(p, cb_n(cb));
                        } else if (stats.isFile()) {
                            do_stuff(p+f, cb_n(cb));
                            // if do_stuff does not have async 
                            // calls inself it might be easier 
                            // to replace line above with
                            //  do_stuff(p+f); cb_n(cb)();
                        }
                    }
                };
            };

            var i;
            for (i = 0; i < files.length; i++) {
                var f = files[i];
                var p = path.join(dir, f);
                fs.stat(p, each(f, p));
            }
        }
    });
}

parse('.', function()
{
    // do some stuff here when async parse completely finished
    console.log('done!!!');
});

Something like this would work -- basic change to your code is the loop turned into a recursive call that consumes a list until it is done. That makes it possible to add an outer callback (where you can do some processing after the parsing is done).

var fs = require('fs'),
  path = require('path');

function parse(dir, cb) {
    fs.readdir(dir, function (err, files) {
        if (err)
          cb(err);
        else 
          handleFiles(dir, files, cb);
    });
}

function handleFiles(dir, files, cb){
  var file = files.shift();
  if (file){
    var p = path.join(dir, file);
    fs.stat(p, function(err, stats){
      if (err)
        cb(err);
      else{
        if (stats.isDirectory())
          parse(p, function(err){
            if (err)
              cb(err);
            else
              handleFiles(dir, files, cb);
          });
        else if (stats.isFile()){
          console.log(p);
          handleFiles(dir, files, cb);
        }
      }
    })
  } else {
    cb();
  }

}


parse('.', function(err){
  if (err)
    console.error(err);
  else {
    console.log('do something else');
  }
});

See following solution, it uses deferred module:

var fs   = require('fs')
  , join = require('path').join
  , promisify = require('deferred').promisify

  , readdir = promisify(fs.readdir), stat = promisify(fs.stat);

function parse (dir) {
    return readdir(dir).map(function (f) {
        return stat(join(dir, f))(function (stats) {
            if (stats.isDirectory()) {
                return parse(dir);
            } else {
                // do some stuff
            }
        });
    });
};

parse('.').done(function (result) {
    // do some stuff here when async parse completely finished
});

I've been using syncrhonize.js with great success. There's even a pending pull request (which works quite well) to support async functions which have multiple parameters. Far better and easier to use than node-sync imho. Added bonus that it has easy-to-understand and thorough documentation, whereas node-sync does not.

Supports two different methods for wiring up the sync, a defered/await model (like what @Mariusz Nowak was suggesting) and a slimmer though not-as-granular function-target approach. The docs are pretty straightforward for each.

Recommend to use node-seq https://github.com/substack/node-seq

installed by npm.

I'm using it, and I love it..

Look for node-sync, a simple library that allows you to call any asynchronous function in synchronous way. The main benefit is that it uses javascript-native design - Function.prototype.sync function, instead of heavy APIs which you'll need to learn. Also, asynchronous function which was called synchronously through node-sync doesn't blocks the whole process - it blocks only current thread!