Thursday, December 21, 2017

Release 0.2.3: Streams and Pipes

This new release focuses on streams by offering two new tools: a way to pipe streams together (and simplify their use) and a CSV parser.

New version

Just before Christmas vacations, here comes the new version:

Release content

Cleaning

A significant effort was put in cleaning the content of the publication repository as it previously contained one plato report folder as well as one documentation folder for each version.

Now everything is consolidated in a single folder and the plato report also includes file history (for instance: regexp.js).

All existing links are adjusted.

gpf.stream.pipe

Combining streams can become tedious when more than two streams must be plugged together. This is greatly simplified using the gpf.stream.pipe helper.

It takes care of chaining the read / write methods and returns a Promise that is resolved when all the data has been processed (including flushable streams).

For example:

// Reading a CSV file var csvFile = gpf.fs.getFileStorage() .openTextStream("file.csv", gpf.fs.openFor.reading), lineAdapter = new gpf.stream.LineAdapter(), csvParser = new gpf.stream.csv.Parser(), output = new gpf.stream.WritableArray(); // csvFile -> lineAdapter -> csvParser -> output gpf.stream.pipe(csvFile, lineAdapter, csvParser, output) .then(function () { return output.toArray(); }) .then(function (records) { // process records });

gpf.stream.csv.Parser

As demonstrated in the previous example, CSV parsing is now possible using the gpf.stream.csv.Parser class.

Lessons learned

There are still some doubts regarding the mechanics of the gpf.stream.pipe implementation.

In particular, if an intermediate read completes before the whole stream has been processed, a new read must be triggered. Also, error handling is unclear even if fully tested. Finally, flushing a stream does not necessarily means its end. Hence it should be possible to continue writing after a flush.

As these behaviors are not yet clearly defined, the code might require adjustments in the future.

In any case, it demonstrates that some 'real life' examples are required to see how this could simplify (or not) the development.

Next release

The next release content will mostly focus on:

  • Finishing the release process improvements
  • Introducing attributes that will be later used to simplify CSV generation
  • Adding new stream helpers to filter / page batch of records

In parallel, the article on gpf.require.define implementation is still in progress and some additional side projects are popping up.