This new release focuses on streams by offering two new tools: a way to pipe streams together (and simplify their
use) and a CSV parser.
New version
Just before Christmas vacations, here comes the new version:
- Stories and bugs implemented
- Sources
- Documentation
- NPM package
Release content
Cleaning
A significant effort was put in cleaning the content of the publication repository as it previously contained one plato report folder as well as one documentation folder for each version.
Now everything is consolidated in a single folder and the plato report also includes file history (for instance: regexp.js).
All existing links are adjusted.
gpf.stream.pipe
Combining streams can become tedious when more than two streams must be plugged together. This is greatly simplified using the gpf.stream.pipe helper.
It takes care of chaining the read / write methods and returns a Promise that is resolved when all the data has been processed (including flushable streams).
For example:
// Reading a CSV file
var csvFile = gpf.fs.getFileStorage()
.openTextStream("file.csv", gpf.fs.openFor.reading),
lineAdapter = new gpf.stream.LineAdapter(),
csvParser = new gpf.stream.csv.Parser(),
output = new gpf.stream.WritableArray();
// csvFile -> lineAdapter -> csvParser -> output
gpf.stream.pipe(csvFile, lineAdapter, csvParser, output)
.then(function () {
return output.toArray();
})
.then(function (records) {
// process records
});
gpf.stream.csv.Parser
As demonstrated in the previous example, CSV parsing is now possible using the gpf.stream.csv.Parser class.
Lessons learned
There are still some doubts regarding the mechanics of the gpf.stream.pipe implementation.
In particular, if an intermediate read completes before the whole stream has been processed, a new read must be triggered. Also, error handling is unclear even if fully tested. Finally, flushing a stream does not necessarily means its end. Hence it should be possible to continue writing after a flush.
As these behaviors are not yet clearly defined, the code might require adjustments in the future.
In any case, it demonstrates that some 'real life' examples are required to see how this could simplify (or not) the development.
Next release
The next release content will mostly focus on:
- Finishing the release process improvements
- Introducing attributes that will be later used to simplify CSV generation
- Adding new stream helpers to filter / page batch of records
In parallel, the article on gpf.require.define implementation is still in progress and some additional side projects are popping up.
No comments:
Post a Comment