A hidden feature: Total.js Chunker

Thu Feb 02 2017 21:55:05 GMT+0100 (Central European Standard Time), Peter Širka

A hidden feature: Total.js Chunker

Total.js Chunker is a very useful helper for importing data from larger data-sources. Chunker and U.streamer() are very big canons and these methods can save a lot of time.

  • Chunker is optimized for heavy and real-time writes
  • Chunker creates multiple files with a specific number of the records
  • Chunker streams all prepared records by pages
  • Chunker documentation

A simple example:

// U.chunker(name, [items_per_page])
var chunker = U.chunker('products', 50);
// Filenames will be in this form: /tmp/products1.json, /tmp/products100.json

// We have a big data stream
somestream.on('data', U.streamer('<product>', '</product>', function(value, index) {
    var product = value.parseXML();

    // Chunker creates multiple files with a specific number of the "chunks" (in our case: 50 products).
    // chunker.write(object);
    chunker.write(product);

}));

// No more data, let's process the files chunker created
somestream.on('end', function() {

    // Flushes last open file
    chunker.end();

    // Reads each file
    chunker.each(function(items, next) {
        // items === 50 products
        // here we can update DB
        // here we have a lot of time :)

        // ....
        // ....

        // Read next items
        next();

    }, function() {
        // This method is called when the chunker doesn't contain any more items (optional).
        // Cleans up hdd and memory
        chunker.destroy();
    });

    // Reads items by page number
    // chunker.read(page, function(err, items) {
    //     items === products (max 50)
    // });

    // Deletes created files
    // chunker.clear();

    // Calls chunker.clear(); internally and deletes in-memory data
    // chunker.destroy();

});

Where to use Chunker?

I use Chunker when I perform imports from different data-sources e.g. XML into my e-commerces. It's very useful helper optimized for low CPU load.


Tags

Follow us

Latest blogs
Flow v6 is here!
Thu Mar 07 2019 11:53:54 GMT+0100 (Central European Standard Time)
A critical security fix
Wed Feb 13 2019 22:15:39 GMT+0100 (Central European Standard Time)
New release: Total.js v3.2
Wed Feb 13 2019 22:14:39 GMT+0100 (Central European Standard Time)
Total.js Wiki v2
Fri Jan 04 2019 22:15:01 GMT+0100 (Central European Standard Time)
Total Year 2018
Thu Jan 03 2019 21:14:00 GMT+0100 (Central European Standard Time)

Latest comments
Nice tip
Mauro Junior
Thu Sep 20 2018 21:41:02 GMT+0200 (Central European Summer Time)
Not only for Total.js. You can communicate with different websocket servers.
Peter Širka
Mon Apr 23 2018 20:08:20 GMT+0200 (Central European Summer Time)
Marko: you need to create a buffer with this codepage and write byte-to-byte string. I recommend ...
Peter Širka
Mon Apr 23 2018 20:06:21 GMT+0200 (Central European Summer Time)
Is WEBSOCKETCLIENT only for internal ws connections between totaljs apps?
Stelios Stephanua
Fri Mar 16 2018 06:04:22 GMT+0100 (Central European Standard Time)
Total.js is amazing! ;)
Leonardo Hessel
Tue Dec 19 2017 19:51:15 GMT+0100 (Central European Standard Time)

Pixabay


Read more

Flow v6 is here!

News: We have released a new version of Flow. This version brings great new features and UI improvements.

Thu Mar 07 2019 11:53:54 GMT+0100 (Central European Standard Time)
A critical security fix

News: We were notified about the critical security bug in Total.js framework. Read a prevention.

Wed Feb 13 2019 22:15:39 GMT+0100 (Central European Standard Time)
New release: Total.js v3.2

News: This new release brings a critical security fix and small new improvements. Update Total.js now.

Wed Feb 13 2019 22:14:39 GMT+0100 (Central European Standard Time)
Total.js Wiki v2

Products: I have released a new version of Total.js Wiki. New version brings new improvements.

Fri Jan 04 2019 22:15:01 GMT+0100 (Central European Standard Time)
Total Year 2018

Business: Last year was perfect for Total.js platform. Total.js platform grows up and it has great results.

Thu Jan 03 2019 21:14:00 GMT+0100 (Central European Standard Time)
Total.js Code Editor v1

Products: Try our real-time collaboration tool for Total Developers. Code Editor offers great features for development.

Fri Dec 07 2018 22:55:13 GMT+0100 (Central European Standard Time)