Total.js v3.0.0

Impossible is possible, this is Total.js v3. This new version brings great new features which will make you love Total.js framework even more. Version 3 is not just an another number, it brings breakthrough features and simplicity.

• Better documentation
• Give a star on GitHub
• Follow us on Twitter

Good backwards compatibility

I guarantee a good backward compatibility with Total.js >= 2.5, but downgrade won't be possible for NoSQL embedded databases since this version doesn't contain NoSQL Views. Some methods have been removed and the framework contains alternatives in most cases.

Bundles

Bundles are one of the best features in this new version of Total.js. Bundles are something like packages, but they are a part of application's core. Imagine that you have a CMS with many files and you don't want to see these files in your e.g. new project because they are useless for you but they are important for the project. And then you use a bundle of CMS which you copy into your app, then you can extend a built-in funcionallity or a theme. Also you can create the bundle from everything e.g. API, eshop, etc. and use it in your projects. Bundles bring clean projects and easy maintenance. You will love bundles.

• Documentation

Async rendering of Total.js Components

This is a very big feature. So a component can contain e.g. loading products from DB or communicate via RESTBuilder with another 3rd party service.

• Documentation

Component:

<script total>
    exports.render = function($) {
        // $.repository
        // $.model
        // $.user
        // $.session
        // $.controller
        // $.query
        // $.body
        // $.files
        // $.options, $.settings --> will contain same value
        // $.callback(model); --> callback with a model
        
        // Model will be rendered in the component view
        $.callback([1, 2, 3, 4]);
    };
</script>

@{foreach m in model)
    <div>@{m}</div>
@{end}

Rendering in a view:

@{component('mycomponent', { customOptions: true })}

Total.js Components support inline files

Next big feature. Total.js parses all inline files and stores them in temporary directory. All files are available on http://blabla.com/~componentname/filename.

• files must be encoded in base64 format
• data are stored in <file name="FILENAME">base64</file> tag
• Documentation

Example:

<file name="logo.svg">base64</file>
<file name="template.html">base64</file>

<script total>
    ...
</script>

NoSQL database

Total.js v3 brings improved, highly optimized database engine with features not seen in any other flat file database. The database is much faster during reading/updating/removing/inserting operations and saves memory as well as CPU. This DB works well on ARM and data is streamed when reading/updating/removing/inserting.

I re-wrote the engine completely and now the engine doens't use Node.js streams but it uses my own implementation which is optimized for better reading and writing.

• New Online NoSQL embedded viewer

New database filters and features

• Documentation

// Performs Regullar Expression
NOSQL('products').find().regexp('name', /iPad/).callback(console.log);

// Performs special search algorithm
NOSQL('products').find().fulltext('name', ['iPad', 'iMac']).callback(console.log);

// Raw JS query
NOSQL('products').find().query('doc.published && doc.price > 0').callback(console.log);

// Reversed "fields" (removes specific fields in the result)
NOSQL('products').find().fields('-description', '-body').callback(console.log);

// "Listing" response will contain a listing object in the form e.g.:
// { page: 1, limit: 50, pages: 50, count: 500, items: [ ..., ... ] }
NOSQL('products').listing().paginate(1, 50).where('confirmed', true).callback(console.log);

// Incremental modifications
NOSQL('products').modify({ '*price': 1.10, '+countupdate': 1 });

NoSQL worker

This is another great feature in Total.js framework. NoSQL worker creates another thread for NoSQL embedded database, so your application will not share the thread with db engine.

Enabling NoSQL worker in /config:

nosql-worker     : true

You can have a problem with some filters like builder.filter() and builder.prepare(). You can't use NoSQL worker in some cases if you use above filters. To get more info about workers feel free to contact me.

NoSQL storage

This is a new file system storage for small big data with MapReduce funcionality. This storage saves documents by days. In other words: the framework creates a new storage file every day. Data are saved in plain text files like NoSQL embedded database file (same structure). Storage is targeted for storing data from IoT sensors.

• Documentation

NOSQL('yourdb').storage.insert(YOUR_DOCUMENT);

Storage doesn't have any method for removing/modifing documents, but you can perform:

Scanning:

// NOSQL('yourdb').storage.scan([from], [to], mapper, [callback]);
NOSQL('yourdb').storage.scan('2018-01-01', '2019-01-01', function(doc, repository, stats) {

    // It's executed for each document in the storage    
    // "repository" {Object} can contain some computed data (average, max/min values, etc.)

}, function(err, repository, stats) {

    // DONE

});

NOSQL('yourdb').storage.scan('2018-01-01', function(doc, repository, stats) {

    repository.avg = repository.avg ? (repository.avg + doc.temperature) / 2 : doc.temperature;

}, function(err, repository, stats) {

    // DONE
    console.log(repository.sum);
    
});

Map Reduce:

This feature performs map/reduce on each document in the storage (only the first time) and then storage performs map/reduce for new documents only. It's very effective. A result called repository is stored on the file system.

// Registers Map/Reduce
NOSQL('yourdb').mapreduce('sum', function(doc, repository) {
    // "repository" {Object} can contain some computed data (average, max/min values, etc.)
    // "repository" is saved on HDD
    // "repository" also contains previous values
    repository.sum = (repository.sum || 0) + doc.price;
});

NoSQL binary & FileStorage

NoSQL binary has also been updated and files are stored under sorted directories. Each directory holds up to 1000 files. Directories are indexed and they are named as 000-000-001 up to 999-999-999. So you can store max. 999 999 999 * 1000 files per DB.

Binary storage has the same methods as before and supports one new method binary.browse([directory], callback(err, files)) for browsing uploaded files.

New File Storage engine supports 100% backward compatibility, but downgrade won't be possible because of a new directory structure for the newest files. NoSQL File Storage can be used without reference to NoSQL database and for file operations only you can use FILESTORAGE(name_of_storage).method_name.

• Documentation

New helpful helpers:

• res.filefs()
• res.filenosql()
• res.imagefs()
• res.imagenosql()
• file.fs()
• file.nosql()
• mailmessage.attachmentfs()
• mailmessage.attachmentnosql()

Improved NoSQL performance

I have improved reading/updating performance of NoSQL database by about 20%.

Reading:

For example 100 000 simple documents like example below can be processed within 1 seconds. Reading introduces short CPU pausing, so it doesn't overload CPU.

{"id":"18040622030002orl0","index":0,"name":"16jz1chy1aonj4inzz13","price":44,"datecreated":"2018-04-06T20:03:57.966Z","body":"Lorem ipsum dolor sit amet, consectetur adipisicing elit. Dolorem debitis, officia beatae. Facilis quas omnis aperiam quae corporis harum perferendis!"}

Updating:

This version of NoSQL performs updates as the data are read. Older version always created a copy of modified NoSQL database, but new version performs all modifications as the file is read (just-in-time), so it's much more effective. Writing also introduces short CPU pausing, so it doesn't overload CPU. IMPORTANT: database can't corrupt DB files when the app is killed since it is designed for unexpected situations.

TABLE - an another NoSQL database

TABLE is a new plain text database like NoSQL embedded but the big difference is preddefined schema. In other words this TABLE DB can store data according to the schema and NOSQL can store schema-less data. It uses same methods/engine like NoSQL embedded but the performance is much more better around (20-30%) and this DB can save CPU, memory and HDD consuption. Table supports following data-types: number, string, boolean, date and object serialized to JSON format. TABLE DB also supports joins like NoSQL embedded database and worker.

• Documentation
• Online TABLE viewer

Schema must be defined in the config:

table.dbname1         : id:string | name:string | price:number     | created:date
table.dbname2         : id:string | name:string | discount:number  | age:number     | created:date

TABLE('products').find().between('price', 100, 200).callback(console.log);

NoSQL embedded can be used in cluster

I have added a simple support for using NoSQL embedded database in the cluster. Database performs locking when updating DB.

Improved middleware

Middleware supports new implementation like Total.js Schema operations. New implementation can simplify the code, for example:

MIDDLEWARE('delay', function($) {
    // $.req
    // $.res
    // $.options
    // $.controller
    // $.next()
    setTimeout($.next, $.options.delay || 1000);
});

Live reload

Live reload automatically reloads website without the need to manually refresh a web browser. It's very easy to enable, just edit debug.js in your app and add options.livereload = true + you need to update view layout.html like the example below.

• Documentation

debug.js

// ===================================================
// FOR DEVELOPMENT
// Total.js - framework for Node.js platform
// https://www.totaljs.com
// ===================================================

const options = {};

// options.ip = '127.0.0.1';
// options.port = parseInt(process.argv[2]);
// options.config = { name: 'Total.js' };
// options.sleep = 3000;
// options.inspector = 9229;
// options.watch = ['private'];
options.livereload = true;

require('total.js/debug')(options);

Layout:

<html>
<head>
    <!-- Total.js inserts live reload scripts from Total.js CDN only in debug mode -->
    @{import('livereload')}
</head>
</html>

Improved schemas

Schemas supports two new improvements: schema.setInsert() and schema.setUpdate().

schema.setInsert(function($) {
    $.success(true);
});

schema.setUpdate(function($) {
    $.success(true);
});

Execution in controller's action:

// A route must have declared schema
function some_action_insert() {
    var self = this;
    self.$insert(self.callback());
}

// A route must have declared schema
function some_action_update() {
    var self = this;
    self.$update(self.callback());
}

Inline execution is very easy:

$INSERT('schema', model, [options], callback, [controller]);
$UPDATE('schema', model, [options], callback, [controller]);

Schema field default value

This version of Total.js simplifies declaring default values for the specific field. It's very easy, just follow the code below:

NEWSCHEMA('User', function() {
    schema.define('name', 'String(30)')('DEFAULT_VALUE');
    schema.define('age', Number, true)(30);
    schema.define('created', Date)(() => new Date());
});

Async schema operations can be async

This version supports async execution of operations in async execution. The example bellow will show you how:

NEWSCHEMA('Test').make(function(schema) {

    schema.addWorkflow('1', function($) {
        setTimeout(function() {
            console.log('1');
            $.success('1');
        }, 100);
    });

    schema.addWorkflow('2', function($) {
        console.log('2');
        $.success('2');
    });

    schema.addWorkflow('3', function($) {
        console.log('3');
        $.callback();
    });

    schema.addWorkflow('4', function($) {
        console.log('4');
        $.success('4');
    });

    var model = schema.create();
    
    // .$workflow(name, options, [callback], [async])
    // .$transform(name, options, [callback], [async])
    // .$operation(name, options, [callback], [async])
    // .$save(options, [callback], [async])
    // .$insert(options, [callback], [async])
    // .$update(options, [callback], [async])
    // etc.
    
    model.$async(NOOP).$workflow('1', null, true).$workflow('2', null, true).$workflow('3', null, true).$workflow('4', null, true);
});

OUTPUT:

2
3
4
1

A simple support for a promises because of async/await

But I don't recommend to use it for web applications because it's slower than the classic way:

NoSQL embedded database:

async function data() {
    var response = await NOSQL('users').find().promise();
    console.log(response);
}

data();

RESTBuilder:

async function data() {
    var response = await new RESTBuilder('https://www.totaljs.com').get().plain().promise();
    console.log(response);
}

data();

Improved routing

I have improved routing a bit by adding two great improvements:

• a HTTP method can be a part of route URL, for some developers this can be better solution than defining method in the flags
• schema flags support @crud_workflow_operation_name, the framework will try to find an operation according to the name in the specified schema, it's not related to workflows file

Write less, do more:

ROUTE('POST /api/products/create/',  ['*Product --> @insert']);  // --> tries to find schema.setInsert()
ROUTE('PUT  /api/products/{id}/',    ['*Product --> @update']);  // --> tries to find schema.setUpdate()
ROUTE('GET  /api/products/{id}/',    ['*Product --> @read']);    // --> tries to find schema.setGet()
ROUTE('GET  /api/products/',         ['*Product --> @query']);   // --> tries to find schema.setQuery()
ROUTE('POST /api/products/import/',  ['*Product --> @exec']);    // --> exec can be workflow, operation or transformation

// OR

ROUTE('POST /api/products/create/    *Product --> @insert');
ROUTE('PUT  /api/products/{id}/      *Product --> @update');
ROUTE('GET  /api/products/{id}/      *Product --> @read');
ROUTE('GET  /api/product/            *Product --> @query');
ROUTE('POST /api/products/import/    *Product --> @exec');

// Multiple operations
ROUTE('POST /api/products/test/      *Product --> @check @save (response) @email'); // responds with "save" output
ROUTE('POST /api/products/test/      *Product --> @check @save @email'); // responds with array of all results

With new dynamic operations you can easily check required fields in the schema directly, for example:

// "op.update" according to the route "*Product --> @update"
schema.required('id', (model, op) => op.update);

// "op" will contain ".check", ".save", ".email" according to the route " *Product --> @check @save (response) @email"
schema.required('id', (model, op) => op.save);

Routing groups:

Can be used in some special cases. I use groups in Total.js CMS for some plugins which are added dynamically as a part of CMS admin and CMS evaluates these routes only for logged-in admin users.

exports.install = function() {
    ROUTE('GET /api/products/', readgroup, ['&admin']);
};

function readgroup() {

    var self = this;
    
    console.log(self.route.groups.admin);
    // output: true
    
    self.empty();    
}

Improved CORS

I have updated CORS() method by adding a new feature which can help you with allowing CORS for all requests. Just execute the method without arguments:

CORS();
// The framework will allow CORS for all routes, methods and origins with enabled credentials.

And the framework tries to merge different CORS conditions to one because of performance. The code below generates usually ROUTE() method with cors flag:

CORS('/api/*', ['get']);
CORS('/api/*', ['post']);
CORS('/api/{id}/', ['delete']);
CORS('/api/{id}/', ['put']);

// Framework will generate in the background:
CORS('/api/*', ['get', 'post', 'delete', 'put']);

Improved domain root changing

This improved feature will help you if you want to provide multiple Total.js apps on the same domain divided by path e.g. /eshop/, /cms/, /helpdesk/.

• all attributes href and src will be modified by adding a new root path
• View Engine supports @{#} command for adding of new root path (it will work in JS and CSS files)

Example:

// Changing root path (default is empty)
default-root    : /eshop/

<a href="/"><img src="/logo.png" /></a>

<script>
    function myRedirect() {
        location.href = '@{#}/orders/';
    }    
</script>

Output:

<a href="/eshop/"><img src="/eshop/logo.png" /></a>

<script>
    function myRedirect() {
        location.href = '/eshop/orders/';
    }    
</script>

Disabling the auto-remaping in a view:

@{noremap}

No remmaping
<a href="/"><img src="/logo.png" /></a>

But bellow code will still work as expected
<a href="@{#}"><img src="@{#}/logo.png" /></a>

Output:

@{noremap}

<a href="/"><img src="/logo.png" /></a>

<a href="/eshop/"><img src="/eshop//logo.png" /></a>

Auto-generating names in "/versions" file

This feature downloads the file in the background and generates MD5 checksum + CRC32. Then it adds the hash to the URL. So from now you don't need to change filename in the /versions file.

• Documentation

/js/default.js      : auto
/css/default.css    : auto

Output will be as shown bellow:

<link type="text/css" rel="stylesheet" href="/css/default-39232023.css" />
<script src="/js/default-842725733.js"></script>

JS and CSS supports a simple markup of view engine

This feature can be helpful in some special cases. It can replace a mark in the script or style directly.

Supports only:

• config values @{config.key}
• resource values @{resource('key')}
• framework globals @{G.key} or F.global.key

yourscript.js:

var name = '@{config.name}';

yourstyle.css:

.container:before { content: '@{config.name}'; }

IMPORTANT: it also works with .html templates processed via LOCALIZE() method.

Improved JavaScript compressor

I have tuned JS compressor and now it optimizes multiple var declarations to one var declaration divided by , comma, so your resulting script will be smaller. Example:

var a = 100;
var b = 'Total.js';
var c = true;

Output:

var a=100,b='Total.js',c=true;

Improved CSS compressor

Improved CSS compressor can optimize margin and padding values. Improvements can decrease your size of CSS. For understanding:

body { margin: 10px 10px 10px 10px; }
p { padding: 10px 0 10px 0; }
div { margin: 10px 0px 0px 0px; }

Output:

body { margin: 10px; }
p { padding: 10px 0; }
div { margin: 10px 0 0; }

Improved UID

I have improved UID([group]) method by adding a new argument group which means a specific counter for defined group.

console.log(UID('users'));
console.log(UID('orders'));
console.log(UID('products'));
console.log();
console.log(UID('users'));

// Output:
// 18050414560001cde1
// 18050414560001cde1
// 18050414560001cde1
// 
// 18050414560002cde1

Convertors

Convertors are a new part of Total.js framework. A convertor is a simple method which can convert some data according to defined schema. Schema supports almost all types from Total.js Schemas. I have tuned convertors for the best performance.

• Documentation

CONVERT({ name: 'Peter', age: '33', address: 'Slovakia' }, 'name:String(30), age:Number, terms: Boolean');
// output: { name: 'Peter', age: 33, terms: false }

What for?

Usage is very simple and really helpful for preparing some filters, etc..

function controller_action() {
    var self = this;
    var filter = CONVERT(self.query, 'page:Number, limit: Number, search: String');
    console.log(filter);
    // output: { page: 13, limit: 0, search: '' }
    // ...
}

RESTBuilder supports proxy

I have extend RESTBuilder by adding proxy option. Now you can use proxy directly in the RESTBuilder. Proxy supports http protocol only which can communicate with http and https endpoints.

RESTBuilder.make(function(builder) {
    builder.url('https://www.totaljs.com');
    builder.proxy('127.0.0.1:8080');
    builder.exec(console.log);
});

Or you can set the proxy globally in the config:

default-proxy    : 127.0.0.1:8080

Auto-generating security.txt

This version of Total.js can generate security.txt file automatically, contact information depends on your app configuration.

• More info about security.txt
• Documentation
• Example of security.txt

// Default
security.txt      : Contact: mailto:support@totaljs.com, Contact: https://www.totaljs.com/contact/

// Disabled
security.txt      :

// Custom:
security.txt      : Contact: mailto:petersirka@gmail.com, Contact: tel:+421903163302, Policy: https://www.totaljs.com/privacy/

Nicer error response messages

I have improved UI of error response messages after 3 years and it complies 21 century. I keep simple and universal design combined in white and black color.

Improved unit-testing

I have added a new global method TESTUSER() for testing purposes. Declared user in this method will be used as an authorized user for all requests. Internally the method replaces F.onAuthorize delegate.

TESTUSER({ id: '1234567890', name: 'Peter Sirka' });

// or with roles
TESTUSER({ id: '1234567890', name: 'Peter Sirka' }, ['@admin']);

// This endpoint is secured via "authorize" flag
TEST('Add user', '/api/users/', function(builder) {
    builder.json({ name: 'John Total' });
    builder.exec(OK());
});

Conclusion

That's not all. I have performed hundreds of small improvements and fixes which make Total.js one of the best Node.js frameworks.

Total.js framework benefits:

• you don't need to use a complicated stack like WebPack, Gulp, Grunt, etc.
• you don't need any dependency, framework is smaller
• you can write apps everywhere and you can only use a Notepad with Node.js platform
• for small projects you can use powerful NoSQL embedded database
• Total.js doesn't use any dependency and contains everything for creating much powerful applications (real-time, web, IoT, REST services, etc.)
• Give a star on GitHub
• Download complete apps

Sponsorship / Partnership

We're looking for stable sponsors and partners. Contact us if you or your company is interested in sponsorship / partnership with Total.js platform. Be a part of us!

Are you interested? Contact us: info@totaljs.com