New release: Total.js v3.1

This version brings bug fixes and some improvements. You can upgrade your current version of Total.js framework from NPM via $ npm install total.js. I recommend to update Total.js now!

Updated default config keys

This is very important change in this version. I have replaced all keys with - chars to keys with _ chars. This change saves a lot of code and personally I think that the performance is may be better. For example:

// OLD:
mail-smtp       : value

// NEW:
mail_smtp       : value

// OLD:
static-url-script       : value

// NEW:
static_url_script       : value

Big differences in the code:

CONF['mail-smtp']
// vs.
CONF.mail_smtp;

I don't know what the compiler really makes, but in the most cases (in different JS engines) can the compiler evaluates ['mail-smtp'] as a string which can affect CPU and memory consumption (maybe). Don't worry about backward compatibility because Total.js solves older keys without any problem and you will be notified about all older keys in app log. Backward compatibility will be removed in Total.js v4.

Config supports a simple encryption of values

New version of Total.js supports a very simple encryption of config values. You can encrypt/encode values via base64 or hex. So now you can encode credentials, secrets or SMTP credentials.

somevalue1           : base64 MTIzNDU2
// Contains 123456

somevalue2           : hex 313233343536
// Contains 123456

somevalue3 (string)  : base64 MTIzNDU2
// Contains 123456

somevalue4 (number)  : hex 313233343536
// Contains 123456

Extended bundles

This version can extend .js, .css and .html files. If you don't want to rewrite the entire content of some file and you want to extend a file for example script then you can use /path/path/--name_of_file.js:

/bundles/cms.bundle
/themes/admin/--ui.js

a content of --ui.js will be added into the /themes/admin/ui.js at the end of file
the file won't be overwritten

External bundles

This version of Total.js supports external bundles which can be downloaded from URL address. Implementation is very easy, just create plain-text file /bundles/cms.url with this content:

https://cdn.totaljs.com/bundles/cms@12.bundle

.bundlesignore

Total.js v3.1 supports .bundlesignore file with a similiar syntax as .gitignore file.

Extended routing

Code less, do more. The framework automatically pair a dynamic value as Total.js Schema and it's case insensitive. So you can merge multiple routes into the one. If the schema doesn't exist then the framework responds with 404.

ROUTE('GET   /api/{schema}/        *{schema} --> @query');
ROUTE('GET   /api/{schema}/{id}/   *{schema} --> @read');
ROUTE('POST  /api/{schema}/{id}/   *{schema} --> @insert');
ROUTE('PUT   /api/{schema}/{id}/   *{schema} --> @update');

CSS variables support a default value

This was a missing feature. It's very helpful for e.g. Total.js CMS. Declaration is similar like or operator in JavaScript:

div { border-radius: $radius || 10px }

Extended NoSQL storage

NoSQL storage is a good choice for storing data from IoT sensors. NoSQL storage creates a new NoSQL database every day. And this version of Total.js extends NoSQL storage by adding three new methods:

.find(beg, end, [threads])
.count(beg, end, [threads])
.scalar(beg, end, type, field, [threads])

This methods return NoSQL DatabaseBuilder. The big benefit is that Total.js can read NoSQL storage with multiple threads, so you can traverse all data / million documents very effective outside of main thread:

NOSQLSTORAGE('weather').find(null, null, 10).take(10).sort('created', true).callback(console.log);

New method: U.reader()

Is a special feature for streaming and filtering data. It uses NoSQL reader filter engine which is optimized for the best performance. U.reader() can be combined with e.g. U.streamer(). It can be used in Total.js Flow.

var reader = U.reader();

reader.find().take(10).search('name', 'Peter').sort('created', true).callback(console.log);

// or
reader.scalar('max', 'price').callback(console.log);

// or
reader.count().callback(console.log);

some_stream.on('data', function(data) {
    // ... processing ...
    // data must be Array of Objects or Object
    reader.push(data);
});

some_stream.on('end', () => reader.push(null));

Cookies support SameSite attribute

controller.cookie('cookie_name', 'cookie_value', '5 days', { security: 'lax' });
controller.cookie('cookie_name', 'cookie_value', '5 days', { security: 'strict' });

// or

controller.cookie('cookie_name', 'cookie_value', '5 days', { security: 1 }); // lax
controller.cookie('cookie_name', 'cookie_value', '5 days', { security: 2 }); // strict

// or

controller.cookie('cookie_name', 'cookie_value', '5 days', { samesite: 'lax' });
controller.cookie('cookie_name', 'cookie_value', '5 days', { sameSite: 'strict' });

TaskBuilder

TaskBuilder can create simple async tasks.

supports great features
supports error handling
supports similar features like Schemas or Operations
Documentation

var tasks = new TaskBuilder();

tasks.push('task1', function($) {
    $.value.task1 = true;
    $.next('task2');
});

tasks.push('task2', function($) {
    $.value.task2 = true;
    $.next('task3');
});

tasks.push('task3', function($) {
    $.value.task3 = true;
    $.done();
});

tasks.exec('task1', function(err, value) {
    console.log('Done', err, value);
});

Usage in Schemas/Operations/Controllers:

// "$" is SchemaOptions or OperationOptions
var tasks = $.tasks();

// or in a controller:
var tasks = self.tasks();

// ...
// your tasks
// ...

tasks.exec('some_task_to_start', $.callback);

Improved OPERATIONS

This version brings better features for Total.js OPERATIONS. Operations are something like Schema methods - operation/workflow/transform but they aren't have any declared schema, they are schemaless.

Documentation

Operations chaining

This is a new powerful feature which can run multiple operations in a row:

NEWOPERATION('1', function($) {
    console.log(1);
    $.success();
});

NEWOPERATION('2', function($) {
    console.log(2);
    $.success();
});

NEWOPERATION('3', function($) {
    console.log(3);
    $.success();
});

// RUN(operations, model, [callback], [options], [controller], [result_name]);
RUN(['1', '2', '3'], { custom: 'data' }, function(err, values) {
    console.log(values);
    // Output: { '1': { success: true }, '2': { success: true }, '3': { success: true }
});

Additional features:

NEWOPERATION('1', function($) {
    console.log(1);
    $.success();
});

NEWOPERATION('2', function($) {

    console.log(2);
    $.invalid('some-error');

}, 3, true);
// "3" means that the operation will be repeated 3 times if the operation returns some error
// "true" (by default) means that the chaining won't continue if the operation returns some error

NEWOPERATION('3', function($) {
    console.log(3);
    $.success();
});

RUN(['1', '2', '3'], { custom: 'data' }, function(err, values) {
    // do stuff
});

Routing

Now are operations part of routing, example:

// Single operation
ROUTE('GET /api/users/     * --> @operation_name');

// Multiple operations, returns all results
ROUTE('GET /api/users/     * --> @operation_1 @operation_2 @operation_3');

// Multiple operations, returns a result from operation_2
ROUTE('GET /api/users/     * --> @operation_1 @operation_2 (response) @operation_3');

Logger for Schemas and Operations

I have added a logger for measuring time of Schemas and Operations. It can be used only for better debugging of Total.js applications.

// Console output
logger        : console

// Output to file /logs/logger.log
logger        : file

// Disabled (default state)
logger        : false

Example of output:

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
GET /api/inventories/all/
2018-09-28 19:39:43 | OPERATION('inventories')                 | 0.243 sec.   | GET /api/inventories/all/           | 172.17.253.202       | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.31
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
GET /api/inventories/all/1/
2018-09-28 19:39:43 | OPERATION('inventories_items')           | 0.250 sec.   | GET /api/inventories/all/1/         | 172.17.253.202       | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.31
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
GET /api/cl/
2018-09-28 19:42:07 | OPERATION('periods')                     | 0.237 sec.   | GET /api/cl/                        | 172.17.253.202       | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.31
2018-09-28 19:42:07 | OPERATION('navigation')                  | 0.260 sec.   | GET /api/cl/                        | 172.17.253.202       | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.31
2018-09-28 19:42:07 | OPERATION('regions')                     | 0.238 sec.   | GET /api/cl/                        | 172.17.253.202       | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.31
2018-09-28 19:42:07 | OPERATION('locations')                   | 0.235 sec.   | GET /api/cl/                        | 172.17.253.202       | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.31
2018-09-28 19:42:08 | OPERATION('dealers')                     | 0.271 sec.   | GET /api/cl/                        | 172.17.253.202       | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.31
2018-09-28 19:42:08 | OPERATION('regionaldivisions')           | 0.222 sec.   | GET /api/cl/                        | 172.17.253.202       | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.31

Multiple HTTP methods in the routing

Tip from Tema Smirnov. Now you can defined a multiple HTTP methods directly in URL argument. IMPORATNT: between methods can't be any empty space.

ROUTE('GET,POST /something/', action);

Improvements

String.slug()

I have improved String.slug() for UTF characters for Chinese/Japan/Arabic/etc. languages. It creates a hash from String. For example:

console.log('牛津热词：邋遢风'.slug());
// Output: sir1di1bss166e8

NoSQL reader

I have improved NOSQL/TABLE reader. I separated filtering engine to independent class and now the class is used for NOSQL / TABLE and NOSQL STORAGE. Of course, engine can be used for streaming data with new utils called U.reader().

UID generator

This version brings a more secured UID() results, Total.js changes ending hash each minute. Now you can use UID results in URL address directly as an argument.

CORS

I have implemented a new config key default-cors which can contain hostnames as origin. It's very helpful for the last restricting of CORS() method. And from now the CORS supports internal caching, so evaluating is a bit faster.

/config:

default_cors       : //www.totaljs.com, //componentator.com

Many thanks to M.Sc. Jens Mueller / Security Researcher for the alert.

Components

Now the framework uses scripts/styles according the group of components, so scripts/styles won't be rendered into the common file. I have enabled caching for components in debug mode (previous versions use caching in release mode only).

New global aliases

New global aliases can improve readability of the code in your Total.js application. Here is a simple comparison:

CONF vs F.config:

// OLD
F.config.something
CONFIG('something');

// NEW
CONF.something

FUNC vs F.functions:

// OLD
F.functions.something();
FUNCTIONS('something')();

// NEW
FUNC.something();

CACHE vs F.cache:

// OLD
F.cache.set('key', 'value', '5 days'); // write
F.cache.set2('key', 'value', '5 days'); // write (persistent)
F.cache.get2('key'); // read

// NEW
CACHE('key', 'value', '5 days'); // write
CACHE('key', 'value', '5 days', true); // write (persistent)
CACHE('key'); // read

Good to know

Total.js framework doesn't processed Total.js files (controllers, definitions, schemas, etc) if the file ends with -bk.js or starts with . dot.

/definitions/database.js     --> will be processed
/definitions/database-bk.js  --> won't be processed
/definitions/.database.js    --> won't be processed

Fixes

a critical bug with storing uploaded files via httpfile.fs() or httpfile.nosql()
a critical bug with JavaScript minificator
a critical bug with rendering of multiple async components
a critical bug with NoSQL counter and freezing app
a critical bug with GZIP compression (sometimes appeared in Safari)
nosql.update() and nosql.modify() methods if the first argument is a function
F.wait() in the test mode
LOCALIZE() didn't work for nested directories
error handling when WebSocketClient is connecting (for example, if the request is unauthorized)
versions with auto value and with enabled F.wait()
LOAD('release') a release mode
SchemaInstance.$clean() for nested schemas
extracting bundles
subdomain routing for localhost
a service for cleaning NoSQL embedded databases
async rendering of components in Total.js View Engine
RESTBuilder cache works only if the response status is 200
compressing CSS with \t tabs
controller.autoclear()

Road to v4

Total.js v4 will remove all older declaration from Schemas, Operations and some internal funcionallity. I have created a lot of projects and I learned a lot, so the core won't be changed.

Happy coding!