Recent versions of ors-amarisoft are able to collect and send xlog data to any fluentd/fluentbit compatible data lake using a common file format.

This was achieved by adding a new parameter to SlapOS Amarisoft software release:

Fluentbit configuration in SlapIOS Amarisoft software:

Selected operation data is collected by the xamari process running nearby Amarisoft enb process on the BBU. Operation data is then converted into a common format (enb.xlog) and sent by fluentbit to a remote server for further processing such as KPI calculation.

A complete example is explained in a notebook. It is a first example of standardisation of operation data of a vRAN infrastructure. A proof-of-concept implementation of a data lake capable of generating 3GPP KPIs from xlog files has been deployed with Wendelin big data hub.

Further standardisation may be needed for environment (ex. temperature). For resource usage (ex. CPU), SlapOS already proposes a standard file format of resource usage (RAM, CPU, disk) per process. However, those file formats are mutually inconsistent, making data sharing in a common data space difficult or impossible.

Ideally, we need a common way to share time sequences, no matter their properties. Possible candidates include:

• HDF5 (exhaustive feature set, limited adoption)
• Parquet (reduced feature set, well adopted)
• ndarray (minimal feature set but native support in python, JS, C, C++, FORTRAN, etc. and widely adopted)
• JSONL (a good replacement to CSV with schemas and native support in python, JS, etc.)
• yet to be defined standard file format from IDSA as part of DIN SPEC 27070
• etc.

Selecting a common standard will be part of next development steps of SlapOS.

References

• KPI calculation notebook - https://nbviewer.org/urls/lab.nexedi.com/kirr/xlte/raw/master/demo/kpidemo.ipynb
• Fluentbit support in Amarisoft SR - https://lab.nexedi.com/nexedi/slapos/blob/master/software/ors-amarisoft/instance-enb.jinja2.cfg#L146

Recent versions of ors-amarisoft are able to collect and send xlog data to any fluentd/fluentbit compatible data lake using a common file format.

This was achieved by adding a new parameter to SlapOS Amarisoft software release:

Selected operation data is collected by the xamari process running nearby Amarisoft enb process on the BBU. Operation data is then converted into a common format (enb.xlog) and sent by fluentbit to a remote server for further processing such as KPI calculation.

A complete example is explained in a notebook. It is a first example of standardisation of operation data of a vRAN infrastructure.

Further standardisation may be needed for environment (ex. temperature). For resource usage (ex. CPU), SlapOS already proposes a standard file format of resource usage (RAM, CPU, disk) per process. However, those file formats are mutually inconsistent, making data sharing in a common data space difficult or impossible.

Ideally, we need a common way to share time sequences, no matter their properties. Possible candidates include:

• HDF5 (exhaustive feature set, limited adoption)
• Parquet (reduced feature set, well adopted)
• ndarray (minimal feature set but native support in python, JS, C, C++, FORTRAN, etc. and widely adopted)
• JSONL (a good replacement to CSV with schemas and native support in python, JS, etc.)
• yet to be defined standard file format from IDSA as part of DIN SPEC 27070
• etc.

Selecting a common standard will be part of next development steps of SlapOS.

References

• KPI calculation notebook - https://nbviewer.org/urls/lab.nexedi.com/kirr/xlte/raw/master/demo/kpidemo.ipynb
• Fluentbit support in Amarisoft SR - https://lab.nexedi.com/nexedi/slapos/blob/master/software/ors-amarisoft/instance-enb.jinja2.cfg#L146

SlapOS is currently based on an API which relies on remote procedure calls in XML (XML-RPC). Although it is very reliable and can be used easily with slapos console, some users have been requesting a pure REST API.

This new API is now closer than ever to be released. It is based on a CRUD (Create Read Update Delete) approach similar to JIO. Rather than a procedural approach, the new SLAP API is content-based. A user creates content (ex. a compute node registration) and tracks the progress of the registration process by looking up how the original content evolves with new properties added as part of this process.

The procedural XML-RPC API of SlapOS has mostly been translated into content-based API:

✔ Done, ❌ Not Done,🛑 Not used and not planned

As we can observe, only three types of contents (Compute Node, Software Installation and Software Instance) are sufficient to describe the whole process of managing a distributed architecture such as a cloud platform, an edge platform or a vRAN. A fourth portal type (Software Product) might be necessary to cover service catalogs and support introspection on software release (ex. equivalence). This remains to be discussed though since it could also be implemented as part of Software Instance additional properties.

The new API is now fully tested with automated tests (see also here and here).

Moving from procedural API to content-based API was actually a challenge posing certains risks and uncertainties. The current state of the ongoing implementation demonstrates that content-based API works well and can actually simplify a lot the description of the SLAP protocol. Content-based API might also simplify disaster recovery procedure whenever a single instance has to handle millions of shared instances (ex. CDN, SDN, etc.). This topic will be discussed in future blog.

Replication manager overview

Replication manager (repman) is an orchestrator for high availability on top of MariaDB 10.x and MySQL and Percona Server 5.7 GTID replication topologies. It supports both interactive and automated failover of the master server. It verifies the integrity of the slave servers before promoting one of them as the replacement master and it also protects the slaves by automatically setting them into read-only mode.

Replication manager cluster has at least 2 nodes, one should be master node and the rest are slaves. It is advised to use at least 3 nodes cluster to keep the cluster tolerant to failures when losing or stopping a slave. Replication manager support for replication tree or relay slaves architecture, in case of master death one of the slaves under the relay is promoted as a master. It also support Maxscale, ProxySQL, Haproxy, etc for routing read/write request to databases.

SlapOS overview

SlapOS provides a compact and unified solution to automate provisioning, orchestration, monitoring and preventive maintenance of software services that can be deployed in data centers, edge nodes, user terminals or embedded systems without the need for virtualization.

SlapOS is composed of two types of servers:

• SlapOS Nodes which run on a minimal GNU/Linux distribution with installed slapos-node package. It consists of SlapGrid daemon which manage Buildout for bootstrapping applications and supervisord to control running processes. Depending on the server ressources, a SlapOS Node can host many Mariadb and Replication Manager application instances.
• SlapOS Master is the orchestrator. It informs each node which software should be build or deployed and which instances of which software should be run. The Node also return to the master the state of services that are deployed as well as information usefull to connect to services. This documentation explain how to request replication manager service with Rapid.Space.

Our goal is to provide an easier way to automatically deploy a full replication manager cluster, with one or more mariadb backups instances then, allow users to request replicated databases into deployed cluster. To do that, we will have to create a replication manager software release for building the software with all required dependencies and provide instances profiles for deploying the cluster and requesting database shared instances.

Software integration and new orchestrator

To provision the cluster, replication manager can generate database and proxy configurations based tags provided and hardware resources description. The prov-orchestrator parameter supplied to repman indicate on which platform the cluster is going to be provisioned. It's useful for generating configuration files according the node platform that will run database and proxies services. Untill now, local, onpremise, OpenSVC and Kubernate orchestrator was supported but none of them where compatible with SlapOS. Instead of regenerating multiple configurations and scripts managed by repman, the easier solution was to add a new orchestrator to repman called slapos for support deployment into Computer Partition. Cluster configuration was updated with new information about the computer partition where the service is deployed, like slapos-proxysql-partitions, proxysql-servers-ipv6, slapos-db-partitions.

We only support Mariadb as database server for clusters. Event if replication manager can support multiple proxies for a  single cluster, we are deploying for now a single ProxySQL per cluster to keep things less complex. The proxy is the entry point to access databases. Initialy, proxies was configured only with IPv4 network, but for slapos support we added support for IPv6 so that, a service deployed in another server can access the database using IPv6 network.

Replication Manager Software Release

The first step of integration is to build Replication manager component, which is mainly writen in go language. By using the golang environment provided by SlapOS, we simply have to write the component buildout profile which extends golang environment and run go build with flags required to build replication manager from sources.

Once we was able to build the component, we create the software release for automating installation of repman and needed dependencies. There is a number of components available to use from the SlapOS repository, but we had to add few more required for replication manager, like: restic (for database backup), sysbench and proxysql.

With the software release, SlapOS node is able to build the entirely software and required components, generate instance configurations and different services, then run them using supervisord. The software release for replication manager was added here.

First release of repman

The software release version 1.0.259 was released on rapid.space, and can be used for deploying clusters. First of all, we should build the SR into servers which will host the cluster. For example, if we want to deploy a cluster with one master and 3 backups (all instances allocated in different servers for good resiliency),  we need to setup 5 slapos node servers. The procedure to add a software release to SlapOS Master and push compiled binary to cache is available here. This other documentation explain how to supply a software release from command line.

Once the software release is built on servers, we can request cluster instance from rapid.space UI or by using command line by running following commands:

This will request a cluster called cluster1 with 4 mariadb instances and one ProxySQL. The variable repman_comp_guid is the computer GUID of replication manager instance, it's the root instance which request mariadb instances. Parameters -sla-XX-computer_guid specify where the mariadb XX will be allocated. Details about all parameters are described in the file instance-repman-input-schema.json.

After few minutes, the cluster will be deployed and connection parameters will be published to rapid.space panel UI.

Request a database instance

Our deployed cluster has no database yet. To add a database to the cluster, we need to request a shared instance which is also called Slave Instance. The slave instance Parameters are sent to the cluster, these parameters are then used to create the database while redeploying the cluster instance. The advantage of requesting a slave instance to create the database is that, different users can create their own databases in the same cluster. Users will only see databases they have created without know any informations about instance of others.

A few minutes after the request, the instance is ready, database was created on master and replicated to slaves. Published connections parameters show URLs for connecting to database, both IPv6 and IPv4 url required SSL connection as database account creation includes a REQUIRE SSL clause in the GRANT statement. IPv6 url allow to access to database from internet.

Setup Wordpress with repman database

Now we have a replicated database deployed, we want to use it with our Wordpress service. Our wordpress instance and replication manager database are on different network so we will use IPv6 for configuration.

After we have validated database information, wordpress will finish configuration and install the database.

Objective and state of the art

As part of OSIE project (Open Source Industrial Edge) we needed a way to implement an industrial line which we could control using entirely open source technologies. In a pure automation world this is usually done using PLC (Programmable Logic Controller) units. Nexedi, being a software company is still far away from the automation world, but one of our OSIE partners (Olimex) was already producing boards that could work together and control electrical circuits. All we needed was to provide an abstract layer of technology so that any automation engineer could use the already existing PLC languages of his choice (ST, Ladder Logic, etc).
Luckily of us Beremiz was already around and it could provide us the PLC control we needed. Thus all we had to do was to hook all together - which is what this article is all about.

Initial coupler prototype

Below the first design of the coupler as it was created by Nexedi.

This coupler consists of the A20-OLinuXino-LIME2 (top) and MOD-IO (bottom) board which are connected over UEXT cable.

The coupler (A20-OLinuXino-LIME2 board) has following characteristics:

1. Running Debian Buster Linux (soon with PREEMPT-RT patch)
2. Allwinner A20/T2 dual core Cortex-A7 processor, each core typically running at 1GHz and dual-core Mali 400 GPU
3. 1GB of RAM
4. 1000MBit native Ethernet

The coupler (MOD-IO) has following characteristics:

1. Support for I2C protocol
2. stackable design (so that multiple boards can be connected to one A20-OLinuXino-LIME2 providing multiple relays)
3. Digital and analog input / output channels

And both boards are completely open source (incl. schematics).

Architecture

Our design had to meet several criteria from the start:

• all components should be open-source
• we should keep existing industry standard like IEC 61131-3
• we should start with Modbus as a communication protocol but later switch to OPC-UA
• We should aim to have a PLC runtime running on an x86_64 architecture and controlling our "dumb" coupler over Modbus. Still if for reasons needed we should also allow runtime to run on MCUs.
• We should use TSN (runtime communicating a with the coupler)
• We should NOT invent yet another ladder logic alike programming language but instead provide an IDE for development which automation engineers already are familiar with
• We should be able to have a component base IDE (pure editor, runtime and coupler) working as a stand-alone unit which we can distribute easy in an abstract software-like fashion

After initially trying with ProviewR we discovered that it is a monolithic product which is quite hard to break into stand-alone components. We continued our search and we found Beremiz

First Goal defined: build a Modbus server at coupler which allows to control relays

Right from the beginning we knew that we need to expose a Modbus interface to the MOD-IO's relays which we wanted to control. Luckily for us using pymodbus and pyA20Lime2 we could provide a Modbus server running on the coupler fairly easy. Still, as a PLC has a deterministic nature where each cycle has a fixed execution interval and where the runtime needs to acquire Modbus status variables over the network from the coupler's Modbus server, we quickly discovered that both network latency and modbus's server implementation overhead can add sometimes significant lags to PLC's program execution - in some cases this leads to Modbus read requests timeouts after more than 30 ms. This finding quickly showed us that we need to spend time to work on improving TSN support for lime2 and improve (if possible) modbus python implementation. For a quick overview of our TSN efforts one can read our report here.

Second goal defined: Write a "Hello World" program 

Once we had a Modbus server exposing relays of MOD-IO it was time to write our first "Hello world!" PLC program - a simple program which switches ON / OFF the existing 4 relays and resets internally some counter values. We used Beremiz in the following setup:

1. Beremiz's IDE running on Kubuntu 20.04 LTS used for writing the program, visualizing its status and controlling the runtime
2. a runtime running on same machine (x86_64 architecture) which controls the coupler over a LAN network (notTSN yet!)
3. a Lime2 coupler running a modbus server with one (for now) attached MOD-IO with 4 relays

Below you can see the source code of our first PLC program:

Below it's a screen shot of the program running inside the IDE itself with real time values.

And finally one can see the video of the coupler being controlled remotely from the runtime.

Third goal: have the PLC control the conveyor of a real machine

As part of OSIE project and the optical (formerly cherry-picking machine) we build there (see here) we will soon use the coupler in a real conveyor belt based machine which should be able to track movements of conveyor belt objects and perform a real-time classification on them to decide whether to select an object or not.

All the selection will be done using machine learning inference algorithms which should run on a close-by powerful server running Wendelin Big Data Platform.

With current coupler we now have the building blocks to finish our production line.

Conclusions

Our proof of concept showed that the existing tools allow us to have a complete open source based system which is not at all vendor locked and free to use.

Objective

As part of OSIE project (Open Source Industrial Edge) we required to have a demonstrator and development system. We decided to implement a small industrial production line using only open source technology. Due to the COVID pandemic and travel and working restrictions, we had no way to visit any of the industrial project partners and thus had to come up with an alternative way so we could implement controls of a production line which we had access to. As one of the project partners and one the Nexedi team leader were located in a region known for cherry production in Bulgaria, we decided to build a cherry picking machine that could sort good from bad cherries and later also fruits in general.

Our long term objectives for this machine are:

• it is entirely built from open source and open hardware components
• it is run using SlapOS distributed cloud platform to deploy and orchestrate services
• it is using the open source industrial edge architecture described in our article on Free Software and Open Hardware for Industrial Automation

State of the Art

Nexedi is a software company first and foremost, so building a mechanical conveyor with all needed electronic components for this was not something we were very familiar with at first. Our initial designs were very naive, our understanding of the problems we faced very limited.

Initial simulation prototype in a virtual environment (Godot)

We decided early on to work with ProviewR - an open source process control software used in large scale industrials. During the first months of the project we had a chance to test how ProviewR would connect to a "soft" modbus interface which our machine would provide but without having the physical machine itself. More details can be found in this blog post on using ProviewR with the Godot 3D game engine.

Below you can see a simulated video of the machine we wanted to build and control with ProviewR.

Architecture of the machine

Our critiera to meet from the get go are listed below:

• Use ONLY open-source components - we could NOT use commercial PLCs and / or software in building this machine
• The machine should be built using standard industrial components (belts, motors, VFDs, gears, etc) so that anyone can rebuild it easily
• The machine should be able to "speak" Modbus (initially) and OPC UA (later)
• The machine should be controllable using ProviewR
• Despite initially targeted at cherry fruit selection, we quickly saw that with proper measurements, gears and motors (and of course software) the machine could be used for other fruit types (and not only fruits)

Based on these criteria we knew that we would have to implement our own versions of already existing industrial components like PLC with modbus protocol support. For this we decided to use Olimex's Lime2 open hardware board

First Goal: Building a PLC

We had to be able to control electrical devices (motors, valves, etc), which is usually done by a PLC. We had to build our own but before starting we had to handle a number of prerequisities:

1. Build an enclosure for Lime2 which can be mounted on a DIN rail
2. Using this enclosure to connect over UEXT to Olimex's MOD-IO board and control electrical relays using I2C
3. Be able to programmatically control a relay (switch it On / Off)

It took some efforts to build a case for Lime2. We did this by starting from an existing FreeCad model which we extended so it could fit an additional component from Olimex called Lime2-shield. The process of printing a box was done using an entry level 3D printer. Luckily for us, Olimex had already developed a library that handles communication over I2c called pyA20Lime2.

As result we had our first first PLC with its own DIN rail mountable box that could be controlled by a Python script. Not fancy but working.

Second goal: Build a conveyor

Nexedi hired an external engineer to build a conveyor belt around which we started to construct the fruit selection machine. Our goal was to have the most simple conveyor with loading and offloading units that:

1. Uses VFD (which "speaks" modbus) so we can control over modbus the linear speed of it
2. Is build from standard industrial components without any welding needed (just using bolts). This way any amateur can rebuild the machine once we open source it

Thanks to industrial standards and existing components this was the easiest part of all. With some good mechanical knowledge we had our first conveyor belt running with START / STOP button within a few weeks.

Third goal: Use the PLC to control the conveyor

At this point in project we could use simple Python script to turn conveyor On / Off. To begin, we did not want to handle VFD's modbus communication and thus we had to use a hard-wired configuration and hard-coded linear speed settings which we then developed to controling the electrical relay using remote code. As part of this phase, we added an air valve controlled via a 12V DC circuit which we would use later in project to blow off bad fruits from the line. We added an air compressor which would supply the compressed air for the valve - of course using existing industrial components like relays, air valves, compressors, etc.

And final wiring:

Fourth goal: Modbus integration. Camera and OpenCV to do an optical inspection

The last part of project was adding Modbus support for our software PLC layer. This would allow a remote application and / or ProviewR setup to control the conveyor. In addition to this we started to experiment with real time camera which uses OpenCV to do shape detection. Since it was a first attempt, the optical inspection we implemented was quite modest and simple: it uses simple masks and countour detection so our conveyor is "trained" to blow away any red items which have more than three countours. This algorithm will of course be improved as the project progresses so it can be used on for real fruits with more complex and realistic models. However for the initial goals we proved that it's possible to have a open source machine within 5 months.

France's National Commission on Informatics and Liberty (CNIL) ruled on 8th October 2020 that it was illegal to store personal health data of EU residents on any cloud operated by a US company or by a European company with business in the US:

Source: https://www.documentcloud.org/documents/7224049-Me-moireCnilHDH.html

Unlike Tik Tok in the US or Facebook in China, US clouds are not banned in the EU. However, using US clouds to process or store personal data of EU residents could be illegal. Using EU clouds submitted to US laws to process or store personal data of EU residents could also be illegal.

This situation is a direct consequence of the "Shrems II" ruling by the Court of Justice of the European Union (CJEU), an equivalent in EU of the US Supreme Court, which applies to all EU.

Source: https://en.wikipedia.org/wiki/Max_Schrems#Schrems_II

France's CNIL had thus to apply "Shrems II" ruling in its decision. Currently, EU residents have no appeal if US authorities remotely access their personal data stored in EU through legal enforcement laws such as CLOUD Act, FISA and Executive Order 12333.

CNIL suggests that in order to preserve fundamental liberties, personal health data should be stored on a cloud operated in Europe by juridical entities that are completely independent of any business activity in the US. CNIL suggests technology licensing approaches for US clouds to operate legally in EU whenever personal data of EU citizens needs to be stored.

Even though CNIL does not say anything about it, the same legal reasoning could apply to other sensitive personal data.

Conseil d'Etat (an equivalent in France of US Supreme Court) confirms in its decision that it is imposible for French government to control unsolicited access by US goverment to personal data stored on US clouds or on clouds subject to US law. Conseil d'Etat confirms that new projects involving personal health data must not use US clouds or clouds subject to US law.

Source: https://www.conseil-etat.fr/actualites/actualites/health-data-hub-et-protection-de-donnees-personnelles-des-precautions-doivent-etre-prises-dans-l-attente-d-une-solution-perenne

It is very likely that similar decision would be taken by CNIL for personal data stored on a Chinese cloud provider for similar reasons realted to China Internet Security Law of 2016.

Understanding the reasoning behind the ruling

Let us consider the following parties

Because country Z can request to Y personal data of K stored by X on the cloud of Y, and because K can not appeal against this request of Z, and because Y can not refuse the request of Z due to its presence in Z and some extra-territorial law, then fundamental rights of K are violated. In application of proportionality principle and if X stores a lot of sensitive personal data, X is reponsible of this situation and should thus stop using Y. It does not matter whether data of K is stored inside EU or outside EU.

Practical mitigation

It is safe to consider that in order to avoid any litigation risk, companies which store personal data of EU residents on the cloud and especially sensitive personal data should migrate to a cloud provider which is not submitted to CLOUD Act, FISA, Executive Order 12333 or possibly China Internet Security Law.

If personal data is not encrypted, it is safe to store it on servers:

• located inside EU, to ensure the absence of foreign jurisdiction;
• controlled by an "independent" EU company.

Here are some examples of "independent" EU companies:

• an EU company with a majority of EU stockholders and no business in US or China;
• an EU company with a majority of EU stockholders and which business in US or China is operated through a subsidiary with an independent governance.

This second example still needs to be confirmed.

If personal data is encrypted, personal data could be stored on servers outside EU in certain cases as long as encryption keys used to access personal data of EU residents do not fall under the control of US or Chinese Law.

Source: https://www.cnil.fr/fr/transferer-des-donnees-hors-de-lue

One should note that any cloud service which does not have access to personal data does not fall under the scope of CJEU/CNIL decission. An EU company which owns its servers could for example deploy on its own infrastructure a remote orchestration service as long as this remote orchestration service does not have access to personal data on the servers and does not itself store personal data.

Legal evolution

CJEU/CNIL decision will likely be reverted by national court and lead to an appeal at CJEU which will likely confirm the original decision of CNIL.

Consequences of CJEU/CNIL could also evolve if a new Privacy Shield is implemented by EU.

Source: https://en.wikipedia.org/wiki/EU%E2%80%93US_Privacy_Shield

Even though some EU governments are deeply influenced by transatlantic social relations and US-EU business opportunities, a new privacy shield is likely to face increased opposition from citizens. Human right values are no for sale for everyone in EU.

Also, due to the nature of CLOUD Act, FISA and Executive Order 12333, it is likely that any new EU Law which authorizes transatlantic data transfer will lead to "Shrems III" unless the US government abandons its global surveillance policy. The same could be said about Chinese governmment and China Internet Security Law. 

In EU, states are prevented from mass collecting data thanks to the independence of CJEU. EU clouds which are not subject US or China

Source: https://www.ft.com/content/71cc07fb-58ff-404d-868c-5dd0e8a97e20

Beyond EU, US and China, some countries such as Russia or India have implemented or may implement digital sovereignty laws that prevent storing or processing data in EU. Digital surveillance defined in French law is probably incompatible with some privacy laws in Japan.

Source: https://fr.wikipedia.org/wiki/Loi_relative_au_renseignement

Russia or China do not allow storing personal data of staff outside Russia. 

Source: https://en.wikipedia.org/wiki/Data_protection_(privacy)_laws_in_Russia

Overall, the consequences of "Shrems II" case and CNIL ruling are just one example of similar situations which will happen again in other parts of the word and result from the contradiction between laws related surveillance, fundamental liberties and digital sovereignty. 

Legal risks

It is safer to design an IT system based on the observation that privacy should be enforced in every country and that all countries try to surveil each other through extra-territorial laws without any global framework. China's initiative to set global data security rules is unlikely to be accepted very soon by EU, US or Japan.

Source: https://www.wsj.com/articles/china-to-launch-initiative-to-set-global-data-security-rules-11599502974

In a digital world with ever evolving and mutually conflicting national laws, the safest design for an IT system is to store data on servers owned by a local independent company and only rely on global cloud services that do not have access to data stored on those servers and do not store themselves personal data. Self-hosting can also be an option.

Just like GDPR did not lead to lawsuits immediately, CJEU/CNIL will not lead to lawsuits immediately.

However, privacy is important for citizens in EU, in Japan and in some other countries. Litigation in EU is not very costly and can be financed by a single person. Justice in EU is independent: national states can not control decisions of CJEU, ECHR, etc.

The risk of litigation is increasing for multinational companies that keep on ignoring EU cloud companies. Recent foreign policy in US ("America First") and in China (wolf diplomacy) is creating resentment among European population. Litigations against companies which store personal data of EU residents on clouds subject to US or China's laws will likely increase.

GDPR fines can be as high as 20,000,000€ or 4% of a company's yearly incom.

Source: https://www.cnil.fr/fr/definition/sanction

In this context, the safest approach for a company operating in the EU is to increase their use of EU cloud services that respect privacy. Another safe solution is to build their own cloud in the EU based on open source software and guarantee full auditability. One way to ensure that an EU cloud service respects privacy is to request access to operation management procedures and audits, something that companies such as BSO or Rapid.Space already provide. Another way is to verify compliance of the service with privacy standards, something which the Gaia-X project plans to provide in the future.

References

Some open source cloud software provided by EU companies:

• OpenNebula - https://opennebula.io/
• OpenSVC - https://www.opensvc.com/
• Proxmox - https://www.proxmox.com/
• SlapOS - https://slapos.nexedi.com/
• XCP-NG - https://xcp-ng.org/

Some EU cloud providers that try to ensure independence from US and China extra-territorial laws:

• BSO - https://www.bsonetwork.com/
• Euris - https://www.euris.com/
• Fuga - https://go.fuga.cloud/
• Hetzner - https://www.hetzner.de/
• Ikoula - https://www.ikoula.com/
• Ionos - https://www.ionos.de/
• Jaguar - https://www.jaguar-network.com/
• OVH - https://www.ovh.com/
• Rapid.Space - https://www.rapid.space/
• Scaleway - https://www.scaleway.com/en/

Some EU cloud providers that provide access to their operation management procedures:

• BSO - https://www.bsonetwork.com/
• Rapid.Space - https://www.rapid.space/

Some unions of cloud providers that meet health data regulations and include EU members:

• AFHADS - https://afhads.fr

Gaia-X project: https://www.data-infrastructure.eu/GAIAX/Navigation/EN/Home/home.html

Objective

The whole idea is to make Godot simulate industrial devices and expose them with the modbus protocol so you can use any modbus-capable industrial control suite out there, in our case it's ProviewR. This way we can build test and our Plc in a virtual shop floor.

Our concrete example is to reproduce a setup in a real fruit selection machine using ProviewR, modbus and different sensors

Technologies

Godot

Godot is a 2D and 3D, cross-platform, free and open-source game engine released under the MIT license. It was initially developed by argentinian Juan Linietsky and Ariel Manzurfor for several companies in Latin America prior to its public release. The development environment runs on multiple operating systems including Linux, macOS, and Windows. Godot can create games targeting PC, mobile, and web platforms.

ProviewR

ProviewR is probably the first Open Source system for process control and automation in the world. Originally developed in Sweden by Mandator and SSAB Oxelösund as a process control system based on standard computers, the system has become a fully-fledged, integrated and low-cost solution that is running on standard PC's with Linux as operating system.
ProviewR is Open Source and the license is GNU/GPL.

Modbus

Modbus is a data communications protocol originally published by Modicon (now Schneider Electric) in 1979 for use with its programmable logic controllers (PLCs). Modbus has become a de facto standard communication protocol and is now a commonly available means of connecting industrial electronic devices. Modbus is popular in industrial environments because it is openly published and royalty-free. It was developed for industdeploy and maintain compared to other standards, and rial applications, is relatively easy to places few restrictions - other than the datagram (packet) size - on the format of the data to be transmitted. Modbus uses the RS485 or Ethernet as its wiring type. Modbus supports communication to and from multiple devices connected to the same cable or Ethernet network.

Godot shop floor

Before we begin, be aware that we will not cover in details how to setup and use Godot, there is a lot of good tutorials out there for you to reproduce or modify our simulation. If you want to test our simulation, just follow the steps in the README of our repo.

Here is our scene:

There are five objects in the scene:

• The hatched cube is a cube spawner. It produces a new cube every three seconds. This cube will fall and land on...
• ... a rectangle with yellow arrow on it which represent a convoyer (which on the real setup has its speed controlled by a VFD using by modbus) belt. The cube is then displaced from left to right on it.
• A cube. obviously. which is in the middle of the conveyor belt on this image. This cube represents the coffee cup of the real world shop floor.
• The blue rectangle with a black sphere in it in the sky is an optical sensor. It can detect a cube on the conveyor belt.
• And finally there is the pusher (which on the real world setup is an air valve with a compressor) on the left side of the conveyor belt. It can be activated to push a cube outside the conveyor belt, making it fall into the underworld.

Cube spawner

Here is the code for spawning a new cube every 3 seconds:

The ``_ready`` is a function called when a scene (in our case the cube spawner) is loaded and "ready" to be used in the 3D world (see Godot documentation) )

Conveyor belt

Nothing fancy here, a force is applied to the cube touching it. This can be done directly in the editor, but as we want the yellow arrow to go at the same speed than the cube, we used a script in order to change the speed in only one place.

Modbus in Godot

This was the simple part. Now let's focus on our modbus devices. Before going into the scripts we are using, we need to understand how modbus work.
Modbus devices are communicating using a client/server relationship (new wording of the Modbus Organization), where clients make requests and servers respond to it (except for Modbus TCP where every devices can be a client). Here are the different data types that a ModBus server can provide:

A modbus client can ask a modbus server to:

• change the value in one of its registers, that is written to Coil and Holding registers.
• read an I/O port: Read data from a Discrete and Coil ports,
• command the device to send back one or more values contained in its Coil and Holding registers.

This is enough technical stuff to understand what we will do in order to communicate data between ProviewR and Godot.

As one can except, Godot does not provide ModBus communication protocol directly. We will need to use libmodbus for this.

To use it, we have created dynamic library which will be loaded and used by some GDScript later on. We will skip the setup part as this not really interesting and very well documented in the Godot documentation. Our dynamic library provides four functions used in Godot:

• start_server(address, port): This function call modbus_mapping_new in order to setup the registers of the modbus servers. Then we spin a new thread using pthread_create which call the server_routine. This routine will wait for a connection from a client and the accept it, then it's waiting for a request sent by the client using modbus_receive. We directly respond to it using modbus_reply which also read or write different register/coils if this what was asked in the request. And that it, we are ready to read and write server register/coils directly in Godot using:
• set_holding_register(value) which set a register with a given value,
• get_holding_register() which returns the value of a register,
• get_coil() which returns the value of a coil.

Here is how we load and wrap our dynamic library symbol in Godot:

Now that we can communicate, let's add some logic to our sensor and pusher

Sensor

Nothing fancy here, we get the distance to the cube using a raycast and store the result in a register of the modbus server corresponding to the sensor.

Pusher

We read the coil of the modbus server corresponding to the pusher and extend the pusher if the coil is set and retract it otherwise.

We used the _physics_process callback because it provides consistent updates over time, regardless of how fast or slow time advances, see Godot documentation.

That the end of the Godot part. We have two modbus server, but nobody is talking to them, what a shame. Let's fix that by using ProviewR.

ProviewR automaton:

ProviewR is a complex system to setup and since we already covered Modbus setup in ProviewR in another article.

Our setup is one Modbusmaster (I can't use client/server wording as ProviewR hasn't made the switch...) with two Modbusslaves, one for the sensor and one for the pusher, with one Modbusmodule containing one signal each. For the Plant we have one Digital Input and one Digital Output connect to their respective node counterpart.

Let's talk about our automaton. We are using grafcet to represent our automaton.

From top to bottom we find:

• ISO: Start step
• T0: Transition triggered by a value of 1 while reading the sensor register
• S0 and S1: Two step running in parallel. S0 set a timer of 1.25 second and S1 set the coil of the pusher to 1.
• T1: Transition triggered by the end of the timer. A coil is unset by default while your not actively pushing a 1 one it, so when the transition happens, the coil of the pusher is set to 0 and the pusher retract itself.
• Loop to ISO

This plc is then compiled (stay tuned for an article on this) to a native executable that ProviewR will run.

If we start both Godot and our ProviewR runtime....

Et voilà!

Thanks for reading this long article.

Sources

• https://en.wikipedia.org/wiki/Modbus
• https://en.wikipedia.org/wiki/Godot_(game_engine)
• http://www.proview.se/v3/e
• https://docs.godotengine.org/en/stable/index.html
• https://lab.nexedi.com/nexedi/godot-modbus-demo

I have always been interested in process isolation technologies such as ZeroVM which provides a form of isolation based on Google's NaCl that even went through formal verifaction at CEA, as part of the Resilience FUI project lead by Nexedi. I did not understand why ZeroVM was not further adopted and had a discussion with one of its creators, Camuel Gilyadov. Based on his experience, the fact that NaCL development stopped prevented the adoption of ZeroVM and has lead other new approaches that solve a similar problem.

Cloudflare Workers: V8: Isolates

Workers is a technology created by Cloudflare and based on V8: Isolates. It was created to solve the lack of isolation in the Lua scripting of Cloudlfare's CDN. Cloudflare markets it as "Cloud without Containers". It is not very different from a user point of view from what has existed for 20 years in Zope: restricted Python scripts. Technically spearking, it consists of running multiple interpreters within the same V8 process (Isolate class). It is not clear however how code is isolated between interpreters and how strong is this isolation.

Support of other languages (C, C++, Rust) is based on WebAssembly. A complete description is availble on "Fine-Grained Sandboxing with V8 Isolates" video.

The V8: Isolates class is used by other libraries such as isolated-vm.

Firecracker: microVMs

Developped by AWS, Firecracker is presented as an alternative to QEMU. Its written in Rust, provides a minimal required device model to the guest operating system while excluding non-essential functionality (only 5 emulated devices are available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal keyboard controller used only to stop the microVM). It also handles resource rate limiting for microVMs, and provides a microVM metadata service to enable the sharing of configuration data between the host and guest.

Firecracker runs in user space and uses the Linux Kernel-based Virtual Machine (KVM) to create microVMs. The fast startup time and low memory overhead of each microVM enables you to pack thousands of microVMs onto the same machine. Each Firecracker microVM is further isolated with common Linux user-space security barriers by a companion program called "jailer". The jailer provides a second line of defense in case the virtualization barrier is ever compromised.

Firecracker is based on musl C library by default and is usually operated through an API which makes it suitable for shared hosting or dynamic CDN, as in SlapOS "slave instance" model. It can also be used with a command line.

Its streamlined kernel loading process enables a < 125 ms startup time and a < 5 MiB footprint. Yet, it is a traditional VM approach which then spaws containers inside the VM with the runc command and supports the OCI image format. It is similar in terms of goals with ChromeOS Linux VMs that are used to run a Debian system.

One should note that there are other small VMs on the market: Jailhouse (Siemens) and XtratuM (fentISS) are quite popular for embedded applications.

gVisor: UML revived

gVisor is a user-space kernel, written in Go, that implements a substantial portion of the Linux system call interface. It provides an additional layer of isolation between running applications and the host operating system. One could view it as a revival of user mode Linux, implemented in Go and with better performance.

gVisor seems to be implemented using the ptrace approach, the same one as proot relies on to deliver user-mode chroot isolation. However, it relies on a much better security model to achieve user-mode process isolation without virtualisation.

What gVisor solves best in our opinion is the problem of poor portability of containers which has lead Nexedi to reject the use of containers in production. This is clearly illustrated in gVisor architecture:

gVisor seems for now focusing primarily on running containers. What would be really nice would be to use gVisor to execute binaries in the same way as qemu does in user space emulation mode or as proot does in combination with qemu.

OSv: back to MacOS 7

OSv is a minimal operaring system made by Cloudius Systems, the company that also does Scylla DB (an altenative to Nexedi's NEO). The idea of OSv is to create very small images that can load much faster than a full blown POSIX OS. OSv achieves this by removing features such as multiple users or process isolation. In a sense, OSv is a single user POSIX system with a memory model closer to μLinux (or MacOS 7) than to Linux. By having a single memory address space and "outsourcing" isolation to kvm, OSv boots fast and runs fast. 

OSv relies on a utility called Capstan to turn an application developped in C, C++, python, Node.js, etc. into a minimal image. OSv maintains a set of base images that can be used as templates for this purpose. OSv shares in this sense the same goals of isolation and portability as SlapOS, but follows a different path: rather than relying on POSIX users (as SlapOS does), it tries to encapsulate a complete application as a standalone OS image.

Another way to understand OSv is as an altenartive to user space emulation mode of qemy. Rather than trying to convert system calls in user space, OSv embeds a minimal OS with enough system calls to run an application, an executes it in a VM. The overhead of spawning an another is thus minimised from a few seconds to much less than a second.

Open Questions

I still feel that there is no equivalent as brillant as ZeroVM was. gVisor seems to be promising to me because of its flexibility and portability, but I wish it could be used for pure process isolation without containers. Overall, this overview of recent virtualisation technologies raises many questions to be addressed in a next post:

• Is "V8: Isolates" isolation as strong as NaCl and ZeroVM? (probably not)
• Can gVisor run SlapOS nanocontainers or plain executables? (probably yes)
• is OSv based on an existing OS source code?
• is OSv POSIX support mature enough to run "real" applications?

This blog post is going to present how Nexedi installed a standard 19'' open rack with Tioga Pass servers in a China data center during the Covid-19 epidemic in March 2020 by cooperating remotely in real time with staff in two cities using Facepro Xpert glasses and systems. 

First installation of Standard 19'' Open Rack in China Telecom Data Center

Nexedi bought two 19'' Rack ESA Kits with OCP Tioga Pass L10 servers for its Rapid Space site in Xinzhou, a city in Shanxi province in mainland China. 

The 19' Rack ESA Kit and Tioga Pass servers are manufactured and supplied by Mitac from Taiwan. Rapid Space also uses Edgecore AS5812 Switches to allow all OCP servers to be connected via 10 Gbps LAN with 1Gbps Internet transit. Mitac shipped all equipment to the China Telecom data center in Xinzhou at the end of 2019 and Nexedi planned to send their Chinese staff in the beginning of 2020 to assemble and install all machines.

Challenge: Covid-19 + Complete Lockdown

In January 2020, Covid-19 started spreading in Wuhan and very soon the virus was detected all around China. By the end of January, Wuhan was completely locked down and other cities were shutting down one-by-one. People were asked to stay in their homes, all unnecessary activities were prohibited, and travel between cities, provinces or districts was strictly forbidden. All business activity in China stopped. All flights from foreign countries were stopped with no date set for restarting schedules. 

One Chinese developer of Nexedi was stuck in his hometown of Anhui. All other Nexedians capable of assembling or installing the servers were in foreign countries. Only staff from our Rapid Space local partner had access to the site but without prior experience in assembling or installing the servers. 

The schedule for the installation was therefore completely stopped.

Remote Installation: real-time cooperation between France and China

In March Edge-Core introduced Softfoundry's FacePro Xpert System to Nexedi, a tool designed for technicians and engineers to assist in remote on-site service, equipment inspection, maintenance and complex manufacturing assembly. A system is composed of one or more experts working on a  PC, tablet or smart phone and a field engineer with smart glasses. Participants can connect to the system using any type of network in order to communicate and visualize the machine and situation in real time. Without knowing when the lock down would be lifted in China, and the Covid-19 situation in France and other countries outside China deteriorating, Nexedi decided to adopt this FacePro Xpert system to handle the assembly and installation. 

The staff onsite brought all the servers and pieces to the data center and wore the Xpert smart glasses to join a visual meeting with Nexedi. 

In a first step, rails to the data center cabinet were fixed with screws before installing a BUS bar. Once panels (both left and right sides) and the BUS bar were installed, the rack (shelf) was inserted and mounted on the side panels.

Next a power supply shelf was installed and fixed on the cabinet to ensure that the power supply module would connect to the BUS bar before inserting the power supply unit itself. After assembling the switch with its pendants and rails by following the manual from Edgecore, staff fixed the switch to the cabinet, as pointed out by the red arrow in the freezed view.

Afterwards servers were plugged into each rack one by one, powered on and connected to the switch to complete the server installation.

Nexedi's engineer in Anhui joined the meeting with his laptop. 

Nexedi's engineers in France joined the meeting with their laptops. 

During the meeting, Nexedi engineers (in France, Poland and in China) were able to get an on-site first person, real-time view and at the same time communicate with the staff transmitting video and audio to give instructions. To assist the on-site staff, the FacePro Xpert System was used to freeze certain views to mark or highlight specific areas for the on-site staff to identify. The field staff had their hands free to assemble machines while reading instruction on the glasses and listening to direct advice from Nexedi experts. They could also voice control the view on the glasses to better communicate and present their view and questions to Nexedi. 

With this real-time cooperation and sharing of first-person perspective, we are able to complete the assembly of all servers and installation of required software. 

In order to configure the AS5812 switch to make computers communicate, we installed AOS (ACCTON Operating System for switches). AOS will be soon open sourced by Edgecore. 

For every machine, we also pre-installed remotely a Debian 10 installation image, from which users can easily install Debian 10 server operating system. Nevertheless, they are also free to install any other server operating systems for their VM according to their needs. 

Nexedi used Raspberry Pi with IPv6 to remote operate and control the installation. First, we installed Re6st (to provide IPv6 access in China thanks to Grandenet) on a Raspberry Pi. Then the on-site staff turned on a mobile hotspot and made the Raspberry Pi connect to the internet by using the hotspot. After that the Raspberry Pi was connected to the debug port of the server with a network cable. With this method, Nexedi engineer could log in to the Raspberry Pi remotely through ssh. 

Success factors: Full tutorial + Facepro Xpert System (smart glasses) 

We are very happy with the result of this remote corporation assembly activity. There are four main factors responsible for this success: 

• Detailed step-by-step tutorials ad procedures are available to on-site staff to consult in advance, or even during the assembly, because we can also share documents while using the FacePro Xpert System. 
• The FacePro Xpert System itself with smart glasses solve the problem of sharing the first-person view while experts are thousand kilometers away from the site. 
• Use the Raspberry Pi as a bridge to access the server to diagnose or debug.
• Re6st allows to give each server a unique IPv6 address, so that all servers are able to expose services on IPv6.

We strongly think this is a solution that could be widely adopted in many industries that need remote corporation on some engineering activities. 

It is possible nowadays to create an "all-in-one" 4G/5G network infrastructure based a generic software and generic PC. This document provides an example of an offer that can be  presented as an alternative to Huawei products in the context of cost-conscious telecommunication operators.

Network description

We consider here a telecommunication operator with an existing bakhaul network (radio, wire) and an infrastructure consisting of:

• 200 big sites (ex. tri-sector antenna);
• 200 small sites (ex. hotel, factory).

The goal of this offer is to upgrade this infrastructure from 2G/3G to 4G/5G or to extend an existing 4G infrastructure with 4G/5G small cells.

Big Site Infrastructure

A Big Site provides 4G/5G to a city or an area of a city. It is connected through an existing TCP/IP redundant network (fiber, radio) to other sites and to the core network.

Big sites can be operated either autonomously with a local EPC or through a central EPC. Both options can be deployed at the same time.

Big sites can provide 4G and 5G radio. Both radio can be used at the same time thanks to DSS technology. 

Small Site Infrastructure

A small site extends the network in hotels, parkings, factories and VIP locations. It is connected through an existing TCP/IP network (radio, xDSL, fiber, LTE) to a big site or to the core network. 

Small sites can be operated either autonomously with a local EPC or through a central EPC. Both options can be deployed at the same time.

Small sites can provide 4G and 5G radio. Both radio can be used at the same time thanks to DSS technology. 

Central Cloud

A modern telecommunication operator should be able to provide cloud services. We thus include in our offer a Rapid.Space on-premise cloud that can be used both as public cloud and private cloud. For resiliency, two sites are needed at least. Sites do not need to be tier-3 thanks to built-in resiliency in the operation management software.

4G vs. 5G: both

4G is the best solution to day but 5G is already coming. Thanks to DSS technology, there is no need to decide which investment makes more sense. We support both.

Local EPC vs. Central EPC: both

A central EPC is the best solution to support handover for mobile applications (voice, data) and a large number of customers (> 10.000). It is also the best way to integrate to an existing core network (S1 protocol). However, it is not resilient and is not compatible with low-latency applications or true edge computing.

A local EPC on each site is the best solution for resiliency, performance (caching, buffering), edge computing (low latency, private networks) and fixed applications (voice, data). However, it has limited handover support (10.000 to 100.000 UEs) which requires either to split a national infrastructure into independent regional areas or to ignore handover.

With support both, at the same time, through different operator IDs managed by the same eNodeB or gNodeB.

Possible use of existing EPC

Amarisoft stack provides its own EPC but also supports existing EPC thanks to S1 interface. Some EPC vendors however use tricky provisions in their maintenance contract to close access to third party vendors. If this is the case, it is always possible to deploy a dedicated EPC for 4G/5G and integrate through roaming with the existing one.

Cost comparison

In order to compare the costs of this solution, site installation and maintenance costs should be taken out of the offer and evaluated independently because there is no fundamental difference between the installation of two pieces of 19inch hardware, no matter the supplier. It is also important to compare both OPEX and CAPEX on the same period of time with the same types of features.

Some vendors will try to hide their real cost by either hiding true OPEX cost that later keep on increasing or by hiding the true cost of on-site installation / maintenance of equipment.

Summary

It is possible to deploy a complete 4G/5G network from radio to billing to using only generic open source hardware (Open Compute Project, Edge-Core, MITAC, ITRenew, etc.), 4G/5G software stack (Amarisoft), SDR hardware (BJT, AW2S) and open source operation management software (Nexedi SlapOS). Overall cost is possibly lower than Huawei. Most components, hardware and software, are open source.

Most cloud services could be built by combining an open source / Free Software (eg. MariaDB) together with a service lifecycle automation (SLM) platform (eg. SlapOS). With one notable exception, this goal has not yet been reached by the Open Source / Free Software community because most projects are still focusing on some kind of virtualisation (eg. virtual machines, containers) or orchestration (eg. Kuberneres) which only represent 10% to 20% of what is necessary to implement a public cloud service.

This may leave SlapOS as the only open source / Free Software project that could possibly match leading public cloud services (AWS, Azure, Alicloud), as the following comparison table highlights:

Nexedi stack: less is more

Nexedi develops and operates complex, scalable applications with less than 15 software: the Nexedi Freee Software stack.

Out of those 15 software, developers actually focus on four of them:

• ERP5 for enterprise records management (including CRM, MRP, MES)
• OfficeJS for Progressive Web Application (PWA) development;
• SlapOS for IT infrastructure automation (on the cloud, at the edge or in the workshop);
• Wendelin for data collection and processing (including AI and conversion).

Since both SlapOS, Wendelin and OfficeJS are just variations of ERP5, Nexedi developers actually only need to learn a single framework (ERP5) and a single language (python). By relying on less tools, Nexedi developers have more time to learn ERP5 in depth. They can reuse their ERP5 knowledge with SlapOS and Wendelin. And thanks to the huge size of python library, most problems that are not already covered by ERP5, SlapOS, Wendelin or OfficeJS can be solved quickly.

AWS vs. Rapid.Space

Amazon AWS provides more than 200 cloud services.

The table below provides a comparison between Amazon AWS cloud services and technologies of the Nexedi Free Software stack which can be used to build simlar services deployed with SlapOS on Rapid.Space high performance, low cost cloud platform. For each AWS category and product,  we provide a possible alternative in Nexedi Free Software stack either as a SlapOS profile (server based) or as Progressive Web App (browser based). We also provide open source / Free Software alternatives we are aware of. 

Missing services

45 features out of the 209 AWS services are not yet covered by the combination of Nexedi stack and Rapid.Space. About 75% of AWS services are thus already covered by Nexedi stack and Rapid.Space, as long as developers accept to adopt Nexedi's tools:

• python on the backend (incl. C/C++/FORTRAN based libraries);
• buildout for cloud automation scripting;
• javascript on the HTML5 frontend;

Out of the 45 missing services, Nexedi does not intend to cover 3 of them:

• blockchain (2): because it has no practical use and can not scale (due to CAP theorem);
• quantum computing (1): because efficient qubit I/O can not exist with the levels of energy available on earth (due Maxwell's demon throught experiment).

12 features are actually already covered through existing Free Software that has not yet been integrated into SlapOS appstore.

Overall, only 30 services (less than 15%) are missing, mainly in relation to:

• Threat detection;
• AI models powered by private or secret datasets;
• Windows/VMWare compatibility.

Based on this analysis, future SlapOS R&D should consider how to encapsulate threat detection into the core of SlapOS design in a cost efficient manner.

Key differences

Nexedi stack focuses on reliability and simplicity. Traditional vendors may prefer to focus on ease of sales, ease of adoption (which often relates to fashion) and complexity (which some executives analyse as completeness). In the context of the previous comparison with AWS stack, one should keep in mind that Nexedi may rely on quite different approaches:

• SlapOS nano-containers instead of Linux name-space based containers;
• content-centric APIs instead of event-centric APIs;
• libraries rather than APIs;
• less is more rather than more and more.

Even though the end-result may be the same, the way it is achieved may be different.

For example, Nexedi considers that Linux name-space based containers are not stable enough for production and are not portable among different hosts with different distribution due to the kernel ABI mismatch problem [RD]. This is why we use SlapOS nano-containers which simply consist of running processes on a POSIX system with an unpriviledged user rather than with the root user. In terms of isolation, it makes no difference. In terms of security, it brings one benefit: some bad code which implies running a process as superuser has to be fixed. In terms of portability, it is superior: the same approach can be applied to FreeBSD or even possibly to embedded OS such as NuttX. It is also more reliable (no ABI mismatch risk) and simpler (no need to add the namespace layer). Yet, the current fashion in 2019 is to use containers, even though reliability problems are now acknowledged [RD] and fashion is fading away [RD].

We prefer in Nexedi to have a single SDK called JIO [RD] and develop all our code based on this SDK and its minimalist API. This API creates a unified abstraction of the world as a repository of contents. Each content has a precise schema, usually a JSON schema. Interacting with services is thus achieved with JIO in the same way as one would interact with a database or a filesystem but with some asynchronism and a wider range of possible operations or data types. Through JIO, we have eliminated most uses of event-based programming. We also stopped defining yet another API for each new problem that needs to be solved. Instead, we rather focus on defining yet another JSON content schema for each new problem that needs to be solved. We also made our code backend-independent and thus portable from one cloud to another in this way.

The difference between a content-based approach and an event-based approach is idempotence, that is the ability for a system to self-recover by repeating a process. Content-based approaches are idempotent by default whereas event-based approaches require extra processes which developers often forget to code. This is similar to the difference between declarative programming and imperative programming. In a distributed infrastructure with thousands of interconnected device, the risk of events is that they are sometimes lost. The advantage of content is that in case it was not accessed at some point, it can always be recovered later, which results in a much more resilient system. The idea of content-centric APIs is not new. It was deployed in military and aerospace in the 1990s with the Data Distribution Service (DDS) protocol [RD]. It can also be found in the OPC-UA PubSub protocol which release lead to an accelerated adoption of OPC-UA standard for Industry 4.0 [RD]. The importance of a concent-centric approach has been well understood in the industry for quite a long time (ex. process control of Japanese chemical factories in the 1990s based on a whiteboard architecture) and in telecommunication research with CCNx [RD]. This is why nearly all software in Nexedi are either content-centric or moving in that direction.

Nexedi also prefers to use a single integrated service (eg. Wendelin) with a single API and different libraries (eg. scikit-learn, scikit-image) to solve different problems (eg. machine learning, image analysis), rather than create multiple independent services with multiple APIs. Nexedi's direction is thus to base modularity on libraries rather than on services or APIs. We consider in particular that the approach called "serverless" can only lead to technical disasters if the meaning of "serverless" is to host on a cloud service hundreds of "single function code" which mutually interact by calling eachother asynchronously through the cloud service infrastructure. That form of "severless" approach can be close to impossible to debug, especially if one starts with a function, adds another which calls the first one and so forth until hundreds of functions are hosted and mutually call eachother asynchronously with no clear structure. It can also lead to latency issues if connection pooling has not be designed into the "serverless" infrastructure. Yet, if "serverless" means hosting code on the cloud and editing code through the Web on a system which is managed by the cloud provider, this is something that Nexedi has been doing since 2001. It has many advantages, especially in terms of efficiency and quality assurance.

Nexedi services which are equivalent to AWS are based on a much smaller list of base services (Wendelin, ERP5, etc.) yet cover a similar scope. It looks like less impressive to an executive than a list of more and more services. It has however the advantage of reducing the learning curve and simplifying integration. One developer can learn a single technology and use it for many different purpose. Less things to learn, more applications.

References

• Free Software and Open Hardware for Industrial Automation
• Top 10 Best Network Inventory Software for Linux
• Top 30 most popular AWS products (thanks to S. Fermigier)

Many companies use SFTP protocol and server (S) to exchange files between two systems and interface two applications (A and B).

The interface logic can be one-way:

A -> S -> B

Or can be two-way:

A <--> S <--> B

In most cases we are aware of, this type of interface is unreliable for various reasons. We will try in this article to list various approaches to make a reliable SFTP interface, based on the experience of other reliable interface techniques.

Reliable interfaces

There are numerous ways to create a reliable interface:

• CFT;
• exchange table in a transactional database;
• transactional HTTP API;
• synchronisation protocol;
• log-file or graph.

CFT is a well known tool that solves many problems found in file transfer between A and B. It provides the guarantee that:

• a file is transferred entirely or not transferred;
• each file sent leaves a trace in a log;
• each file received leaves a trace in a log.

It is thus possible to ensure that all files sent have been received.

An exchange table in a relational database has similar advantages:

• data is inserted into the table entirely or not at all (transactional database);
• the list of inserted data can be queried any time;
• reception of data can be acknowledged too in a transactional way by insert an acknowledgement line.

Transactional HTTP protocol (ex. Zope transactions) is a way to implement HTTP in an application server which ensures through HTTP similar transactionality as with a database. Data is either transferred entirely or not at all. Each transfer (upload) leaves a trace in a log file. Each transfer (download) leaves a trace. If data is in addition self-certified (using an SHA or signature for consistency), downloaded data can be checked for consistency.

Synchronisation protocols such as SyncML, JIO and to some extend embulk ensure that all parties share the same view on the same data subset at any time. By running synchronisation recurringly, the state of A and B eventually becomes consistent, despite synchronisation not being transactionnal.

Protocols such as git or any kind of shared, read-only log file or graph, ensures that all parties share the same data consistently. All parties can thus ensure that the state of applications (A or B) is consistent with the common log file or graph.

All cases we described ensure the integrity and traceability of data transferred by A and B. However, this does not ensure that the interface between A and B is fully consistent. 

Each data D that is transferred should in addition be well formed, by matching for example a JSON schema or an XML DTD. Beyond being well formed, the semantic of the data exchange should be validated using validation functions FA, FB which compares the state of applications (ex. FA for A) to the history of all data that were transferred:

FA(A(t), D(0), D(1), ..., D(t)) == True

It is mandatory to define schema and validation function FA, FB  to ensure that an interface is consistent.

Without that, project teams behind A and B endlessly spend their time trying to prove that the other party is wrong instead of both having a way to prove they are both right. Schema and validation functions play the same role in interfaces as unit or functional tests in application development.

The role of validation function FA is essential in particular to prevent a common problem: B never receives data D from A because A did not even transfer it. Without FA, this problem can never be solved.

The problems of SFTP

SFTP servers have a few issues that lead to unreliable interfaces:

• there is no guarantee that all data was uploaded (by A) because the file might be visible before the end of the upload by A;
• there is no guarantee that all data was downloaded (by B) because the file might be visible before the end of upload by A for example or because of network glitch;
• some SFTP servers erase data after the first download (by B) and thus there is no way to list all data that was exchanged;
• there is no way for A to know if B downloaded the file;
• there is no way for B to know which files A uploaded if files are erased after download (by a rogue user for example which has same credentials as B);
• there is no way to get a full copy of files exchanged by A and B if SFTP servers erases files after first download.

There is thus no way to ensure a reliable interface between A and B by using only SFTP without extra protocols.

SFTP is a bit like UDP compared to TCP. It does not guarantee any kind of integrity in communication.

Towards reliable SFTP interfaces

Obviously, the first step towards reliability is to design a well specified interface:

• define schema for each type of data D which is exchanged;
• define a function F for each application (A, B) which ensures consistency of the interface on each side.

This step is not specific to SFTP. Without it, it is impossible to ever reach reliability.

Let us now solve problems specific to SFTP.

There is no guarantee that all data was uploaded

This problem can be solved with different techniques:

• rename each file after upload (ex. day1.upload -> day1.data)
• upload a hash for each file (ex. day1.sha).

There is no guarantee that all data was downloaded

Compare SHA(day1.data) and day1.sha.

There is no way to list all data that was exchanged

Solution 1: remove the "erase after download" feature of the SFTP server.

Solution 2: add to the interface

• a "message list query" that lets B query A the list of files which A ever sent.

Solution 3:

• A sends every day to B the list of all files that were ever uploaded

There is no way for A to know if B downloaded the file

Solution 1: rename each file after download (ex. day1.data -> day1.received)

Solution 2: 

• a "message received query" that lets A request B the list of files that B ever downloaded.

Solution 3:

• B sends every day to A the list of all files that were ever downloaded

There is no way to get a full copy of files exchanged by A and B

Solution 1: remove the "erase after download" feature of the SFTP server.

Solution 2: add to the interface

• a "message resend query" that lets B request A to resend a message.

Additional thoughts

Defining a schema for Excel files or for COBOL-style data can be difficult. Yet, it is not impossible and it is mandatory.

Here are some suggestions of approaches:

• convert Excel to XML and use DTD or XSD
• use a COBOL conversion utility to XML or JSON (ex. xfw tool by Nexedi, used in many applications)

About the validation function, it is implemented in ERP5 in a simple way: we simply store all files we receive and all files we send in specific modules. And each document that relies on them (ex. Packing List) has an aggregate relation to it. This way, we can ensure that ERP5 received everything. ERP5 can also send again everything. We even keep a trace of all access logs to SFTP server to ensure that we can prove for example that an SFTP directory was empty at a given date.

On systems that do not keep a trace of what they send or receive, the validation function needs to be developed. Yet, without it, there is no way to ensure that the interface works reliably.

Some people might consider blockchains as a kind of log file or graph. Actually, only the merkle tree is really useful here to ensure that data is consistent and equal on both sides, not the proof of work. Blockchains without proof of work are actually very similar to git. It is thus easier to use git.

Can OpenStreetMap provide routing information in real time or simply based on the time of the day? Probably not. But Google Map does.

Does ERP5 CRM automatically suggest incoming leads based on past track record. Probably not. But Salesforce.com can or could through data.com.

Do medical students have access to large annotated cancer data sets at the core of A.I. based prevention. Probably not. But some private A.I. medical companies do.

Free Software is no longer sufficient to regulate the digital economy through open and interoperable standards. It is no longer in the position to compete with data-augmented applications.  Open Content Text Books - the Free Software of education - are no longer sufficient to transfer knowledge to future generations because this knowledge is now increasingly hidden in secret data owned by private companies and protected by privacy laws. In the case of medecine, impossibility to transfer knowledge to future generations is a violation of the 2nd principle of Hippocratic Oath ("I will respect the hard-won scientific gains of those physicians in whose steps I walk, and gladly share such knowledge as is mine with those who are to follow").

Market does no maximise data welfare

Market does not seem to provide any valid solution to the new issues posed by data economy.

The first theorem of welfare economics does not apply to data or in general to any non rival goods (software, culture, army, police, etc.). One of its consequence is that data welfare is maximised if and only if access to data is free. Unless access to data is free, some sub-optimal behaviour eventually happen, such as the formation of monopolies. But free access to personal data is prohibited by privacy laws. Free access to commercial data is prohibited by trade secret laws.

Current legislation is also biased in the advantage of large companies.

Whenever a data driven company with a data set A acquires a company with a data set B, the value of the merged company is equal to the value of each data set plus the value of possible correlations between A and B. Since there are virtually no diminishing returns in a data driven company, market eventually leads to data monopolies, because data correlation between A and B is only legal after merger. Small companies operating independently can not benefit from such correlations because any collaboration or data exchange among them would be against laws that protect privacy or trade secret.

Solution 1: Free Big Data

Some companies may consider that protection of trade secret has a value that justifies not using data sources that may disclose their trade secrets, even though not using those services also means being less efficient.

For example, a European company willing keep its prospects secret may request its sales workforce to use OpenStreetMap with OSRM self-hosted routing rather than Google Map, and to use smartphones that do not connect to Google, Apple or any US based company. This could be useful for a European company in defence or aerospace, considering the fact that under US Law, any US company has an obligation to deliver specific data to the NSA (no matter the location of its data centres), and that there has been a fairly long track record of US government actions to defeat European, Japanese, etc. competitors in foreign export markets (eg. by providing a copy of competing offers to US companies involved in a market).

For this type of case, it would make sense to build a repository of routing information with average times for each route based on the day of the week, the month, weather conditions, etc. Even though this information is less precise than real time updates, it helps offering a service similar to Google Map navigation but that can operate offline and protects trade secret. It also empowers many researchers or small businesses to contribute new routing algorithms that might even be better than those of Google.

We can generalise this example to any situation where collecting in a public repository freely accessibly data sets that are smaller or less "fresh" that those of data monopolies can be compensated by some competitive advantage resulting from better privacy or more competition. We call this approach "Free Big Data". This approach is similar to Free Software. Anyone can download, process or contribute to any big data set. Each big data set is governed by a process of merge requests.

A first implementation of a Free Big Data repository will be presented on November 12, 2018 by the "Wendelin IA" project sponsored by French Government's FUI programme, following the release in 2010 by Nexedi and partners of Data Publica, one of the first repositories of downloadable public open data. All source code of "Wendelin IA" is licensed as Free Software.

Initial lack of awareness or will is evolving

There are currently two limitations to the "Free Big Data" approach: lack of awareness in the industry of trade secret protection risks and lack of will of governments to open their data.

On the one hand, some large European companies tend to underestimate the risks involved in sharing their data using US based cloud platforms. A typical example is a European aerospace company, which started a corporate-wide "Google Only" policy shortly after its CEO hired an ex-Googler as CTO then later adopted Google Suite and Palantir. It is now nearly certain that NSA has access to some parts of corporate data (due to extra-territorial effects of US Law that no contract can mitigate). Some government experts who witnessed how the Concorde programme was eventually killed by US government actions fear that the adoption of US based cloud platforms will ease similar attacks in the future and question why the company rejected equivalent European cloud solutions licensed as Free Software.

Government on the other hand have been trying to monetise their public data rather than provide free access to it. Legal obligations which were voted in favour of open data are still not always met. Governments in charge of enforcing open data may even try ignore it. Yet, the need to share big data as a key factor for the development of A.I. outside big platforms was recognised. French government published a Request for Information on A.I. data sharing. And some governmental organisations famous for their long history of rejection of open data are now providing initial access to their data.

Solution 2: Big Data Appstore

Not all big data sets can be published or downloaded for free by anyone. Health data has to remain secret in application (also) of Hippocratic Oath unless a proper method exists to anonymise it and publish it in a Free Big Data repository.

Until that happens, this data could be shared in another way: using a Big Data Appstore (see "Have AI. Need Data: How Big Data App Stores close the Data Divide between Startups and Industry"). A Big Data Appstore lets users upload scripts that can process an entire data set, searching for correlation or A.I. models. Models produced by scripts can be downloaded, but not the data that was used to build models. Models can also be executed online.

The business model of a Big Data appstore is clear: developers who write scripts can upload scripts for free but need to pay for processing time on the big data appstore.

Typical organisations that can run a Big Data appstore are: 

• state organisations (ex. health, taxes, etc.);
• large companies in any industry (ex. automotive, aerospace, industrial control, transportation, telecommunications, etc.).

Any large data set that can not legally be transfered to anyone is a potential candidate for a Big Data appstore operated by its owner. Big Data appstore is thus a very effective and self-financed solution to open research to all types of data that do not fit into the "Free Big Data" model.

Solution 3: Paid Curation

Every person or company is a potential source of very useful data published intentionally and explicitly. We could thus consider aggregating all data published intentionally by all companies and individuals into a large data set. Could we still keep a high quality?

Here comes the idea of curation: a company or person is entitled to publish any information he or she is willing to. This can be address, phone number, a list of diseases of that person, place of birth, etc. It could even be the log of IP adresses that have accessed their web site (if this is legal).

Each data that is published by that person or company can be curated on demand, at a modest price. If a search engine is built from that data, curated data will be often displayed first, non curated data afterwards. Any person willing to provide precise information for some reason (usually advertising, but no only) is thus going to pay for extensive curation. 

The Paid Curation model is suitable as an alternative to "Pay-per-click" model of current large platforms to aggregate and finance the curation of aggregated data.

References

• Have AI. Need Data: How Big Data App Stores close the Data Divide between Startups and Industry
• Free Big Data for Free AI
• Free Big Data for Free AI in the context of International Medical Innovation
• NIH Clinical Center provides one of the largest publicly available chest x-ray datasets to scientific community
• Big data sets available for free
• 70+ websites to get large data repositories for free

To many, Cloud Computing is just about storing data online (iCloud, Google Drive) or renting virtual machines (AWS, OVH). But it actually much more: it is in essence the complete automation of IT service provisioning through software which replace human tasks. Cloud Computing is to IT what mechanisation has been to agriculture or robots to mechanical industries.

It therefore does not matter whether cloud is provided from a few central data-centres, from millions of technical rooms or from billions of individual on-premise device. What matters is full automation of the complete life cycle of an IT service. And the core value in Cloud Computing is the software that is used to produce Cloud Computing, not the hardware infrastructure.

This understanding of Cloud Computing was first formalised in 2010 and presented at IEEE International Conference on Services Computing in 2011 (see SlapOS: A Multi-Purpose Distributed Cloud Operating System Based on an ERP Billing Model). SlapOS proves that any cloud computing system requires at least two components, and can be made actually from only two simple software components:

• an ERP that automates ordering, delivering, accounting and billing a service;
• a devops software that automates building, provisioning, monitoring and orchestrating a service.

SlapOS has been deployed commercially to provide Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), Disaster Recovery as a Service (DRaaS), Content Delivery Network (CDN), IoT Management System (IMS), etc. Each time SlapOS plays the same role: automate what IT engineers used to do.

In 2011, the idea of IT service automation was extended to IT consulting service automation as part of a Eurostars project: Cloud Consulting. The PhD thesis of Klaus Woelfel  demonstrated in 2014 that automation of complex IT consulting tasks was possible through machine learning or Artificial Intelligence (see "Automating ERP Package Configuration for Small Businesses"). This type IT automation differs however from IT service  life cycle automation since it consists primarily in automating all tasks that lead to the specification of configuration parameters of an IT service.

We will thus distinguish in this article two categories of automation:

• life cycle automation of already specified IT services (Cloud Computing);
• specification automation of IT service (AI Consulting).

We will focus only on the first case (Cloud Computing) and introduce five major evolutions that are currently happening.

Evolution 1: Free Software Cloud that really works (not OpenStack)

The first notable evolution of Cloud Computing is that there is today no economic rationale to depend on proprietary software and platforms created by Amazon, Google, Microsoft, Qingcloud, Alicloud, UCloud, etc. Anyone willing to create its own cloud operator can achieve this goal in a couple of hours or days thanks to SlapOS. The process is entirely documented with rapid.space, including tutorial, billing and business plan. A company can save up to 90% costs by operating its own cloud.

There has always been a competition between proprietary cloud and Free Software cloud. VMWare PC virtualisation was released in 1998 as proprietary software. Qemu PC virtualisation was released as Free Software in 2003.

SlaopOS bare metal nano-container provisioning and orchestration platform was released as Free Software in 2010 by Nexedi's subsidiary "VIFIB" (see "ViFiB Wants You to Host Cloud Computing At Home"). The proprietary equivalent, dotcloud, was an internal project in 2010 that later became the Free Software called Docker in 2013 with orchestration available after 2015.

Free Software has very often been in advance to implement simple solutions to problems that had been solved previously by complex proprietary software. Using level 3 routing and indirection is an example of a simpler solution (re6st) compared to traditional level 2 switching (open vswitch, virtual switch) which are hard to scale unlike routing protocols without loops (see p. 8 of "Protocoles Réseau V" and Bellman-Ford algorithm).

There is thus no significant innovation in proprietary cloud platforms compared to Free Software based cloud, with one exception: OpenStack. OpenStack was designed as a giant committee driven software suite based on requirements expressed by entreprise users mostly (a bit like OSF/1 in the 90s). It is thus a kind of inferior copy of what VMWare does (or what NiftyName did). And because its design is based on messaging passing architecture which ignores fundamental "promise theory" concept and autonomous computing of Marc Burgess, it is still not stable despite billions of dollar spent and tends to reboot quite often. This fact is now well known by CIOs of large companies who failed creating their own cloud platform. Proprietary cloud vendors even use OpenStack in their marketing as a proof that "open source cloud does not work" and as a way to convince prospects to migrate to their proprietary platforms.

So, if one selects Free Software for its cloud, one should ignore OpenStack as a solution and consider numerous other Free Software solutions that exist, most of which share some of its technical components. Such solutions include SlapOS and to some extent Proxmox. Container frameworks (Kubernetes, Docker) might also be considered in very specific cases where the risk of instability can be mitigated (see "Are Linux containers stable enough for production and why Nexedi uses SlapOS nano-containers instead?").

Evolution 2: Open Hardware not only at Facebook

A second evolution of Cloud Computing is the growing adoption of hardware self-designed by big platforms, some of which was released under open hardware licenses.

Previously, Cloud Computing infrastructure was based on servers from brands such as IBM, HP, Dell, Supermicro, etc. Supermicro was actually the No1 choice of most cloud companies because it provided excellent features at very competitive price. It did not have the "extra layer" of marketing and non standard features targeted at entreprise market which is actually useless for cloud infrastructure. It had the right performance based on simple to grasp standards.

But Supermicro was still too expensive and sometimes not secure enough (just like other big brands actually). Supermicro servers stick to the 19 inch rack standard, which requires a fairly expensive data centre infrastructure (cabinets, power, air conditioning, etc.). And some companies such as Apple had to destroy 45.000 Supermicro servers for security reasons back in 2015.

Under the leadership of Facebook, a community of hardware designers has created a new way to deploy servers and data centres in a cost efficient way: the Open Compute Project (OCP). Rather than following the traditional approach of 19 inch racks and computers with a closed enclosure, OCP considers the data centre itself as the enclosure, and servers in the data centre as mere components that require the most efficient air flow, networking and electricity at the lowest cost. 

An OCP data centre is about 20% to 75% cheaper to deploy and operate.

OCP also opens the door to BIOS replacement with a non proprietary firmware: Linuxboot.  Thanks to Linuxboot, the boot process of the server can be controlled more precisely. Many errors or security holes found in proprietary firmware can be fixed. Linuxboot is now adopted by Google, which incidentally joined OCP after designing its own hardware.

Evolution 3: Edge Computing for low latency and resiliency

Edge computing - also known as Fog Computing - is the third evolution of Cloud Computing. It is a kind of Cloud without data centre. It is based on the same automation tools as traditional Cloud Computing. But instead of having those tools deployed inside servers located in a central location, they are deployed on servers that are located closer to the user.

From a business point of view, Edge Computing can be considered as an opportunity for new players, especially telecommunication industry, to acquire significant market share taken from existing Cloud giants (AWS, Azure, Aliyun, OVH, UCloud, etc.).

Nexedi is one of the inventors of Edge computing with SlapOS about 10 years ago. Nexedi's primary goal at that time was resiliency against terrorist attacks that could easily hit a few data centres and cause of massive collapse of IT infrastructure. The first commercial deployment of SlapOS was made with a French telecommunication company.

Nowadays, Edge Computing can be useful to provide Cloud services for applications that require any of the following;

• low latency;
• resiliency (especially to network outage);
• low cost Internet transit.

Edge Computing servers can be located in different locations:

1. at the regional core network of a telecommunication company: < 10 ms latency;
2. at the antena (LTE/NR) or central office (FTTH): < 5 ms latency;
3. on premise: < 1 ms latency;
4. at the sensor or at the client side: 0 ms latency.

For most applications, 10 ms is enough. Regional core network seems the most efficient place to deploy ab edge architecture. A country such as Germany could be covered with 4 locations per telecom operator. In China, this would be about 100 locations per telecom operator. 

For applications that require low latency or network resiliency, on premise deployment of edge servers should to be the solution. Applications such as content delivery, buffering or resilient routing could be implemented by a small gateway. More complex applications such as hard real time control of actuators may require bigger on premise servers.

And for applications that require no latency at all, direct deployment into the sensor seems to be the only way.

The case of deployment at the antena (more precisely, at the eNodeB / gNodeB) is less clear. With solutions such as Saguna, it is now possible for a 4G/5G network operator to redirect part of S1 traffic from nearby eNodeB/gNodeB to on premise network without having to modify its core network architecture (this is what AT&T does). This "S1 redirection" is sufficient to provide latency of less than 5 ms to nearby campuses without installing any server at the antena. 

Another form of Edge deployment to consider in the future consists of central offices at a sub-regional level with software defined radio (PHY) emulation hosted by powerful servers. This type of architecture is equivalent to the regional core network approach but with more sites. Although it is cost efficient and handles densification much better, it is still uncertain when and how it will be adopted because it departs from current architectures of telecommunication companies. In case it is adopted, it would make sense to merge the concept of Edge Computing and Network Management System (NMS) as it was introduced by Nexedi at MWC 2018: SDR PHY becomes an orchestrated service, just as any other service of a Cloud architecture.

Evolution 4: Service Workers eating 90% of Cloud use cases

Service Workers are a new feature available since 2014 in modern HTML5 Web browser (but not on iOS as of November 2018). A Service Worker virtually turns a Web Browser into a Web Server. It is thus possible to implement complex applications that run entirely offline or which dynamic code is entirely executed in the Web Browser.

Service Workers are a form of implementation of Edge Computing (Case 4) "at the sensor or at the client". With Service Workers, servers and traditional cloud computing in general become virtually useless in 90% of use cases.

If one looks at OfficeJS suite and especially to its word processor based on OnlyOffice code, he or she will realise that it is executing entirely inside the web browser (with the exception of format conversion for now). This means that OfficeJS can serve millions of users at virtually no cost for Nexedi (the maintainer of OfficeJS). This should be compared to Google Drive of Microsoft Office Online which both require huge data centres and cloud infrastructures to operate, because in their case, code is executed mostly on the server side.Service Workers should thus be considered as the fourth evolution of cloud computing. They will eventually kill many server based business models and replace them with client (Web Browser) based approaches. They will let small companies such as Nexedi compete against big Cloud players without having to depend on costly infrastructure.

One of the few things which a Service Worker application can not handle is processing large data with both low latency and consistency. A Service Worker can process small data with low latency (see "OfficeJS Iodide: Ubiquitous Data Science Notebooks for Business and Education") by keeping a copy of all data in RAM. A Service Worker could process big data with high latency through some form of distributed computing such as Web Torrent torrent or (possibly) Tron blockchain. But the combination of low latency and big data is impossible to implement consistently using only distributed computing in our opinion.

Evolution 5: low cost Big Computing

Based on the assumption that Service Workers can not implement both low latency and big data, and that everything else can be implemented with Service Workers, an immediate consequence is that servers with the highest growth will likely be big servers (ex. 256 GB, 20 core) rather than small servers (ex. 2 GB, 2 core) or mid-size (ex. 32 GB, 6 core). Server based infrastructure at the data centre or at the edge (regional, premise) will thus likely look like increasingly similar to OCP hardware used by Facebook for its own big data infrastructure.

This is the reason why Rapid.Space focuses on big servers with low cost and high performance and will likely provide even bigger performance to serve market needs with GPU, TPU, NVME disks, etc. that are now becoming essential parts of any infrastructure.

Conclusion: The Future of Cloud

Cloud Computing has little or no future for any use case that can be implemented with Service Workers. Any use case that Service Workers can not cover will remain on the Cloud.

Here is a tentative list use cases that will remain on the Cloud:

• read oriented applications that analyse large randomly accessed data and require low latency (ex. finite elements, OLAP reporting);
• write oriented applications with high concurrency and high number of transactions per second (ex. big accounting, inventory management).

We could also add some cases where networking is the blocking factor:

• low latency push notification (whenever distributed algorithm consumes too much bandwidth or is incompatible with the design of smartphone OS);
• content distribution (whenever WebTorrent protocols are blocked by legacy infrastructure or DPI);
• IoT buffering (whenever no buffering gateway is available on premise).

Last, any applications where Big Computing must be located close enough to "sensors / actuators" to handle hard real time constraints will still require a form of Edge Computing:

• software defined radio (SDR) where physical virtualisation is implemented by a generic PC;
• software defined industrial automation (SDIA) where physical virtualisation is implemented by a generic PC.

In all those use cases that can not be implemented with service workers or distributed algorithms, a stable cloud operation system such as SlapOS will be needed to automate  ordering, delivering, building, provisioning, monitoring, orchestration, accounting and billing of services deployed on the infrastructure in a way that unifies all use cases under the same paradigm, no matter whether services are deployed in a data centre, on premise or in a sensor.

Amsterdam, October 1st, 2018

Nexedi launched today the beta phase of rapid.space, a low cost, high performance cloud service entirely based on Open Hardware, Open Source / Free Software and circular economy. For 25EUR, developers can reserve immediately a dedicated Open Compute Project (OCP) server with 256 GB of RAM, 20 Intel Xeon core, 4 TB SSD, 10 Gbps network and 1 Gbps shared internet transit. The server is charged 195EUR / month as soon as it is delivered with 390EUR paid upfront.

Jean-Paul Smets, CEO of Nexedi, explains: "rapid.space slashes cloud costs for developers, startup companies, government and corporations. Just like a low cost airline, everything was designed in rapid.space to reach the lowest possible price by removing any non essential feature, component or process. Compared to traditional dedicated hosting and public clouds, rapid.space is 30% to 95% cheaper."

Jean-Marie Verdun of ITRenew, says: "rapid.space utilizes our groundbreaking Sesame products. By applying circular economy practices, we have created the first and only line of high-performance, recertified compute servers and storage systems that meets the same business objectives as new hardware. We start with systems built for the most sophisticated hyperscale data centers and designed according to OCP standards. Then, using our unique processes and IP, we engineer the hardware into a line of products with reliability, quality, and LTV performance comparable new OEM systems but available at less than half the price."

Steve Helvie, VP of Channel Development at Open Compute Project: "Jean Marie and the team at ITRenew continue to do a great job enabling local partners like Nexedi through a circular economy business model focused on OCP solutions."

Rafael Monnerat, VP for rapid.service, explains: "rapid.space is entirely based on Open Source / Free Software. This means that any of our customers can contribute to any part of rapid.space platform: provisioning, monitoring, disaster recovery, accounting, billing, etc. It also means that any of our customers could copy our platform and become their own cloud provider. And, thanks to Zero Knowledge technology, we make sure that we do not store in our platform any sensitive customer information."

Jean-Paul Smets concludes: "rapid.space is the first cloud operator that implements full transparency of its platform through full source code disclosure. Our target market consists of developers looking for big servers at low cost. rapid.space applications include: big data, A.I., continuous integration, disaster recovery, ERP backend, e-commerce backend, Virtual Radio Access Network (VRAN), etc."

References

• rapid.space official web site (beta)
• Business Model of a Low Cost Cloud Operator
• High Performance Virtual Machines for Database Applications
• Unleashing the Full Economic Potential of Cloud Computing in China
• SlapOS: edge cloud computing and orchestration
• Open Compute Project

Permanent Link

• https://www.nexedi.com/NXD-Press.Release.Rapid.Space.Beta.Phase

Contact

• Jean-Paul Smets -  Telephone: +33(0)-6-29-024425 -  Email:  jp@nexedi.com
• Rafael Monnerat -  Telephone: +33(0)-6-09034119 -  Email:  rafael@nexedi.com

Pictures

Open Compute Compute Foundation endorsement of Rapid.Space as new community member.

Steve Helvie, VP of Compute Compute Foundation, with Rafael Monnerat, COO of Rapid.Space.

IT Renew booth at OCP Summit in Amsterdam.

About Nexedi

About Rapid.Space

About ITRenew | Data Center Asset Disposition and ITAD

About Open Compute Project

Notebooks are a simple, fun and efficient way to create business reports or play with data science libraries such as scikit-learn or pandas. They are also a great tool for interactive visualisation of data. We have been using Jupyter notebooks at Nexedi as part of Wendelin out-of-core Big Data project. Using Jupyter, we could increase the productivity of engineers in charge of producing reports about the structural health of wind turbines in Gemany. Using Jupyter, we could also provide a tool to analyse and visualise sales trends of a trading company in China. We are currently integrating Jupyter-Lab as the default IDE of SlapOS Edge Computing software.

Jupyter has been of tremendous help at Nexedi. We will keep on using it, especially Jupyter-Lab for server-based software development. But Jupyter has two major drawbacks: it does not scale and it is not ubiquitous. It does not scale because serving Jupyter notebooks to thousands or millions of users requires powerful servers and complex software setup, which tends to break the kind of efficiency or simplicity one can experience as a single user. It is not ubiquitous because one has to install Jupyter to use it, and most users today no longer want to install anything and instead expect everything to be free.

Mozilla Iodide: HTML5 notebook

Along came Iodide, a project originated by Mozilla and lead by Hamilton Ulmer and Brendan Colloran.

Iodide is an HTML5 and Javascript notebook that works similar to Jupyter. But instead of processing data on server side, it processes data directly in the browser. Iodide supports dynamic loading of virtually any Javascript library, including plot.y and mpld3 for visualisation and numjs for linear algebra. It also supports natively ndarray data structures which are commonly used in Javascript (see for example scijs).

Iodide is distributed as a collection of static files that can be hosted on a low-cost Content Delivery Network (CDN). There is no need of any kind of application server to deploy Iodide and disseminate it to million users.

The absence of server side component is what makes Iodide so different and so much better than Jupyter for most applications. With Iodide, a multinational corporation can distribute to all its employees a business reporting framework packed with the best of A.I. libraries without virtually no investment in any form of license or infrastructure.

Pyodide: lightspeed data science programming

Although Iodide is written in Javascript, it can load extensions and support other languages. Pyodide is an extension for Iodide initiated by Michael Droettboom that provides a python interpreter compiled in Web Assembly (WASM). Pyodide comes with a growing number of libraries for visualisation and data sciences. Thanks to Web Assembly, those libraries run at near native speed inside a Web browser.

It is therefore possible to run within the same runtime - the Web Browser - both Javascript and Python code which share the same ndarray data structures.

A surprising consequence of this dual headed runtime is that it slashes drastically the time it takes to develop interactive data analysis and visualisation Web based applications. During our initial evaluation of Iodide, we tried to develop a simple application which we had previously developped using two other frameworks: Bokeh and Wendelin.  Here are our results.

In both cases, the assigned developer was using the framework for the first time. Before Iodide existed, the choice was simple: Wendelin provided enterprise grade features that are needed to industrialise a data collection and processing project whereas Bokeh provided rapid development. It was one or the other.

We found that with Iodide, we were able to provide even faster development than Bokeh. The reason is simple: a Javascript function in Iodide can call a Python function which can itself call a Javascript function. All the extra layers or wrappers that had to be considered between Python and Javascript in both Wendelin or Bokeh were gone. Developing interactive applications that combine Python and Javascript becomes instant in Iodide.

It is also much easier to integrate Iodide to Wendelin than Bokeh or other frameworks based on an application server. With RenderJS, Iodide can become a UI gadget component, just like other existing components (graphs, spreadsheets, etc.). This is because one of the key concepts of Iodide is to have no dependency to any application server and to be based only on static files.

Nexedi supports Iodide

Nexedi therefore decided to invest in Iodide so that it becomes the standard rapid application development environment of the next generation Wendelin platform by combining benefits of Wendelin and of Iodide in the same environment. Iodide will also become a standard reporting tool of OfficeJS HTML5 suite with close integration to ERP5 open source ERP/CRM.

A senior developer, Roman Yurchak, was sponsored by Nexedi to contribute to the core of of Pyodide. This includes adding dynamic module loading (from arbitrary Web URL), automating Numpy test coverage and porting scikit-learn.

Meanwhile, Richard Szczerba, a young developer has been finalising and extending the work of Laurent Sebellin (Nexedi GmbH) to integrate Iodide into OfficeJS. OfficeJS Iodide can now save notebooks in ERP5 or Dropbox, stream data from remote storages and - of course - operate offline.

This investment will keep on in 2018 and 2019 thanks to funding secured from France's FUI public research fund. Any developer or intern is welcome to join Nexedi for short to long time and contribute to the development of Iodide or Pyodide (write to jobs@nexedi.com).

OfficeJS Iodide Notebook

OfficeJS Iodide Notebook is a new member of the OfficeJS appstore that provides a simple way to use, edit and manage, online or offline, multiple Iodide notebooks stored locally or on remote online storages.

OfficeJS is an HTML5 appstore that includes an OpenXML compatible office suite (text, spreadsheet, presentation), an HTML5 compatible office suite (text, spreadsheet, illustration, imaging) and a few applications for daily use (expense tracking, bookmarks, PDF, music player, etc.).

Thanks to service worker technology powered by CribJS, OfficeJS HTML5 applications can operate both online and offline (only on latest IOS). This is how OfficeJS Iodide Notebook can operate entirely offline.

Thanks to storage abstraction powered by JIO, OfficeJS HTML5 applications can store data locally inside the browser (IndexedDB), remotely onto online storages (Dropbox, Google Drive, WebDAV, ERP5, etc.) and synchronise both. This is how OfficeJS Iodide Notebook can store and retrieve the notebook's jsmd text to and from a wide variety of storage without depending on any application server or changing Iodide's code (currently, the last run of any cell is stored in localStorage by Iodide).

And thanks to RenderJS, a lightweight component framework, OfficeJS applications run fast on slow devices (low-end smartphone, ARM based chromebooks, etc.) and can integrate well with other frameworks (Angular, REACT, etc.). RenderJS is the framework that provides to OfficeJS Iodide Notebook the ability to add and display a list of notebooks that support full text search.

All OfficeJS applications are implemented as a collection of static assets (HTML, CSS, JS, etc.) that can be encapsulated into a ZIP file and hosted on any static HTTPS server. No application server is needed. Same applies to OfficeJS Iodide Notebook.

Building OfficeJS Iodide Notebook

The build process involves the following steps:

• build Iodide on a remote Linux virtual machine together with various python libraries required (see https://www.erp5.com/erp5-HowTo.Build.Pyodide for a complete description);
• combine into a single zip file Iodide static files built previously together with JIO, RenderJS  and a manifest file used for Progressive Web Applications (PWA);
• upload the resulting zip file into OfficeJS appstore;
• wait for publication validation.

The complete OfficeJS Iodide Notebook consists of about 130 files for a total of 40 MB of assets. It was tested on a load end ARM based laptop with reasonable performance.

Iodide vs. Jupyter Benchmark

We then decided to run some tests on Iodide to ensure that it can do the job in a real business context. 

To compare Iodide and Jupyter, we selected a python based Jupyter notebook that retrieves sales records from an ERP (ERP5), processes them and visualises results with matplolib. This report is used in a daily business situation to analyse best sales in a trading company and display trends.

We thus created an equivalent notebook using Iodide and Pyodide. We used JIO to retrieve sales record from ERP5 and Plotly.js to visualise results. The rest of the code, mainly in python, is similar to the original Jupyter notebook.

We achieved in rather short time to make a Pyodide notebook that provides same results as the oiginal Jupyter notebook.:

Here are our observations:

• Running Pyodide requires a lot of RAM (ex. 8 GB) for typical business data processing, else the web browser can crash;
• We could not use chinese characters because the xlrd library failed to read them properly when importing the data into pandas (we had to convert data first to CSV);
• Initialising pandas took 3 minutes on Chrome while it took 10 seconds on Firefox;
• graph visualisation is based on seaborn library which is not available (yet) in Pyodide  since it has scipy as a prerequisite (we used matplotlib instead);
• dynamic loading of python modules is possible by calling something like pyodide.loadPackage('https://foo/bar/numpy.js' where numpy.js is a Javascript representation of a python module.

Dynamic loading of python modules from a URL is really a useful feature since anyone can convert existing python code into a Javascript representation and publish it somewhere on the Web. There is thus no longer any need to rebuild Pyodide to extend it. This feature, sponsored by Nexedi, is now described in Pyodide's documentation.

JIOdide: Data Access Jedi

Interfacing Iodide with data sources automatically has been a kind of challenge and a dilemma until now:

• anything would be possible through a server based data access proxy, but then Iodide would lose its unique "static asset only" nature;
• or data would have to be published over https in an ad-hoc way.

We solved this dilemma in OfficeJS by integrating JIO as part of our standard Iodide build. JIO is a library that provides a unified way to local and remote data sources from the Web browser:

• create data;
• read data;
• write data;
• query data.

It applies to both data (ex. files as in a file system) and records (ex. lines in a relational database) through a unified API.

Thanks to JIO, we were able to eliminate the need of exporting data to CSV/Excel. We could instead to load data straight from our ERP (ERP5) through simple JIO calls.

The illustration bellow shows an example of accessing ERP5 data from Iodide:

This data can then be turned into a graph:

JIO can operate without any server side proxy or adapter. It can be extended to support more data sources (ex: Google Drive, Amazon S3, Qiniu,etc.). Its only (important) limitation is that it requires good support of cross-origin resource sharing (CORS), a standard feature of HTTP protocol that some cloud providers still refuse to support. Without CORS, a simple HTTP proxy is a must.

Daily Iodide at Nexedi

A few programmers at Nexedi started using Iodide and Pyodide as a scratchpad to test snippets of js and python code.

Iodide was introduced as a playground for third party system integrators to discover JIO library and to access  ERP5 records from Javascript or python applications.

Iodide notebooks are saved and synchronised in Nexedi's ERP5 instance, just like any other corporate document.

We hope to increasingly use Iodide to generate business reports for our own use or for our customers. Thanks to RenderJS, we plan to integrate Iodide into our business applications as iframes in the same way as we do currently with complex spreadsheets.

Future improvements

Based on our current use of Iodide and Pyodide, we would like to see or contribute to the following improvements:

• support of scikit-learn (ongoing);
• generic and extensible import / export API in iodide (rather than ?gist= type URLs) so that we can directly plug JIO into Iodide's core;
• generic and extensible python module loading API so that it is possible load python modules from different kinds of Web based repositories;
• Iodide API for read access to JSMD and Tools, with a callback function to inform if the state has been changed in any way so that it is possible to inform the user that they have unsaved changes. 

Article Contributors: Richard Szczerba, Sven Franck, Valentin Benozillo, Jean-Paul Smets.

American companies have been forbidden to supply components to ZTE, especially in areas related to 4G and 5G infrastructure. Rather than slowing down the industry, this commercial incident could accelerate the adoption of a new technology for 4G and 5G mobile infrastructure: virtual radio access networks (VRAN).

Traditional 4G and 5G infrastructure depends on electronic components, some of which can only be sourced in the USA. However, a European company called Amarisoft was able recently to replace most of those components with pure software that runs on a generic PC rather than on specialised hardware. Amarisoft was able to emulate the entire 4G and 5G protocols including the physical radio (PHY) using a standard micro-processor. Emulated protocols include recent 5G NR protocol, carrier aggregation over 1 Gbps, 8x8 MIMO, eNodeB, core network, etc. Initial benchmarks even show better performance than traditional hardware based solutions that are currently deployed by telecommunication operators, at a cost which is 5 to 10 times cheaper.

By combining standard PC, remote radio unit (RRU), Amarisoft licensed code and open source network management system (NMS), anyone could potentially provide telecom infrastructure at a fraction of current typical costs, with no dependency to american suppliers. Companies such as AW2S (France) can provide open standard RRU designs. Similar designs are also available from Chinese suppliers. Transistors are usually sourced from NXP (Netherlands) which merger with Qualcomm has been de facto cancelled by Chinese government. Inspur (China) has a complete range of PC designs based on open standards.

Overall, nothing would actually prevent a company such as ZTE to drop current suppliers from USA and adopt VRAN technology instead based on a combination of European and Chinese suppliers. Although some industry experts initially had doubts about the possibility to emulate 4G or 5G entirely with software, various field tests conducted in various countries in Europe and America demonstrated that VRAN... works better for cheaper. The article published in 2017 by Iain Morris "Startup Could Replace Ericsson, Says Orange" was the first sign of industry acceptance of VRAN and software defined radio (SDR) for 4G and 5G.

Yet, some components of Amarisoft VRAN ecosystem are still sourced in the USA:

• Intel micro-processors used to run Amarisoft software;
• FPGA (Altera, Xilinx, Lattice, etc.) used by most RRU vendors;
• analog-digital converters provided by Analog Device.

Replacing those components with other sources is not a big challenge. Amarisoft stack could be easily ported to ARM architecture, Sunway or - even better - to the recent open instruction architecture RISC-V. For example, the cost of turning RISC-V design into an actual product is only between 40 and 80 million RMB. Many companies in China that produce hardware for government could easily bear that cost. 

FPGA used in RRU could be replaced in a first step by ASIC. ASIC can be developed very quickly nowadays by companies such as Zoro (Israel) or in Shenzen area using software simulation. Using ASIC would buy time to prepare an alternative to current FPGA vendors in a second step and break the current dependency to USA on this particular component.

About high performance analog-digital converters, some are already produced in Europe for specific applications. Larger orders could lower their cost down to a level comparable to Analog Device.

Decisions to ban exports rarely produce expected results. Shortly after China decided to ban exports of rare earths to USA and Japan, Japan was pushed to search for an alternative supplier and discovered a semi-infinite' trove of rare-earth metals which means that China no longer controls 93-97% of the market. The US decision to ban exports to ZTE may also push China to adopt alternate suppliers and technologies which will eventually stop the world's current dependency on key american suppliers in a context of increasingly unstable international commercial relations.

Back in 2017, Nexedi tried to move to move to HTTP2 using Apache's HTTP2 module. It was a failure: named servers would show the content of another web site after switching to HTT2. Also, we found that Apache has complex process leak bugs, which although are supposed to be fixed, actually are not after 7 years. Since those bugs were too hard for us to fix considering our limited know how in C/C++ concurrent memory management, we decided to consider alternatives.

We first thought using NGINX which is already available as part of our Open Source / Free CDN Software in SlapOS. But NGINX has its own share of issues too which are not easier to fix than those of Apache.

Then, we discovered a recurring pattern: every 10 years, a new HTTP server is released. It becomes the most popular HTTP server within 10 years then declines. Apache was released 20 years ago, NGINX was released 10 years ago. Both Apache and NGINX brought significant progress to the Web. Apache is declining. NGINX became trendy 10 years ago and is still growing, but does not really solve our problems with Apache.

So, we considered what would be the successor to Apache and NGINX.

We believe that Caddy could be that successor because it brings to the Web two features that are absent in both Apache and NGINX:

• a stable implementation of HTTP2/QUIC;
• a programming language that drastically reduces the risk of memory leaks or process leaks when implementing extensions.

With HTTP servers nowadays able to process 400.000 requests per second on low end hardware, memory leaks and process leaks have become a major risk. Caddy does it better because its programming language, Go, does it better. It includes a garbage collector that solves the problems of memory allocation. And it is based on a clean model of concurrency with the algorithm of scheduling: work stealing. All this at nearly the same raw speed as compiled C or C++.

We thus considered that Caddy was the best option for us. Rather than fixing problems in Apache that could not be solved after years, we decided that we would rather adopt Caddy, benefit immediately from good HTTP2/QUIC and then contribute to Caddy whichever extensions are still missing. Programming in Go language without memory errors or process leaks is much easier for most python programmers than programming in C or C++. By adopting Caddy, we actually adopt a high performance HTTP2 server to which anyone in Nexedi can contribute without much risk. This is invaluable in our opinion.

Since we use SlapOS everywhere in Nexedi for devops, we wanted to achieve an important goal: seamless transition from Apache to Caddy, with maximum feature compatibility. In order to do so, we started by adding tests for our current Apache frontend, testing Apache compilation, instantiation and automatic configuration based on SlapOS Master input. Each test tries to cover one specific feature which we are using in our frontends: how files care cached, how named virtual hosts are created, etc.

Once we covered our Apache frontend with enough test cases, we used Test-Driven-Development (TDD) approach to achieve the same functionality with Caddy as with Apache for selected minimal viable features. Since we have been using the same test suite for both Apache and Caddy, we could be sure sure that the transition would be seamless and automatic at SlapOS level. We could be sure that nothing had to be changed on instance configuration, yet behaviour would be the same.

Once all test cases passed with Caddy, we decided to start the automated migration process on about 3000 named virtual hosts. It completed after about 20 minutes. No incident happened. All sites suddenly started to provide faster response thanks to HTTP2 optimisations.

Our next step will be to extend Caddy and add QUIC support. 

And maybe some day, we will move Caddy to a Cython based HTTP server, if we can pass the same test suite.

The text below is based on my 2016 notes on Python, uvloop and Go after a colleague came back from Europython conference. It shows that on python side, even though uvloop lightly wraps libuv that is known to be fast, the performance drops significantly right after pure python code starts to be added into request handler, and the end result is that python server becomes more than an order of magnitude slower compared to go version.

However in 2018 we still hope that python world could be improved for implementing modern networked servers via bringing in combination of Cython and coroutine/stackless-based approaches such as gevent and pygolang. All the pieces are there, but scattered. We just need to have the focal point where they all could be tightly integrated together.

Back from Europython 2016

It is good to hear that Python world is trying to catch. As I was using Python for many years and also was playing with Go in recent times and know both, on my side I'd like to provide some feedback. But before we begin, let me show you something:

Above are benchmarks for HTTP servers taken from uvloop blogpost for which HTTP handlers were modified to do some simple text processing instead of doing only 100% I/O. The graph shows that once we start adding python code - even very simple one - to server when handling requests, the performance is killed in Python case.

Now here is the explanation:

Modern web-servers are built around so-called reactors. These are specialized event-loops which communicate with kernel in efficient ways (solving c10k problem) using things like epoll() on Linux, kqueue() on FreeBSD etc. Reactors are basically loops when you subscribe for events on file-descriptors, and receive notifications via getting corresponding callbacks. Then every CPU is tried to be loaded which leads to something like M:N schemes where M is threads spawned for every CPU and N is many connections which corresponding thread handle via callback-events.

libuv is a C library which wraps OS-specific mechanisms for organizing reactors in uniform way. uvloop wraps libuv via Cython and expose its service to Python. Go has builtin network scheduler in its runtime which is similar to what libuv is doing.

Now original MagicStack benchmarks (python, go) actually only receive request and send back the payload. Since those two paths are all I/O and are handled 99% by only reactors, what those benchmarks really measure is how well underlying reactor libraries perform. I'm sure libuv should be a good library as well as Go runtime is well done and tuned, and MagicStack benchmarks confirm that.

However any real server has some control logic and things to do in its HTTP handlers and that is not there in MagicStack benchmarks. And once we are starting to add code for handling requests it becomes not only I/O even for cases when people tend to think performance should be I/O bound.

In Python case executing pure-python code is known to be slow. The fact that in original MagicStack benchmarks performance dropped on the floor while they were using HTTP parser written in Python justifies it -- performance recovered only when they actually switched to using C library to parse HTTP requests. For python the trend is: whenever we need performance we need to move that code to C and wrap it. But experience also shows that not all code can be so well localized and slowness often remains scattered throughout whole python codebase. (I'm not considering here cases when we move everything to C and wrap only something like top-level main() because then there is nothing left on Python side)

So let's simulate, at least in part, of being real web-server and doing some work in HTTP handler. For an example workload I choose to analyze characters from request path and see how close they are to name of our several websites. Here is the workload for Python case (full source):

and the same for Go (full source):

For every request we create classifier object, and then for characters in request path do some several method/function calls and lookups in website-name strings. In the end classification statistic is returned to client:

This is not a big workload - rather small one - and doubtfully it is useful, but it shows what start to happen performance-wise when there is some non-trivial work in the handler. About performance: benchmarking was done via e.g.:

on the same machine (my old Core2-Duo notebook) with output like:

For N(input characters) > 11 "helloworld" was simply repeated on request path needed number of times. For those interested actual benchmarks run and numbers are here.

At this point I'd like to ask you to look once again at the picture in the beginning of the post: when there is only 1 input character (only "/" in path) Python and Go perform close to each other, though python is already slower. For 11 input characters ("/helloworld" in path) the difference is ~ 2.8x. For 101 input characters the difference is ~14x -- python becomes more than an order of magnitude slower.

So to me this once again shows that even with having libuv integration, for any real use-case, as long as there is python code in the request handler performance will be much slower compared to Go.

On the other hand, the Go case shows that it performs rather well, usually without wrapping anything as most of the things can be and are actually written in Go itself. For example 90% of Go runtime and in particular libuv analog - Go's network scheduler - is implemented in Go itself which shows the language can be used to get good close to native speed while working on high-level-enough-language similar to python.

I'd like to also add that MagicStacks benchmarks are not reasonable as they set GOMAXPROCS=1. In simple words this means that while Go support for multicore machines is very good, it is artificially limited to be using only 1 CPU on the system. In other words Go is artificially constrained to behave with something like GIL in Python world. Without above GOMAXPROCS=1 setting, Go by default uses all available CPUs and for cases when there is not much contention between handlers performance usually scales close to be linearly. I'd like to remind that we are already using 8-CPU machines at Vifib and Python practically cannot use more than 1 CPU in single process because of GIL.

Some thoughts on concurrency

I want to also add some words about Python approach to concurrency and building parallel servers. To me with asyncio / async/await they are just creating different world for no reason - as every part of software has to be adapted to async & co stuff, and there becomes, yes a bit mitigated, callback spaghetti.

On the other hand in Go it is still the same serial world and we add channels via which you can connect goroutines. Each goroutine runs serially but can send data via channels just like via pipe. To me it is significantly more better and uniform approach even in terms of human thinking so in this case Go adds not only performance but more productivity. And I can tell this with confidence, because I was in reactors/asynchronous programming for a while even implementing my own reactors sometimes.

The thing is: computer is already a state machine (it runs each assembler instruction via leveraging state machine inside CPU) then we have state machine in OS to run several programs / threads. But it is harder for humans to implement state machines compared to serial programming and communication. I mean it is harder for humans to understand and harder to implement state machines. Thus what makes sense is to implement state machine at lowest-possible level and then give programmers the feeling of that they have a lot of serial processes and adequate communication primitives.

The reactor is itself a state machine. Go does it at runtime and hides providing to user serial goroutines and channels while other parties just throw the complexity of "asynchronousity" to developers. For me as a developer, what would be a better approach for python is to actually integrate libuv deeply in its runtime and give developers green (= very cheap) threads and communication primitives. The sad thing about it is that stackless (2) was doing things in this direction for years (it started around beginning of 2000's - the same time when first GIL removal patches started to appear), but despite this approach was with us for a long time there is seemingly almost zero chances for it be merged into CPython and people reinvent "cute" things (async/await asyncio) and throw twisted complexity of programming to developers.

Future directions

In 2016, I would have said that low performance and lack of adequate approach for concurrent programming sadly makes Python to be a not so appropriate language to implement loaded webservers today.

However, the existence of cython combined with a clean concurrency model based on technologies such as gevent and pygolang could change the situation if both can be tightly integrated into cython's static cdef code rather than being all scattered as it is today.

The growing importance of Artificial Intelligence in medical and healthcare industry is raising concerns about possible market excess that could lead to the end of hypocratic oath as we know it today. Just like Free Software which could efficiently regulate the software industry at the end of the 90s and reduce the risk posed by monopolies, Free Big Data will play a key role in efficiently regulating the medical Artificial Intelligence market and create the conditions for Free Artificial Intelligence that can reach every human being on earth.

The first presentation, Free Big Data for Free AI in the context of International Medical Innovation, demonstrated how the Wendelin IA project and platform jointly developed with INRIA, Telecom ParisTech and Abilian can contribute the Free Big Data movement.

The second presentation, Teralab: AI Infrastructure for Research, provides a summary of a human and technical success story of public policy in the field of Big Data and Artificial Intelligence that was initiated by Institut Mines Telecom. Nexedi could contribute to this success story through the SlapOS edge computing and cloud orchestration system.

Press Release

Barcelona, February 26th, 2018

Nexedi is demonstrating during Mobile World Congress 2018 the first preview of a Network Management System (NMS) designed for low latency 5G and 4G networks and published under Free Software licenses. Combined with Amarisoft's 5G / 4G stack, it is now possible to deploy a full featured commercial telecommunication network that relies entirely on software, runs on commodity hardware and supports the provisioning of Edge Computing value added services such as artificial intelligence (A.I.) offloading for autonomous cars or IoT buffering gateways.

Nexedi's Network Management System (NMS) is part of the Open Source Telecom Vendor (OSTV) project, a joint initiative of Amarisoft, AW2S, BJT Partners, University of Paris-Diderot, Nexedi and Splitted Desktop Systems to create an open source framework for the deployment of 4G and 5G networks based on a distributed architecture. OSTV project is sponsored by French government's innovation programme "Gands Défis du Numérique".

Jean-Paul Smets, CEO of Nexedi, explains: "Ten years ago, Nexedi was one of the inventors of distributed cloud computing, also known as Edge Computing. We first extended our SlapOS multi-tenant distributed cloud architecture into a Network Management System (NMS) by supporting the orchestration of Amarisoft LTE / NR virtual radio access network (VRAN) services at the edge. Our NMS supports subscriber management, accounting, monitoring, billing, issue tracking workflow, inventory management and provisionning of value added services at the edge.".

Rafael Monnerat, COO of VIFIB, adds: "SlapOS relies on re6st and babel protocol created at IRIF to optimise latency on a hybrid wired and wireless IPv6 mesh network that acts as a resilient backhaul. By eliminating the need for centralised core network, we already reach effective end-to-end application average latencies of less than 30 ms over LTE with standard user equipment (UE). With the upcoming NR radio announced by Amarisoft at MWC 2018, we expect to reach effective end-to-end application average latencies of less than 5 ms. We believe that both are sufficient for critical applications such as autonomous driving." 

Jean-Paul Smets concludes: "In addition to leading innovation and low latencies in telecommunication infrastructure industry, the OSTV project also leads to dramatic cost cuts. By using OSTV, France's white areas could be covered with 4G+ for 200 million euros instead of 3 billion euros with mainstream technologies."

References

• OSTV Network Management System Leaflet (PDF)
• Live demonstration of Amarisoft new 5G NR software, 4G gigabit range eNodeB and tiny NB-IoT eNodeB

Contacts

• English, French:
  • Jean-Paul Smets, Nexedi CEO, Tel: +33(0)-6-29-02-44-25, Email: jp (at) nexedi (dot) com
• Chinese:
  • Ni Yan, Nexedi, Tel: +86-13871490101, Email: ni (dot) yan (at) nexedi (dot) com
• German:
  • Sven Franck, Nexedi, Tel: +33 (0)-6-61-76-64-23, Email: sven (dot) franck (at) nexedi (dot) com
• Japanese:
  • Yusei Tahara, Nexedi KK, Tel: +81(0)-804068-9210, Email: yusei (at) nexedi (dot) com

About Nexedi

Nexedi is the largest publisher of Open Source / Free Software in Europe with 15 million lines of original source code. Nexedi enterprise software portfolio covers business applications (ERP5), edge cloud computing (SlapOS), big data (Wendelin), distributed transactional NoSQL database (NEO), HTML5 productivity (OfficeJS), progressive offline web applications (RenderJS, JIO), software defined resilient networking (re6st), devops (Webrunner) and multimedia conversion (cloudooo). 

With presence in Europe, Asia and Americas, Nexedi addresses a wide range of industries ranging from aerospace, apparel, banking, telecommunication, healthcare to government sectors. The Free Software nature of Nexedi solutions eliminates licensing costs, provides full freedom to update or customise the system as business requirements change and let corporations capitalise their know how with no single vendor lock-in. Nexedi provides 24/7 support to corporations and governments wishing to migrate their mission critical applications to Free Software solutions.

More information: http://www.nexedi.com

About SlapOS

Born in 2009, SlapOS is the only Open Source / Free Sofware solution for edge computing that has been deployed commercially and successfully. Based on a Hyperconverged Orchestration System (HyOS) that consistently integrates provisionning, devops, accounting, billing, monitoring, orchestration and automated disaster recovery, SlapOS can be used to implement in a few days or weeks public clouds, distributed mesh clouds, big data clouds, hyperconverged infrastructures, IoT appstores, 4G/5G networks or edge computing. SlapOS technology is at the core of Teralab, a big data platform that was awarded by the Big Data Value Association and used by dozens of multinational corporations. SlapOS has been deployed together with ERP5 at Airbus, Mitsubishi, Aide et Action, Capago, etc.

More information: http://slapos.nexedi.com

Legalese

ERP5, SlapOS, Wendelin, NEO, OfficeJS, Re6st and Cloudoo are registered trademarkes of Nexedi. All other trademarks are the property of their respective owners.

NEO, the distributed transactional NoSQL object database of Nexedi, has reached on November 24th 2017 a storage size of 100 TB on a redundant array of inexpensive computers. This success demonstrates that both ERP5 open source ERP/CRM and Wendelin out-of-core Big Data platform can power the biggest commercial bulkloads, both for transactional and data science applications.

The data stored in NEO consists of 30 big data streams of about 3.3 TB, each of which can be accessed sequentially or randomly.

In order to achieve this result, Nexedi has been running a NEO ingestion test drive for 40 days on 6 inexpensive Dedibox computers, each of which with 3 SATA disks of 6TB, 1x Intel® Xeon® E5 1410 v2 and 64 GB RAM. 30 concurrent fluentd streams of data have been ingested into ERP5 / Wendelin platform powered by NEO at an average growth rate of 2.5 TB per day. The NEO cluster was configured with 30 independent Zope application servers and 18 independent replicated storages for a total disk usage of 88.6 TB. A compression factor of 43.5% was observed on the random data that was ingested in this test run.

NEO database relies on an innovative protocol that turns a cluster of independent storage engines into a single transactional storage space. NEO currently supports MariaDB, MySQL, SQLite and POSIX filesystem as possible storage engines. For the current tests, MariaDB has been used with two different storage backends: RocksDB and TokuDB. Both RocksDB and TokuDB have shown similar peformance: RocksDB write performance was more consistent whereas TokuDB read performance was a bit better. Detailed test report will be published soon.

Nexedi has developed for about ten years a distributed transactional replicated object store powered by MariaDB: NEO. We use NEO to store all kinds of data: records for ERP5 or large arrays of floating points for Wendelin big data platform. NEO, ERP5 and Wendelin are implemented entirely in python.

NEO's architecture

NEO is an implementation of ZODB with features such as multi-master replication. ZODB is an ACID key-value DB that preserves history by default. ZODB's simplest backend is a simple logging file.

NEO's architecture consists of:

• storage nodes: their role is to access or store data in a sharded and redundant way;
• master nodes: their role is to ensure consistency of transactions;
• control nodes: their role is to control the operation of the cluster.

NEO storage nodes are the only nodes that persist data. Storage nodes rely on MariaDB database. You can see the code of our MariaDB backend at https://lab.nexedi.com/nexedi/neoppod/blob/master/neo/storage/database/mysqldb.py

The _setup method contains the schema of tables:

• config, pt: these 2 tables are so small that we can ignore them
• trans: transaction metadata (1 row per transaction)
• obj: object metadata (1 row per object)
• data: raw data (we do data deduplication so several rows of obj can refer to the same row in data)
• bigdata: in addition to max_allowed_packet, both InnoDB (by configuration) and TokuDB (hardcoded) have a maximum row size limit, so the biggest records are split here
• ttrans/tobj, which are similar to trans/obj, is a temporary place for metadata being committed (they're a replay log, and they're empty when there's no transaction being committed)

Since ZODB is key-value database, it translates in NEO terms as:

• key maps to obj.oid
• value maps to data.value
• tid is transaction id

NEO is an ACID database with 2-phase commit and we do 2 or 3 SQL commits per transaction:

• a first one (tpc_vote) to make sure that all the data to commit is actually stored durably somewhere
• a second one (tpc_finish) to mark the transaction as committed (ideally, that would only flip 1 bit): lockTransaction method
• after a successful commit, the storage nodes are notified that metadata can be moved from ttrans/tobj to trans/obj, making the new transaction readable: this is done in the unlockTransaction method, and since there's no urgency to commit after that (we're able to replay the transaction in case of crash), we wait at most 1 second in case that another transaction would be committed before

The _connect method disables autocommit. The performance would be catastrophic if we didn't minimize the number of fsyncs.

Currently, each NEO storage node accesses through a Unix socket to an independent MariaDB server. We use MariaDB mainly to store BLOBs. But, as part of ERP5 and Wendelin's architecture, we also use MariaDB independently of NEO as a kind of separate datawarehouse that it used to index certain specific properties hidden in each BLOB.

We focus in this document mainly on NEO and the storage of BLOBs in MariaDB.

MariaDB storage engines: InnoDB, TokuDB, RocksDB

We have compared in 2017 the ability of the different storages of MariaDB to handle a large quantity of BLOBs. Here are the results:

• InnoDB: good data integrity but poor scalability;
• TokuDB: good scalability but data integrity issues;
• RocksDB: good data integrity, mixed results about performance.

We also found that all MariaDB storage engines failed to optimize property index selection and even to take into account hints for forced index selection.

Here are some details from the experiences we have conducted.

Issues

Most of them are reported upstream and aren't described here. They are listed at the end of this document.

Optimizer

It seems that in the case of NEO, optimizer issues aren't as severe as might have been feared:

• FORCE INDEX hints would work as expected and they have been added everywhere if the code.
• The only queries with potential full scan despite appropriate index could be rewritten differently. The workaround is enough for the moment.

However, optimizer issues can remain problematic for queries other than SELECT: DELETE/UPDATE offer no way to specify index hints.

Besides NEO, we do experience sub-optimal index selection within boolean fulltext search as described in MDEV-7250.

TokuDB: DELETE sometimes do nothing

This concerns the unlockTransaction method mentioned above. Randomly and very rarely, when a NEO DB restarts and checks for transactions to recover, it sees lines in ttrans/tobj for transactions that were committed long time ago. This results in integrity errors because NEO moves these lines again to trans/obj, which already have them.

As shown above, the move from ttrans/tobj to trans/obj is really done in a transactional way. A row must never exist in 2 tables at the same time.

It even happened for a DB with internal replication: the data of 2 MariaDB instances are modified identically, and the bug happened for only 1 of them.

We don't know if DELETE really did nothing or if lines are resurrected. Further investigation, with an external tool watching the ttrans table every minute, showed that a deleted line was still there ~21 seconds later. And this line was alone in the table when it should have been deleted.

MariaDB 10.1.25 (tokudb_version: 5.6.36-82.0) is the most recent version for which we got this bug.

Reported to Percona at https://jira.percona.com/browse/PS-4291

Configuring RocksDB

Due to the difficulty to configure RocksDB, we use it with default options. In particular, this means that all experiments were done without compression enabled (in fact, that's because we built MariaDB without Snappy). We think that MariaDB should provide a way for applications to enable it on their own, like the compression option of TokuDB when creating table.

Compression is important for obj and trans, and it's enabled for TokuDB.

Performance

The superiority of TokuDB over InnoDB in terms of performance is quite well-known (e.g. https://www.percona.com/blog/2012/09/27/three-ways-that-fractal-tree-indexes-improve-ssd-for-mysql/). To name a few, here are some advantages of TokuDB that we could observe:

• good compression, which means 2 things: less disk space used and more efficient disk cache
• less fragmentation (optimizations for HDD tend to be useful for SSD)
• better query plans (I mean, probably better stats)
• faster queries even for the same query plan, and IIRC, even if all data is in cache
• less configuration required: it's horrible that one have to be an expert in InnoDB so that the engine is close to optimal configuration, given the machine specifications and actual load

Nexedi did a comparison between RocksDB, TokuDB and InnoDB, on a machine with SSD and 16GB RAM, and innodb_buffer_pool_size/tokudb_cache_size/rocksdb_block_cache_size at 2GB (InnoDB would have been faster with bigger values but it does not matter: we're going to have DB of several hundred GB, and we'll be far from having everything in RAM).

The template for MariaDB configuration resulted in:

Tests

Test 1: importing 126 GB of data to a NEO DB with 1 storage node

In this test, we import a lot of data and automatically deduplicate it thanks to NEO. This tests gives a good idea of the write performance of the storage engine.

126 GB is the size of the source DB file, which includes metadata. Size of raw data:

• before NEO compression: 120 GB
• after NEO compression: 26.2 GB
• after deduplication: 23.7 GB

Test 2: balancing from 1 node to 2 and then to 4

In this test, we change the topology of a NEO cluster by redistributing data over available nodes: adding nodes is a way to increase available space. A NEO database is split into a fixed number of partitions (NEO term, i.e. not to be confused with MariaDB partition) and some of them are moved to other nodes. A partition on a node is a cell, cells are first copied ("tweak" below) and then dropped from source nodes.

Test 3: reclaim free space on the first 2 nodes

The actual deletion of dropped cells is still an experimental feature:

• it doesn't scale for big databases (1 DELETE per partition per table)
• it's not done in background: it only happens when the cluster is going to be operational

At last, we use OPTIMIZE TABLES so that MariaDB really frees space.

Results

A first comparison between InnoDB and TokuDB was done in February 2017 (at that time, NEO didn't use index hints). One result to keep is that InnoDB performed really bad with innodb_file_per_table=0, to a point it could not end. After the first tweak from 1 to 2 nodes, the index stats were too bad, and OPTIMIZE/ANALYZE didn't help. Optimizing the first DB took 7h17 for nothing.

Tests were repeated recently to also compare with RocksDB (MariaDB 10.2.8), with worse results although the hardware is the same. Maybe the SSD has aged.

1. The SQL queries were completed in 15h55 but InnoDB remained busy in background.
2. Out-Of-Memory killed but not really a problem: we already know that NEO must delete data in a smarter way.
3. SHOW STATUS reports 26.8 GB free (1.9 GB free before tweak).

A few notes about deduplication:

• It's internal to each node, so it is normal that the DB is bigger after being tweak.
• Before the tweak, 15% of records are deduplicated.
• Each deduplicated record put RocksDB at a disadvantage, due to MDEV-13739.

Without deduplication:

Ongoing effort with MariaDB team

We have contacted MariaDB core developer team about the different sub-optimal behaviours that we discussed and have received a very positive and constructive reply.

Most behaviours are either being fixed or already fixed. They can be tracked through the bug tracker of MariaDB:

• RocksDB: Memory leak with only SELECT statements (fixed)
• RocksDB: Huge slowness at inserting rows, CPU-bound (pending)
• RocksDB: "Too many open files" (pending)
• Optimizer: FORCE INDEX hints are not always respected (pending)
• Optimizer: Full scan despite appropriate index (pending)
• Optimizer: MATCH...AGAINST is always estimated to return 1 row (pending)

Reported to Facebook:

• RocksDB: Column family documentation is extremely sparse (pending, major issue)
• ALTER TABLE ... ENGINE=ROCKSDB uses too much memory (pending)

The last one would explain the OOM reported during the tests.

Future improvements of NEO'architecture

Here is a list of ongoing, planned or considered improvements:

• use MariaDB embedded (considered);
• use MariaDB embedded's ability to access BLOB through pointer and reduce number of copies (considered);
• reimplement NEO in golang (ongoing);
• run automated scalability tests over 1 PB of data (ongoing).