Skip to content

E2C - Edge to Cloud Connector

Powered by

Software GmbH

Project Links
Software GitHub Repository https://github.com/ds2-eu/e2c.git
Progress GitHub Project https://github.com/orgs/ds2-eu/projects/24

General Description

Purpose: Dataspaces allow for data to be shared between data providers and data consumers. This includes data coming from sensors and devices at a high rate. This module is used to establish a secure edge-to-cloud connectivity for data providers to cloud-based IoT platforms like Azure IoT or Cumulocity IoT or AWS IoT via a MQTT bridge. The data providers will decide which data to share and with whom. In addition, the data quality can be monitored as well.

Description: The DS2 Edge-to-Cloud Connector module (DS2 E2C) will use Software AG’s open-source product thin-edge.io as background. The underlying communication relies on MQTT, a standards-based messaging protocol used for machine-to-machine communication. The data is locally collected on the edge device, mainly IoT-based but not exclusively, and published to the MQTT broker supplied by this module. Depending on the specific cloud platform used by the data provider or data consumer, appropriate rules will get applied to map the generic format to the specific cloud format. The data will constantly be monitored for loss of data quality which will be reported. Using the MQTT-bridge functionality the data and quality information gets mapped to the chosen cloud platform. The communication is encrypted via SSL/TSL using local certificates.

Architecture

The figure below represents the module fit into the DS-DS environment. DS2 IDT Architecturefits

The figure below represents the actors, internal structure, primary sub-components, primary DS2 module interfaces, and primary other interfaces of the module. DS2 IDT Architecture

Component Definition

In E2C, data is collected via a MQTT broker and can be sent to different cloud environments such as Cumulocity, Azure or AWS via adapters built on top of the open-source software thinedge-io. Thin-edge.io is supporting any Linux distribution, in most cases even with the appropriate package manager. In DS2, thin-edge.io will mainly be used to connect on-premises edge devices to cloud environments of the Data Providers.

This module has the following subcomponent and other functions:

Edge-to-Cloud Connector Module – Core

  • MQTT Broker: This is the central message bus of the module, based on the open-source Mosquitto implementation.

  • Data Converter: Reads data which was published under a certain broker topic, and transforms it to the specific format for the configured cloud platform. The data converter can use a pipeline of custom routines for specific tasks before the transformed data is re-published to the MQTT broker under a different topic.

  • Data Quality Module: The data read by the Data Converter component is constantly analysed by the Analyze Data component of this module to check if the data quality goals are met. The information is collected and sent back to the MQTT broker via the Report Data Quality component.

  • Other Custom Routines: Users can write their own routines to perform tasks on their data published to the MQTT broker before it is sent to the cloud platform via the cloud connector. This could be e.g. calculate moving averages, many examples are available in the thin-edge.io plugins directory.

  • Cloud Connector: Message bus where data can be published under a topic and data consumer can subscribe to these topics.

External Components Used

  • Data Source: The data provider configures and selects where the data to be offered and shared comes from, most likely an edge device.

  • Tier 1 Service Stack for Marketplace and Development: The module uses the marketplace especially for the configuration of the desired cloud adapter.

  • Tier 3 Inter-Dataspace Sharing: The module uses the Discover and Trust components for the configuration of the desired cloud adapter.

  • Cumulocity, Azure, AWS, Custom: These components represent the possible cloud adapters that can be configured. Besides connecting to the commercial clouds Cumulocity, Azure and AWS, one can write a custom adapter for any IoT platform having a MQTT connector, e.g. the open-source ThingsBoards platform.

Screenshots

DS2 E2C Screenshots

Commercial Information

Organisation (s) License Nature License
Software GmbH Open Source Apache 2.0 (pending)

Top Features

1. Data streaming: Upload relevant data to be shared in dataspaces from devices or existing APIs.

2. Cloud platform agnostic: E2C allows to stream data to any cloud platform by it's extensible connectivity feature.

3. No vendor lock-in: Most platforms provide dedicated software to connect devices and data sources - by using those, migrating to another platform is a tedious process. In E2C, switching to another supported platform can be as easy as UI-led configuration.

4. OpenAPI specification: Import your existing OpenAPI specs to define new data streams from existing APIs.

5. Query Builder: Build queries and filters in the user interface to select only the data you need on an attribute level.

6. Time windows: Define time windows for data streaming.

7. Data mapping: Map data from source to target data models.

8. Configure once: Once the streaming services are configured, they can be executed on demand.

How To Install

Requirements

No particular requirements for CPU, memory or storage have been identified. The machine running the streaming component needs to have network access to the endpoints that are to be streamed as well as network access to the cloud platform that data is to be streamed to.

Software

  • Docker runtime

  • Python3, pip

  • Git

Summary of installation steps

  1. Clone Git repository
  2. Set cloud platform user credentials
  3. Build & run E2C streamer container
  4. Execute E2C streamer
  5. Configure E2C streamer using E2C UI

Detailed steps

Run container

Clone E2C Git repository. docker-compose.yaml runs two containers. One contains an MQTT-Broker while the other contains the E2C streamer component. Both containers have access to the 'thin-edge' environment and can pub/sub to the broker.

Prior to starting the infrastructure, credentials to target cloud platforms have to be set in the docker-compose.yaml file. Once the credentials have been set, the infrastructure can be started by executing 

docker compose up --build

Running E2C stream component

Currently, E2C stream component is to be started manually.

To start the E2C streamer, switch into the tedge-container by running:

docker exec -it tedge-container bash

Once inside the container, the E2C streamer can be started using the following command:

/venv/bin/python3 /app/main.py --config config.json

The parameter --config config.json is optional and allows to pass an existing configuration file, while the streamer can be configured entirely using the E2C UI.

Once the E2C streamer is running, it's configuration can be retrieved and updated by E2C UI. To start the UI, clone the UI Git repository and execute 

pip install -r requirements.txt

E2C streamer is now ready to stream data to the configured cloud platform.

How To Use

To display the configuration UI, run 

python main.py
In the configuration UI, you can upload an Open API specification by clicking on the button"Load OpenAPI Spec from...":

DS2 E2C HTU1

Select the specification "itc-dataspace-provider-iot.yaml" in the "resources" directory. The dialog will change to the attribute selection screen:

DS2 E2C HTU2

Select the data endpoints that are to be streamed to a cloud platform. Activate the "Get IoT data" endpoint. The endpoint requires organizational units as parameters. Here, a subquery can be carried out. In the Query Parameters section, click on "Stream from..." and select the endpoint "/organizations" and select the matching attribute "uuid". Additionally, the data endpoint requires two additonal parameters: "dateFrom" and "dateTo" that allows to narrow down the query.

Here, date queries can be used, i.e. "Custom -> DateQuery" using "Today-2". This will select all data that was created within the last two days. In the "dateTo", you can select "Custom -> DateNow" to get all data up to the current time.

Next, select all streaming attributes that you are interested in. In the example, we selected "temperature", "airPressure", "humidity", "soilMoisture", "leafWetness", "batteryLevel" and "status".

In the "Group By" section, the data messages can be grouped by an attribute. This is commonly used to assign all sensor data to a device in the target platform. Select "sensorDisplayname" to obtain a readable device name in the cloud platform.

In the "Date attribute" section, you can set the attribute that contains the sensor reading's timestamp. Here, we set it to "dateCreated".

The endpoint configuration is now complete. Click on "Next" in the lower right corner. In the next step, the target cloud environment is configured:

DS2 E2C HTU3

This allows to define technical information required to stream the data to the desired cloud environment. Once the information is entered, click on "Publish Configuration" to configure the E2C streaming component.

The data is now being streamed to the configured cloud.

Other Information

No other information at the moment for E2C.

OpenAPI Specification

N/A

N/A