Skip to content

E2C - Edge to Cloud Connector

Powered by

Bluebridgesolutions UG (haftungsbeschränkt)

Project Links
Software GitHub Repository https://github.com/ds2-eu/e2c.git
Progress GitHub Project https://github.com/orgs/ds2-eu/projects/24

General Description

Purpose: Dataspaces allow for data to be shared between data providers and data consumers. This includes data coming from sensors and devices at a high rate. This module is used to establish a secure edge-to-cloud connectivity for data providers to cloud-based IoT platforms like Azure IoT or Cumulocity IoT or AWS IoT via a MQTT bridge. The data providers will decide which data to share and with whom. In addition, the data quality can be monitored as well.

Description: The DS2 Edge-to-Cloud Connector module (DS2 E2C) will use Software AG’s open-source product thin-edge.io as background. The underlying communication relies on MQTT, a standards-based messaging protocol used for machine-to-machine communication. The data is locally collected on the edge device, mainly IoT-based but not exclusively, and published to the MQTT broker supplied by this module. Depending on the specific cloud platform used by the data provider or data consumer, appropriate rules will get applied to map the generic format to the specific cloud format. The data will constantly be monitored for loss of data quality which will be reported. Using the MQTT-bridge functionality the data and quality information gets mapped to the chosen cloud platform. The communication is encrypted via SSL/TSL using local certificates.

Architecture

The figure below represents the module fit into the DS-DS environment. DS2 IDT Architecturefits

The figure below represents the actors, internal structure, primary sub-components, primary DS2 module interfaces, and primary other interfaces of the module. DS2 IDT Architecture

Component Definition

In E2C, data is collected via a MQTT broker and can be sent to different cloud environments such as Cumulocity, Azure or AWS via adapters built on top of the open-source software thinedge-io. Thin-edge.io is supporting any Linux distribution, in most cases even with the appropriate package manager. In DS2, thin-edge.io will mainly be used to connect on-premises edge devices to cloud environments of the Data Providers.

This module has the following subcomponent and other functions:

Edge-to-Cloud Connector Module – Core

  • MQTT Broker: This is the central message bus of the module, based on the open-source Mosquitto implementation.

  • Data Converter: Reads data which was published under a certain broker topic, and transforms it to the specific format for the configured cloud platform. The data converter can use a pipeline of custom routines for specific tasks before the transformed data is re-published to the MQTT broker under a different topic.

  • Data Quality Module: The data read by the Data Converter component is constantly analysed by the Analyze Data component of this module to check if the data quality goals are met. The information is collected and sent back to the MQTT broker via the Report Data Quality component.

  • Other Custom Routines: Users can write their own routines to perform tasks on their data published to the MQTT broker before it is sent to the cloud platform via the cloud connector. This could be e.g. calculate moving averages, many examples are available in the thin-edge.io plugins directory.

  • Cloud Connector: Message bus where data can be published under a topic and data consumer can subscribe to these topics.

External Components Used

  • Data Source: The data provider configures and selects where the data to be offered and shared comes from, most likely an edge device.

  • Tier 1 Service Stack for Marketplace and Development: The module uses the marketplace especially for the configuration of the desired cloud adapter.

  • Tier 3 Inter-Dataspace Sharing: The module uses the Discover and Trust components for the configuration of the desired cloud adapter.

  • Cumulocity, Azure, AWS, Custom: These components represent the possible cloud adapters that can be configured. Besides connecting to the commercial clouds Cumulocity, Azure and AWS, one can write a custom adapter for any IoT platform having a MQTT connector, e.g. the open-source ThingsBoards platform.

Screenshots

DS2 E2C Screenshots

Commercial Information

Organisation (s) License Nature License
Software GmbH Open Source Apache 2.0
bluebridgesolutions UG (haftungsbeschränkt) Open Source Apache 2.0

Top Features

1. Data streaming: Upload relevant data to be shared in dataspaces from devices or existing APIs.

2. Cloud platform agnostic: E2C allows to stream data to any cloud platform by it's extensible connectivity feature.

3. No vendor lock-in: Most platforms provide dedicated software to connect devices and data sources - by using those, migrating to another platform is a tedious process. In E2C, switching to another supported platform can be as easy as UI-led configuration.

4. OpenAPI specification: Import your existing OpenAPI specs to define new data streams from existing APIs.

5. Query Builder: Build queries and filters in the user interface to select only the data you need on an attribute level.

6. Time windows: Define time windows for data streaming.

7. Data mapping: Map data from source to target data models.

8. Configure once: Once the streaming services are configured, they can be executed on demand.

How To Install

Requirements

No particular requirements for CPU, memory or storage have been identified. The machine running the streaming component needs to have network access to the endpoints that are to be streamed as well as network access to the cloud platform that data is to be streamed to.

Software

  • Docker runtime
  • Git

Summary of installation steps

  1. IDT based installation
  2. Running containers separately

Detailed steps

1. IDT based installation

TODO - needs to be documented

2. Running containers separately

TODO - needs to be updated

Run container

Clone E2C Git repository. ds2-e2c/docker-compose.yaml runs three containers. One contains an MQTT-Broker, one a PostgreSQL and one the E2C Configuration Management.

Start them by executing 

docker compose up --build

Running E2C stream component

Currently, E2C stream component is to be started manually.

To start the E2C streamer, switch into the tedge-container by running:

docker exec -it tedge-container bash

Once inside the container, the E2C streamer can be started using the following command:

/venv/bin/python3 /app/main.py --config config.json

The parameter --config config.json is optional and allows to pass an existing configuration file, while the streamer can be configured entirely using the E2C UI.

Once the E2C streamer is running, it's configuration can be retrieved and updated by E2C UI. To start the UI, clone the UI Git repository path 'ds2-e2c-stream' and execute 

pip install -r requirements.txt

E2C streamer is now ready to stream data to the configured cloud platform.

How To Use

Open the application in your browser.

To start using E2C, follow these simple steps: * Define Data Sources * Define Cloud Platforms * Define Configurations * Assign configrations to Workers * Optional: define your Data Assets in your catalog.

Define Data Sources

In the top menu of E2C Connectors Management, navigate to "Data Sources" and select "Add new..." in the top right corner of the screen. You can now select upload an OpenAPI description of the data source you want to use. In our example, select ""itc-dataspace-provider-iot.yaml". The name, description and endpoint details are imported. You can also assign the credentials for the data source there.

Define Cloud Platforms

In the top menu of E2C Connectors Management, navigate to "Cloud Platforms" and select "Add new..." in the top right corner of the screen. Fill in the details of the platform and create the platform by clicking the "Save" button.

Define Configurations

In the previous steps, the data sources and the data destination cloud platform was defined. Define configurations in which you can select which data to stream to your platforms. Start by selecting "Add new..." and follow the wizard guiding you through the process. In the "Group By" section, the data messages can be grouped by an attribute. This is commonly used to assign all sensor data to a device in the target platform.

Assign configurations to Workers

Once configurations are defined, you can assign them to workers that perform the ETL task of connecting to the data sources, gathering and filtering the data based on the configuration and forwards them to the cloud platform of your choice.

Start by creating a new worker by clicking "Add new...". For each worker, you can select the configuration and the target cloud platform. Using the toggle switch, you can pause and resume worker.

Congratulations, the data is now being streamed to the configured cloud.

Optional: Define Data Assets

TODO.

Other Information

No other information at the moment for E2C.

OpenAPI Specification

N/A

N/A