IDT - IDT Broker

Powered by

Project Links
Software GitHub Repository https://github.com/ds2-eu/idt.git
Progress GitHub Project https://github.com/orgs/ds2-eu/projects/3

General Description

IDT is the core enabler of DS2 who purpose is to be deployed in front of participants data source/spaces and network connected to any other IDT-enabled data source. As such its aim is to run all DS2 modules, including the DS2 Connector, the core module for Inter-Dataspace communication and data transfer, and the Containerisation module for DS2 module deployment. The IDT contains the core Kubernetes runtime to run all containerised modules and a series of additional open-source software for module management.

The IDT contains the Kubernetes runtime that is the core service to run all modules in a containerised way. Those modules descriptors are uploaded to the DS2 Portal Marketplace and then, using the IDT Kubernetes UI, are deployed to the Kubernetes runtime. The Containerisation module kicks-in and then converts those module descriptors to full module charts effectively deploying the DS2 modules. The Kubernetes UI, alongside the Management and Monitoring Controller, will be used to manage and monitor all DS modules running on a participants IDT. Additional services will also be run as part of the IDT such as certificate management, ingress network traffic management using traditional ingress controllers with ingress resources to expose modules and eventually transitioning to service mesh and gateway API, storage management and possibly other useful open-source tools. The other key component of the IDT is the DS2 Connector used for Dataspace like communications following current IDSA and Gaia-X standards. An additional IDT UI will be provided for module navigation and Connector management.

Architecture

The figure below represents the module fit into the DS-DS environment. DS2 IDT Architecturefits

The figure below represents the actors, internal structure, primary sub-components, primary DS2 module interfaces, and primary other interfaces of the module. DS2 IDT Architecture

Component Definition

This module has the following subcomponent and other functions:

Kubernetes (Module Runtime): Kubernetes is the leading technology in container orchestration and the choice and key component of the IDT for deployment and integration of the DS2 modules. This is the core IDT subcomponent that runs and orchestrates the DS2 modules and all the other IDT subcomponents, as containers. This is Open-Source software and the current distribution being used is K3s, a lightweight version of Kubernetes easy to install, half the memory, all in a binary, in less than 100 MB among other enhancements. One of the main advantages is the flexibility of installation, since it can be deployed at a participant edge, onPremise, InCloud, etc.
Kubernetes UI: The Kubernetes UI is open-source software based on Rancher that allows to install and manage Kubernetes in a more user-friendly way both onPremise and InCloud. The Kubernetes UI will provide the management interface for platform administrators to manage the Kubernetes cluster and resources running on top of it. The Kubernetes UI will also provide the monitoring interface to the IDT Kubernetes subcomponent and the DS2 modules.
Management and Monitoring Controller: This is the main interface from the Kubernetes UI to the Kubernetes subcomponent and also for external integrations. The Management and Monitoring Controller is open-source software based on the Kubernetes and Rancher API and the Rancher agents. It is used as the primary interface to the IDT Kubernetes subcomponent for cluster management. It will also be used as the primary interface for monitoring which will potentially be integrated with the DRM Blockchain module for traceability. In addition, further research on using other modules for monitoring such as Prometheus-Grafana will be conducted for enhanced monitoring.
Ingress – Gateway: The ingress or gateway resource provides the entry point to the IDT Kubernetes subcomponent via the Ingress controller, thus, the IDT network, for all network traffic from external apps, being an external app, any system external to the IDT. It describes how the DS2 modules are exposed outside of the IDT. Initially the modules will use a Kubernetes Ingress resource to expose the modules but further research will be conducted to examine the use of the Gateway API and Service Mesh technology.
Ingress Controller – Service Mesh: Based on Open Source, the Ingress Controller is the Kubernetes controller dealing with Ingress resources, that is, managing the entry point to the IDT and how the DS2 modules are exposed outside of IDT. Further research is expected for replacement of the Ingress Controller with Service Mesh technology and the Kubernetes Gateway API, that adds a transparent layer to provide the IDT with enhanced connectivity, security, control and observability. The use of the Service Mesh could also be a key feature for using more secure communication via mutual TLS protocol (mTLS) in all DS2 communications which provides and additional trust layer. This could also be integrated with the DS2 trust and identity system.
Storage Manager: Open-source software to provide the interface between the IDT Kubernetes subcomponent and the physical storage for DS2 stateful modules. This will use Kubernetes native storage technology to allow highly available stateful module deployments in IDT. When data from DS2 modules need to be persisted in a participant backend storage system, the Storage Manager will be used to map current deployment and Kubernetes Persistent Volumes to external storage systems. This is not a storage system or technology for modules… If DS2 modules need to use storage, the DS2 modules need to provide them by packaging them in their module Chart.
CertManager: Based on Open-source software, it provides management of SSL certificates for secure connectivity ie. HTTPS, with verified signed certificates using Let’s Encrypt CertificateAuthority (CA) and configures them for the Ingress or Gateway resource. The CertManager integrates with the Ingress Controller and/or Service Mesh subcomponents and in addition, further research on integration with DS2 Trust system will be explored.
DS2 Connector: The DS2 Connector in the IDT is the key element that will allow for DS2 transactions and data exchange, following the IDSA and Gaia-X standards. The Open-source Eclipse EDC Connector and the necessary components (control plane, data plane …) will be used to provide interoperability between Dataspaces and secure, trustworthy exchange of data. Following existing Dataspace principles and protocols, the DS2 Connector will use the DS2 Trust system for identity management and will connect to other participants IDT DS2 Connectors in other Dataspaces for data exchange. The Connector will also integrate with the DS2 Catalog or a Dataspace level Metadata Broker for participant and data discovery.
IDT UI: This is the main interface for the IDT from the user point of view. The interface is actually two interfaces, one is the Participant Portal, which is a local version of the DS2 Portal and displays the list of modules running in an IDT and user and role management. The other is a more Connector focused UI, with Connector configuration and management features.
IDT Toolkit / Platform Configuration: This is a toolkit box where common tools will be deployed for use by the modules, including platform configuration to be used mainly by the Containerisation module and the Chart templates, but also can be used by other modules. The first tool in the box will be an email server to be used by modules.
Tier 0 Support Service Stack:
- DRM and API: For further exploration integration of the monitoring controller with the Blockchain will be considered.
Tier 1 Service Stack for Marketplace and deployment and API: The full stack will be implemented as generically described elsewhere in this document. Exceptions: The IDT uses the DS2 Portal and Marketplace to retrieve the ChartDescriptors of modules and deploy them via the Kubernetes UI. Then the Containerisation module uses the descriptors to deploy the full module Helm Chart. In addition, The DS2 Connector in the IDT integrates with other IDT DS2 Connectors for data exchange.
Tier 3 Trust Stack and Catalog and API: The IDT will make use of the relevant parts of the DS2 Trust Stack for certificates in the Ingress Controller – Service Mesh and identities in the DS2 Connector. The IDT will also connect via the DS2 Connector to the Catalog.
External Apps: External Apps refer to any software application external to the IDT and DS2 ecosystem that uses the DS2 Connector in the IDT for any DS2 data transaction. It’s the application that can trigger a data exchange via the Connector, either as a consumer or producer.
External Storage Systems: This refers to any external storage system, physical or software defined, that a participant has already in place and where data from the IDT and DS2 ecosystem can be persisted, thus, is mapped via the Storage Manager into the IDT Kubernetes

Screenshots

DS2 IDT Screenshots

Commercial Information

Organisation (s)	License Nature	License	Marketplace Link
ICE	Open Source	Apache 2.0	idt

Top Features

Kubernetes Platform: The IDT provides a Kubernetes based platform for ease of integration and deployment of the modules
Flexible Installation Process: The IDT provides a user friendly installation process for easy installation to non-experienced users. In any case, management of the platform itself, will require some expertise. In addition, it supports different installation modes from on-cloud to on-prem and edge.
Management Interface: A Rancher based UI is provided for the Kubernetes cluster management. Management of Kubernetes itself will require some expertise. The Management UI provides the interface and API for module management : deployment, deletion, upgrade
Monitoring of Platfom and Apps: Provides the interface and API to monitor the cluster itself and the modules using Rancher UI
Seamless Integration with Containerisation Deployment: IDT integrates seamlessly with the Containerisation module for ease of module deployment
Networking: Provides secure networking and connectivity among the installed apps and to and from outside the cluster
Log management: Ability to retrieve module logs for troubleshooting and debugging
Native Storage: Provides Kubernetes native storage for stateful applications
IDT UI: Provides the DLM UI as the entry point to the IDT, single sign on and module navigation which includes all the IDT core module links: containerisation, localcatalog, dlm, credentialmanager and dlmstore
DS2 Connector: The IDT incorporates the DS2 Connector for DS2 data exchange

How To Install

The IDT installs a pre-packaged enterprise ready Kubernetes cluster along with some extra features for management and deployment.

Requirements

Provision a Linux Node or VM (Ubuntu 18.04, 20.04, 22.04)

The VM has to be run on top of AMD architectures ie. don't run a vm on a mac host machine since typically module images won't support arm architectures

Resources:

Kubernetes Node

Minimum: 2 cpu cores, 4 GB RAM and 10 GB disk capacity.
Recommended: 4 cpu cores, 8 GB RAM and 50 GB disk capacity.
Recommended when installgin idt core bundle: 8 cpu cores, 16 GB RAM and 100 GB disk capacity

These numbers may change since a number of IDT components will be deployed, check specific requirements for specific components.

Software

IDT installs these software utilities and specific tested compatible versions:

Docker
K3s (Kubernetes)
Helm
Cert-manager
Rancher
Creates signed certificates to use by the ingress controller from various providers (digitalocean, cloudflare), from files or self signed
Nginx Ingress Controller
Nginx docker (load balancer - optional)
DS2 Connector
Core DS2 modules

DS2 Installation

The DS2 Installation of IDT includes not only the IDT itself but the installation of all core modules

Summary of installation steps

Purchase the IDT Core bundle
From the purchased bundle, download the idtinstaller module
Unzip and untar the installer in the VM to be installed
Configure the installation
Run the installer

Detailed Steps

Purchase the IDT Core bundle from the DS2 Marketplace at Marketplace . This includes the "idtinstaller" and all the core modules
From the purchased bundle, download and copy the idtinstaller module to the VM where the IDT is going to be installed.
The idtinstaller is a tar.gz file which is packaged as a zip file for the Marketplace. First unzip the zip file and then untar the tar file
```
tar -xzvf idtinstaller.tar.gz
```
Once the installer is unpacked, navigate to the idtinstaller folder and edit the global.yaml file to configure the installation
```
# idt params
nodeip: 192.168.50.241
iface: eth0
idt_domain: ybeer.ds2.icelab.cloud
custom_ssl: 1
issuer: digitalocean

# cont params
githubuser: 
githubtoken: 
organisation: myorganisation

# connector
connectorversion: ds2connectorv24pae
```
where the parameters are
nodeip: ip of the node where idt is being installed

iface: the network interface of the ip

idt_domain: the domain as configured in the Portal

custom_ssl: optional parameter to create the ssl certificates for the idt domain with values 0, 1 and 2. * 0: the installer creates a self-signed certificate for the domain * 1: the installer creates a valid let's encrypt signed certificate for an exisiting domain using the dns challenge. If this option is selected, the parameter issuer needs to be added to the yaml file with the name of the dns server that hosts the domain. Supported issuers so far are: * cloudflare
```
custom_ssl: 1
issuer: cloudflare
```
Edit the /idtinstaller/minids23.0/issuers/cloudflare-secret.yaml and add your cloudflare API token api-token: API Token * digitalocean
```
custom_ssl: 1
issuer: digitalocean
```
Edit the /idtinstaller/minids23.0/issuers/digitalocean-secret.yaml and add your digitalocean access token base64 encoded ie. echo -n $accesstoken | base64 access-token: “Acess Token” * 2: the installer creates a secret based on an existing valid signed certificate owned by the user. If this option is selected, the parameters public_key and private_key need to be added to the yaml file with the path to the crt and key files of the certificate. ```yaml custom_ssl: 2 public_key: idt.crt private_key: idt.key githubuser: the github username as configured in the Portal

githubtoken: create a githubtoken for the github user with pull permissions from the repo. It will be used to pull charts and images for the modules

organisation: the organisation registered in the portal

connectorversion: leave as it is
Run the installer from the idtinstaller folder. The installer will install the IDT and all the core modules acquired with the Marketplace core bundle: IDT, Containerisation, Connector, DLM, LCAT, CredManager, DLM Store
```
./idtinstaller.sh
```
Access the Portal and navigate via DashButton to Containerisation module
Refresh the GitHub repository to refresh all installed modules
Navigate to all core modules via DashButton or DLM

Update existing installation without re-installting Kubernetes and Flux

Update existing installation reinstalling everything

Standalone Installation

Summary of installation steps

Clone the repo
Deploy IDT
- Create the parameters yaml file
- Run the command minids23.1.sh as per instructions. This deploys the Kubernetes platform
Access Rancher at https://rancher.$nodeip.ds2.sslip.io where $nodeip uses '-' instead of '.' (The domain can be changed if needed)
1. If http access is needed, run patch_nginx.sh
Deploy IDT modules from the Rancher UI (Using Containerisation module at a later stage)
- Any script required before deploying any component should be in the idt/modules folder
- Upon deployment of first module import the self-signed certificate in the trusted CA's store

Detailed steps

Clone the repository

git clone https://github.com/ds2-eu/idt.git

Navigate to idt folder
```
cd idt/idt3.0/minids23.0
```
Create the parameters file minids2.yaml with this parameters:

nodeip: the ip of the vm where idt and k3s will be installed ie. 192.168.50.10

iface: the network interface of the nodeip ie. enp0s8

idt_domain: optional parameter to configure the domain of the idt and modules that will later be deployed to organisation domain. This domain will usually be public. If not set, the domain defaults to "$ip.idt.ds2.sslip.io" where $ip is the nodeip configured but using '-' between the numbers ie. 192-168-50-10

rancher_domain: optional parameter to configure the Rancher domain to a different domain that the one from the organisation. This domain will usually be private since it has access to the IDT Kubernetes cluster. If not set, the domain defaults to "$ip.ds2.sslip.io" where ip is the same as the idt_domain example.

custom_ssl: optional parameter to create the ssl certificates for the idt domain with values 0, 1 and 2. 0: the installer creates a self-signed certificate for the domain 1: the installer creates a valid let's encrypt signed certificate for an exisiting domain using the dns challenge. If this option is selected, the parameter > issuer needs to be added to the yaml file with the name of the dns server that hosts the domain. Supported issuers so far are: cloudflare 2: the installer creates a secret based on an existing valid signed certificate owned by the user. If this option is selected, the parameters public_key and > private_key need to be added to the yaml file with the path to the crt and key files of the certificate.

An example of a yaml file for issuer type of ssl:

nodeip: 192.168.50.5
iface: enp0s8
idt_domain: idt.myexternaldomain.com
rancher_domain: idt.myinternaldomain.com
custom_ssl: 1
issuer: cloudflare

Another example using existing certificate:

iface: enp0s8
idt_domain: idt.myexternaldomain.com
rancher_domain: idt.myinternaldomain.com
custom_ssl: 2
public_key: idt.crt
private_key: idt.key

Run idt
```
./minids23.1.sh
```
The script will install the software utilities in this order:
First, IDT installs Docker
Then, k3s Kubernetes cluster is installed. Once installed, the installation process will wait and check for K3s to be up and running. Helm is installed together with K3s and kubectl.
Next cert-manager is deployed in order to provide a ssl certificate for Rancher. Process will wait and check that cert-manager is running.
Then Rancher is deployed in the cluster. Process will wait and check that Rancher is running.
Next, the nginx ingress controller is deployed. Before this, a self signed ssl certificate is created using certmanager. Notice that the domain that is configured in the certificate, is the one to be used as domain for the modules when deployed to IDT which defaults to *.$nodeip.modules.ds2.sslip.io . This domain can be changed.
An external to Kubernetees nginx server is also run as nginxlb docker image, as the entry point to the cluster, to serve as an external load balancer.
DS2 IDT is now ready.

Users can also select which components need to be installed, ie. maybe docker is already installed in the vm, or this is a re-installation, or the nginxlb does not need to be installed. To select the components, create or edit the file minids2.cfg and add remove the components to be installed. If the name of the component is not in the file, that component will be installed, if the name is in the file, that component will not be installed. * docker * k3s * certmanager * rancher * ingress * nginxlb

An example of a file for an installation where docker and nginxlb are not installed:

docker
nginxlb

The IDT is then installed.

IDT-SH

Access Rancher by accessing the Rancher url in the browser (https://rancher.$nodeip.ds2.sslip.io)
Once in the Rancher UI, the admin password is set
Then navigate to the workloads in the system project

Rancher

The nginx ingress controller is by default set to only accept https connections and redirect to https. In order to use http, run the script patch_nginx.sh which will configure nginx ingress controller to accept http (optional).
Now that the cluster is up and running, the Rancher UI can be accessed in order to manage the cluster and install modules.

How To Use

The IDT is mainly the Kubernetes based platform with some extra features and enhancements where all the modules are deployed and managed. So, the how to use in this case is focused first on managing the Kubernetes cluster which is the core of IDT itself and requires some IT expertise. In addition, the DLM module is included which is the entry point to all the core modules in IDT. Some instructions on how to use the DS2 connector will also be included.

Rancher UI

Once the IDT has been installed, the Rancher UI along with the IDT Portal (local Portal) and the core modules (to be done) can be accessed. The Rancher UI is the main entry point for Kubernetes cluster management but won't be used to manage modules since the Containerisation module is the module to install and delete modules from IDT.

Inspect the cluster and check number of nodes, resources, etc
Module logs: In order to review the modules logs, navigate to Apps, Installed Apps, and the list of installed modules is displayed. Select the module to be monitored and the list of Kubernees resources of that module are displayed. Select the Deployment and the Pod is displayed. Click on the three dots on the right and select View Logs. The logs of the module are displayed.

Kubectl commands

Experienced IT users can also use kubectl commands, installed during the IDT installation in order to manage the cluster. Please see the Kubernetes documentation in the additional links section.

DLM

The DLM is a module or sub-module of IDT which is a web application that includes all the links to the core modules running in IDT. * From the dash button, click on modules and click on dlm IDT-dlmbutton * You will be redirected to the DLM module where you have the available links for the IDT core modules IDT-dlm

DS2 Connector

Once the IDT is installed, the DS2 Connector is deployed automatically and exposed using the domain selected when installing the IDT. The main module that allows management of the connector ie. creating data assets, policies, data offers, etc. is the LCAT localcatalog module which is a core module deployed with IDT installation. This is the recommended way of managing the DS2 Connector for production, please read the instructions at LCAT.

The DS2 Connector can also be managed via API and mainly for testing purposes, included in the idtinstaller package, we have created a set of scritps that allow you to perform some of the basic operations of the connector: 1. Create a data asset 2. Create a policy 3. Create a data offer (contract definition) 4. Fetch a connector catalog 5. Negotiate a contract 6. Transfer data (http push and pull)

For details on how to work with the DS2 Connector and use those scripts please red the [Guide of the DS2 Connector](../guides(ds2connector.md).