Skip to content

ORC - Orchestration

Powered by

DS2 ORC IceLogo

Project Links
Software GitHub Repository https://github.com/ds2-eu/orchestration
Progress GitHub Project https://github.com/orgs/ds2-eu/projects/9

General Description

The Orchestration module is made to design and then orchestrate at runtime In-Dataspace, Inter-Dataspace, internal, and third-party services which facilitate common data-orientated operations such as transformation of data, checks on data, data updates etc. The orchestrator contains a flexible GUI to design workflows and decision points on these services, and run time component to implement the workflow

Services are added to and then selected from a Service catalog from a participant’s local service catalog (In-Dataspace deployment), the DS2 service intermediary catalog (Inner-Dataspace), or other available catalog/service knowledge. These services can be graphically linked together to form a workflow and where decision pathways, decision points, and other operators can be deployed to determine the workflow. Error and exit points should be predetermined with defaults ensuring that failures and error conditions allow flows to be closed automatically. One class of operator is for user defined forms for human input but most often the flows contain backend services. In the context of DS2 the operators will, if necessary, be expanded based on novel usecase peculiarities as will the forms designer. Primarily the design interface is orientated around service interconnectivity, but this will be augmented with a data pre-viewer to help interconnect and understand the results of interconnecting data-orientated services. The orchestrator will be available as a module and for interparticipant service orchestration will extend the connectors.

Architecture

The figure below represents the module fit into the DS-DS environment. DS2 ORC Architecturefits

The figure below represents the actors, internal structure, primary sub-components, primary DS2 module interfaces, and primary other interfaces of the module. DS2 IDT Architecture

Component Definition

This module has the following subcomponent and other functions:

Orchestration Module - Core

  • Orchestration Core: This is the runtime heart of orchestration which conducts a process (workflow) triggering other services and via a BPMN module from the Service Composition designer repository. For the tier 1 standard connections (Portal etc) it can be perceived as the entry point. If new orchestration design methods are needed it will use them. Runtime events are connected to the logging components and for inter-participant dataflows it will interface with the DS2 Cross Participant Orchestration subcomponent. This is currently ICE background and will see little development except for a DS2 compliant UI.

DS2 Service Registry:

  • In-participant: This is a local registry of all services which a participant may potentially use in a workflow, composed together in the designer, and executed in the runtime. Registration can be automatic in the case of IDT installed services. It will also expose services in the inter-participant DS2 registry. It exists now but will be rebuilt in the context of DS2 and IDT.

  • Between Participant: 95% the same functionality but can function similarly to a metadata broker to host services from multiple participants which can be shared in a controlled way to the In-participant registry to allow participant-participant service interactions

  • Service Composition Designer (DS2 Upgrade): This is the main UI for the Orchestration Designer based on existing ICE background. It allows a user to select or drag various elements from a toolbox (services/APIs from the DS2 Service Registry, methods), which can be placed on a canvas where they can then begin to start designing their orchestration by dragging and connecting various elements together. The saved BPMN2.0 notation model will then be used by the runtime orchestration core. The DS2 upgrade will be mainly for UI and inclusion of New Data Previewer and Forms designer blocks.

  • Forms designer: Many orchestrations have a need for user input and whilst some might come from other systems this can be complex when only limited information. The forms designer will allow the easy inclusion of simple form in any Service composition and also ensure that it respect the data flow as well as service needs.
  • Data Previewer: This is also a new subcomponent which will be rendered via the Service composition designer. Currently services are connected but when designing it is useful to know at design time what might be the inputs and the expect result. In a data orientated project this is especially useful, and this utility will allow some rendering of data to help show flow operations between building blocks before they are deployed.
  • New Orchestration methods (from pilots): Many methods – eg choice boxes, selections are already implemented in the orchestrator, but it is possible that the pilot might suggest further ones that could be interesting to implement – although at this stage of the analysis it seems there is not. The new methods will be exposed in the orchestrator runtime & designer.
  • Orchestration track/log: Currently this is rudimentary and especially in the trustworthy context of dataspace an major overhaul is necessary to extract more granular logging information at runtime.
  • Services and API: These are the services that can be orchestrated, and the API block is the interface to each which includes other external (non DS2) modules/services, DS Service Intermediaries and Tier 2 in dataspace DS modules/services.
  • Tier 0 Support Service Stack: This includes the DRM and API: For further exploration, but if room to implement and a match of requirements to feature the blockchain part of the DRM module to enhance logging
  • DARC & API: As with DRM but in this case to use DARC to help configuration
  • Culture and Language Module and API: As with DRM but in this case to use this modules ontology engine to help auto-link services
  • Tier 1 Service Stack for Marketplace and deployment and API: The full stack will be implemented as generically described elsewhere in this document. Exceptions: The Platform will only be needed for inter-participant service orchestrations if used

Inter-Participant orientated:

  • DS2 Cross DS Orchestration: This is a new runtime module which will act as a bridge between the orchestration within each participant through interconnections to the Inter-Participant Service Registry and the Orchestration core at each participant
  • Tier 3 Trust Stack and API: For interparticipant service the module will use relevant parts of the DS2 trust stack

Screenshots

DS2 IDT Screenshots

Commercial Information

Organisation (s) License Nature License
ICE Open Source Apache 2.0

Top Features

  1. Process Designer: A custom Camunda BPMN Modeller
  2. API Repository: repository of API services
  3. Service API: an API that allows use of services in the API Repository externally
  4. User Tasks: a unique type of task that allows for workflows to request user input
  5. User Task Manager: a front end that allows for management of user tasks
  6. Control Panel: a web app that allows users to view, manage and interact with workflows
  7. Process View: a view that allows the user to monitor the status of workflows
  8. Single-Sign On: integration with the Dash Button that allows the ORC module to be linked to the Portal

How To Install

Requirements

N/A

Software

  • Git
  • Docker

Summary of installation steps

  1. Clone the git repository
  2. In each of the subcomponents, run the build.sh script and select the local build
  3. In each of the subcomponents, run docker pull xxx:latest where xxx is the name of the subcomponent
  4. In each of the subcomponents, run docker-compose up
  5. Navigate to localhost:4200

Detailed steps

Each of the subcomponents contains a build.sh file that allows the creation of the docker images in such a way that you can change it depending on project, location or production.

In addition, each of the subcomponents are individual, which means each subcomponent can be run on their and interacted with (although some features may be unavailable).

The current list of ORC front end docker images includes:

  • welcome:latest
  • process-designer:latest
  • form-designer:latest
  • control-panel:latest
  • service-directory:latest
  • task-manager:latest

How To Use

General Overview

The Orchestration Module is made up of six sub comonents that all interact with each other in different ways. Together, they offer the full suite of features that the ORC tool offers independantly.

Process Designer

The Process Designer is the largest single subcomponent of the ORC Module. It is a Camunda based BPMN Modeller that allows users to create workflows.

Process Designer View

Users can drag and drop elements from the panel on the right side in order to create workflows. When exported, the workflows, the processes, can be viewed in the Control Panel.

Form Designer

The Form Designer is where the user can create forms for use with User Tasks. User Tasks are tasks dicated by the Process Designer. When triggered in an active workflow, the User Task requires a user interact with a form created via the Form Designer through the Task Manager.

Form Designer View

Service Directory

The Service Directory is a repository of APIs that can then be accessed through the Process Designer to pass data through a workflow or recieve data from a workflow. These services can also be accessed through the Service Directory API.

Service Directory View

Control Panel

The Control Panel shows all the active workflows; active processes can be found here, and each instance of a process can also be found. Each process instance also has visual view that allows users to view various elements of a workflow after it has been created.

Control Panel Processes List A Screenshot of the Control Panel - List of Processes

Control Panel Process Instances List A Screenshot of the Control Panel - List of Process Instances

Control Panel Process Instance View List A Screenshot of the Control Panel - List of Process Instance View

Task Manager

The Task Manager is the component that handles the front of the User Tasks. When a User Task is triggered, the assigned user will be able to complete the User Task from this view. Administrators are also able to view all User Tasks.

Task Manager Task View A Screenshot of the Task Manager Task List View

Task Manager Task View A Screenshot of the Task Manager Task View

Making and Running a Simple Process

Other Information

No other information at the moment for ORC module.

OpenAPI Specification

N/A

N/A