Skip to content

DS2 Connector

The DS2 Connector Guide

Connector Scripts

There are several test scripts to test the use of a connector. The current version of the connector and its API is not secured, so no need to use any token, or user/pass to call teh API. In all cases, the scripts need to be edited to change the URL of the connector.

  • Create an asset as a provider: run the script create_asset.sh in order to create an asset in the connector local catalog. You can edit the create-asset.json to change some of the properties.
  • Create a policy as a provider: same as the asset, you can create a new policy on the connector. You can edit the create-policy.json to change some of the policy or use it as it is. Then run the create_policy.sh script.
  • Create a contract definition as a provider: the contract definition is applied to a policy and an asset or group of assets. You can adapt the create-contract-definition.json to your needs, or if you changed the id of the policy. If not, the contract definition will be applied to all assets using the previous policy. Run the sript create_contract.sh.
  • Fetch the provider catalog as a consumer: In this case, you need to use another IDT with another connector as a consumer. You should have at least two IDTs that may represent two dataspaces and can exchange data. Edit the fetch-catalog.json and add the provider connector URL ie. the other IDTs connector URL. Then run the script fetch_catalog.sh with the consumer connector URL. This script retrieves the catalog with the assets that were created at the provider.
  • Negotiate a contract: to initiate the negotation of a contract on an asset run the script negotiate_contract.sh with the contract_offer_id as main parameter: 
    ./negotiate_contract.sh $contract_offer_id

    where contract_offer_id is obtained from the previous output of the fetch catalog, at the path dcat:dataset.odrl:hasPolicy.@id

  • Getting the contract agreement id: the previous call returns the id of the on-going contract negotiation. To check the status of the negotiation, run the script get_contract.sh with parameter contract_id: 
    get_contract.sh $contract_id

    where contract_id is obtained from the id of output of the previous script.

    The state of the output of get_contract should be FINALIZED if the contract negotiation is successful

These scripts and steps have been developed following the steps at the EDC Transfer Samples with all the details at https://github.com/eclipse-edc/Samples/blob/main/transfer/transfer-01-negotiation/README.md

Now a transfer can be started. Since the sample transfer is using a Provider Push, there are some prerequisites such as running an http server etc. For this, please follow steps at https://github.com/eclipse-edc/Samples/blob/main/transfer/transfer-02-provider-push/README.md. There are some scripts that have been developed for this in the IDT: start_transfer.sh, start-transfer.json.template, check_transfer.sh .

As a quick summary:

  • Build and run an http server from the EDC Samples repository
  • Run the start_transfer.sh script at the IDT repository using the contract_agreement_id from the previous step 
    start_transfer.sh $contract_agreement_id

    where contract_agreement_id is obtained as output from the get_contract.sh script

  • Check the status of the transfer with check_transfer.sh and parameter transfer_id from the output of the start_transfer step 
    check_transfer.sh $transfer_id

    where transfer_id is obtained as output from start_transfer.sh script

The DS2 Connector API can be used to perform custom integrations and transfers.

TBD: the Connector swagger API.

Connector Scripts v21

With the release of ds2connectorv21, scripts from previous version have been updated to support the new connector and apis. The scripts can be found in the IDT repository at https://github.com/ds2-eu/idt.git in the /ds2connectorv21scripts folder.

The steps are still the same as in the previous version with additional transfer processes: * Create an asset as a provider * Create a policy as a provider * Create a contract definition (data offer) as a provider * Fetch the catalog as a consumer * Negotiate a contract as a consumer * Start data transfer pull or push as a consumer

The scripts used some input parameters to create the different payloads needed for the request, but users can customize the payloads and run the requests standalone without the scripts ie. using curl or postman or integrating the calls in an application.

Create the asset, policies and contract definitions as a consumer

  • Create an asset: Run the create_asset.sh with $mydomain as parameter where $mydomain is the domain used when installing IDT. You can edit the assetId and the baseUrl of the asset in the create-asset.json file 
    {
"@context": {
  "@vocab": "https://w3id.org/edc/v0.0.1/ns/"
},
"@id": "assetId", --> Edit this
"properties": {
  "name": "product description",
  "contenttype": "application/json"
},
"dataAddress": {
  "type": "HttpData",
  "name": "Test asset",
  "baseUrl": "https://jsonplaceholder.typicode.com/users", --> Edit this
  "proxyPath": "true"
}
}
    Run the script: 
    ./create_asset.sh idtb.ds2.icelab.cloud
    Sample output: 
    {
"@type": "IdResponse",
"@id": "assetId",
"createdAt": 1773227328096,
"@context": {
  "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
  "edc": "https://w3id.org/edc/v0.0.1/ns/",
  "odrl": "http://www.w3.org/ns/odrl/2/"
  }
}
  • Create a policy: You can edit the policy id in the create-policy.json file. The permission, prohibition and obligation can also be edited to enhance the policy. Examples of this will be provided in a later release where connector will be integrated with PAE and other credentials. 
    {
"@context": {
  "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
  "odrl": "http://www.w3.org/ns/odrl/2/"
},
"@id": "aPolicy", --> Edit this
"policy": {
  "@context": "http://www.w3.org/ns/odrl.jsonld",
  "@type": "Set",
  "permission": [],
  "prohibition": [],
  "obligation": []
}
}
    Run the script:
    ./create_policy.sh $mydomain
    Sample output: 
    {
  "@type": "IdResponse",
  "@id": "aPolicy", --> Edit this
  "createdAt": 1773227339744,
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "odrl": "http://www.w3.org/ns/odrl/2/"
  }
}
  • Create a contract definition (data offer). You can edit the create-contract-definition.json file to change the contract definition id, asset selector and the policy. In this example the asset selector will select all assets created and the policy used is the policy created in the previous command aPolicy. 
    {
"@context": {
  "@vocab": "https://w3id.org/edc/v0.0.1/ns/"
},
"@id": "1", --> Edit this
"accessPolicyId": "aPolicy", --> Edit this
"contractPolicyId": "aPolicy", --> Edit this
"assetsSelector": [] --> Edit this
}

Run the script: 

./create_contract_definition.sh $mydomain
Sample output: 
{
"@type": "IdResponse",
"@id": "1",
"createdAt": 1773227372361,
"@context": {
  "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
  "edc": "https://w3id.org/edc/v0.0.1/ns/",
  "odrl": "http://www.w3.org/ns/odrl/2/"
}
}

Fetch the Catalog and Negotiate the Contract as a provider

  • Fetch the Catalog: to fetch the catalog as a consumer you need to query your control plane agains the provider control plane. Run the script: 
    ./fetch_catalog.sh $providerdomain $mydomain

    where $providerdomain is the provider connector domain and mydomain is my connector domain if we want to query the catalog from ds2controlplane.domainA.com and our connector is ds2controlplane.domainB.com, you need to run ./fetch_catalog.sh domainA.com domainB.com

Sample output:

{
"@id": "86d784cf-bbf4-44ec-b7be-c3e5a68510d0",
"@type": "dcat:Catalog",
"dcat:dataset": {
  "@id": "assetId",
  "@type": "dcat:Dataset",
  "odrl:hasPolicy": {
    "@id": "MQ==:YXNzZXRJZA==:ZDQ2NzdjZDYtYzExOS00YjE4LTkwYzUtZTU1MzllYjY1YjRk",
    "@type": "odrl:Offer",
    "odrl:permission": [],
    "odrl:prohibition": [],
    "odrl:obligation": []
  },
  "dcat:distribution": [
    {
      "@type": "dcat:Distribution",
      "dct:format": {
        "@id": "HttpData-PULL"
      },
      "dcat:accessService": {
        "@id": "ba6c43bf-ce54-42d3-9273-9e2b27e89fac",
        "@type": "dcat:DataService",
        "dcat:endpointDescription": "dspace:connector",
        "dcat:endpointURL": "https://ds2controlplane.idtb.ds2.icelab.cloud/api/dsp"
      }
    },
    {
      "@type": "dcat:Distribution",
      "dct:format": {
        "@id": "HttpData-PUSH"
      },
      "dcat:accessService": {
        "@id": "ba6c43bf-ce54-42d3-9273-9e2b27e89fac",
        "@type": "dcat:DataService",
        "dcat:endpointDescription": "dspace:connector",
        "dcat:endpointURL": "https://ds2controlplane.idtb.ds2.icelab.cloud/api/dsp"
      }
    }
  ],
  "name": "product description",
  "id": "assetId",
  "contenttype": "application/json"
  },
  ...
* Negotiate Contract: from the provider catalog you can get the offer ids at 
 "@id": "assetId",
"@type": "dcat:Dataset",
"odrl:hasPolicy": {
  "@id": "MQ==:YXNzZXRJZA==:ZDQ2NzdjZDYtYzExOS00YjE4LTkwYzUtZTU1MzllYjY1YjRk",
the offer id is "MQ==:YXNzZXRJZA==:ZDQ2NzdjZDYtYzExOS00YjE4LTkwYzUtZTU1MzllYjY1YjRk" for asset "assetId"

You can now try to reach a contract agreement with that provider for that offer id. This is something done by the connector and is an accept the policy defined for the offer or not, there is no negotiation in place.

Run the script: 

./negotiate_contract.sh $providerdomain $providerorganisation $offerid $mydomain

As an example: ./negotiate_contract.sh domainA.com A "MQ==:YXNzZXRJZA==:ZDQ2NzdjZDYtYzExOS00YjE4LTkwYzUtZTU1MzllYjY1YjRk" domainB.com where A is the name of the provider organisation -> this is used to create the provider did

Sample output: 

{
 ...
 "@id": "c4dc522a-3c21-4cb7-a431-949b8f5bd0e9",
 "createdAt": 1685525281848,
 ...
}
This is the id of the contract negotation, now it's time to check the status of the contract Run the script: 
./get_contractagreementid.sh $mydomain $negotiationid

where $negotiationid is the id from the previous step output As an example: ./get_contractagreementid.sh domainB.com c4dc522a-3c21-4cb7-a431-949b8f5bd0e9

Sample output: 

{
"@type": "ContractNegotiation",
"@id": "c4dc522a-3c21-4cb7-a431-949b8f5bd0e9",
"type": "CONSUMER",
"protocol": "dataspace-protocol-http",
"state": "FINALIZED",
"counterPartyId": "did:web:ds2identityhub.idtb.ds2.icelab.cloud:idtb",
"counterPartyAddress": "https://ds2controlplane.idtb.ds2.icelab.cloud/api/dsp",
"callbackAddresses": [],
"createdAt": 1773159906921,
"assetId": "assetId",
"contractAgreementId": "5cd3c215-7ce4-4e21-b7e4-f02551e8c528",
"correlationId": "8131f187-a6b3-4ac2-807f-a3d545563ab1",
"@context": {
  "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
  "edc": "https://w3id.org/edc/v0.0.1/ns/",
  "odrl": "http://www.w3.org/ns/odrl/2/"
}
}
The sate should be "FINALIZED", which means there is a contract in place with agreement id contractAgreementId. The contractAgreementId is needed in the next steps when running the transfers. From this users can select to run a pull or a push type data transfer, meaning that for the pull transfer the consumer will pull the data from the provider data plane vs the push transfer where the provider connector will push the data to a specific consumer location.

Run the Pull Transfer

Run the Push Transfer

Before running the push transfer, users need to have a location where the provider will push the data. For our examples, ds2 includes an http server that can be deployed on the IDT and used as the server where the provider will push the data

To install the http demo server run the same steps to install the new connector, without having to delete any module, for the chart demohttp/ in the containerisation repository. No parameters are needed to install it using Containerisation.