05 Setup and Installation

Setup and Installation Guide

This document contains the installation and configuration information required to deploy the Data Exchange (DX) DMP APD Server.

Configuration

In order to connect the DX DMP APD Server with PostgreSQL, RabbitMQ, Razorpay Service, DX Catalogue Server, DX Auth Server, etc please refer Configurations. It contains appropriate information which shall be updated as per the deployment.

Dependencies

In this section we explain about the dependencies and their scope. It is expected that the dependencies are met before starting the deployment of DX DMP APD Server.

External Dependencies

Software Name	Purpose
PostgreSQL	For storing information related to policy, linked account, orders, resources, users, payment status
RabbitMQ	To publish auditing related data to auditing server via RabbitMQ exchange
Java Razorpay Client	To onboard providers, create order, checkout page for consumers and process the payment

Internal Dependencies

Software Name	Purpose
DX Authentication Authorization and Accounting (AAA) Server	Used to download certificate for JWT token decoding and to get user info
DX Catalogue Server	Used to fetch the list of resource and provider related information

Prerequisites

Keycloak registration for DX DMP-APD as trustee and APD

The trustee user must be registered on Keycloak as a user
- This can be done via the keycloak admin panel, or by using Data Exchange (DX) UI
- The trustee user need not have any roles beforehand
The COS admin user must call the create APD API : Link to the API with the name as the name of the APD, owner as email address of trustee (as registered on Keycloak) and URL as the domain of the APD
Once the APD has been successfully registered, the trustee user will gain the trustee role scoped to that particular APD.
- They can verify this by calling the list user roles API : Link to the API
The trustee can get client credentials to be used in APD Operations by calling the get default client credentials API : Link to the API

DMP-APD Server registering with Razorpay

DMP APD Server could acts a sub merchant to onboard providers (merchants) for creating-linked account by creating an account in Razorpay : link to the dashboard
The live account mode needs to be activated in Razorpay by completing the KYC, filling other details
Live account needs to be activated in order to allow the real-time transactions
To use Linked account APIs through DMP, a dashboard needs to be created and a initial linked account should be created from the dashboard : Link to pre-requisite Otherwise, Route feature not enabled for the merchant / Internal Server Error could be one the errors

Provider's registration with Razorpay

Provider acts a merchant to the DMP-APD Server
They need to create linked account in order to route the payments to their accounts
Linked account creation at DMP involves :
Registration as provider at Data Exchange (DX) Platform
Approval of provider
- Link to the get provider registration API : here
- Link to the update provider registration API : here
Registration as merchant at Razorpay
- Link to the dashboard : here
Creation of resource items at Catalogue Server
- Link to create item : here
Creation of Linked account at Data Marketplace server
Creation of Product at Data Marketplace server

Webhook Creation at Razorpay

2 Webhooks are to be created on Razorpay dashboard to capture information about 2 events from Razorpay asynchronously
- order.paid
- payment.failed
A secret should be added while creating the webhook and could be stored in config with field name : webhook_secret
baseUrl could be replaced with appropriate Data Marketplace APD URL

Name	Endpoint on DMP	Event on Razorpay
Order paid	https://baseUrl/order-paid-webhooks	order-paid
Payment failed	https://baseUrl/payments-failed	payment-failed

Testing Payment flow in Razorpay’s test mode

When the mode in Razorpay dashboard is test mode, the payment flow could be simulated by consumer buying a product-variant from UI
When consumer clicks on check-out and buys a product variant on UI, a payment page from Razorpay pops up
After the consumer selects options like netbanking and clicks on pay now, the consumer could select Success or Failed button to simulate the success or failure flow
In Success flow, the consumer’s purchase is successful and a policy is created for the resources listed under the given product variant
During failure flow, the consumer’s purchase fails and could be fetched in list failed purchase

RabbitMQ

To setup RabbitMQ refer the setup and installation instructions available here
After deployment of RabbitMQ, we need to ensure that there are certain prerequisites met. In case if it is not met, please login to RabbitMQ management portal using the RabbitMQ management UI and create a the following

Create vHost

Type	Name	Details
vHost	IUDX-INTERNAL	Create a vHost in RabbitMQ

Create Exchange

Exchange Name	Type of exchange	features	Details
auditing	direct	durable	Create an exchange in vHost IUDX-INTERNAL to allow audit information to be published

Create Queue and Bind to Exchange

Exchange Name	Queue Name	vHost	routing key	Details
auditing	direct	durable	#	Create a queue in vHost IUDX-INTERNAL to allow audit information to be consumed. Ensure that the queue is bound to the auditing exchange

User and Permissions

Create a DX DMP APD user using the RabbitMQ management UI and set write permission. This user will be used by DX DMP APD server to publish audit data

API	Body	Details
/api/users/user/permissions	As shown below	Set permission for a user to publish audit information

Body for the API request

 "permissions": [
        {
          "vhost": "IUDX-INTERNAL",
          "permission": {
            "configure": "^$",
            "write": "^auditing$",
            "read": "^$"
          }
        }
]

PostgresQL

To setup PostgreSQL refer setup and installation instructions available here
Note : PostgreSQL database should be configured with a RBAC user having CRUD privileges

Table Name	Purpose
user_table	To store user related information like first name, last name, email etc, that is fetched from AAA Server
resource_entity	To store resource information fetched from catalogue
policy	To store policy related information, resource info, consumer and provider info
product	For storing provider, product related information
merchant_table	For storing linked account related information of the provider along with razorpay account_id, account product ID, business details
product_resource_relation	For mapping the product ID with resource ID(s)
product_variant	For storing product variants of a given product
order_table	For storing order ID from Razorpay, amount of the product variant, currency used etc.,
payment_table	To store payment related information like paymentId, payment signature, orderId from Razorpay to verify a payment
invoice	To store purchase related information

Auditing

Auditing is done using the DX Auditing Server which uses Immudb and Postgres for storing the audit logs
To setup immuclient for immudb please refer immudb setup guide
The schema for auditing table in PostgreSQL is present here - postgres auditing table schema
The schema for Immudb table, index for the table is present here - immudb schema in DX Auditing Server
Auditing is currently done for Create Linked account, Update Linked account and Create Policy

Table Name	Purpose	DB
auditing_dmp	To store logged information about endpoint, caller of the endpoint, timestamp when the POST, DELETE, PUT requests respond with 200 success response	Immudb, PostgreSQL

Note : User ID, endpoint and epoch time are indexed in Immudb to retrieve the logs faster

Database Migration using Flyway

Database flyway migrations help in updating the schema, permissions, grants, triggers etc., with the latest version
Each flyway schema file is versioned with the format V<majorVersion>_<minorVersion>__<description>.sql, ex : V1_1__init-tables.sql
Schemas for PostgreSQL tables are present here - Flyway schema
Values like DB URL, database user credentials, user and schema name should be populated in flyway.conf
The following commands shall be executed
- mvn flyway:info -Dflyway.configFiles=flyway.conf To get the flyway schema history table
- mvn clean flyway:migrate -Dflyway.configFiles=flyway.conf To migrate flyway schema
- mvn flyway:repair To resolve some migration errors during flyway migration
Please find the reference to Flyway migration here

Installation Steps

Maven

Install java 11 and maven
Use the maven exec plugin based starter to start the server. Goto the root folder where the pom.xml file is present and run the below command. mvn clean compile exec:java@rs-apd-server

JAR

Install java 11 and maven
Set Environment variables

export DMP_APD_URL=https://<dmp-apd-domain-name>
export LOG_LEVEL=INFO

Use maven to package the application as a JAR. Goto the root folder where the pom.xml file is present and run the below command. mvn clean package -Dmaven.test.skip=true
2 JAR files would be generated in the target/ directory
- iudx.data.marketplace-cluster-1.0.0-SNAPSHOT-fat.jar - clustered vert.x containing micrometer metrics
- iudx.data.marketplace-dev-1.0.0-SNAPSHOT-fat.jar - non-clustered vert.x and does not contain micrometer metrics

Running the clustered JAR

Note: The clustered JAR requires Zookeeper to be installed. Refer here to learn more about how to set up Zookeeper. Additionally, the zookeepers key in the config being used needs to be updated with the IP address/domain of the system running Zookeeper. The JAR requires 3 runtime arguments when running:

--config/-c : path to the config file
--host/-i : the hostname for clustering
--isClustered/-C : true for clustering and false for non-clustered mode
--modules/-m : comma separated list of module names to deploy

e.g. java -jar target/iudx.iudx.apd.acl.server-cluster-0.0.1-SNAPSHOT-fat.jar --host $(hostname) -C true -c secrets/all-verticles-configs/config.json -m iudx.data.marketplace.authenticator.AuthenticationVerticle, iudx.data.marketplace.apiserver.ApiServerVerticle, iudx.data.marketplace.consumer.ConsumerVerticle, iudx.data.marketplace.product.ProductVerticle, iudx.data.marketplace.auditing.AuditingVerticle

e.g. java -jar target/iudx.data.marketplace-cluster-1.0.0-SNAPSHOT-fat.jar -c secrets/all-verticles-configs/config.json --isClustered true --host localhost --modules all

Use the --help/-h argument for more information. You may additionally append an ACL_APD_JAVA_OPTS environment variable containing any Java options to pass to the application.

e.g.

$ export DMP_APD_JAVA_OPTS="Xmx40496m"
$ java $DMP_APD_JAVA_OPTS -jar target/iudx.data.marketplace-cluster-1.0.0-SNAPSHOT-fat.jar ...

Running the non-clustered JAR

The JAR requires 1 runtime argument when running

--config/-c : path to the config file

e.g. java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory -jar -target/iudx.data.marketplace-dev-1.0.0-SNAPSHOT-fat.jar -c secrets/all-verticles-configs/config.json -C false

Use the --help/-h argument for more information. You may additionally append an RS_JAVA_OPTS environment variable containing any Java options to pass to the application.