Skip to content

Commit

Permalink
Added examples for Portal APIs using IAM authz
Browse files Browse the repository at this point in the history
* This effectively support without needing setup PORTAL_TOKEN
  environment variable but using AWS CLI credential or IAM role.
* Updated README and a couple of examples for possible backend
  and end user ad-hoc use case code snippet.
* R example is still using Python for v4 signing facility and http
  `requests` package; through `reticulate` R library. This can be
  improved to pure R with `httr` and `cloudyr`.
* Related to #415 #377
  • Loading branch information
victorskl committed Mar 9, 2022
1 parent 6598e76 commit 0d85ae1
Show file tree
Hide file tree
Showing 6 changed files with 855 additions and 6 deletions.
68 changes: 65 additions & 3 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,75 @@ _Required:_ You will need [curl](https://curl.se/) and [jq](https://stedolan.git
- PROD: `https://api.data.prod.umccr.org`
- DEV: `https://api.data.dev.umccr.org`

### Setup Portal Token
## Authorization

Portal currently support 2 types of API authorization.
1. Portal Token
2. AWS IAM

### Portal Token

- Follow setting up [Portal Token](PORTAL_TOKEN.md)
- Use appropriate Portal Token depending on PROD or DEV environment
- If you receive `Unauthorised` or similar then Portal Token has either expired or invalid token for target env.
- Token valid for 24 hours (1 day)

### AWS IAM

> NOTE: AWS IAM is for those who have direct access to UMCCR AWS resources and, need to closely knit their solution within UMCCR AWS environment. And, re-using AWS SSO facility for accessing Portal APIs for conveniences. For one-off and out-of-band use cases, please use Portal Token with JWT OAuth flow.
- Basically, Portal APIs has mirrored `/iam/` prefix to all endpoints.
- Required: **AWS IAM credentials** or, assume role for **service-to-service** use case.
- For local dev, this goes along with your AWS CLI setup.
- For service user, you will need to add appropriate permission to "assume-role" policy (see below).
- Append Prefix: `/iam/` to the endpoint. For example:
```
https://api.data.prod.umccr.org/iam/lims
```
- You will then need to make [AWS Signature v4 singed request](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) with AWS credentials to the Portal endpoints. There are readily available existing drop-in v4 signature signing library around. Some pointers are as follows.

#### Assume Role

- Required: Attach the following policy to the service role (IAM Role) permission.
```
{
"Effect": "Allow",
"Action": [
"execute-api:Invoke"
],
"Resource": [
"*"
]
}
```

#### CLI

- Recommend to use [awscurl](https://github.com/okigan/awscurl) in-place of `curl` for IAM endpoints.
- As simplest case, you can wrap the `awscurl` and bash scripting it to query the Portal APIs; Then pipe to post-process with `jq` or any choice of text processor for transformation!
- Example:
- Install `brew install awscurl`
- Login AWS CLI using `ProdOperator` role as per normal
- Then
```
awscurl --profile prodops --region ap-southeast-2 -H "Accept: application/json" "https://api.data.prod.umccr.org/iam/lims" | jq
```

#### Python

- Recommend to use [aws-requests-auth](https://github.com/davidmuller/aws-requests-auth) or [requests-aws4auth](https://github.com/tedder/requests-aws4auth)
- See [examples/portal_api_sig4.py](examples/portal_api_sig4.py)

#### Node.js

- Recommend to use [Amplify.Signer](https://aws-amplify.github.io/amplify-js/api/classes/signer.html) or [aws4](https://github.com/mhart/aws4) or [aws4-axios](https://github.com/jamesmbourne/aws4-axios)
- See [examples/portal_api_sig4.js](examples/portal_api_sig4.js)

#### R
- Recommend to use Python [requests-aws4auth](https://github.com/tedder/requests-aws4auth) through [reticulate](https://rstudio.github.io/reticulate/)
- See [examples/portal_api_sig4.R](examples/portal_api_sig4.R)

## Endpoints

### LIMS Endpoint

Expand Down Expand Up @@ -249,8 +313,6 @@ curl -s -H "Authorization: Bearer $PORTAL_TOKEN" "https://api.data.prod.umccr.or

Similarly, you can filter request parameters on `run_id`, `lane`.

## Experimental

### Reports Endpoint

_(*potentially large JSON response)_
Expand Down
80 changes: 80 additions & 0 deletions docs/examples/portal_api_sig4.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# -*- coding: utf-8 -*-
# Use Case:
# Using Portal API without needing PORTAL_TOKEN but, just natively with your existing AWS CLI credentials.
#
# Note:
# Instead of going through Python library with `reticulate`, we could do the _pure_ R approach with
# just `aws.signature` and `httr`.
#
# Usage:
# conda activate data-portal-apis
# pip install requests-aws4auth
# export AWS_PROFILE=prodops
# Rscript portal_api_sig4.R

depedencies <- list("reticulate", "aws.signature", "jsonlite", "testthat")
invisible(lapply(depedencies, function(d) if (!require(d, character.only = TRUE)) install.packages(d, repos = "https://cran.ms.unimelb.edu.au")))

library(reticulate)
library(aws.signature)
library(jsonlite)
library(testthat)

print_dash <- function(times = 64) {
cat(strrep("-", times))
cat("\n")
}

use_condaenv(condaenv = "data-portal-apis")
py_config()

print_dash()

# ---

py_awsauth <- import("requests_aws4auth")
py_requests <- import("requests")
region <- "ap-southeast-2"
service <- "execute-api"

credentials <- aws.signature::locate_credentials()

authr <- py_awsauth$AWS4Auth(
credentials$key,
credentials$secret,
region,
service,
session_token = credentials$session_token
)

url <- "https://api.data.prod.umccr.org/iam/lims" # using iam endpoint

params <- reticulate::py_dict(
c("rowsPerPage", "subject_id"),
c("1000", "SBJ01651")
)

response <- py_requests$get(url, auth = authr, params = params)

response_list <- response$text
# response_list <- response$json()
# typeof(response_list)
# response_list

response_df <- fromJSON(response_list)

test_that("SBJ01651 should have only 3 samples/libraries", {
expect_equal(response_df$pagination$count, 3)
})
print_dash()

results <- response_df$results
results

print_dash()

cat("All SBJ01651 samples:\n\t")
results$sample_id
results$sample_id[1]
results$sample_id[2]
results$sample_id[3]
27 changes: 27 additions & 0 deletions docs/examples/portal_api_sig4.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// -*- coding: utf-8 -*-
/**
Usage:
yarn add aws4-axios axios aws-sdk
export AWS_PROFILE=prodops
node portal_api_sig4.js
**/
import axios from 'axios';
import AWS from 'aws-sdk';
import {aws4Interceptor} from 'aws4-axios';

// https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html
const credentials = new AWS.SharedIniFileCredentials();

const interceptor = aws4Interceptor(
{
region: 'ap-southeast-2',
service: 'execute-api',
},
credentials
);

axios.interceptors.request.use(interceptor);

axios.get('https://api.data.prod.umccr.org/iam/lims').then((res) => {
console.log(res.data)
});
26 changes: 26 additions & 0 deletions docs/examples/portal_api_sig4.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# -*- coding: utf-8 -*-
"""
Usage:
pip install aws-requests-auth botocore
export AWS_PROFILE=prodops
python portal_api_sig4.py
"""
from urllib.parse import urlparse, ParseResult

import requests
from aws_requests_auth.boto_utils import BotoAWSRequestsAuth

if __name__ == '__main__':
obj: ParseResult = urlparse("https://api.data.prod.umccr.org/iam/lims")

auth = BotoAWSRequestsAuth(
aws_host=obj.hostname,
aws_region="ap-southeast-2",
aws_service="execute-api"
)

response = requests.get(obj.geturl(), auth=auth)

resp_dict: dict = response.json()

print(resp_dict)
9 changes: 7 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@
"description": "UMCCR Data Portal API backend in cloud native serverless",
"license": "MIT",
"version": "0.1.0",
"type": "module",
"dependencies": {
"aws-sdk": "^2.1089.0",
"aws4-axios": "^2.4.9",
"axios": "^0.26.0"
},
"devDependencies": {
"husky": "^7.0.4",
"serverless": "^3.1.1",
Expand All @@ -15,6 +21,5 @@
"scripts": {
"prepare": "husky install"
},
"resolutions": {
}
"resolutions": {}
}
Loading

0 comments on commit 0d85ae1

Please sign in to comment.