Skip to content

Latest commit

 

History

History
71 lines (49 loc) · 2.94 KB

README.md

File metadata and controls

71 lines (49 loc) · 2.94 KB

Goal

Demonstrate how to create a set of tasks with dependencies that runs on a docker container using AWS Batch

Files

  • components/component1.py: Defines a set of functions that works on the container demo2:latest
  • components/component2.py: Defines a set of functions that works on the container demo2:latest
  • attr.py: Meta data for the experiment/workflow
  • batchconfig.json: Batch configuration to specify region and job queue name
  • deployment.py: Implementation of a selection of tasks to be run in the Docker container

Prerequisites

  1. AWS CLI needs to be installed and configured (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html)
  2. AWS Batch environment, job queues and definitions need to be configured (https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html)
  3. Ensure that you have successfully executed demo_2 example and pushed the container from this example to your Elastic Container Registry (ECR) using push_to_ecr.py

How to run the example

Do the following in the order listed below

  • Configure user
aws configure
AWS Access Key ID: 
AWS Secret Access Key: 
Default region name: 
Default output format:
  • Configure the batch queue Modify the batchconfig.json to reflect your chosen region and queue name and other configurations
{
    "region_name": "<AWS-REGION-ID>",
    "jobQueue": "twingraph-run-queue",
    "logGroupName": "/aws/batch/job",
    "vCPU": 1,
    "Mem": 2048
}

An AWS Batch queue/compute environment needs to be set up before this example can be executed. For this initial demo, please set up the compute environment and queue to be 'EC2/ECS'. Fargate options require Role ARN and other information to be passed in as parameters (see API documentation) and EKS on Batch is not currently supported, please use plain EKS example instead from Demo 5.

Remember to modify the jobQueue variable in the above json script to your choice of the AWS Batch queue name.

Run the orchestration pipeline

python deployment.py

If your tasks fail to launch on AWS Batch, ensure your Region ID and Job Queue Names are correct. Additionally ensure that AWS Batch is able to run the getting-started-wizard-job. Please ensure that your Batch instance role has sufficient IAM permissions to access other services such as Amazon CloudWatch, Elastic Container Registry, S3, etc.

If you are new to AWS Batch, we recommend that you use the wizard (https://.console.aws.amazon.com/batch/home?region=#wizard) to setup your AWS Batch environment.

If your task fails to run on AWS Batch, you need clear your local celery task runners before attempting to launch a new AWS Batch job. This can be achieved by running the script as shown below

Change directory to the utilities to clean up:

cd {YOUR DIRECTORY FOR TwinGraph}/examples/utils
python stop_and_delete.py