Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nightly playground #136

Merged
merged 2 commits into from
Jan 24, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/nightly-playground-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Nightly Playground Build and Test

on:
push:
paths:
- nightly-playground/**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think pr trigger is enough here.

pull_request:
paths:
- nightly-playground/**

jobs:
build-and-test:
runs-on: ubuntu-latest
defaults:
run:
working-directory: nightly-playground

steps:
- uses: actions/checkout@v3

- uses: actions/setup-node@v4
with:
node-version: 16.x

- name: Run CDK Build and Test
run: |
npm install
npm run build

- name: Run test coverage
run: |
npm test -- --coverage

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
37 changes: 37 additions & 0 deletions nightly-playground/.eslintrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
module.exports = {
env: {
browser: false,
es6: true,
jest: true,
},
extends: [
'airbnb-base',
],
globals: {
Atomics: 'readonly',
SharedArrayBuffer: 'readonly',
},
parser: '@typescript-eslint/parser',
parserOptions: {
ecmaVersion: 2018,
},
plugins: [
'@typescript-eslint',
],
rules: {
hasTrailingComma: 'off',
indent: ['error', 2],
'import/extensions': 'error',
'import/no-namespace': 'error',
'import/no-unresolved': 'error',
'import/no-extraneous-dependencies': 'error',
'import/prefer-default-export': 'off',
'max-classes-per-file': 'off',
'no-unused-vars': 'off',
'no-new': 'off',
'max-len': ['error', { 'code': 160, 'ignoreComments': true }],
"no-param-reassign": 0,
"no-shadow": "off",
"@typescript-eslint/no-shadow": ["error"]
},
};
9 changes: 9 additions & 0 deletions nightly-playground/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
*.js
!jest.config.js
*.d.ts
node_modules

# CDK asset staging directory
.cdk.staging
cdk.out
cdk.context.json
6 changes: 6 additions & 0 deletions nightly-playground/.npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
*.ts
!*.d.ts

# CDK asset staging directory
.cdk.staging
cdk.out
27 changes: 27 additions & 0 deletions nightly-playground/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# OpenSearch Nightly Playground

This project is an extension of [opensearch-cluster-cdk](https://github.com/opensearch-project/opensearch-cluster-cdk) that deploys nightly built artifacts daily. The source code concentrates on taking care of regular deployments, permissions, access, etc. For more customization, please feel free to directly use [opensearch-cluster-cdk](https://github.com/opensearch-project/opensearch-cluster-cdk).

## Getting Started

- Requires [NPM](https://docs.npmjs.com/cli/v7/configuring-npm/install) to be installed
- Install project dependencies using `npm install` from this project directory
- Configure [aws credentials](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html#getting_started_prerequisites)

## Deployment

### Required context parameters

In order to deploy the stack the user needs to provide a set of required parameters listed below:

| Name | Requirement | Type | Description |
|-------------------------------|:------------|:------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| distVersion | Required | string | The OpenSearch distribution version (released/un-released) the user wants to deploy |
| distributionUrl | Required | string | OpenSearch tarball distribution URL plugin |
| dashboardsUrl | Required | string | OpenSearch-Dashboards tarball distribution URL version |

#### Sample command to set up multi-node cluster with security enabled

```
npm run cdk deploy "*" -- -c distVersion=2.3.0 -c distributionUrl=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.3.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.3.0-linux-x64.tar.gz -c dashboardsUrl=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.3.0/latest/linux/x64/tar/dist/opensearch-dashboards/opensearch-dashboards-2.3.0-linux-x64.tar.gz
```
20 changes: 20 additions & 0 deletions nightly-playground/bin/nightly-playground.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/env node

/* Copyright OpenSearch Contributors
SPDX-License-Identifier: Apache-2.0

The OpenSearch Contributors require contributions made to
this file be licensed under the Apache-2.0 license or a
compatible open source license. */

import { App } from 'aws-cdk-lib';
import { NightlyPlaygroundStack } from '../lib/nightly-playground-stack';

const app = new App();
const region = app.node.tryGetContext('region') ?? process.env.CDK_DEFAULT_REGION;
const account = app.node.tryGetContext('account') ?? process.env.CDK_DEFAULT_ACCOUNT;
const playgroundId = app.node.tryGetContext('playgroundId') ?? '2x';

new NightlyPlaygroundStack(app, playgroundId, {
env: { account, region },
});
40 changes: 40 additions & 0 deletions nightly-playground/cdk.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"app": "npx ts-node --prefer-ts-exts bin/nightly-playground.ts",
"watch": {
"include": [
"**"
],
"exclude": [
"README.md",
"cdk*.json",
"**/*.d.ts",
"**/*.js",
"tsconfig.json",
"package*.json",
"yarn.lock",
"node_modules",
"test"
]
},
"context": {
"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
"@aws-cdk/core:stackRelativeExports": true,
"@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
"@aws-cdk/aws-lambda:recognizeVersionProps": true,
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true,
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
"@aws-cdk/core:checkSecretUsage": true,
"@aws-cdk/aws-iam:minimizePolicies": true,
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
"@aws-cdk/core:target-partitions": [
"aws",
"aws-cn"
]
}
}
10 changes: 10 additions & 0 deletions nightly-playground/jest.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
module.exports = {
testEnvironment: 'node',
roots: ['<rootDir>/test'],
preset: 'ts-jest',
testMatch: ['**/*.test.ts'],
transform: {
'^.+\\.tsx?$': 'ts-jest',
},
testTimeout: 50000,
};
57 changes: 57 additions & 0 deletions nightly-playground/lib/nightly-playground-stack.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/* Copyright OpenSearch Contributors
SPDX-License-Identifier: Apache-2.0

The OpenSearch Contributors require contributions made to
this file be licensed under the Apache-2.0 license or a
compatible open source license. */

import { InfraStack } from '@opensearch-project/opensearch-cluster-cdk/lib/infra/infra-stack';
import { NetworkStack } from '@opensearch-project/opensearch-cluster-cdk/lib/networking/vpc-stack';
import { Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class NightlyPlaygroundStack {
public stacks: Stack[] = []; // only required for testing purpose

constructor(scope: Construct, id: string, props: StackProps) {
const distVersion = scope.node.tryGetContext('distVersion');
if (distVersion === 'undefined') {
throw new Error('distVersion parameter cannot be empty! Please provide the OpenSearch distribution version');
}
const distributionUrl = scope.node.tryGetContext('distributionUrl');
peterzhuamazon marked this conversation as resolved.
Show resolved Hide resolved
if (distributionUrl === 'undefined') {
throw new Error('distributionUrl parameter cannot be empty! Please provide the OpenSearch distribution URL');
}
const dashboardsUrl = scope.node.tryGetContext('dashboardsUrl');
peterzhuamazon marked this conversation as resolved.
Show resolved Hide resolved
if (dashboardsUrl === 'undefined') {
throw new Error('dashboardsUrl parameter cannot be empty! Please provide the OpenSearch-Dashboards distribution URL');
}

// @ts-ignore
const networkStack = new NetworkStack(scope, `networkStack-${id}`, {
...props,
serverAccessType: 'prefixList',
restrictServerAccessTo: 'pl-f8a64391',
});

this.stacks.push(networkStack);

// @ts-ignore
const infraStack = new InfraStack(scope, `infraStack-${id}`, {
...props,
vpc: networkStack.vpc,
securityGroup: networkStack.osSecurityGroup,
cpuArch: 'x64',
opensearchVersion: distVersion,
minDistribution: false,
securityDisabled: false,
enableMonitoring: true,
distributionUrl,
singleNodeCluster: false,
dashboardsUrl,
});
this.stacks.push(infraStack);

infraStack.addDependency(networkStack);
}
}
Loading
Loading