Use for CI/CD? Recipe and use-case idea for documenting.

[nelsonjchen]

I wonder if anyone has tried hooking this stuff up to their CI/CD system. Leasing a pre-nuked account sounds really great for start-up speed and disposing the environment immediately afterwards keeps tests running in clean environments.

[eschwartz]

@nelsonjchen this project actually uses DCE in it's own CI/CD system, to deploy a to a test environment. Take a look at the Azure Pipelines config in pipeline.yml. Every PR will:

• Lease a new AWS account from a pre-configured pool of test accounts (managed by a dedicated DCE test instance)
• Generate credentials against the leased account using dce leases login
• Deploy a new instance of DCE to that test account, using the PR code (terraform + go packages). It's a single account, with master and child merged.
• Run functional tests against the PR-version of DCE
• Tear down the PR-version of DCE
• Unlease the account, to go back into the testing pool.

I think it's a pretty neat use-case, if you can get it working. I'm no longer active on this project, but it was a great setup at the time.

[nelsonjchen]

I think it's a pretty neat use-case, if you can get it working. I'm no longer active on this project, but it was a great setup at the time.

Hmm, is there a better setup nowadays?

[eschwartz]

There's always room for improvement! Give it a go, I'd be interested to hear how it works out.

[nelsonjchen]

I was debating whether or not to throw in a yo-dawg joke when I made this issue. Good to know we're on the same page!

[jayanandagit]

Hi @nelsonjchen,
You mentioned an ideal implementation scenario for DCE. We use DCE that is CI/CD enabled. Our CI leases one of the Pool's aws account and deploys the DCE artifacts to that account. It then runs applicable unit tests and functional tests, and upon successful execution, it finally destroys the artifacts and the lease. You may also want to look at the azure DevOps pipeline for reference. I hope this insight helps. Let us know if you come up with any enhancement ideas, and we would love to add them to our product backlog.

[nelsonjchen]

Probably nothing specifically about enhancement to DCE but this recipe thing at the moment.

I might be interested in producing in a proof of concept pipeline using GitHub Actions though and writing up about it. Azure Pipelines is great and was very useful when GitHub Actions wasn't available or mature but a GitHub Action-based setup is the lowest friction IMO since it is integrated and an "external" service doesn't need to be setup.

[bytebounder]

Your choice of GitHub actions makes sense to me! We have been comparing these two recently and are pretty excited about the gh actions potential, especially around managing compliance and security for CI/CD and automated workflows. When you can, We'd love to hear about your experience with DCE and GitHub Actions.

Thanks!

[nelsonjchen]

I'm a more interested in making this or the recipe usable for a GitHub OSS project with the security issues that come by as well as some sort of limiter or maybe even a spinlock fallback on DCE account leasing acquisition for capacity reasons.

For security, I'm interested in and desire something that allows the maintainers of a project to mark and approve a PR to run with DCE secrets. I'm looking into using something like Bors-NG or something that'll allow a marked and trusted PR to run with secrets. Unfortunately, Bors-NG imposes a workflow. It maybe "common sense" to them but it might not be right for everyone.

For capacity, GitHub can provide many more "runners" than the default account AWS Organization account limit of 4. There's currently no way to limit the parallelism of free runners purely on GitHub. I don't think there's a way to block GitHub actions from running so maybe a spinlock on getting leases is needed. Or maybe this can be helped by that Bors bot or something similar.

Anyway, those are just some thoughts for now.

[hutchic]

Could I interest optum in taking over the github action I wrote on behalf of my employer observeinc?

It's a little rough around the edges but for for the purposes of a 1.0 it gets the job done