Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accept CLI option for the number of parallel ops in a test run's plan/apply #36323

Merged
merged 6 commits into from
Feb 5, 2025
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changes/unreleased/ENHANCEMENTS-20250123-101838.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: ENHANCEMENTS
body: Terraform Test command now accepts a -parallelism=n option, which sets the number of parallel operations in a test run's plan/apply operation.
time: 2025-01-23T10:18:38.979866+01:00
custom:
Issue: "34237"
6 changes: 3 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,12 @@ require (
github.com/hashicorp/go-plugin v1.6.0
github.com/hashicorp/go-retryablehttp v0.7.7
github.com/hashicorp/go-slug v0.16.3
github.com/hashicorp/go-tfe v1.70.0
github.com/hashicorp/go-tfe v1.74.1
github.com/hashicorp/go-uuid v1.0.3
github.com/hashicorp/go-version v1.7.0
github.com/hashicorp/hcl v1.0.0
github.com/hashicorp/hcl/v2 v2.23.1-0.20250203194505-ba0759438da2
github.com/hashicorp/jsonapi v1.3.1
github.com/hashicorp/jsonapi v1.3.2
github.com/hashicorp/terraform-registry-address v0.2.3
github.com/hashicorp/terraform-svchost v0.1.1
github.com/hashicorp/terraform/internal/backend/remote-state/azure v0.0.0-00010101000000-000000000000
Expand Down Expand Up @@ -256,7 +256,7 @@ require (
go.opentelemetry.io/proto/otlp v1.0.0 // indirect
golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 // indirect
golang.org/x/sync v0.10.0 // indirect
golang.org/x/time v0.7.0 // indirect
golang.org/x/time v0.9.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/api v0.126.0 // indirect
google.golang.org/appengine v1.6.8 // indirect
Expand Down
16 changes: 8 additions & 8 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1123,8 +1123,8 @@ github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerX
github.com/hashicorp/go-sockaddr v1.0.2 h1:ztczhD1jLxIRjVejw8gFomI1BQZOe2WoVOu0SyteCQc=
github.com/hashicorp/go-sockaddr v1.0.2/go.mod h1:rB4wwRAUzs07qva3c5SdrY/NEtAUjGlgmH/UkBUC97A=
github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4=
github.com/hashicorp/go-tfe v1.70.0 h1:R5a9Z+jdVz6eRWtSLsl1nw+5Qe/swunZcJgeKK5NQtQ=
github.com/hashicorp/go-tfe v1.70.0/go.mod h1:2rOcdTxXwbWm0W7dCKjC3Ec8KQ+HhW165GiurXNshc4=
github.com/hashicorp/go-tfe v1.74.1 h1:I/8fOwSYox17IZV7SULIQH0ZRPNL2g/biW6hHWnOTVY=
github.com/hashicorp/go-tfe v1.74.1/go.mod h1:kGHWMZ3HHjitgqON8nBZ4kPVJ3cLbzM4JMgmNVMs9aQ=
github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
Expand All @@ -1141,8 +1141,8 @@ github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/hashicorp/hcl/v2 v2.23.1-0.20250203194505-ba0759438da2 h1:JP8y98OtHTujECs4s/HxlKc5yql/RlC99Dt1Iz4R+lM=
github.com/hashicorp/hcl/v2 v2.23.1-0.20250203194505-ba0759438da2/go.mod h1:k+HgkLpoWu9OS81sy4j1XKDXaWm/rLysG33v5ibdDnc=
github.com/hashicorp/jsonapi v1.3.1 h1:GtPvnmcWgYwCuDGvYT5VZBHcUyFdq9lSyCzDjn1DdPo=
github.com/hashicorp/jsonapi v1.3.1/go.mod h1:kWfdn49yCjQvbpnvY1dxxAuAFzISwrrMDQOcu6NsFoM=
github.com/hashicorp/jsonapi v1.3.2 h1:gP3fX2ZT7qXi+PbwieptzkspIohO2kCSiBUvUTBAbMs=
github.com/hashicorp/jsonapi v1.3.2/go.mod h1:kWfdn49yCjQvbpnvY1dxxAuAFzISwrrMDQOcu6NsFoM=
github.com/hashicorp/logutils v1.0.0 h1:dLEQVugN8vlakKOUE3ihGLTZJRB4j+M2cdTm/ORI65Y=
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
github.com/hashicorp/mdns v1.0.4/go.mod h1:mtBihi+LeNXGtG8L9dX59gAEa12BDtBQSp4v/YAJqrc=
Expand Down Expand Up @@ -1460,8 +1460,8 @@ github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.563/go.mod h1:7sCQWVkxcsR38nffDW057DRGk8mUjK1Ing/EFOK8s8Y=
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.588 h1:DYtBXB7sVc3EOW5horg8j55cLZynhsLYhHrvQ/jXKKM=
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.588/go.mod h1:7sCQWVkxcsR38nffDW057DRGk8mUjK1Ing/EFOK8s8Y=
Expand Down Expand Up @@ -1917,8 +1917,8 @@ golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxb
golang.org/x/time v0.0.0-20220922220347-f3bd1da661af/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.1.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.7.0 h1:ntUhktv3OPE6TgYxXWv9vKvUSJyIFJlyohwbkEwPrKQ=
golang.org/x/time v0.7.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/time v0.9.0 h1:EsRrnYcQiGH+5FfbgvV4AP7qEZstoyrHB0DzarOQ4ZY=
golang.org/x/time v0.9.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
Expand Down
2 changes: 1 addition & 1 deletion internal/backend/remote-state/kubernetes/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ require (
golang.org/x/sys v0.29.0 // indirect
golang.org/x/term v0.28.0 // indirect
golang.org/x/text v0.21.0 // indirect
golang.org/x/time v0.7.0 // indirect
golang.org/x/time v0.9.0 // indirect
golang.org/x/tools v0.25.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
Expand Down
4 changes: 2 additions & 2 deletions internal/backend/remote-state/kubernetes/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -492,8 +492,8 @@ golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.7.0 h1:ntUhktv3OPE6TgYxXWv9vKvUSJyIFJlyohwbkEwPrKQ=
golang.org/x/time v0.7.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/time v0.9.0 h1:EsRrnYcQiGH+5FfbgvV4AP7qEZstoyrHB0DzarOQ4ZY=
golang.org/x/time v0.9.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
Expand Down
5 changes: 5 additions & 0 deletions internal/cloud/test.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,10 @@ type TestSuiteRunner struct {
// Verbose tells the runner to print out plan files during each test run.
Verbose bool

// OperationParallelism is the limit Terraform places on total parallel operations
// during the plan or apply command within a single test run.
OperationParallelism int

// Filters restricts which test files will be executed.
Filters []string

Expand Down Expand Up @@ -204,6 +208,7 @@ func (runner *TestSuiteRunner) Test() (moduletest.Status, tfdiags.Diagnostics) {
Filters: runner.Filters,
TestDirectory: tfe.String(runner.TestingDirectory),
Verbose: tfe.Bool(runner.Verbose),
Parallelism: tfe.Int(runner.OperationParallelism),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to sanity check, that this being 0 will be overridden with a default at the API level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the value is not given, i.e when the API calls the CLI, we use a default value. That override is done here, not at the API level

Variables: func() []*tfe.RunVariable {
runVariables := make([]*tfe.RunVariable, 0, len(variables))
for name, value := range variables {
Expand Down
75 changes: 75 additions & 0 deletions internal/cloud/test_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,81 @@ Success! 2 passed, 0 failed.
}
}

func TestTest_Parallelism(t *testing.T) {

streams, _ := terminal.StreamsForTesting(t)
view := views.NewTest(arguments.ViewHuman, views.NewView(streams))

colorize := mockColorize()
colorize.Disable = true

mock := NewMockClient()
client := &tfe.Client{
ConfigurationVersions: mock.ConfigurationVersions,
Organizations: mock.Organizations,
RegistryModules: mock.RegistryModules,
TestRuns: mock.TestRuns,
}

if _, err := client.Organizations.Create(context.Background(), tfe.OrganizationCreateOptions{
Name: tfe.String("organisation"),
}); err != nil {
t.Fatalf("failed to create organisation: %v", err)
}

if _, err := client.RegistryModules.Create(context.Background(), "organisation", tfe.RegistryModuleCreateOptions{
Name: tfe.String("name"),
Provider: tfe.String("provider"),
RegistryName: "app.terraform.io",
Namespace: "organisation",
}); err != nil {
t.Fatalf("failed to create registry module: %v", err)
}

runner := TestSuiteRunner{
// Configuration data.
ConfigDirectory: "testdata/test",
TestingDirectory: "tests",
Config: nil, // We don't need this for this test.
Source: "app.terraform.io/organisation/name/provider",

// Cancellation controls, we won't be doing any cancellations in this
// test.
Stopped: false,
Cancelled: false,
StoppedCtx: context.Background(),
CancelledCtx: context.Background(),

// Test Options, empty for this test.
GlobalVariables: nil,
Verbose: false,
OperationParallelism: 4,
Filters: nil,

// Outputs
Renderer: &jsonformat.Renderer{
Streams: streams,
Colorize: colorize,
RunningInAutomation: false,
},
View: view,
Streams: streams,

// Networking
Services: nil, // Don't need this when the client is overridden.
clientOverride: client,
}

_, diags := runner.Test()
if len(diags) > 0 {
t.Errorf("found diags and expected none: %s", diags.ErrWithWarnings())
}

if mock.TestRuns.parallelism != 4 {
t.Errorf("expected parallelism to be 4 but was %d", mock.TestRuns.parallelism)
}
}

func TestTest_JSON(t *testing.T) {

streams, done := terminal.StreamsForTesting(t)
Expand Down
34 changes: 31 additions & 3 deletions internal/cloud/tfe_client_mock.go
Original file line number Diff line number Diff line change
Expand Up @@ -1673,9 +1673,10 @@ type MockTestRuns struct {
client *MockClient

// TestRuns and modules keep track of our tfe.TestRun objects.
TestRuns map[string]*tfe.TestRun
modules map[string][]*tfe.TestRun
logs map[string]string
TestRuns map[string]*tfe.TestRun
modules map[string][]*tfe.TestRun
logs map[string]string
parallelism int

// delayedCancel allows a mock test run to cancel an operation instead of
// completing an operation. It's used
Expand Down Expand Up @@ -1767,6 +1768,9 @@ func (m *MockTestRuns) Create(ctx context.Context, options tfe.TestRunCreateOpti
"test.log",
)
m.modules[tr.RegistryModule.ID] = append(m.modules[tr.RegistryModule.ID], tr)
if options.Parallelism != nil {
m.parallelism = *options.Parallelism
}

return tr, nil
}
Expand Down Expand Up @@ -2170,6 +2174,30 @@ func (m *MockWorkspaces) UpdateByID(ctx context.Context, workspaceID string, opt
return w, nil
}

func (m *MockWorkspaces) ListEffectiveTagBindings(ctx context.Context, workspaceID string) ([]*tfe.EffectiveTagBinding, error) {
w, ok := m.workspaceIDs[workspaceID]
if !ok {
return nil, tfe.ErrResourceNotFound
}
var effectiveTagBindings []*tfe.EffectiveTagBinding
for _, tb := range w.TagBindings {
effectiveTagBindings = append(effectiveTagBindings, &tfe.EffectiveTagBinding{
Key: tb.Key,
Value: tb.Value,
})
}
return effectiveTagBindings, nil
}

func (m *MockWorkspaces) DeleteAllTagBindings(ctx context.Context, workspaceID string) error {
w, ok := m.workspaceIDs[workspaceID]
if !ok {
return tfe.ErrResourceNotFound
}
w.TagBindings = nil
return nil
}

func updateMockWorkspaceAttributes(w *tfe.Workspace, options tfe.WorkspaceUpdateOptions) error {
// for TestCloud_setUnavailableTerraformVersion
if w.Name == "unavailable-terraform-version" && options.TerraformVersion != nil {
Expand Down
9 changes: 9 additions & 0 deletions internal/command/arguments/test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ type Test struct {
// will be executed.
Filter []string

// OperationParallelism is the limit Terraform places on total parallel operations
// during the plan or apply command within a single test run.
OperationParallelism int

// TestDirectory allows the user to override the directory that the test
// command will use to discover test files, defaults to "tests". Regardless
// of the value here, test files within the configuration directory will
Expand Down Expand Up @@ -55,6 +59,7 @@ func ParseTest(args []string) (*Test, tfdiags.Diagnostics) {
cmdFlags.BoolVar(&jsonOutput, "json", false, "json")
cmdFlags.StringVar(&test.JUnitXMLFile, "junit-xml", "", "junit-xml")
cmdFlags.BoolVar(&test.Verbose, "verbose", false, "verbose")
cmdFlags.IntVar(&test.OperationParallelism, "parallelism", DefaultParallelism, "parallelism")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking of the future, if we ever want the users to control parallelism of the test runs themselves. Perhaps this should actually be operation-parallelism instead of just parallelism. Or do you think it should be parellelism here and the other one could be test-parallelism?

Copy link
Contributor Author

@dsa0x dsa0x Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are going to have that as a flag, then yes operation-parallelism makes sense. However, I think we can let runs configuration be defined in the file config block that we will be introducing. If we then ever introduce file parallelism, then file-parallelism seems fine. What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's likely we'll still get a request to control the parallelism from the CLI even if we provide the option in the configuration. For example, we have #34359 asking to override the command attribute of run blocks from the CLI.

But, perhaps keeping parallelism to mean the operation parallelism across all commands (so it's the same flag for plan, apply, test, etc) is a good idea. We can then lean into file-parallelism and run-parallelism as options for that level of customisation.


// TODO: Finalise the name of this flag.
cmdFlags.StringVar(&test.CloudRunSource, "cloud-run", "", "cloud-run")
Expand All @@ -73,6 +78,10 @@ func ParseTest(args []string) (*Test, tfdiags.Diagnostics) {
"The -junit-xml option is currently not compatible with remote test execution via the -cloud-run flag. If you are interested in JUnit XML output for remotely-executed tests please open an issue in GitHub."))
}

if test.OperationParallelism < 1 {
test.OperationParallelism = DefaultParallelism
}

switch {
case jsonOutput:
test.ViewType = ViewJSON
Expand Down
Loading
Loading