Specify publish dependencies on deployments

docs/docs/helm/deployments.mdx

@@ -87,7 +87,9 @@ There are quite a few things to notice in the previous example:
 
 The `helm_deployment` target has many additional fields including the target kubernetes namespace, adding inline override values (similar to using helm's `--set` arg) and many others. Please run `pants help helm_deployment` to see all the possibilities.
 
-## Dependencies with `docker_image` targets
+## Dependencies
+
+### With `docker_image` targets
 
 A Helm deployment will in most cases deploy one or more Docker images into Kubernetes. Furthermore, it's quite likely there is going to be at least a few first party Docker images among those. Pants is capable of analysing the Helm chart being used in a deployment to detect those required first-party Docker images using Pants' target addresses to those Docker images.
 
@@ -156,6 +158,18 @@ Pants' will rely on the behaviour of the `docker_image` target when it comes dow
 It's good practice to publish your Docker images using tags other than `latest` and Pants preferred behaviour is to choose those as this guarantees that the _version_ of the Docker image being deployed is the expected one.
 :::
 
+### Other publish dependencies
+
+You can automatically `publish` items before deploying. Add the publishable target to the `publish_dependencies` field. This is useful if Pants cannot infer the dependency on a `docker_image` target.
+
+```python tab={"label":"BUILD"}
+docker_image(name="my_container")
+
+helm_chart(name="my_chart")
+
+helm_deployment(name="my_deployment", chart=":my_chart", publish_dependencies=[":my_container"])
+```
+
 ## Value files
 
 It's very common that Helm deployments use a series of files providing with values that customise the given chart. When using deployments that may have more than one YAML file as the source of configuration values, the Helm backend needs to sort the file names in a way that is consistent across different machines, as the order in which those files are passed to the Helm command is relevant. The final order depends on the same order in which those files are specified in the `sources` field of the `helm_deployment` target. For example, given the following `BUILD` file:

docs/docs/terraform/index.mdx

@@ -129,9 +129,9 @@ Pants can generate multi-platform lockfiles for Terraform. The setting `[downloa
 platforms = ["windows_amd64", "darwin_amd64", "linux_amd64"]
 ```
 
-### Basic Operations
+## Basic Operations
 
-#### Formatting
+### Formatting
 
 Run `terraform fmt` as part of the `fix`, `fmt`, or `lint` goals.
 
@@ -142,7 +142,7 @@ pants fix ::
 ✓ terraform-fmt made no changes.
 ```
 
-#### Validate
+### Validate
 
 Run `terraform validate` as part of the `check` goal.
 
@@ -161,7 +161,7 @@ Success! The configuration is valid.
 terraform_module(skip_terraform_validate=True)
 ```
 
-#### Deploying
+### Deploying
 
 :::caution Terraform deployment support is in alpha stage
 Many options and features aren't supported yet.
@@ -192,3 +192,15 @@ To run `terraform plan`, use the `--dry-run` flag of the `experimental-deploy` g
 ```
 pants experimental-deploy --dry-run ::
 ```
+
+#### Publishing before deploying
+
+You can automatically `publish` items before deploying. Add the publishable target to the `publish_dependencies` field. An example usecase is publishing a Docker image before deploying resources that will use it.
+
+```python tab={"label":"BUILD"}
+docker_image(name="my_container")
+
+terraform_module(name="my_terraform")
+
+terraform_deployment(name="my_deployment", root_module=":my_terraform", publish_dependencies=[":my_container"])
+```

docs/docs/writing-plugins/common-plugin-tasks/adding-a-deployment.mdx

@@ -0,0 +1,120 @@
+---
+    title: Adding a Deployment
+    sidebar_position: 11
+---
+
+How to create a custom deployment that is deployed with the `experimental-deploy` goal
+
+---
+
+The `experimental-deploy` goal is experimental. Changes may be sudden, frequent, and breaking.
+
+This guide will walk you through implementing a target that supports being deployed. The advantage of this over implementing a `run` goal are sandboxed execution and the pre-publishing dependent targets.
+
+## 1. Create target and fieldsets for the target
+
+These allow for specifying the target in the BUILD file and referencing its contents
+
+```python  tab={"label": "pants-plugins/my_deployment/target_types.py"}
+from dataclasses import dataclass
+
+from pants.core.goals.deploy import DeployFieldSet, DeploymentPublishDependencies
+from pants.engine.target import COMMON_TARGET_FIELDS, Dependencies, DescriptionField, Target
+
+
+# Example for fields your deployment might have
+class MyDeploymentDependenciesField(Dependencies):
+    pass
+
+
+@dataclass(frozen=True)
+class MyDeploymentTarget(Target):
+    alias = "my_deployment"
+    core_fields = {
+        *COMMON_TARGET_FIELDS,
+        DeploymentPublishDependencies,  # This enables manually specifying dependencies to publish before deploying
+        MyDeploymentDependenciesField,
+    }
+
+
+@dataclass(frozen=True)
+class MyDeploymentFieldSet(DeployFieldSet):
+    required_fields = (
+        MyDeploymentDependenciesField,
+    )
+    description: DescriptionField
+    dependencies: MyDeploymentDependenciesField
+```
+
+## 2. Create a DeployFieldSet subclass
+
+A subclass of DeployFieldSet will ensure that it has the general fields of a deployment.
+
+```python tab={"label": "pants-plugins/my_deployment/deploy.py"}
+@dataclass(frozen=True)
+class DeployMyDeploymentFieldSet(MyDeploymentFieldSet, DeployFieldSet):
+    pass
+```
+
+
+## 3. Rules to deploy
+
+Create a rule from the `DeployFieldSet` subclass to `DeployProcess` to actually deploy
+
+```python tab={"label": "pants-plugins/my_deployment/deploy.py"}
+from pants.core.goals.deploy import DeployProcess, DeploySubsystem
+from pants.engine.process import InteractiveProcess, Process
+from pants.option.global_options import KeepSandboxes
+
+@rule(desc="Deploy my deployment")
+async def run_my_deploy(
+    field_set: DeployMyDeploymentFieldSet,
+    deploy_subsystem: DeploySubsystem,
+    keep_sandboxes: KeepSandboxes,
+) -> DeployProcess:
+    # If your deployment supports dry-run, you can hook into the flag here
+    if deploy_subsystem.dry_run:
+        ...
+    else:
+        ...
+
+    deploy_process = InteractiveProcess.from_process(
+        Process(...),  # Implementation of the command invocation
+        keep_sandboxes=keep_sandboxes
+    )
+
+    publish_dependencies = ...  # you can infer dependencies that need to be published before the deployment
+
+    return DeployProcess(
+        name=field_set.address.spec,
+        process=deploy_process,
+        publish_dependencies=publish_dependencies,  # these will be published before the deployment
+    )
+```
+
+## 4. Register rules
+
+At the bottom of the file, let Pants know what your rules and types do. Update your plugin's `register.py` to tell Pants about them.
+
+```python tab={"label": "pants-plugins/my_deployment/deploy.py"}
+from pants.core.goals.deploy import DeployFieldSet
+from pants.engine.rules import collect_rules
+from pants.engine.unions import UnionRule
+
+
+def rules():
+    return (
+        *collect_rules(),
+        UnionRule(DeployFieldSet, DeployMyDeploymentFieldSet)
+    )
+```
+
+```python tab={"label": "pants-plugins/my_deployment/register.py"}
+from my_deployment import deploy
+
+def rules():
+    return [
+        ...,
+        *deploy.rules()
+    ]
+```

docs/docs/writing-plugins/common-plugin-tasks/index.mdx

@@ -13,4 +13,5 @@
 - [Add Tests](./run-tests.mdx)
 - [Add lockfile support](./plugin-lockfiles.mdx)
 - [Custom `setup-py` kwargs](./custom-python-artifact-kwargs.mdx)
+- [Adding a deployment](./adding-a-deployment.mdx)
 - [Plugin upgrade guide](./plugin-upgrade-guide.mdx)

docs/docs/writing-plugins/common-plugin-tasks/plugin-lockfiles.mdx

@@ -100,7 +100,7 @@ async def generate_lockfile_from_sources(
 
 ## 4. Register rules
 
-At the bottom of the file, let Pants know what your rules and types do. Update your plugin's `register.py` to tell Pants about them/
+At the bottom of the file, let Pants know what your rules and types do. Update your plugin's `register.py` to tell Pants about them.
 
 ```python tab={"label": "pants-plugins/fortran/lockfiles.py"}
 

docs/notes/2.24.x.md

@@ -40,6 +40,10 @@ The "legacy" system will be removed in the 2.25.x series.
 
 Many tools that Pants downloads can now be exported using [the new `export --bin` option](https://www.pantsbuild.org/2.24/reference/goals/export#bin). For example, `pants export --bin="helm"` will export the `helm` binary to `dist/export/bin/helm`. For each tool, all the files are exported to a subfolder in `dist/export/bins/`, and the main executable is linked to `dist/export/bin/`.
 
+#### Experimental-deploy
+
+The new field `publish_dependencies` allows for specifying dependencies that should be published before the deployment is run. Publishing dependencies before deploying can still be temporarily suppressed with the `--no-experimental-deploy-publish-dependencies` option.
+
 ### Backends
 
 #### JVM

src/python/pants/backend/helm/target_types.py

@@ -8,6 +8,7 @@
 
 from pants.backend.helm.resolve.remotes import ALL_DEFAULT_HELM_REGISTRIES
 from pants.base.glob_match_error_behavior import GlobMatchErrorBehavior
+from pants.core.goals.deploy import DeploymentPublishDependencies
 from pants.core.goals.package import OutputPathField
 from pants.core.goals.test import TestTimeoutField
 from pants.engine.internals.native_engine import AddressInput
@@ -526,6 +527,7 @@ class HelmDeploymentTarget(Target):
     alias = "helm_deployment"
     core_fields = (
         *COMMON_TARGET_FIELDS,
+        DeploymentPublishDependencies,
         HelmDeploymentChartField,
         HelmDeploymentReleaseNameField,
         HelmDeploymentDependenciesField,

src/python/pants/backend/terraform/goals/deploy.py

@@ -29,7 +29,7 @@
 
 
 @dataclass(frozen=True)
-class DeployTerraformFieldSet(TerraformDeploymentFieldSet, DeployFieldSet):
+class DeployTerraformFieldSet(TerraformDeploymentFieldSet):
     pass
 
 

src/python/pants/backend/terraform/target_types.py

@@ -5,6 +5,7 @@
 
 from dataclasses import dataclass
 
+from pants.core.goals.deploy import DeployFieldSet, DeploymentPublishDependencies
 from pants.engine.internals.native_engine import AddressInput
 from pants.engine.rules import collect_rules, rule
 from pants.engine.target import (
@@ -117,14 +118,15 @@ class TerraformDeploymentTarget(Target):
     alias = "terraform_deployment"
     core_fields = (
         *COMMON_TARGET_FIELDS,
+        DeploymentPublishDependencies,
         TerraformDependenciesField,
         TerraformRootModuleField,
     )
     help = "A deployment of Terraform"
 
 
 @dataclass(frozen=True)
-class TerraformDeploymentFieldSet(FieldSet):
+class TerraformDeploymentFieldSet(DeployFieldSet):
     required_fields = (
         TerraformDependenciesField,
         TerraformRootModuleField,

src/python/pants/core/goals/deploy.py

@@ -11,6 +11,7 @@
 
 from pants.core.goals.package import PackageFieldSet
 from pants.core.goals.publish import PublishFieldSet, PublishProcesses, PublishProcessesRequest
+from pants.engine.addresses import UnparsedAddressInputs
 from pants.engine.console import Console
 from pants.engine.environment import EnvironmentName
 from pants.engine.goal import Goal, GoalSubsystem
@@ -22,17 +23,33 @@
     FieldSetsPerTarget,
     FieldSetsPerTargetRequest,
     NoApplicableTargetsBehavior,
+    SpecialCasedDependencies,
     Target,
     TargetRootsToFieldSets,
     TargetRootsToFieldSetsRequest,
+    Targets,
 )
 from pants.engine.unions import union
 from pants.option.option_types import BoolOption
-from pants.util.strutil import pluralize, softwrap
+from pants.util.docutil import bin_name
+from pants.util.strutil import help_text, pluralize, softwrap
 
 logger = logging.getLogger(__name__)
 
 
+class DeploymentPublishDependencies(SpecialCasedDependencies):
+    alias = "publish_dependencies"
+    help = help_text(
+        f"""
+        Addresses to targets that should be packaged with the `{bin_name()} experimental-deploy` goal
+        and whose resulting artifacts should be published.
+        Pants will publish the artifacts as if you had run `{bin_name()} publish`.
+
+        For example, this will allow you to publish Docker images that will be used when deploying a Helm chart.
+        """
+    )
+
+
 @union(in_scope_types=[EnvironmentName])
 @dataclass(frozen=True)
 class DeployFieldSet(FieldSet, metaclass=ABCMeta):
@@ -42,6 +59,8 @@ class DeployFieldSet(FieldSet, metaclass=ABCMeta):
     result of the deploy rule.
     """
 
+    publish_dependencies: DeploymentPublishDependencies
+
 
 @dataclass(frozen=True)
 class DeployProcess:
@@ -199,12 +218,28 @@ async def run_deploy(console: Console, deploy_subsystem: DeploySubsystem) -> Dep
         for field_set in target_roots_to_deploy_field_sets.field_sets
     )
 
-    publish_targets = (
+    deploy_publish_dependencies = await MultiGet(
+        Get(
+            Targets,
+            UnparsedAddressInputs,
+            field_set.publish_dependencies.to_unparsed_address_inputs(),
+        )
+        for field_set in target_roots_to_deploy_field_sets.field_sets
+    )
+    specified_publish_targets = (
+        set(chain.from_iterable(deploy_publish_dependencies))
+        if deploy_subsystem.publish_dependencies
+        else set()
+    )
+
+    inferred_publish_targets = (
         set(chain.from_iterable([deploy.publish_dependencies for deploy in deploy_processes]))
         if deploy_subsystem.publish_dependencies
         else set()
     )
 
+    publish_targets = inferred_publish_targets | specified_publish_targets
+
     logger.debug(f"Found {pluralize(len(publish_targets), 'dependency')}")
     publish_processes = await _all_publish_processes(publish_targets)