Initial commit

.github/workflows/main.yml

@@ -0,0 +1,39 @@
+# This workflow will install Python dependencies, run tests and lint with a single version of Python
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: ci
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.10
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.10"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8 pytest
+        pip install .
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        pytest

.gitignore

@@ -0,0 +1,14 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+.coverage
+# uv files
+.python-version
+# Virtual environments
+.venv
+# Development
+local_dev/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Ali Masri
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,86 @@
+# Cloud Build YAML Validator
+
+[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)
+
+A robust and extensible tool for validating Google Cloud Build YAML configuration files against schema specifications and custom rules.
+
+## Features
+
+This validator performs the following checks:
+
+- **YAML Syntax**: Ensures the file is a valid YAML document.
+- **Schema Compliance**: Validates the YAML structure against [Cloud Build specifications](https://cloud.google.com/build/docs/build-config-file-schema).
+- **Step Dependencies**: Verifies that all `waitFor` references point to valid step IDs.
+- **Substitution Variables**: Checks for unreferenced substitution variables and ensures they start with an underscore (`_`).
+- **Custom Validations**: Easily extendable with additional custom validation rules.
+
+## Installation
+
+You can install the Cloud Build YAML Validator using `pip`, `uv`, or your preferred Python package manager:
+
+```bash
+git clone https://github.com/alimasri/google-cloudbuild-yaml-validator
+cd google-cloudbuild-yaml-validator
+pip install -e .
+```
+
+## Usage
+
+```md
+usage: cloudbuild-validator [-h] [-s SCHEMA] -f FILE
+
+options:
+  -h, --help            show this help message and exit
+  -s SCHEMA, --schema SCHEMA
+                        Path to the schema file to validate against
+  -f FILE, --file FILE  Path to the content file to validate
+```
+
+### Example
+
+```bash
+cloudbuild-validator -f /path/to/cloudbuild.yaml
+```
+
+## Specifications
+
+The validator enforces schema specifications for Google Cloud Build YAML configuration files, based on the [official Cloud Build documentation](https://cloud.google.com/build/docs/build-config-file-schema). Users can provide a custom schema file using the `-s` or `--schema` option. The default schema file is located at `src/cloudbuild_validator/data/cloudbuild-specifcations.json`, which can be used as a reference for creating custom schemas. By adhering to this schema, users ensure their Cloud Build configuration files are valid and correctly interpreted by the Cloud Build service. Example modifications could include adding organization-specific patterns for image names, environment variables, or other configuration options.
+
+## Adding New Validations
+
+### Extending the default validations
+
+The validator will automatically discover and execute all `Validator` subclasses in the validators.py file.
+To add a new validation rule, create a new class that inherits from `cloudbuild_validator.validators.Validator` and implement the `validate` method. The `validate` method should accept a dictionary representing the Cloud Build configuration file and raise a `cloudbuild_validator.exceptions.CloudBuildValidationError` if the validation fails. 
+
+#### Example
+
+```python
+class StepIdPrefixValidator(Validator):
+    """Makes sure that step IDs start with a specific prefix."""
+
+    def __init__(self, prefix: str):
+        super().__init__()
+
+    def validate(self, content: dict) -> None:
+      if not step_id.startswith(''):
+        raise CloudBuildValidationError(f"Step ID '{step_id}' does not start with the expected prefix {self.prefix}.")
+```
+
+### Using the `add_validator` method
+
+The `CloudbuildValidator` class provides an `add_validator` method that allows users to add custom validation rules. This method accepts a `Validator` subclass and adds it to the list of validators that will be executed during the validation process.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+Distributed under the MIT License. See LICENSE for more information.
+
+## Acknowledgments
+
+- [Google Cloud Build documentation](https://cloud.google.com/build)
+- [Google Cloud Build YAML schema specifications](https://cloud.google.com/build/docs/build-config-file-schema)

pyproject.toml

@@ -0,0 +1,31 @@
+[project]
+name = "cloudbuild-validator"
+version = "0.1.0"
+description = "A robust and extensible tool for validating Google Cloud Build YAML configuration files against schema specifications and custom rules."
+readme = "README.md"
+authors = [
+    { name = "Ali Masri", email = "[email protected]" }
+]
+requires-python = ">=3.10"
+dependencies = [
+    "loguru>=0.7.2",
+    "pydantic-settings>=2.6.1",
+    "pydantic>=2.9.2",
+    "yamale>=5.2.1",
+]
+
+[project.scripts]
+cloudbuild-validator = "cloudbuild_validator:main.run"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[dependency-groups]
+dev = [
+    "coverage>=7.6.4",
+    "pytest>=8.3.3",
+    "pytest-cov>=6.0.0",
+    "pytest-sugar>=1.0.0",
+    "ruff>=0.7.3",
+]

src/cloudbuild_validator/config.py

@@ -0,0 +1,31 @@
+from pydantic_settings import BaseSettings
+
+
+class Settings(BaseSettings):
+    DEFAULT_SUBSTITUTIONS: list[str] = [
+        "PROJECT_ID",
+        "BUILD_ID",
+        "PROJECT_NUMBER",
+        "LOCATION",
+        "TRIGGER_NAME",
+        "COMMIT_SHA",
+        "REVISION_ID",
+        "SHORT_SHA",
+        "REPO_NAME",
+        "REPO_FULL_NAME",
+        "BRANCH_NAME",
+        "TAG_NAME",
+        "REF_NAME",
+        "TRIGGER_BUILD_CONFIG_PATH",
+        "SERVICE_ACCOUNT_EMAIL",
+        "SERVICE_ACCOUNT",
+        "_HEAD_BRANCH",
+        "_BASE_BRANCH",
+        "_HEAD_REPO_URL",
+        "_PR_NUMBER",
+    ]
+
+    SUBSTITUTION_VARIABLE_PATTERN: str = r"\$\{(_\w+)\}"
+
+
+settings = Settings()

src/cloudbuild_validator/core.py

@@ -0,0 +1,46 @@
+from pathlib import Path
+from typing import List
+
+import yamale
+
+from cloudbuild_validator import validators
+
+
+class CloudBuildValidator:
+    def __init__(self, speficifactions_file: Path, add_default_validators: bool = True):
+        self.validators = []
+        self.schema = yamale.make_schema(speficifactions_file)
+        if add_default_validators:
+            for validator in dir(validators):
+                if (
+                    isinstance(getattr(validators, validator), type)
+                    and issubclass(getattr(validators, validator), validators.Validator)
+                    and getattr(validators, validator) != validators.Validator
+                ):
+                    self.add_validator(getattr(validators, validator)())
+
+    def add_validator(self, validator: validators.Validator):
+        self.validators.append(validator)
+
+    def remove_validator(self, validator: validators.Validator):
+        self.validators.remove(validator)
+
+    def validate(self, yaml_file_path: Path) -> List[str]:
+        content = yamale.make_data(yaml_file_path)
+        if len(content) > 1:
+            raise validators.CloudBuildValidationError(
+                "Multiple documents found in the file"
+            )
+        try:
+            yamale.validate(self.schema, content)
+        except yamale.YamaleError as e:
+            raise validators.CloudBuildValidationError(e) from e
+        content = content[0][0]
+
+        errors = []
+        for validator in self.validators:
+            try:
+                validator.validate(content)
+            except validators.CloudBuildValidationError as e:
+                errors.append(str(e))
+        return errors

src/cloudbuild_validator/data/cloudbuild-specifications.yaml

@@ -0,0 +1,50 @@
+steps: list(include('Step'), min=1)
+timeout: str(required=False)
+queueTtl: str(required=False)
+logsBucket: str(required=False)
+options:
+  env: list(str(), required=False)
+  secretEnv: str(required=False)
+  volumes: include('Volume', required=False)
+  sourceProvenanceHash: enum('MD5', 'SHA256', 'SHA1', required=False)
+  machineType: enum('UNSPECIFIED', 'N1_HIGHCPU_8', 'N1_HIGHCPU_32', 'E2_HIGHCPU_8', 'E2_HIGHCPU_32', required=False)
+  diskSizeGb: int(required=False)
+  substitutionOption: enum('MUST_MATCH', 'ALLOW_LOOSE', required=False)
+  dynamicSubstitutions: bool(required=False)
+  automapSubstitutions: bool(required=False)
+  logStreamingOption: enum('STREAM_DEFAULT', 'STREAM_ON', 'STREAM_OFF', required=False)
+  logging: enum('GCS_ONLY', 'CLOUD_LOGGING_ONLY', required=False)
+  defaultLogsBucketBehavior: str(required=False)
+  pool: map(required=False)
+  requestedVerifyOption: enum('NOT_VERIFIED', 'VERIFIED', required=False)
+  workerPool: str(required=True)
+substitutions: map(str(), str(), required=False)
+tags: list(str(), required=False)
+serviceAccount: str()
+secrets: map(required=False)
+availableSecrets: map(required=False)
+artifacts: include('Artifact', required=False)
+images: list(list(str()), required=False)
+---
+Artifact:
+  mavenArtifacts: list(map(), required=False)
+  pythonPackages: list(map(), required=False)
+  npmPackages: list(map(), required=False)
+Volume: list(map(name=str(), path=str()), required=False)
+TimeSpan:
+  startTime: str()
+  endTime: str()
+Step:
+  name: str()
+  args: list(str(), required=False)
+  env: list(str(), required=False)
+  allowFailure: bool(required=False)
+  dir: str(required=False)
+  id: str()
+  waitFor: list(str(), required=False)
+  entrypoint: str(required=False)
+  secretEnv: list(str(), required=False)
+  volumes: include('Volume', required=False)
+  timeout: str(required=False)
+  script: str(required=False)
+  automapSubstitutions: bool(required=False)

src/cloudbuild_validator/main.py

@@ -0,0 +1,49 @@
+from argparse import ArgumentParser
+from pathlib import Path
+
+from loguru import logger
+
+from cloudbuild_validator.core import CloudBuildValidator
+
+
+def main(schema: Path, content: Path):
+    logger.info("Program started")
+
+    logger.info(f"Validating {content} against {schema}...")
+    validator = CloudBuildValidator(schema)
+    errors = validator.validate(content)
+    if not errors:
+        logger.info("Validation passed")
+        raise SystemExit(0)
+
+    logger.error("Validation failed")
+    for error_msg in errors:
+        logger.error(f"\t{error_msg}")
+
+    raise SystemExit(1)
+
+
+def run():
+    default_schema = Path(__file__).parent / "data" / "cloudbuild-specifications.yaml"
+    parser = ArgumentParser()
+    parser.add_argument(
+        "-s",
+        "--schema",
+        type=Path,
+        help="Path to the schema file to validate against",
+        required=False,
+        default=default_schema,
+    )
+    parser.add_argument(
+        "-f",
+        "--file",
+        type=Path,
+        help="Path to the content file to validate",
+        required=True,
+    )
+    args = parser.parse_args()
+    main(args.schema, args.file)
+
+
+if __name__ == "__main__":
+    run()