Skip to content

Commit

Permalink
Create e2e provisioning API test (#1371)
Browse files Browse the repository at this point in the history
* Create e2e provisioning API test

* Test bindings API

* Test secret existence with kubeconfig from binding

* Refactor files structure

* Rename folder

* Update docs

* Rename test

* Change require no error to assertion

* Update end-2-end tests docs

* Extract paths to functions

* Update docs

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs/contributor/05-10-e2e_tests.md

Co-authored-by: Iwona Langer <[email protected]>

* Update docs

* Update docs

* Update docs

---------

Co-authored-by: Iwona Langer <[email protected]>
  • Loading branch information
KsaweryZietara and IwonaLanger authored Oct 28, 2024
1 parent 251b8bc commit 9244517
Show file tree
Hide file tree
Showing 9 changed files with 808 additions and 40 deletions.
117 changes: 78 additions & 39 deletions docs/contributor/05-10-e2e_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@

## Overview

The end-to-end (E2E) tests cover Kyma Environment Broker (KEB) and SAP BTP, Kyma runtime.
There are three tests:
The following end-to-end (E2E) tests cover Kyma Environment Broker (KEB) and SAP BTP, Kyma runtime:

* `skr-tests` for testing the following operations on different cloud service providers: Kyma provisioning, BTP Manager Secret reconciliation, updating OIDC, updating machine type, and Kyma runtime deprovisioning
* `skr-aws-upgrade-integration` for checking Kyma runtime provisioning, upgrading, and deprovisioning
* `keb-endpoints-test` for checking if `kyma-environment-broker` endpoints require authorization
* `skr-aws-networking` for checking if provisioning a Kyma runtime with custom networking parameters works as expected
* `skr-trial-suspension-dev` for testing the following operations: Kyma provisioning, Kyma suspension, and Kyma runtime deprovisioning
* `skr-aws-binding` for testing the following operations: Kyma provisioning, fetching Kyma Binding, using Kyma Binding, deleting Kyma Binding, and Kyma runtime deprovisioning
* `provisioning-service-aws-stage` for checking if Cloud Management Service Provisioning API works as expected

## E2E SKR Tests

Expand Down Expand Up @@ -61,68 +63,69 @@ In this mode, the test executes the following steps:
make skr SKIP_PROVISIONING=true
```

## E2E SKR AWS Upgrade Integration Test
## KEB Endpoints Test

### Usage

The test executes the following steps:

1. Provisions a Kyma runtime cluster.
2. Runs a Kyma runtime upgrade.
3. Deprovisions the Kyma runtime instance and cleans up the resources.
1. Calls KEB endpoints without an authorization token.
2. Checks whether the call was rejected.

### Test Execution

1. Before you run the test, prepare the `.env` file based on this [`.env.template`](/testing/e2e/skr/skr-aws-upgrade-integration/.env.template).
1. Before you run the test, prepare the `.env` file based on this [`.env.template`](/testing/e2e/skr/keb-endpoints-test/.env.template).
2. To set up the environment variables in your system, run:

```bash
export $(xargs < .env)
```

3. Run the test scenario:

3. Run the test scenario.
```bash
make skr-aws-upgrade-integration
make keb-endpoints
```

## KEB Endpoints Test
## Networking Parameter Tests

### Usage

The test executes the following steps:
1. Calls KEB endpoints without an authorization token.

1. Calls KEB endpoints with invalid networking parameters.
2. Checks whether the call was rejected.
3. Provisions a cluster with custom networking parameters.
4. Deprovisions the cluster.

### Test Execution

1. Before you run the test, prepare the `.env` file based on this [`.env.template`](/testing/e2e/skr/keb-endpoints-test/.env.template).
1. Before you run the test, prepare the `.env` file based on this [`.env.template`](/testing/e2e/skr/skr-networking-test/.env.template).
2. To set up the environment variables in your system, run:

```bash
export $(xargs < .env)
```

3. Run the test scenario.
3. Run the test scenario:

```bash
make keb-endpoints
make skr-networking
```

## Networking Parameter Tests
## E2E SKR Suspension Test

### Usage

The test executes the following steps:

1. Calls KEB endpoints with invalid networking parameters.
2. Checks whether the call was rejected.
3. Provisions a cluster with custom networking parameters.
4. Deprovisions the cluster.
1. Provisions a Kyma runtime cluster.
2. Waits until Trial Cleanup CronJob triggers suspension.
3. Waits until suspension succeeds.
4. Deprovisions the Kyma runtime instance and cleans up the resources.

### Test Execution

1. Before you run the test, prepare the `.env` file based on this [`.env.template`](/testing/e2e/skr/skr-networking-test/.env.template).
1. Before you run the test, prepare the `.env` file based on this [`.env.template`](/testing/e2e/skr/skr-test/.env.template).
2. To set up the environment variables in your system, run:

```bash
Expand All @@ -132,7 +135,7 @@ The test executes the following steps:
3. Run the test scenario:

```bash
make skr-networking-test
make skr-trial-suspension
```

## Binding Tests
Expand All @@ -142,13 +145,16 @@ The test executes the following steps:
The test executes the following steps:

1. Provisions a Kyma runtime cluster.
2. Creates a binding using Kubernetes TokenRequest and saves the returned kubeconfig.
2. Creates a Kyma Binding and saves the returned kubeconfig.
3. Initializes a Kubernetes client with the returned kubeconfig.
4. Tries to fetch a Secret using the binding from Kubernetes TokenRequest.
5. Creates a binding using Gardener and saves the returned kubeconfig.
6. Initializes a Kubernetes client with the returned kubeconfig.
7. Tries to fetch a Secret using the binding from Gardener.
8. Deprovisions the Kyma runtime instance and cleans up the resources.
4. Fetches the `sap-btp-manager` Secret using the Kyma Binding.
5. Fetches the created Kyma Binding.
6. Deletes the created Kyma Binding.
7. Tries to fetch the `sap-btp-manager` Secret using the deleted Kyma Binding.
8. Tries to create a Kyma Binding using invalid parameters.
9. Tests response status codes.
10. Tries to create more than 10 Kyma Bindings.
11. Deprovisions the Kyma runtime instance and cleans up the resources.

### Test Execution

Expand All @@ -162,19 +168,52 @@ The test executes the following steps:
3. Run the test scenario:

```bash
make skr-binding-test
make skr-binding
```

## Provisioning Service Tests

### Usage

The test executes the following steps:

1. Sends a call to Provisioning API to provision a Kyma runtime. The test waits until the environment is created.
2. Creates a Kyma Binding.
3. Fetches the `sap-btp-manager` Secret using the kubeconfig from the created Kyma Binding.
4. Fetches the created Kyma Biding.
5. Deletes the created Kyma Binding.
6. Tries to fetch the `sap-btp-manager` Secret using the invalidated kubeconfig.
7. Tries to fetch the deleted Kyma Binding.
8. Sends a call to Provisioning API to deprovision the Kyma runtime. The test waits until the environment is deleted.

### Test Execution

1. Before you run the test, prepare the `.env` file based on this [`.env.template`](/testing/e2e/skr/provisioning-service-test/.env.template).
2. To set up the environment variables in your system, run:

```bash
export $(xargs < .env)
```

3. Run the test scenario:

```bash
make provisioning-service
```

## CI Pipelines

The tests are run once per day at 01:05 by the given ProwJobs:
The tests are run daily.

* `keb-endpoints-test` - KEB endpoints test
* `skr-aws-integration-dev` - SKR test
* `skr-aws-binding` - Kyma Bindings test
* `skr-aws-networking` - networking parameters test
* `skr-azure-integration-dev` - SKR test
* `skr-azure-lite-integration-dev` - SKR test
* `skr-trial-integration-dev` - SKR test
* `skr-preview-dev` - SKR test
* `skr-free-aws-integration-dev` - SKR test
* `skr-aws-integration-dev` - SKR test
* `skr-aws-upgrade-integration-dev` - SKR AWS upgrade integration test
* `keb-endpoints-test` - KEB endpoints test
* `skr-networking-test` - networking parameters test
* `skr-preview-dev` - SKR test
* `skr-sap-converged-cloud-integration-dev` - SKR test
* `skr-trial-integration-dev` - SKR test
* `skr-trial-suspension-dev` - SKR suspension test
* `provisioning-service-aws-stage` - Provisioning API test
6 changes: 5 additions & 1 deletion testing/e2e/skr/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,8 @@ skr-binding:
curl -fLSs -o /usr/local/bin/kcp https://storage.googleapis.com/kyma-development-artifacts/kcp/master/kcp-linux
chmod +x /usr/local/bin/kcp
npm install
npm run skr-binding-test
npm run skr-binding-test

.PHONY: provisioning-service
provisioning-service:
cd provisioning-service-test/cmd && go test -timeout 130m
9 changes: 9 additions & 0 deletions testing/e2e/skr/provisioning-service-test/.env.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
APP_PROVISIONING_URL=
APP_PROVISIONING_CLIENT_ID=
APP_PROVISIONING_CLIENT_SECRET=
APP_PROVISIONING_UAA_URL=
APP_PROVISIONING_PLAN_NAME=
APP_PROVISIONING_PLAN_ID=
APP_PROVISIONING_USER=
APP_PROVISIONING_INSTANCE_NAME=
APP_PROVISIONING_REGION=
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
package cmd

import (
"context"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func TestProvisioningService(t *testing.T) {
suite := NewProvisioningSuite(t)

suite.logger.Info("Creating a new environment")
environment, err := suite.provisioningClient.CreateEnvironment()
require.NoError(t, err)

err = suite.provisioningClient.AwaitEnvironmentCreated(environment.ID)
assert.NoError(t, err)
suite.logger.Info("Environment created successfully", "environmentID", environment.ID)

suite.logger.Info("Creating a new binding")
createdBinding, err := suite.provisioningClient.CreateBinding(environment.ID)
assert.NoError(t, err)
assert.NotEmpty(t, createdBinding.Credentials.Kubeconfig)

if len(createdBinding.Credentials.Kubeconfig) != 0 {
suite.logger.Info("Creating a new K8s client set")
clientset, err := suite.K8sClientSetForKubeconfig(createdBinding.Credentials.Kubeconfig)
assert.NoError(t, err)

suite.logger.Info("Fetching a secret", "Secret namespace", secretNamespace, "Secret name", secretName)
_, err = clientset.CoreV1().Secrets(secretNamespace).Get(context.TODO(), secretName, metav1.GetOptions{})
assert.NoError(t, err)

suite.logger.Info("Fetching a binding", "Binding ID", createdBinding.ID)
fetchedBinding, err := suite.provisioningClient.GetBinding(environment.ID, createdBinding.ID)
assert.NoError(t, err)
assert.Equal(t, createdBinding.Credentials.Kubeconfig, fetchedBinding.Credentials.Kubeconfig)

suite.logger.Info("Deleting a binding", "Binding ID", createdBinding.ID)
err = suite.provisioningClient.DeleteBinding(environment.ID, createdBinding.ID)
assert.NoError(t, err)

suite.logger.Info("Trying to fetch a secret using invalidated kubeconfig", "Secret namespace", secretNamespace, "Secret name", secretName)
_, err = clientset.CoreV1().Secrets(secretNamespace).Get(context.TODO(), secretName, metav1.GetOptions{})
assert.Error(t, err)

suite.logger.Info("Trying to fetch a deleted binding", "Binding ID", createdBinding.ID)
_, err = suite.provisioningClient.GetBinding(environment.ID, createdBinding.ID)
assert.EqualError(t, err, "unexpected status code 404: body is empty")
}

suite.logger.Info("Deleting the environment", "environmentID", environment.ID)
_, err = suite.provisioningClient.DeleteEnvironment(environment.ID)
require.NoError(t, err)

err = suite.provisioningClient.AwaitEnvironmentDeleted(environment.ID)
assert.NoError(t, err)
suite.logger.Info("Environment deleted successfully", "environmentID", environment.ID)
}
66 changes: 66 additions & 0 deletions testing/e2e/skr/provisioning-service-test/cmd/suite_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package cmd

import (
"context"
"log/slog"
"os"
"testing"

"github.com/kyma-project/kyma-environment-broker/testing/e2e/skr/provisioning-service-test/internal"

"github.com/stretchr/testify/require"
"github.com/vrischmann/envconfig"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
)

const (
secretNamespace = "kyma-system"
secretName = "sap-btp-manager"
)

type Config struct {
Provisioning internal.ProvisioningConfig
}

type ProvisioningSuite struct {
t *testing.T
logger *slog.Logger

provisioningClient *internal.ProvisioningClient
}

func NewProvisioningSuite(t *testing.T) *ProvisioningSuite {
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))

ctx := context.Background()

var cfg Config
err := envconfig.InitWithPrefix(&cfg, "APP")
require.NoError(t, err)

logger.Info("Creating a new provisioning client")
provisioningClient := internal.NewProvisioningClient(cfg.Provisioning, logger, ctx, 60)
err = provisioningClient.GetAccessToken()
require.NoError(t, err)

return &ProvisioningSuite{
t: t,
logger: logger,
provisioningClient: provisioningClient,
}
}

func (p *ProvisioningSuite) K8sClientSetForKubeconfig(kubeconfig string) (kubernetes.Interface, error) {
config, err := clientcmd.RESTConfigFromKubeConfig([]byte(kubeconfig))
if err != nil {
return nil, err
}

clientset, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, err
}

return clientset, nil
}
Loading

0 comments on commit 9244517

Please sign in to comment.