DisposableRequest request returning partial results

[arunkpskpm]

What happened?

I'm trying to use DisposableRequest to gather some data, and then I plan to iterate through it using go-templating, applying some logic based on the data. However, the response includes only partial results (the original output has hundreds of items in an array/list). Is this behavior expected? Could there be any trimming occurring at the provider level?

Please find the manifest which used retriving the account groups from Prisma Cloud.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xawsaccount1s.composite.prismacloud.io   # pluralname + provider + groupname
spec:
  compositeTypeRef:
    apiVersion: composite.prismacloud.io/v1alpha1  # groupname/versionname
    kind: XAwsAccount1
  mode: Pipeline
  pipeline:
    - step: environmentConfigs
      functionRef:
        name: function-environment-configs
      input:
        apiVersion: environmentconfigs.fn.crossplane.io/v1beta1
        kind: Input
        spec:
          environmentConfigs:
          - type: Reference
            ref:
              name: profile-config
    - step: aws-account-onboarding
      functionRef:
        name: function-go-templating
      input:
        apiVersion: gotemplating.fn.crossplane.io/v1beta1
        kind: GoTemplate
        source: Inline
        inline:
          template: |
            {{ $resource := .observed.composite.resource }}
            {{- $accountId := $resource.spec.accountID }}
            {{- $accountName := $resource.spec.accountName }}            
            {{- $env := (index $.context "apiextensions.crossplane.io/environment") }}
            ---
            apiVersion: http.crossplane.io/v1alpha2
            kind: DisposableRequest
            metadata:
              name: get-group-list
              annotations:
                gotemplating.fn.crossplane.io/composition-resource-name: get-group-list             
            spec:
              deletionPolicy: Orphan
              forProvider:
                body: ''
                expectedResponse: .response.statusCode == 200
                headers:
                  Content-Type:
                    - 'application/json; charset=UTF-8'
                  x-redlock-auth:
                    - "{{`{{`}} response-token:crossplane-system:token {{`}}`}}"
                insecureSkipTLSVerify: true
                method: GET
                shouldLoopInfinitely: true
                url: https://api.eu.prismacloud.io/cloud/group/name
                waitTimeout: 5m
              providerConfigRef:
                name: http-conf
            ---
            apiVersion: composite.prismacloud.io/v1alpha1
            kind: XAwsAccount1
            status:
              ResourcesStatus: |
                {{ $.observed.resources |toJson  }}
    - step: automatically-detect-ready-composed-resources
      functionRef:
        name: function-auto-ready

What environment did it happen in?

Crossplane version: v1.17.2

Include at least the version or commit of Crossplane you were running. Consider
also including your:

• Cloud provider or hardware configuration
• Kubernetes version (use kubectl version)

iehje9m@AMC ~ % kubectl version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.31.0

• Kubernetes distribution (e.g. Tectonic, GKE, OpenShift) - Kind
• OS (e.g. from /etc/os-release) - MacOS
• Kernel (e.g. uname -a)

[arielsepton]

Hi @arunkpskpm,

The DisposableRequest resource does not perform any trimming of any data.

Out of curiosity, I tested the behavior by mimicking the API you used, as documented here, with the following API:

from flask import Flask, Response
import json
import random
import uuid

app = Flask(__name__)

def generate_items(count=1000):
    return [
        {
            "autoCreated": random.choice([True, False]),
            "id": str(uuid.uuid4()),  # Generate a random UUID for the ID
            "name": f"Item {i}"
        }
        for i in range(count)
    ]

@app.route('/v1/test', methods=['GET'])
def test():
    items = generate_items(1000)  # Generate 1000 items
    return Response(json.dumps(items), status=200, mimetype='application/json')

if __name__ == '__main__':
    app.run(debug=True)

And created the following DisposableRequest:

apiVersion: http.crossplane.io/v1alpha2
kind: DisposableRequest
metadata:
  name: test
spec:
  deletionPolicy: Orphan
  forProvider:
    url:  http://127.0.0.1:5000/v1/test
    method: GET
    insecureSkipTLSVerify: true
  providerConfigRef:
    name: http-conf

As expected, all 1000 items appeared in the DisposableRequest resource's status.

Have you tried creating the DisposableRequest individually (not as part of a composition)? It could help us determine whether the issue lies with the request itself or with the processing done by the Go templating.

Looking forward to hearing your thoughts on this!

[arunkpskpm]

Hi @arielsepton ,

I had another look at the issue, and it seems to be working just fine now. I'm not sure what went wrong during the earlier tests, but everything is functioning as it should. DisposableRequest is returning all the objects as expected. Sorry for any trouble this may have caused. I'll go ahead and close the request.

Thanks,
Arun