[BUG] Non published references are returned by DataSync when using include([Array of refs]) function and the UIDs are mixed up then.

[idmedia-kaiser]

Hi, we are facing an issue with DataSync when using the include() function to include only specific references.
https://contentstack.github.io/datasync-filesystem-sdk/global.html#include

We have a form content type with a reference field for fields
https://app.contentstack.com/#!/stack/blt81b4b787a1850697/content-type/form/de-de/entry/blt0e8af3662cf665b0/edit?branch=main

Some of those references are not published. If we check in the DEV Tool widget we correctly don't get this fields back that are not published.

https://cdn.contentstack.io/v3/content_types/form/entries/blt0e8af3662cf665b0?api_key=blt81b4b787a1850697&access_token=&environment=development&locale=de-de

{
...
  "fields": [
    {
      "uid": "blte52259e1308823f8",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "bltf45bb3e8d8c04dd1",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "bltb673a18f783fa5b6",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "blt9788f975762ad3e0",
      "_content_type_uid": "input_field"
    },
    {
      "uid": "blt4984c9008d23711a",
      "_content_type_uid": "input_field"
    }
  ]
}

If we use the include(["fields"]) function then the not published entries get mixed up with entries that are actually published and we get duplicated content

[{
    "content_type_uid": "form",
    "locale": "de-at",
    "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1"
                },
                {
                    "uid": "bltb673a18f783fa5b6"  // -> Not published should be "blta8fed94360b45377" or actually not present at all
                },
                {
                    "uid": "blt9788f975762ad3e0"  // -> Not published should be "blta22b07fa5fc5fce9" or actually not present at all
                },
                {
                    "uid": "bltb673a18f783fa5b6"
                },
                {
                    "uid": "blt9788f975762ad3e0"
                },
                {
                    "uid": "blt4984c9008d23711a"
                }
            ]
        }
    ]
}]

Composed query

let blogQuery = Stack.contentType("form");
blogQuery = blogQuery.equalTo("uid", "blt0e8af3662cf665b0");
blogQuery = blogQuery.include(["fields"]);
blogQuery.find().then((result: {}) => {
	res.json(result);
})

This works correctly if we use replace
blogQuery = blogQuery.include(["fields"]);
with
blogQuery = blogQuery.includeReferences();

but we don't want all references but only some of them.

[idmedia-kaiser]

@harshithad0703 As discussed I opened this as a new issue with the correct description of the error.
@aman19K @netrajpatel @cs-raj @shafeeqd959 FYI

[aman19K]

Hi @idmedia-kaiser ,
We’ve tested the include() function in DataSync, and it’s working as expected. To include the content of referenced entries in your response, you need to use the include[] parameter and provide the UID of the reference field as the value. It looks like you're currently using fields instead of UIDs, which is why it's not working correctly. Please try using UIDs and let us know how it goes?

For example:-
Stack.contentType('example').entries().include(['authors','categories']).find()
If we have three CTs (authors, categories, and blogs) referenced in example, and we just want to retrieve the authors and categories, then we can use include() function.

FYI
DataSync:-
https://github.com/contentstack/datasync-filesystem-sdk/blob/master/src/stack.ts#L800
https://github.com/contentstack/datasync-filesystem-sdk/blob/master/src/stack.ts#L1090
API:- https://www.contentstack.com/docs/developers/apis/content-delivery-api#include-reference

CC: @netrajpatel @harshithad0703

[idmedia-kaiser]

Hi @aman19K, as I wrote also in the ticket.If you check the demo
API:- https://www.contentstack.com/docs/developers/apis/content-delivery-api#include-reference
categories is the name of the reference field and the content type uid inside is category. And we do the same.
fields is the reference field that holds the content type uid input_field.
Therefore we pass fields and not input_field to the include[] same as in the demo we pass categories and not category

And the issue is not the its not working, the issue is with unpublished entries getting returned partially and mixing up with existing entries

"entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "title": "Straße (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1", <---- unpublished entry which actuallay has UID blt983d6cdb2ec347cd see below
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "title": "Hausnummer (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "title": "Adresszusatz",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "title": "PLZ (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "title": "Ort (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                }
            ]
        }

same resposne without the include(["fields"]

 "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt983d6cdb2ec347cd", <-- this is the unpublished entry
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "_content_type_uid": "input_field"
                }
            ]
        }
    ]

expected is this

"entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "title": "Straße (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "title": "Hausnummer (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "title": "Adresszusatz",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "title": "PLZ (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "title": "Ort (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                }
            ]
        }

[aman19K]

Yes @idmedia-kaiser , We are checking the demo. As soon as we can reproduce the problem, we'll let you know.
CC: @harshithad0703

[aman19K]

Hi @idmedia-kaiser ,We can reproduce the issue on my end, and it is functioning correctly now with this fix. Would you please try that and let me know if it functions properly on your end as well?

[idmedia-kaiser]

Hi @aman19K, I just applied this fix locally, and it's not really working, its just removes the next to last element.
Before the fix

{
    "content_type_uid": "form",
    "locale": "de-de",
    "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "title": "Straße (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "title": "Hausnummer (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "title": "Adresszusatz",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "title": "PLZ (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "title": "Ort (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                }
            ]
        }
    ]
}

After the fix

{
    "content_type_uid": "form",
    "locale": "de-de",
    "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "title": "Straße (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "title": "Hausnummer (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "title": "Adresszusatz",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "title": "Ort (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                }
            ]
        }
    ]
}

[aman19K]

Hi @idmedia-kaiser , We’re re-verifying the fix and will let you know if we able to reproduce it or need any help from your side.
CC: @netrajpatel @harshithad0703

[harshithad0703]

Hi @idmedia-kaiser
We were able to reproduce the issue and here is the fix, please do try and let us know, if this is functioning as expected.

cc: @aman19K @netrajpatel

[idmedia-kaiser]

Hi @harshithad0703 @aman19K @netrajpatel , that looks good! blt983d6cdb2ec347cd is no longer present and bltf45bb3e8d8c04dd1 is there once and the correct field.

After this fix

{
    "content_type_uid": "form",
    "locale": "de-de",
    "entries": [
        {
            "fields": [
                {
                    "uid": "blte52259e1308823f8",
                    "title": "Straße (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltf45bb3e8d8c04dd1",
                    "title": "Hausnummer (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "bltb673a18f783fa5b6",
                    "title": "Adresszusatz",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt9788f975762ad3e0",
                    "title": "PLZ (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                },
                {
                    "uid": "blt4984c9008d23711a",
                    "title": "Ort (Pflichtfeld)",
                    "_content_type_uid": "input_field"
                }
            ]
        }
    ]
}

Please let us know when this will be released :)

[harshithad0703]

Thank you for confirming! It’s great to hear that the issue has been fixed and the correct field is now present

This fix is scheduled to be released by 20th January.
cc: @cs-raj