Optimize Shopify Metafields Sync Performance

[sgandhi1311]

Problem

The current implementation of metafields sync is inefficient because:

1. It fetches all parent IDs (products, orders, etc.) first
2. Makes additional API calls for each parent ID to fetch its metafields
3. Many API calls are wasted on parents that have no metafields
4. Results in high sync duration and excessive API usage

Solution

Implemented a hybrid approach that:

1. Directly queries metafields list without fetching parent IDs first
2. Only falls back to individual parent ID fetching when metafield count exceeds the max page limit (default set to 175)
3. Uses GraphQL pagination for efficient data retrieval
4. Maintains existing functionality for edge cases

Implementation Details

• Added new GraphQL queries to fetch metafields directly
• Implemented pagination handling at both resource and metafield levels
• Added fallback mechanism for parents with > max page limit metafields
• Maintained backwards compatibility with existing bookmark logic

Additional Changes

1. Add retry for the interruptible sync error from the server side.
2. Include the new field (= inventoryItem) in the product variants schema.

Benefits

• Significantly reduced API calls
• Faster sync completion time
• Lower resource utilization

Testing

• Verified data consistency with old approach
• Confirmed proper handling of bookmarks
• Performed PR-alpha on the customer connection

Future Considerations

• Monitor performance on very large stores
• Consider implementing parallel processing for large datasets or using bulk APIs
• Add metrics collection for optimization tracking

Related Issues

Closes #[TDL-26998]

AI generated code

https://internal.qlik.dev/general/ways-of-working/code-reviews/#guidelines-for-ai-generated-code

• this PR has been written with the help of GitHub Copilot or another generative AI tool

[somethingmorerelevant]

the base class implementation has more explicit error handling, it would be a good idea to carry it over to the modified implementation of the class.

ref: gql_base.py


    @shopify_error_handling
    def call_api(self, query_params):
        """
        - Modifies the default call api implementation to support GraphQL
        - Returns response Object dict
        """
        try:
            query = self.get_query()
            LOGGER.info("Fetching %s %s", self.name, query_params)
            response = shopify.GraphQL().execute(query=query, variables=query_params)
            response = json.loads(response)
            if "errors" in response.keys():
                raise ShopifyAPIError(response['errors'])
            data = response.get("data", {}).get(self.data_key, {})
            return data
        except ShopifyAPIError as gql_error:
            LOGGER.error("GraphQL Error %s", gql_error)
            raise ShopifyAPIError("An error occurred with the GraphQL API.") from gql_error
        except Exception as exception:
            LOGGER.error("Unexpected error occurred.",)
            raise exception

[sgandhi1311]

I added the exception handling over here as well.

[somethingmorerelevant]

this is not the right way to use a static method with a self reference
please change the method implementation or update the function call.

Metafields.get_resource_type_query(gql_resource)()

[sgandhi1311]

I have updated the function to the instance method.

[rdeshmukh15]

This is correct but the function self.get_bookmark_by_name(f"{self.name}_{resource_type}") seems to be called twice — once to set current_bookmark_value and once more when updating current_bookmarks.
Can't we write like this, instead?
current_bookmarks[resource_type] = current_bookmark_value

[sgandhi1311]

makes sense, made the change

[rdeshmukh15]

Here,

• You're checking if resource_type exists in current_bookmarks. If not, you fetch the initial bookmark value from self.get_bookmark_by_name(f"{self.name}_{resource_type}") and store it in current_bookmarks[resource_type]
• You then check if replication_value >= current_bookmarks[resource_type]. If this condition is True, you update the bookmark for this resource_type by taking the maximum value between the current replication_value and the existing current_bookmarks[resource_type].
  Essentially, you're checking and possibly updating current_bookmarks[resource_type] twice.
  You can combine the check and update process in a single step by using the dict.get() method and reducing the redundancy:

if replication_value >= current_bookmarks.get(resource_type, current_bookmark_value):
    current_bookmarks[resource_type] = max(replication_value, current_bookmarks.get(resource_type, replication_value))

[sgandhi1311]

Sounds logical.
I optimized it further.

if replication_value >= current_bookmarks.get(resource_type, current_bookmark_value):
    current_bookmarks[resource_type] = replication_value

Let me know if you think something else.

[rdeshmukh15]

It is not impacting anything at all, still these 2 empty lines should be there before class def.

[somethingmorerelevant]

Suggested change

	* Include the new field (= inventoryItem) in the product variants schema.
	* Include new field inventoryItem in the product_variants stream.

If this was not intended