Two document paths overlap with each other #315

ktwbc · 2023-10-22T21:03:12Z

ktwbc
Oct 22, 2023

Trying to use electro on a small ddb table that has two GSIs. Right now the table is blank, and doing an upsert() should insert but I'm getting a failure Invalid UpdateExpression: Two document paths overlap with each other; must remove or rewrite one of these paths; path one: [updated_at], path two: [user_id]" - For more detail on this error reference: https://electrodb.dev/en/reference/errors/#aws-error

index structure:

indexes: {
      default: {
        pk: {
          field: 'pk',
          composite: ['token_id', 'user_id']
        }
      },
      byLastUpdated: {
        index: 'last-updated',
        pk: {
          field: 'pk',
          composite: ['token_id', 'user_id']
        },
        sk: {
          field: 'updated_at',
          composite: ['updated_at']
        }
      },
      byUser: {
        index: 'user_token',
        pk: {
          field: 'user_id',
          composite: ['user_id'],
          template: '${user_id}'
        },
        sk: {
          field: 'updated_at',
          composite: ['updated_at']
        }
      }
    }

table in ddb:

I've purposely deleted token_id and user_id from my other fields, trying to not get them to duplicate but the electrodb db logger does it anyway:

    Reflect.deleteProperty(params, 'token_id');
    Reflect.deleteProperty(params, 'user_id');
    console.log(params);

    await ActiveUsers.upsert({ token_id, user_id, ...params }).go();

logger

{
    "type": "query",
    "method": "upsert",
    "params": {
        "TableName": "dev-active-users2",
        "UpdateExpression": "SET #__edb_e__ = :__edb_e___u0, #__edb_v__ = :__edb_v___u0, #token_id = :token_id_u0, #browser = :browser_u0, #user_id = :user_id_u0, #name = :name_u0, #course = :course_u0, #language = :language_u0, #initial_signup = :initial_signup_u0, #location = :location_u0, #on_page = :on_page_u0, #on_page_id = :on_page_id_u0, #status = :status_u0, #used = :used_u0, #to_expire = if_not_exists(#to_expire, :to_expire_u0), #created_at = if_not_exists(#created_at, :created_at_u0), #updated_at = :updated_at_u0, #updated_at = :updated_at_u1, #user_id = :user_id_u1",
        "ExpressionAttributeNames": {
            "#__edb_e__": "__edb_e__",
            "#__edb_v__": "__edb_v__",
            "#token_id": "token_id",
            "#browser": "browser",
            "#user_id": "user_id",
            "#name": "name",
            "#course": "course",
            "#language": "language",
            "#initial_signup": "initial_signup",
            "#location": "location",
            "#on_page": "on_page",
            "#on_page_id": "on_page_id",
            "#status": "status",
            "#used": "used",
            "#to_expire": "to_expire",
            "#created_at": "created_at",
            "#updated_at": "updated_at"
        },
        "ExpressionAttributeValues": {
            ":__edb_e___u0": "ActiveUsers",
            ":__edb_v___u0": "01",
            ":token_id_u0": "01hdcfrkjnhac1wcaq090bwb03",
            ":browser_u0": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
            ":user_id_u0": "01hdcfqw6bp2ywzs4skd3qjhgq",
            ":name_u0": "[email protected]",
            ":course_u0": "testing",
            ":language_u0": "en",
            ":initial_signup_u0": 1698004988.109,
            ":location_u0": "New York. NT",
            ":on_page_u0": "introduction",
            ":on_page_id_u0": "introduction",
            ":status_u0": "purchased",
            ":used_u0": "card",
            ":to_expire_u0": 1698055199.999,
            ":created_at_u0": 1698005028124,
            ":updated_at_u0": 1698005028124,
            ":updated_at_u1": 1698005028124,
            ":user_id_u1": "01hdcfqw6bp2ywzs4skd3qjhgq"
        },
        "Key": {
            "pk": "$database$activeusers_01#token_id_01hdcfrkjnhac1wcaq090bwb03#user_id_01hdcfqw6bp2ywzs4skd3qjhgq"
        }
    },

I can clearly see it duplicating user_id as both user_id_u0 and user_id_u1, as well as updated_at, I just can't figure out why or what to do to fix it.

tywalch · 2023-10-22T23:14:29Z

tywalch
Oct 22, 2023
Maintainer

Hey @ktwbc 👋

This definitely isn't working as expected, thank you for opening this ticket.

Are you using this for new data? I wouldn't think you're using this with existing data given that you're using the default key for the pk field?

5 replies

ktwbc Oct 22, 2023
Author

yes it's actually just a table of daily activity, so every morning it starts blank. It's used for a dashboard panel. and TTL is set to wipe all the entries overnight.

tywalch Oct 22, 2023
Maintainer

The edge-case you're seeing is because your model uses an attribute as an index field (user_id) and the same attribute as a composite in another index (pk). This is leans into two escape hatches Electro has for those use with pre-existing tables, which is pretty atypical, but should still be addressed.

In the interest of getting you something working, is it possible for you to update your attribute names or table schema? I can provide an example. I understand changing one or either are not always easy to do.

ktwbc Oct 22, 2023
Author

this is a new table i'm testing with, changes are easy. Yes, examples would be helpful. Thanks!

tywalch Oct 22, 2023
Maintainer

New partition and sort key names; keeps your original index names, but it is most common for those to also be made generic:

https://electrodb.fun/?#code/PQKgBAsg9gJgpgGzARwK5wE4Es4GcA0YuccYGeqCALgUQBYCG5YA7llXWAGbZwB2MXGBDAAUKKwBbAA5QMVMAG8wAUT5V2AT0IBlTADcsAY1IBfbhiiSwAIkRwjVSzABGNgNzijUPrgVUGFwRSAF5bTShUDAB9AKC4aL4GSTgPcVAwABUGXABrITUNKk1hMW9ffxz8sDC+OBZVdS0AClEwJTb2sElYRAAuDq6u-iLNAZsAvNwbfE6h-UxcLB9xgEYZua7iDEMTccnchmlpG03TWaGGKicsF1QqPAHFTfaqKFz+aKwYJ5eu4ukcHGfmwfAA5qchu1zn9UNsvj9BlDXppAcCbuDIVCYcjUNIYFc4DBoldfsiUWjbHxUJIXJgNsjTGcLl0sAI4AAPR5IobwLgMShUMnk6S5YXk7g4BCIgDkoplLIl3hkUCWDwGAG0ZW8PnwEQqwDK4Zh9QBdRXYv44qEuTQAGRyVAAqvjCYjnhK2fAOQMZQhHQBaPEEh4wBV-dqi8XkrhS2Xyi3I5WyNVAsBanWfb4Go3w7PmiNga3IvLR5GxxCyvLhiXtZOq9hprXBt0kqgygu1sBGR2+6m0zA18lMxmJsC2p3bMus9k+w3GmKZvhD5FRnkxuO+hexd6fBOFutWFONzW5k35sdDB4yf3qw0AEkU2++phX2MvRDF6-Lm8N1Y-h4qqmp4tqGbYdgB3a9oa-Z0hgb5DCOlqIZ01rKHEwRFqIACUniiAcuAAHR4tsVDNIoS4IuMgRGDMYCgUSbYDAAIoShF8FALDNNhhDPoiNg0TYpjYYRYJQNx7hAA

tywalch Oct 22, 2023
Maintainer

This will work with your existing dynamodb table but won't error because the attribute names do not match the same as your index names (see my reasoning in my previous message):

https://electrodb.fun/?#code/PQKgBAsg9gJgpgGzARwK5wE4Es4GcA0YuccYGeqCALgUQBYCG5YA7llXWAGbZwB2MXGBDAAUKKwBbAA5QMVMAG8wAUT5V2AT0IBlTADcsAY1IBfbhiiSwAIkRwjVSzABGNgNzijUPrgVUGFwRSAF5bTShUDAB9AKC4aL4GSTgPcVAwABUGXABrITUNKk1hMW9ffxz8sDC+OBZVdS0AClEwJTb2sElYRAAuDq6u-iLNAZsAvNwbfE6h-UxcLB9xgEYZua7iDEMTccnchmlpG03TWaGGKicsF1QqPAHFTfaqKFz+AEkYJ5eu4ukcHGfmwfAA5hshkMuDgED9bG8PnxvpCoec-qhtt9flD-ppAcCbuDUbiYYh4TZMZgURc0bShqhpDArnAYABBKg43FgAFA2x8VCSFyYElQslw8aM5kPdlUUXtUxnelgLACOAAD0egyh8C4DEonO1uOkuS53O4sPhAHITVblbjvDIoEsHgMANpWxFfGB2sBWqkYb5WgC6fwVf3RuJcmgAMjkqABVJks+HPc2q+DqgZWhDxgC0UpTdrDYBNZu54uttvtUMdshdfI9XuRPsI-qxPtD5sj3Ly5dJluzhZl0SuxfN7TrzvYjf9yZlHJDNaGRnj2YFQswVpLitxPaG0YT237XQzGqH21i7344+5ZaNFcHfoD0SwrZLk6s9Zn7vb1M7y5gPuUJ9g+A7kkO86sqOVC3uaU4Nr+w6souwaAZOa5+huwoYNu3YRmcnQ9socTBEBogAJSeKIBy4AAdIy2xUM0ijNtitiBEYMxgMhsoDAAIiydF8FALDNBRhABuxNicTYpgUXRYJQOJ7hAA

ktwbc · 2023-10-25T00:00:31Z

ktwbc
Oct 25, 2023
Author

well it got past the initial issue but attempting to use the byUser in a query isn't working.

  let results = await ActiveUsers.query.byUser({ user_id }).go();

debug:

{
    "type": "query",
    "method": "query",
    "params": {
        "KeyConditionExpression": "#pk = :pk",
        "TableName": "dev-active-users2",
        "ExpressionAttributeNames": {
            "#pk": "user_token_pk"
        },
        "ExpressionAttributeValues": {
            ":pk": "01hdhztva2q6g1acd26s4f2241"
        },
        "IndexName": "user_token"
    },
    "config": {
        "includeKeys": false,
        "originalErr": false,
        "raw": false,
        "params": {},
        "page": {},
        "lastEvaluatedKeyRaw": false,
        "pager": "named",
        "unprocessed": "item",
        "response": "default",
        "cursor": null,
        "data": "attributes",
        "ignoreOwnership": false,
        "_providedIgnoreOwnership": false,
        "_isPagination": true,
        "_isCollectionQuery": false,
        "pages": 1,
        "listeners": [],
        "preserveBatchOrder": false,
        "attributes": [],
        "terminalOperation": "go",
        "formatCursor": {},
        "hydrate": false,
        "_includeOnResponseItem": {}
    }
}

error:

{
    "type": "results",
    "method": "query",
    "config": {
        "includeKeys": false,
        "originalErr": false,
        "raw": false,
        "params": {},
        "page": {},
        "lastEvaluatedKeyRaw": false,
        "pager": "named",
        "unprocessed": "item",
        "response": "default",
        "cursor": null,
        "data": "attributes",
        "ignoreOwnership": false,
        "_providedIgnoreOwnership": false,
        "_isPagination": true,
        "_isCollectionQuery": false,
        "pages": 1,
        "listeners": [],
        "preserveBatchOrder": false,
        "attributes": [],
        "terminalOperation": "go",
        "formatCursor": {},
        "hydrate": false,
        "_includeOnResponseItem": {}
    },
    "success": false,
    "results": {
        "name": "ValidationException",
        "$fault": "client",
        "$metadata": {
            "httpStatusCode": 400,
            "requestId": "ACGU1VO3IV96009F970LOG2FOBVV4KQNSO5AEMVJF66Q9ASUAAJG",
            "attempts": 1,
            "totalRetryDelay": 0
        },
        "__type": "com.amazon.coral.validate#ValidationException",
        "message": "Query condition missed key schema element: user_id"
    }
}

1 reply

tywalch Oct 25, 2023
Maintainer

Can you post the schema you ended up with? If it was the first one (which it looks like it might be), that example requires you to change your table definition to use new key field names. Sorry for the confusion. The new definition for that entity would look something like this:

{
  "TableName": "your-table-name",
  "KeySchema": [
    {
      "AttributeName": "pk",
      "KeyType": "HASH"
    }
  ],
  "AttributeDefinitions": [
    {
      "AttributeName": "pk",
      "AttributeType": "S"
    },
    {
      "AttributeName": "sk",
      "AttributeType": "N"
    },
    {
      "AttributeName": "user_token_pk",
      "AttributeType": "S"
    }
  ],
  "GlobalSecondaryIndexes": [
    {
      "IndexName": "last-updated",
      "KeySchema": [
        {
          "AttributeName": "pk",
          "KeyType": "HASH"
        },
        {
          "AttributeName": "sk",
          "KeyType": "RANGE"
        }
      ],
      "Projection": {
        "ProjectionType": "ALL"
      }
    },
    {
      "IndexName": "user_token",
      "KeySchema": [
        {
          "AttributeName": "user_token_pk",
          "KeyType": "HASH"
        },
        {
          "AttributeName": "sk",
          "KeyType": "RANGE"
        }
      ],
      "Projection": {
        "ProjectionType": "ALL"
      }
    }
  ],
  "BillingMode": "PAY_PER_REQUEST"
}

ktwbc · 2023-10-28T21:57:23Z

ktwbc
Oct 28, 2023
Author

I hadn't updated the actual db with the new user_token_pk name, so I did that and did get it working.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Two document paths overlap with each other #315

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Two document paths overlap with each other #315

ktwbc Oct 22, 2023

Replies: 3 comments · 6 replies

tywalch Oct 22, 2023 Maintainer

ktwbc Oct 22, 2023 Author

tywalch Oct 22, 2023 Maintainer

ktwbc Oct 22, 2023 Author

tywalch Oct 22, 2023 Maintainer

tywalch Oct 22, 2023 Maintainer

ktwbc Oct 25, 2023 Author

tywalch Oct 25, 2023 Maintainer

ktwbc Oct 28, 2023 Author

ktwbc
Oct 22, 2023

Replies: 3 comments 6 replies

tywalch
Oct 22, 2023
Maintainer

ktwbc Oct 22, 2023
Author

tywalch Oct 22, 2023
Maintainer

ktwbc Oct 22, 2023
Author

tywalch Oct 22, 2023
Maintainer

tywalch Oct 22, 2023
Maintainer

ktwbc
Oct 25, 2023
Author

tywalch Oct 25, 2023
Maintainer

ktwbc
Oct 28, 2023
Author