Dynamo Stream Handler - Semantics Help

[barqco]

Hi,
Love the library, I have a quick question on how to semantically handle dynamodb streams. I know about the parse section of the documentation

The sample code shows the following:

import DynamoDB from "aws-sdk/clients/dynamodb";
import { DynamoDBStreamEvent } from 'aws-lambda';
import { MyEntity } from "./my-entity";

function main(event: DynamoDBStreamEvent) {
  const parsed = event.Records
    .map(record => DynamoDB.Converter.unmarshall(record.dynamodb.NewImage))
    .map(item => MyEntity.parse({ Item: item }))

  console.log(parsed)
}

However, I want to create a generic dynamostream handler where I am not sure yet what the Entity type of the event record is. So I can't parse since I don't know what entity it is yet. There is a private field called __edb_e__, that does show up in the new image. I am wondering if it is advisable to rely on this, or if there are plans in the future to expose something that will "get" the entity type from a potential JSON object.

My Code looks something like this:

export const dynamoStreamHandler = async (req: DynamoDBStreamEvent) => {
  for (const record of req.Records) {
    if (!record.dynamodb?.NewImage) {
      continue;
    }
    const potentialDomainModelItem = unmarshall(
      record.dynamodb.NewImage as Record<string, AttributeValue>
    );

    console.log("Potential domain event: ", potentialDomainModelItem);
    return "ok";
  }
};

I'm hoping to use Zod and create a switch statement

export const DomainModelNames = z.enum(ELECTRO_DB_DOMAIN_MODEL_NAMES);
export type DomainModelName = z.infer<typeof DomainModelNames>;

  const { __edb_e__ : potentialEntityType } = insertedDynamoItem;
  switch (potentialEntityType) {
    case "User":
    console.log('user logic')
      break;

    default:
      console.log(
        "No match implemented yet for this entity type: ",
        potentialEntityType
      );
      break;
  }

let me know what you think!

[zirkelc]

This is one thing I also have to figure out before switching over to a single table design. I think it would be good to have a static function on the Entity class like Entity.fromItem/Image/Record() that either returns the correct model.entity type or the instantiated Entity object given multiple Schemas.

Something like this maybe:

const Task = new Entity({
  model: {
    entity: "task",
    version: "1",
    service: "taskapp",
  },
  attributes: {
    // ...
  },
  indexes: {
    // ...
  },
}, { table, client });

const Book = new Entity({
  model: {
    entity: 'book',
    version: '1',
    service: 'store'
  },
  attributes: {
    // ...
  },
  indexes: {
    // ...
  }
}, { client, table });

function main(event: DynamoDBStreamEvent) {
  const parsed = event.Records
    .map(record => DynamoDB.Converter.unmarshall(record.dynamodb.NewImage))
    .map(item =>
      Entity.fromImage( //> returns an instance of Task or Book
        { Item: item },
        [Task, Book] // pass all your entities here
      )
    )

  console.log(parsed)
}

[zirkelc]

I just looked at a Service example and its constructor supports passing multiple entity instances. Maybe it could be extended to return the right entity instance for a given DynamoDB record image:

const TaskApp = new Service({
	employee: Employee, // available at TaskApp.entities.personnel
	task: Task,         // available at TaskApp.entities.directives
}, {table, client});

function main(event: DynamoDBStreamEvent) {
  const parsed = event.Records
    .map(record => DynamoDB.Converter.unmarshall(record.dynamodb.NewImage))
    .map(item => TaskApp.fromItem({ Item: item }));

  console.log(parsed)
}

[tywalch]

Hi @barqco 👋

The __edb_e__ and __edb_v__ fields can definitely be used to identify an item. In fact, they can be customized via the setIdentifier() method on your Entity. It is named the way it is to limit its potential for conflicting with user defined attributes.

[tywalch]

I am working right now actually on methods to help translate:

• items to keys
• items to cursors
• cursors to keys
• cursors to composite attributes
• keys to composite attributes
• keys to cursors

@zirkelc a parse() on Service is a great idea 👍

[zirkelc]

Do you mind if I submit a PR for the function Service.parse()? I have an idea how it could be implemented, but I wanted to check with you first, in case you already implement such a function at the moment.