at_type property value specification for collections in app metadata

[keighrim]

Because

Currently in v1.1.1 (3a9bfc8), input and output annotation specs in app metadata only support simple atomic types for specifying an additional property and their fixed value. One can add multiple type specifications with different property values to represent a range of possible values for the property ("one-of" spec, see #77), as for the output section of the app metadata always implies "any-of" the specified types.

clams-python/clams/appmetadata/__init__.py

Line 15 in 3a9bfc8

real_valued_primitives = Union[int, float, bool, str]

clams-python/clams/appmetadata/__init__.py

Lines 72 to 75 in 3a9bfc8

	properties: Dict[str, real_valued_primitives] = pydantic.Field(
	{},
	description="(optional) Specification for type properties, if any."
	)

However, aside from the fact that this hacky representation for "one-of" types is hardly agreeable to be particularly readable (as in https://apps.clams.ai/swt-detection/v3.0/), currently metadata schema doesn't allow a collection of values to an output type property that presents either "all-of" specification or more complex data types. (see clamsproject/mmif-python#252)

Done when

Based on the conclusion from clamsproject/mmif-python#252, the app metadata schema is updated and relevant helper methods are also updated.

Additional context

No response

[keighrim]

Until now, the most frequent use case of the "one-of" output spec was to record possible classification results. That is, we used "one-of" representation to represent the labelset information in the app metadata. However, as discussed in clamsproject/mmif#218 , we are likely to add a dedicated labelset prop to all Annotation subtypes, and I'd assume that would practically solve the problems with "one-of" representation, at least.

[keighrim]

tl;dr

In app metadata,

1. a dev can't say that this app wants a X type with y property without specifying the value of y.
   • proposal: "*" string to have a special wild card meaning (any value)
2. a dev can say that this app outputs a X type with y property of which value is one of [a, b, c], but it's utterly ugly.
   • proposal: none at the time.

Open to any other suggestions!

---

to specify a property with "a" value

This is the only scenario that the current app metadata I/O spec is designed for.

on input side

In the simplest case, suppose an app wants a TimeFrame with label="speech" property (and a AudioDocument without any further property specification). The app metadata can be written as

"input": [
    { "@type": "AudioDocument", "required": true },
    { "@type": "TimeFrame", "required": true, "properties":  { "label": "speech" }}
]

on output side

As the output, the same app returns an MMIF with TextDocument and Alignment for each speech-labeled time frame.

"output": [
    { "@type": "Alignment"},  # no `required`: all items in this output list always are "optional" (app can always fail to generate any annotations)
    { "@type": "TextDocument", "properties": {"@lang": "en"}}  
]

(Yes, this app is an English ASR app, but to simplify the example, it doesn't generate any "linguistic" unit annotations)

to specify a property with "any" value from a set of possible values

This scenario is the reason this issue is originally opened. For instance, the current SWT app (v4) outputs

...
"annotations": [
  { "@type": "TimeFrame",
    "properties": {
      "label": "chyron",
      "classification": { "chyron": 0.8210625767707824 },
      "targets": [ "tp_1", "tp_2", "tp_3", "tp_4", "tp_5" ],
      "representatives": [ "tp_2" ],
      "id": "tf_1"
    }
  },
  { "@type": "TimePoint",
    "properties": {
      "timePoint": 2000,
      "label": "I",
      "classification": {
        "bars": 3.4400006825308083e-06,
        "slate": 1.3836956497925712e-05,
        "chyron": 0.966004173271358,
        "credits": 0.0013315067626535892,
        "NEG": 0.03264704300880794
      },
      "id": "tp_1"
    }
  },
  ...  # and more
]

on output side

So currently the output is specified in (https://apps.clams.ai/swt-detection/v4.0/metadata.json)

  "output": [{
    "@type": "TimeFrame",
    "properties": {
      "timeUnit": "milliseconds",
      "labelset": ["bars", "slate", "chyron", "credits", "NEG"]
    }
  }, {
    "@type": "TimePoint",
    "properties": {
      "timeUnit": "milliseconds",
      "labelset": ["bars", "slate", "chyron", "credits", "NEG"]
    }
  }],

which doesn't say anything about label, classification, or representatives properties in the actual output annotations.
But that should be just fine, as those properties are "variables" and the values are different every time.

on input side

Now, let's take a simplified OCR app that's developed to target the SWT app. Namely, this OCR app wants MMIF input with TimeFrames with label="slate" or label="chyron" orlabel="credits". Based on "one-of"/disjunctive interpretation of lists in input list (#77), we can use something like this (@snewman-aa please confirm if this is the case now for the OCR apps you recently worked)

  "input": [
    [  # nested list has "disjunctive" interpretation
      { "@type": "TimeFrame",
        "properties": { "label": "slate"}
      },
      { "@type": "TimeFrame",
        "properties": { "label": "chyron"}
      },
      { "@type": "TimeFrame",
        "properties": { "label": "credit"}
      }
    ]
  ],

Now, we can infer just form the TimeFrame::labelset spec of the app metadata of the SWT that it generates MMIF that's suitable for this OCR app, it might not be so obvious to a type-theoretical workflow engine, if some intelligent type coercion system is missing. To ensure an "easier" type matching, we can coordinate the SWT app to specify all possible values of label in the output section, as we used to do in v3 of the SWT app.

  "output": [{
    "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v1",
    "properties": {
      "timeUnit": "milliseconds",
      "labelset": ["bars", "slate", "chyron", "credits", "NEG"],
      "label": "bars"
    }
  }, {
    "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v1",
    "properties": {
      "timeUnit": "milliseconds",
      "labelset": ["bars", "slate", "chyron", "credits", "NEG"],
      "label": "slate"
    }
  }, {
    "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v1",
    "properties": {
      "timeUnit": "milliseconds",
      "labelset": ["bars", "slate", "chyron", "credits", "NEG"],
      "label": "chyron"
    }
  }, {
    "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v1",
    "properties": {
      "timeUnit": "milliseconds",
      "labelset": ["bars", "slate", "chyron", "credits", "NEG"],
      "label": "credits"
    }
  }, {
    "@type": "http://mmif.clams.ai/vocabulary/TimePoint/v1",
    "properties": {
      "labelset": ["bars", "slate", "chyron", "credits", "NEG"],
      "timeUnit": "milliseconds"
    }
  }],

Of course it's ugly, and actually very much redundant now due to the labelset property of the TimeFrame annotation. Alternatively, we can specify the OCR app to look for TFs with labelset= super set of ["slate", "chyron", "credit"] (which we don't have a way to specify "super-set-of" part). Since currently such type matching in workflows is done all by human, it's not a critical concern at the moment. So we can come back to this issue later, when we have a better understanding of the actual type-theoretical workflow engine.

to specify a property with "any" value

Now, let's say the OCR app uses existence of representatives property in the TF annotation to fine target annotations, instead of label. As pointed out in the above, representatives is a "variable" and since we can't specify a fixed value (or a fixed list of values) of it in the app metadata, we can't specify that this app requires TFs with representatives property. Namely, we can't do this.

  "input": [
    { "@type": "TimeFrame",
      "properties": { "representatives": ??? }  # we still want to make sure the this app wants TF with `representatives` prop!
    }
  ],

To solve this problem, we can introduce a special string "*" to have a special wild card meaning (any value). One obvious potential problem is that a property can actually have an asterisk as its value (e.g. Specifying a delimiter in a NLP app that handles some structured text).

[keighrim]

fixed via clamsproject/apps@b620729 and d77cf64