fix: required fields in nested pydantic models need to follow parent nullability

[joscha]

Description

Fields of nested pydantic models that are not kept as JSON via skip_nested_types=True need to follow the "requiredness" of their parent.

Example:

class A(BaseModel):
  id: int

class B(BaseModel):
  id: int
  child_a: Optional[A]
  dlt_config: ClassVar[DltConfig] = {"skip_nested_types": True}

results in a table that looks like this:

table B:

name	nullable
id	false
child_a__id	false

which is incorrect as I can serialize an instance of B that does not have a child_a, and thus, by extension, no id for child_a.

The change in this pull request retains the nullability of the field that contains the nested model, effectively making any required field of a nested, flattened model nullable as well if the parent model field is optional, i.e. the resulting table with the change in this PR is:

table B:

name	nullable
id	false
child_a__id	true

If we agree this is a bug and the fix in this PR is the way to go I can add a test & fix for deeply nested models as well.

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	4855ec6
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/67b39fd70535820008c7efea

[sh-rp]

@rudolfix I think it makes sense to forward the nullability to flattened fields, wdyt?

[sh-rp]

Just small changes, then we can merge this.

[sh-rp]

@joscha could you add the reverse test? Another field with a non optional child model and test that this is not nullable? Then make sure the conflicts are solved and the linter passes, then we can merge this :)

[joscha]

ah yes, good point, will do!

[joscha]

inverse test in 74af54c

[joscha]

Changes are in @sh-rp. Unsure about the failing dest tests - maybe an access issue? I can't see how my change would have affected them?

[sh-rp]

@joscha looks good to me, thanks! Could you merge in devel (or rebase) to make sure the tests pass?

[joscha]

@joscha looks good to me, thanks! Could you merge in devel (or rebase) to make sure the tests pass?

all green @sh-rp