Add KDVH migration package

[Lun4m]

This PR tackles the first part of issue #14.
I adapted some scripts originally written by Ketil for ODA to dump tables from KDVH and import them into LARD. Almost happy with the code...

Things I'm still not 100% sure about:

1. I'm using the KDVH proxy to dump the tables, but I couldn't find T_MONTH_INTERPOLATED and T_DIURNAL_INTERPOLATED, probably I need to connect to KDVH directly?
2. Is blobData used for non-scalar parameters (KLOBS for example)?
3. Do we need to dump observations that have both NULL data and flags? Related to ->
4. in data_functions.go we do a lot of postprocessing that I haven't touched since I don't really understand it. My main concern is with the insertion of "null" values (i.e. -32767, etc.), are these used somewhere downstream? Otherwise I might simply drop them, since our schema can handle real NULLs.

@ketilt, do you have any insight regarding these points? Or comments about the code, if you have time to take a look at it?

Still left to do:

• remove CorrKDVH field, since original and corrected have the same value
• Double check that the derived controlinfo and useinfo are correct

[ketilt]

1. It makes sense that they are not available in the proxy. Do not worry about dumping them. The tables are not in active use by anything. I can make a dump of them.
2. Yes
3. Related to what? There should be no reason to dump nulls. They likely exist only because of the table format of KDVH (empty cells in a row where something is not null)
4. This is a kvalobs feature. The missing indicator exists in two versions: -32767 means expected value missing and -32766 means value was found but was censored in QC. These numbers will not feature in the KDVH data, but they are all over the queues and all over ODA. If you've found a lossless way to handle this differently, it's probably better to do that instead

[ketilt]

Flag definitions:
https://dokit.met.no/klima/bjornn/kvalobs/flagg_i_kvalobs_versjon_8.7.1

[Lun4m]

3. Related to what? There should be no reason to dump nulls.
   They likely exist only because of the table format of KDVH (empty cells in a row where something is not null)

Related to point 4, sorry for the confusion! I was specifically asking since in makeDataPage it's fine if both are null:

lard/kdvh_importer/data_functions.go

Lines 28 to 39 in 648d5f8

	if kdvh.flagsAreInvalid() {
	useinfo = []byte("9999900900000000")
	} else {
	useinfo = []byte(kdvh.Flags + "00900000000")
	}
	if !nullData {
	controlinfo = []byte("0000000000000000")
	} else {
	controlinfo = []byte("0000003000000000")
	floatval = -32767
	}

But I'm with you, it doesn't make sense to dump them 👌

[ketilt]

3. Related to what? There should be no reason to dump nulls.
   They likely exist only because of the table format of KDVH (empty cells in a row where something is not null)

Related to point 4, sorry for the confusion! I was specifically asking since in makeDataPage it's fine if both are null:

lard/kdvh_importer/data_functions.go

Lines 28 to 39 in 648d5f8

	if kdvh.flagsAreInvalid() {
	useinfo = []byte("9999900900000000")
	} else {
	useinfo = []byte(kdvh.Flags + "00900000000")
	}
	if !nullData {
	controlinfo = []byte("0000000000000000")
	} else {
	controlinfo = []byte("0000003000000000")
	floatval = -32767
	}

But I'm with you, it doesn't make sense to dump them 👌

Hmm, could this be in the case of blob data?

[intarga]

@Lun4m Do you think there would be a point in removing the indices while importing?

I would think it's maybe redundant with COPY, but this doc page suggests it: https://www.postgresql.org/docs/current/populate.html#POPULATE-RM-INDEXES

In any case we should probably run ANALYZE after importing

[ketilt]

@Lun4m Are you filtering open and closed data? I don't remember seeing anything about that when reading the code, but perhaps I missed it

I can provide a list of which stnr / tables are defined as closed

[intarga]

I can provide a list of which stnr / tables are defined as closed

Thanks, but I don't think that's necessary, we can fetch it from stinfosys

[Lun4m]

Ah, I forgot to drop the indices. Edit: Done!