Certain files from the StatFi database fail to import

[ljleppan]

Certain files from the StatFi database, f.ex. 046_syyttr_tau_111_fi.px fail to import. The problem persists whether the file is downloaded using the python script or via browser.

The issues is that the dimensions of the generated row and column indices do not match the size of the DATA section in the input file. While reading in data, some of the data points also seem to contain tens of thousands of null bytes in their string representations.

A similar issue is encountered when using the pxR package to try to process the file:

> my.px.object <- read.px( "database/StatFin/oik/syyttr/046_syyttr_tau_111_fi.px" )
Error in read.px("database/StatFin/oik/syyttr/046_syyttr_tau_111_fi.px") : 
  The input file is malformed: data and varnames length differ
In addition: Warning message:
In scan(filename, what = "character", sep = "\n", quiet = TRUE,  :
  embedded nul(s) found in input

As the R library suffers from the same issue, I'm inclined to believe that this issue is with the input file, rather than the libraries themselves. A more in-depth investigation is needed to determine whether this is the case.