Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem flattening data #9

Open
tdunning opened this issue Sep 4, 2014 · 1 comment
Open

Problem flattening data #9

tdunning opened this issue Sep 4, 2014 · 1 comment

Comments

@tdunning
Copy link
Owner

tdunning commented Sep 4, 2014

Hi Ted,

Here's the error I get:

$ ./synth -schema msg/clickstream.json
id,user_id,program_id,timestamp,last_five_actions,action,device,br,la,st,os
Exception in thread "main" java.lang.IllegalArgumentException: Cannot flatten type class com.fasterxml.jackson.databind.node.ObjectNode
at com.mapr.synth.samplers.FlattenSampler.sample(FlattenSampler.java:33)
at com.mapr.synth.samplers.FlattenSampler.sample(FlattenSampler.java:12)
at com.mapr.synth.samplers.SchemaSampler.sample(SchemaSampler.java:69)
at com.mapr.synth.Synth.main(Synth.java:34)

Here's the schema, which is pretty much just lifted from the example in README.md, except for the fact I'm using the lookup class for the value of the "base" key:

[
{"name":"id", "class":"id"},
{"name":"user_id", "class": "foreign-key", "size": 100000 },
{"name":"program_id", "class": "foreign-key", "size": 125 },
{"name":"timestamp", "class": "date", "format": "yyyy-MM-dd HH:MM:ss.SS", "start": "2014-08-12 00:00:00.00" },

{"name":"last_five_actions", "class": "flatten", "value":
{
"class": "sequence", "length": 5, "base": {
"class": "lookup", "file": "msg/actions.csv"
}
}
},

{"name":"action", "class":"string", "dist":{
"play":21, "stop":19, "pause":16, "ff": 12, "rw": 7, "replay": 2}
},

{"name":"device", "class":"string", "dist":{
"large":25, "phone":45, "tablet":25, "other": 5}
},

{"name":"br", "class":"browser"},
{"name":"la", "class":"language"},
{"name":"st", "class":"state"},
{"name":"os", "class":"os"}
]

I'm proceeding without the lookup stuff, since that produces data that I think is good enough for my purpose.

@tdunning
Copy link
Owner Author

tdunning commented Nov 9, 2014

Vince,

I am taking a look at this again. What was the desired result. I am not quite clear what you wanted to happen. What flatten does is take a list of lists and removes one level of nesting. For a list of objects, that isn't so simple. One thought I had about what you might have wanted is that you might have wanted to have a single output record for each element of the sequence with all other fields in the record replicated identically. If so, the short walk to that result is to simply use python to reprocess the output.

Can you say more about what you want?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant