Commits on Jul 30, 2023

1. apacheGH-36375: [Java] Added creating MapWriter in ComplexWriter. (ap… …

   …ache#36351) (#32)
   
   Added new method rootAsMap() to ComplexWriter and implement it in ComplexWriterImpl for supporting map type.
   Previously in dremio side:
   
   When i trying to return map like output ComplexWrite
   with this code:
   
   org.apache.arrow.vector.complex.writer.BaseWriter.MapWriter mapWriter = out.rootAsList().map(false);
   
   mapWriter.startMap();
   for (java.util.Map.Entry<java.lang.Integer, java.lang.Integer> element : map.entrySet()) {
       mapWriter.startEntry();
       mapWriter.key().integer().writeInt((Integer) element.getKey());
       mapWriter.value().integer().writeInt((Integer) element.getValue());
       mapWriter.endEntry();
   }
   mapWriter.endMap();
   It use UnionMapWriter and generate schema like:
   EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key: Int(32, true) not null, value: Int(32, true) not null> not null, map: Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)> not null>>>
   But in OutputDerivation impl class where i should create output Complete type
   
   List<Field> children = Arrays.asList( CompleteType.INT.toField("key", false), CompleteType.INT.toField("value", false));
   return new CompleteType(CompleteType.MAP.getType(), CompleteType.struct(children).toField(MapVector.DATA_VECTOR_NAME, false));
   (This is only one valid case, because MapVector.initializeChildrenFromFields())
   return
   EXPR$0::map<key::int32, value::int32> I found a place where it start using union - PromotableWriter.promoteToUnion.
   And in the end i have
   
   SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was 
   schema(EXPR$0::map<key::int32, value::int32>)
    but then changed to 
   schema(EXPR$0::map<struct::struct<key::int32, value::int32>, map::map<key::int32, value::int32>>)
   * Closes: apache#36375
   
   Authored-by: Ivan Chesnov <[email protected]>
   
   Signed-off-by: David Li <[email protected]>

   

   xxlaykxx committed Jul 30, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 1f84c08
   • Browse repository at this point

   Copy the full SHA

   1f84c08 View commit details

   Browse the repository at this point in the history