Read access to NameMap mapping.

[vladiulianbogdan]

Hello,

I have to write a custom encoder from Protobuf to something else. I've noticed that the Visitor protocol is public so I was planning to create a custom Visitor that will do the encoding.

My problem is that I don't have any way to get the name of the field using the fieldNumber that I receive in the visitor call. That's because the names(for number: Int) -> Names? is internal.

Would it make sense to add a public method names(for number: Int) -> String? that can be used for writing custom encoders/visitors?

Thanks!

[thomasvl]

Can you explain a little more what the use for this is? Are you trying to transform something from a proto into another format? i.e. - what is proto bringing to the table in the first place?

The _NameMap is an internal detail that we don't want to expose. There has been some discussion about expose the Descriptor that describes the message like some other languages do, and that could possibly be used.

[vladiulianbogdan]

I'm trying to migrate from a proprietary binary format to Protobuf without breaking an API contract. Legacy client code is also sending and receiving NSDictionary through the module I am working on.

Using Protobuf schemas with field names matching the dictionary keys and going through JSON nearly allowed me to just switch out the serialization format. But when parsing a Protobuf Message to a dictionary going through JSON, type information is lost.

Example:

ProtoBufMessage {
    enum Status {
        ON = 0;
        OFF = 1;
    }
    optional Status status = 1;
    optional bytes name = 1;
}

Expected: ["status":  0, "name": Data(...)]
Actual: ["status": "ON", "name": "cmFuZG9tX25hbWVfYXNfc3RyaW5n"]

Writing a custom Visitor (that could even skip JSON) seemed to be the natural way to solve this problem. However, writing my own Visitor does not allow me to access the property names like the internal JSON Encoder can.

[mickeyreiss-visor]

I have a similar use case.

I am receiving a domain object, which is defined in proto and encoded to JSON on my server and receiving it on the iOS side as a [String,Any]?

To do this, I am implementing a custom Decoder.

It would be helpful to get access to number(forJSONName: String) -> Int?. This would allow me to implement nextFieldNumber more easily in my Decoder.

[thomasvl]

If the server and client are using protos, why not just use the built in support for binary or json? i.e. - why do you need a custom decoder?

A custom decoder sorta implies you are doing your own format thing, in which case, nothing would seem to guarantee the name transforms done by protos would match your custom encoding, no?

[mickeyreiss-visor]

In my case, I have limited control over the transport layer and its Swift SDK. I am using proto3 standard JSON encoding on the write side, but, unfortunately, I receive a Swift dictionary on the read side.

Specifically, I am using firestore (gcp docs/firebase docs).

My main motivation to use protobuf is to increase type-safety across my client and server codebases.

[thomasvl]

The the keys match, then you might want to try serialize the JSON again (to bytes) and feed it to SwiftProtobuf, no custom decoder needed, so it could be list total code for you to maintain.

[mickeyreiss-visor]

Agreed in theory... That idea works for many cases, however some types encoded in the DocumentSnapshot's data dictionary are not all handled by NSJSONSerialization or JSONEncoder. For example, timestamps needed to be mapped from Timestamp to Date. Some mapping code needs to exist somewhere (if I want to use this data sync stack).

[nwparker]

@mickeyreiss, I'm in the same boat as you (also for Firebase). Interesting that we independently reached the same conclusion. Perhaps that implies something.

What did you end up doing here?

[mickeyreiss]

@nwparker we ended up writing our own code generation that keeps different languages' firestore models in sync based on a schema.

[nwparker]

Thanks for the reply, @mickeyreiss. Let me know if you guys plan to ever share any of that code out. It sounds useful.

For now I've decided to just go with writing back and forth via JSON since it's easiest to implement. Although it is sad to lose some types in Firebase. For now I'm only storing Timestamps and I suppose it's fine to store them as ISO8601 strings (they preserve sorting this way anyways)

[mickeyreiss]

@nwparker that plan makes sense. In retrospect, I’ve noticed that firestore does use protos behind the scenes. They hide this fact in their client SDKs. It might make sense to construct a Write directly, which sets fields as a typed map<string, Value>.

Feel free to contact me directly regarding our codegen if you’re interested.

[wzio]

Hei, @mickeyreiss same situation is here, I am want to custom the code generation for my own, like generate some extentsion which has method like func getValue(by filedNumber: Int)-> Any? func setValue(by filedNumber: Int, value: Any?). Did you use sone meta-programming tools like Sourcery or just modify the protoc-gen-swift
Can you give some advice?

[mickeyreiss]

(My advice is about 1 year out of date, so please take it with a grain of salt.)

1. We ended up depending on Objective-C protos, rather than Swift. This allowed us to a avoid an additional dependency (because Firestore ios already relies on the Objective-C protobuf runtime.)

2. We ended up generating a very explicit decoder that maps and casts each values from Firestone’s public API to our struct. I believe most values go through an intermediate Foundation data structure. This is not ideal, but it offloads some of the heavy lifting.

3. Yes, I would recommend using a type-safe code generation tool like Sourcery. We use a mix of approaches for different languages, and the type-safe tools tend to be more maintainable than the string template based codegen.

I’m essence, there is no metaprogramming in the runtime. All of the type mapping occurs at compile time.

[tbkka]

Did you ... just modify the protoc-gen-swift ...

You should not modify protoc-gen-swift or its output. If you do, then you may have problems in the future when you need to upgrade to a new version.

However, you can use SwiftProtobufPluginLibrary in your own code generator. This library has some of the logic needed to interface with protoc so you can create additional code that augments protoc-gen-swift output.

[sewerynplazuk]

Are there any plans on allowing the read access to NameMap mapping? As many other stated, this would be extremely valuable for debugging and logging purposes.

For now, we need to use workarounds like the reflection based approach which unnecessarily complicates the extremely simple task (getting the value's description) by the order of magnitude, is less performant and much more error-prone.

[thomasvl]

nothing planned at the moment. What's your usecase for needing the data?

[sewerynplazuk]

@thomasvl Sorry for the very late reply. We have an analytics system that uses proto messages under the hood. The idea is that clients should be able to inspect recorded events in the debug menu.