Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ConstructorConstructor creating mismatching Collection and Map instances #2068

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions gson/src/main/java/com/google/gson/Gson.java
Original file line number Diff line number Diff line change
Expand Up @@ -1104,8 +1104,7 @@ public JsonReader newJsonReader(Reader reader) {
* @see #fromJson(String, TypeToken)
*/
public <T> T fromJson(String json, Class<T> classOfT) throws JsonSyntaxException {
T object = fromJson(json, TypeToken.get(classOfT));
return Primitives.wrap(classOfT).cast(object);
return fromJson(json, TypeToken.get(classOfT));
}

/**
Expand Down Expand Up @@ -1196,8 +1195,7 @@ public <T> T fromJson(String json, TypeToken<T> typeOfT) throws JsonSyntaxExcept
*/
public <T> T fromJson(Reader json, Class<T> classOfT)
throws JsonSyntaxException, JsonIOException {
T object = fromJson(json, TypeToken.get(classOfT));
return Primitives.wrap(classOfT).cast(object);
return fromJson(json, TypeToken.get(classOfT));
}

/**
Expand Down Expand Up @@ -1358,7 +1356,19 @@ public <T> T fromJson(JsonReader reader, TypeToken<T> typeOfT)
JsonToken unused = reader.peek();
isEmpty = false;
TypeAdapter<T> typeAdapter = getAdapter(typeOfT);
return typeAdapter.read(reader);
T object = typeAdapter.read(reader);
Class<?> expectedTypeWrapped = Primitives.wrap(typeOfT.getRawType());
Copy link
Collaborator Author

@Marcono1234 Marcono1234 Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems all other fromJson methods delegate to this one, so I have added the type check (which was previously Primitives.wrap(classOfT).cast(object) in the other fromJson methods) here instead.
However, this method did not actually have this check before.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this change against all of Google's internal tests and found a lot of failures with this exception, or with an exception about not being able to instantiate the abstract class com.google.common.collect.ImmutableList. In both cases I think the code in question should be fixed, because it is relying on accidental behaviour. A pattern I saw quite often was this:

@AutoValue
public abstract Foo {
  public abstract ImmutableList<Bar> bars();

  public static TypeAdapter<Foo> typeAdapter(Gson gson) {
    return new AutoValue_Foo.GsonTypeAdapter(gson);
  }

  @AutoValue.Builder
  public interface Builder {
    public Builder setBars(List<Bar> bars);
    public Foo build();
  }
}

This is using auto-value-gson to serialize and deserialize AutoValue classes. It works even if no TypeAdapterFactory has been registered for ImmutableList, because Gson deserializes a plain List. Auto-value-gson constructs the Foo instance using its builder, and the deserialized List works with the setBars method. That method calls ImmutableList.copyOf(List) so we do get the right ImmutableList type. But the additional checks in this PR don't let us get that far.

As I say, code like that only works accidentally. My plan is to add a GuavaCollectionsTypeAdapterFactory to extras and update the internal code to use that. I already found several custom ImmutableListTypeAdapterFactory and ImmutableMapTypeAdapterFactory classes which could be switched to use this.

Some of the other failures were just plain using the wrong types, which again happened to work through erasure and the like.

So in short, I think this change is OK, but it may require some client code to be updated. Most of the updates should be straightforward and will lead to client code that is more correct.

I think it will be a while before we can complete this because of all the adjustments that will need to be made.

if (object != null && !expectedTypeWrapped.isInstance(object)) {
throw new ClassCastException(
"Type adapter '"
+ typeAdapter
+ "' returned wrong type; requested "
+ typeOfT.getRawType()
+ " but got instance of "
+ object.getClass()
+ "\nVerify that the adapter was registered for the correct type.");
}
return object;
} catch (EOFException e) {
/*
* For compatibility with JSON 1.5 and earlier, we return null for empty
Expand Down Expand Up @@ -1403,8 +1413,7 @@ public <T> T fromJson(JsonReader reader, TypeToken<T> typeOfT)
* @see #fromJson(JsonElement, TypeToken)
*/
public <T> T fromJson(JsonElement json, Class<T> classOfT) throws JsonSyntaxException {
T object = fromJson(json, TypeToken.get(classOfT));
return Primitives.wrap(classOfT).cast(object);
return fromJson(json, TypeToken.get(classOfT));
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,15 +36,9 @@
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Queue;
import java.util.Set;
import java.util.SortedMap;
import java.util.SortedSet;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.ConcurrentNavigableMap;
import java.util.concurrent.ConcurrentSkipListMap;

/** Returns a function that can construct an instance of a requested type. */
Expand Down Expand Up @@ -94,7 +88,19 @@ static String checkInstantiable(Class<?> c) {
return null;
}

/** Calls {@link #get(TypeToken, boolean)}, and allows usage of JDK Unsafe. */
public <T> ObjectConstructor<T> get(TypeToken<T> typeToken) {
return get(typeToken, true);
}

/**
* Retrieves an object constructor for the given type.
*
* @param typeToken type for which a constructor should be retrieved
* @param allowUnsafe whether to allow usage of JDK Unsafe; has no effect if {@link #useJdkUnsafe}
* is false
*/
public <T> ObjectConstructor<T> get(TypeToken<T> typeToken, boolean allowUnsafe) {
Type type = typeToken.getType();
Class<? super T> rawType = typeToken.getRawType();

Expand Down Expand Up @@ -142,12 +148,19 @@ public <T> ObjectConstructor<T> get(TypeToken<T> typeToken) {
};
}

if (!allowUnsafe) {
String message =
"Unable to create instance of "
+ rawType
+ "; Register an InstanceCreator or a TypeAdapter for this type.";
return () -> {
throw new JsonIOException(message);
};
}

// Consider usage of Unsafe as reflection, so don't use if BLOCK_ALL
// Additionally, since it is not calling any constructor at all, don't use if BLOCK_INACCESSIBLE
if (filterResult == FilterResult.ALLOW) {
// finally try unsafe
return newUnsafeAllocator(rawType);
} else {
if (filterResult != FilterResult.ALLOW) {
String message =
"Unable to create instance of "
+ rawType
Expand All @@ -158,6 +171,9 @@ public <T> ObjectConstructor<T> get(TypeToken<T> typeToken) {
throw new JsonIOException(message);
};
}

// finally try unsafe
return newUnsafeAllocator(rawType);
}

/**
Expand Down Expand Up @@ -290,7 +306,6 @@ private static <T> ObjectConstructor<T> newDefaultConstructor(
}

/** Constructors for common interface types like Map and List and their subtypes. */
@SuppressWarnings("unchecked") // use runtime checks to guarantee that 'T' is what it is
private static <T> ObjectConstructor<T> newDefaultImplementationConstructor(
Type type, Class<? super T> rawType) {

Expand All @@ -303,33 +318,84 @@ private static <T> ObjectConstructor<T> newDefaultImplementationConstructor(
*/

if (Collection.class.isAssignableFrom(rawType)) {
if (SortedSet.class.isAssignableFrom(rawType)) {
return () -> (T) new TreeSet<>();
} else if (Set.class.isAssignableFrom(rawType)) {
return () -> (T) new LinkedHashSet<>();
} else if (Queue.class.isAssignableFrom(rawType)) {
return () -> (T) new ArrayDeque<>();
} else {
return () -> (T) new ArrayList<>();
}
@SuppressWarnings("unchecked")
ObjectConstructor<T> constructor = (ObjectConstructor<T>) newCollectionConstructor(rawType);
return constructor;
}

if (Map.class.isAssignableFrom(rawType)) {
if (ConcurrentNavigableMap.class.isAssignableFrom(rawType)) {
return () -> (T) new ConcurrentSkipListMap<>();
} else if (ConcurrentMap.class.isAssignableFrom(rawType)) {
return () -> (T) new ConcurrentHashMap<>();
} else if (SortedMap.class.isAssignableFrom(rawType)) {
return () -> (T) new TreeMap<>();
} else if (type instanceof ParameterizedType
&& !String.class.isAssignableFrom(
TypeToken.get(((ParameterizedType) type).getActualTypeArguments()[0]).getRawType())) {
return () -> (T) new LinkedHashMap<>();
} else {
return () -> (T) new LinkedTreeMap<>();
}
@SuppressWarnings("unchecked")
ObjectConstructor<T> constructor = (ObjectConstructor<T>) newMapConstructor(type, rawType);
return constructor;
}

// Unsupported type; try other means of creating constructor
return null;
Comment on lines +332 to +333
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #419 (comment) the concern was mentioned that the fallback will be Unsafe which might leave the Collection or Map implementation in a broken state because fields have not been initialized. And the built-in Collection and Map adapters might then encounter confusing exceptions when trying to add elements.

Not sure if we should throw an exception then in case of Collection or Map? Or we could maybe add an allowUnsafe parameter here (which is considered in addition to GsonBuilder#disableJdkUnsafe), which the built-in Collection and Map adapters can set to false because calling add and put will likely fail afterwards.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eamonnmcmanus, do you think this is something to consider?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the concern here. If we reach this point then rawType is neither a Collection nor a Map. Could you explain what sort of type you might consider throwing an exception for here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is a bug with GitHub and it is not showing the comment I was referring to above; here is an archived version:

I'm anxious about using UnsafeAllocator on map and collection types. Types like HashMap expect their constructors to run (and initialize state) but they won't necessarily get run if we rely on UnsafeAllocator. (We'd be relying on the presence of a no-args constructor).

When users define custom Collection or Map implementations without no-args constructor, then Unsafe would be used to create the instance. But because the Collection and Map adapter calls Collection and Map methods during deserialization, that most likely leads to confusing exceptions because Unsafe did not properly initialize the instances.

I have pushed a commit now which prevents Unsafe usage in this case; I hope that is fine. If you don't think it should be included please let me know and I revert it again.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I think I understand the concern now. It is not about this statement particularly, but the fact that the earlier return statements can return null if newCollectionConstructor or newMapConstructor does, and then we'll fall back to the Unsafe path.

We don't see this issue with Guava's ImmutableList etc because the public types are abstract, so Unsafe obviously can't instantiate them. I generally hate the nonsense with Unsafe allocation so I'm all for changes that allow less of it. However I'll want to run this latest version past Google's tests again to see if there are further unforeseen consequences.

}

private static ObjectConstructor<? extends Collection<? extends Object>> newCollectionConstructor(
Class<?> rawType) {

// First try List implementation
if (rawType.isAssignableFrom(ArrayList.class)) {
return () -> new ArrayList<>();
}
// Then try Set implementation
else if (rawType.isAssignableFrom(LinkedHashSet.class)) {
return () -> new LinkedHashSet<>();
}
// Then try SortedSet / NavigableSet implementation
else if (rawType.isAssignableFrom(TreeSet.class)) {
return () -> new TreeSet<>();
}
// Then try Queue implementation
else if (rawType.isAssignableFrom(ArrayDeque.class)) {
return () -> new ArrayDeque<>();
}

// Was unable to create matching Collection constructor
return null;
}

private static boolean hasStringKeyType(Type mapType) {
// If mapType is not parameterized, assume it might have String as key type
if (!(mapType instanceof ParameterizedType)) {
return true;
}

Type[] typeArguments = ((ParameterizedType) mapType).getActualTypeArguments();
if (typeArguments.length == 0) {
return false;
}
return $Gson$Types.getRawType(typeArguments[0]) == String.class;
}

private static ObjectConstructor<? extends Map<? extends Object, Object>> newMapConstructor(
Type type, Class<?> rawType) {
// First try Map implementation
/*
* Legacy special casing for Map<String, ...> to avoid DoS from colliding String hashCode
* values for older JDKs; use own LinkedTreeMap<String, Object> instead
*/
if (rawType.isAssignableFrom(LinkedTreeMap.class) && hasStringKeyType(type)) {
return () -> new LinkedTreeMap<>();
} else if (rawType.isAssignableFrom(LinkedHashMap.class)) {
return () -> new LinkedHashMap<>();
}
// Then try SortedMap / NavigableMap implementation
else if (rawType.isAssignableFrom(TreeMap.class)) {
return () -> new TreeMap<>();
}
// Then try ConcurrentMap implementation
else if (rawType.isAssignableFrom(ConcurrentHashMap.class)) {
return () -> new ConcurrentHashMap<>();
}
// Then try ConcurrentNavigableMap implementation
else if (rawType.isAssignableFrom(ConcurrentSkipListMap.class)) {
return () -> new ConcurrentSkipListMap<>();
}

// Was unable to create matching Map constructor
return null;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,10 @@ public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> typeToken) {
TypeAdapter<?> elementTypeAdapter = gson.getAdapter(TypeToken.get(elementType));
TypeAdapter<?> wrappedTypeAdapter =
new TypeAdapterRuntimeTypeWrapper<>(gson, elementTypeAdapter, elementType);
ObjectConstructor<T> constructor = constructorConstructor.get(typeToken);
// Don't allow Unsafe usage to create instance; instances might be in broken state and calling
// Collection methods could lead to confusing exceptions
boolean allowUnsafe = false;
ObjectConstructor<T> constructor = constructorConstructor.get(typeToken, allowUnsafe);

@SuppressWarnings({"unchecked", "rawtypes"}) // create() doesn't define a type parameter
TypeAdapter<T> result = new Adapter(wrappedTypeAdapter, constructor);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,10 @@ private static Object createAdapter(
// TODO: The exception messages created by ConstructorConstructor are currently written in the
// context of deserialization and for example suggest usage of TypeAdapter, which would not work
// for @JsonAdapter usage
return constructorConstructor.get(TypeToken.get(adapterClass)).construct();
// TODO: Should probably not allow usage of Unsafe; instances might be in broken state and
// calling adapter methods on them might lead to confusing exceptions
boolean allowUnsafe = true;
return constructorConstructor.get(TypeToken.get(adapterClass), allowUnsafe).construct();
}

private TypeAdapterFactory putFactoryAndGetCurrent(Class<?> rawType, TypeAdapterFactory factory) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,10 @@ public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> typeToken) {
TypeAdapter<?> valueAdapter = gson.getAdapter(TypeToken.get(valueType));
TypeAdapter<?> wrappedValueAdapter =
new TypeAdapterRuntimeTypeWrapper<>(gson, valueAdapter, valueType);
ObjectConstructor<T> constructor = constructorConstructor.get(typeToken);
// Don't allow Unsafe usage to create instance; instances might be in broken state and calling
// Map methods could lead to confusing exceptions
boolean allowUnsafe = false;
ObjectConstructor<T> constructor = constructorConstructor.get(typeToken, allowUnsafe);

@SuppressWarnings({"unchecked", "rawtypes"})
// we don't define a type parameter for the key or value types
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ public String toString() {
return adapter;
}

ObjectConstructor<T> constructor = constructorConstructor.get(type);
ObjectConstructor<T> constructor = constructorConstructor.get(type, true);
return new FieldReflectionAdapter<>(
constructor, getBoundFields(gson, type, raw, blockInaccessible, false));
}
Expand Down
55 changes: 55 additions & 0 deletions gson/src/test/java/com/google/gson/GsonTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,61 @@ public Object read(JsonReader in) {
}
}

@Test
public void testFromJson_WrongResultType() {
class IntegerAdapter extends TypeAdapter<Integer> {
@Override
public Integer read(JsonReader in) throws IOException {
in.skipValue();
return 3;
}

@Override
public void write(JsonWriter out, Integer value) {
throw new AssertionError("not needed for test");
}

@Override
public String toString() {
return "custom-adapter";
}
}

Gson gson = new GsonBuilder().registerTypeAdapter(Boolean.class, new IntegerAdapter()).create();
// Use `Class<?>` here to avoid that the JVM itself creates the ClassCastException (though the
// check below for the custom message would detect that as well)
Class<?> deserializedClass = Boolean.class;
var exception =
assertThrows(ClassCastException.class, () -> gson.fromJson("true", deserializedClass));
assertThat(exception)
.hasMessageThat()
.isEqualTo(
"Type adapter 'custom-adapter' returned wrong type; requested class java.lang.Boolean"
+ " but got instance of class java.lang.Integer\n"
+ "Verify that the adapter was registered for the correct type.");

// Returning boxed primitive should be allowed (e.g. returning `Integer` for `int`)
Gson gson2 = new GsonBuilder().registerTypeAdapter(int.class, new IntegerAdapter()).create();
assertThat(gson2.fromJson("0", int.class)).isEqualTo(3);

class NullAdapter extends TypeAdapter<Object> {
@Override
public Object read(JsonReader in) throws IOException {
in.skipValue();
return null;
}

@Override
public void write(JsonWriter out, Object value) {
throw new AssertionError("not needed for test");
}
}

// Returning `null` should be allowed
Gson gson3 = new GsonBuilder().registerTypeAdapter(Boolean.class, new NullAdapter()).create();
assertThat(gson3.fromJson("true", Boolean.class)).isNull();
}

@Test
public void testGetAdapter_Null() {
Gson gson = new Gson();
Expand Down
Loading