Add support for GraphQL

[fubhy]

The README mentions that there are plans to add GraphQL support "once it matures". I think we have reached that point. Actually quite some time ago to be honest. We are using GraphQL in production successfully for a while now. The module has good test coverage (although there is room for improvement of course). We've been very careful with doing a stable release because we are still actively working on additional features. The core APIs have been stable for a while, however. A few days ago we released the first alpha version. I'd like to highlight, however, that the module is considerably more stable and production ready than the average module published on d.o.

So... I'd like to kickstart a discussion of how we can add GraphQL to Reservoir. How do y'all envision support for various service models side-by-side in the first place?

I am open for chatting about this in person at Decoupled Dev Days or in a call at your convenience.

/cc @pmelab

[wimleers]

I think that ideally, we ship with both JSON API and GraphQL enabled automatically out of the box. You just choose which to use, or you can even use both. Because that results in the lowest possible bar to entry. We want to make Reservoir as smooth, pleasant, simple and configurationless as possible.

Thoughts?

[tedbow]

@fubhy thanks for starting this discussion up. I will be at Decoupled Dev Days so interested in talking their.

So would adding GraphQL be as simple as including the module and it's dependencies. Are there any other Drupal modules that you would recommend installing with the module?

The other thing I think we should think about is documentation? Is there need for or the ability to create insite documentation like OpenAPI provides for JSON API. Does this provide that? https://github.com/fubhy/graphql-drupal/tree/8.x-3.x/modules/graphql_voyager
I just wondering because we hopefully have or will have soon full auto generated docs for JSON. It might seem weird to have nothing for GraphQL

[fubhy]

Hi @tedbow!

Yes, simply installing the sub modules (or rather, the sub-modules that you need) should do. We have two special sub-modules for supporting persisted queries (query maps) (https://dev-blog.apollodata.com/persisted-graphql-queries-with-apollo-client-119fd7e6bba5) and two which simply provide schema introspection capabilities (voyager via graphql_voyager and graphiql via graphql_graphiql). Those four modules don't add anything to the schema but add functionality around it.

The other modules all provide or extend schema definitions.

As for the documentation... We are currently working on a blog post series that started a few weeks ago. It is about GraphQL, Drupal and React. The first few blog posts have already been published and anothre blog post (Extending GraphQL in Drupal - Part 2) is going to be published on Wednesday. We are going to use these blog posts as a starting point for the documentation included in the module and plan to submit a PR for that in the next few days (basically copying the content of the blog posts over in a format that works for gitbooks.io). You can find the blog post series here:

1. GraphQL Introduction: https://www.amazeelabs.com/en/blog/graphql-introduction
2. Drupal, GraphQL, React and Apollo: https://www.amazeelabs.com/en/blog/drupal-graphql-react-apollo
3. Drupal and GraphQL - Batteries included: https://www.amazeelabs.com/en/blog/drupal-graphql-batteries-included
4. Extending GraphQL - Part 1: https://www.amazeelabs.com/en/blog/extending-graphql-part1-fields
5. Extending GraphQL - Part 2: Will be published on Wednesday

[tedbow]

@fubhy thanks for the info and the update on the API First call.

We need to solve the problem of making sure the bundles we need exposed to graphql are exposed by default. GraphQL module provides this configuration page for exposing entity types and bundles to the graph.

The configuration is based on View modes. Obviously this would be very confusing in Reservoir since we hide the manage display page.

So without changing the GraphQL module like we could either

1. Update ship with default configuration to expose the user, file, tag and article bundle and update this configuration each time a content type is saved. or
2. Add our own GraphQL Plugin with a weight that will override the existing one and just expose the raw values. @fubhy had mentioned how to do this but I can't remember now.

I think 2 is probably the way to go but that would mean we would want to take away access to the screen above because it would probably make the form not work as expected.

If we did 2 I don't think we should put it in Reservoir UI module as it is not concerned with the UI. So either make a generic Reservoir module or a Reservoir GraphQL module. I would vote for Reservoir GraphQL module so that it would easier to figure out.

[pmelab]

@tedbow But the default view mode is still there in the background?

Getting rid of the confit screen is trivial. These values are accessed through one service with three methods: isEntityTypeExposed, isEntityBundleExposed and getExposedViewMode. We could just swap in another one that returns true, true and 'default'. If your default view mode still picks up all new fields as it does in vanilla Drupal, you should always have all content entities with all fields as rendered strings in you schema.

Obviously we don't want strings for everything. Custom field plugins (image fields, links ... please have a look at blogpost nr. 3) use the FieldFormatterDeriver to apply on fields that use the formatter defined in the plugin's field_formatter annotation property.
Currently these are set to the graphql_* formatters because there was a lot of confusion why some fields are "special" and some not, and it's more clear if they are configured explicitly.
But we could swap the implementation class of the graphql_core.field_manager service with a subclass that replaces this property with the field types default formatter (graphql_image => image, graphql_link => link). Then the special fields would apply per field type and don't need extra configuration.

For raw values we would probably have to create a new deriver that specifically targets the fields that you want exposed as typed data properties.

TL;DR: No changes to the graphql module necessary. The reservoir_graphql module would consist of three simple classes.

[wimleers]

I'm pretty concerned by https://www.amazeelabs.com/en/blog/drupal-graphql-batteries-included and https://github.com/fubhy/graphql-drupal/tree/8.x-3.x/modules.

That's a lot of modules. And I don't understand why many of them exist at all. For example: graphql_boolean. Why is that even a separate module? Same for graphql_entity_reference. And graphql_image.
Of course, for Reservoir, this is not necessarily a concern, because we can just enable all of these.

I'm also confused why the set of exposed fields depends on a view mode.

I'm asking this because we're very very very focused on simplicity. The jsonapi module is much simpler. (I helped change the direction of the jsonapi module to ensure simplicity.)

Don't get me wrong — I'm very excited about the prospect of GraphQL! But the module looks very different since I last saw it, and has grown significantly in complexity. If we can keep the complexity completely hidden, then it's fine. I want to better understand which complexity is necessary (and can't be hidden), versus which can be hidden.

I hope that makes sense!

[pmelab]

The number of modules: The reason is simply that they either are advanced/optional/dev-only (mutations, views, graphiql, explorer), or they depend on different core modules (image, file, link ...). And we want the schema to remain as slim as possible. Enable just what you need. And if your data hub doesn't store images, you don't need the image modules.

Confusing fields: The graphql_boolean field is going away since it's covered by the raw-value feature by now. But the other field types provide functionality on top of the pure typed data layer.
The GraphQL module started out based on typed data too, but this didn't work out well. This approach sacrifices too much of what Drupal is and you end up rebuilding all that features in your client.

Simple example: JSON API by default exposes the unsanitized HTML string and the input format for formatted text fields. The input format has literally no value outside of Drupal, unless you are willing to rebuild input filters in the consumer. So what we chose to do by default, is exposing the rendered field value instead.
This fits a surprisingly high number of cases and keeps a lot more of Drupal's features in the game (sanitation, localisation ...). And for situations where this doesn't make sense, we provide other options. Like the graphql_image field that provides the fields for requesting specific image styles, or graphql_entity_reference that resolves to the referenced entity directly so you can continue to query it's properties. And there is the GraphQL Raw formatter that goes back down to typed data properties if necessary.
JSON API has enhancers which is basically the same concept. We are just delivering common ones out of the box.

View modes: We need a way to restrict and configure what and how to expose. Not every use of the GraphQL Module is a pure decoupled data hub. There are lots traditional (head-ful?) projects that need to expose just a subset of their information, and we don't want to leak data by default, so everything is opt-in.
View modes perfectly fit the "field"-approach above, the audience had a decade to familiarise with them and even core ships with a "RSS" view mode, which is a very similar use case. So it's simply the biggest bang for the buck.
@fubhy is working on an RFC for a "Graph configuration UI", which might replace (or augment) this concept in the long run, since it will be more flexible and powerful. But I doubt it will be easier to understand and use.

I totally agree that the GraphQL module is not simple, but thats not our goal. We want to provide options for a lot of use cases. Reservoir can fill the "opinionated" role and preselect what you seem fit and hide the rest.

[fubhy]

I am going to publish a technical blog post / RFC this week about the plans for the configuration format and user interface for enabling fully customizable schema configuration. This is going to take some time to implement and I am trying to gather funds / sponsoring for allowing more focused work on it.

One more idea for potentially reducing the number of modules would be to include all of them in the same module and then conditionally enable the contained plugins based on the existence of the corresponding core module. Not sure about that... @pmelab?

Maybe it's best to set up a call with you @tedbow and @pmelab and me to discuss how to best move forward and find solutions for the integration problems especially around the view modes together.

[wimleers]

It's very very interesting to see how GraphQL made different choices than rest + jsonapi :) jsonapi already influenced rest and vice versa. Looking forward to make both of those better thanks to graphql, and to make graphql better too thanks to the other two!

Great discussion here!

---

The GraphQL module started out based on typed data too, but this didn't work out well. This approach sacrifices too much of what Drupal is and you end up rebuilding all that features in your client.

Interesting! Where can we read more about this?

The input format has literally no value outside of Drupal, unless you are willing to rebuild input filters in the consumer.

This is only true if you're building a read-only application (pure consumer). As soon as you need to do mutations, you need to access the raw value, because that's what users can edit.

View modes: We need a way to restrict and configure what and how to expose. Not every use of the GraphQL Module is a pure decoupled data hub.

What's exposed by rest and jsonapi depends on field access. Relying solely on view modes is dangerous because it may result in field access not being respected depending on how GraphQL implements it (I'm assuming it also respects field access).

And more importantly, relying on view modes at all may result in required fields not being made available, which means that reading some entity X, then mutating it may result in a validation error because some required field's value is not exposed during reading, but is required during writing.

This fits a surprisingly high number of cases and keeps a lot more of Drupal's features in the game (sanitation, localisation ...).
Can you explain how using view modes brings sanitization + localization? (And what else?)

even core ships with a "RSS" view mode, which is a very similar use case
I disagree this is similar. RSS is read-only. Plus, RSS is specifically about a reading experience, not a developer experience/API.

I totally agree that the GraphQL module is not simple, but thats not our goal. We want to provide options for a lot of use cases. Reservoir can fill the "opinionated" role and preselect what you seem fit and hide the rest.

Yep! But part of the role of Reservoir is ensuring the mental model remains consistent and simple. So I'm being critical/asking these questions precisely because I want to ensure that I can fit in GraphQL without needing to ask more of developers than we have so far. It's totally possible the GraphQL module came up with sensible additional requirements, but it's equally possible that the GraphQL module could be simplified :) Let's find out!

---

I am going to publish a technical blog post / RFC this week about the plans for the configuration format and user interface for enabling fully customizable schema configuration.

Configuration format? UI for customizable schema?

I can see how you would want aliasing (renaming Drupal fields to something more sensible for the GraphQL API consumer), and the ability to show/hide certain computed fields/properties. But other than that, I'm inclined to say it should be governed by field access. Because that ensures consistent operation across everything: UI + REST + JSON API + GraphQL.

If that configuration format/UI therefore is effectively a UI for tweaking field access: then yes, that'd make sense to me! But it'd be useful not just for GraphQL — it'd be useful outside that too :)

Looking forward to read more about this :)

[pmelab]

This is only true if you're building a read-only application (pure consumer). As soon as you need to do mutations, you need to access the raw value, because that's what users can edit.

I'm sorry if this was not clear. View modes/fields are only affecting output, not input. Input and output types are different things in GraphQL, and for the latter we use typed data directly.

Can you explain how using view modes brings sanitization + localization? (And what else?)

Fields and formatters are a huge source of functionality in Drupal core and contrib, and Drupal developers are already familiar with them. More than once we found ourselves in a "damn, we have to implement this in javascript" situation for things we took for granted. It's hard to draw the line where Drupal ends and where the consumer starts, and often it moves during development. Displays and fields give us more room to move it in our favor.

Yep! But part of the role of Reservoir is ensuring the mental model remains consistent and simple.

GraphQL is very different to REST and JSON API (not document oriented in the first place), and it already requires a very different mental model. You can get closer by reducing GraphQL to typed data, but this also means to leave out a lot.

[fubhy]

Configuration format? UI for customizable schema?

I can see how you would want aliasing (renaming Drupal fields to something more sensible for the GraphQL API consumer), and the ability to show/hide certain computed fields/properties. But other than that, I'm inclined to say it should be governed by field access. Because that ensures consistent operation across everything: UI + REST + JSON API + GraphQL.

There is much more to it than just aliasing:

Adding the same graph edge multiple times with different configuration (e.g. the same field twice with different names and different output formatter)
Deciding with edge input values are exposed and which are statically pre-configured
Deciding which edges exactly show up in the graph in the first place
Merging edges (e.g. directly accessing delta 1 on a field)
Merging multiple edges (field -> delta 1 -> specific typed data property)
Overriding field access

If that configuration format/UI therefore is effectively a UI for tweaking field access: then yes, that'd make sense to me! But it'd be useful not just for GraphQL — it'd be useful outside that too :)

It's definitely not just about field access.

Looking forward to read more about this :)

I briefly mentioned the core idea already quite some time ago (nearly 2 years ago). Basically the goal is to expose the entirety of the Drupal Data Model as a Graph of Possibilities and then allowing users to create Graph Derivatives through configuration. This would be entirely decoupled from the GraphQL module. These Graph Derivatives could then be used to generate Search API configuration, a GraphQL schema or a JSON API schema.

The user interface for the configuration would also be visualized as a convenient graph interface built with JavaScript and something like D3 or another Graph Visualization library.

More about that in the post that I am preparing.

[wimleers]

I'm sorry if this was not clear. View modes/fields are only affecting output, not input. Input and output types are different things in GraphQL, and for the latter we use typed data directly.

There is much more to it than just aliasing:

There is much more to it than just aliasing:

Adding the same graph edge multiple times with different configuration (e.g. the same field twice with different names and different output formatter)
Deciding with edge input values are exposed and which are statically pre-configured
Deciding which edges exactly show up in the graph in the first place
Merging edges (e.g. directly accessing delta 1 on a field)
Merging multiple edges (field -> delta 1 -> specific typed data property)
Overriding field access

These two statements seem to suggest GraphQL is focused on/favoring/primarily designed for read-only applications. Because it's effectively impossible to send the same data that was fetched, if A) edges are omitted (or merged, which implies omitting), B) fields use a particular "output formatter" (which must be a GraphQL-specific @FieldFormatter plugin then).

Many of the things I'm reading here really only make sense for applications that only read data, or at least have far more reading than writing. That's fine, of course — it's just a design decision. But it'd be good to see that acknowledged/documented — or if my understanding is wrong, I'd be happy to read that too :)

---

You can get closer by reducing GraphQL to typed data, but this also means to leave out a lot.

That's indeed the angle I'm coming from: Typed Data as the source of truth, and rest, jsonapi and graphql (and rendered HTML of course) as possible representations. I'd love to better understand what it is that that would be leaving out. I'm sure you have documentation somewhere where these big decisions/insights are documented, with clear real-world examples?

[fubhy]

These two statements seem to suggest GraphQL is focused on/favoring/primarily designed for read-only applications. Because it's effectively impossible to send the same data that was fetched, if A) edges are omitted (or merged, which implies omitting), B) fields use a particular "output formatter" (which must be a GraphQL-specific @FieldFormatter plugin then).

None of the statements suggest that it's only about reading. Merging edges does not prevent you from doing read AND write operations through the same pipeline. The information about which edges were merged is obviously not lost. So if you merge two edges (e.g. Field "foo" => delta 1) into one edge, it's essentially a new edge that contains the information and static values configured for both "nested" edges. It's like a sub-graph that is simply not exposed in the resulting schema. Of course that can potentially support both read and write operations.

Also, output formatter in this context does NOT mean field formatter. Disregard fields for a moment. Not all edges in the Drupal data graph are fields. A possible output formatter for an edge could obviously be one that can be configured to use a field formatter. That's just one type of output formatter plugin though and would only apply to / be configurable for things that are fields.

By the way, GraphQL is not REST and it has clear separation between "queries" and "mutations" and it makes a lot of sense. Write and Read operations are fundamentally different in any case.

That's indeed the angle I'm coming from: Typed Data as the source of truth
Typed data is extremely limiting and can not serve as the single source of truth. It can certainly serve as a source of information for a subset of the graph but certainly not everything.

[fubhy]

Just to let you know: We've added a much tighter Views integration recently as well as support for XML parsing of text fields. We've also further worked on our test coverage. In general the module is becoming more and more stable every day while also constantly gaining additional powerful features.

With the new improved Views integration we now also support Fielded Views as well as Views with custom data providers like e.g. Search API Views.

[wimleers]

I wanted to cross-post contentacms/contenta_jsonapi#101 (comment) when I posted that two days ago, but obviously that failed. GitHub--

I met with @fubhy and @pmelab at DrupalCon Vienna. We discussed my criticism/feedback that the GraphQL module needed to become simpler. Because for:

• this to be usable by Drupal users that aren't experts
• this to eventually be committable to Drupal core
• this to be maintainable

this simplification was IMHO essential. (See #59 (comment) for the detailed original concerns.)

@fubhy and @pmelab took this feedback and then made an incredible push forward.

I'm happy to say that in alphas 7 and 8, they addressed all of the above: https://twitter.com/wimleers/status/922472920626204673. This also came up during Monday's API-First Initiative meeting. During that meeting, @e0ipso also confirmed those same concerns: that's the reason why Contenta hadn't yet adopted GraphQL (this issue). (I think @e0ipso was referring to contentacms/contenta_jsonapi#101 (comment) in particular.)

And just a few hours ago, they released https://www.drupal.org/project/graphql/releases/8.x-3.0-beta1 — which further cleaned things up!

The situation today is:

• no more special "GraphQL" view mode to be configured for every entity type + bundle, instead it respects the TypedData definition
• after installing, you should be immediately able to make GraphQL queries
• no more dozen separate modules to install (for BC, https://github.com/drupal-graphql/graphql-legacy exists): just install graphql + graphql_core (the latter to implement various plugins on behalf of core modules)
• rather than having one monster module with lots of submodules for various edge cases, move those submodules into their own projects. Especially because they would take a very long time to become stable, if ever. Separate modules now exist for graphql_xml, graphql_json, graphql_twig and graphql_views

Very exciting!