Skip to content

Commit

Permalink
Merge branch 'main' into analyze_port
Browse files Browse the repository at this point in the history
  • Loading branch information
@daveads
daveads committed Jul 23, 2024
2 parents 9cccb7c + da28f61 commit 3fb8dfc
Show file tree
Hide file tree
Showing 9 changed files with 67 additions and 58 deletions.
29 changes: 19 additions & 10 deletions .github/workflows/bench.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,24 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
strategy:
matrix:
service: [apollo_server, caliban, netflix_dgs, gqlgen, tailcall, async_graphql, hasura, graphql_jit]
service:
[
apollo_server,
caliban,
netflix_dgs,
gqlgen,
tailcall,
async_graphql,
hasura,
graphql_jit,
]
steps:
- name: Checkout (GitHub)
uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}

- name: Build devcontainer and run setup and benchmark
uses: devcontainers/[email protected]
with:
Expand All @@ -37,27 +47,26 @@ jobs:
- name: List benchmark files
run: |
ls -la bench*.txt || echo "No matching files found"
- name: Upload benchmark results
uses: actions/upload-artifact@v3
with:
name: benchmark-results
path: bench*.txt


analyze:
needs: benchmark
needs: build
runs-on: benchmarking-runner
steps:
- name: Checkout (GitHub)
uses: actions/checkout@v4

- name: Download all benchmark results
uses: actions/download-artifact@v3
with:
name: benchmark-results
path: .

- name: List downloaded artifacts
run: ls -la bench*.txt || echo "No matching files found"

Expand All @@ -67,19 +76,19 @@ jobs:
- name: Print benchmark results
run: cat ./results.md

- name: Comment benchmark results on PR
if: github.event_name == 'pull_request_target'
uses: peter-evans/commit-comment@v3
with:
sha: ${{ github.event.pull_request.head.sha }}
body-path: "results.md"
reactions: eyes

- name: Commit and push changes (on main branch)
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: stefanzweifel/git-auto-commit-action@v5
with:
branch: main
commit_author: Author <[email protected]>
commit_message: "Update performance results in README.md"
commit_message: "[ci skip] update performance results in README.md"
48 changes: 24 additions & 24 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,32 +51,32 @@ Get started with the benchmarks:
| Query | Server | Requests/sec | Latency (ms) | Relative |
|-------:|--------:|--------------:|--------------:|---------:|
| 1 | `{ posts { id userId title user { id name email }}}` |
|| [Tailcall] | `29,496.60` | `3.38` | `196.74x` |
|| [async-graphql] | `1,805.45` | `55.70` | `12.04x` |
|| [Caliban] | `1,547.91` | `64.33` | `10.32x` |
|| [GraphQL JIT] | `1,335.26` | `74.60` | `8.91x` |
|| [Gqlgen] | `772.80` | `128.41` | `5.15x` |
|| [Netflix DGS] | `358.06` | `177.42` | `2.39x` |
|| [Apollo GraphQL] | `268.25` | `366.16` | `1.79x` |
|| [Hasura] | `149.93` | `549.07` | `1.00x` |
|| [Tailcall] | `29,173.60` | `3.41` | `211.81x` |
|| [async-graphql] | `2,019.59` | `49.49` | `14.66x` |
|| [Caliban] | `1,744.80` | `57.28` | `12.67x` |
|| [GraphQL JIT] | `1,342.93` | `74.16` | `9.75x` |
|| [Gqlgen] | `797.87` | `124.41` | `5.79x` |
|| [Netflix DGS] | `366.18` | `181.49` | `2.66x` |
|| [Apollo GraphQL] | `263.39` | `372.49` | `1.91x` |
|| [Hasura] | `137.74` | `587.80` | `1.00x` |
| 2 | `{ posts { title }}` |
|| [Tailcall] | `58,939.60` | `1.69` | `67.77x` |
|| [Caliban] | `9,238.76` | `11.16` | `10.62x` |
|| [async-graphql] | `9,138.76` | `11.34` | `10.51x` |
|| [Gqlgen] | `2,136.19` | `48.48` | `2.46x` |
|| [Apollo GraphQL] | `1,775.87` | `56.26` | `2.04x` |
|| [Netflix DGS] | `1,593.80` | `69.82` | `1.83x` |
|| [GraphQL JIT] | `1,399.68` | `71.35` | `1.61x` |
|| [Hasura] | `869.69` | `114.81` | `1.00x` |
|| [Tailcall] | `58,485.30` | `1.70` | `67.37x` |
|| [Caliban] | `9,751.71` | `10.62` | `11.23x` |
|| [async-graphql] | `9,444.13` | `10.73` | `10.88x` |
|| [Gqlgen] | `2,181.26` | `47.48` | `2.51x` |
|| [Apollo GraphQL] | `1,760.69` | `56.72` | `2.03x` |
|| [Netflix DGS] | `1,598.09` | `70.06` | `1.84x` |
|| [GraphQL JIT] | `1,392.45` | `71.71` | `1.60x` |
|| [Hasura] | `868.17` | `114.99` | `1.00x` |
| 3 | `{ greet }` |
|| [Caliban] | `67,637.00` | `1.07` | `25.83x` |
|| [Tailcall] | `59,557.80` | `1.69` | `22.75x` |
|| [Gqlgen] | `47,525.20` | `5.21` | `18.15x` |
|| [async-graphql] | `47,150.60` | `2.21` | `18.01x` |
|| [Netflix DGS] | `8,261.44` | `15.02` | `3.16x` |
|| [Apollo GraphQL] | `8,186.12` | `12.54` | `3.13x` |
|| [GraphQL JIT] | `5,273.45` | `18.93` | `2.01x` |
|| [Hasura] | `2,618.18` | `38.13` | `1.00x` |
|| [Caliban] | `67,512.70` | `1.07` | `26.49x` |
|| [Tailcall] | `59,015.70` | `1.71` | `23.16x` |
|| [Gqlgen] | `47,378.10` | `5.25` | `18.59x` |
|| [async-graphql] | `47,361.00` | `2.25` | `18.58x` |
|| [Netflix DGS] | `8,297.54` | `14.39` | `3.26x` |
|| [Apollo GraphQL] | `8,128.40` | `12.43` | `3.19x` |
|| [GraphQL JIT] | `5,241.04` | `19.05` | `2.06x` |
|| [Hasura] | `2,548.54` | `39.16` | `1.00x` |

<!-- PERFORMANCE_RESULTS_END -->

Expand Down
Binary file modified assets/latency_histogram1.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/latency_histogram2.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/latency_histogram3.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/req_sec_histogram1.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/req_sec_histogram2.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/req_sec_histogram3.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
48 changes: 24 additions & 24 deletions results.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,31 @@
| Query | Server | Requests/sec | Latency (ms) | Relative |
|-------:|--------:|--------------:|--------------:|---------:|
| 1 | `{ posts { id userId title user { id name email }}}` |
|| [Tailcall] | `29,496.60` | `3.38` | `196.74x` |
|| [async-graphql] | `1,805.45` | `55.70` | `12.04x` |
|| [Caliban] | `1,547.91` | `64.33` | `10.32x` |
|| [GraphQL JIT] | `1,335.26` | `74.60` | `8.91x` |
|| [Gqlgen] | `772.80` | `128.41` | `5.15x` |
|| [Netflix DGS] | `358.06` | `177.42` | `2.39x` |
|| [Apollo GraphQL] | `268.25` | `366.16` | `1.79x` |
|| [Hasura] | `149.93` | `549.07` | `1.00x` |
|| [Tailcall] | `29,173.60` | `3.41` | `211.81x` |
|| [async-graphql] | `2,019.59` | `49.49` | `14.66x` |
|| [Caliban] | `1,744.80` | `57.28` | `12.67x` |
|| [GraphQL JIT] | `1,342.93` | `74.16` | `9.75x` |
|| [Gqlgen] | `797.87` | `124.41` | `5.79x` |
|| [Netflix DGS] | `366.18` | `181.49` | `2.66x` |
|| [Apollo GraphQL] | `263.39` | `372.49` | `1.91x` |
|| [Hasura] | `137.74` | `587.80` | `1.00x` |
| 2 | `{ posts { title }}` |
|| [Tailcall] | `58,939.60` | `1.69` | `67.77x` |
|| [Caliban] | `9,238.76` | `11.16` | `10.62x` |
|| [async-graphql] | `9,138.76` | `11.34` | `10.51x` |
|| [Gqlgen] | `2,136.19` | `48.48` | `2.46x` |
|| [Apollo GraphQL] | `1,775.87` | `56.26` | `2.04x` |
|| [Netflix DGS] | `1,593.80` | `69.82` | `1.83x` |
|| [GraphQL JIT] | `1,399.68` | `71.35` | `1.61x` |
|| [Hasura] | `869.69` | `114.81` | `1.00x` |
|| [Tailcall] | `58,485.30` | `1.70` | `67.37x` |
|| [Caliban] | `9,751.71` | `10.62` | `11.23x` |
|| [async-graphql] | `9,444.13` | `10.73` | `10.88x` |
|| [Gqlgen] | `2,181.26` | `47.48` | `2.51x` |
|| [Apollo GraphQL] | `1,760.69` | `56.72` | `2.03x` |
|| [Netflix DGS] | `1,598.09` | `70.06` | `1.84x` |
|| [GraphQL JIT] | `1,392.45` | `71.71` | `1.60x` |
|| [Hasura] | `868.17` | `114.99` | `1.00x` |
| 3 | `{ greet }` |
|| [Caliban] | `67,637.00` | `1.07` | `25.83x` |
|| [Tailcall] | `59,557.80` | `1.69` | `22.75x` |
|| [Gqlgen] | `47,525.20` | `5.21` | `18.15x` |
|| [async-graphql] | `47,150.60` | `2.21` | `18.01x` |
|| [Netflix DGS] | `8,261.44` | `15.02` | `3.16x` |
|| [Apollo GraphQL] | `8,186.12` | `12.54` | `3.13x` |
|| [GraphQL JIT] | `5,273.45` | `18.93` | `2.01x` |
|| [Hasura] | `2,618.18` | `38.13` | `1.00x` |
|| [Caliban] | `67,512.70` | `1.07` | `26.49x` |
|| [Tailcall] | `59,015.70` | `1.71` | `23.16x` |
|| [Gqlgen] | `47,378.10` | `5.25` | `18.59x` |
|| [async-graphql] | `47,361.00` | `2.25` | `18.58x` |
|| [Netflix DGS] | `8,297.54` | `14.39` | `3.26x` |
|| [Apollo GraphQL] | `8,128.40` | `12.43` | `3.19x` |
|| [GraphQL JIT] | `5,241.04` | `19.05` | `2.06x` |
|| [Hasura] | `2,548.54` | `39.16` | `1.00x` |

<!-- PERFORMANCE_RESULTS_END -->

0 comments on commit 3fb8dfc

Please sign in to comment.