This repository has been archived by the owner on Oct 11, 2024. It is now read-only.
andy-neuma triggered nightly on refs/heads/main #69
andy-neumanightly.yml
on: schedule
AWS-AVX2-32G-A10G-24G-Benchmark
/
BENCHMARK
7h 21m
NIGHTLY-MULTI
/
...
/
BUILD
24m 30s
NIGHTLY-SOLO
/
...
/
BUILD
45m 6s
Accuracy-Smoke-AWS-AVX2-32G-A10G-24G
/
LM-EVAL-SMOKE
1h 45m
AWS-AVX2-32G-A10G-24G-Benchmark
/
NM_GH_ACTION_BENCHMARK
21s
Matrix: NIGHTLY-MULTI / TEST
Matrix: NIGHTLY-SOLO / TEST
Annotations
4 errors and 3 warnings
|
NIGHTLY-MULTI / TEST (aws-avx2-192G-4-a10g-96G) / TEST
Process completed with exit code 1.
|
|
NIGHTLY-SOLO / TEST (aws-avx2-192G-4-a10g-96G) / TEST
Process completed with exit code 1.
|
|
NIGHTLY-SOLO / TEST (aws-avx2-192G-4-a10g-96G) / TEST
Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run
|
|
AWS-AVX2-32G-A10G-24G-Benchmark / NM_GH_ACTION_BENCHMARK
# :warning: **Performance Alert** :warning:
Possible performance regression was detected for benchmark **'smaller_is_better'**.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold `1.10`.
| Benchmark suite | Current: 788b4e526d379aa6b910cb1932a756d17cbcc997 | Previous: dcd4973217a1e9ddf1d45145ee7814bb3073b525 | Ratio |
|-|-|-|-|
| `{"name": "median_ttft_ms", "description": "VLLM Serving - Dense\nmodel - teknium/OpenHermes-2.5-Mistral-7B\nmax-model-len - 4096\nsparsity - None\nbenchmark_serving {\n \"nr-qps-pair_\": \"300,1\",\n \"dataset\": \"sharegpt\"\n}", "gpu_description": "NVIDIA A10G x 1", "vllm_version": "0.2.0", "python_version": "3.10.12 (main, Mar 7 2024, 18:39:53) [GCC 9.4.0]", "torch_version": "2.2.1+cu121"}` | `94.45406099985121` ms | `84.74403700029143` ms | `1.11` |
This comment was automatically generated by [workflow](https://github.com/neuralmagic/nm-vllm/actions?query=workflow%3ANightly) using [github-action-benchmark](https://github.com/marketplace/actions/continuous-benchmark).
Comment was generated at https://github.com/neuralmagic/nm-vllm/commit/788b4e526d379aa6b910cb1932a756d17cbcc997#commitcomment-140944844
|
|
NIGHTLY-MULTI / TEST (aws-avx2-192G-4-a10g-96G) / TEST
This job failure may be caused by using an out of date self-hosted runner. You are currently using runner version 2.314.1. Please update to the latest version 2.315.0
|
|
NIGHTLY-SOLO / TEST (aws-avx2-192G-4-a10g-96G) / TEST
This job failure may be caused by using an out of date self-hosted runner. You are currently using runner version 2.314.1. Please update to the latest version 2.315.0
|
|
AWS-AVX2-32G-A10G-24G-Benchmark / NM_GH_ACTION_BENCHMARK
Performance alert! Previous value was 84.74403700029143 and current value is 94.45406099985121. It is 1.1145806164453362x worse than previous exceeding a ratio threshold 1.1
|
Artifacts
Produced during runtime
| Name | Size | |
|---|---|---|
|
3.10.12-nm-vllm-0.2.0.tar.gz
Expired
|
465 KB |
|
|
3.11.4-nm-vllm-0.2.0.tar.gz
Expired
|
465 KB |
|
|
8677578660-aws-avx2-32G-a10g-24G
Expired
|
124 KB |
|
|
cc-vllm-html-aws-avx2-192G-4-a10g-96G
Expired
|
1.7 MB |
|
|
gh_action_benchmark_jsons-8677578660-aws-avx2-32G-a10g-24G
Expired
|
28.8 KB |
|
|
nm_vllm-0.2.0-cp310-cp310-manylinux_2_17_x86_64.whl
Expired
|
88.1 MB |
|
|
nm_vllm-0.2.0-cp311-cp311-manylinux_2_17_x86_64.whl
Expired
|
88.1 MB |
|