Skip to content

Commit

Permalink
Merge branch 'main' into yutji/directx-deferredrender
Browse files Browse the repository at this point in the history
  • Loading branch information
yukirora authored Aug 14, 2023
2 parents 2bd047f + 67f2aa7 commit 82b2987
Show file tree
Hide file tree
Showing 34 changed files with 562 additions and 77 deletions.
11 changes: 11 additions & 0 deletions .github/workflows/build-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,17 @@ jobs:
sudo apt-get clean
sudo docker rmi $(sudo docker images --format "{{.Repository}}:{{.Tag}}" --filter=reference="node" --filter=reference="buildpack-deps")
df -h
- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
uses: jlumbroso/free-disk-space@main
with:
# This might remove tools that are actually needed, if set to "true" but frees about 6 GB
tool-cache: false
# All of these default to true, but feel free to set to "false" if necessary for your workflow
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: Prepare metadata
id: metadata
run: |
Expand Down
29 changes: 25 additions & 4 deletions .github/workflows/build-win.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ on:

jobs:
docker:
name: Docker build win2004
name: Docker build win directx12
runs-on: [self-hosted, windows, x64, win2004]
steps:
- name: Checkout
Expand All @@ -24,6 +24,25 @@ jobs:
docker system prune -a -f
docker volume prune -a -f
shell: pwsh
- name: Set TAG variable based on the branch
run: |
if ($env:GITHUB_EVENT_NAME -match "release") {
$version = $env:GITHUB_REF.Substring($env:GITHUB_REF.LastIndexOf('/') + 1)
echo "TAG=superbench/superbench:$version-directx12" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
} elseif ($env:GITHUB_REF -match "refs/heads/release/(.*)") {
$version = $Matches[1]
echo "TAG=superbench/release:$version-directx12" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
} elseif ($env:GITHUB_BASEREF -match "release/(.*)"){
$version = $Matches[1]
echo "TAG=superbench/release:$version-directx12" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
} else {
echo "TAG=superbench/main:directx12" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
}
shell: pwsh
env:
GITHUB_REF: ${{ github.ref }}
GITHUB_BASEREF: ${{ github.base_ref }}
GITHUB_EVENT_NAME: ${{ github.event_name }}
- name: Build Docker image
working-directory: .
shell: pwsh
Expand All @@ -37,7 +56,7 @@ jobs:
--isolation=process `
--tag $env:TAG .
env:
TAG: superbench/main:win2004
TAG: ${{ env.TAG }}
- name: Push Docker image
if: ${{ github.event_name != 'pull_request' }}
shell: pwsh
Expand All @@ -46,7 +65,7 @@ jobs:
docker push $env:TAG
docker logout
env:
TAG: superbench/main:win2004
TAG: ${{ env.TAG }}
USER: ${{ secrets.DOCKERHUB_USERNAME }}
PASS: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Add bash to PATH
Expand All @@ -64,7 +83,9 @@ jobs:
docker run --rm `
--isolation process `
--device class/5B45201D-F2F2-4F3B-85BB-30FF1F953599 `
-e CI=true $ci_env -e SB_TEST_CUDA="0" -e SB_TEST_ROCM="0" -e SB_TEST_PYTORCH="0" -e SB_TEST_DIRECTX="1" -e CODECOV_TOKEN superbench/main:win2004 cmd /c $command
-v C:/Windows/System32/DriverStore:C:/Windows/System32/DriverStore `
-e CI=true $ci_env -e SB_TEST_CUDA="0" -e SB_TEST_ROCM="0" -e SB_TEST_PYTORCH="0" -e SB_TEST_DIRECTX="1" -e CODECOV_TOKEN --entrypoint "cmd" $env:TAG "/c python dockerfile/directx/enable-graphics-apis.py && cmd /c $command"
shell: pwsh
env:
TAG: ${{ env.TAG }}
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

__SuperBench__ is a validation and profiling tool for AI infrastructure.

📢 [v0.8.0](https://github.com/microsoft/superbenchmark/releases/tag/v0.8.0) has been released!
📢 [v0.9.0](https://github.com/microsoft/superbenchmark/releases/tag/v0.9.0) has been released!

## _Check [aka.ms/superbench](https://aka.ms/superbench) for more details._

Expand Down
4 changes: 2 additions & 2 deletions dockerfile/directx12.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -64,5 +64,5 @@ RUN make -C third_party directx_amd

# Run the entrypoint script for enabling vendor-specific graphics APIs
RUN powershell -Command "Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine -Force"
CMD [ "python", "dockerfile/directx/enable-graphics-apis.py" ]
ENTRYPOINT [ "cmd.exe" ]
ENTRYPOINT [ "python", "dockerfile/directx/enable-graphics-apis.py" ]
CMD [ "cmd.exe" ]
2 changes: 1 addition & 1 deletion docs/getting-started/installation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ You can clone the source from GitHub and build it.
:::note Note
You should checkout corresponding tag to use release version, for example,

`git clone -b v0.8.0 https://github.com/microsoft/superbenchmark`
`git clone -b v0.9.0 https://github.com/microsoft/superbenchmark`
:::

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/run-superbench.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ sb deploy -f remote.ini --host-password [password]
:::note Note
You should deploy corresponding Docker image to use release version, for example,

`sb deploy -f local.ini -i superbench/superbench:v0.8.0-cuda12.1`
`sb deploy -f local.ini -i superbench/superbench:v0.9.0-cuda12.1`

You should note that version of git repo only determines version of sb CLI, and not the sb container. You should define the container version even if you specified a release version for the git clone.

Expand Down
2 changes: 1 addition & 1 deletion docs/superbench-config.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ superbench:
<TabItem value='example'>
```yaml
version: v0.8
version: v0.9
superbench:
enable: benchmark_1
monitor:
Expand Down
14 changes: 14 additions & 0 deletions docs/user-tutorial/container-images.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,15 @@ available tags are listed below for all stable versions.
values={[
{label: 'CUDA', value: 'cuda'},
{label: 'ROCm', value: 'rocm'},
{label: 'DirectX', value: 'directx'},
]
}>
<TabItem value='cuda'>

| Tag | Description |
|-------------------|------------------------------------|
| v0.9.0-cuda12.1 | SuperBench v0.9.0 with CUDA 12.1 |
| v0.9.0-cuda11.1.1 | SuperBench v0.9.0 with CUDA 11.1.1 |
| v0.8.0-cuda12.1 | SuperBench v0.8.0 with CUDA 12.1 |
| v0.8.0-cuda11.1.1 | SuperBench v0.8.0 with CUDA 11.1.1 |
| v0.7.0-cuda11.8 | SuperBench v0.7.0 with CUDA 11.8 |
Expand All @@ -45,6 +48,10 @@ available tags are listed below for all stable versions.

| Tag | Description |
|-------------------------------|--------------------------------------------------|
| v0.9.0-rocm5.1.3 | SuperBench v0.9.0 with ROCm 5.1.3 |
| v0.9.0-rocm5.1.1 | SuperBench v0.9.0 with ROCm 5.1.1 |
| v0.9.0-rocm5.0.1 | SuperBench v0.9.0 with ROCm 5.0.1 |
| v0.9.0-rocm5.0 | SuperBench v0.9.0 with ROCm 5.0 |
| v0.8.0-rocm5.1.3 | SuperBench v0.8.0 with ROCm 5.1.3 |
| v0.8.0-rocm5.1.1 | SuperBench v0.8.0 with ROCm 5.1.1 |
| v0.8.0-rocm5.0.1 | SuperBench v0.8.0 with ROCm 5.0.1 |
Expand All @@ -66,5 +73,12 @@ available tags are listed below for all stable versions.
| v0.3.0-rocm4.2-pytorch1.7.0 | SuperBench v0.3.0 with ROCm 4.2, PyTorch 1.7.0 |
| v0.3.0-rocm4.0-pytorch1.7.0 | SuperBench v0.3.0 with ROCm 4.0, PyTorch 1.7.0 |

</TabItem>
<TabItem value='directx'>

| Tag | Description |
|-------------------------------|--------------------------------------------------|
| v0.9.0-directx12 | SuperBench v0.9.0 with DirectX12, Windows10-2004 |

</TabItem>
</Tabs>
2 changes: 1 addition & 1 deletion docs/user-tutorial/data-diagnosis.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ superbench:
example:
```yaml
# SuperBench rules
version: v0.8
version: v0.9
superbench:
rules:
failure-rule:
Expand Down
2 changes: 1 addition & 1 deletion docs/user-tutorial/result-summary.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ superbench:
```yaml title="Example"
# SuperBench rules
version: v0.8
version: v0.9
superbench:
rules:
kernel_launch:
Expand Down
2 changes: 1 addition & 1 deletion superbench/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
Provide hardware and software benchmarks for AI systems.
"""

__version__ = '0.8.0'
__version__ = '0.9.0'
__author__ = 'Microsoft'
4 changes: 4 additions & 0 deletions superbench/benchmarks/micro_benchmarks/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@
from superbench.benchmarks.micro_benchmarks.sharding_matmul import ShardingMatmul
from superbench.benchmarks.micro_benchmarks.tcp_connectivity import TCPConnectivityBenchmark
from superbench.benchmarks.micro_benchmarks.tensorrt_inference_performance import TensorRTInferenceBenchmark
from superbench.benchmarks.micro_benchmarks.directx_gpu_encoding_latency import DirectXGPUEncodingLatency
from superbench.benchmarks.micro_benchmarks.directx_gpu_copy_performance import DirectXGPUCopyBw
from superbench.benchmarks.micro_benchmarks.directx_mem_bw_performance import DirectXGPUMemBw
from superbench.benchmarks.micro_benchmarks.directx_gemm_flops_performance import DirectXGPUCoreFlops

Expand Down Expand Up @@ -63,6 +65,8 @@
'ShardingMatmul',
'TCPConnectivityBenchmark',
'TensorRTInferenceBenchmark',
'DirectXGPUEncodingLatency',
'DirectXGPUCopyBw',
'DirectXGPUMemBw',
'DirectXGPUCoreFlops',
]
132 changes: 132 additions & 0 deletions superbench/benchmarks/micro_benchmarks/directx_gpu_copy_performance.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

"""Module of the DirectXGPUCopyBw performance benchmarks."""

import os
from superbench.common.utils import logger
from superbench.benchmarks import BenchmarkRegistry, Platform
from superbench.benchmarks.micro_benchmarks import MemBwBenchmark


class DirectXGPUCopyBw(MemBwBenchmark):
"""The GPUCopyBw benchmark class."""
def __init__(self, name, parameters=''):
"""Constructor.
Args:
name (str): benchmark name.
parameters (str): benchmark parameters.
"""
super().__init__(name, parameters)
self._mem_types = ['htod', 'dtoh']
self._bin_name = 'DirectXGPUCopyBw.exe'

def add_parser_arguments(self):
"""Add the specified arguments."""
super().add_parser_arguments()

self._parser.add_argument(
'--size',
type=int,
required=False,
default=None,
help='Size of data for GPU copy.',
)
self._parser.add_argument(
'--warm_up',
type=int,
required=False,
default=20,
help='Number of warm up copy times to run.',
)
self._parser.add_argument(
'--num_loops',
type=int,
required=False,
default=1000,
help='Number of copy times to run.',
)
self._parser.add_argument(
'--minbytes',
type=int,
required=False,
default=64,
help='Run size from min_size to max_size for GPU copy.',
)
self._parser.add_argument(
'--maxbytes',
type=int,
required=False,
default=8 * 1024 * 1024,
help='Run size from min_size to max_size for GPU copy.',
)
self._parser.add_argument(
'--check',
action='store_true',
help='Whether check data after copy.',
)

def _preprocess(self):
"""Preprocess/preparation operations before the benchmarking.
Return:
True if _preprocess() succeed.
"""
if not super()._preprocess():
return False

for mem_type in self._args.mem_type:
# Prepare the command line.
command = os.path.join(self._args.bin_dir, self._bin_name)
command += f' --{mem_type}'
command += ' --warm_up ' + str(self._args.warm_up)
command += ' --num_loops ' + str(self._args.num_loops)
if self._args.size is not None:
command += ' --size ' + str(self._args.size)
else:
command += ' --minbytes ' + str(self._args.minbytes)
command += ' --maxbytes ' + str(self._args.maxbytes)
if self._args.check:
command += ' --check'
self._commands.append(command)
return True

def _process_raw_result(self, cmd_idx, raw_output):
"""Function to process raw results and save the summarized results.
Args:
cmd_idx (int): the index of command corresponding with the raw_output.
raw_output (str): raw output string of the micro-benchmark.
Return:
True if the raw output string is valid and result can be extracted.
"""
self._result.add_raw_data('raw_output', raw_output, self._args.log_raw_data)

try:
lines = raw_output.splitlines()
for line in lines:
if 'GB' in line:
type = line.split()[0].strip(':')
size = int(line.strip().split()[1].strip('B'))
bw = float(line.strip().split()[2])
self._result.add_result(f'{type}_{size}_bw', bw)
if 'error' in line.lower():
logger.error(
'The result format is invalid - round: {}, benchmark: {}, raw output: {}.'.format(
self._curr_run_index, self._name, raw_output
)
)
return False
return True
except Exception as e:
logger.error(
'The result format is invalid - round: {}, benchmark: {}, raw output: {}, exception: {}.'.format(
self._curr_run_index, self._name, raw_output, str(e)
)
)
return False


BenchmarkRegistry.register_benchmark('directx-gpu-copy-bw', DirectXGPUCopyBw, platform=Platform.DIRECTX)
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,14 @@
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<TargetName>DirectXGPUCopyBw</TargetName>
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v143</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<TargetName>DirectXGPUCopyBw</TargetName>
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v143</PlatformToolset>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ int main(int argc, char *argv[]) {
} else {
// Run all sizes
for (SIZE_T usize = option.min_size; usize <= option.max_size; usize += usize) {
option.size = usize;
GPUCopyBw benchmark(&option);
benchmark.Run();
}
Expand Down
Loading

0 comments on commit 82b2987

Please sign in to comment.