This repository provides the official implementation of FlashAttention from the following paper.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
Paper: https://arxiv.org/abs/2205.14135
IEEE Spectrum article about our submission to the MLPerf 2.0 benchmark using FlashAttention.
We've been very happy to see FlashAttention being widely adopted in such a short time after its release. This page contains a partial list of places where FlashAttention is being used.
Phil Tillet (OpenAI) has an experimental implementation of FlashAttention in Triton: https://github.com/openai/triton/blob/master/python/tutorials/06-fused-attention.py
As Triton is a higher-level language than CUDA, it might be easier to understand and experiment with. The notations in the Triton implementation are also closer to what's used in our paper.
To install (requiring CUDA 11, NVCC, and an Turing or Ampere GPU):
pip install flash-attnAlternatively you can compile from source:
python setup.py install
Interface: src/flash_attention.py
To run the benchmark against PyTorch standard attention:
PYTHONPATH=$PWD python benchmarks/benchmark_flash_attention.py
FlashAttention currently supports:
- Turing or Ampere GPUs (e.g., A100, RTX 3090, T4, RTX 2080).
- fp16 and bf16 (bf16 requires Ampere GPUs).
- Head dimensions that are multiples of 8, up to 128 (e.g., 8, 16, 24, ..., 128). Head dim > 64 backward requires A100.
Our tentative roadmap:
[Jun 2022] Make package pip-installable[Done, thanks to lucidrains].[Jun 2022] Support SM86 GPUs (e.g., RTX 3080, 3090)[Done].- [Jun 2022] Refactor to use Cutlass.
[Jun 2022] Support SM75 GPUs (e.g. T4)[Done].[Jun 2022] Support bf16[Done].[Jul 2022] Implement cross-attention[Done].[Jul 2022] Support head dimension 128[Done].- [Jul 2022] Support SM70 GPUs (V100).
[Aug 2022] Fuse rotary embedding[Done].- [Aug 2022] Support attention bias (e.g. ALiBi, relative positional encoding).
We present expected speedup (combined forward + backward pass) and memory savings from using FlashAttention against PyTorch standard attention, depending on sequence length, on different GPUs (speedup depends on memory bandwidth - we see more speedup on slower GPU memory).
We currently have benchmarks for these GPUs:
We display FlashAttention speedup using these parameters (similar to BERT-base):
- Batch size 8
- Head dimension 64
- 12 attention heads
Our graphs show sequence lengths between 128 and 4096 (when standard attention runs out of memory on an A100), but FlashAttention can scale up to sequence length 64K.
We generally see 2-4X speedup at sequence lengths between 128 and 4K, and we see more speedup when using dropout and masking, since we fuse the kernels. At sequence lengths that are popular with language models like 512 and 1K, we see speedups up to 4X when using dropout and masking.
We show memory savings in this graph (note that memory footprint is the same no matter if you use dropout or masking). Memory savings are proportional to sequence length -- since standard attention has memory quadratic in sequence length, whereas FlashAttention has memory linear in sequence length. We see 10X memory savings at sequence length 2K, and 20X at 4K. As a result, FlashAttention can scale to much longer sequence lengths.
We show speedup with head dimension 128. Here we show batch size 16 with 12 heads. Speedup is less than with the smaller head sizes, since we have to make the block size smaller in the tiling. But speedup is still significant, especially with a causal mask.
For the RTX 3090, we use batch size 12 with 12 attention heads. Memory savings are the same as on an A100, so we'll only show speedup here.
We see slightly higher speedups (between 2.5-4.5x) on the GTX 3090, since memory bandwidth on the GDDR6X is lower than A100 HBM (~900 GB/s vs. ~1.5 TB/s).
We again use batch size 12 with 12 attention heads.
T4 SRAM is smaller than the newer GPUs (64 KB), so we see less speedup (we need to make the block sizes smaller, so we end up doing more R/W). This matches the IO complexity analysis from section 3.2 of our paper.
T4 GPUs are commonly used for inference, so we also measure speedup on the forward pass only (note that these are not directly comparable to the graphs above):
We see speedups between 2.5x-4.5x on the forward pass.
We test that FlashAttention produces the same output and gradient as a reference implementation, up to some numerical tolerance. In particular, we check that the maximum numerical error of FlashAttention is at most twice the numerical error of a baseline implementation in Pytorch (for different head dimensions, input dtype, sequence length, causal / non-causal).
To run the tests:
pytest -q -s tests/test_flash_attn.py
To install (requiring ROCm, and MI210 or MI250 GPU): You can compile from source:
Launch docker rocm/pytorch:rocm5.4.2_ubuntu20.04_py3.8_pytorch_2.0.0_preview
Enter flash_attention
$patch /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/utils/hipify/hipify_python.py hipify_patch.patch
$python setup.py install
Launch docker rocm/pytorch:rocm5.4_ubuntu20.04_py3.8_pytorch_1.12.1
or any pytorch 1.13.1 docker
Enter flash_attention
$patch /opt/conda/lib/python3.8/site-packages/torch/utils/hipify/hipify_python.py hipify_patch_1.12.patch
$python setup.py install
Alternatively you can build the whole docker image with flash attention automatically.
docker build . -f Dockerfile.rocm -t [IMAGE NAME you like]
Run the container using the following command:
docker run -it --network host --ipc host --device /dev/dri --device /dev/kfd --cap-add SYS_PTRACE --group-add video --security-opt seccomp=unconfined [IMAGE NAME you like]
Flash-attention in the dockerfile will have the best performance automatically. To run the benchmark against PyTorch standard attention:
PYTHONPATH=$PWD python benchmarks/benchmark_flash_attention.py
Benchmark results(MI250, deterministic off, unit test mode off, RTZ):
PYTHONPATH=$PWD python benchmarks/benchmark_flash_attention.py
FlashAttention - Forward pass
8.23 ms
1 measurement, 30 runs , 128 threads
FlashAttention - Backward pass
29.06 ms
1 measurement, 30 runs , 128 threads
FlashAttention - Forward + Backward pass
37.88 ms
1 measurement, 30 runs , 128 threads
PyTorch Standard Attention - Forward pass
26.28 ms
1 measurement, 30 runs , 128 threads
PyTorch Standard Attention - Backward pass
63.20 ms
1 measurement, 30 runs , 128 threads
PyTorch Standard Attention - Forward + Backward pass
89.37 ms
1 measurement, 30 runs , 128 threads
For passing unit tests compile flash-attention from source which may take a while:
FLASH_ATTENTION_INTERNAL_USE_RTZ=0 python setup.py install
Before running unit tests, the unit test mode and deterministic flags should be both turned on by setting the environment variables:
export FLASH_ATTENTION_INTERNAL_DETERMINISTIC=1
export FLASH_ATTENTION_INTERNAL_UNIT_TEST_MODE=1Run the unit tests:
pytest tests/test_flash_attn.pyUnit tests results(MI250, deterministic on, unit test mode on, RTN):
2113 passed, 2848 skipped in 128.24s (0:02:08)
FlashAttention currently supports:
- MI200 GPUs (MI210, MI250).
- fp16 and bf16.
- Head dimensions that are multiples of 8, up to 128 (e.g., 8, 16, 24, ..., 128).
This alpha release of FlashAttention contains code written for a research project to validate ideas on speeding up attention. We have tested it on several models (BERT, GPT2, ViT). However, there might still be bugs in the implementation that we hope to iron out in the next few months.
If you encounter any of these bugs, please open a respective GitHub Issue!
Our implementation uses Apex's FMHA code as a starting point.
We thank Young-Jun Ko for the in-depth explanation of his FMHA implementation and for his thoughtful answers to our questions about CUDA.
If you use this codebase, or otherwise found our work valuable, please cite:
@inproceedings{dao2022flashattention,
title={FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness},
author={Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{\'e}, Christopher},
booktitle={Advances in Neural Information Processing Systems},
year={2022}
}