Repo for the ASPLOS'23 paper Vidi: Record Replay for Reconfigurable Hardware
This is forked from the aws/aws-fpga that contains the implementation of a record/replay framework called Vidi. For artifact-eval, check here.
For portability to platforms other than the AWS F1 platform but still uses AXI interfaces, there is no polished write-up available as of this point. But feel free to email (preferrablely the first author) or post questions on github. Briefly, you need to hack the simulation/synth script used in our setup to alternative IPs in your setup. And change the Vidi wrapper, which currently assumes the standard I/O interfaces on the AWS F1 platform. The skeleton of the Vidi wrapper basically contains:
- CSR control signals in module rr_csrs
- Various AXI interconnect that setup the backend AXI trace storage and AXIL control registers
- Conversion of ad-hoc AXI signals to systemverilog interfaces.
- Wrappers of individual AXI interface to record/replay. The macro
REG_AXI_MSTR_INTF_RRandREG_AXI_MSTR_INTF_RR(defined here) contain example wrappers as instantiations of other Vidi components. - Codegen of some binary merge tree that would balance the signal widths accordign to the width of tracked interfaces.
- example of specifying the tracked interfaces:
hdk/cl/examples/cl_fpgarr/design/scripts/cl_fpgarr_autogrouping.py --help - example of generating the merge tree:
hdk/cl/examples/cl_fpgarr/design/scripts/sweep_mergetree.sh
The following is the original document that describes how the AWS framework works in general.
- Overview of AWS EC2 FPGA Development Kit
- Amazon EC2 F1 platform features
- Getting Started
- Documentation Overview
- Developer Support
AWS EC2 FPGA Development Kit is a set of development and runtime tools to develop, simulate, debug, compile and run hardware accelerated applications on Amazon EC2 F1 instances. It is distributed between this github repository and FPGA Developer AMI - Centos/AL2 provided by AWS with no cost of development tools.
After creating an FPGA design (also called CL - Custom logic), developers can create an Amazon FPGA Image (AFI) and easily deploy it to an F1 instance. AFIs are reusable, shareable and can be deployed in a scalable and secure way.
| Development Environment | Description | Accelerator Language | Hardware Interface | Debug Options | Typical Developer |
|---|---|---|---|---|---|
| Software Defined Accelerator Development using Vitis/SDAccel | Development experience leverages an optimized compiler to allow easy new accelerator development or migration of existing C/C++/openCL, Verilog/VHDL to AWS FPGA instances | C/C++/OpenCL, Verilog/VHDL (RTL) | OpenCL APIs and XRT | SW/HW Emulation, Simulation, GDB, Virtual JTAG (Chipscope) | SW or HW Developer with zero FPGA experience |
| Hardware Accelerator Development using Vivado | Fully custom hardware development experience provides hardware developers with the tools required for developing AFIs for AWS FPGA instances | Verilog/VHDL | XDMA Driver, peek/poke | Simulation, Virtual JTAG | HW Developer with advanced FPGA experience |
| IP Integrator/High Level Design(HLx) using Vivado | Graphical interface development experience for integrating IP and high level synthesis development | Verilog/VHDL/C | XDMA Driver, peek/poke | Simulation, Virtual JTAG | HW Developer with intermediate FPGA experience |
For on-premise development, SDAccel/Vitis/Vivado must have the correct license and use one of the supported tool versions.
The FPGA Developer AMI is available on the AWS marketplace without a software charge and includes tools needed for developing FPGA Designs to run on AWS F1.
Given the large size of the FPGA used inside AWS F1 Instances, Xilinx tools work best with 32GiB Memory. z1d.xlarge/c5.4xlarge and z1d.2xlarge/c5.8xlarge instance types would provide the fastest execution time with 30GiB+ and 60GiB+ of memory respectively. Developers who want to save on cost, could start coding and run simulations on low-cost instances, like t2.2xlarge, and move to the aforementioned larger instances to run the synthesis of their acceleration code.
AWS marketplace offers multiple versions of the FPGA Developer AMI. The following compatibility table describes the mapping of currently supported developer kit versions to AMI versions:
| Developer Kit Version | Tool Version Supported | Compatible FPGA Developer AMI Version |
|---|---|---|
| 1.4.18+ | 2020.2 | v1.10.X (Xilinx Vivado/Vitis 2020.2) |
| 1.4.16+ | 2020.1 | v1.9.0-v1.9.X (Xilinx Vivado/Vitis 2020.1) |
| 1.4.13+ | 2019.2 | v1.8.0-v1.8.X (Xilinx Vivado/Vitis 2019.2) |
| 1.4.11+ | 2019.1 | v1.7.0-v1.7.X (Xilinx Vivado/SDx 2019.1) |
| 1.4.8 - 1.4.15a | 2018.3 | v1.6.0-v1.6.X (Xilinx Vivado/SDx 2018.3) |
| 1.4.3 - 1.4.15a | 2018.2 | v1.5.0-v1.5.X (Xilinx Vivado/SDx 2018.2) |
| 1.3.7 - 1.4.15a | 2017.4 | v1.4.0-v1.4.X (Xilinx Vivado/SDx 2017.4) |
For software-defined development please look at the runtime compatibility table based on the Xilinx toolset in use: SDAccel or Vitis
The HDK directory contains documentation, examples, simulation, build and AFI creation scripts to start building Amazon FPGA Images (AFI).
The HDK can be installed on any on-premises server or an EC2 instance.
The developer kit is not required if you plan to use a pre-built AFI shared from another developer.
With Amazon EC2 FPGA instances, each FPGA is divided into two partitions:
-
Shell (SH) – AWS platform logic implementing the FPGA external peripherals, PCIe, DRAM, and Interrupts.
-
Custom Logic (CL) – Custom acceleration logic created by an FPGA Developer.
At the end of the development process, combining the Shell and CL creates an Amazon FPGA Image (AFI) that can be loaded onto the Amazon EC2 FPGA Instances.
The following table provides the shells currently available to develop your CL with. Each shell provides specific interfaces and features and currently needs to be used with the Dev Kit branch listed in the table.
| Shell Name | Shell Version | Dev Kit Branch | Description |
|---|---|---|---|
| F1 XDMA Shell | F1.X.1.4 | master | Provides all the interfaces listed here, includes DMA |
| F1 Small Shell | F1.S.1.0 | small_shell | Provides all the interfaces listed here. This shell does not include DMA engine and provides significant reduction in Shell resource usage. |
For more details, check the FAQ
The software-defined development environment allows customers to compile their C/C++/OpenCL code into the FPGA as kernels, and use OpenCL APIs to pass data to the FPGA. Software developers with no FPGA experience will find a familiar development experience that supercharges cloud applications.
In addition, this development environment allows for a mix of C/C++ and RTL accelerator designs into a C/C++ software based development environment. This method enables faster prototyping using C/C++ while supporting manual optimization of critical blocks within RTL. This approach is similar to optimizing time critical functions using software compiler optimization methods.
To get started with Xilinx SDAccel, review the Software-defined development environment readme. To get started with Xilinx Vitis, review the Vitis unified development environment readme.
The SDK directory includes the runtime environment required to run on EC2 FPGA instances. It includes the drivers and tools to manage the AFIs that are loaded on the FPGA instance. The SDK isn't required during the AFI development process; it is only required once an AFI is loaded onto an EC2 FPGA instance. The following sdk resources are provided:
- Linux Kernel Drivers - The developer kit includes three drivers:
- XDMA Driver - DMA interface to/from HDK accelerators.
- FPGA Libraries - APIs used by C/C++ host applications.
- FPGA Management Tools - AFI management APIs for runtime loading/clearing FPGA image, gathering metrics and debug interface on the F1 instance.
- 1-8 Xilinx UltraScale+ VU9P based FPGA slots
- Per FPGA Slot, Interfaces available for Custom Logic(CL):
- One x16 PCIe Gen 3 Interface
- Four DDR4 RDIMM interfaces (with ECC)
- AXI4 protocol support on all interfaces
- User-defined clock frequency driving all CL to Shell interfaces
- Multiple free running auxiliary clocks
- PCI-E endpoint presentation to Custom Logic(CL)
- Management PF (physical function)
- Application PF
- Virtual JTAG, Virtual LED, Virtual DIP Switches
- PCI-E interface between Shell(SH) and Custom Logic(CL).
- SH to CL inbound 512-bit AXI4 interface
- CL to SH outbound 512-bit AXI4 interface
- Multiple 32-bit AXI-Lite buses for register access, mapped to different PCIe BARs
- Maximum payload size set by the Shell
- Maximum read request size set by the Shell
- AXI4 error handling
- DDR interface between SH and CL
- CL to SH 512-bit AXI4 interface
- 1 DDR controller implemented in the SH (always available)
- 3 DDR controllers implemented in the CL (configurable number of implemented controllers allowed)
If you have never used AWS before, we recommend you start with AWS getting started training, and focus on the basics of the AWS EC2 and AWS S3 services. Understanding the fundamentals of these services will make it easier to work with AWS F1 and the FPGA Developer Kit.
FPGA Image generation and EC2 F1 instances are supported in the us-east-1 (N. Virginia), us-west-2 (Oregon), eu-west-1 (Ireland) and us-gov-west-1 (GovCloud US) regions.
⚠️ NOTE: By default, your AWS Account will have an EC2 F1 Instance launch limit of 0. Before using F1 instances, you will have to open a Support Case to increase the EC2 Instance limits to allow launching F1 instances.
You have the choice to develop on AWS EC2 using the FPGA Developer AMI or on-premise.
ℹ️ INFO: We suggest starting with the FPGA Developer AMI with build instances on EC2 as it has Xilinx tools and licenses setup for you to be able to quickly get into development.
ℹ️ INFO: For on-premise development, you will need to have Xilinx tools and licenses available for you to use
-
Start a Build Instance first to start your development.
💡 TIP: This instance does not have to be an F1 instance. You only require an F1 instance to run your AFI's(Amazon FPGA Image) once you have gone through your design build and AFI creation steps.
ℹ️ INFO: If you need to follow GUI Development flows, please checkout our Developer Resources where we provide Step-By-Step guides to setting up a GUI Desktop.
-
Clone the FPGA Developer Kit on your instance.
git clone https://github.com/aws/aws-fpga.git -
Follow the quickstarts from the next section.
Before you create your own AWS FPGA design, we recommend that you go through one of the step-by-step Quickstart guides:
| Description | Quickstart | Next Steps |
|---|---|---|
| Software Defined Accelerator Development using Xilinx Vitis | Vitis hello_world Quickstart | 60+ Vitis examples, Vitis Library Examples |
| Software Defined Accelerator Development using Xilinx SDAccel | SDAccel hello_world Quickstart | 60+ SDAccel examples |
| Custom Hardware Development(HDK) | HDK hello_world Quickstart | CL to Shell and DRAM connectivity example, Virtual Ethernet Application using the Streaming Data Engine |
| IP Integrator/High Level Design(HLx) | IPI hello_world Quickstart | IPI GUI Examples |
ℹ️ INFO: For more in-depth applications and examples of using High level synthesis, Vitis Libraries, App Notes and Workshops, please refer to our Example List
| How To | Description |
|---|---|
| Migrate Alveo U200 designs to F1 | This application note shows the ease of migrating an Alveo U200 design to F1. |
Documentation is located throughout this developer kit and the table below consolidates a list of key documents to help developers find information:
| Topic | Document Name | Description |
|---|---|---|
| AWS setup | Setup AWS CLI and S3 Bucket | Setup instructions for preparing for AFI creation |
| Developer Kit | RELEASE NOTES, Errata | Release notes and Errata for all developer kit features, excluding the shell |
| Developer Kit | Errata | Errata for all developer kit features, excluding the shell |
| F1 Shell | AWS Shell RELEASE NOTES | Release notes for F1 shell |
| F1 Shell | AWS Shell ERRATA | Errata for F1 shell |
| F1 Shell | AWS Shell Interface Specification | Shell-CL interface specification for HDK developers building AFI |
| F1 Shell - Timeout and AXI Protocol Protection | How to detect a shell timeout | The shell will terminate transactions after a time period or on an illegal transaction. This describes how to detect and gather data to help debug CL issues caused by timeouts. |
| Vitis | Debug Vitis Kernel | Instructions on debugging Vitis Kernel |
| Vitis | Create Runtime AMI | Instructions on creating a runtime AMI when using Xilinx Vitis |
| Vitis | XRT Instructions | Instructions on building, installing XRT with MPD daemon considerations for F1 |
| SDAccel | Debug RTL Kernel | Instructions on debugging RTL Kernel with SDAccel |
| SDAccel | Create Runtime AMI | Instructions on creating a runtime AMI when using Xilinx SDAccel |
| HDK - Host Application | Programmer View | Host application to CL interface specification |
| HDK - CL Debug | Debug using Virtual JTAG | Debugging CL using Virtual JTAG (Chipscope) |
| HDK - Simulation | Simulating CL Designs | Shell-CL simulation specification |
| HDK - Driver | README | Describes the DMA driver (XDMA) used by HDK examples and includes a link to an installation guide |
| AFI | AFI Management SDK | CLI documentation for managing AFI on the F1 instance |
| AFI - EC2 CLI | copy_fpga_image, delete_fpga_image, describe_fpga_images, fpga_image_attributes | CLI documentation for administering AFIs |
| AFI - Creation Error Codes | create_fpga_image_error_codes | CLI documentation for managing AFIs |
| AFI - Power | FPGA Power, recovering from clock gating | Helps developers with understanding FPGA power usage, preventing power violations on the F1 instance and recovering from a clock gated slot. |
| On-premise Development | Tools, Licenses required for on-premise development | Guidance for developer wanting to develop AFIs from on-premises instead of using the FPGA Developer AMI |
| Frequently asked questions | FAQ | Q/A are added based on developer feedback and common AWS forum questions |
-
The Amazon FPGA Development User Forum is the first place to go to post questions, learn from other users and read announcements.
- We recommend joining the AWS forums to engage with the FPGA developer community, AWS and Xilinx engineers to get help.
-
You could also file a Github Issue for support. We prefer the forums as this helps the entire community learn from issues, feedback and answers.
- Click the "Watch" button in GitHub upper right corner to get regular updates.