2. F1 Instances¶
Summary of Steps¶
Unlike simulation, running on the F1 requires a few simple manual steps. These depend on your personal AWS account (EC2 and S3). Specifically, following synthesis Amazon requires the bitstream to be uploaded to your S3 account, and an EC2 account is needed to launch an F1 instance to run the spatial application in hardware.
This tutorial describes the following steps:
- Authenticating your AWS account
- Generating and synthesizing a Spatial design. In our experience, synthesis/place/route takes 4-12 hours depending on design size
- Uploading the bitstream (AKA design checkpoint, or DCP) to Amazon S3 and waiting approximately 1 hour for the Amazon FPGA Image (AFI) associated with this bitstream to become valid
- Opening an F1 instance through your EC2 account
- Running the spatial application
Step 1: Authenticating your AWS account¶
Follow these steps to create a file rootkey.csv
. This file can be placed anywhere, and will be needed for later steps to run commands associated with your AWS account.
Then add the path to that file to your .bashrc as follows:
export AWS_CONFIG_FILE=/path/to/rootkey.csv
Step 2: Generate and Synthesize your application¶
The first step is the same as compiling a Spatial application for any other target, shown here for the MatMult_outer
example:
cd ${SPATIAL_HOME}
bin/spatial MatMult_outer --synth
You can replace MatMult_outer
with any application, as described in the previous tutorial.
Now generate the F1 design and run synthesis/place/route to begin creating the Amazon FPGA Image (AFI):
cd ${SPATIAL_HOME}/gen/MatMult_outer
make aws-F1-afi
We tested this both using Amazon’s FPGA Developer AMI and also locally using their instructions.
Notice that once this command completes, Vivado synthesis will be running in the background. A text file called create_spatial_AFI_instructions.txt
has also been created. Follow the instructions in this text file (also described below) once Vivado completes to upload the Design Checkpoint (DCP) and finish creating the AFI.
Vivado will have completed once the *.vivado.log
file in aws-fpga/hdk/cl/examples/MatMult_outer/build/scripts
has printed Finished creating final tar file in to_aws directory.
. You can also check if the vivado
process has stopped.
Step 3: Creating the AFI¶
See the generated file create_spatial_AFI_instructions.txt
, which guides the user step-by-step on how to upload the DCP to S3 and then run the create-fpga-image
command.
Running these commands will require the AWS Command Line Interface. We tested with version 1.11.78.
Once create-fpga-image
has been run, wait for the logs
directory in S3 to be filled, and ensure that the AFI state in the file called State
is “available”.
Step 4: Opening an F1 instance¶
Start an F1 instance in the AWS console. We tested with Amazon’s FPGA Developer AMI although this might not be necessary.
If you already have an existing F1 instance (e.g. for a previous Spatial application), skip to Step 5. If this is your first time starting the F1 instance, follow the one-time setup steps below.
Clone Amazon’s EC2 FPGA Hardware and Software Development Kit to any location, e.g. /home/centos/src/project_data
:
git clone https://github.com/aws/aws-fpga.git
Put the following (replacing with your chosen path above) in your .bashrc
:
cd /home/centos/src/project_data/aws-fpga/
source /home/centos/src/project_data/aws-fpga/sdk_setup.sh
Source the .bashrc
:
source ~/.bashrc
Then follow these instructions to build and install the required EDMA driver. Summary:
cd sdk/linux_kernel_drivers/edma/
make
echo 'edma' | sudo tee --append /etc/modules-load.d/edma.conf
sudo cp edma-drv.ko /lib/modules/`uname -r`/
sudo depmod
sudo modprobe edma-drv
Step 5: Running the Spatial application¶
Generate the host (software) binary using:
make aws-F1-sw
You can do this on your local machine and copy over the binary to the F1 (this might require changing permissions to run it), or compile the binary on the F1 instance.
To do it on the F1 instance, you only need the software/runtime
and software/include
directories of the generated Spatial AWS application (e.g. aws-fpga/hdk/cl/examples/MatMult_outer
), and can compile
using make all
in software/runtime
.
Also modify the file load.sh
in software/runtime
to paste in the agfi ID returned above. Eventually this will be automated.
Run the application using the commands below in the runtime
directory. Eventually the call to load.sh
will be automated within Top
:
bash load.sh
sudo ./Top arg1 arg2 ...
Notes on the above commands:
- Currently we require a board reset (part of
load.sh
) prior to running an application. Eventually this will not be needed. - Eventually the
agfi
above will be automatically written to a file which the Spatial application reads. For now it is part ofload.sh
.