2. F1 Instances

Summary of Steps

Unlike simulation, running on the F1 requires a few simple manual steps. These depend on your personal AWS account (EC2 and S3). Specifically, following synthesis Amazon requires the bitstream to be uploaded to your S3 account, and an EC2 account is needed to launch an F1 instance to run the spatial application in hardware.

This tutorial describes the following steps:

  • Authenticating your AWS account
  • Generating and synthesizing a Spatial design. In our experience, synthesis/place/route takes 4-12 hours depending on design size
  • Uploading the bitstream (AKA design checkpoint, or DCP) to Amazon S3 and waiting approximately 1 hour for the Amazon FPGA Image (AFI) associated with this bitstream to become valid
  • Opening an F1 instance through your EC2 account
  • Running the spatial application

Step 1: Authenticating your AWS account

Follow these steps to create a file rootkey.csv. This file can be placed anywhere, and will be needed for later steps to run commands associated with your AWS account.

Then add the path to that file to your .bashrc as follows:

export AWS_CONFIG_FILE=/path/to/rootkey.csv

Step 2: Generate and Synthesize your application

The first step is the same as compiling a Spatial application for any other target, shown here for the MatMult_outer example:

cd ${SPATIAL_HOME}
bin/spatial MatMult_outer --synth

You can replace MatMult_outer with any application, as described in the previous tutorial.

Now generate the F1 design and run synthesis/place/route to begin creating the Amazon FPGA Image (AFI):

cd ${SPATIAL_HOME}/gen/MatMult_outer
make aws-F1-afi

We tested this both using Amazon’s FPGA Developer AMI and also locally using their instructions.

Notice that once this command completes, Vivado synthesis will be running in the background. A text file called create_spatial_AFI_instructions.txt has also been created. Follow the instructions in this text file (also described below) once Vivado completes to upload the Design Checkpoint (DCP) and finish creating the AFI.

Vivado will have completed once the *.vivado.log file in aws-fpga/hdk/cl/examples/MatMult_outer/build/scripts has printed Finished creating final tar file in to_aws directory.. You can also check if the vivado process has stopped.

Step 3: Creating the AFI

See the generated file create_spatial_AFI_instructions.txt, which guides the user step-by-step on how to upload the DCP to S3 and then run the create-fpga-image command.

Running these commands will require the AWS Command Line Interface. We tested with version 1.11.78.

Once create-fpga-image has been run, wait for the logs directory in S3 to be filled, and ensure that the AFI state in the file called State is “available”.

Step 4: Opening an F1 instance

Start an F1 instance in the AWS console. We tested with Amazon’s FPGA Developer AMI although this might not be necessary.

If you already have an existing F1 instance (e.g. for a previous Spatial application), skip to Step 5. If this is your first time starting the F1 instance, follow the one-time setup steps below.

Clone Amazon’s EC2 FPGA Hardware and Software Development Kit to any location, e.g. /home/centos/src/project_data:

git clone https://github.com/aws/aws-fpga.git

Put the following (replacing with your chosen path above) in your .bashrc:

cd /home/centos/src/project_data/aws-fpga/
source /home/centos/src/project_data/aws-fpga/sdk_setup.sh

Source the .bashrc:

source ~/.bashrc

Then follow these instructions to build and install the required EDMA driver. Summary:

cd sdk/linux_kernel_drivers/edma/
make
echo 'edma' | sudo tee --append /etc/modules-load.d/edma.conf
sudo cp edma-drv.ko /lib/modules/`uname -r`/
sudo depmod
sudo modprobe edma-drv

Step 5: Running the Spatial application

Generate the host (software) binary using:

make aws-F1-sw

You can do this on your local machine and copy over the binary to the F1 (this might require changing permissions to run it), or compile the binary on the F1 instance. To do it on the F1 instance, you only need the software/runtime and software/include directories of the generated Spatial AWS application (e.g. aws-fpga/hdk/cl/examples/MatMult_outer), and can compile using make all in software/runtime.

Also modify the file load.sh in software/runtime to paste in the agfi ID returned above. Eventually this will be automated.

Run the application using the commands below in the runtime directory. Eventually the call to load.sh will be automated within Top:

bash load.sh
sudo ./Top arg1 arg2 ...

Notes on the above commands:

  • Currently we require a board reset (part of load.sh) prior to running an application. Eventually this will not be needed.
  • Eventually the agfi above will be automatically written to a file which the Spatial application reads. For now it is part of load.sh.