Chapter 9. Running and Debugging Your Application

This chapter describes how to run and debug your application and covers the following topics:

Loading the Bitstream

The Device Manager (devmgr) maintains a bitstream registry of algorithm bitstreams that can be loaded and executed using the RASC abstraction layer. The devmgr user command is used to add, delete, and query algorithms in the bitstream registry. An algorithm is identified by its user supplied name and consists of a set of files, as follows:

  • Algorithm bitstream

  • Configuration file for the algorithm bitstream

  • Core services configuration file

All of these files must be accessible by the devmgr user command when the algorithm is added to the bitstream registry.

The devmgr user command executes on the target Altix system, but the algorithm bitstream and configuration files are built on a PC using one of several FPGA development tools.


Note: For detailed information using the devmgr user command, see “RASC Device Manager”.

To make the algorithm files available to the devmgr command, perform the following:

  1. Prepare to FTP three files from the PC to the Altix system.

    • The files can reside anywhere on the Altix system that can be accessed by the devmgr command. A convenient location is, as follows:

      /usr/share/rasc/bitstream/your_algorithm_directory
      

    • Create the bitstream and optionally a user specific subdirectory in /usr/src/RASC, if they do not already exist on the Altix system.

  2. Execute the FTP command from the PC to transfer the algorithm bitstream, its configuration file, and the core services configuration file to the user generated subdirectory.

  3. Login to the Altix system and execute the devmgr user command to add the algorithm to the bitstream registry giving the algorithm a user supplied name. All references to the bitstream in the bitstream registry is made using this name. Note that the bitstream file suffix must be *.bin and the configuration files suffix must be *.cfg.

    For example, assume that an algorithm bitstream and configuration files have been FTP'd to the Altix system in subdirectory dog. A list of the files in dog could look like the following:

    /usr/share/rasc/bitstream/dog/spot.bin
    /usr/share/rasc/bitstream/dog/spot.cfg
    /usr/share/rasc/bitstream/dog/core_services.cfg
    

  4. This devmgr command could be executed on the Altix system to add the algorithm to the bitstream registry with the name bark, as follows:

    devmgr -a -n bark -b /usr/share/rasc/bitstream/dog/spot.bin
    

    Notice that the full path name of the bitstream must be specified. If the algorithm bitstream configuration file is not called spot.cfg, the devmgr -c option must also specify the full path name of the algorithm configuration file. Similarly, if the core services file is not core_services.cfg, the -s option must specify its full path name.

RASC Device Manager

This section provides detailed information on using the Device Manager (devmgr) software and covers the following topics:

RASC Device Manager Overview

The RASC Device Manager (devmgr) maintains a bitstream registry and a FPGA hardware inventory. You can add, delete, and list bitstreams in the bitstream registry using the devmgr command and can allocate available FPGAs by making requests using the RASC abstraction layer software. You can also use the devmgr command to list the FPGAs in the inventory.

You can manually load an algorithm into an FPGA on demand using the devmgr user command -l option.

You can use the devmgr -r option to direct the Device Manager to always reload an algorithm into an FPGA, even when that FPGA is already loaded with the current version of the requested algorithm.

When an FPGA is allocated to a user, the requested bitstream is loaded into those FPGAs. Allocation requests must be made using abstraction software (see Chapter 4, “RASC Abstraction Layer” for specific API calls) and are passed to the device manager.

Each entry in the bitstream registry is identified by the name, which is limited to 63 characters, that is assigned by the user when the bitstream is added to the bitstream registry. The entry also includes copies of the bitstream binary file, the bitstream configuration file, and the cores services configuration file. The location of these files on the local machine is also specified by the user when the entry is added.

You can use the devmgr -m unavail option to mark an FPGA device node as unavailable for application allocation. You can use the devmgr -m avail option to mark an FPGA device node as available for application allocation.

The devmgr -v option shows the devmgr command version, and build date and time. This information can be helpful when diagnosing Device Manager software problems.

RASC Device Manager Structure

The Device Manager is structured as client and server components that communicate over a TCP/IP connection. Both the devmgr user command and the services requested by the abstraction layer interface with the client side of the device manager. The bitstream registry management and FPGA allocation and loader services occur on the server side. Before a bitstream can be added to the bitstream registry, all of the bitstream files must exist on the machine where the server is running.

Using the Device Manager Command (devmgr)

The devmgr user command requires one action argument that may require additional arguments, and accepts an optional client debugging argument. The command actions are, as follows:

Guidelines for using the devmgr command are, as follows:

  • The add and delete actions require a bitstream registry entry name; the add action also requires the binary bitstream file name, which must have the ".bin" suffix.

  • If the bitstream configuration file name is not specified, it defaults to the bitstream file basename with a ".cfg" suffix in the same directory as the binary bitstream file.

  • If the cores services file name is not specified, it defaults to "core_services.cfg" in the same directory as the binary bitstream file.

Add a Bitstream To the Bitstream registry

The devmgr command syntax to add a bitstream to the bitstream registry is, as follows:

devmgr -a -n algorithm name -b bitstream file name [-c bitstream config file name] [-s core services config file name] [-y on | off]

The devmgr command accepts the following options:

-a  

Adds an entry to the bitstream registry.

-n algorithm name 

Names an entry in the bitstream registry.

-b bitstream file name 

Indicates the location of the bitstream file.

-c bitstream config file name  

Indicates the location of the configuration file.

-s core services config file name 

Indicates the location of the core services config file.

-y on | off 

Turns client debugging on or off.

Delete a Bitstream From the Bitstream registry

The devmgr command syntax to delete a bitstream from the bitstream registry is, as follows:

devmgr -d -n algorithm name [-y on | off]

The devmgr command accepts the following options:

-d 

Deletes an entry from the bitstream registry.

-n algorithm name  

Identifies the name of an entry in the bitstream registry.

-y on | off 

Turns client debugging on or off.

List the Contents of a Bitstream registry

The devmgr command syntax to list a specific entry or all entries in the bitstream registry is, as follows:

devmgr -q [-n algorithm name][-y on | off]

The devmgr command accepts the following options:

-q 

Lists all entries in the bitstream registry.

-n algorithm name 

List a specific algorithm in the bitstream registry.

-y on | off 

Turns client debugging on or off.

Update an Algorithm in the Bitstream registry

You can use the -u option to update an existing algorithm in the Device Manager bitstream registry with a new version of the bitstream file, bitstream configuration file, and core services configuration file. All three files are replaced in the bitstream registry with the new files specified with the command. The -u option is logically equivalent to a -d option followed by the -a option, where both options specify the same algorithm name.

After a successful update of an algorithm in the bitstream registry, any FPGAs currently loaded with the same algorithm name will be reloaded with the new version of the algorithm the next time that FPGA is allocated.

The devmgr command syntax to update an algorithm in the bitstream registry is, as follows:

devmgr -u -n algorithm name -b bitstream file name [-c bitstream config file name] [-s core services config file name] [-y on | off]

The devmgr command accepts the following options:

-u  

Updates the entry in the bitstream registry.

-n algorithm name 

Names an entry in the bitstream registry.

-b bitstream file name 

Indicates the location of the bitstream file.

-c bitstream config file name  

Indicates the location of the configuration file.

-s core services config file name 

Indicates the location of the core services config file.

-y on | off 

Turns client debugging on or off.

List the FPGAs in the Inventory

You can use the devmgr command to list the FPGAs in the inventory. The list indicates whether the FPGA is available or in use, the system node where the FPGA resides (its device node ID), part number, and manufacture (mfg) number. FPGAs that are in use also list the Linux process ID (pid) and Pthread thread ID (thread id) of the current FPGA user, and the name of the algorithm currently loaded in to the FPGA.

The devmgr command syntax to list a specific entry or all entries in the FPGA inventory is, as follows:

devmgr -i [-N odd integer][-y on | off]

The devmgr command accepts the following options:

-i 

Lists all entries in the inventory.

-N odd integer 

List the specific FPGA device node ID.

-y on | off 

Turns client debugging on or off.

Mark an FPGA as Available or Unavailable

You can mark an FPGA device node as unavailable for use by an application with the devmgr -m unavail option. If the FPGA is currently in use, the mark action will remain pending until the FPGA is freed; then the FPGA is marked as unavailable.

The devmgr -m avail option marks an FPGA device node as available for allocation by an application.

You must specify the target FPGA device node ID using the -N option whenever you use the devmgr -m option.

You can use the devmgr -i option to display the availability of an FPGA device (see “List the FPGAs in the Inventory”).

The devmgr command syntax to to mark an FPGA device as available or unavailable is, as follows:

devmgr -m avail | unavail -N odd integer [-y on | off]

The devmgr command accepts the following options:

-m avail | unavail 

Marks an FPGA device node as available or unavailable for use.

-N odd integer 

Specifies the target FPGA device node ID.

-y on | off 

Turns client debugging on or off.

Turn Debugging On or Off

The devmgr command syntax to turn server or client debugging on or off is, as follows:

devmgr -x on | off [-y on | off]

The devmgr command accepts the following options:

-x on | off 

Turns server debugging on or off.

-y on | off 

Turns client debugging on or off.

Device Manager server and client debugging can also be turned on or off using environment variables. This is the only way to control Device Manager client debugging for RASC applications. The devmgr_server command -x and -y options override any environment variable settings. The devmgr_server command is used to start the server component of the device manager. For more information on the devmgr_server command, see “Device Manager Server Command”.

To turn client debugging on or off, set the RASC_DEVMGR_CLIENT_DEBUG variable, as follows (for the Korn shell):

export RASC_DEVMGR_CLIENT_DEBUG=on
export RASC_DEVMGR_CLIENT_DEBUG=ON
export RASC_DEVMGR_CLIENT_DEBUG=off
export RASC_DEVMGR_CLIENT_DEBUG=OFF

To turn server debugging on or off set the RASC_DEVMGR_SERVER_DEBUG variable, as follows (for the Korn shell):

export RASC_DEVMGR_SERVER_DEBUG=on
export RASC_DEVMGR_SERVER_DEBUG=ON
export RASC_DEVMGR_SERVER_DEBUG=off
export RASC_DEVMGR_SERVER_DEBUG=OFF

Device Manager Load FPGA Command

You can manually load an algorithm into an FPGA on demand using the devmgr user command -l option. Usually the algorithm is automatically loaded into the FPGA by the RASC abstraction layer as part of the co–processor (COP) allocation request. FPGAs loaded by the abstraction layer are marked as in use until the RASC abstraction layer frees the FPGA. FPGAs manually loaded using the devmgr command are not marked in use and their algorithm can be immediately changed by another user command or the RASC abstraction layer after the devmgr command terminates.

The -l option requires an algorithm name specified with the -n option and a system node ID identifying the FPGA location with the -N option.


Note: Note that I/O nodes, including FPGA nodes, are always an odd number. Use the devmgr command with the -i option to view the FPGA inventory and their node IDs.

The devmgr command syntax to manually load an FPGA with an algorithm is, as follows:

devmgr -l -N FPGA odd numbered node ID -n algorithm name [-y on | off] 

The devmgr command accepts the following options:

-l 

Loads this FPGA with this algorithm.

-N FPGA odd numbered node ID 

FPGA device odd numbered node ID.

-y on | off 

Turns client debugging on or off.

Device Manager Reload FPGA Command

By default, the Device Manager only loads an algorithm into an FPGA under the following conditions:

  • The FPGA is not already loaded with the requested algorithm, that is, the name of the algorithm loaded in the FPGA is not the same as the requested algorithm name.

  • A newer version of the algorithm is available in the bitstream registry.

Otherwise, the FPGA is not loaded, eliminating the latency needed to load the FPGA.

You can use the -r option to direct the Device Manager to always reload an algorithm into an FPGA, even when that FPGA is already loaded with the current version of the requested algorithm. Always reloading an FPGA can be useful when debugging a suspected hardware or software FPGA load problem. Specify on to enable always reload an FPGA and off to only reload an FPGA when needed.

To determine whether the always reload option is on or off, use the devmgr -i command.

The devmgr command syntax to always reload an FPGA or only reload an FPGA when it is needed is, as follows:

devmgr -r on | off [-y on | off]

The devmgr command accepts the following options:

-r on | off 

Turns the always reload an FPGA action on or off.

-y on | off 

Turns client debugging on or off.

Device Manager Version Information

You can use the devmgr -v option to show the version, and build date and time of the devmgr command. This information can help you diagnose Device Manager software problems.

The devmgr command syntax to show version information is, as follows:

devmgr -v [-y on | off]

The devmgr command accepts the following options:

-v  

Shows devmgr version information.

-y on | off 

Turns client debugging on or off.

Device Manager Server Command

The Device Manager includes a server component that manages the algorithm bitstream registry, allocates and frees FPGA devices from its inventory, and loads algorithm bitstreams into the FPGAs. The devmgr_server command is used to start the server component of the device manager.

Both devmgr user command and abstraction layer requests are sent over a TCP connection to the server to be processed, and the response is sent back over the same connection to the requester.

By default, the server listens for incoming connection requests on port number 9999. The port number can be changed by the system administrator by adding a port definition to the TCP/IP services file. Usually this file is located, as follows:

/etc/services

An example of how to change the listening port to 9998 is, as follows:

rasc_devmgr      9998/tcp          # RASC Device Manager Server


Note: After you make a port number change, you must restart the server (see the “Using the Device Manager Server (devmgr_server) Command”).


Using the Device Manager Server (devmgr_server) Command

After changing a port number as described in “Device Manager Server Command”, the server must be started before any requests can be processed. The devmgr_server command is used to start the server component of the device manager. The debug option enables server initialization debugging, which cannot be done using the devmgr command with the -x on action. By the time a devmgr command can request that server debugging be turned on, the server initialization has already completed.

The devmgr_server command syntax is, as follows:

devmgr_server [-x on | off]

where the -x option turns server debugging on or off. The default setting is off.

Device Manager Logging Facility

The Device Manager client and server provides an optional logging facility that tracks Device Manager requests and responses. Client or server logging is independently enabled by specifying the log file name using an environment variable. Log entries are time stamped and marked with the owning process and thread identifiers. The server log entries also include the process and thread identifiers of the client.

A summary of client application Device Manager activity can be obtained by enabling client logging for the application. Each client should use its own log file by specifying a unique file name for its log. This makes finding and browsing the log for a particular application easier.

The current working directory of the device manager's server is always root. Unless a path is specified, the server log file is created in the root directory.

To enable client logging set the RASC_DEVMGR_CLIENT_LOG variable to the name of the file that is to contain the client log entries, as shown below for the Korn shell:

export RASC_DEVMGR_CLIENT_LOG=client_log_file_name

To disable client logging, unset the RASC_DEVMGR_CLIENT_LOG variable, as shown below for the Korn shell:

unset RASC_DEVMGR_CLIENT_LOG

To enable server logging set the RASC_DEVMGR_SERVER_LOG variable to the name of the file that is to contain the server log entries, as shown below for the Korn shell:

export RASC_DEVMGR_SERVER_LOG=server_log_file_name

To disable server logging, unset the RASC_DEVMGR_SERVER_LOG variable, as shown below for the Korn shell:

unset RASC_DEVMGR_SERVER_LOG

Using the GNU Project Debugger (GDB)


Note: Access to memories configured as 64-bit wide is not supported at this time in GDB FPGA (gdbfpga).

This document describes extensions to the GNU Debugger (GDB) command set to handle debugging of one or more FPGAs. Normal GDB commands and facilities are unchanged.

Brackets [ ] are used in this chapter to indicate that the value is optional. The [ ] here is not something a GDB user types; it is a syntactic convention used in this document to express an optional command field.

We use N here to indicate a number such as 0 or 1. The FPGA number is assigned by GDB when the FPGA is opened (loaded into the GDB command session). The first FPGA opened is assigned number zero, the next is assigned one, and so on. Numbers are not reused. GDB interacts with the RASC abstraction layer to implement several of the GDB features documented below. The N FPGA number is not the same as the cop_desc integer assigned in a user's application (that cop_desc integer is printed in the 'info fpga' output).

Some FPGA data is treated as if it were in registers. Such things use GDB normal register syntax, so one types a prefix $ (and GDB shows a prefix $) to the data name. Other FPGA data is treated as variables or arrays and normal C-like language syntax is used.


Note: For this release, GDB FPGA (gdbfpga) does NOT support DMA stream_in and stream_out functionality described in “Streaming Direct Memory Access” in Chapter 3.


GDB Commands

The commands added to GDB unique to RASC are, as follows:

  • fpgaactive [on/off]

  • set fpga fpganum = N

  • info fpgaregisters [regname]

  • info fr (alias for 'info fpgregisters')

  • info fpga

  • fpgastep

  • fpgacont

  • fpgatrace [on/off]

Examples of standard commands and expressions with special meanings for FPGAs are, as follows:

print $a_0
print a_in[12]

where the RASC configuration has specified a_0 as being in the Debug Port Interface of the Algorithm Block, and a_in is an array in an SRAM (possibly N buffered).

These are FPGA-specific and visible only when actively stepping an FPGA. For more information, see “Adding Extractor Directives to the Source Code” in Chapter 7.

fpgaactive

The syntax for the fpgaactive command is, as follows:

fpgaactive [on,off]

With no operands, shows the activity-state (on or off) of the current FPGA. Defaults to off, meaning FPGA is inactive and registers are invisible. The RASC Abstraction layer normally maintains the off/on state when you load GDB FPGA with an executable that allocates an FPGA. The state can be set on or off manually (in GDB, not the device) with an optional operand of on or off, however, doing so is not useful, and could break an application.

set fpga fpganum

The syntax for the fpganum command is, as follows:

set fpga fpganum = N

This sets the current fpga number. This value provides an identity so when there are multiple FPGAs one knows which one is being acted/reported on. The N is the FPGA number assigned by GDB, not the cop_desc integer set in the application executable. The FPGA number is not 'recycled' in a GDB session.

fpgaregisters

The syntax for the fpgaregisters command is, as follows:

info fpgaregisters [regname]

Prints the list of register names and values for the current FPGA. If regname is given prints just that register name and value. It ignores normal processor registers.

info fpga

The syntax for the info fpga command is, as follows:

info fpga [full] [N]

This reports information about the FPGA set in use. That is, it displays overview information about each FPGA. The full option expands the information shown. The N option restricts output to a specfic FPGA (N being the GDB-assigned FPGA number).

fpgastep

The syntax for the fpgastep command is, as follows:

fpgastep [N]

This command does a single step by default. The N option tells GDB to step the FPGA N steps. Because hardware does the N steps, N steps go as fast as a single step.

fpgacont

The syntax for the fpgacont command is, as follows:

fpgacont

Sets the current FPGA running to completion.

fpgatrace

The syntax for the fpgatrace command is, as follows:

fpgatrace [on/off]

You can use the fpgatrace to dynamically report FPGA use.

Turn fpga start/stop tracking (reporting) on or off. With no operands this reports the current state of tracking. With tracking on, each start/stop of an FPGA prints a short line giving the FPGA configuration file name, GDB number, and the ID assigned by the application. It has not escaped our attention that the mechanism could be elaborated to provide detailed information live or to a postprocessing tool to aid in understanding when multiple FPGAs are in use or the FPGA use is otherwise complicated.

Registers

FPGA values that have been mapped to debug ports and labelled by extractor directives are visible as if in the processor register set. If the user space configuration file says it is an output register (using a REG_OUT extractor directive) , then (when the FPGA is active) the command 'print $a' will print that register.

You must specifiy a register as REG_IN or REG_OUT. The register can be defined as R,W, or RW, but on a bit basis, a direction must be defined.

The requirements for this operation are that the register name is unique per FPGA register and across the entire set of register names in the processor. The FPGA registers are an extended set appended to the processor registers. It is important to note that these registers are only visible when the FPGA is active. This is a normal GDB construct with an FPGA aware extension and it adheres to all typical register syntax.

Values and Stepping

When an FPGA is running one may want to see the internal values if one is stepping the FPGA. Internal values that are exposed depends on the following:

  • FPGA programming

  • What the FPGA version of core-services supports

  • What the device can do

  • Linux device driver support

As with normal language programming, there is 'stepping'as known at the hardware level, and 'stepping' at a higher level. This is particularly relevant for a device like an FPGA because whether a given value is even 'current' or 'valid' depends on details of the FPGA state. The device program exposes 'step points' that are meaningful at a software level. Programming to the interface ensures values are meaningful at those step points, and those step points are what is iterated fpgastep. Some device programs may not expose such logical-step points and for those device programs values printed via GDB will be more difficult to interpret sensibly (since it is not possible to print the entire FPGA internal state).

FPGA Run Status

When the GDB break command is used on an application executable that is enabled with RASC API calls, rasclib_brkpt_start() and rasclib_brkpt_done() functions, the GDB inserts break points when an FPGA is started or done, respectively.

To be able to see the FPGA state when alg_done occurs, you would expect to set rasclib_brkpt_done. However, you actually have to first set rasclib_brkpt_start. If you do not do so, you will not be able to see any of the FPGA registers when you hit rasclib_brkpt_done. The sequence is similar to the following:

break rasclib_brkpt_start

break rasclib_brkpt_done

handle SIGUSR1 nostop pass noprint

run

-> stop at rasclib_brkpt_start and probe registers as desired

fpgacont

->stop at rasclib_brkpt_done and probe registers as desired

delete

cont

-> success

An alternative method of determining FPGA run status is the fpgatrace on command that reports details of FPGA starts and stops with more detail than the break command.

Changes To GDB Commands

The gdb cont command has a new side effect in gdbfpga of implying fpgacont on all FPGAs on which you can apply the fpgastep or fpgacont commands. The side effect makes cont get the entire application running as you would expect no matter what state FPGAs are in.