For PIO purposes (the CPU loading and storing in device space), memory space defined by each PCI device in its configuration registers is allocated in the upper two gigabytes of the PCI address space, above 0x8000 0000. These addresses are allocated dynamically, based on the contents of the configuration registers of active devices. The I/O address space requested by each PCI device in its configuration registers is also allocated dynamically as the system comes up.

It is possible for a PCI device to request (in the initial state of its Base Address Registers) that its address space be allocated in the first 1 MB of the PCI bus. This request cannot be honored in the O2 workstation. Devices that cannot decode bus addresses above 0x8000 0000 are not supported.

Device drivers get a virtual address to use in PIO access by creating a PIO map (see “Managing PIO Maps for PCI”).

As described under “Driver Flag Constant”, the pfxdevflag public name is a byte containing flags for driver characteristics. Since a PCI driver is inevitably a new driver, with no heritage in older versions of IRIX or UNIX, Silicon Graphics, Inc. strongly recommends that you design it from the start to be compatible with multiprocessors. The implications of this are discussed under “Planning for Multiprocessor Use”.

When the system boots up, the kernel probes the PCI bus configuration space and takes a census of active devices. For each device it notes

The kernel assigns starting bus addresses for memory and I/O space and sets these addresses in the Base Address Registers (BARs) in the device. Then the kernel looks for a driver that has registered a matching set of vendor and device IDs using pciio_driver_register().

If no matching driver has registered, the device remains inactive. For example, the driver might be a loadable driver that has not been loaded as yet. When the driver is loaded and registers, the kernel will match it to any unattached devices.

When the kernel matches a device to its registered driver, the kernel calls the driver's pfxattach() entry point. It passes one argument, a vertex_hdl_t, which acts as an opaque handle to a kernel object that describes this device. This handle is used to:

A driver needs to save information about each device, usually in a structure.Fields in a typical structure might include:

A problem is that at initialization time a driver does not know how many devices it will be asked to manage. For a workstation such as O2 you can expect the number will be small, but you should allow for portability to server-class systems that support dozens of devices. In the past this problem has been handled by allocating an array of a fixed number of information structures, indexed by the device minor number.

This is not a good solution for a PCI driver because PCI configuration is so dynamic. In addition, a loadable driver loses the contents of its global variables when it unloads. The IRIX PCI support gives you a different way.

In a PCI driver, you dynamically allocate memory for an information structure to hold information about the one device being attached. (See “General-Purpose Allocation”.) You save the address of the structure in the kernel's hardware vertex, using the device_info_set() function, which associates an arbitrary pointer with a vertex_hdl_t.

extern void device_info_set(vertex_hdl_t, arbitrary_info_t);

The information structure can easily be recovered in any top-half routine; see “Locating Device Information”.

For almost any device you need at least one PIO map. You use a PIO map to read the configuration space or the I/O space of a device. For a bus-master device you will need at least one DMA map. These maps can be allocated when the device is attached, and the addresses of the maps can be stored in the device information structure.

A PIO map is created by pciio_piomap_alloc(). It takes a vertex_hdl_t, a flag for the bus address space, and an offset in that space as its principal arguments.

extern pciio_piomap_t pciio_piomap_alloc (
      vertex_hdl_t vhdl,       /* set up mapping for this device */
      device_desc_t dev_desc,  /* device descriptor (null) */
      pciio_space_t space,     /* which address space */
      iopaddr_t pcipio_addr,   /* starting offset in space */
      size_t byte_count,
      size_t byte_count_max,   /* maximum size of a mapping */
      unsigned flags);         /* defined in sys/pio.h */

The arguments are as follows:

The device descriptor structure type dev_desc_t is declared in pciio.h. Only one field is inspected, intr_swlevel. It must be set to one of the interrupt levels of type pl_t as declared in ddi.h, typically plhi.

A PIO map that you will use to access device configuration registers is based on a a space of PCIIO_PIOMAP_CFG. The space selection PCIIO_PIOMAP_WIN(n) means that this map is to be based on Base Address register n, from 0 through 5, in the PCI configuration space. The device configuration registers specify whether a given base address register defines memory or I/O space. When the space is defined by a 64-bit base address register, use the lower number, the index of the word that contains the configuration bits.

Example 15-2 shows a function that allocates a PIO map. The configuration space is passed as an argument, as is the size of the space to map. The function assumes the map should start at offset 0 in the selected space.

pciio_piomap_t makeMap(vertex_hdl_t dev, int BAR, size_t size)
{
   struct device_desc_s ddesc;
   ddesc.intr_swlevel = plhi;
   return pciio_piomap_alloc(
         dev,         /* vertex handle */
         &ddesc,      /* dev descriptor w/ in level in it */
         BAR,         /* space, _CFG or _WIN(n) */
         0,           /* starting offset */
         size,size,   /* size to map */
         0);          /* default endian */
}

A DMA map is created by pciio_dmamap_alloc(), which takes a vertex_hdl_t, a size, and flags regarding the treatment of the mapping.

extern pciio_dmamap_t pciio_dmamap_alloc(
      vertex_hdl_t dev,         /* set up mappings for this device */
      device_desc_t dev_desc,  /* device descriptor */
      size_t byte_count_max,   /* max size of a mapping */
      unsigned flags);         /* defined in dma.h */

Other map functions are discussed under “Managing PIO Maps for PCI” and “Managing DMA Maps for PCI”.

Typically a PCI driver needs to read the device configuration registers and possibly write to them. These are PIO operations. To access the configuration, create a PIO map for the configuration space and extract an address from it. Present this address to pciio_config_get() to fetch a word from configuration space.

pciio_piomap_t cfg_map;
typedef volatile __uint32_t cfg_reg;
cfg_reg *cfg_ptr;
__uint32_t cfg_value;
cfg_map = pciio_piomap_alloc(
               vhdl,  /* as received in attach() */
               NULL,  /* default device descriptor */
               PCIIO_SPACE_CFG, /* map to config space */
               (iopaddr_t)0, /* at offset 0 */
               64,64, /* size of PCI config header */
               0 ); /* no flags */
cfg_ptr = (cfg_reg *) pciio_piomap_addr(
               cfg_map, /* the map to use */
               0,    /* start at the beginning */
               64 ); /* amount I care about */
cfg_value = pciio_config_get(cfg_ptr,4); /* read the register */

For a PCI bus master device, the pfxattach() function should set the Cache Line Size register to 128 (the size of a cache line in all Silicon Graphics systems).

For devices that can interrupt, a key step during pfxattach() is to register an interrupt handler for the device. This is done in a two-step process. First you create an interrupt connection object; then you use that object to specify the interrupt handling function.

The interrupt connection is created with pciio_intr_alloc(), which takes a vertex_hdl_t and a flag for the interrupt line that the device uses.

extern pciio_intr_t pciio_intr_alloc(
         vertex_hdl_t dev,         /* which PCI device */
         device_desc_t dev_desc,   /* device descriptor */
         pciio_intr_line_t lines,  /* which line(s) will be used */
         vertex_hdl_t owner_dev);         /* owner of this intr */

The interrupt object is used in establishing a handler, and it is needed later to stop taking interrupts. You will probably want to save its address in the device information structure for later use (see “Inactivating an Interrupt Handler”).

After creating the interrupt object, you establish a handler using pciio_intr_connect(). Its principal arguments are the interrupt object, a handler address, and a value to be passed to the handler when it is called (this will typically be the address of the device information structure you are preparing).

extern int pciio_intr_connect(
         pciio_intr_t intr_hdl,      /* pciio intr resource handle */
         intr_func_t intr_func,     /* pciio intr handler */
         intr_arg_t intr_arg,       /* arg to intr handler */
         void *thread);             /* intr thread to use */

If a device will interrupt on line C, interrupt setup could resemble Example 15-4.

pciio_intr_t intobj;
extern void int_handler(devinfo*);
int retcode;
intobj = pciio_intr_alloc(
               vhdl, /* as received in attach() */
               0,    /* device descriptor is n.a. for pci */
               PCIIO_INTR_LINE_C, /* the line it uses */
               (vertex_hdl_t) 0);
retcode = pciio_intr_connect(
               intobj, /* the interrupt object */
               (intr_func_t) int_handler, /* the handler */
               (intr_arg_t) pDevInfo, /* dev info as input */
               (void*)0 ); /* threads are next release */
if (!retcode) cmn_err(CE_WARN,"oh fiddlesticks");

---

Tip: Interrupts are enabled when the pfxattach() entry point is called. If the PCI device is in a state that can produce an interrupt, the interrupt handling function can be called before pciio_intr_connect() returns. Make sure that all global data used by the interrupt handler has been initialized.

---

---

Note: PCI devices can share the four PCI interrupt lines. As a result, in some cases the kernel cannot tell which device caused an interrupt. When there is any doubt, the kernel calls all the interrupt handlers that are registered to that interrupt line. For this reason, your interrupt handler must not assume that its device did cause the interrupt. It should always test to see if an interrupt is really pending, and exit immediately when one is not.

---

The return code from pfxattach() is tested by the kernel. The driver can reject an attachment. When your driver cannot allocate memory, or fails due to another problem, it should:

More than one driver can register to support the same vendor ID and device ID. When the first driver fails to complete the attachment, the kernel continues on to test the next, until all have refused or one accepts. The pfxdetach() entry point can only be called if the pfxattach() entry point returns success (0).

The driver upper-half entry points are called to implement requests from user processes or filesystems that need to open, read, write, map or control the device. These calls can occur at any time; and on a multiprocessor, they can occur multiple times concurrently, on parallel CPUs.

The user process refers to a device through a file descriptor opened to a device special file. The primary argument to any upper-half entry point is the dev_t, a value that distinguishes the device. Traditional drivers extract device numbers from the dev_t (see “The Device Number Types”).

The first thing any upper-half entry point needs to do is to locate the per-device information structure that was prepared in the pfxattach() entry point (see “Allocating Storage for Device Information”). You do this by calling device_info_get(). However, that function takes a vertex_hdl_t. You get that from dev_to_vhdl(). The code, which is repeated over and over in a PCI driver, resembles Example 15-6.

vertex_hdl_t vhdl = dev_to_vhdl(dev);
my_dev_info_t *pDev = (my_dev_info_t)device_info_get(vhdl);
   if (!pDev) return(ENXIO);
   if (!(pDev->status & USABLE)) return(ENXIO);

The only variation is the data type of the device information structure. In the pfxopen() entry point, the dev_t is received as a reference argument, not by value.

In the event that device_info_get() returns NULL, this device has not been attached, or the pfxattach() entry point did not allocate an information structure; or the pfxdetach() entry point has been called. In such cases, the upper-half routine should return ENXIO (no such device). This test is shown in Example 15-6.

Example 15-6 also shows another test. In the next release of IRIX, a driver will be able to implement a pfxdisable() entry point, called to make a device temporarily unusable. Even in the current release, your driver might find reasons, such as a persistent device error, to force a device offline. A single flag bit in the device information structure represents this state. Again, a return of ENXIO is appropriate.

The functions that are used to manage PIO maps are summarized in Table 15-2.

Maps are allocated with pciio_piomap_alloc(). Its use is covered under “Allocating PIO Maps”, because you typically will allocate the PIO maps you need while attaching the device.

You use a PIO map by calling pciio_piomap_addr(). This function takes a map, an offset within the PCI address space described by the map, and the number of bytes of space that the address will be used to retrieve.

extern caddr_t pciio_piomap_addr(
            pciio_piomap_t pciio_piomap,   /* mapping resources */
            iopaddr_t pciio_addr,   /* map for this pcipio address */
            size_t byte_count);    /* map this many bytes */

The pciio_addr argument is added to the base offset specified to pciio_piomap_alloc(), and that in turn is added to the base address assigned by the kernel to this device, to arrive at the bus address needed. The byte_count argument specifies how many bytes beyond the bus address you may access.

The returned value is a kernel virtual address that is mapped to the requested PCI bus address for at least that many bytes. Use the address as a pointer to a “volatile” variable. In systems in which PCI space is hard-wired to specific memory addresses, pciio_piomap_alloc() is a simple function and pciio_piomap_addr() is a trivial one, so they should be used for portability's sake even in systems where you know the mapping.

An example of using a PIO map to access configuration space is shown under “Reading the Device Configuration”.

Once you have extracted an address using pciio_piomap_addr(), the map is active. It remains active until you call either pciio_piomap_done() or pciio_piomap_free(). In some systems, it costs nothing to keep a PIO map active. In other systems, an active PIO map may tie up global hardware resources. It is is a good idea to call pciio_piomap_done() when the address is no longer needed.

Some systems also support a one-step translation function, pciio_piotrans_addr(). This function takes a combination of the arguments of pciio_piomap_alloc() and pciio_piomap_addr(), and returns a translated address. In effect, it combines creating a map, using the map, and freeing the map, into a single step. However, this function can fail in systems that do not use hard-wired bus maps. The two-step process of allocating a map and then interrogating it is more general.

Access to configuration space is done in two steps. First you use pciio_piomap_addr() or pciio_piotrans_addr() to get a virtual address; then you pass the address to pciio_config_get() or pciio_config_get(). In some systems, the configuration space can be accessed directly with fetch and store instructions. In these systems, these two functions are implemented as macros.

When you use PIO to fetch or store 32-bit values on 32-bit-aligned PCI addresses, PIO works as you would expect, and a 32-bit value is fetched or returned.

However, when you use PIO to fetch or store less than 32 bits—either a 16-bit value or an 8-bit value—you must use an address that takes account of byte-swapping. The least significant address bits are summarized in Table 15-3.

You can deal with this complication in either of three ways, as follows:

The functions that are used to manage simple DMA maps are summarized in Table 15-4.

Maps are allocated with pciio_dmamap_alloc(). Its use is covered under “Allocating PIO Maps”, because you typically will allocate the maps you need while attaching the device.

You obtain a map for a single, contiguous span of virtual memory by calling pciio_dmamap_addr(). It takes principle arguments of a map, a memory address, and a length.

extern iopaddr_t pciio_dmamap_addr(
         pciio_dmamap_t dmamap,      /* use these mapping resources */
         paddr_t paddr,             /* map for this address */
         size_t byte_count);        /* map this many bytes */

The value returned is a bus address that you can program into a bus master device. When the device accesses that address, it is accessing the specified memory location.

Once you have extracted an address using pciio_dmamap_addr(), the map is active. It remains active until you call either pciio_dmamap_done() or pciio_dmamap_free(). In some systems, it costs nothing to keep a DMA map active. In other systems, an active map may tie up global hardware resources. It is is a good idea to call pciio_dmamap_done() when the I/O operation is complete.

In systems in which PCI space is hard-wired to specific memory addresses, pciio_dmamap_alloc() is a short function and pciio_dmamap_addr() is a trivial one. However, these systems also support a one-step translation function, pciio_dmatrans_addr(). This function takes a combination of the arguments of pciio_dmamap_alloc() and pciio_dmamap_addr(), and returns a translated address. In effect, it combines creating a map, using the map, and freeing the map, into a single step.

In some cases you are not sure whether a memory buffer is contiguous, or perhaps you are sure that it is not. In this case you need to create a list of memory addresses and lengths—one address and length for each segment of memory. Then you need to translate the segments into a list of bus addresses. The list of bus addresses can be programmed into the device one at a time or, if the device supports scatter/gather, you can program all of the list of addresses for transfer in sequence.

Support for these cases is proved by address-length lists, an abstract data type that is supported by a family of functions. The next major release of IRIX contains a complete family of functions for address-length lists. The current release supports a subset necessary to use with DMA maps. The functions related to address-length lists are summarized in Table 15-5.

The function buf_to_alenlist() is called in a pfxstrategy() entry point. It takes a buf_t and returns an address-length list that describes each segment of memory in the buffer that the buf_t describes (see “Structure buf_t” and “Entry Point strategy()”. The function kvaddr_to_alenlist() takes the address and length of any buffer in kernel virtual memory and returns an address-length list to describe that extent of memory.

When you are ready to perform DMA to a buffer, you create an address-length list to describe the buffer, and pass that through pciio_dmamap_list(). This function returns a new address-length list in which the memory addresses have been replaced by PCI bus addresses.

You step through the contents of the converted address-length list using alenlist_get(), which returns successive pairs of values—an address and a length—from the list. You program each pair of values into the bus master device.

The functions for managing interrupt handlers are summarized in Table 15-6.

The allocation of an interrupt handler is covered under “Registering an Interrupt Handler”. When detaching a device, call pciio_intr_disconnect() to break the association between the interrupt and the handler function.

There are typically various allocated objects—PIO and DMA maps, interrupt objects—that are addressed from the device information structure stored for the device. All such objects should be released at this time.