The user-level interrupt (ULI) facility allows a hardware interrupt to be handled by a user process. The ULI facility is intended to simplify and streamline the response to external events. ULIs can be written to respond to interrupts initiated from the VME bus, the PCI bus, or external interrupt ports. ULIs are essentially Interrupt Service Routines (ISRs) that reside in the address space of a user process. As shown in Figure 7-1, when an interrupt is received that has been registered to a ULI, it calls the user function from the interrupt level. For function prototypes and other details, see the uli(3) man page.
 | Note: The uli(3) man page and the libuli library are installed as part of the REACT/pro package. The features described in this chapter are supported in REACT/pro version 3.2, which must be installed in order to use them. |
In the past, PIO could be only synchronous: the program wrote to a device register, then polled the device until the operation was complete. With ULI, the program can manage a device that causes interrupts on the VME or PCI bus. You set up a handler function within your program. The handler is called whenever the device causes an interrupt.
In IRIX 6.2, user-level interrupts were introduced for VME bus devices and for external interrupts on the Challenge and Onyx systems. In IRIX 6.5, user-level interrupts are also supported for PCI devices, and for external interrupts on Origin 2000, Origin 200, and Onyx 2 systems.
When using ULI with a VME or PCI device, you use PIO to initiate device actions and to transfer data to and from device registers. When using ULI to trap external interrupts, you enable the interrupts with ioctl() calls to the external interrupt handler. All these points are covered in much greater detail in the IRIX Device Driver Programmer's Guide (see “Related Publications and Sites”).
The ULI handler is a function within your program. It is entered asynchronously from the IRIX kernel's interrupt-handling code. The kernel transfers from the kernel address space into the user process address space, and makes the call in user (not privileged kernel) execution mode. Despite this more complicated linkage, you can think of the ULI handler as a subroutine of the kernel's interrupt handler. As such, the performance of the ULI handler has a direct bearing on the system's interrupt response time.
Like the kernel's interrupt handler, the ULI handler can be entered at almost any time, regardless of what code is being executed by the CPU—a process of your program or a process of another program, executing in user space or in a system function. In fact, the ULI handler can be entered from one CPU while the your program executes concurrently in another CPU. Your normal code and your ULI function can execute in true concurrency, accessing the same global variables.
Because the ULI handler is called in a special context of the kernel's interrupt handler, it is severely restricted in the system facilities it can use. The list of features the ULI handler may not use includes the following:
Any use of floating-point calculations. The kernel does not take time to save floating-point registers during an interrupt trap. The floating-point coprocessor is turned off, and an attempt to use it in the ULI handler causes a SIGILL (illegal instruction) exception.
Any use of IRIX system functions. Because most of the IRIX kernel runs with interrupts enabled, the ULI handler could be entered while a system function was already in progress. System functions do not support reentrant calls. In addition, many system functions can sleep, which an interrupt handler may not do.
Note: Elsewhere in this book you will read that interrupt handlers in IRIX 6.5 run as “threads” and can sleep. While true, this privilege has not yet been extended to user-level interrupt handlers, which are still required never to sleep. Any storage reference that causes a page fault. The kernel cannot suspend the ULI handler for page I/O. Reference to an unmapped page causes a SIGSEGV (memory fault) exception.
Any calls to C library functions that might violate the preceding restrictions.
There are very few library functions that you can be sure use no floating point, make no system calls, and do not cause a page fault. Unfortunately, library functions such as sprintf(), often used in debugging, must be avoided.
In essence, the ULI handler should do only these things, as shown in Figure 7-2:
Store data in program variables in locked pages, to record the interrupt event.
For example, a ring buffer is a data structure that is suitable for concurrent access.
Program the device as required to clear the interrupt or acknowledge it.
The ULI handler has access to the whole program address space, including any mapped-in devices, so it can perform PIO loads and stores.
Post a semaphore to wake up the main process.
This must be done using a ULI function.
Since the ULI handler can interrupt the program at any point, or run concurrently with it, the program must be prepared for concurrent execution. There are two areas to consider: global variables, and library routines.
The asynchronous, possibly concurrent entry to the ULI handler can confuse a debugging monitor such as dbx. Some strategies for dealing with this are covered in the uli(3) man page.
When variables can be modified by both the main process and the ULI handler, you must take special care to avoid race conditions.
An important step is to specify -D_SGI_REENTRANT_FUNCTIONS to the compiler, so as to get the reentrant versions of the C library functions. This ensures that, if the main process and the ULI handler both enter the C library, there is no collision over global variables.
You can declare the global variables that are shared with the ULI handler with the keyword “volatile,” so that the compiler generates code to load the variables from memory on each reference. However, the compiler never holds global values in registers over a function call, and you almost always have a function call (such as ULI_block_intr()) preceding a test of a shared global variable.
The ULI feature allows a program to open more than one interrupting device. You register a handler for each device. However, the program can only wait for a specific interrupt to occur; that is, the ULI_sleep() function specifies the handle of one particular ULI handler. This does not mean that the main program must sleep until that particular interrupt handler is entered, however. Any ULI handler can waken the main program, as discussed under “Interacting With the Handler”.
A program initializes for ULI in the following major steps:
Open the device special file for the device.
For a PCI or VME device, map the device addresses into process memory (see the IRIX Device Driver Programmer's Guide (see “Related Publications and Sites”).
Lock the program address space in memory.
Initialize any data structures used by the interrupt handler.
Register the interrupt handler.
Interact with the device and the interrupt handler.
Any time after the handler has been registered, an interrupt can occur, causing entry to the ULI handler.
Devices are represented by device special files (see the IRIX Device Driver Programmer's Guide (see “Related Publications and Sites”)). In order to gain access to a device, you open the device special file that represents it. The device special files that can generate user-level interrupts include:
The external interrupt line on a Challenge, Onyx, or Origin 200 system, or the base module's external interrupt in an Origin 2000 or Onyx 2 system is /dev/ei. Other external interrupt source devices in an Origin 2000 or Onyx 2 system are mentioned in the IRIX Device Driver Programmer's Guide.
The files that represent PCI bus address spaces are summarized in the pciba(7) man page and the IRIX Device Driver Programmer's Guide.
The files that represent VME control units are summarized in the IRIX Device Driver Programmer's Guide.
The program should open the device and verify that the device exists and is active before proceeding.
The ULI handler must not reference a page of program text or data that is not present in memory. You prevent this by locking the pages of the program address space in memory. The simplest way to do this is to call the mlockall() system function:
if (mlockall(MCL_CURRENT|MCL_FUTURE)<0) perror (“mlockall”); |
The mlockall() function has two possible difficulties. One is that the calling process must have either superuser privilege or CAP_MEMORY_MGT capability (see the mlockall(3C) man page). This may not pose a problem if the program needs superuser privilege in any case, for example, to open a device special file. The second difficulty is that mlockall() locks all text and data pages. In a very large program, this may be so much memory that system performance is harmed.
The mlock() or mpin() functions can be used by unprivileged programs to lock a limited number of pages. The limit is set by the tunable system parameter maxlkmem. (Check its value—typically 2000—in /var/sysgen/mtune/kernel. See the systune(1) man page for instructions on changing a tunable parameter.)
In order to use mlock() or mpin(), you must specify the exact address ranges to be locked. Provided that the ULI handler refers only to global data and its own code, it is relatively simple to derive address ranges that encompass the needed pages. If the ULI handler calls any library functions, the library DSO needs to be locked as well. The smaller and simpler the code of the ULI handler, the easier it is to use mlock() or mpin().
When the program is ready to start operations, it registers its ULI handler. The ULI handler is a function that matches the prototype
void function_name(void *arg); |
The registration function takes arguments with the following purposes:
The file descriptor of the device special file.
The address of the handler function.
An argument value to be passed to the handler on each interrupt. This is typically a pointer to a work area that is unique to the interrupting device (supposing the program is using more than one device).
A count of semaphores to be allocated for use with this interrupt.
An optional address, and the size, of memory to be used as stack space when calling the handler.
Additional arguments for VME and PCI devices.
You can ask the ULI support to allocate a stack space by passing a null pointer for the stack argument. When the ULI handler is as simple a function as it normally is, the default stack size of 1024 bytes is ample.
The semaphores are allocated and maintained by the ULI support. They are used to coordinate between the program process and the interrupt handler, as discussed under “Interacting With the Handler”. You should specify one semaphore for each independent process that can wait for interrupts from this handler. Normally one semaphore is sufficient.
The value returned by the registration function is a handle that is used to identify this interrupt in other functions. Once registered, the ULI handler remains registered until the program terminates (there is no function for un-registration).
The ULI_register_ei() function takes the arguments described in the preceding topic. Once it has successfully registered your handler, all external interrupts are directed to that handler.
It is important to realize that, so long as a ULI handler is registered, none of the other interrupt-reporting features supported by the external interrupt device driver operate any more (see the IRIX Device Driver Programmer's Guide and the ei(7) man page). These restrictions include the facts that:
The per-process external interrupt queues are not updated.
Calls to ioctl(EIIOCRECV) sleep until they are interrupted by a timeout, a signal, or because the program using ULI terminated and an interrupt arrived.
Calls to the library function eicbusywait_f() do not terminate.
Clearly you should not use ULI for external interrupts when there are other programs running that also use them.
The ULI_register_vme() function takes the following additional arguments:
The interrupt level that the device uses
A word that contains, or receives, an interrupt vector number (sometimes referred to as the status or ID)
The interrupt level used by a device is normally set by hardware and documented in the VECTOR line that defines the device (see the IRIX Device Driver Programmer's Guide).
Some VME devices have a fixed interrupt vector number; others are programmable. You pass a fixed vector number to the function. If the number is programmable, you pass 0, and the function allocates a number. You must then use PIO to program the vector number into the device.
The program process and the ULI handler synchronize their actions using two functions.
When the program cannot proceed without an interrupt, it calls ULI_sleep(), specifying the following:
The handle of the interrupt for which to wait
The number of the semaphore to use for waiting
Typically only one process ever calls ULI_sleep() and it specifies waiting on semaphore 0. However, it is possible to have two or more processes that wait. For example, if the device can produce two distinct kinds of interrupts—normal and high-priority, perhaps—you could set up an independent process for each interrupt type. One would sleep on semaphore 0, the other on semaphore 1.
When an ULI handler is entered, it wakes up a program process by calling ULI_wakeup(), specifying the semaphore number to be posted. The handler must know which semaphore to post, based on the values it can read from the device or from program variables.
The ULI_sleep() call can terminate early, for example if a signal is sent to the process. The process that calls ULI_sleep() must test to find the reason the call returned—it is not necessarily because of an interrupt.
The ULI_wakeup() function can be called from normal code as well as from a ULI handler function. It could be used within any type of asynchronous callback function to wake up the program process.
The ULI_wakeup() call also specifies the handle of the interrupt. When you have multiple interrupting devices, you have the following design choices:
You can have one child process waiting on the handler for each device. In this case, each ULI handler specifies its own handle to ULI_wakeup().
You can have a single process that waits on any interrupt. In this case, the main program specifies the handle of one particular interrupt to ULI_sleep(), and every ULI handler specifies that same handle to ULI_wakeup().
The program can gain exclusive use of global variables with a call to ULI_block_intr(). This function does not block receipt of the hardware interrupt, but does block the call to the ULI handler. Until the program process calls ULI_unblock_intr(), it can test and update global variables without danger of a race condition. This period of time should be as short as possible, because it extends the interrupt latency time. If more than one hardware interrupt occurs while the ULI handler is blocked, it is called for only the last-received interrupt.
There are other techniques for safe handling of shared global variables besides blocking interrupts. One important, and little-known, set of tools is the test_and_set() group of functions documented in the test_and_set(3) man page. These instructions use the Load Linked and Store Conditional instructions of the MIPS instruction set to safely update global variables in various ways.
This section contains two programs to show how user-level interrupts are used.
The program listed in Example 7-1 is a hypothetical example of how user-level interrupts can be used to handle interrupts from the PCI bus in an Onyx 2/Origin 2000 system
The program listed in Example 7-2 is a hypothetical example of how user-level interrupts can be used to handle external interrupts in a Challenge and Onyx system.
Example 7-1. Hypothetical PCI ULI Program
/* * pci40_uli.c - PCI User Level Interrupt (ULI) test using the * Greenspring PCI40 IP carrier card to generate * interrupts. * * This version for Onyx 2/Origin 2000 systems (Origin 200 systems * will have a different hwgraph path.) * * link with -luli * * Make sure that the latest 6.5 REACT/pro, PCI and kernel * roll-up patches are installed. * */ #include <sys/types.h> #include <sys/mman.h> #include <sys/fcntl.h> #include <sys/prctl.h> #include <unistd.h> #include <stdio.h> #include <sys/syssgi.h> #include <sys/sysmp.h> #include <sched.h> #include <sys/uli.h> #define INTRPATH "/hw/module/1/slot/io2/pci_xio/pci/2/intr" #define PCI40_PATH "/hw/module/1/slot/io2/pci_xio/pci/2/base/2" #define PLX_PATH "/hw/module/1/slot/io2/pci_xio/pci/2/base/0" #define PCI40_SIZE (1024*1024) #define PLX_SIZE 128 #define PCI_INTA 0x01 #define NUM_INTS 1000000 #define BAD_RESPONSE 30 #define PROC 0 extern int errno; int intr; static void *ULIid; volatile uchar_t *pci40_addr; /* definitions for timer */ typedef unsigned long long iotimer_t; __psunsigned_t phys_addr, raddr; unsigned int cycleval; volatile iotimer_t begin_time, end_time, *timer_addr; int timer_fd, poffmask; float usec_time; int bad_responses = 0; float longest_response = 0.0; float average_response = 0.0; static void intrfunc(void *arg) { end_time = *timer_addr; /* Set the global flag indicating to the main thread that an * interrupt has occurred, and wake it up */ intr++; /* * clear the interrupt on the mothercard by clearing CNTRL0 * adding 1 to offset for big endian access */ *(unsigned char *)(pci40_addr+0x501) = 0x00; } main(int argc, char *argv[]) { int fd; int pci_fd; int plx_fd; int cpu; int multi_cpus = 0; volatile uint_t *plx_addr; volatile uint_t x; float fres; double total = 0; struct sched_param sparams; struct timespec wait_time; /* * do the appropriate real-time things */ sparams.sched_priority = sched_get_priority_max(SCHED_FIFO); if (sched_setscheduler(0, SCHED_FIFO, &sparams) < 0) { perror("psched: ERROR - sched_setscheduler"); exit(1); } if (mlockall( MCL_CURRENT | MCL_FUTURE )){ perror ("mlockall"); } /* * be sure there are multiple cpus present before * attempting to run on an isolated cpu - once * verified, isolate and make non-preemptive * the cpu, then force the process to execute there * */ cpu = sysmp(MP_NPROCS) - 1; if (cpu>0) { multi_cpus = 1; if (sysmp(MP_ISOLATE,cpu)) { perror("sysmp-MP_ISOLATE"); exit(1); } if (sysmp(MP_NONPREEMPTIVE,cpu)) { perror("sysmp-MP_NONPREEMPTIVE"); exit(1); } if (sysmp(MP_MUSTRUN,cpu)) { perror("sysmp-MP_MUSTRUN"); exit(1); } } /* * memory map the hardware cycle-counter */ poffmask = getpagesize() - 1; phys_addr = syssgi(SGI_QUERY_CYCLECNTR, &cycleval); raddr = phys_addr & ~poffmask; timer_fd = open("/dev/mmem", O_RDONLY); timer_addr = (volatile iotimer_t *)mmap(0, poffmask, PROT_READ, MAP_PRIVATE, timer_fd, (off_t)raddr); timer_addr = (iotimer_t *)((__psunsigned_t)timer_addr + (phys_addr & poffmask)); fres = ((float)cycleval)/1000000.0; /* * open the PCI user interrupt device/vertex */ fd = open(INTRPATH, O_RDWR); if (fd < 0 ) { perror(INTRPATH); exit (1); } /* * open the PLX register space on the PCI40 card */ plx_fd = open(PLX_PATH, O_RDWR); if (plx_fd < 0 ) { perror(PLX_PATH); exit (1); } /* * open the PCI40 memory space for device registers */ pci_fd = open(PCI40_PATH, O_RDWR); if (pci_fd < 0 ) { perror(PCI40_PATH); exit (1); } /* * map in the PLX register space on the PCI40 card */ plx_addr = (volatile uint_t *) mmap(0, PLX_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, plx_fd, 0); if (plx_addr == (uint_t *) MAP_FAILED) { perror("mmap plx_addr"); exit (1); } /* * set up the PLX register to pass through the interrupt */ x = *(volatile uint_t *)(plx_addr + 0x1a); *(volatile uint_t *)(plx_addr + 0x1a) = x | 0x00030f00; /* * map in the PCI40 memory space for device registers */ pci40_addr = (volatile uchar_t *) mmap(0, PCI40_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, pci_fd, 0); if (pci40_addr == (uchar_t *) MAP_FAILED) { perror("mmap"); exit (1); } /* * clear the interrupt on the mothercard by clearing CNTRL0 * adding 1 to offset for big endian access */ *(unsigned char *)(pci40_addr+0x501) = 0x00; /* * Register the pci interrupt as a ULI source. */ ULIid = (int *)ULI_register_pci(fd, /* the pci interrupt device */ intrfunc, /* the handler function pointer */ 0, /* the argument to the handler */ 0, /* the # of semaphores needed */ 0, /* the stack to use */ 0, /* the stack size to use */ PCI_INTA);/* PCI interrrupt line */ if (ULIid == 0) { perror("register uli"); exit(1); } printf ("Registered successfully for PCI INTA - Sending interrupts\n"); /* * Ask for 200 usec wait time - resolution on Origin is * really only ~1.5 ms instead */ wait_time.tv_sec = 0; wait_time.tv_nsec = 200000; while(intr < NUM_INTS) { /* * then, enable the interrupt on the PCI carrier * card - adding 1 to offset for big endian access */ begin_time = *timer_addr; *(unsigned char *)(pci40_addr+0x501) = 0xc0; nanosleep(&wait_time,NULL); usec_time = (end_time-begin_time)*fres; if (usec_time > BAD_RESPONSE) { bad_responses++; } if ((usec_time > longest_response) && (intr > 5)) longest_response = usec_time; total += usec_time; average_response = total/(float)intr; if (!(intr % 1000)&&(intr>0)) { printf(" Average ULI Response (%d interrupts):\t %4.2f usecs\n", intr,average_response); printf(" Number of Interrupts > %d usecs:\t\t %d \n", BAD_RESPONSE,bad_responses); } } printf(" Average ULI Response (%d interrupts):\t %4.2f usecs \n", intr,average_response); printf(" Number of Interrupts > %d usecs:\t\t %d \n", BAD_RESPONSE,bad_responses); printf(" Longest ULI Response:\t\t\t\t\t %4.2f \n", longest_response); if (multi_cpus) { sysmp( MP_PREEMPTIVE, cpu ); sysmp( MP_UNISOLATE, cpu ); } }Example 7-2. Hypothetical External Interrupt ULI Program
/* This program demonstrates use of the External Interrupt source * to drive a User Level Interrupt. * * The program requires the presence of an external interrupt cable looped * back between output number 0 and one of the inputs on the machine on * which the program is run. */ #include <sys/ei.h> #include <sys/uli.h> #include <sys/lock.h> #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <fcntl.h> /* The external interrupt device file is used to access the EI hardware */ #define EIDEV "/dev/ei" static int eifd; /* The user level interrupt id. This is returned by the ULI registration * routine and is used thereafter to refer to that instance of ULI */ static void *ULIid; /* Variables which are shared between the main process thread and the ULI * thread may have to be declared as volatile in some situations. For * example, if this program were modified to wait for an interrupt with * an empty while() statement, e.g. * while(!intr); * the value of intr would be loaded on the first pass and if intr is * false, the while loop will continue forever since only the register * value, which never changes, is being examined. Declaring the variable * intr as volatile causes it to be reloaded from memory on each iteration. * In this code however, the volatile declaration is not necessary since * the while() loop contains a function call, e.g. * while(!intr) * ULI_sleep(ULIid, 0); * The function call forces the variable intr to be reloaded from memory * since the compiler cannot determine if the function modified the value * of intr. Thus the volatile declaration is not necessary in this case. * When in doubt, declare your globals as volatile. */ static int intr; /* This is the actual interrupt service routine. It runs * asynchronously with respect to the remainder of this program, possibly * simultaneously, on an MP machine. This function must obey the ULI mode * restrictions, meaning that it may not use floating point or make * any system calls. (Try doing so and see what happens.) */ static void intrfunc(void *arg) { /* Set the global flag indicating to the main thread that an * interrupt has occurred, and wake it up */ intr = 1; ULI_wakeup(ULIid, 0); } /* This function creates a new process and from it, generates a * periodic external interrupt. */ static void signaler(void) { int pid; if ((pid = fork()) < 0) { perror("fork"); exit(1); } if (pid == 0) { while(1) { if (ioctl(eifd, EIIOCSTROBE, 1) < 0) { perror("EIIOCSTROBE"); exit(1); } sleep(1); } } } /* The main routine sets everything up, then sleeps waiting for the * interrupt to wake it up. */ int main() { /* open the external interrupt device */ if ((eifd = open(EIDEV, O_RDONLY)) < 0) { perror(EIDEV); exit(1); } /* Set the target cpu to which the external interrupt will be * directed. This is the cpu on which the ULI handler function above * will be called. Note that this is entirely optional, but if * you do set the interrupt cpu, it must be done before the * registration call below. Once a ULI is registered, it is illegal * to modify the target cpu for the external interrupt. */ if (ioctl(eifd, EIIOCSETINTRCPU, 1) < 0) { perror("EIIOCSETINTRCPU"); exit(1); } /* Lock the process image into memory. Any text or data accessed * by the ULI handler function must be pinned into memory since * the ULI handler cannot sleep waiting for paging from secondary * storage. This must be done before the first time the ULI handler * is called. In the case of this program, that means before the * first EIIOCSTROBE is done to generate the interrupt, but in * general it is a good idea to do this before ULI registration * since with some devices an interrupt may occur at any time * once registration is complete */ if (plock(PROCLOCK) < 0) { perror("plock"); exit(1); } /* Register the external interrupt as a ULI source. */ ULIid = ULI_register_ei( eifd, /* the external interrupt device */ intrfunc, /* the handler function pointer */ 0, /* the argument to the handler */ 1, /* the number of semaphores needed */ NULL, /* the stack to use (supply one) */ 0); /* the stack size to use (default) */ if (ULIid == 0) { perror("register ei"); exit(1); } /* Enable the external interrupt. */ if (ioctl(eifd, EIIOCENABLE) < 0) { perror("EIIOCENABLE"); exit(1); } /* Start creating incoming interrupts. */ signaler(); /* Wait for the incoming interrupts and report them. Continue * until the program is terminated by ^C or kill. */ while (1) { intr = 0; while(!intr) { if (ULI_sleep(ULIid, 0) < 0) { perror("ULI_sleep"); exit(1); } printf("sleeper woke up\n"); } }