When a Frame Scheduler takes over scheduling and dispatching threads on one CPU, it isolates the CPU (see “Isolating a CPU from TLB Interrupts” in Chapter 3), and completely supersedes the operation of the normal IRIX scheduler on that CPU. Only threads queued to the Frame Scheduler can use the CPU. IRIX thread dispatching priorities are not relevant on that CPU.

The execution of normal processes, daemons, and pending timeouts are all migrated to other CPUs—typically to CPU 0, which cannot be owned by a Frame Scheduler. All interrupt handling is usually directed away from a Frame Scheduler CPU as well (see “Preparing the System”). However, a Frame Scheduler CPU can be used to handle interrupts, although doing so runs a risk of causing overruns.

The Frame Scheduler in REACT/pro version 3.2 supports two thread programming models: sprocs and pthreads. Both threading models allow multiprogramming, but sprocs are proprietary to Silicon Graphics, while pthreads are standardized by the IEEE POSIX 1003.1c specification.

In this guide, a thread is defined as an independent flow of execution that consists of a set of registers (including a program counter and a stack).

A traditional IRIX process has a single active thread that starts once the program is executed and runs until the program terminates. A multithreaded process may have several threads active at one time. Hence, a process can be viewed as a receptacle that contains the threads of execution and the resources they share (that is, data segments, text segments, file descriptors, synchronizers, and so forth).

Instead of scheduling threads according to priorities, the Frame Scheduler dispatches them according to a strict, cyclic rotation governed by a repetitive time base. The time base determines the fundamental frame rate. (See “Selecting a Time Base”.) Some examples of the time base are as follows:

The interrupts from the time base define minor frames. Together, a fixed number of minor frames make up a major frame. The length of a major frame defines the application's true frame rate. The minor frames allow you to divide a major frame into subframes. Major and minor frames are shown in Figure 4-1.

In the simplest case, you have a single frame rate, such as 60 Hz, and every activity your program does must be done once per frame. In this case, the major and minor frame rates are the same.

In other cases, you have some activities that must be done in every minor frame, but you also have activities that are done less often: in every other minor frame or in every third one. In these cases, you define the major frame so that its rate is the rate of the least-frequent activity. The major frame contains as many minor frames as necessary to schedule activities at their relative rates.

Figure 4-1. Major and Minor Frames

As pictured in Figure 4-1, the Frame Scheduler maintains a queue of threads for each minor frame. Queue each activity thread of your program to a specific minor frame. Determine the order of cyclic execution within a minor frame by the order in which you queue threads. You can do the following:

All these options are controlled by scheduling disciplines you specify for each thread as you queue it (see “Using the Scheduling Disciplines”).

Typically a frame scheduler is driven by a single interrupt source and contains minor frames having the same duration, but a variable frame scheduler may be used to implement an FRS having multiple interrupt sources and/or minor frames of variable duration (see the frs_create_vmaster() function).

The relationship between threads and a Frame Scheduler depends upon the thread model in use, as follows:

See “Implementing a Single Frame Scheduler” for details.

An overview of the Frame Scheduler API can be found in the frs(3) man page, which provides a complete listing of all the FRS functions. Separate man pages for each of the FRS functions provide the details of the Frame Scheduler API. The API elements are declared in /usr/include/sys/frs.h. The following are some important types that are declared in /usr/include/sys/frs.h:

Additionally the pthreads interface adds the following types, as declared in /usr/include/sys/pthread.h:

Variable frame schedulers may drive each minor frame with a different interrupt source, as well as define a different duration for each minor frame. These two characteristics may be used together or separately, and are defined using an interrupt information template.

An interrupt information template consists of an array of frs_intr_info_t data structures, where each element in the array represents a minor frame. For example, the first element in the array represents the interrupt information for the first minor frame, and so on for n minor frames.

The frs_intr_info_t data structure contains two fields for defining the interrupt source and its qualifier: intr_source and intr_qualifier.

The following example demonstrates how to define an interrupt information template for a frame scheduler having minor frames of different duration. Assume the application requires four minor frames, where each minor frame is triggered by the synchronized clock timer, and the duration of each minor frame is as follows: 100 ms, 150 ms, 200 ms, and 250 ms. The interrupt information template may be defined as follows:

frs_intr_info_t intr_info[4];
intr_info[0].intr_source    = FRS_INTRSOURCE_CCTIMER;
intr_info[0].intr_qualifier = 100000;
intr_info[1].intr_source    = FRS_INTRSOURCE_CCTIMER;
intr_info[1].intr_qualifier = 150000;
intr_info[2].intr_source    = FRS_INTRSOURCE_CCTIMER;
intr_info[2].intr_qualifier = 200000;
intr_info[3].intr_source    = FRS_INTRSOURCE_CCTIMER;
intr_info[3].intr_qualifier = 250000;

The following example demonstrates how to define an interrupt information template for a frame scheduler using multiple interrupt sources. Assume the application requires two minor frames, where the first minor frame is triggered by the vertical retrace interrupt and the second minor frame is triggered by the CPU timer. Also assume the vertical retrace interrupt is running at 60 Hz (every 16.6 ms). The following interrupt information template defines the CPU timer interrupt of the second frame to fire 8.3 ms after the vertical retrace interrupt:

frs_intr_info_t intr_info[2];
intr_info[0].intr_source    = FRS_INTRSOURCE_VSYNC;
intr_info[0].intr_qualifier = 0;
intr_info[1].intr_source    = FRS_INTRSOURCE_CPUTIMER;
intr_info[1].intr_qualifier = 8300;

Note that 8.3 ms was chosen in the example because it is known that the timer interrupt will fire before the next major frame's vsync interrupt. If 20 ms were chosen for the timer instead, then a sequence error would occur (see section “Sequence Error”) and an error signal would be sent to the controller thread.

Detailed programming examples are available, demonstrating use of variable frame schedulers, which can be found in the /usr/share/src/react/examples directory. For more information, see the frs_create_vmaster(3) man page.

System Call Interface for Fortran and Ada

Each Frame Scheduler function is available in two ways: as a system call to schedctl(), or as one or more library calls to functions in the frs library, /usr/lib/libfrs.a. The system call is accessible from FORTRAN and Ada programs because both languages have bindings for schedctl() (see the schedctl(2) man page). The correspondence between the library functions and schedctl() calls is shown in Table 4-2.

---

Note: The pthread functions for the Frame Scheduler are not supported for FORTRAN applications.

---

Table 4-2. Frame Scheduler schedctl() Support

Library Function	Schedctl Syntax
frs_create()	int schedctl( MPTS_FRS_CREATE, frs_info_t* frs_info);
frs_enqueue()	int schedctl( MPTS_FRS_ENQUEUE, frs_queue_info_t* frs_queue_info);
frs_join()	int schedctl( MPTS_FRS_JOIN, pid_t frs_master);
frs_start()	int schedctl( MPTS_FRS_START, pid_t frs_master);
frs_yield()	int schedctl( MPTS_FRS_YIELD);
frs_stop()	int schedctl( MPTS_FRS_STOP, pid_t frs_master);
frs_resume()	int schedctl( MPTS_FRS_RESUME, pid_t frs_master);
frs_destroy()	int schedctl( MPTS_FRS_DESTROY, pid_t frs_master);
frs_getqueuelen()	int schedctl( MPTS_FRS_GETQUEUELEN, frs_queue_info_t* frs_queue_info);
frs_readqueue()	int schedctl( MPTS_FRS_READQUEUE, frs_queue_info_t* frs_queue_info, pid_t* pidlist);
frs_premove()	int schedctl( MPTS_FRS_PREMOVE, frs_queue_info_t* frs_queue_info);
frs_pinsert()	int schedctl( MPTS_FRS_PINSERT, frs_queue_info_t* frs_queue_info, pid_t *base_pid);
frs_getattr()	int schedctl (MPTS_FRS_GETATTR, frs_attr_info_t* frs_attr_info);
frs_setattr()	int schedctl( MPTS_FRS_SETATTR, frs_attr_info_t* frs_attr_info);

An activity thread that is queued to a Frame Scheduler has the basic structure shown in Example 4-1.

/* Initialize data structures etc. */
frs_join(scheduler-handle)
do
{
   /* Perform the activity. */
   frs_yield();
} while(1);
_exit(); 

When the thread is ready to start real-time execution, it calls frs_join(). This call blocks until all queued threads are ready and scheduling begins (see “Starting Multiple Schedulers”). When frs_join() returns, the thread is running in its first minor frame. For more information about frs_join(), see frs_join(3).

The thread then performs whatever activity is needed to complete the minor frame and calls frs_yield(). This gives up control of the CPU until the next minor frame where the thread is queued and executes. For more information about frs_yield(), see frs_yield(3).

An activity thread is never preempted within a minor frame. As long as it yields before the end of the frame, it can do its assigned work without interruption from other threads (it can be interrupted by hardware interrupts, if any hardware interrupts are allowed in that CPU). The Frame Scheduler preempts the thread at the end of the minor frame.

---

Tip: Because an activity thread cannot be preempted, it can often use global data without locks or semaphores. When the thread that modifies a global variable is queued in a different minor frame than the threads that read the variable, there can be no access conflicts between them.

---

Conflicts are still possible between two threads that are queued to the same minor frame in different, synchronized Frame Schedulers. However, such threads are guaranteed to be running concurrently. This means they can use spin-locks (see “Locks” in Chapter 2) with high efficiency.

---

Tip: When a very short minor frame interval is used, it is possible for a thread to have an overrun error in its first frame due to cache misses. A simple variation on the basic structure shown in Example 4-1 is to spend the first minor frame touching a set of important data structures in order to “warm up” the cache. This is sketched in Example 4-2.

---

/* Initialize data structures etc. */
frs_join(scheduler-handle); /* Much time could pass here. */
/* First frame: merely touch important data structures. */
do
{
   frs_yield();
   /* Second and later frames: perform the activity. */
} while(1);
_exit(); 

When an activity thread is scheduled on more than one minor frame in a major frame, it can be designed to do nothing except warm the cache in the entire first major frame. To do this, the activity thread function has to know how many minor frames it is scheduled on, and calls frs_yield() that many times in order to pass the first major frame.

Threads in a minor frame queue are dispatched in the order they appear on the queue (priority is irrelevant). Queue ordering can be modified as follows:

See the man pages frs_enqueue(3), frs_pinsert(3), frs_premove(3), and “Managing Activity Threads”.

When the activities of one frame cannot be completed by one CPU, you need to recruit additional CPUs and execute some activities concurrently. However, it is important that each of the CPUs have the same time base, so that each starts and ends frames at the same time.

You can create one master Frame Scheduler, which owns the time base and one CPU, and as many synchronized (slave) Frame Schedulers as you need, each managing an additional CPU. The slave schedulers take their time base from the master, so that all start minor frames at the same instant.

Each FRS requires its own controller thread. Therefore, to create multiple, synchronized Frame Schedulers, you must create a controller thread for the master and each slave FRS.

Each Frame Scheduler has its own queues of threads. A given thread can be queued to only one CPU. (However, you can create multiple threads based on the same code, and queue each to a different CPU.) All synchronized Frame Schedulers use the same number of minor frames per major frame, which is taken from the definition of the master FRS.

A single Frame Scheduler is created when the FRS controller thread calls frs_create_master() or frs_create(). The FRS controller calls frs_pthread_enqueue() or frs_enqueue() one or more times to notify the new Frame Scheduler of the threads to schedule in each of the minor frames. The FRS controller calls frs_start() when it has queued all the threads. Each scheduled thread must call frs_join() after it has initialized and is ready to be scheduled.

Each activity thread must be queued to at least one minor frame before it can join the FRS via frs_join(). Once all activity threads have joined and the FRS is started by the controller thread, the first minor frame begins executing. For more information about these functions, see the frs_enqueue(3), frs_join(3), and frs_start(3) man pages.

A Frame Scheduler cannot start dispatching activities until the following has occurred:

When multiple Frame Schedulers are used, none can start until all are ready.

Each FRS controller notifies its Frame Scheduler that it has queued all activities by calling frs_start(). Each activity thread signals its Frame Scheduler that it is ready to begin real-time processing by calling frs_join().

A Frame Scheduler is ready when it has received one or more frs_pthread_enqueue() or frs_enqueue() calls, a matching number of frs_join() calls, and an frs_start() call for each Frame Scheduler. Each slave Frame Scheduler notifies the master Frame Scheduler when it is ready. When all the schedulers are ready, the master Frame Scheduler gives the downbeat, and the first minor frame begins.

Any Frame Scheduler can be made to pause and restart. Any thread (typically but not necessarily the FRS controller) can call frs_stop(), specifying a particular Frame Scheduler. That scheduler continues dispatching threads from the current minor frame until all have yielded. Then it goes into an idle loop until a call to frs_resume() tells it to start. It resumes on the next time-base interrupt, with the next minor frame in succession. For more information, see the frs_stop(3) and frs_resume(3) man pages.

---

Note: If there is a thread running Background discipline in the current minor frame, it continues to execute until it yields or is blocked on a system service.

---

Since a Frame Scheduler does not stop until the end of a minor frame, you can stop and restart a group of synchronized schedulers by calling frs_stop() for each one before the end of a minor frame. There is no way to restart all of a group of schedulers with the certainty that they start up on the same time-base interrupt.

The FRS control thread identifies the initial set of activity threads by calling frs_pthread_enqueue() or frs_enqueue() prior to starting the Frame Scheduler. All the queued threads must call frs_join() before scheduling can begin. However, the FRS controller can change the set of activity threads dynamically while the Frame Scheduler is working, using the following functions:

Using these functions, the FRS controller can change the queueing discipline (overrun, underrun, continuable) of a thread by removing it and inserting it with a new discipline. The FRS controller can suspend a thread by removing it from its queue; or can restart a thread by putting it back in its queue.

---

Note: When an activity thread is removed from the last or only queue it was in, it is returned to the normal IRIX scheduler and can begin to execute on another CPU. When an activity thread is removed from a queue, a signal may be sent to the removed thread (see “Handling Signals in an Activity Thread”). If a signal is sent to it, it begins executing in its specified or default signal handler; otherwise, it simply begins executing following frs_yield(). Once returned to the IRIX scheduler, a call to an FRS function such as frs_yield() returns an error (this also can be used to indicate the resumption of normal scheduling).

---

The FRS controller can also queue new threads that have not been scheduled before. The Frame Scheduler does not reject an frs_pthread_insert() or frs_pinsert() call for a thread that has not yet joined the scheduler. However, a thread must call frs_join() before it can be scheduled. For more information, see the frs_pinsert(3) man page.

If an queued thread should be terminated for any reason, the Frame Scheduler removes the thread from all queues in which it appears.

Each processor chip contains a free-running timer that is used by IRIX for normal process scheduling. This timer is not synchronized between processors, so it cannot be used to drive multiple synchronized schedulers. The on-chip timer can be used as a time base when only one CPU is used.

To use the on-chip timer, specify FRS_INTRSOURCE_CPUTIMER as the interrupt source, and the minor frame interval in microseconds, to frs_create_master() or frs_create().

The high-resolution timer and clock is a timer that is synchronous across all processors, and is ideal to drive synchronous schedulers. On Origin, Onyx 2, CHALLENGE, and Onyx systems, this timer is based on the high-resolution counter discussed under “Hardware Cycle Counter” in Chapter 2.

To use this timer, specify FRS_INTRSOURCE_CCTIMER, and specify the minor frame interval in microseconds to frs_create_master() or frs_create().

The IRIX kernel uses this timer for managing timer events. When your program creates the master Frame Scheduler, the Frame Scheduler migrates all timeout events to CPU 0, leaving the timer on the scheduled CPU free.

The high-resolution timers in all CPUs are synchronized automatically.

An interrupt is generated for every vertical retrace by the graphics subsystem (see “Understanding the Vertical Sync Interrupt” in Chapter 3). The frame rate is either 50 Hz or 60 Hz, depending on the installed hardware. This interrupt is especially appropriate for a visual simulator, since it defines a frame rate that matches the graphics subsystem frame rate.

To use the vertical sync interrupt, specify FRS_INTRSOURCE_VSYNC to frs_create_master() or frs_create(). An error is returned if this system is not configured with a graphics subsystem.

When multiple synchronized schedulers are used, the master Frame Scheduler distributes the vertical sync interrupt.

An external interrupt is generated via a signal applied to the external interrupt socket on systems supporting such a hardware feature, such as Origin, Challenge, and Onyx systems (see “External Interrupts” in Chapter 6). To use external interrupts as a time base, use the following steps:

For example, in irix.sm, a directive similar to the following causes PCI interrupt 4 of the first I/O slot to be handled by CPU 1. (The actual directive depends on the hardware configuration of the target platform.)

DEVICE_ADMIN: /hw/module/1/slot/io1/baseio/pci/4 INTR_TARGET=/hw/cpunum/1

When multiple synchronized schedulers are used, the master Frame Scheduler receives the interrupt and allocates it simultaneously to the synchronized schedulers.

---

Note: When the external interrupt is routed to a CPU, all of the interrupt threads that are triggered from the same physical interrupt will be bound to that CPU as well. When the FRS assumes scheduling responsibility for that CPU, to avoid jitter and potential delay, those threads will be prevented from running. If you still need access to the other devices in the IOC3 module that holds the external interrupt port, you should use another IOC3 module.

---

A user-written, kernel-level device driver can supply the time-base interrupt (see “FRS Kernel-Level Device Driver Interface”). The Frame Scheduler registers the driver and assigns it a unique registration number, then allocates an interrupt group. The device driver must direct interrupts to it.

To use a device driver as a time base, specify FRS_INTRSOURCE_DRIVER and the device driver's registration number to frs_create_master() or frs_create(). See “Implementing a Single Frame Scheduler”.

A programmed, software-generated interrupt can be used as the time base. Any user process can send this interrupt to the master Frame Scheduler by calling frs_userintr().

---

Note: Software interrupts are primarily intended for application debugging. It is not feasible for a user process to generate the low-latency and determinism for interrupts required by a real-time application.

---

To use software interrupts as a time base, specify FRS_INTRSOURCE_USER to frs_create_master() or frs_create().

---

Caution: The use of software interrupts has a potential for causing a system deadlock if the interrupt-generating process contends for a resource that is also used by a frame-scheduled activity thread. If any activity thread calls IRIX system functions, the only way to be absolutely sure of avoiding deadlock is for the interrupt-generating process to avoid using any IRIX system functions. Note that C library functions such as printf() invoke system functions, and can lead to deadlocks in this case.

---

The user level interrupt (ULI) facility allows a hardware interrupt to be handled by a user process, enabling device drivers to reside at the user-level; see the uli(3) man page for details. To use ULI to drive the frame scheduler, specify FRS_INTRSOURCE_ULI as the interrupt source to the frs_create_master() or frs_create_vmaster() functions.

The ULI capability is supported only for Origin and Onyx 2 platforms. The PCI or VME interrupt must be routed to the master Frame Scheduled processor in order for the frame scheduler to recognize it.

---

Note: The frame scheduler is invoked after the user-level portion of the interrupt handler has completed servicing the interrupt.

---

In the simplest case, an activity thread starts during the minor frame in which it is queued, and completes its work and yields within the same minor frame.

If the thread is not ready to run (for example, blocked on I/O) during the entire minor frame, an underrun exception is said to occur. If the thread fails to complete its work and yield within the minor frame interval, an overrun exception is said to occur.

The Frame Scheduler calls this strict discipline the Real-time scheduling discipline.

This model could describe a simple kind of simulator in which certain activities—poll the inputs; calculate the new status; update the display—must be repeated in that order during every frame. In this scenario, each activity must start and must finish in every frame. If one fails to start, or fails to finish, the real-time program is broken in some way and must take some action.

However, realistic designs need the flexibility to have threads with the following characteristics:

The other disciplines are used, in combination with Real-time and with each other, to allow these variations.

The Background discipline is mutually exclusive with the other disciplines. The Frame Scheduler dispatches a Background thread only when all other threads queued to that minor frame have run and have yielded. Since the Background thread cannot be sure it will run and cannot predict how much time it will have, the concepts of underrun and overrun do not apply to it.

---

Note: A thread with the Background discipline must be queued to its frame following all non-Background threads. Do not queue a real-time thread after a Background thread.

---

You specify Underrunable discipline with Real-time discipline to prevent detection of underrun exceptions. You specify Underrunable in the following cases:

When you specify Real-time+Underrunable, the thread is not required to start in that minor frame. However, if it starts, it is required to yield before the end of the frame or an overrun exception is raised.

You specify Overrunnable discipline with Real-time discipline to prevent detection of overrun exceptions. You specify it in the following cases:

When you specify Overrunnable+Real-time, the thread is not required to call frs_yield() before the end of the frame. Even so, the thread is preempted at the end of the frame. It does not have a chance to run again until the next minor frame in which it is queued. At that time it resumes where it was preempted, with no indication that it was preempted.

You specify Continuable discipline with Real-time discipline to prevent the Frame Scheduler from clearing the flags at the end of this minor frame (see “Scheduling Within a Minor Frame”).

The result is that, if the thread yields in this frame, it need not run or yield in the following frame. The residual frs_yield flag value, carried forward to the next frame, applies. You specify Continuable discipline with other disciplines in order to let a thread execute just once in a block of consecutive minor frames.

There are cases when a thread sometimes or always requires more than one minor frame to complete its work. Possibly the work is lengthy, or possibly the thread could be delayed by a system call or a lock or semaphore wait.

You must decide the absolute maximum time the thread could consume between starting up and calling frs_yield(). If this is unpredictable, or if it is predictably longer than the major frame, the thread cannot be scheduled by the Frame Scheduler. Hence, it should probably run on another CPU under the IRIX real-time scheduler.

However, when the worst-case time is bounded and is less than the major frame, you can queue the thread to enough consecutive minor frames to allow it to finish. A combination of disciplines is used in these frames to ensure that the thread starts when it should, finishes when it must, and does not cause an error if it finishes early.

The discipline settings for each frame should be as follows:

A thread can be queued for one or more of these multiframe sequences in one major frame. For example, suppose that the minor frame rate is 60 Hz, and a major frame contains 60 minor frames (1 Hz). You have a thread that should run at a rate of 5 Hz and can use up to 3/60 second at each dispatch. You can queue the thread to 5 sequences of 3 consecutive frames each. It could start in frames 0, 12, 24, 36, and 48. Frames 1, 13, 25, 37 and 49 could be intermediate frames, and 2, 14, 26, 38 and 50 could be final frames.

The first Frame Scheduler provides the time base for the others. It is called the master scheduler. The other schedulers take their time base interrupts from the master, and so are called slaves. The combination is called a sync group.

No single thread may create more than one Frame Scheduler. This is because every Frame Scheduler must have a unique FRS controller thread to which it can send signals. As a result, the program has three types of threads:

The master Frame Scheduler must be created before any slave Frame Schedulers can be created. Slave Frame Schedulers must be specified to have the same time base and the same number of minor frames as the master.

Slave Frame Schedulers can be stopped and restarted independently. However, when any scheduler, master or slave, is destroyed, all are immediately destroyed.

A variety of program designs are possible but the simplest is possibly the set of steps described in the following paragraphs.

The master controller thread performs these steps:

Each slave controller thread performs these steps:

For an example of this kind of program structure, refer to “Examples of Multiple Synchronized Schedulers” in Appendix A.

---

Tip: In this design sketch, the knowledge of which activity threads to create, and on which frames to queue them, is distributed throughout the code of multiple threads, where it might be hard to maintain. However, it is possible to centralize the plan of schedulers, activities, and frames in one or more arrays that are statically initialized. This improves the maintainability of a complex program.

---

The Overrun exception indicates that a thread failed to yield in a minor frame where it was expected to yield, and was preempted at the end of the frame. An Overrun exception indicates that an unknown amount of work that should have been done was not done, and will not be done until the next frame in which the overrunning thread is queued.

The Underrun exception indicates that a thread that should have started in a minor frame did not start. Possibly the thread has terminated. More likely it was blocked in some kind of wait because of an unexpected delay in I/O, or a deadlock on a lock or semaphore.

The FRS control thread can establish one of four policies for handling overrun and underrun exceptions. When it detects an exception, the Frame Scheduler can do the following:

The default action is to send a signal (the specific signals are listed under “Setting Frame Scheduler Signals”). The scheduler continues to run. The FRS control thread can then take action, for example, terminating the Frame Scheduler.

The f rs_pthread_setattr() or f rs_setattr() function is used to change exception policies. This function must be called before the Frame Scheduler is started. After scheduling has begun, an attempt to change the policies or signals is rejected.

In order to allow for future enhancements, frs_pthread_setattr() or frs_setattr() accepts arguments for minor frame number and thread ID; however it currently allows setting exception policies only for all policies and all minor frames. The most significant argument to it is the frs_recv_info structure, declared with these fields.

typedef struct frs_recv_info {
    mfbe_rmode_t  rmode;      /* Basic recovery mode */
    mfbe_tmode_t  tmode;      /* Time expansion mode */
    uint          maxcerr;    /* Max consecutive errors */
    uint          xtime;      /* Recovery extension time */
} frs_recv_info_t;

The recovery modes and other constants are declared in /usr/include/sys/frs.h. The function in Example 4-3 sets the policy of injecting a repeat frame. The caller specifies only the Frame Scheduler and the number of consecutive exceptions allowed.

int
setInjectFrameMode(frs_t *frs, int consecErrs)
{
  frs_recv_info_t work;
  bzero((void*)&work,sizeof(work));
  work.rmode = MFBERM_INJECTFRAME;
  work.maxcerr = consecErrs;
  return frs_setattr(frs,0,0,FRS_ATTR_RECOVERY,(void*)&work);
}

The function in Example 4-4 sets the policy of stretching the current frame (a function to set the policy of stealing time from the next frame is nearly identical). The caller specifies the Frame Scheduler, the number of consecutive exceptions, and the stretch time in microseconds.

int
setStretchFrameMode(frs_t *frs,int consecErrs,uint microSecs)
{
  frs_recv_info_t work;
  bzero((void*)&work,sizeof(work));
  work.rmode = MFBERM_EXTENDFRAME_STRETCH;
  work.tmode = EFT_FIXED; /* only choice available */
  work.maxcerr = consecErrs;
  work.xtime = microSecs;
  return frs_setattr(frs,0,0,FRS_ATTR_RECOVERY,(void*)&work);
}

When you set a policy that permits exceptions, the FRS controller thread can query for counts of exceptions. This is done with a call to frs_pthread_getattr() or  frs_getattr(), passing the handle to the Frame Scheduler, the number of the minor frame, and the thread ID of the thread within that frame.

The values returned in a structure of type frs_overrun_info_t are the counts of overrun and underrun exceptions incurred by that thread in that minor frame. In order to find the count of all overruns in a given minor frame, you must sum the counts for all threads queued to that frame. If a thread is queued to more than one minor frame, separate counts are kept for it in each frame.

The function in Example 4-5 takes a Frame Scheduler handle and a minor frame number. It gets the list of thread IDs queued to that minor frame, and returns the sum of all exceptions for all of them.

Example 4-5. Function to Return a Sum of Exception Counts (pthread Model)

#define THE_MOST_TIDS 250 int totalExcepts(frs_t * theFRS, int theMinor) { int numTids = frs_getqueuelen(theFRS, theMinor); int j, sum; pthread_t allTids[THE_MOST_TIDS]; if ( (numTids <= 0) || (numTids > THE_MOST_TIDS) ) return 0; /* invalid minor #, or no threads queued? */ if (frs_pthread_readqueue(theFRS, theMinor, allTids) == -1) return 0; /* unexpected problem with reading IDs */ for (sum = j = 0; j<numTids; ++j) { frs_overrun_info_t work; frs_pthread_getattr(theFRS  /* the scheduler */ theMinor, /* the minor frame */ allTids[j], /* the threads */ FRS_ATTR_OVERRUNS, /* want counts */ &work); /* put them here */ sum += (work.overruns + work.underruns); } return sum; }

---

Tip: The FRS read queue functions return the number of threads present on the queue at the time of the read. Applications can use this returned value to eliminate calls to frs_getqueuelen().

---

When a process is scheduled by the IRIX kernel, it receives a pending signal the next time the process exits from the kernel domain. For most signals, this could occur under the following conditions:

(SIGALRM is delivered as soon as the kernel is ready to return to user processing after the timer interrupt, in order to preserve timer accuracy.) Thus, for a process that is ready to run, in a CPU that has not been made nonpreemptive, normal signal latency is at most 10 milliseconds, and SIGALARM latency is less. However, when the receiving process is not ready to run, or when there are competing processes with higher priorities, the delivery of a signal is delayed until the next time the receiving process is scheduled.

When the CPU is nonpreemptive (see “Making a CPU Nonpreemptive” in Chapter 3), there are no clock tick interrupts, so signals can only be delivered following a system call.

Signal latency can be greater when running under the Frame Scheduler. Like the normal IRIX scheduler, the Frame Scheduler delivers pending signals to a process when it next returns to the process from the kernel domain. This can occur under the following conditions:

The upper bound on signal latency in this case is the interval between the minor frames to which that process is queued. If the process is scheduled only once in a major frame, it might not receive a signal until a full major frame interval after the signal is sent.

When a Frame Scheduler detects an Overrun or Underrun exception that it cannot recover from, and when it is ready to terminate, it sends a signal to the FRS controller.

---

Tip: Child processes inherit signal handlers from the parent, so a parent should not set up handlers prior to sproc() or fork() unless they are meant to be inherited.

---

The FRS controller for a synchronized Frame Scheduler should have handlers for Underrun and Overrun signals. The handler could report the error and issue frs_destroy() to shut down its scheduler. An FRS controller for a synchronized scheduler should use the default action for SIGHUP (Exit) so that completion of the frs_destroy() quietly terminates the FRS controller.

The FRS controller for the master (or only) Frame Scheduler should catch Underrun and Overrun exceptions, report them, and shut down its scheduler.

When an FRS is terminated with frs_destroy(), it sends SIGKILL to its FRS controller. This cannot be changed; and SIGKILL cannot be handled. Hence frs_destroy() is equivalent to termination for the FRS controller.

A Frame Scheduler can send a signal to an activity thread when the thread is removed from any queue using  frs_pthread_remove() or frs_premove() (see “Managing Activity Threads”). The scheduler can also send a signal to an activity thread when it is removed from the last or only minor frame to which it was queued (at which time a thread is returned to normal IRIX scheduling).

In order to have these signals sent, the FRS controller must set nonzero signal numbers for them, as discussed in the following topic, “Setting Frame Scheduler Signals”.

The Frame Scheduler sends signals to the FRS controller.

---

Note: In earlier versions of REACT/pro, the Frame Scheduler sent these signals to all processes queued to that Frame Scheduler as well as the FRS controller. That is no longer the case. You can remove signal handlers for these signals from activity processes, if they exist.

---

The signal numbers used for most events can be modified. Signal numbers can be queried using frs_pthread_getattr(FRS_ATTR_SIGNALS) or frs_getattr(FRS_ATTR_SIGNALS) and changed using frs_pthread_setattr(FRS_ATTR_SIGNALS) or frs_setattr(FRS_ATTR_SIGNALS), in each case passing an frs_signal_info structure. This structure contains room for four signal numbers, as shown in Table 4-3.

Signal numbers must be changed before the Frame Scheduler is started. All the numbers must be specified to frs_pthread_setattr() or frs_setattr(), so the proper way to set any number is to first file the frs_signal_info_t using frs_pthread_getattr() or frs_getattr(). The function in Example 4-6 sets the signal numbers for Overrun and Underrun from its arguments.

int
setUnderOverSignals(frs_t *frs, int underSig, int overSig)
{
  int error;
  frs_signal_info_t work;
  error = frs_pthread_getattr(frs,0,0,FRS_ATTR_SIGNALS,(void*)&work);
  if (!error)
  {
    work.sig_underrun = underSig;
    work.sig_overrun = overSig;
    error = frs_pthread_setattr(frs,0,0,FRS_ATTR_SIGNALS,(void*)&work);
  }
  return error;
}

When frs_create_vmaster() is used to create a frame scheduler triggered by multiple interrupt sources, a sequence error signal is dispatched to the controller thread if the interrupts come in out of order. For example, if the first and second minor frame interrupt sources are different, and the second minor frame's interrupt source is triggered before the first minor frame's interrupt source, then a sequence error has occurred.

This type of error condition is indicative of unrealistic time constraints defined by the interrupt information template.

The signal code that represents the occurrence of a sequence error is SIGRTMIN+1. This signal cannot be reset or disabled using the frs_setattr() interface.

The following sequence of actions occurs when a device driver is used as a source of time-base interrupts for the Frame Scheduler.

Device driver names, device driver structure, configuration files, and related topics are covered in the IRIX Device Driver Programming Guide.

A device driver must register two interface functions to make them known to the Frame Scheduler. This call, which occurs during the device driver's own initialization, also makes the driver known as a source of time-base interrupts:

frs_driver_export( int frs_driver_id,
                  void (*frs_func_set)(intrgroup_t*),
                  void (*frs_func_clear)(void));

The parameter frs_driver_id is the driver's identification number. A real-time program specifies the same number to frs_create_master() or frs_create() to select this driver as the source of interrupts. The identifier is an integer between 0 and 7. Different drivers in the same system must use different identifiers. A typical call resembles the code in Example 4-8.

/*
** Function called by the example driver to export
** its Frame Scheduler interface functions.
*/
frs_driver_export(3, example_frs_func_set, example_frs_func_clear);

The device driver must provide a function with the following prototype:

void pfx_frs_func_set ( intrgroup_t* intrgroup ) ;

A skeleton of an initialization function for a Challenge or Onyx system running under IRIX 6.2 is shown in Example 4-9. The function is called by a new master Frame Scheduler—one that is created with an interrupt source parameter of FRS_INTRSOURCE_DRIVER and an interrupt qualifier specifying this device driver's number (see “Device Driver Interrupt”). A device driver is used by only one Frame Scheduler at a time.

The argument intrgroup is passed by the Frame Scheduler to identify the interrupt group it has allocated. A VME device driver must set the hardware devices it manages so that interrupts are directed to this interrupt group. The actual group identifier may be obtained using the macro:

intrgroup_get_groupid(intrgroup)

The effective destination may be obtained using the following macro:

EVINTR_GROUPDEST(intrgroup_get_groupid(intrgroup))

/*
** Frame Scheduler initialization function
** for the External Interrupts Driver
*/
int FRS_is_active = 0;
int FRS_vme_install = 0;
void
example_frs_func_set(intrgroup_t* intrgroup)
{
   int s;
   ASSERT(intrgroup != 0);
   /*
   ** Step 1 (VME only):
   ** In a VME device driver, set up the hardware to send
   ** the interrupt to the appropriate destination.
   ** This is done with vme_frs_install() which takes:
   ** * (int) the VME adapter number
   ** * (int) the VME IPL level
   ** * the intrgroup as passed to this function.
   */
   FRS_vme_install = vme_frs_install(
      my_edt.e_adap, /* edt struct from example_edtinit */
      ((vme_intrs_t *)my_edt.e_bus_info)->v_brl,
      intrgroup);
   /*
   ** Step 2: any hardware initialization required.
   */
   /*
   ** Step 3: note that we are now in use.
   */
   FRS_is_active = 1;
}

Only VME device drivers on the CHALLENGE/Onyx need to call vme_frs_install() — do not call it on Origin systems. As suggested by the code in Example 4-9, the arguments to vme_frs_install() can be taken from data supplied at boot time to the device driver's pfxedtinit() function:

The pfxedtinit() entry point is documented in the IRIX Device Driver Programming Guide.

---

Tip: The vme_frs_install() function is a dynamic version of the VECTOR configuration statement. You are not required to use the IPL value from the configuration file.

---

The device driver must provide a function with the following prototype:

void prfx_frs_func_clear ( void ) ;

A skeleton for this function is shown in Example 4-10. The Frame Scheduler that initialized a device driver calls this function when the Frame Scheduler is terminating. The Frame Scheduler deallocates the interrupt group to which interrupts were directed.

The device driver should clean up data structures and make sure that the device is in a safe state. A VME device driver must call vme_frs_uninstall().

/*
** Frame Scheduler termination function
*/
void
example_frs_func_clear(void)
{
   /*
   ** Step 1: any hardware steps to quiesce the device.
   */

   /*
   ** Step 2 (VME only):
   ** Break the link between interrupts and the interrupt
   ** group by calling vme_frs_uninstall() passing:
   ** * (int) the VME adapter number
   ** * (int) the VME IPL level
   ** * the value returned by vme_frs_install()
   */	
   vme_frs_uninstall(
      my_edt.e_adap, /* edt struct from example_edtinit */
      ((vme_intrs_t *)my_edt.e_bus_info)->v_brl,
      FRS_vme_install);
   /*
   ** Step 3: note we are no longer in use.
   */
   FRS_is_active = 0;
}

A driver has to call the Frame Scheduler interrupt handler from within the driver's nonthreaded interrupt handler using code similar to that shown in Example 4-11. It delivers the interrupt to the Frame Scheduler on that CPU. The function to be invoked is

void frs_handle_driverintr(void);

void example_intr()
{
   /*
   ** Step 1: anything required by the hardware
   */
   /*
   ** Step 2: if connected to the Frame Scheduler, send
   ** an interrupt to it. Flag FRS_is_active is set in
   ** 

Example 4-9 and cleared in Example 4-10.

   */
   if (FRS_is_active) frs_handle_driverintr();
   /*
   ** Step 3: any additional processing needed.
   */
   return;
}

It is possible for an interrupt handler to be entered at a time when the Frame Scheduler for its processor is not active; that is, after frs_destroy() has been called and before the driver termination function has been entered. The frs_handle_driverintr() function checks for this and does nothing when nothing is required.

The call to frs_handle_driverintr() must be executed on a CPU controlled by the FRS that is using the driver. The only way to ensure this is to ensure that the hardware interrupt used by this driver is directed to that CPU. In IRIX 6.4 and later, you direct a hardware interrupt to a particular CPU by placing a DEVICE_ADMIN directive in the file /var/sysgen/system/irix.sm. See comments in that file for the syntax.

Threaded interrupt handlers experience problems when run on a CPU with an FRS. If the physical interrupt is not directed to the CPU, the thread is forced to run elsewhere. If the interrupt is directed, the thread does not get a chance to run until the FRS exits. The FRS schedules only the user threads that are registered with it on its CPUs; it ignores kernel and driver threads bound to those CPUs. Currently, by default, all VME and PCI drivers for IP27 and IP35 systems are threaded.

When writing drivers to be used with the FRS, you should use nonthreaded interrupt handlers. Nonthreaded handlers run out of the interrupt stack and therefore, do not need to be scheduled by the CPU, as do threaded handlers. However, the physical interrupt must still be routed to the desired CPU.

SGI's response time guarantee is not honored when used with nonthreaded interrupt handlers because IRIX has little control over them. Therefore, authors of such drivers should be careful not to spend too much execution time within their handlers. Nonthreaded handlers must not attempt to take any form of blocking lock or call any function that might block.

Example 4-12 shows how to modify the standard VME interrupt registration to use nonthreaded interrupts. The code is based on the sample VME driver found in the IRIX Device Driver Programmer's Guide, chapter 13, in the section titled “Sample VME Device Driver.”

...
#include <sys/iobus.h>
...

int
rfm_edtinit(edt_t * e)
{
        ...

        /*
         * Get the device descriptor for our device
         */
        example_dev_desc = device_desc_dup(conn);

        /*
         * Set the non-threaded flag on the device descriptor
         */
        device_desc_flags_set(example_dev_desc,
               (device_desc_flags_get(conn) | D_INTR_NOTHREAD));

        /*
         * Allocate the interrupt for the device
         */
        intr = vmeio_intr_alloc(conn, example_dev_desc,
                                ivec, ilev, rfm, 0);

        ...
}