This chapter explains how to configure real-time CPUs that are restricted from running scheduled processes and isolated from load-balancing considerations. It discusses the following:
For information about creating an external interrupt character special device file, see “Opening the Device Special File” in Chapter 8. For information about potential problems, see Chapter 12, “Troubleshooting”.
To configure and control REACT, you will use the react(8) command. Configurable items include:
REACT stores most configuration information supplied via the react command in the /etc/react.conf file; however, permissions are stored in the /etc/sysconfig/sgi-react.conf file.
The react command has the following options:
/sbin/react -a "arguments" /sbin/react -c cpu [-I command [--arg]] [-v] /sbin/react -d [-v] /sbin/react -e [-v] /sbin/react -h /sbin/react -i irqlist|RR [-v] /sbin/react -p group:permission /sbin/react -r cpulist [-i irqlist|RR] [-v] /sbin/react -s /sbin/react -o kernel /sbin/react -w "entire_commandline" /sbin/react -x "arguments" |
| -a " arguments" | Adds or changes the specified kernel arguments (separated by white space) to the default kernel arguments supplied by REACT. If you include multiple arguments, you must enclose them within quotation marks. | |||
| -c | Specifies the CPU on which a command will be invoked (see -I). The CPU must be configured to be real-time. There is no default. | |||
| -d | ||||
| -e | Enables the configuration stored in the /etc/react.conf file. For more information, see “Initially Configuring REACT”. | |||
| -h | Displays the usage statement (the default for react without any options). | |||
| -I cmd [-- arg ] ... | Invokes the specified command line on the CPU specified by -c.
If you specify -c without -I, by default react will execute the value of the SHELL environment variable if it is set or else use /bin/sh to invoke a subshell. If you specify -I without -c, it is ignored. | |||
| -i irqlist|RR | Specifies the interrupt requests (IRQs) to be redirected. The specification is either:
To minimize latency of real-time interrupts, it is often necessary to direct some IRQs to specific real-time processors and to direct other interrupts away from specific real-time processors. You should only redirect IRQs if you must move them away from CPUs that must be real-time. However, redirected IRQs often have higher latency, so it is preferable to select CPUs for real-time in such a way as to not require interrupt redirection. By default (if you do not enter -i), REACT assumes that the IRQs should be moved off of the real-time CPUs. REACT causes IRQs that can be moved to be evenly dispersed among CPUs in the bootcpuset in a round-robin (-i RR) fashion. | |||
| -o kernel | Specifies a kernel number (as ordered in the /boot/grub/menu.lst file, beginning with 0 for the first entry) or a kernel label (as shown in the /etc/elilo.conf file). By default, the default kernel in the menu.lst or elilo.conf file is used.
| |||
| -p group:permission | Specifies group ownership and permissions with regard to running react, where:
| |||
| -r cpulist | Specifies the real-time CPUs. cpulist takes one of the following formats:
If you do not specify -r cpulist, no real-time CPUs are identified. | |||
| -s | Shows the REACT configuration. See “Showing the Configuration”. | |||
| -v | Specifies verbose mode, which sends tracing messages to the console. | |||
| -w "commandline" | Substitutes the specified entire kernel command line for the default kernel command line supplied by REACT. You must enclose the command line within quotation marks. | |||
| -x "arguments" | Deletes the specified kernel arguments (separated by white space) from the default kernel arguments supplied by REACT. If you include multiple arguments, you must enclose them within quotation marks. |
For more information, see the react(8) man page.
To initially configure REACT, do the following:
Specify the real-time CPUs and optionally any interrupt requests (IRQs) to be redirected:
[root@linux root]# react -r cpulist [-i irqlist]
For example, to restrict CPUs 8-32 and (by default) redirect IRQs away from CPUs 8-32:
[root@linux root]# react -r 8-32
In another example, to restrict CPUs 2, 3, 4, 5, 6, and 7, to redirect IRQ 59 to CPU 2, and to redirect IRQ 66 to CPU 5:
[root@linux root]# react -r 2-7 -i 59:2,66:5
Reboot the system (react will add the required kernel command-line options).
When the system comes back up, REACT is automatically enabled by the /etc/init.d/sgi_react script (which runs the react -e command).
The enable (-e) option does the following:
Creates a container cpuset named rtcpus and cpusets (labeled rtcpuN) for each CPU that is not part of the bootcpuset (such as /rtcpus/rtcpu1 for CPU1). You can use these cpusets to run your real-time threads. You will find these cpusets in /dev/cpuset, along with the bootcpuset set up by react -r in step 1 and stored in /etc/react.conf.
Configures the cpuset's memory nodes by setting the values in the following files:
/dev/cpuset/rtcpus/rtcpu N/mems
/dev/cpuset/boot/mems
Redirects interrupts if specified with the -i option in step 1. The proper hexadecimal mask values are echoed to the file /proc/irq/interrupt/smp_affinity .
After the system is rebooted with the real-time configuration and REACT is automatically enabled, you can make changes to the real-time and bootcpusets dynamically without additional reboots.
For example, to change the list of real-time CPUs to CPU 2 and CPU 4 and return to the default round-robin handling of IRQs, enter the following:
[root@linux root]# react -r 2,4 -i RR |
To change the IRQ configuration without altering the real-time CPUs, use just the -i option. For example, to redirect IRQ 4340 to CPU 3 and to redirect IRQ 66 to CPU 5:
[root@linux root]# react -i 4340:3,66:5 |
 | Note: To temporarily change the running REACT system, you can call libreact from a user program to add or remove real-time CPUs. However, these changes will not be stored in /etc/react.conf . For more information, see the libreact(3) man page. |
To disable REACT and return the system to normal, do the following:
Stop the real-time processes.
Enter the disable option:
[root@linux root]# react -d
The disable option does the following:
Removes the rtcpuN cpusets and adjusts /boot to behave like /cpuset on a system without REACT.
Starts the IRQ balancer, which will move any changed IRQs to CPUs based on the IRQ balancer's policies. For more information, see the irqbalance(1) man page.
To reenable a previously configured REACT system that has been disabled and use the configuration that is stored in /etc/react.conf, enter the following:
[root@linux root]# react -e |
If you enter react -e on a currently enabled REACT system whose configuration has been modified by a user program that calls libreact, react enables the configuration stored in the /etc/react.conf file.
REACT supplies specific kernel boot parameters for optimum interrupt response times. It is possible to change these default values or any other kernel boot parameter. Also, REACT will typically will operate on the default kernel in the corresponding boot configuration file (either /etc/elilo.conf or /boot/grub/menu.lst). The following options help facilitate modifying the kernel command-line options:
Example boot parameters:
nohz=off noirqdebug init=/sbin/react-init.sh isolcpus=1-8 |
To add options, use the -a option:
# react -a cgroups_disable=memory nohz=off noirqdebug cgroups_disable=memory init=/sbin/react-init.sh isolcpus=1-8 |
To change an option's value:
# react -a nohz=on nohz=on noirqdebug cgroups_disable=memory init=/sbin/react-init.sh isolcpus=1-8 |
To delete an option, use the -x option:
# react -x cgroups_disable=memory nohz=on noirqdebug init=/sbin/react-init.sh isolcpus=1-8 |
To add or change multiple options, separate them with white space and enclose them within quotation marks:
# react -a "cgroups_disable=memory nohz=off" nohz=off noirqdebug cgroups_disable=memory init=/sbin/react-init.sh isolcpus=1-8 |
To enable a nondefault kernel, use the -o option to specify the kernel by its label:
# react -e -o SLES11_SP2_1 |
To disable:
# react -d -o SLES11_SP2_1 |
To specify the third kernel listed in the /boot/grub/menu.lst file:
# react -e -o 2 |
To disable:
# react -d -o 2 |
The cpusets, devices, and control files associated with REACT are normally accessible only by the root user. The -p option lets you specify a group of users that have access to the following REACT features:
This option generates the /etc/udev/rules.d/99-sgi-react.rules file and a new /etc/sysconfig/sgi-react.conf configuration file, which initially holds the group ID and permissions. It changes the group ownership and file mode permissions for REACT /dev, /sys/class/extint, and /dev/cpuset files, both immediately and across reboots.
After you use the -p option, the specified users can run REACT applications without having the ability to overwrite any file on the system. (That is, the specified users do not have CAP_DAC_OVERRIDE authority.)
| Note: The specified users will not have access to native system calls that require specific capabilities, such as sched_setscheduler() . To directly use those system calls, a user must have the required process capabilities set via the cpu_sysrt_set_allowed_caps and cpu_sysrt_set_caps routines |
To use the -p option, you specify an explicit group (using either the group name or the group ID) and the explicit permissions to set:
-p group:permission |
For example, suppose that the /etc/group file has an entry for a group named usersA that has a numerical ID of 100 and another group named usersB that has a numerical group ID of 222:
[root@linux root]# grep users /etc/group usersA:x:100: usersB:x:222: |
When using the -p option, you could specify either 100 or usersA for the group value.
Suppose that no changes have yet been made to the /etc/sysconfig/sgi-react.conf file:
[root@linux root]# ls -al /dev/cpuset/rtcpus/tasks /dev/frs -rw-r--r-- 1 root root 0 2011-08-18 10:11 /dev/cpuset/rtcpus/tasks crw-rw---- 1 root root 10, 54 2011-08-18 10:11 /dev/frs [root@linux root]# cat /etc/sysconfig/sgi-react.conf [root@linux root]# |
To allow the group usersA to run REACT with execute, read and write permission (0777), enter the following and show the results:
[root@linux root]# react -p usersA:0777
[root@linux root]# ls -al /dev/cpuset/rtcpus/tasks /dev/frs
-rwxrwxrwx 1 root usersA 0 2011-08-22 08:08 /dev/cpuset/rtcpus/tasks
crwxrwxrwx 1 root usersA 10, 54 2011-08-22 08:08 /dev/frs
[root@linux root]# cat /etc/sysconfig/sgi-react.conf
group 100
mode 0777 |
To allow read permission (0644) while leaving the current assigned groups untouched, enter the following and show the results:
[root@linux root]# react -p -1:0644 [root@linux root]# ls -al /dev/cpuset/rtcpus/tasks /dev/frs -rw-r--r-- 1 root usersA 0 2011-08-22 08:08 /dev/cpuset/rtcpus/tasks crw-r--r-- 1 root usersA 10, 54 2011-08-22 08:08 /dev/frs [root@linux root]# cat /etc/sysconfig/sgi-react.conf group 100 mode 0777 |
To allow the group usersB to run REACT using the permissions set in the previous command line (0644), enter the following and show the results:
[root@linux root]# react -p usersB:-1 [root@linux root]# ls -al /dev/cpuset/rtcpus/tasks /dev/frs -rw-r--r-- 1 root usersB 0 2011-08-22 08:08 /dev/cpuset/rtcpus/tasks crw-r--r-- 1 root usersB 10, 54 2011-08-22 08:08 /dev/frs [root@linux root]# cat /etc/sysconfig/sgi-react.conf group 100 mode 0777 |
Any changes made to a particular device or tasks file under the control of REACT will be reset to the values in /etc/sysconfig/sgi-react.conf with a reboot or the use of the -e or -r options. For example:
[root@linux root]# ls -al /dev/frs /dev/cpuset/rtcpus/tasks -rwxr-xr-x 1 root usersA 0 2011-08-18 10:17 /dev/cpuset/rtcpus/tasks crwxrwxrwx 1 root usersB 10, 54 2011-08-18 10:11 /dev/frs [root@linux root]# react -r 4-7 -p -1:-1 [root@linux root]# ls -al /dev/frs /dev/cpuset/rtcpus/tasks -rwxrwxrwx 1 root usersA 0 2011-08-18 10:17 /dev/cpuset/rtcpus/tasks crwxrwxrwx 1 root usersA 10, 54 2011-08-18 10:11 /dev/frs |
Enabling react will use the last values set in /etc/sysconfig/sgi-react.conf and result in the following:
[root@linux root]# react -e [root@linux root]# ls -al /dev/frs /dev/cpuset/rtcpus/tasks -rwxrwxrwx 1 root usersA 0 2011-08-22 08:08 /dev/cpuset/rtcpus/tasks crwxrwxrwx 1 root usersA 10, 54 2011-08-22 08:08 /dev/frs |
The -s option displays the configuration that is running and the configuration that is stored in /etc/react.conf.
| Note: These may be different if you have called libreact from a user program to add or remove real-time CPUs. |
For example:
[root@linux root]# react -s ++++ REACT is ENABLED ++++ Live configuration: =========================== bootcpuset cpus: 0-7,32-39 real-time cpus: 8-31,40-63 Configuration in /etc/react.conf: ========================================= bootcpuset cpus: 0-7 32-39 real-time cpus: 8-31 40-63 IRQ configuration: 21:0 23:7 54:4 63:45 |
If you add -v to the command line with -d, -e, -r, or -i, the react command prints a trace of its actions to the console. The verbose output will detail the steps taken by react and is useful in understanding its behavior and analyzing problems. (The amount of output will vary greatly depending on the number of CPUs and the number of IRQs.)
For example (line breaks shown here for readability:
[root@linux root]# react -ve Default label = '0' kernel kernel /boot/vmlinuz-2.6.32-70.el6.x86_64 ro root=LABEL=uv41-sysR12 ... Current Kernel Command line: ro root=LABEL=uv41-sysR12 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM ... rtcpus 8-31,40-63 bootcpus 1-7,32-39 Acquiring Lock... Lock Acquired cpuset_delete: /rtcpuN does not exist ... cpuset /rtcpus cpu 8 mem 2 ... modified cpu list 8-31,40-63 modified mem list 2-7 cpuset: modify /rtcpus DUP cpu 1 ... cpuset: modify /boot Releasing Lock Lock Released Acquiring Lock... Lock Acquired DUP cpu 8 ... cpuset /boot cpu 0 mem 0 ... cpuset /rtcpus/rtcpu63 cpu 63 mem 7 modified cpu list 63 modified mem list 7 cpuset: modify /rtcpus/rtcpu63 Releasing Lock Lock Released ++++ REACT is ENABLED ++++ Live configuration: =========================== bootcpuset cpus: 0-7,32-39 real-time cpus: 8-31,40-63 Configuration in /etc/react.conf: ========================================= bootcpuset cpus: 0-7 32-39 real-time cpus: 8-31 40-63 IRQ configuration: 21:0 23:7 54:4 63:45 Total Nodes = 8 CPUs on node 0 - 0-3,32-35 ... IRQ 0 is on node 0, bootcpu on same node == 0 ... IRQ 94 is on node 3, No bootcpu available on node, using bootcpu == 32 ... **** Manually config'd IRQs **** IRQ 21 cpu 0 IRQ 23 cpu 7 IRQ 54 cpu 4 IRQ 63 cpu 45 |
To run a process on a real-time CPU, you must invoke or attach it to a real-time cpuset (that is, a cpuset containing a CPU that does not exist in the bootcpuset, such as the /dev/cpuset/rtcpus/rtcpu N cpusets created above). Examples:
To execute the foo -l command on CPU 4:
[root@linux root]# react -c 4 -I ./foo -- -l
Note: The double-minus "-- " is required so that react command does not interpret the -l option but instead passes it to the foo command.
To execute the foo command on real-time CPU 4:
[root@linux root]# cpuset --invoke /rtcpus/rtcpu4 -I ./foo
To discover the real-time CPUs and then attach the foo process to CPU 1 (which is the second real-time CPU, not the second CPU on the system):
[root@linux root]# echo $$ | cpuset -a /rtcpus [root@linux root]# dplace -c 1 ./foo
To attach an existing process to real-time CPU 2:
[root@linux root]# echo $$ | cpuset --attach /rtcpus/rtcpu2
For more information, see the cpuset(1), dplace(1), libreact(3), and libcpuset(3) man pages.
The following command invokes the date(1) command without arguments on CPU 6:
# react -c 6 -I date |
The following command invokes the date(1) command and displays the date in Greenwich Mean Time (GMT) universal time on CPU 6:
# react -c 6 -I date -- -u |
| Note: The double-minus "--" is required here so that react command does not interpret the -u option but instead passes it to the date command. |
The following command invokes an sh(1) subshell (by default) on CPU 6:
# react -c 6 |