In order to run ClusterFly, start an identical copy of the program cfly on each of the channel nodes. You can either run cfly manually on each node, or you can use a script that runs cfly on them from a single node. The following subsections show each one of these operation modes using a data file called town_ogl.pfb.

% setenv PFSERVER master-channel
% cfly town_ogl.pfb

The ClusterFly GUI provides a control panel with an assortment of controls. You can operate the control panel using the mouse buttons and the keyboard attached to node channel0 ( or the node that started running cfly first).

Many other keys on the keyboard are active and can be used to control ClusterFly even when the control panel is not displayed. For full details, the perfly man page contains a list of these key sequences and their effects, as well as details on motion models.

Entering cfly -H at the command line displays a list of the command-line options. For instance, the ClusterFly program allows several motion models; the –d on the command line tells the program to start in the Drive model, which provides an easy way to drive or walk through a scene while maintaining a fixed height above the ground.

Look around the scene using the mouse. First, place the cursor in the center of the simulation window. Now depress the middle mouse button and move the mouse to the left or right to turn in place; you will continue to pivot until you place the cursor back in the center of the screen.

Do not worry if you inadvertently start moving around, lose sight of the building, or otherwise lose position or control. Just move the cursor into the control panel area and click the Reset All button on the control panel to get back to the original setup.

To approach the City Hall model, turn until you are facing it (if you are not already facing it) and then center the mouse in the screen. Depress the left mouse button briefly to start accelerating forward. When you release the button, you will continue gliding forward at constant speed and can hold down the middle mouse button to steer. The ClusterFly application shows you how the basic visual simulation tools work. This example uses a section of the New Jerusalem City Hall (see Figure 3-1).

Figure 3-1. Section of the New Jerusalem City Hall

Tap the middle mouse button to stop in front of the building (if you actually entered the building, remember the Reset All button). Now accelerate backward by pressing the right button. When you are as far back as you want to go, hold down the left mouse button to gradually slow down, or tap the middle mouse button to stop immediately.

Now use the left mouse button again to start moving forward and drive slowly into the model. Notice that the walls closest to you are cut away at first so you can see inside; when you are completely inside the building, those walls reappear. Drive around and explore the building. Tap the middle mouse button to stop before you run into anything (but do not worry—at this point you will bounce off any walls you hit). If the walls get in your way, you can turn off collision detection with the button labeled Collide on the control panel, or press the c key on the keyboard.

To see the underlying geometry used to create the model, click the Style button in the control panel, or press the w key on the keyboard. This changes the display to wireframe mode. In this mode you can more easily see how many polygons are used to represent an object. This information can be helpful when you are tuning a database, because it is important to know when the number of polygons becomes a limiting factor. To turn wireframe mode off, just click the Style button (or press w) again. The W key can be used to cycle through several different drawing styles.

To close the entire control panel (and devote the entire screen to the model), click the GUI Off button at the upper right of the control panel, or just press the F1 key. Press the F1 key again to restore the control panel. The GUI is part of libpfutil. See the sample program, /usr/share/Performer/src/pguide/libpfutil/utilui.c.

If you click the Stats button in the control panel, a transparent panel showing scene statistics appears overlaid on the screen. The buttons next to the Stats button allow you to choose one of the available statistical displays. Try moving around in the scene while watching how the statistics change. Note in particular that the number of triangles being considered for rendering changes drastically depending on where you look; this demonstrates OpenGL Performer's use of culling to ignore objects that are completely outside the field of vision. For more information about culling and the statistics panels, see the OpenGL Performer Programmer's Guide.

The control panel's field-of-view slider can be used to select a wide angle view, up to 100 degrees.

As you travel through the building, try turning on the fog effect by clicking the Fog button. Experiment with the fog density and other controls. (Remember: If you have closed the control panel, the F1 key restores it.)

So far you have been driving. There are other default motion models provided through the libpfui library. These motion models can be subclassed to create your own models. You can find the source code for these motion models in the directory /usr/share/performer/src/lib/libpfui/.

There are other approaches to traveling through a scene than the models described here. For instance, you can build a specific path into the viewer, to prevent the user from straying outside your model. The Path model is supported by a general path-following system in the libpfutil library. Many simulation applications require path support for such objects as cars, trucks, and people (in driver-training software); waiting aircraft both on the ground and in the air (in flight simulation); and opposing forces in military trainers. Path support in libpfutil allows paths of varying speeds to be built from line segments and arcs with automatic fillet construction between segments for smooth transitions.

When you want to quit cfly, either press the Esc key on the keyboard of channel0 (or the node that started running cfly first) or click the Quit button on the ClusterFly GUI of channel0 (or the node that started running cfly first).

This subsection describes the changes made to perfly in order to create cfly.

In general, all the global information in perfly is shared among the cluster nodes. This information is stored in a global variable named ViewState. The naive approach would assign ViewState for sharing via the DataSync API. Unfortunately, ViewState contains pointers to node-local objects such as pfDCS nodes. Sharing the exact contents of ViewState would therefore share the pointers to the pfDCS nodes (incorrect). Moreover, the intent is to share information in two phase-shifted stages:

There are two new memory structures containing data for the two phase shifted stages:

The intent was to keep the changes to the perfly source code to the necessary minimum. The new file cluster.C contains all the code accessing the DataSync cluster API. The original code contains new function calls in three places:

The perfly code contains a few other changes. In multiple places a node with clusterRole ROLE_SLAVE should skip some standard operations—for example, setting camera position.

The code starts by initializing the DataSync library. Initializing requires initialization parameters as in the following code:

dsInitParams *initParams = new dsInitParams();
dsInit(initParams);

The program must now get a handle to the DataSync server. We use the environment variables PFSERVER and PFPORT to select the server host and port. Once selected, the program calls dsOpen() and receives a handle to the server as shown in the following:

dsOpenParams *openParams = new dsOpenParams();
if ((server = getenv ("PFSERVER")) != NULL)
	openParams -> setAddress(server);
else
	openParams -> setAddress(server_name);

if ((port = getenv ("PFPORT")) != NULL)
	openParams -> setPort(atoi(port));
dsServer *svr = dsOpen( openParams );

The program now allocates memory in the Performer shared arena for all the data structures that it wants to share via DataSync. These include the pre-pfFrame() and post-pfFrame() structures as well as a DataSync session information block. The following code shows the memory allocation:

ViewState -> SessionData = (ClusterSessionData *)
                     pfCalloc (sizeof (ClusterSessionData), 1,
                     pfGetSharedArena());
ClusterViewState_pre = (ClusterSharedViewState_pre *)
                     pfCalloc (sizeof (ClusterSharedViewState_pre), 1,
                     pfGetSharedArena());
ClusterViewState_post = (ClusterSharedViewState_post *)
                     pfCalloc (sizeof (ClusterSharedViewState_post), 1,
                     pfGetSharedArena());

The program tries to add a session to the DataSync server. Only one cluster node can add a session of a given name. The DataSync server returns a non-NULL result to that node. The program designates this node to be the master of the cluster. All other nodes are slaves of that master. The following code shows adding a session and finding the master:

dsSession *session = svr->addSession( session_name );   
if (session) // Success - I'm the first use of this session.
{
    ViewState -> clusterRole = ROLE_MASTER;
    sprintf (ViewState -> UserName, "master");
    ViewState -> SessionUser = session->addUser(ViewState -> UserName);
    ....
}
else
{
    ViewState -> clusterRole = ROLE_SLAVE;
    sprintf (ViewState -> UserName, "slave[%d]", getpid());
    dsSession *session = svr->login( session_name );
    ViewState -> SessionUser = session->addUser(ViewState -> UserName);
    .....
}

A master receives a pointer to the newly created session and is considered logged in to the session. A slave has to log in to a session in order to receive a pointer. Once a session pointer is obtained, the master and the slave have to add themselves as users to a session. A dsUser is the only way a process can interact with a session. One process may have multiple dsUsers to a single session. In the cfly implementation, each process has only one dsUser.

During the execution of cfly, the program has to receive DataSync events when other cluster nodes enter or leave the session. The program informs the DataSync server what events we wish to handle by calling the following function:

ViewState -> SessionUser -> handleEvent (session_events);

On the master side, the program has to create all the DataSync shared components: a session-info block, shared-memory buffers, and barriers. The program allocates shared data IDs for these elements in the following code:

dsDataId session_data_id = ViewState -> SessionUser
                   ->alloc( DS_MEMORY, DS_PRESERVED );
ViewState -> SessionData -> data_pre = ViewState -> SessionUser
                   ->alloc( DS_MEMORY, DS_UNPRESERVED );
ViewState -> SessionData -> data_post = ViewState -> SessionUser
                   ->alloc( DS_MEMORY, DS_UNPRESERVED );
ViewState -> SessionData -> barrier_pre = ViewState -> SessionUser
                   ->alloc(DS_BARRIER, DS_UNPRESERVED);
ViewState -> SessionData -> barrier_post = ViewState -> SessionUser
                   ->alloc(DS_BARRIER, DS_UNPRESERVED);

The program only allocates these shared Data IDs on the master. The slaves receive these the IDs using a session-info block as described in the following paragraphs.

For barrier objects, the previous allocations are all that are needed. However, DataSync does not allocate any memory for memory objects. We have to attach a local memory buffer (allocated earlier by the application) to each shared data ID in order to enable sharing. The program does this in the following calls (on the master):

        ViewState -> SessionUser -> mapMemory(
                            ViewState -> SessionData -> data_pre,
                            0,
                            ClusterViewState_pre,
                            sizeof(ClusterSharedViewState_pre),
                            DS_WRONLY);
         ViewState -> SessionUser -> mapMemory(
                            ViewState -> SessionData -> data_post,
                            0,
                            ClusterViewState_post,
                            sizeof(ClusterSharedViewState_post),
                            DS_WRONLY);
        ClusterViewState_post -> exitFlag = 0;
        ViewState -> SessionUser -> mapMemory(
                            session_data_id,
                            0,
                            ViewState -> SessionData,
                            sizeof(ClusterSessionData),
                            DS_WRONLY );

On the slaves, the program maps the above shared memory blocks as read-only.

The only pre-defined shared memory block on the DataSync server is the session info. The master has to notify the slaves of its newly allocated shared data IDs by storing their IDs in the session info as follows:

session->putInfo( session_data_id );

Slaves can call dsSession::getInfo() to retrieve the shared data ID used for the session info and map a local memory block to it:

dsDataId cluster_data_id = session->getInfo();

ViewState -> SessionUser -> mapMemory(
          cluster_data_id,
          0,
          ViewState -> SessionData,
          sizeof(ClusterSessionData),
          DS_RDONLY|DS_COPY);

When done with its initialization, the master calls the following function:

session->realize();

All pending slaves continue running their initialization code.

The functions sendClusterViewState_pre() and sendClusterViewState_post() package portions of the ViewState structure and inform DataSync to propagate them to the slaves. This section will focus on sendClusterViewState_pre() because it is shorter.

As explained earlier, some members of ViewState are pointers to local objects; so, the program has to package them before sharing across a cluster. One such example is ViewState -> sceneDCS. The program cannot share its pointer over the cluster. Instead, the program calls the following function:

    ViewState -> sceneDCS -> getMat(ClusterViewState_pre -> sceneDCS);

Once done packing all latency-critical members in ClusterViewState_pre, the program informs DataSync that the buffer is not clear:

ViewState -> SessionUser -> markMemory (
          ViewState -> SessionData ->data_pre,
          0,
          sizeof (ClusterSharedViewState_pre));

The program then makes DataSync propagate all marked buffers across the cluster as shown in the following:

ViewState -> SessionUser -> syncMemory();

DataSync does not have any API for receiving data. The local memory blocks that each slave creates are updated transparently. Slaves can assume that their input buffers are up-to-date and process them. The cfly slaves unpack input buffers into ViewState and make the necessary changes in the displayed GUI.

In order to force synchronization between the master and the slaves, each joins two barriers each frame. The master joins barriers right after calling dsUser::syncMemory(). The slaves join the same barriers before reading their shared memory copies. This ensures that slaves do not run too fast and process the same shared memory before the master finished updating it. It also ensures that slaves do not run too slow and miss complete updates of their shared memory.

Joining a barrier requires knowing its shared data ID and the number of users that should also join this barrier. The following call joins a barrier:

ViewState -> SessionUser -> barrier (
          ViewState -> SessionData -> barrier_pre,
          ViewState -> numUsers);

In order to maintain an up-to-date count on the number of users in a session, the program makes the following calls:

dsDirectory *dir = (dsDirectory *)
                   (ViewState -> ClusterSession -> directory());
ViewState -> numUsers = dir -> size();

The function queryClusterRank() computes the size of a cluster and the relative position of the current node in the cluster.

The program cfly uses shared memory to share the contents of ViewState. Another possible approach would send messages every time some value changes in the ViewState structure.