This chapter describes the data and configuration files that are required for the Scatter Visualizer. You can also generate these files automatically by using the Tool Manager. The subjects discussed are:
Read about the Scatter Visualizer in the MineSet Enterprise Edition User's Guide before using this chapter.
In its simplest form, the data file consists of a list of lines, each containing a set of fields, each separated by one tab. (Other separators are also allowed, but only one per file can separate each field. See “Input Options”.) All lines in the file must contain the same fields. The interpretation of the fields is specified by the configuration file, described in the next sections.
Sample data files are provided as part of the Scatter Visualizer package. For Windows these are in the directory in which MineSet is installed, under the Examples\scatterviz directory. For UNIX, they are in the /usr/lib/MineSet/scatterviz/examples/ directory. One such file is the store sales data file. In this file the first few lines appear as:
LIQUOR STORE 4300,4460,4800,4900,4700,4200,4250,4200 2700,2800,2750,3000,2900,2600,2500,2650 1600,1650,1900,1950,2000,2200,2300,2300 GROCERY STORE 700,900,600,800,877,755,800,600 3000,2900,3100,2800,2899,2950,3400,3300 10000,11000,9000,9800,9700,9650,9770,9700 |
In this file listing, each line consists of four fields, separated by tabs. The first field is a string that identifies a store type. The second field is an array of eight numbers, separated by commas, which might be sales of alcohol over an eight-day period. The third and fourth fields are also arrays of eight numbers that could represent sales of tobacco and food, respectively, over the same eight-day period.
The sample data file has other fields in the same format, but these are not shown. These additional fields correspond to sales of other products (see the configuration file store-type.scatterviz for a listing of all the fields). Windows users find this file in the directory in which MineSet is installed, under \Examples\scatterviz\. UNIX users find the file in /usr/lib/MineSet/scatterviz/examples/store-type.scatterviz.
The data file cannot contain blank lines or comments. Missing or extra data on a line causes an error.
The Scatter Visualizer supports integer, floating-point number, and string data types, as well as arrays of these types. See Chapter 5, “Data and Configuration File Basics,” for more information on data types.
With the Scatter Visualizer, you can use fields that are one- or two-dimensional arrays of fixed size. In a fixed-sized array field, all entries of the given field are arrays with the same number of values. Arrays contain the data values across one or two independent variables (those dimensions controlled by the sliders). In the listing from the file store-type.data, the second, third, and fourth fields are arrays.
The configuration file format is flexible. You must separate the words with spaces, and the file is case-sensitive. Except for the include statement and text within quoted strings, spacing and line breaks are irrelevant.
As each section is encountered, a special configuration file (referred to as a defaults file) is also read in. Defaults files normally contain options statements. These files are read in the following order:
The scatterviz directory which usually contains system defaults. Windows users can find this in the directory in which MineSet was installed, under \config\scatterviz. UNIX users can find the directory in /usr/lib/MineSet/scatterviz.
The ~/.MineSet directory in which you can set up personal defaults (the tilde, ~, indicates your home directory).
The current directory, which lets you set up defaults for each directory.
Files with the same name can appear in more than one of the above-named directories; in this case, the order given is the one in which the directories are read. If the same option is found in multiple files, the last option read is used. Note that the appropriate section in the configuration file is read after all the defaults files; thus, options in the configuration file override those in the defaults files.
The first section of a configuration file is normally the input section. It defines the name and format of the data file. A typical input section might appear as follows:
input {
file "company.data";
string company;
slider int income from 20000 to 60000 by 10000;
slider date “%N %Y” purchaseDate from “Jan 1990” to “Dec
1992” by “1 month”;
options array separator `,';
float lifeSales[income][purchaseDate];
float autoSales[income][purchaseDate];
float homeSales[income][purchaseDate];
string location;
}
|
This example states that the input file is called company.data, and that there are five fields: company, lifeSales, autoSales, homeSales, and location. The company and location fields are of type string, and the other three fields are two-dimensional arrays of type float. Two slider dimensions are declared:
income, which is of type int, ranges from 20000 to 60000 in increments of 10000.
purchaseDate, which is of type date and ranges from January 1990 to December 1992 in increments of one month.
The arrays lifeSales, autoSales, and homeSales contain values for each income and purchase date. Individual values within the arrays are separated by commas.
When the input section is entered, the defaults file inputDefaults is read in.
The file statement names the data file to be read. This statement is required. Its form is as follows:
file "filename"; |
filename must be in double quotation marks. If it is a relative pathname (no leading slash), it is first sought in the directory containing the current configuration file. If include statements are present, this might not be the same as the initially loaded configuration file. If it is not found in the current configuration file's directory, the file is sought in the current directory.
Enumeration statements declare enumerations, or enums, that index into array fields. The enum statement has three forms:
enum type name from value1 to value2 by increment;
This declares an enum with values starting at value1 and incremented by increment until they reach or exceed value2. For example, the following statement declares age as an enum with the values 20, 30, 40, 50, 60, and 70:
enum int age from 20 to 70 by 10;
Type must be a number type (int, float, or double) or date (see “Dates”).
enum type name from value1 to value2 across numberOfValues;
This declares an enum with values ranging from value1 to value2. The numberOfValues is an integer specifying the number of values. For example, the following statement declares age as an enum with the values 20, 30, 40, 50, 60, and 70:
enum int age from 20 to 70 across 6;
Type must be a number type (int, float, or double) or date (see “Dates”).
enum type name {value1, value2, ..., valueN};
Type can be any type or date (see “Dates”).
The enum statement includes special support for a date type that handles date and time values from January 1, 1753 forward. The date type is valid only within enum statements. A date enum statement can have the following types syntax:
enum date “format” name from “value1” to “value2” across numberOfValues;
enum date “format” name { value1, value2, ..., valueN };
enum date “format” name from “value1” to “value2” by “increment”;
|
The format string specifies the format of the values; it is useful for controlling how dates are displayed in the animation control panel. The syntax of the format string is similar to the scanf function in C. Various units of time are represented by special characters preceded by the percent symbol (%). For example:
enum date cq “Calendar Q%Q, %Y” from “Calendar Q1, 1980” to “Calendar Q3, 1985” by “1 quarter”; |
The “Calendar Q” in the format string matches the “Calendar Q” in value1 and value2. The %Q in the format string indicates that the next number in value1 and value2 is the calendar quarter. The comma and space in the format string match the commas and spaces in the values. Finally, the %Y in the format string specifies that the year values are next.
Table 9-1 lists the characters that can follow the percent symbol and the units of time they represent.
Table 9-1. Characters That Can Follow the Percent Symbol in the Format String
Character | Time Unit | Precision |
|---|---|---|
Y | 4 | |
Q | 1 | |
M | 2 | |
N | month name | >= 3 |
D | 2 | |
h | 2 | |
m | 2 | |
s | 2 |
With the exception of N, each character matches an integer of the specified precision. N matches 3 or more characters giving the English name of the month.
The from-to-by form of the enum statement includes an increment value. For dates, the increment is a quoted string containing an integer, an optional space, and one of the special characters in Table 9-1 or one of the symbols year, quarter, month, day, hour, minute, or second. The plural forms of these symbols are also accepted. These symbols are not keywords, because they have special meaning only in the increment string. The following are examples of valid increments:
“1 year” “7 days” “4h” |
The data statements declare the fields in the data file. You must declare the fields in the order they appear in the data file. The format of most data statements is as follows:
type name; |
type is int, float, double string, dataString, date, or fixedString(n), where n is an integer representing the width of the string; name is the variable name. Unlike in C, you can declare only one variable per statement.
You can also base a data field an enumeration. The syntax is as follows:
enum enumName name; |
The field must contain ints corresponding to the values of the enum. You can declare the enum ageGroup as:
enum string ageGroup {"below 30", "30-39", "40-49", "50-59", "60 or above"};
|
In this case, you can declare the field age:
enum ageGroup age; |
The field should contain ints between 0 and 4, where 0 is displayed as “below 30,” 1 as “30-39,” and so forth.
You can declare only one variable per statement.
Arrays are also declared using data declarations. The declaration syntax for one-dimensional arrays is one of the following:
type name [ number ] ; type name [ enumName ] ; type name [ null enumName ] ; |
For example:
float revenue [50]; |
The declaration syntax for two-dimensional arrays is one of the following:
type name [ number1 ][ number2 ] ; type name [ enumName1 ][ enumName2 ] ; type name [ null enumName1 ][ null enumName2 ] ; |
For example:
float revenue [50][10]; |
When enums are used, the number of values in the array is taken from the declaration of the enum. For example, given the following statements, the array clothingPurchases must have six values, corresponding to the enum values 20, 30, 40, 50, 60, and 70:
enum int age from 20 to 70 by 10; float clothingPurchases[age]; |
The keyword null indicates an extra value at the beginning of the array, corresponding to null. Thus, the following statements declare clothingPurchases as an array with seven values: the first value corresponding to null or unknown age values, and the remaining six values corresponding to age values 20, 30, 40, 50, 60, and 70:
enum int age from 20 to 70 by 10; float clothingPurchases[null age]; |
You can override the separator between values in an array by declaring it as:
type name [ number ] separator `char'; |
For example:
float revenue [50][10] separator `:'; |
If you do not specify a separator, the default separator (usually a tab) is used.
All options statements begin with the word “options” and have one or more comma-separated options.
The separator option defines the separator between fields in the data file. The default separator is a tab. The syntax is:
options separator `char';
For example:
options separator `:';
Note: The separator is used also to separate values within arrays; however, arrays can override the separator. The backslash option controls whether backslashes in the input data are treated specially or like other characters. The syntax is:
options backslash off; options backslash on;
The default is off. If backslash processing is on, separators in the input data preceded by backslashes are treated as regular characters rather than separators. Within strings, this causes standard C-style backslash processing.
The expressions section of a configuration file lets you define additional fields that are expressions of existing fields. For example, you can define a new field as the sum of two other fields.
The format of the expressions section is:
expressions
{
expressionDeclaration;
...
}
|
expressionDeclaration has the following form:
type name = expression ; |
The following is a sample expression section. This section assumes two existing array fields of type double: “male” and “female”. These represent spending by males and females on various goods over time (one independent dimension). Two fields are added: “total” represents the total dollars spent, and “pctFemale” represents the percentage of dollars spent by females.
expressions
{
double total[36] = male+female;
double pctFemale[36] = divide (female*100, total, 50.0);
}
|
The expressions section has no options; thus, no defaults file is read in for it.
The view section of a configuration file describes how the data is displayed, including the mapping of sizes, colors, axes, and so on. The default values for these options are in view.scatterviz.options. The Windows options are in the directory in which MineSet is installed under \Programs\scatterviz\view.scatterviz.options. UNIX users can find the file in /usr/lib/MineSet/scatterviz/view.scatterviz.options. The syntax is as follows:
view
{
viewStatement;
...
}
|
A sample view section is:
When entering the view section, the viewDefaults file is read in.
The slider statement identifies an enum to be used as a slider dimension. Its syntax is one of the following:
slider enumName; slider null enumName; |
The enum name is declared in the input section. If the keyword null is present, the slider includes a position at the beginning that corresponds to null or unknown values of the enum. You must declare arrays indexed by the slider to match the null in the slider statement.
There can be one, two or no slider statements. The first slider statement applies to the horizontal slider, the second applies to the vertical slider. If there is no slider statement, the resulting display does not include animation.
The entity statement lets you specify a variable that uniquely identifies the entities in the display. The entity statement consists of a series of clauses, separated by commas:
entity clause1, clause2,... |
Alternatively, the clauses can be given in separate entity statements.
The first clause of the entity statement normally contains the name of the entity variable (for instance, brand in the example on page 119).
This clause defines how the entities are labeled. It has the following forms:
label off
label on
This turns on the labels. The default labels use the entity variable as the label for each entity.
label variable
This turns on the labels and uses the given variable to label the entities. When this form is used, it is not necessary to specify label on.
This clause turns on the labels and specifies their color. It has the following form:
label color “colorname” |
colorname is the name of a color in a special format. (Color naming is explained in “Color Statement”.) The default label color is gray.
The legend clause explains what the entities are. Any string can be placed in the entity legend. The legend clause has the following forms:
legend off
This turns off the entity legend.
legend on
This turns on the entity legend (this is the default). The default legend is:
Entity: varname
varname is the name of the entity variable.
legend label “string”
This turns on the legend and explicitly sets the legend string. If this form is used, legend on is unnecessary.
The size statement describes how a field of data is mapped to the sizes of entities. The size statement consists of a series of clauses, separated by commas:
size clause1, clause2,... |
Alternatively, the clauses can be given in separate size statements.
The first clause normally contains the name of a field to be mapped to size (for instance, total$, in the view example in “Configuration File View Section”). The field must be of a number type (int, float, or double), of which float is the most efficient. The field can be an array that is indexed by slider dimensions. If no size field is specified, all entities are the same size.
Normally, the size variable is mapped to the size of the entities, so that the biggest entity has a size of 5. You can change this size by specifying a different value. If there is no size variable, the default maximum size is 5. The max clause has the form:
max float |
Instead of the max clause, you can use the scale clause to scale size values; all values are multiplied by the scale. The scale clause's syntax is:
scale float |
The legend clause defines the meaning of the size mappings. You can place any string in the size legend. The legend clause has the following forms:
legend off
This turns off the size legend.
legend on
This turns on the size legend (this is the default). The default legend is:
size:varname
varname is the name of the variable that is mapped to size.
legend label “string”
This turns on the legend and explicitly sets the legend string. If you use this form, legend on is unnecessary.
The color statement describes how values are mapped to colors. The format is similar to the size statement, consisting of several clauses that you can separate with commas, or enter as multiple statements. The syntax is:
color clause1, clause2,... |
Color names follow the conventions of the X Window system, except that the names must be in quotes. Examples of valid colors are “green,” “hot pink,” and “#77ff42.” The latter is in the form “#rrggbb,”, in which the red, green, and blue components of the color are specified in hexadecimal value. Pure saturation is represented by ff, a lack of color by 00. For example, “#000000” is black, “#ffffff” is white, “#ff0000” is red, and “#00ffff” is cyan.
Windows users should only use the “#ff0000”form, as only some of the named colors are supported (for example, white, black, gray, red, yellow, green, and so on).
UNIX users can find the color list file at /usr/lib/X11/rgb.txt.
As with size, you can specify a single field to be mapped to an entity color. The field can be an array that is indexed by slider dimensions. If the field is an array, it must be a number type. If the field is a number type, you can use the scale and buckets clauses described below to map a range of colors to the values of the field. If the field is not a number type, it is sorted, and each unique value is assigned a color.
The colors clause specifies the colors to be used. The syntax is:
colors "colorname" "colorname"... |
The format for colorname is described in “Color Naming”. There are no commas between the colors, because commas are used to separate clauses in the color statement. A sample colors clause is:
colors "red" "gray" "blue" |
Colors in the list are subsequently referred to by their index, starting at zero. In the above example, red is color 0, gray is color 1, and blue is color 2.
If there is no colors statement, colors are chosen randomly. If there is a colors statement, at least as many colors must be specified as will be mapped.
The scale clause allows assignment of values to a continuous range of colors. For example, when percentage is displayed, you can assign red to 0%, gray to 50%, and blue to 100%. Intermediate values are interpolated; for example 25% is pinkish, and 55% is a slightly bluish gray.
The syntax for the scale clause is:
scale float float ... |
The first value is mapped to color 0, the second to color 1, and so forth. The colors statement must contain at least as many colors as will be mapped to the largest index.
Values in this statement must be in increasing order. Any value less than the first color is assigned the value of the first color. Any value greater than the last value is assigned the last color. Intermediate values are interpolated.
For example, assume the pctFemale field indicates what percentage of the group is female, and you want to map a group that is 100% female to red, 100% male to blue, and 50% each to gray. The colors statement for this is:
colors pctFemale, colors "blue" "gray" "red", scale 0 50 100; |
Use the scale clause only in conjunction with a numeric color variable.
The buckets clause is similar to the scale clause without interpolation. All values are rounded down to the highest value in the clause, and that exact color is used. Values less than the first value use the first color.
The syntax for the buckets clause is:
buckets float float ... |
The syntax and assignment of colors is the same as for the scale clause.
If, in the previous example, you used the buckets clause instead of the scale clause, the statement would be:
colors pctFemale, colors "blue" "gray" "red", buckets 0 50 100; |
All values greater than or equal to 100 are colored red. Values greater than, or equal to, 50 but less than 100, are gray. All other values are then blue.
Use the buckets clause only with a numeric color variable.
The legend clause creates a legend of the colors. The legend clause syntax can be any of the following:
legend off legend on legend "string" "string" ... legend label "string" |
The legend off clause turns the off legend. The legend on clause turns on the legend. You can omit it if other legend statements are included. Specifying only legend on generates the default legend.
The default legend includes a single label to the left (with the name of the field that is mapped to color), and a list of colored labels on the right (with values obtained from the scale clause, the buckets clause, or from the field). To override the strings in the colored labels, specify the strings as:
legend "string" "string" |
To override the label on the left, specify it following the word label. To eliminate this label, specify an empty string; as follows:
legend label "" |
The axis statement causes a variable to be used as an axis in the 3D landscape. The variable's values determine where the entities are positioned on the axis. There can be up to three axis statements. Like the size and color statements, the axis statement contains a series of comma-separated clauses, but you must specify all of them in a single statement.
axis clause1, clause2,... |
As with size and color, you can specify a field to be used as an axis. The field can be an array that is indexed by slider dimensions. If the field is an array, it must be of numeric type. If the field is non-numeric, it is sorted, and each unique value is assigned a position along the axis.
The label clause has the following form:
label "string" |
The string is used to label the axis. It appears in the landscape, at the end of the axis line. The default label is the name of the axis variable.
Normally, the axis variable is mapped directly to the position of the entities along the axis. The max clause lets you normalize the values of the axis variable, so that the maximum value is mapped to the specified max. The max clause's syntax is:
max float |
Instead of using the max clause to affect position values, you can use the scale clause to scale the values. All values are multiplied by the scale. The scale clause syntax is:
scale float |
The color clause specifies the color used for the axis line and label. It has the following form:
color "colorname" |
The extend clause specifies whether the axis should be extended automatically to include the value zero. It has the following form:
extend on extend off |
The summary statement specifies a summation to be calculated over all the entities. You can use the summary to color the drawing window in the animation control panel. Like the size and color statements, the summary statement has several clauses that you can specify in one statement, separated by commas, or in separate statements as follows:
summary clause1, clause2,... |
You can specify the variable to be used in the summary. This variable must be of numeric type. Typically, the summary variable is an array indexed by slider dimensions, so that the summary value varies across the slider dimensions.
The color clause specifies the color used to display the summary values in the drawing window. It has the following form:
color “colorname” |
Various shades of the color, from white to the specified color, are used to represent summary values. The minimum summary value is mapped to white, while the maximum summary value is mapped to the specified color. The default summary color is red.
The legend clause creates a legend of the summary colors. The legend clause syntax can be any of the following:
legend off legend on legend label "string" |
The legend off clause turns off the legend. The legend on clause turns on the legend. You can omit it if other legend statements are included. Specifying only legend on generates the default legend.
The legend includes a single label to the left (which defaults to the aggregation function and variable used in the summary), and two colored labels on the right (with the minimum and maximum summary values). To override the label on the left, specify it following the word label. To eliminate this label, specify an empty string as follows:
legend label "" |
The drillthrough statement specifies a column to be used to define the drill through expression. This column must be string valued, and should specify an expression that defines the object when selected (assuming each row defines an object).
The syntax for this statement is of the following form:
drillthrough stringColumnName; |
Drill through mapping is used in the .scatterviz file generated by the associations mining algorithm. It is used here because the table in the datafile contains rules and properties of the rules, not the columns in the original dataset.
The Selections > Drill Through Columns menu in the Scatter Visualizer has no effect when a specific drill through column has been mapped.
The message statement specifies the message displayed when an entity is selected. The syntax is similar to that of the C printf statement. A sample message statement is:
message "%s: $%f, %.0f%% of target, %.0f%% of last year",
product, sales, pctTarget, pctLastYear;
|
This could produce the following message:
furniture: $2425.37, 23% of target, 87% of last year |
The formats must match the type of data being used:
Strings must use %.
Ints must use integer formats (such as %d).
Floats and doubles must use floating point formats (such as %f).
For details of the printf format, UNIX users can see the printf (1) reference (man) page (type man printf at the shell prompt).
A special format type has been added to printf. If the percent sign is followed by a comma (for example, “%,f”), commas are inserted in the number for clarity. Only the U.S. convention of d,ddd,ddd.dddd is supported, with the decimal point represented by a period, and commas separating every three places to the left of the decimal point. For example, the format could be:
message "%s: $%,f, %,.0f%% of target, %,.0f%% of last year",
product, sales, pctTarget, pctLastYear;
|
In this case, it would produce this message:
furniture: $2,425.37, 23% of target, 87% of last year |
The $, *, h, l, ll, L, and n printf format options are not supported.
All values, including the format string, are expressions. Therefore, if you had a pctFemale field, but wanted a more gender-neutral message, you could use the following:
message pctFemale>50?"%f%% females":"%f%% males",
pctFemale>50?pctFemale:100-pctFemale;
|
If pctFemale is 70, the message “70% females” is displayed; if pctFemale is 30, the message “70% males” is displayed. In this case, you can also achieve the same result with a single format string:
message "%f%% %s", pctFemale>50?pctFemale:100-pctFemale,
pctFemale>50?"females":"males";
|
If you do not specify a message, a default message containing the names and values of all the fields is used.
The execute statement lets you execute a shell command by double-clicking an object. The syntax is similar to that of the message statement.
The following is a sample execute statement you can use on UNIX that uses xconfirm to show a window with information about the item. The command line (string) is shown as three lines. In an actual file, this should be on a single line. Multi-line strings are not supported.
execute "xconfirm -t '%s' -t 'population %,.0f' -t '%,.0f per
sq mile' -t '%,.0f sq-miles' > /dev/null", states, population,
density, sqMiles;
|
This might produce a the following message:
CA 64 per sq mile 266,807 sq-miles |
The following is a sample Windows execute statement (a simple xconfirm utility is provided with the installation):
execute "xconfirm -t '%s has a population of %,.0f' -t 'averaging %,
.1f per sq mile' -t 'across %,.0f sq-miles'",provinces,
population, density, sqMiles;
|
If there is no execute statement, double-clicking an object has the same effect as single-clicking it.
The filter statement specifies that only entities meeting certain filter criteria are displayed initially. The filter criteria are in the form of expressions whose values must all be true or nonzero for an entity to be displayed (expressions are described Chapter 5, “Data and Configuration File Basics,” in “MineSet Expression Language”).
The syntax of the filter statement is:
filter expression, expression,... |
For example, the following statement specifies that only records from California or Washington, with sales greater than 9000 and a pctTarget value greater than or equal to 90, should be displayed initially:
filter state == "CA" || state == "WA", sales > 9000, pctTarget >= 90; |
After the Scatter Visualizer is invoked, you can change or remove the filter criteria interactively by using the filter panel.
The view section of the configuration file has several options for controlling parameters of the display. These options can appear in a single options statement, separated by commas, or in separate options statements. The syntax of the options statement is:
options option, option,... |
The following options are available:
entity label size float
axis label size float
hide entity label distance float
Controls the distance at which entity labels become invisible. Smaller distances might improve performance, but the labels disappear more quickly.
grid color “colorname”
grid size float float float
Controls the spacing between grid lines. It applies the three values to grid lines along the x, y, and z axes, respectively.
entity shape shapeName
Specifies the shape used to display entities. shapeName can be “cube,” “bar,” or “diamond.”
background color “colorname”
Specifies the initial background color.
orientation top right front float float float
Specifies the initial orientation of the scene. You may use three floating point values to specify an arbitrary orientation vector.
perspective on off
Specifies whether or not to use perspective initially. The default is to use it.
upvector float float float
Specifies a camera upvector for defining a frame of reference. The default is 0.10.1. This option combined with the orientation option allows any initial viewing orientation.