The first part of this appendix describes the types and formats of data supported by the Splat Visualizer. Data input to the Splat Visualizer must be provided as a single file containing raw data, in a tab-separated ASCII text.
The second part discusses the configuration file, which describes how the Splat Visualizer reads in, and displays, the data file.
Both the data and configuration files can be generated automatically by the Tool Manager (see Chapter 3).
In its simplest form, the data file consists of a list of lines, each containing a set of fields, each separated by one tab. (Other separators are also allowed, but only one per file can separate each field. See “Input Options”.) All lines must contain the same fields. The interpretation of the fields is specified by the configuration file, described in the next section. Using the adultJobs data file provided as part of the Splat Visualizer package (file /usr/lib/MineSet/splatviz/examples/adultJobs.data), the first few lines of the input file appear as:
Bachelors Adm-clerical 3 3 51189.4869565217 115 Bachelors Exec-managerial 2 5 70722.6271186441 59 Bachelors Adm-clerical 2 3 37876.328358209 134 Bachelors Exec-managerial 3 0 34436.8 5 Bachelors Tech-support 1 2 37583.66667 3 Bachelors Tech-support 1 3 13711.33333 3 Bachelors Tech-support 1 4 29878.74193 31 |
In this sample file listing, each line consists of six fields, separated by tabs. The first field is a string that identifies level of education. The second field is a string which identifies occupation. The third field identifies the age bin. The fourth field identifies the number of hours per week worked bin. The fifth field quantifies the average gross income. The sixth field is the weight of records in the aggregate (ie the record count, unless record weighting has been used). This data file was derived from /usr/lib/MineSet/data/adult94.data by performing Tool Manager operations (specifically binning and aggregation).
The data file cannot contain blank lines or comments. Missing or extra data on a line causes an error.
The Splat Visualizer supports the following seven data types:
float represents a single-precision floating point number. The decimal point is optional. Numbers in exponential “e” notation are also accepted.
double represents a double-precision floating point number. The decimal point is optional when representing a floating point number. Numbers in exponential “e” notation are also accepted. The superior precision of double can be useful for accurately representing large numbers, since float can represent only seven or eight significant digits accurately. This superior accuracy, however, consumes twice the memory space of float.
dataString represents a string that is unlikely to appear multiple times. If it appears multiple times, several copies are made. A dataString is typically used to store an address. Addresses are unlikely to be compared, and each record can have a different address.
string represents a string of characters that can appear multiple times in the data file. Unlike a dataString, only a single copy of a given string is stored in memory, no matter how many times it appears in the data. This saves much memory for strings appearing many times.
Comparing strings is also much quicker than comparing dataStrings. Processing is somewhat slower when looking for duplicate strings as they are read in. An example of string use is for a division name that appears once for each department in the division. If you are unsure whether to use a string or a dataString, use a string.
fixed string represents a string of fixed length. Like a dataString, if a fixed string appears multiple times, multiple copies are made. In general, fixed strings are used internally for representations of data from data bases, and are generally better to use than strings or dataStrings.
date represents a date and time. In the data file, date must appear in the format MM/DD/YY HH:MM:SS.
Any field element in the data file can also have the value “?” (question mark), indicating an unknown or null value (see the discussion of nulls in Appendix J).
The configuration file format is flexible. Words in it must be separated by spaces, and it is case-sensitive. Except for the include statement and text within quoted strings, spacing and line breaks are irrelevant.
The configuration file consists of a series of sections, each of which has the form:
sectionKeyword
{
statements...
}
|
where sectionKeyword names the section. The order of the sections is significant, since sections can refer to variables defined in previous sections.
As each section is encountered, a special configuration file (referred to as a defaults file) is also read in. Defaults files normally contain options statements. These files are read in the following order:
Files with the same name can appear in more than one of the above-named directories; in this case, the order given is the one in which the directories are read. If the same option is found in multiple files, the last option read is used. Note that the appropriate section in the configuration file is read after all the defaults files; thus, options in the configuration file override those in the defaults files.
A statement has the following form:
statementKeyword info ; |
where statementKeyword defines the statement, and info varies according to the keyword.
A variable name can appear in two formats:
In the first format, it is a letter followed by a number of letters, digits, or underscores. It cannot be a keyword, and should not be quoted.
In the alternate form, the variable name should be surrounded by back quotes (`). In this form, the variable name can match a keyword, and can contain even non-alphanumeric characters. Configuration files generated automatically by the Tool Manager use this form.
There is no scoping of variable names; a given variable name can only be declared once in the configuration file.
Many sections have options statements, which have the form
options optionName info, optionName info... ; |
where optionName defines the specific option, and info depends on the option. In some cases, optionName can be more than one word. To maximize the number of allowable variable names, most option names are meaningful only within the appropriate options statement; option names do not conflict with variable names. You can declare several options on the same line, separating them by commas or placing them in several options statements. If two conflicting values for the same option appear, the last value is taken.
The configuration file can contain lines of the form
include "filename" |
These lines can appear anywhere in the configuration file, but each must be on its own line. The filename must be in quotation marks; anything after the closing quote is ignored. The number of nested includes is unlimited. If a relative pathname (one not beginning with a slash) is specified, the file is first sought in the directory containing the current configuration file. If include statements are present, this might not be the same as the initially loaded configuration file. If it is not found in the directory containing the current configuration file, the include file is sought in the current directory.
A statement similar to an include is sinclude, which has the form
sinclude "filename" |
This is identical to the include statement, except that no error is given if the file does not exist; instead, the sinclude statement is ignored.
Strings and characters in the configuration file follow C conventions. Strings are in double quotation marks ("), and characters are in single quotation marks ('). All standard backslash conventions are followed (for example, \n represents a new line).
Comments begin with a pound (#) symbol at the beginning of a line; anything after this symbol to the end of the line is ignored, up to the end of the line.
The keywords recognized by the Splat Visualizer are listed in Table E-1. Variables cannot have these names unless they are surrounded by back quotes (`). Tokens appearing only in option statements are not keywords, and can be used for variable names.
Table E-1. Splat Visualizer Keywords
|
|
|
|
|---|---|---|---|
across | average | axis | buckets |
by | color | colors | weight |
dataString | date | divide | double |
execute | expressions | file | float |
from | include | input | int |
key | label | legend | max |
message | min | modulus | monitor |
off | on | options | opacity |
scale | separator | sinclude | size |
slider | string | sum | summary |
time | to | view | background |
Currently, the keywords execute, monitor, weight, and time are not used by the Splat Visualizer.
The first section of a configuration file is normally the input section. It defines the name and format of the data file. A typical input section might look like this:
input {
file "adultJobs.data";
enum string `age_bin_k`{"- 20", "20-30", "30-40",
"40-50", "50-60", "60-70", "70+"};
enum string `hours_per_week_bin_k`{"- 20", "20-25", "25-30", "30-35", "35-40", "40-45", "45-50", "50-55", "55-60", "60-65", "65-70", "70+"};
string `education`;
string `occupation`;
enum `age_bin_k``age_bin`;
enum `hours_per_week_bin_k``hours_per_week_bin`;
double `avg_gross_income`;
int `count_gross_income`;
}
|
This example states that the input file is called adultJobs.data, and that there are six fields: education, occupation, age_bin, hours_per_week_bin, avg_gross_income, and count_gross_income. The education and occupation fields are of type string. The age_bin and hours_per_week_bin are of type enum, where the values of these enums is defined by age_bin_k and hours_per_week_bin_k respectively. The column avg_gross_income is of type double and the field count_gross_income is of type int.
When the input section is entered, the defaults file inputDefaults is read in.
The file statement names the data file to be read. This statement is required. Its form is:
file "filename"; |
filename must be in double quotation marks. If it is a relative pathname (no leading slash), it is first sought in the directory containing the current configuration file. If include statements are present, this might not be the same as the initially loaded configuration file. If it is not found in the current configuration file's directory, the file is sought in the current directory.
Enumeration statements declare enumerations, or enums. The enum statement has three forms.
The first enum statement form is
enum type name from value1 to value2 by increment;
This declares an enum with values starting at value1 and incremented by increment until they reach or exceed value2. For example, the statement
enum int age from 20 to 70 by 10;
declares age as an enum with the values 20, 30, 40, 50, 60, and 70.
Type must be a number type (int, float, or double) or date (see “Dates”).
The second enum statement form is
enum type name from value1 to value2 across numberOfValues;
This declares an enum with values ranging from value1 to value2. The numberOfValues is an integer specifying the number of values. For example, the statement:
enum int age from 20 to 70 across 6;
declares age as an enum with the values 20, 30, 40, 50, 60, and 70.
Type must be a number type (int, float, or double) or date (see “Dates”).
The third enum statement explicitly lists the enum values. Its form is:
enum type name { value1, value2, ..., valueN };Type can be any type or date (see “Dates”).
The enum statement includes special support for a date type that handles date and time values starting Jan 1, 1753. The date type is valid only within enum statements. A date enum statement can have the following syntaxes:
enum date “format” name from “value1” to “value2” across numberOfValues;
enum date “format” name { value1, value2, ..., valueN };
enum date “format” name from “value1” to “value2” by “increment”;
|
The format string specifies the format of the values; it is useful for controlling how dates are displayed in the animation control panel. The syntax of the format string is similar to the scanf function in C. Various units of time are represented by special characters preceded by the percent symbol (%). For example,
enum date cq “Calendar Q%Q, %Y” from “Calendar Q1, 1980” to “Calendar Q3, 1985” by “1 quarter”; |
The “Calendar Q” in the format string matches the “Calendar Q” in value1 and value2. The %Q in the format string indicates that the next number in value1 and value2 is the calendar quarter. The comma and space in the format string match the commas and spaces in the values. Finally, the %Y in the format string specifies that the year values are next.
Table E-2 lists the characters that can follow the percent symbol and the units of time they represent.
Table E-2. Characters That Can Follow the Percent Symbol in the Format String
Character | Time Unit | Precision |
|---|---|---|
Y | 4 | |
Q | 1 | |
M | 2 | |
N | month name | >= 3 |
D | 2 | |
h | 2 | |
m | 2 | |
s | 2 |
With the exception of N, each character matches an integer of the specified precision. N matches 3 or more characters giving the English name of the month.
The from-to-by form of the enum statement includes an increment value. For dates, the increment is a quoted string containing an integer, an optional space, and one of the special characters in Table E-2 or one of the symbols year, quarter, month, day, hour, minute, and second. The plural forms of these symbols are also accepted. Note that these symbols are not keywords, since they have special meaning only in the increment string. The following are examples of valid increments:
“1 year” “7 days” “4h” |
The data statements declare the fields in the data file. The fields must be declared in the order they appear in the data file. The format of most data statements is
type name; |
where type is int, float, double string, dataString, date, and fixedString(n), where n is an integer representing the width of the string; name is the variable name. Unlike in C, only one variable can be declared per statement.
A data field can also be based on an enumeration. The syntax is
enum enumName name; |
The field must contain ints corresponding to the values of the enum. For example, if the enum ageGroup is declared as
enum string ageGroup {"below 30", "30-39", "40-49", "50-59", "60 or above"};
|
the field age can be declared as
enum ageGroup age; |
The field should contain ints between 0 and 4, where 0 is displayed as “below 30,” 1 as “30-39”, and so forth.
Only one variable can be declared per statement.
All options statements begin with the word “options” and have one or more comma-separated options.
The separator option defines the separator between fields in the data file. The default separator is a tab. The syntax is
options separator `char';
For example:
options separator `:';
The backslash option controls whether backslashes in the input data are treated specially or like other characters. The syntax is:
options backslash off; options backslash on;
The default is off. If backslash processing is on, separators in the input data preceded by backslashes are treated as regular characters rather than separators. Within strings, this causes standard C-style backslash processing.
The view section of a configuration file describes how the data is displayed, including the mapping of sizes, colors, axes, and so on. The default values for these options are in /usr/lib/MineSet/splatviz/view.splatviz.options. Its form is
view
{
viewStatement;
...
}
|
view {
slider `age_bin`;
opacity `count_gross_income`;
color `avg_gross_income`;
axis `education`, color "grey";
axis `occupation`, color "grey";
axis `hours_per_week_bin`, max 100, color "grey";
options grid size 0 0 0;
summary `count_gross_income`, color "red";
}
|
When entering the view section, the viewDefaults file is read in.
The slider statement identifies an enum column to be used as a slider dimension. Its syntax is one of the following:
slider columnName; |
The columnName name is declared in the input section. If this column contains nulls, the slider includes a beginning position corresponding to those null values.
There can be 0, 1, or 2 slider statements. The first slider statement applies to the horizontal slider, the second to the vertical slider. If there is no slider statement, the resulting display does not include animation.
In the Splat Visualizer, the opacity is based on counts or record weights. If a column is mapped to this requirement, it is used to weight each record (rather than using 1), when computing a value for the opacity. Thus, if you had a column with values for population, density, or the result of a count aggregation, you might want to map this column to the opacity (weight) requirement. If you had no such column, the requirement is left unmapped, and a column of 1's is used by default.
An opacity type for each axis. (This can be Max Opacity, or Scale Opacity.)
Max Opacity lets you specify that an axis is scaled independently to a specified opacity. If one axis has a Max Opacity that is twice as opaque as the other, it will be twice as opaque, regardless of the data values. This option is most useful when comparing axes that are in different units (for example, comparing income to age). This option has no effect on non-numeric data.
Scale Opacity lets you specify that the axis is scaled based on its maximum value. If two axes have the same Scale Opacity, but one has a maximum that is twice the value of the other, the former will be twice as opaque as the latter. This option is useful for comparing axes with the same units (for example, income vs. expenses). This option does affect the opacity of non-numeric axes.
In the Splat Visualizer, this column (if present) is sum aggregated, and its name is prepended with "sum_". If no column is present, the record count is used. In either case, the values of this column are used to compute opacity. The legend at the bottom of the main window shows this column name after "opacity:".
The opacity statement describes how a column is mapped to the opacity of the splats. The opacity statement consists of a series of clauses, separated by commas:
opacity clause1, clause2,... |
Alternatively, the clauses can be given in separate opacity statements.
The first clause normally contains the name of a field to be mapped to opacity (count_gross_income, in the view example in “The View Section”). The field must be of a number type (int, float, or double).
The legend clause defines the meaning of the opacity mapping. The legend clause has the following forms:
legend off
This turns off the opacity legend.
legend on
This turns on the opacity legend (this is the default). The default legend is:
opacity:count
where count is a column that the tool has created by counting the number of records in each aggregate. If a column was mapped to opacity, the name of this column prepended with "sum_" is shown in the legend. This new column is computed by sum aggregating the column mapped to the opacity requirement.
legend label “string”
This turns the legend on and explicitly sets the legend string. If this form is used, legend on is unnecessary.
The color statement describes how values are mapped to colors. The format is similar to the opacity statement, consisting of several clauses that can be separated by commas, or entered as multiple statements. The syntax is:
color clause1, clause2,... |
Color names follow the conventions of the X window system, except that the names must be in quotes. Examples of valid colors are “green,” “Hot Pink,” and “#77ff42.” The latter is in the form “#rrggbb”, in which the red, green, and blue components of the color are specified in hexadecimal value. Pure saturation is represented by ff, a lack of color by 00. For example,”#000000” is black, “#ffffff” is white, “#ff0000” is red, and “#00ffff” is cyan.
As with opacity, you also can specify a column to be mapped to splat color. If the column is a number type, the scale and buckets clauses described below can be used to map a range of colors to the values of the field.
The colors clause specifies the colors to be used. The colors clause's syntax is:
colors "colorname" "colorname"... |
The format for colorname is described in “Color Naming”. Note that there are no commas between the colors, because commas are used to separate clauses in the color statement. A sample colors clause is:
colors "red" "gray" "blue" |
Colors in the list are subsequently referred to by their index, starting at zero. In the above example, red is color 0, gray is color 1, and blue is color 2.
If there is no colors statement, colors are chosen randomly. If there is a colors statement, at least as many colors must be specified as are to be mapped.
The scale clause allows assignment of values to a continuous range of colors. For example, when displaying a percentage, red can be assigned to 0%, gray to 50%, and blue to 100%. Intermediate values are interpolated; for example 25% is pinkish, and 55% is a slightly bluish gray.
The syntax for the scale clause is
scale float float ... |
The first value is mapped to color 0, the second to color 1, and so forth. The colors statement must contain at least as many colors as are to be mapped to the largest index.
Values in this statement must be in increasing order. Any value less than the first color is assigned the value of the first color. Any value greater than the last value is assigned the last color. Intermediate values are interpolated.
For example, assume the pctFemale field indicates what percentage of the group is female, and you want to map a group that is 100% female to red, 100% male to blue, and 50% each to gray. The colors statement for this is:
colors pctFemale, colors "blue" "gray" "red", scale 0 50 100; |
Use the scale clause only in conjunction with a numeric color variable.
The buckets clause is similar to the scale clause without interpolation. All values are rounded down to the highest value in the clause, and that exact color is used. Values less than the first value use the first color.
The syntax for the buckets clause is
buckets float float ... |
The syntax and assignment of colors is the same as for the scale clause.
If, in the above example, you used the buckets clause instead of the scale clause, the statement would be:
colors pctFemale, colors "blue" "gray" "red", buckets 0 50 100; |
All values greater than or equal to 100 are colored red. Values greater than, or equal to, 50 but less than 100, are gray. All other values are then blue.
Use the buckets clause only with a numeric color variable.
The legend clause creates a legend of the colors. The legend clause syntax can be any of the following:
legend off legend on legend "string" "string" ... legend label "string" |
The legend off clause turns the legend off. The legend on clause turns the legend on. It can be omitted if other legend statements are included. Specifying only legend on generates the default legend.
The default legend includes a single label to the left (with the name of the field that is mapped to color), and a list of colored labels on the right (with values obtained from the scale clause, the buckets clause, or from the field). To override the strings in the colored labels, specify the strings as:
legend "string" "string" |
To override the label on the left, specify it following the word label. To eliminate this label, specify an empty string; that is:
legend label "" |
The axis statement causes a variable to be used as an axis in the 3D landscape. The variable's values determine where the entities are positioned on the axis. There can be up to three axis statements. Like the size and color statements, the axis statement contains a series of comma-separated clauses, but all of them must be specified in a single statement.
axis clause1, clause2,... |
As with size and color, you can specify a field to be used as an axis. The field can be an array that is indexed by slider dimensions. If the field is an array, it must be of type number. If the field is not of type number, it is sorted, and each unique value is assigned a position along the axis.
The summary statement specifies aggregate information to be calculated for all data defined by the slider position. The summary is used to color the drawing window in the animation control panel. Like the opacity and color statements, the summary statement has several clauses that can be specified in one statement, separated by commas, or in separate statements.
summary clause1, clause2,... |
You can specify the variable to be used in the summary. This variable must be of number type. If no summary variable is specified, sum of counts is used. If a variable is specified, then the weighted average of that variable (for all the data at the slider location) is used.
The color clause specifies the color used to display the summary values in the drawing window. It has the form
color “colorname” |
Various shades of the color, from white to the specified color, are used to represent summary values. The minimum summary value is mapped to white, while the maximum summary value is mapped to the specified color. The default summary color is red. If no slider variable is specified, this statement has no effect.
The legend clause creates a legend of the summary colors. The legend clause syntax can be any of the following:
legend off legend on legend label "string" |
The legend off clause turns the legend off. The legend on clause turns the legend on. It can be omitted if other legend statements are included. Specifying only legend on generates the default legend.
The legend includes a single label to the left (which defaults to the aggregation function and variable used in the summary), and two colored labels on the right (with the minimum and maximum summary values). To override the label on the left, specify it following the word label. To eliminate this label, specify an empty string; that is
legend label "" |
The view section of the configuration file has several options for controlling parameters of the display. These options can appear in a single options statement, separated by commas, or in separate options statements. The syntax of the options statement is
options option, option,... |
The following options are available:
axis label size float
background color “colorname”
hide label distance float
controls the distance at which axis labels become invisible. Smaller distances might improve performance, but the labels disappear more quickly.
grid color “colorname”
grid size float float float
controls the spacing between grid lines. It applies the three values to grid lines along the x, y, and z axes, respectively.
shape splatType
specifies the type of splat used. The shapeName can be “constant,” “linear,” "gaussian","texture", or “sphere.”