Chapter 1. Getting Started

This introduction provides an overview of MineSet, an integrated suite of data mining and visualization tools, and describes the basic tool execution scenario.


Note: Before using any of the MineSet tools, follow the installation and licensing instructions in the MineSet release notes. Then your system administrator must set up the DataMover configuration file. You also can choose to set up various options. The setup details are described in Chapter 2.


MineSet Tools Suite

The MineSet suite of tools lets you mine and graphically display quantitative information in ways that can help you better visualize, explore, and understand your data. This suite of data mining and analysis tools can help you organize and examine your data in new and meaningful ways. The mining tools automatically find patterns and build models that can be viewed using the visualization tools. The visualization tools can also be applied directly to the data for further insights. These tools provide an enabling power that lets you gain a deeper, intuitive understanding of your data, and helps you discover hidden patterns and important trends.

These tools provide a highly interactive, three-dimensional (3D) visual interface that lets you manipulate visual objects on the screen, as well as search, filter and perform animations. This ability to visualize and survey complex data patterns can prove invaluable for decision support, in business intelligence and knowledge management.

The MineSet suite consists of three basic components:

  • a centralized control module, consisting of a graphical user interface tool called the Tool Manager, and a process called the DataMover, which runs on the server part of MineSet's client/server architecture.

  • analytical data mining, with nine data mining tools:

    • Association Rules Generator

    • Automatic Binning

    • Cluster Generator

    • Column Importance

    • Decision Table Inducer and Classifier

    • Decision Tree Inducer and Classifier

    • Evidence Inducer and Classifier

    • Option Tree Inducer and Classifier

    • Regression Tree Inducer and Regressor

  • visualization tools, which let you view your data using ten different visual metaphors:

    • Cluster Visualizer

    • Decision Table Visualizer

    • Evidence Visualizer

    • Map Visualizer

    • Record Viewer

    • Rules Visualizer

    • Scatter Visualizer

    • Splat Visualizer

    • Statistics Visualizer

    • Tree Visualizer

The following sections provide a brief description of each of the above-mentioned components.

Tool Manager

Each of the mining and visualization tools described below can be configured and started via a consistent graphical user interface known as the Tool Manager. The Tool Manager

  • connects you to the server on which the analytical mining and transformations are performed

  • lets you access, query and transform data

  • creates configuration files for each tool

DataMover

The DataMover is a process that runs on the server on behalf of the user. The DataMover

  • connects to databases, flat files (ASCII or binary), and retrieves the data

  • invokes the mining tools

  • performs additional data manipulation such as binning and aggregation

  • returns the data to the Tool Manager for distribution to the visualization tools

  • can store the data in files on the server or client for future operations.

Association Rules Generator

The Association Rules Generator processes an input file, then generates an output file consisting of rules. These rules indicate the frequency with which one item occurs in a record along with another item. The strength of the association is quantified by three numbers.

  • The first number, the predictability of the rule, quantifies how often an item X and an item Y occur together as a fraction of the number of records in which X occurs. For example, given that someone has bought milk, how often do they also buy eggs.

  • The second number, the prevalence of the rule, quantifies how often X and Y occur together in the file as a fraction of the total number of records. For example, how often were milk and eggs bought together.

  • The third number is expected predictability. This gives an indication of what the predictability would be if there were no relationship between the items in the record. For example, how often were eggs bought, regardless of whether milk was bought as well.

Automatic Binning

Automatic Binning groups together closely spaced numerical data into discrete categories. Some data mining algorithms, such as the Decision Tree Inducer, require some discrete (categorical) data; similarly, visualization tools such as the Splat Visualizer may need data categorized in this way.

MineSet can automatically determine these categories, or you can determine how you need it done. Requirements can be as simple as dividing the data into three equal ranges; or as complex as having MineSet choose ranges differentiated according to some chosen attribute, at the same time discarding the outer five percent of the data as outliers.

Clustering

Clustering segments data into similar groups or clusters. For example, you can ask MineSet to suggest a segmentation of customers into five distinct groups, without giving any further parameters. Once the clustering operation has been run, you can view the results in the Cluster Visualizer; or apply the clustering model to the current data, then analyze the resulting clusters in any MineSet visualization or mining tool.

Column Importance

Column Importance determines how important various attributes are for determining the value of a given label attribute. For example, you can ask MineSet to select automatically the best three attributes that help determine whether someone is a good credit risk. The system might select income, own-house, and car-cost. These attributes can then be used to configure various visualizers.

Column Importance has an advanced mode that provides additional capabilities. First, it lets you determine how important each of the attributes is. (For example, you could determine that both income and salary are similar in importance in determining credit risk. Although income might be slightly better in determining importance, you might prefer to use salary because it is easier to obtain.) Second, once you explicitly choose an attribute, you can determine what other attributes are important in conjunction with it. (For example, if you have chosen salary rather than income, house-cost might become more important than own-house, and income would have a very low importance.)

Decision Table Inducer and Classifier

The Decision Table Classifier classifies data by making a series of consecutive decisions leading to the classification based on a record's attributes. It can be used to predict events such as whether a bank customer is likely to default on a loan, or a homeowner is likely to refinance their mortgage.

The Decision Table Inducer creates a Decision Table Classifier from the data. Attributes are tested to classify the data, and you have the option to set the order in which the tests are run as well. The resulting Decision Table Classifier can be viewed using the Decision Table Visualizer, so you can simultaneously explore multiple attribute tests, two at a time.

Decision Tree Inducer and Classifier

The Decision Tree Classifier classifies data according to a set of attributes by making a series of decisions based on those attributes. Applying this classifier to determine the profile of someone with credit worthiness, for example, a decision tree might determine if someone who owns a home, owns a car that cost between $15,000 and $23,000, and has two children, is a good credit risk.

The Decision Tree Inducer generates a Decision Tree Classifier, the structure of which is displayed using the Tree Visualizer, each decision being represented by a node of the tree. The graphical representation helps you understand the model, as well as gives valuable insight into the data, by using visual searching and filtering.

Evidence Inducer and Classifier

The Evidence Classifier classifies data by examining the probabilities of a specified result occurring based on a given attribute. For example, it might determine that someone who owns a car that cost between $15,000 and $23,000 has a 70% chance of being a good credit risk, and a 30% chance of being a bad credit risk. The classifier predicts the class with the highest probability based on a simple probabilistic model.

The model is displayed using the Evidence Visualizer, which shows pie charts illustrating the different probabilities. This graphical representation can help the user understand the classification algorithm, as well as providing valuable insights into the data and answering “what if” questions.

Option Tree Inducer and Classifier

The Option Tree Classifier classifies data using a technique similar to the Decision Tree Classifier. Unlike decision trees, option trees can contain special option nodes, which allow the classifier to consider the influence of splitting on multiple attributes simultaneously. For example, an option node in an option tree built to identify a car's country of origin might choose miles per gallon, horsepower, number of cylinders, and weight as informative attributes. In a decision tree, a node can choose at most one attribute for consideration at a time. In an option tree, the results of all options are “voted” when performing classification. Option trees are often more accurate than decision trees; however, they generally are much larger.

The Option Tree Inducer generates an Option Tree Classifier from a training set in much the same way that the Decision Tree inducer generates a Decision Tree. The induced option tree is displayed using the Tree Visualizer. This visualization helps you understand the classifier, and provides insight into which attributes are important in determining the value of the label.

Regression Tree Inducer and Regressor

The Regression Tree Regressor predicts continuous attributes, in the same the way that the Decision Tree and Option Tree Classifiers predict discrete attributes. While a classifier predicts an event, such as whether a customer will churn (leave you) or not, a regressor predicts specific numerical values, such as the profit margin for a business for the next financial quarter.

The Regression Tree Inducer builds a Regression Tree Regressor model from your data. As with Decision and Option Trees, this model can be viewed and analyzed using the Tree Visualizer, so you can understand the basis from which its predictions are made.

Cluster Visualizer

The Cluster Visualizer displays statistics about the clusters or groups that are generated by the clustering mining tool. It places these statistics side-by-side with those for the entire data set, so that you can see which features make each cluster unique.

The Cluster Visualizer places the attributes in the display in the order of importance for understanding the clustering. When you select one particular cluster, Cluster Visualizer produces an ordering which is the most useful for discriminating between that cluster and the remainder of the data set.

Decision Table Visualizer

The Decision Table Visualizer allows you to view the distribution of data from a discrete column at multiple levels of a hierarchy. For example, you can examine the profitability of a business along dimensions of product class, geography, sales promotions and sales-representative compensation plan. The Decision Table Visualizer distributes the data two attributes at a time, allowing you to drill-down to further pairs of attributes at each level.

The Decision Table Visualizer explores the results of the Decision Table Inducer, so that the discrete column you examine is the label that the inducer classifies. When this is done, the Decision Table Inducer arranges the attributes to determine which pair to display first, and how to drill down from that top level to subsequent levels.

Evidence Visualizer

The Evidence Visualizer visually represents the model generated by the Evidence Classifier. It initially shows cake charts that represent how the various attributes contribute to the decision, and allow “what-if” analysis.

Map Visualizer

The Map Visualizer lets you visualize data relationships that exist across geographically meaningful areas. For example, you can visualize different areas of a country, showing the relative impact of a marketing program. The Map Visualizer's drill-down capabilities let you focus on designated regions and perform a more detailed analysis in smaller geographical elements. One application might be analyzing how one or more products are being sold across different geographies. A powerful animation feature, coupled with a capability to connect different views of the same or related data, permits fast comparisons and difference analyses. This tool lets you visually examine patterns in your data that are difficult to detect when that data is shown in a tabular, two-dimensional form.

Record Viewer

The Record Viewer lets you view the data in the current table in a row/column spreadsheet-like tool.

Rules Visualizer

The Rules Visualizer visually represents the model of the Association Rules Generator mining tool. It provides detailed data analysis that lets you examine relationships across data elements in new ways. In doing so, you might discover relationships that significantly differ from what you might have expected; this, in turn, can lead to important discoveries about your data or the processes behind that data. This tool's visualization capabilities let you discover additional patterns of co-occurrence between these data elements. For example, you can use the analysis of products sold during the last sales promotion to guide your advertising campaign for the next sales period. The Rules Visualizer's high performance would let you analyze the results from today's sales data in time to alter the advertising campaign for the future.

Scatter Visualizer

The Scatter Visualizer lets you examine the behavior of data across eight different dimensions. The data is shown in a grid representing up to three dimensions. Extra dimensions can map to the size, color, and label of each displayed entity. Two further independent dimensions can be assigned as dynamic dimensions. A slider can be used to select specific values along those dimensions, or a path can be traced through those dimensions, for animation. During the path traversal, the display changes automatically to reflect the change in the independent variables.

Splat Visualizer

The Splat Visualizer produces 3D plots of very large data sets. Instead of showing individual data points, it renders the density of data using varying opacity. It has many of the same features as the Scatter Visualizer.

Statistics Visualizer

The Statistics Visualizer computes and displays summary information for the current dataset (maximum, minimum, median, standard deviation, distinct values, and quartiles).

Tree Visualizer

The Tree Visualizer helps you analyze data that has hierarchical relationships. It provides an interactive “fly-through” capability for examining relationships among data at different hierarchical levels. For example, the Tree Visualizer can be used to examine a company's product line, graphically displaying each product's contribution to the company's total revenue. Each branch of the hierarchy displays information at increasing levels of detail, breaking revenues down by product lines and, eventually, individual products. Another example of using the Tree Visualizer is to show company sales revenue, displaying a company-wide total as well as sub-totals at regional and other levels. The fly-through capability in the Tree Visualizer lets you rapidly reposition your view of the data. The Tree Visualizer's filtering and searching capabilities let you focus on specific data elements and queries.

The Tree Visualizer is also used to view the resulting models of the Decision Tree and Option Tree Classifiers, and the Regression Tree Regressor; with each decision being represented by a separate node in the tree. Each node also contains bars showing how the data is modeled based on the decisions up to that point (for example, 73% of people who own a home and have two children are good credit risks, while 27% are not).

Basic Tool Execution Scenario

Each of the MineSet tools is started, configured, and run in a consistent manner. The sequence of actions you follow at your MineSet client and at the MineSet server is shown schematically in Figure 1-1. A description of the steps inherent in this figure follows.

Figure 1-1. Tool Execution Sequence

Figure 1-1 Tool Execution Sequence

The following steps describe a “typical” interaction with a MineSet tool, and the sequence of the tool's actions. Depending on your requirements, some steps might be skipped (for instance, if the data and configuration files have been generated in a previous work session).

  1. Start the Tool Manager, which is the graphical interface for generating and specifying the configuration file, data file, and tools to be used. The Tool Manager runs on your MineSet client.

  2. The Tool Manager opens a network connection to the DataMover, which runs on the MineSet server, which in some cases may be the same as your client workstation, and in others is a separate machine.

  3. Use the Tool Manager to specify

    • the database and table, or a binary or ASCII flat file containing the data on either the client or the server

    • which mining or visualization tools are to be applied

    • how that data is to be displayed, through tool options

    • a session file to save the history of your work

    Information retrieved via the DataMover is used to guide this interaction. As a result, the Tool Manager generates a configuration file. This file contains the user-defined parameters that determine the execution of the following steps.

  4. The Tool Manager transmits a copy of the configuration file from step 3 to the DataMover. The DataMover processes the file by

    • accessing the database or flat file

    • performing the specified data transformations

    • running the mining tools when requested

    • generating the visualization files when requested

    These visualization files consist of your data in a specific format readable by the MineSet tool. Then a copy of these visualization files is transferred to the MineSet client.

  5. The Tool Manager invokes the appropriate MineSet visualization tool.

  6. The tool accesses the visualization files and displays the data.

  7. If you generated a model, that model can be applied to additional data (see Figure 10-5).


Note: The MineSet client and server can run on different machines, using a network to communicate. Because network bandwidth is often scarce, you should be cautious about transferring large files between client and server regularly. If you are doing mining operations on a large database or file, you can achieve greater efficiency by storing that file on the server, where the DataMover runs, rather than on the client.