The Decision Tree, Option Tree, and Regression Tree visualizers look and feel similar, even though the algorithms behind them vary considerably. This chapter covers the similarities and the differences in the use of these three tree-based models. To learn more about the algorithms used by the inducers, and possible options, see the MineSet Enterprise Edition Reference Guide .
This chapter contains the following sections:
Sample configuration and data files have been provided to demonstrate the tree visualizers' features and capabilities. These files are in the \examples directory under MineSet, in the location where MineSet is installed.
This section explains the use of the three types of tree, and some of the reasons for choosing one over the other. Each tree tool consists of an inducer, which builds the model, and a visualizer, which displays the resulting tree. The inducer is either a classifier (Decision Tree and Option Tree) or a regressor (Regression Tree).
The Decision Tree is a predictive model. It makes predictions by using the dependent, or known, attribute values to help determine the value of the label, or unknown attribute. The task of predicting the value of a nominal value (usually character strings such as “yes” and “no”), or an attribute that can only take on a small number of values, is referred to as classification. A decision tree classifies data by predicting the label for each record. The underlying structure used for classification is a decision tree, such as the one shown in Figure 8-1. Once the Decision Tree Inducer has built a classifier the data, the Tree Visualizer displays its structure.
The first element of the tree is the root node, representing all of the data. From there, the tree spits into two or more branches, each representing data with different values for a specific attribute (column). For example, Figure 8-1 is a decision tree visualization of the cars dataset, where the records are classified by cubic inches. The first split is on 169.5 inches or less on the right branch, and those over 169.5 cubic inches on the left. The tree can split on the same attribute at more than one node. The node can also split into more than two branches. For instance, for the mushroom dataset, with the records being classified into edible and non-edible, the first split is into nine branches, corresponding to the nine odor types.
The object is to reach nodes at the ends of the branches (leaf nodes) where the records all, or nearly all, have the same class (label). In Figure 8-1, the classes are the different ranges of cubic inches values listed at the bottom of the Decision Tree window.
Like Decision Tree classifiers, Option Tree classifiers also assign each record to a class. The underlying structure used for classification is a decision tree, as described in the previous section. Once the Option Tree Inducer has built the classifier, the Tree Visualizer displays its structure. An option tree actually consists of several decision trees. Instead of picking an attribute to split on for the root node, it picks several, and makes a decision tree for each.
Figure 8-2 shows an option tree for the cars dataset, where the goal is to predict the origin of a car (U.S., Japan, or Europe). The Option Tree extends the Decision Tree Classifier by allowing Option Nodes. An Option Node gives several options that can be chosen at a decision node in the tree. For example, in Figure 8-2, the root is an option node with five options: cubic inches, number of cylinders, weight (in pounds), miles per gallon, and brand name.
Option nodes serve two purposes:
They enhance comprehensibility of the factors affecting the determination of class label by showing several choices that can be made. Instead of using a single attribute at a node, an option node provides you with several options. When examining the tree, you can choose to follow an option that you believe is easier to understand or is better for predictions based upon your previous experience, or one you select based on the error estimate.
In the cars dataset shown in Figure 8-2, you can follow the cylinders subtree because it has few values or the weightlbs subtree because its estimated error is lower (1.53). Error estimates are only estimates; generally, if the error difference between two options is less than twice their mean standard deviation, then statistically the errors are not different.
They reduce the risk of creating an inaccurate classifier by averaging the votes of each option subtree. Every option leads to a subtree that can be thought of as an “expert.” The option node averages these experts' votes. Such averaging can lead to a better classifier with a lower error rate.
In the cars dataset, shown in Figure 8-2, the root node has an estimated error rate of 0.76%, which is lower than any of its children. Sometimes, the obvious choice for an option is not necessarily the best one to use. For instance, while brand might seem like an obvious attribute for this task, the training set might not contain all brands (and in fact, it does not contain all of them). For an unseen brand, Decision Tree guesses the majority class (U.S.) and makes two errors. However, when there are other options, they are averaged, and the error is reduced.
Option Trees, however, have two disadvantages:
The time necessary to build an option tree under the default setting is about 10 to 15 times longer than that needed to build a decision tree.
The Tree Visualizer file that is created is very large, containing 10 to 15 times as many nodes as a regular decision tree.
Run the Option Tree inducer on your dataset to determine whether the advantages in comprehensibility and error rates justify the longer induction time. You might gain additional insight as to which attributes to remove or use when building a decision tree.
Regression is the task of predicting a continuous label value, given a set of descriptive attributes. Regression and classification are similar, except that in classification the predicted label can take on only a small number of discrete values, for example, an iris may be classified by type as iris-setosa, iris-versicolor, or iris-virginica. In regression, the predicted label can be any value in a continuous range, for example, an individual's annual income may be any amount greater than zero.
When a regressor is generated, the Option Tree Visualizer displays the tree. This visualization can help you understand the regressor and how it makes predictions. In addition, it can provide valuable insight into the data itself. Once generated, a regressor can be used to predict the label value for unlabeled records.
The underlying structure used for regression is a regression tree, such as the one shown in Figure 8-3.
The easiest way to start the tree tools is from the Tool Manager:
From the Tool Manager File menu, connect and log in to a server; from the same menu choose Open New Data File and select or type in the desired filename.
In the Data Destinations pane, click Mining Tools (Figure 8-4) and click either Classify or Regress in the lower row of tabs:
From the popup Mode menu, choose a mode. See the “Error Estimation” entry in the MineSet Enterprise Edition Reference Guide for more information about the four available modes.
From the popup Inducer menu, choose the inducer.
From the Discrete Label (Continuous Label for Regression Tree) popup menu, choose the attribute (column) you wish to use for the label. For instance, for the mushroom dataset, a logical choice is edibility. For the churn dataset, a logical choice is churned.
Click Go to start the inducer. The Status pane at the bottom of the Tool Manger window shows the progress and resulting statistics. Once the MineSet finishes it's calculations, the visualizer window appears.
If you have the Viz Tool icon on your desktop, you can drag an existing MineSet visualization file icon and drop it on the Viz Tool icon. The visualization will then display in the Viz window.
You can also drag a visualization file into an open Viz Tool window. If you have your preferences set to Single Document mode, you must drop the file icon onto the title bar of the Viz window. If you have your preferences set to Multiple Document mode, you can drop the file icon anywhere within the window, and it will display.
Decision Tree and Option Tree visualizations are similar. They both consist of two types of nodes connected by edges (the lines): decision nodes and leaf nodes.
Labels on the decision nodes specify the attribute that is tested at that node. The values (or ranges of values) against which the attributes are tested are shown at the edges. Each possible value for the attribute matches exactly one edge. For example, the root of the decision tree in Figure 8-1 tests the attribute weightlbs; the two edges emanating from the node specify the ranges of values for that attribute (less than or equal to 3018 and greater than 3018) so that every possible value matches either the right branch or the left branch.
Leaf nodes in a decision tree specify a class. If you follow the right branch in Figure 8-1 from the root to a leaf labeled (... 16.5], you see that Decision Tree classifies records with weightlbs greater than 3018 pounds, horsepower greater than 141, and cubicinches greater than 311.5, as getting 16.5 miles per gallon or less.
The color of the base indicates the error estimate of the subtree: indigo shows high error, grey indicates medium, white indicates low error. If no test set records reached a node (which means there is no error estimate), the color of the base is black.
The vertical bars atop each node show the distribution of the classes at the node. The base of each node has a height and a color. The height corresponds to the weight of the training set records that have reached this node (this is the number of records if weight was not set). In general, the higher the weight, the more reliable the class distribution at every node (see “Weighting Records” in Chapter 3 for more details about weighting records).
Placing your mouse arrow over a node displays the following information:
Subtree weight—The weight of the training set records in the subtree below the node pointed to. This value is mapped to the height of the base.
Test set error/loss—An estimate of the subtree error (or loss if a loss matrix was given). The number after the +/- is the standard deviation of the estimate. The higher the standard deviation, the less accurate the error estimate. The error/loss estimate and the standard deviation are less reliable for leaves with few records or when the test set error is close to 0% or 100%.
Test set weight—The weight of records from the test set that reached the node (number of records if weight was not set).
Purity—A number from 0 to 100 indicating the skewness (see the Skewness entry in the glossary) of the label value distribution at the node. If a node has records from a single class, the purity is 100. If the label values have the same weight, the purity is 0. The purity is computed after backfitting (see the Backfitting entry in the glossary).
Only Classify and Error mode yields the test set error/loss and weight. You can use the Test Classifier option to generate a visualization based on an existing classifier and a test set.
To classify a record with an unknown label, start at the root of the tree, and follow the branches dictated by the attribute values for that record. By following the appropriate edges based on the record's attribute values, you reach a leaf node. The label, or class, associated with the leaf node is the predicted classification of the record.
Some decisions are made quickly and take a shorter path. Other decisions can take a longer path. In general, every leaf corresponds to a rule that is the conjunction of all tests at the decision nodes and all the values (or ranges of values) on the edges leading to it from the root.
The decision tree produced for the iris dataset (Figure 8-5) is very simple, and works well for illustration purposes. This file can be opened from the Tool Manager Visual Tools menu; select the 3D Visualizer, then \examples\ iris-dt.treeviz. At the root of this tree, the error rate is 6%, with a standard deviation of 3.39%. The standard deviation is high because the file is small, and the test set has only 50 records. The purity is 0.0, indicating that the distribution is uniform.
The left child of the root has 0 test set error and a purity of 100 because all records with petal_length less than or equal to 2.6 inches are of the iris-setosa class; thus, the prediction of iris-setosa is likely to be very accurate for all records with petal_length less than or equal to 2.6 inches. The right child of the root has an estimated error of 8.57%. In this child, which matches records whose petal lengths are greater than 2.6 inches, there are no records belonging to the iris-setosa class; thus, the class is more likely to be iris-versicolor or iris-virginica. Because only two possibilities exist at this node, there is a higher purity than at the root (36.91).
The Decision Tree leaf nodes segment the data into clusters sharing the same classification rule (path that leads to each leaf). By looking at the leaf nodes, it is possible to see clusters that share the same set of properties.
The Decision Tree Visualizer uses the Tree Visualizer for its display. The Tree Visualizer features several useful devices, such as a search panel, a filter panel, and an overview window, accessible from the View menu of the Tool Manager. These are described in Chapter 5, “Examining Data with the Tree Visualizer,” and the “Tree Visualizer” entry in the MineSet Enterprise Edition Reference Guide .
Decision Tree also has various options available to fine-tune the induction algorithms, pruning, and evaluation metrics. These are covered in the “Decision Tree” entry in the MineSet Enterprise Edition Reference Guide .
The Decision and Option Tree visualizations are fundamentally similar. However, there are a few differences between the two types of tree:
The leftmost option in an option tree would have been the only option chosen by the Decision Tree inducer. As you go right, the options are ranked in decreasing order by the fitness scoring. The fitness scores do not necessarily match the test-set error shown. This is expected, as the inducer is using a non-perfect scoring function. The test-set estimate also has natural variability: the larger the test-set, the more accurate the estimate.
The option node can have a different error rate from that of every one of its children. Because the option node averages the children's predictions, its error rate can be different. In some cases, its error is lower than that of every child, showing that averaging helps.
The distribution of instances (shown in bars) at every child of an option node is exactly the same as that of the option node itself. This is because there was no decision made by the option node: options are being presented as children.
One feature of the Option Tree visualization that is very useful for navigating through the tree is clicking the right mouse button on an option node. This presents the list of children, which are the options.
A simple Option Tree worth examining can be found in MineSet 3.1\examples\cars-odt.treeviz. The Option Tree Visualizer uses the Tree Visualizer for its display. The Tree Visualizer features several useful devices, such as a search panel, a filter panel, and an overview window. These are described in Chapter 5, “Examining Data with the Tree Visualizer.”
Option Tree also has various options available to fine-tune the induction algorithms, pruning, and evaluation metrics. These are covered in the “Option Tree” entry in the MineSet Enterprise Edition Reference Guide .
Regression Tree and Decision Tree visualizations are similar. They both consist of two types of nodes connected by edges: decision nodes and leaf nodes.
Decision nodes specify the attribute that is tested at the node. Values (or ranges of values) against which the attributes are tested are shown at the edges. Each possible value for the attribute matches exactly one edge. For example, the root of the regression tree in Figure 8-3 tests the attribute age; the two edges emanating from the node partition values for that attribute (less than 27.5 and greater than or equal to 27.5) so that every possible value matches either the right branch or the left branch. If the value is unknown and there is no edge labeled with a question mark, the mean or median label value at the current node is predicted.
Leaf nodes in a regression tree predict a value. A useful example can be found in \examples\adult-rt.treeviz, discussed here. If you follow the leftmost branch in Figure 8-3 from the root to the leaf labeled 4002.300, you find that the Regression Tree predicts that people under 19.5 years old who work less than 35.5 hours per week average a gross income of $4002.30.
Each bar on a node in a Regression Tree corresponds to a subrange of continuous label values. The range of continuous label values covered by each node can be different. The bars at each node form a histogram indicating how the weight (number) of records is distributed over this range. The number of bars is determined by the weight of records at the root. The leftmost bar always corresponds to the lowest value. The size and midpoint, therefore, may be different at every node. The color of a bar indicates the midpoint of the subrange that bar covers. The maximum range is indicated by blue on the left and red on the right. A node that covers only records with a limited range of label values has a histogram that does not range from blue to red.
The base of each node has a height. The height corresponds to the weight of the training set records that have reached this node (this is the number of records if weight was not set). In general, the higher the weight, the more reliable the distribution at a node.
Placing your mouse arrow over a node displays the following information (see the glossary for definitions of the terms):
Subtree weight—The weight of the training set records in the subtree below the node pointed to. This value is mapped to the height of the base.
Mean: the mean of the continuous label.
Standard deviation: the standard deviation of the continuous label. The higher the standard deviation, the less reliable the model.
Median: the median of continuous label.
Absolute deviation: absolute deviation of the continuous label. The higher the absolute deviation, the less reliable the model.
To predict a value for a record, start at the root of the tree, and follow the branches dictated by the attribute values for that record. By following the appropriate edges based on the record's attribute values, you reach a leaf node. The prediction associated with the leaf node is the predicted value of the record.
When evaluating a classifier, the natural metric is error (the number of examples for which the classifier predicts the wrong label). When a loss matrix is supplied, different types of misclassification errors may have different associated costs. In this situation loss is a natural measure. For more information, see “Defining a Loss Matrix” in Chapter 11.
For regression, where the task is to predict a real value, there is no single natural evaluation metric. The two measures that are frequently used are standard deviation and absolute deviation. The standard deviation is the square root of the mean squared error. The absolute deviation is the mean of the absolute value of the difference between the predicted label value and the actual label value.
The Regression Tree Visualizer utilizes the Tree Visualizer for its display. The Tree Visualizer features several useful devices, such as a search panel, a filter panel, and an overview window. These are described in Chapter 5, “Examining Data with the Tree Visualizer,”and the “Tree Visualizer” entry in the MineSet Enterprise Edition Reference Guide .
The Regression Tree also has various options available to fine-tune the induction algorithms, pruning, and evaluation metrics. These are covered in the “Regression Trees” entry in the MineSet Enterprise Edition Reference Guide .
Up to this point, this chapter has discussed classifying data with known outcomes (for instance, churned or not churned and edible or poisonous), but Decision, Option, and Regression Trees can be most useful for predicting unknown or future outcomes. You can build a model based on data where the class is known, and then use that model to classify new data where the class is unknown. See Chapter 7, “Understanding Predictive Modeling,” and Chapter 11, “Refining Predictive Modeling,” for more information about building and applying models.