This chapter introduces specific tasks involved in the knowledge discovery process. The knowledge discovery process is iterative (represented in Figure 2-1) where you go back to earlier stages once you discover new patterns and improve your understanding of the data.
The common steps of this process are:
Identify the source of the data—see “Identifying the Data”.
Prepare the data—see “Preparing the Data”.
Build a model—see “Building a Model”.
Evaluate the model—see “Evaluating a Model”.
Deploy the model—see “Deploying a Model”.
The task of identifying the data begins by deciding what data is needed to solve a problem. For example, predictability about customer behavior is often a necessary goal recast in terms of a problem. In defining the problem, the investigator must identify the data needed to solve that problem and explore other possible sources of data.
Data may be in a difficult location or in an obscure form. Sometimes there are several initial databases that may be incompatible with each other. Further, if data is scanty or incomplete, more data may be needed. The form in which new data is to be collected depends on the form of existing data. MineSet supports native interfaces to several commercial databases (Oracle, Informix, SQL), ODBC interface, as well as reading data from different file formats (tab-separated flat file, MineSet binary file, Excel, SPSS, Mutable, etc.)
Data may need modification before loading into MineSet (a step often called cleaning.) Specifically, the following problems are common:
Data may be in a format incompatible with MineSet representation (for example, binary, encoded, or EBCDIC strings from old mainframe computers).
Data may be misspelled or erroneous, or have incomplete, or erroneous values.
Field descriptions may be unclear or confusing, or may mean different things depending on the source. For example, order date may mean the date that the order was sent, postmarked, received, or keyed in.
Data may be out of date; for example, customers may have moved, changed households, or changed spending patterns.
Even clean data may need to be transformed before it is suitable for mining and visualization.
Transformations can greatly improve model performance. If you were analyzing telephone company data, for instance, you may find that long distance rate (sales divided by total minutes used) is a better predictor of customer behavior than either element given separately. Data transformations are at the heart of developing a sound model. As you progress, you may even go back and transform the data differently. You can transform the data by:
Adding columns, usually by applying a mathematical formula to existing data to create a new field.
Removing columns that are not pertinent, are redundant, or contain obvious, uninteresting predictors.
Filtering visualizations. For example, you may want to see only the strongest rules or the most profitable customer segments.
Binning data—breaking up a continuous range of data into discrete segments (for instance, [1 - 10], [11 - 20], and so on).
Aggregating data—grouping records together, and finding the sum, maximum, minimum, or average values.
Sampling the data to get a random subset of the data (by percentage or count).
Applying a classifier that you have previously created, to label new records with a class label, or to estimate the probability of a given label value.
In MineSet, most of these transformations take place using the Data Transformation pane in Tool Manager.
At the core of the knowledge discovery process is model building, automatically done by analytical data mining algorithms. This is clarified in Chapter 3.
Evaluating the accuracy of a model refines your understanding of that model and its usefulness. This allows you to improve the model by filtering the data, eliminating columns, creating new columns, and so on.
MineSet implements four model assessment methods: error estimation, confusion matrix, lift curve, and ROI (return-on-investment) curve.
A model can be deployed by applying it to new data. New data can give rise to further questions, which may require further refinements.
In the telecommunications example in Chapter 3, a model can be created to determine which customers are likely to leave their phone carrier. Customer records can then be evaluated through the model to identify the specific customers most likely to leave. These customers can be given incentives to stay.
The next two chapters step you through the knowledge discovery process on the churn dataset—a prepared dataset of telecommunication customers. As you work through the examples, think of the process presented here and how your operations progress forward and loop back as shown in Figure 2-1.