Screen Risk Factors by Custom Criteria
This example shows how to use the Screen Risk Factors task to automatically exclude risk factors from a table based on their predictive power.
This example also shows how to set up the screening criteria.
Feature selection is an important step in the development of a statistical model. Input data can have hundreds or thousands of variables, and discarding some variables often improves model interpretability, training times, and other important attributes.
This example loads the Screen Risk Factors data set, which contains a table of customer information such as age, income, and employment status. This example uses pre-defined metrics to assess risk factors individually and analyzes the predictive power of each variable relative to a given (binary) response variable. This example then shows you how to select variables automatically or semi-automatically using the Screen Risk Factors task. This example also shows you how to customize the screening criteria used to assess the risk factors.
Load Data and Pre-defined Screening Criteria
Load the example data from ScreenRiskFactorsData.mat.
Construct pre-defined screening criteria in your workspace. Use
ExampleScreeningCriteria to generate myCriteria object. This function returns a
ScreeningCriteria object defined by an
import mrm.data.selection.* myCriteria = ExampleScreeningCriteria();
These criteria have been set up as follows:
For each variable, the Information Value and Chi-squared p-value are calculated.
These values are compared against certain thresholds that assign the metric a Pass, Fail or Undecided classification. In this case, the thresholds for the metrics are hard-coded but you can obtain the thresholds from the appropriate data in the development environment.
The overall classification works on the 'worst-of' basis. If the status for either Information Value or the Chi-squared p-value is a Fail, the overall status will be Fail, and so on.
TestSuiteFactory sets the
StatusInterpreter of the metrics handler to
overallScreeningStatus. This is where the auto-generated exclusions and comments are set. For the exclusions, the function must assign to each
MetricsHandler state an
mrm.data.selection.ScreeningStatus object (or an
NullStatus) to ensures that the Screen Risk Factors task automatically marks the variable for exclusion.
In addition, the percentage of missing entries is displayed. This value does not affect the overall rating.
Launch Screen Risk Factors
Open a new live script and launch the Screen Risk Factors task. This can be done in two ways:
1) Start typing 'Screen' and select the task from the drop-down menu
2) Search for Screen Risk Factors in the Live Task gallery
The task opens in a reduced view until the required inputs are selected:
Input table must be a table or a timetable; the drop-down shows all such objects in the workspace. For this example, select
Response variable drop-down shows all the binary variables in the input table. For this example, select
Criteria should be the
ScreeningCriteriaobject you wish to apply - in this case
Analyze and Remove Risk Factors
The task now expands.
The task calculates the screening metrics for each risk factor in the input table. The summary of the results is shown in the 'Analyze data variables' section. The table contains one row for each variable in the input table.
'Status' shows the overall classification of the variable based on the screening metrics.
'Exclude' shows whether the variable is to be removed from the data set.
'Comment' contains the reasons for excluding the variable, or for leaving the variable included.
The live task auto populates the 'Exclude' and 'Comment' columns based on the criteria. In this example, the 'Fails' are automatically excluded and 'Passes' are automatically included with automatically generated comments. The 'Undecided' risk factors are left blank for the user to analyze. You can overwrite these auto-completed values and sort the table according to any of these columns.
The area underneath the table is specific to the risk-factor variable and displays the screening metrics, as well as a double histogram that demonstrates how well (or not) the variable discriminates between the two possible responses. To switch the view to another variable, click the variable name in the table.
Document with Modelscape Reporting
The live task dynamically produces two outputs:
filteredTable: This is a subtable of the input table without the excluded risk factors. Use this subtable in the next step of the model development process.
exclusionTable: This table includes all the data of the input table together with the exclusion flags and comments in the Live Task. To view this information, tick the 'Preview summary tables' box in the 'Display results' section. This information is stored in
You can insert the above tables into model documentation using the Modelscape Reporting feature. To achieve this, create document holes with titles, say
, in the Word document.
To create document holes in a Word document, view the Developer tab, and click the 'Rich Text Content Control' symbol Aa in the Controls area. Then click 'Properties', and fill in the Title fields.
import mrm.data.filter.* [ExclusionSummary, ProgressSummary] = summarizeExclusionTable(exclusionTable)
After you have created holes, pick up the new variables from the MATLAB workspace and insert them into the model document using
For examples of creating document holes and for more details on the use of
fillReportFromWorkspace, see Model Documentation in Modelscape.
Set Up Custom Criteria
You can customize the criteria used to screen variables in the Screen Risk Factors Live Task. The criteria must be in an
mrm.data.selection.ScreeningCriteria object. For the class definition, run:
This class is a holder for a handle to a function f.
f(inputData, 'PredictorVar', varName, 'ResponseVar', respVar)
f call must be well-defined and produce an
.MetricsHandler object for any table or timetable
inputData, any predictor variable
varName, and for a given binary response variable
TestSuiteFactory has this signature for the function call.
To see examples of these functions in the Modelscape package, run
edit mrm.data.selection.ExampleScreeningCriteria; edit mrm.data.selection.TestSuiteFactory; edit mrm.data.selection.overallScreeningStatus;