isanomaly
Find anomalies in data using isolation forest
Syntax
Description
finds anomalies in the table tf
= isanomaly(forest
,Tbl
)Tbl
using the IsolationForest
object forest
and returns the logical array tf
,
whose elements are true
when an anomaly is detected in the corresponding
row of Tbl
. You must use this syntax if you create
forest
by passing a table to the iforest
function.
specifies options using one or more name-value arguments in addition to any of the input
argument combinations in the previous syntaxes. For example, set
tf
= isanomaly(___,Name=Value
)
to identify observations with
scores above 0.5 as anomalies.ScoreThreshold
=0.5
Examples
Detect Novelties
Create an IsolationForest
object for uncontaminated training observations by using the iforest
function. Then detect novelties (anomalies in new data) by passing the object and the new data to the object function isanomaly
.
Load the 1994 census data stored in census1994.mat
. The data set consists of demographic data from the US Census Bureau to predict whether an individual makes over $50,000 per year.
load census1994
census1994
contains the training data set adultdata
and the test data set adulttest
.
Train an isolation forest model for adultdata
. Assume that adultdata
does not contain outliers.
rng("default") % For reproducibility [Mdl,tf,s] = iforest(adultdata);
Mdl
is an IsolationForest
object. iforest
also returns the anomaly indicators tf
and anomaly scores s
for the training data adultdata
. If you do not specify the ContaminationFraction
name-value argument as a value greater than 0, then iforest
treats all training observations as normal observations, meaning all the values in tf
are logical 0 (false
). The function sets the score threshold to the maximum score value. Display the threshold value.
Mdl.ScoreThreshold
ans = 0.8600
Find anomalies in adulttest
by using the trained isolation forest model.
[tf_test,s_test] = isanomaly(Mdl,adulttest);
The isanomaly
function returns the anomaly indicators tf_test
and scores s_test
for adulttest
. By default, isanomaly
identifies observations with scores above the threshold (Mdl.ScoreThreshold
) as anomalies.
Create histograms for the anomaly scores s
and s_test
. Create a vertical line at the threshold of the anomaly scores.
histogram(s,Normalization="probability") hold on histogram(s_test,Normalization="probability") xline(Mdl.ScoreThreshold,"r-",join(["Threshold" Mdl.ScoreThreshold])) legend("Training Data","Test Data",Location="northwest") hold off
Display the observation index of the anomalies in the test data.
find(tf_test)
ans = 15655
The anomaly score distribution of the test data is similar to that of the training data, so isanomaly
detects a small number of anomalies in the test data with the default threshold value. You can specify a different threshold value by using the ScoreThreshold
name-value argument. For an example, see Specify Anomaly Score Threshold.
Specify Anomaly Score Threshold
Specify the threshold value for anomaly scores by using the ScoreThreshold
name-value argument of isanomaly
.
Load the 1994 census data stored in census1994.mat
. The data set consists of demographic data from the US Census Bureau to predict whether an individual makes over $50,000 per year.
load census1994
census1994
contains the training data set adultdata
and the test data set adulttest
.
Train an isolation forest model for adultdata
.
rng("default") % For reproducibility [Mdl,tf,scores] = iforest(adultdata);
Plot a histogram of the score values. Create a vertical line at the default score threshold.
histogram(scores,Normalization="probability"); xline(Mdl.ScoreThreshold,"r-",join(["Threshold" Mdl.ScoreThreshold]))
Find the anomalies in the test data using the trained isolation forest model. Use a different threshold from the default threshold value obtained when training the isolation forest model.
First, determine the score threshold by using the isoutlier
function.
[~,~,U] = isoutlier(scores)
U = 0.7449
Specify the value of the ScoreThreshold
name-value argument as U
.
[tf_test,scores_test] = isanomaly(Mdl,adulttest,ScoreThreshold=U); histogram(scores_test,Normalization="probability") xline(U,"r-",join(["Threshold" U]))
Input Arguments
forest
— Trained isolation forest model
IsolationForest
object
Trained isolation forest model, specified as an IsolationForest
object.
Tbl
— Predictor data
table
Predictor data, specified as a table. Each row of Tbl
corresponds to one observation, and each column corresponds to one predictor variable.
Multicolumn variables and cell arrays other than cell arrays of character vectors are
not allowed.
If you train forest
using a table, then you must provide
predictor data by using Tbl
, not X
. All
predictor variables in Tbl
must have the same variable names and
data types as those in the training data. However, the column order in
Tbl
does not need to correspond to the column order of the
training data.
Data Types: table
X
— Predictor data
numeric matrix
Predictor data, specified as a numeric matrix. Each row of X
corresponds to one observation, and each column corresponds to one predictor
variable.
If you train forest
using a matrix, then you must provide
predictor data by using X
, not Tbl
. The
variables that make up the columns of X
must have the same order as
the training data.
Data Types: single
| double
Name-Value Arguments
Specify optional pairs of arguments as
Name1=Value1,...,NameN=ValueN
, where Name
is
the argument name and Value
is the corresponding value.
Name-value arguments must appear after other arguments, but the order of the
pairs does not matter.
Example: ScoreThreshold=0.75,UseParallel=true
sets the threshold for
the anomaly score to 0.75 and instructs the function to run computations in
parallel.
ScoreThreshold
— Threshold for anomaly score
forest.ScoreThreshold
(default) | numeric scalar in the range [0,1]
Threshold for the anomaly score, specified as a numeric scalar in the range
[0,1]
. isanomaly
identifies observations
with scores above the threshold as anomalies.
The default value is the ScoreThreshold
property value of forest
.
Example: ScoreThreshold=0.5
Data Types: single
| double
UseParallel
— Flag to run in parallel
false
(default) | true
Flag to run in parallel, specified as true
or
false
. If you specify UseParallel=true
, the
isanomaly
function executes for-loop iterations in parallel
by using parfor
. This option requires
Parallel Computing Toolbox™.
Example: UseParallel=true
Data Types: logical
Output Arguments
tf
— Anomaly indicators
logical column vector
Anomaly indicators, returned as a logical column vector. An element of
tf
is true
when the observation in the
corresponding row of Tbl
or X
is an anomaly,
and false
otherwise. tf
has the same length as
Tbl
or X
.
isanomaly
identifies observations with
scores
above the threshold (the
ScoreThreshold
value) as anomalies.
scores
— Anomaly scores
numeric column vector in the range [0,1]
Anomaly scores, returned as a numeric column vector whose values are in the
range [0,1]
. scores
has the same length as
Tbl
or X
, and each element of
scores
contains an anomaly score for the observation in the
corresponding row of Tbl
or X
. A score value
close to 0 indicates a normal observation, and a value close to 1 indicates an
anomaly.
More About
Isolation Forest
The isolation forest algorithm [1] detects anomalies by isolating anomalies from normal points using an ensemble of isolation trees.
The iforest
function builds an isolation forest (ensemble of isolation
trees) for training observations and detects outliers (anomalies in the training data). Each
isolation tree is trained for a subset of training observations, sampled without
replacements. iforest
grows an isolation tree by choosing a split
variable and split position at random until every observation in a subset lands in a
separate leaf node. Anomalies are few and different; therefore, an anomaly lands in a
separate leaf node closer to the root node and has a shorter path length (the distance from
the root node to the leaf node) than normal points. The function identifies outliers using
anomaly scores defined based on the
average path lengths over all isolation trees.
The isanomaly
function uses a trained isolation forest to detect
anomalies in data. For novelty detection (detecting anomalies in new data with
uncontaminated training data), you can train an isolation forest with uncontaminated
training data (data with no outliers) and use it to detect anomalies in new data. For each
observation of the new data, the function finds the average path length to reach a leaf node
from the root node in the trained isolation forest, and returns an anomaly indicator and
score.
For more details, see Anomaly Detection with Isolation Forest.
Anomaly Scores
The isolation forest algorithm computes the anomaly score s(x) of an observation x by normalizing the path length h(x):
where E[h(x)] is the average path length over all isolation trees in the isolation forest, and c(n) is the average path length of unsuccessful searches in a binary search tree of n observations.
The score approaches 1 as E[h(x)] approaches 0. Therefore, a score value close to 1 indicates an anomaly.
The score approaches 0 as E[h(x)] approaches n – 1. Also, the score approaches 0.5 when E[h(x)] approaches c(n). Therefore, a score value smaller than 0.5 and close to 0 indicates a normal point.
Algorithms
isanomaly
considers
NaN
, ''
(empty character vector),
""
(empty string), <missing>
, and
<undefined>
values in Tbl
and
NaN
values in X
to be missing values.
isanomaly
does not use observations with all missing values. The function assigns the anomaly score of 1 and anomaly indicator offalse
(logical 0) to the observations.isanomaly
uses observations with some missing values to find splits on variables for which these observations have valid values.
References
[1] Liu, F. T., K. M. Ting, and Z. Zhou. "Isolation Forest," 2008 Eighth IEEE International Conference on Data Mining. Pisa, Italy, 2008, pp. 413-422.
Extended Capabilities
C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.
Usage notes and limitations:
Use
saveLearnerForCoder
,loadLearnerForCoder
, andcodegen
(MATLAB Coder) to generate code for theisanomaly
function. Save a trained model by usingsaveLearnerForCoder
. Define an entry-point function that loads the saved model by usingloadLearnerForCoder
and calls theisanomaly
function. Then usecodegen
to generate code for the entry-point function. For an example, see Code Generation for Anomaly Detection.To generate single-precision C/C++ code for
isanomaly
, specify the name-value argument"DataType","single"
when you call theloadLearnerForCoder
function.Strict single-precision calculations are not supported. In the generated code, single-precision inputs produce single-precision outputs. However, variables inside the function might be double-precision.
This table contains notes about the arguments of
isanomaly
. Arguments not included in this table are fully supported.Argument Notes and Limitations Tbl
The entry-point function must do the following:
Accept data as arrays.
Create a table from the data input arguments and specify the variable names in the table.
Pass the table to
isanomaly
.
For an example of this table workflow, see Generate Code to Classify Data in Table. For more information on using tables in code generation, see Code Generation for Tables (MATLAB Coder) and Table Limitations for Code Generation (MATLAB Coder).
The number of rows, or observations, in
Tbl
can be a variable size, but the number of columns inTbl
must be fixed.
X
The number of rows, or observations, in
X
can be a variable size, but the number of columns inX
must be fixed.ScoreThreshold
Names in name-value arguments must be compile-time constants. UseParallel
This name-value argument is not supported, but the function supports parallel computation through OpenMP.
The generated code of
isanomaly
usesparfor
(MATLAB Coder) to create loops that run in parallel on supported shared-memory multicore platforms in the generated code. If your compiler does not support the Open Multiprocessing (OpenMP) application interface or you disable OpenMP library, MATLAB® Coder™ treats theparfor
-loops asfor
-loops. To find supported compilers, see Supported Compilers. To disable OpenMP library, set theEnableOpenMP
property of the configuration object tofalse
. For details, seecoder.CodeConfig
(MATLAB Coder).
For more information, see Introduction to Code Generation.
Automatic Parallel Support
Accelerate code by automatically running computation in parallel using Parallel Computing Toolbox™.
To run in parallel, set the UseParallel
name-value argument to
true
in the call to this function.
For more general information about parallel computing, see Run MATLAB Functions with Automatic Parallel Support (Parallel Computing Toolbox).
Version History
Introduced in R2021b
Open Example
You have a modified version of this example. Do you want to open this example with your edits?
MATLAB Command
You clicked a link that corresponds to this MATLAB command:
Run the command by entering it in the MATLAB Command Window. Web browsers do not support MATLAB commands.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)