Find Events in Timestamped Data
This example shows how to use events to work with timestamped data stored in timetables. An event consists of a timestamp (when something happened), often a description (what happened), and sometimes additional information about the event. The timestamp for an instantaneous event is a single point in time, either a
duration value. The timestamp for an interval event consists of two points in time that define the start and end of the event.
You can use events to find or mark data of interest for plotting and analysis. This example shows how you can define events in data using information that is within your data set. To add events from an external data source to your timestamped data, see Add Events from External Data to Timetable.
You can store events separately from the main data set. A timetable is a convenient way to store events because it can hold the event timestamps along with their descriptions and any other information. You can use the event timestamps as subscripts into the main data set to select data at, before, or after an event, or between two events. You can also store events in the same timetable as the data set they describe by using one or more additional timetable variables. This example shows how to use both representations. The representation that is more useful depends on the data analysis you plan to perform.
Define Data Set Using Variations in Length of Day
By definition a day is 86,400 seconds long, where the second has a precise definition in the International System of Units (SI). However, the length of a day actually varies due to several physical causes. It varies with the seasons by as much as 30 seconds over and 21 seconds under the SI definition because of the eccentricity of Earth's orbit and the tilt of its axis. Averaging these seasonal effects enables the definition of the mean solar day, which does not vary in length over a year.
Also, there is a very long-term slowing in the rotational speed of the Earth due to tidal interaction with the moon; a smaller, opposite, shorter-term component believed to be due to melting of continental ice sheets; very short-term cycles on the order of decades; and unpredictable fluctuations due to geological events and other causes. Because of those effects, the length of a mean solar day might increase or decrease. In recent decades, it has fluctuated up and down, but has mostly been 1–3 milliseconds longer than 86,400 seconds. That difference is known as the excess Length of Day, or excess LOD.
For this example, create a timetable that contains the excess LOD for every day from January 1, 1962, to the present. The International Earth Rotation and Reference Systems Service (IERS) collects and publishes this data. However, this data needs preprocessing before storing in a MATLAB timetable because the dates are modified Julian dates. To read the IERS data into a table, use the
readtable function. Rename the two variables of interest to
file = "https://datacenter.iers.org/data/latestVersion/223_EOP_C04_14.62-NOW.IAU1980223.txt"; IERSdata = readtable(file,"NumHeaderLines",14); IERSdata.Properties.VariableNames([4 8]) = ["MJD","ExcessLOD"];
To store the excess LOD values in a timetable, convert the modified Julian dates to
datetime values. Use the
datetime function with the
"ConvertFrom","mjd" name-value argument. Then convert
IERSdata from a table to a timetable using the
IERSdata.Date = datetime(IERSdata.MJD,"ConvertFrom","mjd"); IERSdata.ExcessLOD = seconds(IERSdata.ExcessLOD); IERSdata = table2timetable(IERSdata(:,["Date","ExcessLOD"]))
IERSdata=22015×1 timetable Date ExcessLOD ___________ ____________ 01-Jan-1962 0.001723 sec 02-Jan-1962 0.001669 sec 03-Jan-1962 0.001582 sec 04-Jan-1962 0.001496 sec 05-Jan-1962 0.001416 sec 06-Jan-1962 0.001382 sec 07-Jan-1962 0.001413 sec 08-Jan-1962 0.001505 sec 09-Jan-1962 0.001628 sec 10-Jan-1962 0.001738 sec 11-Jan-1962 0.001794 sec 12-Jan-1962 0.001774 sec 13-Jan-1962 0.001667 sec 14-Jan-1962 0.00151 sec 15-Jan-1962 0.001312 sec 16-Jan-1962 0.001112 sec ⋮
Plot the excess LOD as a function of time.
plot(IERSdata.Date,IERSdata.ExcessLOD,"b-"); ylabel("Excess LOD");
Find Events in Data Set
Since the 1960s there have been several periods when the excess LOD decreased over the short term. If you smooth the excess LOD data, you can see this local behavior more easily.
To smooth the excess LOD, use the
smoothdata function. Then plot the smoothed data over the excess LOD.
IERSdata.SmoothedELOD = smoothdata(seconds(IERSdata.ExcessLOD),"loess","SmoothingFactor",.4); plot(IERSdata.Date,IERSdata.ExcessLOD,"b-"); hold on plot(IERSdata.Date,IERSdata.SmoothedELOD,"g-","LineWidth",2); hold off ylabel("Excess LOD");
The peaks and troughs of the smoothed data show where the short-term trend changed direction. After reaching a peak, the excess LOD decreases. After reaching a trough, the excess LOD increases. The peaks and troughs are notable events in this data set.
To identify the peaks and troughs in the smoothed data, use the
islocalmin functions. Then get the date and the value of the excess LOD for each peak and trough. Create a categorical array with two types,
trough, that describe these two types of events.
peaks = find(islocalmax(IERSdata.SmoothedELOD)); troughs = find(islocalmin(IERSdata.SmoothedELOD)); Date = IERSdata.Date([peaks;troughs]); Value = IERSdata.SmoothedELOD([peaks;troughs]); Type = categorical([zeros(size(peaks)); ones(size(troughs))],[0 1],["peak","trough"]);
Save Events in Timetable
Create a timetable that contains the peak and trough events. Each row of the timetable contains the event date, the type of event, and the value of the excess LOD at that event.
extremaEvents = sortrows(timetable(Date,Type,Value))
extremaEvents=9×2 timetable Date Type Value ___________ ______ __________ 16-Jul-1972 peak 0.0030145 04-Aug-1975 trough 0.0027592 25-Jun-1977 peak 0.0028619 11-Jan-1987 trough 0.0012623 31-Oct-1993 peak 0.0023055 11-Nov-2003 trough 0.00031077 13-Feb-2008 peak 0.00087789 18-Jul-2010 trough 0.00074339 24-Dec-2015 peak 0.0012426
Plot Events Against Data
Mark the peaks and troughs on the plot, using a triangle pointed upward for peaks and a triangle pointed downward for troughs.
hold on isPeak = (extremaEvents.Type == "peak"); hpeaks = plot(extremaEvents.Date(isPeak),extremaEvents.Value(isPeak),"y^","MarkerFaceColor","y"); isTrough = (extremaEvents.Type == "trough"); htroughs = plot(extremaEvents.Date(isTrough),extremaEvents.Value(isTrough),"yv","MarkerFaceColor","y"); hold off
Find Data Between Two Events
Select the subset of data between the second peak and the second trough.
First specify the time range of these events by using the
timerange function and the dates on which the peak and trough occurred.
between2peak2trough = timerange(extremaEvents.Date(3),extremaEvents.Date(4))
between2peak2trough = timetable timerange subscript: Select timetable rows with times in the half-open interval: [25-Jun-1977 00:00:00, 11-Jan-1987 00:00:00) See Select Timetable Data by Row Time and Variable Type.
IERSdata using the time range. Display the start and the end of the subset using the
segment4 = IERSdata(between2peak2trough,:); head(segment4)
ans=8×2 timetable Date ExcessLOD SmoothedELOD ___________ ____________ ____________ 25-Jun-1977 0.002277 sec 0.0028619 26-Jun-1977 0.002236 sec 0.0028619 27-Jun-1977 0.002147 sec 0.0028619 28-Jun-1977 0.002032 sec 0.0028619 29-Jun-1977 0.001936 sec 0.0028619 30-Jun-1977 0.001871 sec 0.0028619 01-Jul-1977 0.001875 sec 0.0028619 02-Jul-1977 0.00193 sec 0.0028619
ans=8×2 timetable Date ExcessLOD SmoothedELOD ___________ _____________ ____________ 03-Jan-1987 0.0015672 sec 0.0012623 04-Jan-1987 0.001762 sec 0.0012623 05-Jan-1987 0.0018557 sec 0.0012623 06-Jan-1987 0.0018304 sec 0.0012623 07-Jan-1987 0.0016989 sec 0.0012623 08-Jan-1987 0.0014924 sec 0.0012623 09-Jan-1987 0.0012496 sec 0.0012623 10-Jan-1987 0.0010093 sec 0.0012623
Create Interval Events from Instantaneous Events
From peak to trough, the excess LOD is decreasing, meaning that the Earth's rotation speeds up during that interval. Store these decreasing periods in a table as interval events. The last peak has no matching trough, so use the last time in the data set as the end time for the last interval.
StartDate = IERSdata.Date(peaks); EndDate = [IERSdata.Date(troughs); IERSdata.Date(end)]; decreasingEvents = table(StartDate,EndDate)
decreasingEvents=5×2 table StartDate EndDate ___________ ___________ 16-Jul-1972 04-Aug-1975 25-Jun-1977 11-Jan-1987 31-Oct-1993 11-Nov-2003 13-Feb-2008 18-Jul-2010 24-Dec-2015 10-Apr-2022
Compute the average decrease in excess LOD during each interval (in units of seconds of daily excess LOD per year). Add that information to the interval events in the table.
dTime = decreasingEvents.EndDate - decreasingEvents.StartDate; dExcess = IERSdata.SmoothedELOD(decreasingEvents.EndDate) - IERSdata.SmoothedELOD(decreasingEvents.StartDate); decreasingEvents.AnnualAvgDecrease = seconds(dExcess ./ years(dTime))
decreasingEvents=5×3 table StartDate EndDate AnnualAvgDecrease ___________ ___________ _________________ 16-Jul-1972 04-Aug-1975 -8.3728e-05 sec 25-Jun-1977 11-Jan-1987 -0.00016756 sec 31-Oct-1993 11-Nov-2003 -0.0001989 sec 13-Feb-2008 18-Jul-2010 -5.5449e-05 sec 24-Dec-2015 10-Apr-2022 -0.00026636 sec
These results show that the mean solar day, averaged over an entire year, has been decreasing over the last few years by about 0.3 milliseconds per year. The mean solar day is currently near or even slightly less than 86,400 seconds. However, many experts believe that this trend will not continue.
Convert Instantaneous Events to State Variable Using
extremaEvents timetable records times at which the smoothed excess LOD reached a peak and began a decrease or reached a trough and began an increase. Another way to represent those changes is as a state variable that indicates whether the data are increasing or decreasing at any given time in the data set. To convert the discrete peak and trough events into a new increasing and decreasing state variable, first assign the peak and trough events into the excess LOD data at their time of occurrence.
The new timetable variable,
State, is a state variable
with three states. The states are
peak at any peak,
trough at any trough, and
<undefined> at any other time between a peak and a trough.
IERSdata.State(extremaEvents.Date) = extremaEvents.Type; IERSdata(timerange("23-Dec-2015","1-Jan-2016"),:)
ans=9×3 timetable Date ExcessLOD SmoothedELOD State ___________ _____________ ____________ ___________ 23-Dec-2015 0.0016556 sec 0.0012426 <undefined> 24-Dec-2015 0.0015032 sec 0.0012426 peak 25-Dec-2015 0.0014242 sec 0.0012426 <undefined> 26-Dec-2015 0.0014033 sec 0.0012426 <undefined> 27-Dec-2015 0.0014386 sec 0.0012426 <undefined> 28-Dec-2015 0.0015334 sec 0.0012426 <undefined> 29-Dec-2015 0.0016433 sec 0.0012426 <undefined> 30-Dec-2015 0.0017386 sec 0.0012425 <undefined> 31-Dec-2015 0.0018316 sec 0.0012425 <undefined>
Now rename the values in
State to represent the state at any time, not just at peaks and troughs. Use the
renamecats function to rename the states as
increasingLOD. These two states represent decreasing LOD and increasing LOD.
To carry these state variable values forward to each time in the data, use the
retime function, specifying the
previous method. After every trough,
retime uses the
previous method to carry
increasingLOD forward as the state, until
retime encounters a peak. Then it carries
decreasingLOD forward as the state.
IERSdata.State = renamecats(IERSdata.State,["decreasingLOD","increasingLOD"]); IERSdata = retime(IERSdata,IERSdata.Date,"previous"); IERSdata(timerange("23-Dec-2015","1-Jan-2016"),:)
ans=9×3 timetable Date ExcessLOD SmoothedELOD State ___________ _____________ ____________ _____________ 23-Dec-2015 0.0016556 sec 0.0012426 increasingLOD 24-Dec-2015 0.0015032 sec 0.0012426 decreasingLOD 25-Dec-2015 0.0014242 sec 0.0012426 decreasingLOD 26-Dec-2015 0.0014033 sec 0.0012426 decreasingLOD 27-Dec-2015 0.0014386 sec 0.0012426 decreasingLOD 28-Dec-2015 0.0015334 sec 0.0012426 decreasingLOD 29-Dec-2015 0.0016433 sec 0.0012426 decreasingLOD 30-Dec-2015 0.0017386 sec 0.0012425 decreasingLOD 31-Dec-2015 0.0018316 sec 0.0012425 decreasingLOD
retime is a valuable tool to convert from instantaneous events to a state variable. In this example, the peaks and troughs occurred at certain row times of the data timetable. But even for externally defined events whose times are not in your data set,
retime enables you to evaluate the state at only the times that do appear in the data. In this example, passing
IERSdata.Date as an argument into
retime shows this capability.
Finally, highlight the segments where excess LOD is decreasing in red. Even though
decreasingEvents contains the start and end times of those segments, you can plot those segments more easily using the state variable.
delete([hpeaks; htroughs]); hold on decreasing = (IERSdata.State == "decreasingLOD"); plot(IERSdata.Date(decreasing),IERSdata.SmoothedELOD(decreasing),'r.'); hold off
Alternatively, highlight the background in those regions. In this case, the interval events are more convenient.
hold on startEnd = [decreasingEvents.StartDate decreasingEvents.EndDate]; h = fill(startEnd(:,[1 2 2 1]),[-.002 -.002 .005 .005],"red","FaceAlpha",.2,"LineStyle","none"); hold off
Find More Complex Events in Data
The excess LOD has both increased and decreased since the 1960s. Indeed, in many years there were short periods when the raw excess LOD was significantly negative. These are only very short-term fluctuations, but during those periods the Earth was rotating one millisecond or more faster than 86,400 SI seconds.
plot(IERSdata.Date,IERSdata.ExcessLOD,"b-"); ylabel("Excess LOD"); hold on line(IERSdata.Date([1 end]),[0 0],"Color","k","LineStyle",":") hold off ylabel("Excess LOD");
Identify the years in which the excess LOD was negative on any day. Then use
retime to find the minimum excess LOD in each of those years and create a timetable of events. These are interval events in one sense but are stored as instantaneous events marked only by their year.
negLOD = IERSdata(IERSdata.ExcessLOD < 0,"ExcessLOD"); negYears = unique(dateshift(negLOD.Date,"start","year")); negYears.Format = "uuuu"; negLODEvents = retime(negLOD,negYears,"min"); negLODEvents.Properties.VariableNames = "MinExcessLOD"
negLODEvents=25×1 timetable Date MinExcessLOD ____ ______________ 1984 -9.38e-05 sec 1986 -1.33e-05 sec 1987 -0.0001492 sec 1988 -7.06e-05 sec 1999 -0.0001063 sec 2000 -0.000311 sec 2001 -0.0007064 sec 2002 -0.0007436 sec 2003 -0.0009769 sec 2004 -0.0010672 sec 2005 -0.0010809 sec 2006 -0.0003865 sec 2007 -0.0006192 sec 2008 -0.0003945 sec 2009 -0.0004417 sec 2010 -0.000784 sec ⋮
Mark the time axis red for each year that had periods when the excess LOD was negative. In this data set, such years happen more frequently after the year 2000.
hold on plot([negLODEvents.Date negLODEvents.Date+calyears(1)],[-.0016 -.0016],"r-","lineWidth",6); ylim(seconds([-.0016 .0045])); hold off
Whether you use instantaneous events, state variables, or switch between them depends on which representation is more convenient and useful for the data analysis that you plan to carry out. Both representations are useful ways to add information about events to your timestamped data in a timetable.