How to parse text data
Show older comments
Hi
I have data in the below format. I need the mechanism to parse the data from below format with expected output.
Input data format:
07/16 12:55:22.012 INFO | test_runner_utils:0812| Began logging to /tmp/test_that_results_hatch_deL3lZ
07/16 12:55:27.477 INFO | test_runner_utils:0259| autoserv| Processing control file
Expected Output format:
Define level of message extraction based on the marker sign ==> |
-Step 1: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|
-Step 2: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>| extract full text in a variable, option to grab variable if associated with value
-Step 3: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|<string>:%1.3f|
-Step 4: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|<string>:%1.3f|extract full text in a variable, option to grab variable if associated with value
-Step 5: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|<string>:%1.3f|<string>|
-Step 6: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|<string>:%1.3f|<string>|extract full text in a variable, option to grab variable if associated with value
Input data format:
07/16 12:55:27.620 DEBUG| utils:0287| [stdout] CHROMEOS_RELEASE_BOARD=hatch
07/16 13:28:58.330 INFO | mode_switcher:0673| -[FAFT]-[ start wait_for_client ]---
Expected Output format:
-Step 1: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|<string>:%1.3f|<[string]>
-Step 2: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|<string>:%1.3f|<[string]> extract full text in a variable, option to grab variable if associated with value
-Step 3: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|<string>:%1.3f|<[string]>
-Step 4: Extract Timestamp in mm/dd HH:MM:sec.millisec <string>|<string>:%1.3f|<[string]> [string] extract full text in a variable, option to grab variable if associated with value
Input data format:
2019-07-16 12:55:30 > string
2019-07-16 12:55:30 powerbtn: released
Expected Output format:
Note the marker >
-Step 1: Extract Timestamp in YYYY:MM:DD HH:mm:sec > < string>
-Step 2: Extract Timestamp in YYYY:MM:DD HH:mm:sec < full string>
Input data format
2019-07-16 12:55:31 > [12074.734997 HC 0x121 err 1]
Expected Output format
-Step 1: Extract Timestamp in YYYY:MM:DD HH:mm:sec > [< %1.3f string extract full text in a variable, option to grab variable if associated with value>]
Thanks a lot
5 Comments
Bob Thompson
on 17 Jul 2019
Life is Wonderful
on 18 Jul 2019
Edited: Life is Wonderful
on 18 Jul 2019
Guillaume
on 19 Jul 2019
I've not looked at this question in details. Does the file format differ much from the one in your previous question?
If not, it should be fairly trivial to adapt the parser I wrote, which would be a lot less effort than starting again from scratch.
Life is Wonderful
on 19 Jul 2019
Edited: Life is Wonderful
on 19 Jul 2019
Life is Wonderful
on 23 Jul 2019
Accepted Answer
Are you still on very old version (please fill the release field next to the question)?. If on a modern version, the file can easily be read with:
VariableNames = {'Date', 'Level', 'delim1', 'PID', 'delim2', 'Message'};
VariableWidths = [19, 5, 1, 23, 2, 5000];
VariableTypes = {'datetime', 'char', 'char', 'char', 'char', 'char'};
opts = fixedWidthImportOptions('VariableNames', VariableNames, 'VariableWidths', VariableWidths, 'VariableTypes', VariableTypes, 'SelectedVariableNames', [1, 2, 4, 6]);
opts = setvaropts(opts, 'Date', 'InputFormat', 'MM/yy hh:mm:ss.SSS');
content = readtable('test_that.txt', opts);
results in:
If on a version fo matlab that doesn't have tables, use textscan with fixed width fields:
fid = fopen('test_that.txt', 'rt');
content = textscan(fid, '%18c%*c%5c%*c%23c%*2c%s', 'Delimiter', '', 'Whitespace', '');
fclose(fid);
content = [cellstr(content{1}), cellstr(content{2}), cellstr(content{3}), content{4}]
23 Comments
Life is Wonderful
on 23 Jul 2019
Edited: Life is Wonderful
on 23 Jul 2019
Guillaume
on 23 Jul 2019
You need to select a product before you can select the release (I know it's silly, we've complained enough to mathworks).
Then use the fixedWidthImportOptions version, which is more reliable than the textscan version. Matlab does all the hard work for you.
Life is Wonderful
on 23 Jul 2019
Edited: Life is Wonderful
on 23 Jul 2019
Guillaume
on 23 Jul 2019
Use readtable as I've shown, not readcell.
A table is better than a structure, you can access the data by table variable exactly as you want:
content.Date
and by rows:
content(1:10, :)
and if you convert the table into a timetable, easily select a time period:
content = table2timetable(content);
content = rmmissing(content); %to remove the bad rows
subcontent = content(timerange('1-Jul-2018 11:30', '1-Jul-2018 12:00'), :)
Life is Wonderful
on 23 Jul 2019
Guillaume
on 23 Jul 2019
I'm not sure what you've done. The code I've shown above doesn't throw that error. Whatever modification you've done is most likely the culprit.
Life is Wonderful
on 23 Jul 2019
Life is Wonderful
on 23 Jul 2019
Edited: Life is Wonderful
on 23 Jul 2019
Guillaume
on 23 Jul 2019
what does
which readData -all
return?
Life is Wonderful
on 23 Jul 2019
Edited: Life is Wonderful
on 23 Jul 2019
Guillaume
on 23 Jul 2019
"Bundle#361 start failed"
Wow, that's some very weird error that I've never come across. There's something very weird going on with your install. If you have active support you should contact Mathworks. which shouldn't error, it's a built-in function.
Life is Wonderful
on 23 Jul 2019
sriram shastry comment posted as an answer moved here:
Thanks .I did the software re-install. I can see the data without any errors.
What is not clear for me now is how to plot the data . I can plot content.Data but rest I can't .Date is not in string format but message,PID and message are string cell.
Please suggest plotting mechanism so that data can be looked more efficiently.
>> content.Date(1:10,1)
ans =
10×1 datetime array
07/18 11:27:02.968
07/18 11:27:02.968
07/18 11:27:02.968
07/18 11:27:02.968
07/18 11:27:02.968
07/18 11:27:02.969
07/18 11:27:02.969
07/18 11:27:02.970
07/18 11:27:02.972
07/18 11:27:02.972
>> content.Level(1:10,1)
ans =
10×1 cell array
{'DEBUG'}
{'DEBUG'}
{'DEBUG'}
{'INFO' }
{'INFO' }
{'DEBUG'}
{'DEBUG'}
{'DEBUG'}
{'DEBUG'}
{'DEBUG'}
>> content.PID(1:10,1)
ans =
10×1 cell array
{'autoserv:0729' }
{'autoserv:0730' }
{'autoserv:0731' }
{'autoserv:0739' }
{'pidfile:0016' }
{'host_info:0263'}
{'host_info:0267'}
{'base_job:0357' }
{'base_job:0357' }
{'base_job:0357' }
>> content.Message(1:10,1)
ans =
10×1 cell array
{'autoserv is running in drone lab_chrome-debug.' }
{'autoserv command was: /build/hatch/usr/local/build/autotest/server/autoserv -p -r /tmp/test_that_results_hatch_iGVg61/results-1-firmware_UpdateKernelSubkeyVersion -m 10.223.131.106 --no_console_prefix -u autotest_system -l ad_hoc_build/ad_hoc_suite/firmware_UpdateKernelSubkeyVersion -s --no_use_packaging /tmp/tmphhqbvd --args servo_host=localhost servo_port=9999' }
{'autoserv parsed options: Namespace(args=' servo_host=localhost servo_port=9999', cleanup=False, client=False, cloud_trace_context='', cloud_trace_context_enabled='False', collect_crashinfo=False, control_filename=None, control_name=None, disable_sysinfo=False, execution_tag='', group_name='', host_attributes={}, host_info_subdir='host_info_store', install_in_tmpdir=False, isolate='', job_labels=None, lab='', label='ad_hoc_build/ad_hoc_suite/firmware_UpdateKernelSubkeyVersion', local_only_host_info=False, machines='10.223.131.106', machines_file=None, no_console_prefix=True, no_logging=False, no_tee=False, no_use_packaging=True, parent_job_id=None, parse_job='', pidfile_label='autoserv', provision=False, repair=False, require_ssp=False, reset=False, results='/tmp/test_that_results_hatch_iGVg61/results-1-firmware_UpdateKernelSubkeyVersion', server=True, skip_crash_collection=False, ssh_options='', ssh_pass='', ssh_port=22, ssh_user='root', ssh_verbosity=0, test_source_build='', use_existing_results=False, user='autotest_system', verbose=False, verify=False, verify_job_repo_url=False, write_pidfile=True)'}
{'Results placed in /tmp/test_that_results_hatch_iGVg61/results-1-firmware_UpdateKernelSubkeyVersion' }
{'Logged pid 23629 to /tmp/test_that_results_hatch_iGVg61/results-1-firmware_UpdateKernelSubkeyVersion/.autoserv_execute' }
{'Committing HostInfo to store InMemoryHostInfoStore[HostInfo[Labels: [], Attributes: {}]]' }
{'HostInfo updated to: HostInfo[Labels: [], Attributes: {}]' }
{'Persistent state global_properties.tag now set to ''' }
{'Persistent state global_properties.fast now set to False' }
{'Persistent state global_properties.max_result_size_KB now set to 40000' }
later comment moved here as well
Any Feedback
- on plotting the data.
- Convert timestamp into CPUtime
Thanks
Guillaume
on 31 Jul 2019
I have no idea what you mean by plotting. How do you plot text? What do you want to plot against what?
I also don't know what you mean by converting to cpu time. cpu time usually mean the amount of time that a process/program spend on calculation. It's a duration, so has nothing to do with timestamps which are points in time.
Life is Wonderful
on 1 Aug 2019
Guillaume
on 1 Aug 2019
Level is either 'DEBUG', 'INFO', 'ERROR', etc. Again, I've no idea what you mean by plotting these. I don't even have a clue what sort of plot you're looking for. A line plot?, A bar plot? A pie chart? A scatter plot? None of these make sense for text.
If you want to convert the date into duration from the start of the log (calling that cputime would be very misleading), then it's trivial:
content.Duration = content.Date - Content.Date(1);
Life is Wonderful
on 1 Aug 2019
Guillaume
on 1 Aug 2019
I am fine with Line plot
A line plot of what? What does 'DEBUG' look like on a line plot? I'm sorry I really don't understand what you want to plot. For me, plotting text makes no sense at all. You plot numbers, not text.
Life is Wonderful
on 1 Aug 2019
Life is Wonderful
on 1 Aug 2019
Guillaume
on 1 Aug 2019
Yes, you can put text on a figure. How do you determine the position of said text. You need at least an x (time?) and a y (????).
Maybe you should draw what you want (on paper, in excel, whatever) because I still don't understand.
On the other, we've completely deviated from your original question which I believe has been answered, so maybe you should start a new question dedicated to your plotting. Others may be more likely to contribute that way.
Life is Wonderful
on 2 Aug 2019
Edited: Life is Wonderful
on 2 Aug 2019
Life is Wonderful
on 2 Aug 2019
More Answers (2)
Bob Thompson
on 18 Jul 2019
0 votes
I need next steps
◾Convert Datacontent into cell's - like timestamp , message data-1,message data-2
◾Put cell in proper format
◾Create Matlab variables
◾Display Matlab variable for good analysis
1) regexp automatically outputs all results in a cell, each containing a string.
2) You can convert strings to date time formats using datetime. To do this 'quickly' I suggest using a loop through your regexp results, or by using cellfun (which is really still a loop).
3) What exactly do you mean by this? I personally do not know of a way to dynamically create variables within Matlab, and I think you would be better served to keep the information in a cell array, or to make a table out of it. It is certainly possible to create new variables in a table from a captured string from regexp.
4) Displaying Matlab variables is simply a matter of not suppressing them, or if specifically wanting to display them then you can use fprintf with no target so it defaults to the command window.
5 Comments
Life is Wonderful
on 18 Jul 2019
Edited: Life is Wonderful
on 18 Jul 2019
Life is Wonderful
on 18 Jul 2019
Bob Thompson
on 18 Jul 2019
You're getting the structure class error because 'names' outputs the results as a structure, rather than a cell, as I was expecting. Personally, I prefer 'tokens' or 'match' as my output flag for regexp.
Cellfun will not work with any input that is not a cell, hence the error.
I would suggest something like the following:
fileData = regexp(filecontent, '^(?<Time_MDY>[^ ]+) (?<Time_HMSsss>[^ ]+) (?<first>[^|]\w+)|\s+(?<last>[^|\r\n]+)|(?<last>[^|\r\n]+),\s+(?<first>[^|]|\w+)', 'tokens', 'lineanchors');
dates = datetime([fileData{1}{1},' ',fileData{1}{2}], 'InputFormat', 'MM/dd HH:mm:ss.SSS');
I did use a singular line to test this, so if you have multiple rows of inputs and outputs from regexp then you may need to investigate using a loop.
Bob Thompson
on 19 Jul 2019
Are you only looking to capture the timestamp? It seems like the issue is more in the initial regexp processing than in the date time conversion.
If you are only looking to capture the timestamp I would suggest doing a regexp call like this:
filedata = regexp(filecontent'(\d\d.\d\d\s\d\d.\d\d.\d\d.\d\d\d)\D+\d\d\d\d\D+\n','tokens');
dates = datetime([filedata{:}], 'InputFormat', 'MM/dd HH:mm:ss.SSS');
If you are looking to capture more than the timestamps then please explain more. I know you outline some more in your OP, but I'm not entirely sure what you're referring to.
Life is Wonderful
on 19 Jul 2019
Edited: Life is Wonderful
on 19 Jul 2019
Life is Wonderful
on 18 Jul 2019
0 votes
Categories
Find more on Cell Arrays in Help Center and File Exchange
on 17 Jul 2019
on 2 Aug 2019
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
