Extracting data from strings with varying delimiters and column widths

I have read in the data as a single column array of strings and stripped out all the extra information and have each set of node temperatures seperated by a "--------------------" row.

My main trouble is dealing with the occurance that can be seen in the final line of this timestep where we have temperatures of 1000 or higher and the space disapears between the temeprature value and the node number and the node and temperature columns are different widths so can't be split by a single width.

Any help on solving this would be great

CONVERGENCE HAS BEEN OBTAINED.
 =============================
    TIME =  1646.00000


 TOTAL TEMPERATURES.
 --------------------
 NODE TEMP.  NODE TEMP.  NODE TEMP.  NODE TEMP.  NODE TEMP.
    1  98.7    2  98.7    3  91.6    4  91.6    5  85.1
    6  85.1    7  79.2    8  79.2    9  73.8   10  73.8
   11  68.8   12  68.8   13  64.3   14  64.3   15  60.0
...
  346  20.0  347  20.0  348  20.0  349  20.0  350  20.0
  351 110.2  352 110.2  353 312.0  354 312.0  355 582.7
  356 582.7  357 753.4  358 753.4  359 854.4  360 854.4
  361 932.8  362 932.8  363 999.5  364 999.5  3651050.0
  3661050.0  3671087.7  3681087.7  3691117.9  3701117.9

[Stephen23]

You should use readtable, which can parse fixed-width text files:

https://www.mathworks.com/help/matlab/ref/readtable.html

Read more on this feature:

https://www.mathworks.com/help/matlab/ref/matlab.io.text.fixedwidthimportoptions.html

[Alastair Temple]

Can I use readtable on data already in matlab (as I need to strip out lot of extra info first)? Or would I need to read in the origional output file, strip out the extra data... write it back to a text file or csv and then re-read it using readtable?

[Stephen23]

As far as I can tell readtable was not designed to parse strings/character vectors.

This quite an oversight, as this was a very very useful feature of textscan.

Answer 1

[Stephen23]

You can insert delimiters between those numbers, e.g. using regexprep:

str = regexprep(str,' *(\d+?) *(\d{1,4}\.\d+)','$1,$2,')

Note that regexprep can be applied to a string array or to a cell array of character vectors (no loop required).

Then use textscan or sscanf or whatever you prefer to convert to numeric, e.g. for each line of text:

sscanf(str,'%f,',[1,Inf])

[Alastair Temple]

Works perfectly thank you.

Just to make sure I am understanding correctly the "matching" part of the expression :

' *(\d+?) *(\d{1,4}\.\d+)'

does the following:

1. Creates 2 tokens to be matched, the first part creates a 2 or 3 digit number (d = digit, + = repeate at least once (giving you 2 digits or more) and the ? meaning repeate previous item once or no times, giving a final of 2 or 3 digits).
2. The second part asks for a 1 to 4 digit number followed by a '.' and more digits.

then the replacement part says to seperate them both with a ',' in the order 'token 1' followed by 'token 2'?

[Stephen23]

Almost.

In this case the ? actually modifies the preceding quantifier to be lazy (by default quantifiers are greedy, i.e. match as much as they can).

' *(\d+?) *(\d{1,4}\.\d+)'
% *                         zero or more spaces
%  (                        start token 1 (for Node #)
%   \d+                     match one or more digits, but...
%      ?                    lazy match, i.e. match as *few* characters as possible
%       )                   end token 1
%         *                 zero or more spaces
%          (                start token 2 (for Temp #)
%           \d{1,4}         match from one to four digits (default = greedy)
%                  \.       match period character
%                    \d+    match one or more digits
%                       )   end token 2

Note that the regular expression is anchored on the period character. Token 2 will (greedy) match from 1 to 4 digits before the period, but no more than that, so even if the space is missing, this fixes the length of token 2 to a maximum of four digits. Because token 1 uses a lazy quantifier, it collects however many digits are remaining before token 2. Read more here:

https://www.mathworks.com/help/matlab/matlab_prog/regular-expressions.html#f0-43073

[Alastair Temple]

Awesome thank you again (also very nice and neat explaination there).