How do I determine the number of headerlines in a text document?

I need to read in data from a text document. The document starts with an unknown number of headerlines (zero up to 20), and the data is broken into two columns with an unknown number of rows.

4 points
PER          AREA
1.000000     0.995458 
0.997791     0.981313 
0.988447     0.980913 
0.987217     0.971326

I would like to determine the number of headerlines there are in the document, and then use textscan to skip them. My code so far is...

fileID = fopen('nameoffile.txt');
n = 0;
tline = fgetl(fileID);
while ( ischar(tline) || isinteger(tline) )
   tline = fgetl(fileID);
   n = n+1;
end
data = textscan( fileID, '%f%f%*[^\n]','Headerlines',n,'CollectOutput',true);

This gives me a header count, 'n', that is equal to the number of lines in the file. Is there a way to do this?

Thanks.

Answer 1

[Jos (10584)]

Read the file as strings and parse the strings. The following might work (at least it works when I copied your example into a file.)

DataFile = 'data.txt' 
C = textread(DataFile, '%s','delimiter','\n')
V = cellfun(@(s) sscanf(s,'%f%f').', C, 'un', 0)
DATA = vertcat(V{:})

[Jos (10584)]

And to obtain the headers:

HEADER = C(cellfun('isempty',V))

Answer 2

[dpb]

TMW has already done it for you...

doc importdata

[Joshua]

I tried that command, and it does work with the example I gave. However, my actual header is more complex and it doesn't want to work with it.

Is there any reason 'importdata' would not work?

   Shape name
Size #, #, #
# of vertices
Perimeter is the first column, Area is the second column.
This was created by 'person' on 'date' using the function 'name'. 
PER          AREA
1.000000     0.995458 
0.997791     0.981313 
0.988447     0.980913 
0.987217     0.971326

[dpb]

Doesn't look like should be...and indeed, it worked on that snippet of a file here--

>> josh=importdata('josh.txt')
josh = 
        data: [4x2 double]
    textdata: {6x2 cell}
  colheaders: {'PER'  'AREA'}
>> josh.data
ans =
  1.0000    0.9955
  0.9978    0.9813
  0.9884    0.9809
  0.9872    0.9713
>>

This is R2012b/32-bit Windows. Your version and if fails consistently on your system would seem to be worthy a bug report.

Answer 3

[Joseph Cheng]

Well if you are having arbitrary headers but you know where the data starts (with the column headers PER and AREA); then you can hunt for the start of data using fgetl and strfind. so you'll open the file, fgetl and given certain conditions look for strfind(line,'PER')==1 and the same for AREA (i don't know the line position of AREA). Additionally you can put in a check to see that the data afterwords is 2 numbers if ~isempty(str2num(nextline));

after you get your heading position use that in dlmread.

Answer 4

[Gabriel Felix]

I had to use \n at the end of each line. Without it I couldn't make textscan() work properly, even thoug the "HeaderLines" was configured according to the text file lines. This was the only solution I found after struggling with the code for an intire day.

This was the text:

!
!
! alfa (graus) =  5.0
!
! Id.      x/s       z/s     alfai       cl     c*cl/cmed    cdi      cmc/4
!                           (graus)
   1      .246      .050    -1.209      .255      .332     .00538     .0170
   2      .292      .150    -1.098      .259      .319     .00496     .0545
   3      .339      .250     -.925      .254      .297     .00410     .0944
   4      .385      .350     -.741      .243      .268     .00315     .1341
   5      .432      .450     -.561      .227      .235     .00223     .1714
   6      .479      .550     -.393      .206      .199     .00141     .2034
   7      .525      .650     -.238      .181      .163     .00075     .2266
   8      .572      .750     -.101      .152      .126     .00027     .2362
   9      .619      .850      .014      .116      .089    -.00003     .2236
  10      .659      .938      .103      .074      .052    -.00013     .1693
!
! CL asa    =  .208
! CDi asa   =  .00258
! e (%)     =  88.9
! CMc/4 asa =  .1339

My code:

%! alfa (graus) =  5.0
P = textscan(fid,'! alfa (graus) = %f','Delimiter',' ','MultipleDelimsAsOne',true,'headerLines',2,'CollectOutput',1);
alpha(1) = P{1};
%! CL asa    =  .208
P = textscan(fid,'! CL asa = %f\n','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'headerLines',4+n);
CL(1) = P{1};
%! CDi asa   =  .00258
P = textscan(fid,'! CDi asa   =  %f\n','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'headerlines',0);
CDi(1) = P{1};
%! CMc/4 asa =  .1339
P = textscan(fid,'! CMc/4 asa = %f','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'HeaderLines',2);
Cmc4(1) = P{1};

[dpb]

I'd go at something like this somewhat differently...this is more an extraction of data substrings than skipping the headerlines--probably worthy of a Q? on its own as the result is likely of more general interest...

GF=textread('GFelix.txt','%s','whitespace','','delimiter','\n');
% OK textread is deprecated, but it's handy 'cuz don't need fopen ...
% reads as cell array each line.
wantvars={'alfa','CL','CDi','e','CMc/4'};   % define our lookup strings..
keepers=GF(startsWith(GF,"! "+wantvars));   % save the subset only out of the file
wantnames=strrep(wantvars,'/','');          % make allowable var names
vals=num2cell(str2double(extractAfter(GF(startsWith(GF,"! "+wantvars)),'='))); 
S=cell2struct(vals,wantnames.',1)

results in

> S
S = 
  struct with fields:

    alfa: 5.00
      CL: 0.21
     CDi: 0.00
       e: 88.90
    CMc4: 0.13
>> 

[Gabriel Felix]

What an elegant solution! Thank you for helping. I posted it here because I thought it was related to this topic's subject. (This topic was one of the many I consulted for trying to solve my problem).

However I should have made a question with it. I would have done it if I hadn't solved the problem with this less elegant solution.

I really liked your approach. Thanks a lot man!

[dpb]

No problem; glad to be of help.

Illustrating some of the newer string class functionality that is useful for such -- even I wasn't sure the lookup syntax was going to work as intended but tried it as being what I knew I wanted! :)

The other really important takeaway is when you have strings for variables, use a struct or a table or one of the similar data structures that has the facility to address the field or variable name within it--don't try to make variables directly that end up with the dreaded eval as only way to use them.