Reading in csv files
1 view (last 30 days)
Show older comments
I have a bit of program to read in csv files but it takes a really long time, most likely because of the "for" loops. Can anyone inform me on ways I can make this program more efficient?
%
% Load and convert a csv file
%
%clear all contents
clear
clc
tic;
%read in file
fileName = 'SmartBowAccelTrunk.csv';
%fileName = 'testDoc.txt';
%the order of the file is as follows
%timestamp;SBname;farmid;tagmac;xval;yval;zval;modul;soft;accX;accY;accZ;absAcc
%20130625182723252;xxxx;182;000008B1;11.01;4.91;;h;5;-71;-583;831;1017.591
%d %s %d %s %f %f %f %s %d %d %d %d %d %f
%read in csv file as txt
[num, txt, raw] = xlsread(fileName); %returns a cell array of strings
%first row is header information read in as string %s
header = textscan(txt{1}, '%s%s%s%s%s%s%s%s%s%s%s%s%s', 'Delimiter', ';'); %13 columns of data... read each of the header titles
header = [header 'time' 'hour' 'minute' 'second' 'ms'];
% loop through remaining rows and extract data (first as text, then as relevant data format)
for iRow=2:length(txt),
%rowData = textscan(txt{iRow}, '%f%s%f%s%f%f%f%s%d%d%d%d%f', 'Delimiter', ';');
rowData = textscan(txt{iRow},'%s%s%s%s%s%s%s%s%s%s%s%s%s','Delimiter', ';'); %just grab as strings, convert later
for iCol=1:length(rowData)
allData(iRow-1,iCol) = rowData{iCol};
end
%change time string to number
allTimes(iRow-1) = allData(iRow-1,1);
%the number to be parsed
a=allTimes';
mstimes=rem(a, 1000000000);
tn=datenum(a,'yyyymmddHHMMSSFFF');
time_hh=str2num(datestr(tn,'HH'));
time_mm=str2num(datestr(tn,'MM'));
time_ss=str2num(datestr(tn,'SS'));
time_ms=str2num(datestr(tn,'FFF'));
end
%add row with time
%allData = [allData times];
cellTime=num2cell(mstimes);
cellHour=num2cell(time_hh);
cellMin=num2cell(time_mm);
cellSec=num2cell(time_ss);
cellMs=num2cell(time_ms);
allData=[allData cellTime cellHour cellMin cellSec cellMs];
allData=sortrows(allData,[3 14]);
output={header; allData};
toc
0 Comments
Answers (3)
Image Analyst
on 16 Jul 2013
And the reason you're not using csvread() to read in your csv file is what?????
0 Comments
Don
on 12 Oct 2018
Edited: Walter Roberson
on 13 Oct 2018
[file, path] = uigetfile('*.csv');
file_name = strcat(path, file);
csv_file = fopen(file_name);
tic;
line = fgetl(csv_file);
% Get the names form the first line
names = textscan(line, '%s');
% this reads 90,002 lines with 20 fields in 4.228 seconds
cntr = 1;
% make a cell to get started
data = cell(10,1);
while ~feof(csv_file)
line2 = fgetl(csv_file);
% Skip the first 2 columns, I don't wnat tme, take this out if you wnat
% them
[DateTime, pos] = textscan(line2, '%s', 2, 'Delimiter', ',');
len = size(line2);
data{cntr} = textscan(line2(pos+1:len(2)), '%f', 'Delimiter', ',');
cntr = cntr+1;
end
fprintf('Read %d lines\n', cntr);
fclose(csv_file);
toc
0 Comments
See Also
Categories
Find more on String Parsing in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!