big dataset: how to find specific data that would require taking the mean?

I have a csv file, millions of rows. It consists of time series data of a few thousand individual households. I need to find various contrasting household consumption profiles. Is there a way I can somehow find the type of profiles I am after? Trying to simply import the dataset is not an option, it is too large.

To look at the profiles I would probably have to put them into an hourly average for the year (ie. average for all the 0000, 0001, ..., 2300hrs) of the year and see how the consumption profiles looks. Though this isn't necessarily essential so long as I can see there is a general trend in the profile for a particular house.

[dpb]

Have you actually tried to import the data? What is the file organization; doesn't seem to match up unless this is something like several years' electric consumption on a minute-by-minute or even finer resolution basis, maybe? There are facilities in Matlab for "tall" arrays beginning in R2016b but there are other facilities prior to that which can be used effectively as well.

Certainly agglomeration is helpful but need a little more detail on what really intend to try to compute.

[Nikhil Sreekumar]

Hi Maitiumc,

There are certain functionalities available in MATLAB for processing very large data. But certain of those functionalities have been introduced recently. If you want to have a quick look on various ways in MATLAB to handle very large data, please have a look into the below link:

• Large Files and Big Data

Also regarding the computation of hourly average a year, you may have to group them and do a grpstats on the data. An example of Statistical and Machine Learning is provided in the following link:

• Statistical and Machine Learning with Big Data

In order to help you out more, could you please specify what you have done until now and the results that you have obtained?

Thanks,

Nikhil