How to calculate an average of a weather variable for a month during daylight hours only?

I need help computing an average value of a weather variable for daylight hours during one month. I have daily data from an instrument for January 2019, which can be found here (ftp://ftp.archive.arm.gov/haegelea1/217376_extractedData/ascii-csv/). The instrument measures cloud base height (which is denoted as "first_cbh" in the data). The instrument takes a measurement every 16 seconds starting at midnight. I have data as daily files for January 19 each with time & cloud base height and various other non-needed variables. The time is measured as an offset from midnight in interval of 16 seconds. I need to calculate an average of the "first_cbh” column for the entire month but only during daylight hours. I have having difficulty computing an average during only the daylight hours. Not sure how to approach the time variable?

[Steven Lord]

Does your data set include information about whether a measurement was taken during daylight hours?

Do you have a function that given a location can determine whether a particular time falls within daylight hours?

• For example, in Natick, MA (where MathWorks HQ is) the sun rose at 7:04 AM and will set at 6:46 PM.
• In Boston, MA, just a few miles east sunrise was at 7:03 AM and sunset will be at 6:45 PM. Pretty close to the Natick times, but if your sampling interval is 16 seconds those minor differences could include or exclude several samples.
• In Washington DC the sun rose at 7:26 AM and will set at 7:10 PM.

Or are you approximating, that say "daylight hours" are between 7 AM and 7 PM?

[Aaron Haegele]

I do not have nor know of a function that given location can determine whther a particular time falls within daylight hours. The instrument location is Lamont Oklahoma though. If there is such function that would be great if not I guess the best way would be to approximate daylight hours?

Answer 1

[Jon]

Here's an example of one way to do it

filename = 'sgpceil10mC1.b1.20190101.000000.custom.csv'
% read data into a table
T = readtable(filename);

% convert base time from posixtime to MATLAB datetime array
T.base_time = datetime(T.base_time,'ConvertFrom','posixtime');

% make new column with absolute time
% (if you don't need it for anything else you could also just make a variable named t
% and work with that, but otherwise might be handy to keep in table)
T.t = T.base_time + seconds(T.time_offset);

% get logical indices of daytime hours between 6:00AM and 6:00PM (18:00)
idlDay = hour(T.t) >= 6 & hour(T.t) <= 18

% now get data of interest, for example background light
first_cbh_daytime = T.first_cbh(idlDay);

% and compute the mean
first_cbh_bar = mean(first_cbh_daytime) 

[Aaron Haegele]

What would be the best way to read all the files (31 files for 31 days in January)? Can I use a loop to repeat this process or do you read all the files first in like a datastore?

[Jon]

I would make a loop to read each of the files into an overall table and then use logical indexing as shown in above to select out the data you want and work with it.

One way to do this is something like this (just an outline here you will have to fill in details)

The idea of using the vertcat on the cell array of individual tables is a trick I learned from one of @Stephen Cobeldicks's posts

% get list of files, assuming they are in the same directory as you are running the code
% otherwise you can generalize this further
list = dir('*.csv')

% determine how many days were read
numDays = length(list)
% make cell array to hold individual day tables
dayTables = cell(numDays,1)

% loop through the day files
for k = 1:numDays
    dayTables{k} = readtable(list(k).name)
end

% combine into overall table
overallTable = vertcat(dayTables{:});