Mat file corrupt? I'm not so sure...

Hi,

I have a random issue where the following line in my code:

   data = load(job_output_filenames{job_index});

ends up generating a message that it's unable to load that mat file, and it may be corrupt. Maybe 50% of the time. Yet every time I drag it into Matlab, it opens fine, and I see 4 variable cell arrays, all looking OK. Could it possibly be memory, or heap space? I have 600mb heap now...if I go too much higher, Matlab says it cannot open, so it reduces back to 128mb.

If it happens again, I can capture exact error messages if it helps....

Dave

[David Pesetsky]

Hello?

[Geoff Hayes]

David - what can you tell us about the file? How large is it? Was it generated with the version of MATLAB that you are using to read it or did you obtain the file from somewhere or someone else?

[David Pesetsky]

OK, I was finally able to replicate the issue.

The file is 252 KB, and was generated by the same Matlab as is trying to read it back in. When I drag and drop it into Matlab, it loads fine. I see 4 variables, one is a 933x52 cell, and the other 3 are 81x69 double. This only became a problem when I added the 933x52 variable. I tried to attach the file to this post.

[David Pesetsky]

Geoff, is there an issue with the mismatching size or type of the AoA array with the other 3 variables?

[Geoff Hayes]

David - I can load the file fine with

 load('job_20150324_212227981_out.mat')

Do you see an error message when you try the above?

[David Pesetsky]

I see no errors when I load it, either by the load command or drad and drop. And now, I ran the script again and it had no issues. It is a seemingly random issue. I think there is a memory issue? Or maybe a reboot I did helped? Do you see anything I could do better with the format of that AoA array?

[Geoff Hayes]

David - I don't see any issue with the AoA, and combining variables with different dimensions shouldn't be a problem (most everyone does this).

Answer 1

[Jan]

The file is valid. If you obeserve random problems, there must be a random source for this, e.g. a corrupted hard disk or a failing network connection. So check this at first.

Another problem, which looks random, although it is not, is omitting the path of the file. Then Matlab searchs the file in the current folder, but e.g. a timer or GUI callback can change the current folder unexpectedly. If the unexpected current folder contains a damaged file with teh same name, it is hard to detect this. Therefore it is strongly recommended to include the full path of a file in evey case.

[David Pesetsky]

Darn. I added the path, and it said it was corrupt again. I am still able to manually execute the load command on that file. I may try to reformat a variable, or some other approach. Can't have it randomly not working.

Crazy.

[David Pesetsky]

Ah ha. I see a pause block just before the erroring block of code. The way this is working, Matlab spawns several parallel processes, and needs to wait until writing some mat files is complete, and then it puts the parallelly created mat files together as one. Here's the block befor the trouble spot:

        % Wait for jobs to finish
        while ~prod(done_flags)
        dir_out=dir('job_*_out.mat');
        for k2=1:length(dir_out)
            job_index = strmatch(dir_out(k2).name, job_output_filenames);
            if ~isempty(job_index) && ~done_flags(job_index)
                done_flags(job_index) = 1;
                disp(['Job FINISHED with output file ' job_output_filenames{job_index}]);
            end
        end
        pause(0.01);
    end

I'll play with the lenghth of puase time. But there's probably a better way to do whatever it's doing....

[David Pesetsky]

...so, putting a 5 second pause between the Wait block, and the block that needed to manipulate the data and was having troubles, seemed to fix it. I guess Matlab was still busy writing to arrays.