Question about parforloop in matlab

Hi, I am using parfor loop on matlab. However, I am having troubles with printing to a file using fprintf command. I am getting the following error:

Invalid file identifier.  Use fopen to generate a valid file identifier.

Please Help!

Here's my code:

tic
i = 1;
 fID = fopen('datafile7.txt', 'W');
fprintf(fID,'name\trow\tcolumn\t25R\t25G\t25B\t1R\t1G\t1B);
     s = ['photo000' int2str(i) '.jpg'];
     A=imread(s);
    [m,n,rgb]=size(A);
    for row=4:(m-3)
        parfor column=4:(n-3)

fprintf(fID, 'image%d\t%d\t%d\t%d\t%d\t%d\t%d\t%d\t%d\t\r\n',i,row,column,A(row-3:row+3,column-3:column+3,1), A(row-3:row+3,column-3:column+3,2), A(row-3:row+3,column-3:column+3,3) ); 

          end
      end

fclose(fID);
toc

[SK]

You have multiple processes all trying to write to the same file simultaneously. Something is bound to go wrong. Why do you want to do this? Even if it worked, the file would be garbled. Just use a normal for loop.

[John]

The matrix I am dealing with is very large (takes me 85 hours approx to run without parfor). That is the reason why I am using parfor.

[Matt J]

It seems like a very bad idea to extract all the 7x7 blocks of your very large image and write them to a text file. This will consume a lot of memory redundantly as well as being very slow. Why not extract the blocks for processing on the fly?

[Adam]

If you use a matfile (doc matfile) you can write to the same file using a parfor loop, but it is not necessarily very performant.

I do it for a computationally expensive algorithm, but I never had time to work out how of it it is possible to have a thread doing the file writes while the rest of the threads get on with their next bit of processing so my algorithm has ~10% downtime writing block results to a matfile. It does work though .

[John]

I dont understand what you mean by processing them on the fly. I have to create a distance matrix for the image an then conduct a KNN search for skin segmentation. I thought it would easier to access data if we already have them on a txt file.

Answer 1

[Sean de Wolski]

Rather than writing results to the same file, write to N files and concatenate them later where N is the number of workers you have.

[John]

I am not familiar with concatenating N files. Could you give me an example. Also, I'll be creating 2000 files then, will concatenating them not take a lot of time?

[Sean de Wolski]

I'd continue the discussion with Matt. There's probably no reason to do this at all and there are probably better file formats that you could us e(such as Tiff) for this type of operation.

The good thing about Tiff is it supports partial loading and direct parallel processing with blockproc.

[John]

So instead for storing every thing in a file, I should calculate the distance matrix directly from the image pixels? Also, will Tiff support RGB pixel values?

[Sean de Wolski]

Yes, Tiff support rgb.

How big is the image?

Answer 2

[Matt J]

I have to create a distance matrix for the image an then conduct a KNN search for skin segmentation.

I don't have a clear picture of what that means, but if the processing operates on one 7x7 image block at a time, why not use

     blockproc(..., 'UseParallel',1)

to obtain the final result directly?