copyfile skips files?

Hello everyone!

I want to create copies of specific files in a new directory. They are coming out of over 500 different subfolder, so I don't want to do it manually.

I have loaded and accessed data within all of them, so I know they exist. For some reason, not all files make it to the destination folder, but I'm not getting an error message. I also have plenty of space on my drive, so that can't be the problem either.

ddir = uigetdir;
AllFiles = dir(fullfile(ddir, '**', 'DT_data*mat'));
destination = 'C:\Users\XXXXX\Desktop\tempFigData';

for i = 1:length(AllFiles)
    sourcefile = fullfile(AllFiles(i).folder, AllFiles(i).name);
    copyfile(sourcefile, destination);
end

Does anyone have an idea what could be the problem here?

[dpb]

Not w/o being able to see the directory structure and what is returned for the dir() and the resulting content of the destination subdirectory there's no way to tell

Start by including the optional return status and message at least...

[status,msg]=copyfile(sourcefile, destination);
if ~status
  warning('Failed to copy %s\nError: %s\n',sourcefile,msg)
end

This continues on, you could error instead or whatever you care to do but should give at least clues why.

One thing to be careful of is proper construction of names if any files have embedded blanks in file name. I think fullfile() is smart enough, but not absolutely positive you may not need to enclose with "" or .. Worth checking.

[kova]

Thank you!

I'm sorry if I'm not giving the info you need, I'm very much a novice! ;-)

I will definitely start adding status messages to my scripts!

In this case however, it didn't return anything. Just to check, I tried it the other way round, without negating status, (which did return a message) and made it count for how many files this was the case. It counted 515 (the number I want), while the destination directory actually only contains 508 files.

I don't know, if this is what you meant, but dir() returns a struct with name and folder as char.

The filenames are automatically generated (they're output files from experiments), so in this case I know they don't contain any spaces. I can load the files using fullfile(), so I think that should be fine.

[dpb]

Do a dir() on the target destination and compare to the AllFiles original.

dDest=dir(fullfile(destination, '**', 'DT_data*mat'));
SrcDestDiff=setdiff({AllFiles.name},{dDest.name})

Answer 1

[Image Analyst]

Try this:

% Get top level folder.
ddir = uigetdir(pwd);
filePattern = fullfile(ddir, '**', 'DT_data*.mat');
AllFiles = dir(filePattern);
destination = 'C:\Users\XXXXX\Desktop\tempFigData';
if ~isfolder(destination)
	mkdir(destination);
end
fprintf('Found %d files.\n', length(AllFiles));
if length(AllFiles) == 0
	warningMessage = sprintf('Did not find any files matching %s', filePattern);
	uiwait(errordlg(warningMessage));
	return;
end

for i = 1:length(AllFiles)
	sourceFileName = fullfile(AllFiles(i).folder, AllFiles(i).name);
	destinationFileName = fullfile(destination, AllFiles(i).name);
	fprintf('Copying file %s\n   to %s\n', sourceFileName, destinationFileName);
	% 	copyfile(sourceFileName, destination);
end

Now what do you see?

[kova]

This is great, thank you!

Well what I'm seeing is "Found 515 files." and presumably 515 "Copying file ...". However, there are still only 508 files in the destination directory. I then used @dpb 's last suggestion (setdiff), which came out to a 1x0 empty cell. So it's not recognising they're different? I mean AllFiles is a 515x1 struct and dDest is a 508x1 struct?

I'm at a loss!

[Les Beckham]

I suspect that you have some files with duplicate names which get copied over each other.

[Stephen23]

I second Les Beckham's comment.

[dpb]

Les' suspection would be what the setdiff result implies...I intended to add one other piece but forgot...

cellfun(@(s) numel(unique(s),[{AllFiles.name},{dDest.name}])

if does show up that both are not same number unique files try again with

cellfun(@(s) numel(unique(lower(s)),[{AllFiles.name},{dDest.name}])

I don't know what copyfile does w/ case; Windows is case-preserving but not case-sensitive so possibly that could have happened somehow?

[Image Analyst]

If you don't have duplicate destination filenames, then it's time to look at the return values from copyfile(). Print them both out with an fprintf() and see what they are. They should all indicate success. I'm not sure if copyfile() throws an error with red text and stops your script. It might just log the error to the status message it returns and it's up to you to check it to decide whether or not to throw an error.

[status,msg] = copyfile(___) also returns the message text for any warning or error that occurs.

[dpb]

He's already done that in response to my first Comment above, IA. Reported no errors, no messages for all 515.

<https://www.mathworks.com/matlabcentral/answers/518934-copyfile-skips-files#comment_830520>

[dpb]

Just for grins, @kova, how about attach (use the paperclip icon) a .mat file that contains the two dir() arrays above (AllFiles and dDest)? Make detailed probing easier if folks had local copies...