Decreasing Computational Time with Parfor and variable slicing

1 view (last 30 days)
Ryan
Ryan on 16 Apr 2013
Good day,
I am fairly new to parallel computing and so far I feel like I have been successful. However I have written a code in parallel with parfor and I am using a HUGE data set on the magnitude of 34000 x n. I was wondering if there is a way to make my computations even more efficient. I also have a message saying that variable is indexed but not sliced in a parfor loop. This might result in unnecessary communication overhead. Here is a copy of my code
softTFIDFMat = zeros(n,n);
parfor i=1:n
temp = zeros(1,n);
for j=i:n
score = tfidfn(i,:)'*tfidfn(j,:).*jMat;
score = sum(score(:));
temp(j) = score;
end
softTFIDFMat(i,:) = temp;
end
tfidfn is a sparse matrix that is 34303 x n, where n is generally > 2000 and jMat is also a sparse double. Any help would be appreciated. Computational time is a little under 24 hours as of now.

Accepted Answer

Jan
Jan on 17 Apr 2013
tfidfn(i,:)'*tfidfn(j,:)
This consumes much more time that a column oriented indexing:
tfidfnT = transpose(tfidfn);
...
score = tfidfnT(:, i) * tfidfnT(:, j)' .* jMat;
  3 Comments
Ryan
Ryan on 17 Apr 2013
I am yes and it actually did make a small difference.

Sign in to comment.

More Answers (2)

Edric Ellis
Edric Ellis on 17 Apr 2013
It looks as though each iteration of your PARFOR loop accesses every element of "tfidfn", so you cannot slice it. Even if "tfidfn" were dense, it's still "only" about 0.5GB, and so the transfer time for that to each worker is very likely to be completely insignificant compared to 24 hours for the complete computation.
  2 Comments
Ryan
Ryan on 17 Apr 2013
I saw a post by you some time ago that suggested a solution that looked like this
for idx = 1:n*n [k,j] = ind2sub([n n],idx); A(idx) = sum(sum(tfidfn(:,k)*tfidfn(:,j).*jMat));
%in my case end I think this might actually work fairly well if i can do the for looping for only the upper triangular values. Do you know how I could do that?

Sign in to comment.


Ryan
Ryan on 17 Apr 2013
since the output matrix is symmetric perhaps I can break up the indexing for loop i or would this not help at all? For a tfidfn matrix that is 253x500 the computational time is about 2.39 seconds

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!