How can we encourage more thoughtful feedback on FEX submissions?
Show older comments
Is there a way that either TMW or File Exchange (FEX) users can encourage more thoughtful feedback on FEX pages? I find that ratings without verbal feedback are not very helpful. I always wonder, "did you give this submission one or two stars because there are errors in the code; because it's computationally inefficient; because you didn't take the time to learn how to use it; because it was designed to solve a different problem than the problem you are trying to solve?" A five-star rating can be a helpful endorsement, but when I see a four-star rating without explanation I am left to wonder about the submission's weaknesses.
Perhaps the most confounding rating is a three-star rating without comment. What does it mean? On more than one occasion I've even noticed users who have created a Mathworks account simply to leave a three-star rating without comment, and have not left any other feedback on FEX. Clearly they're driven to express their lack of opinion, but there's no telling why.
It's counter-intuitive, but I think reducing the five-star rating system to a simple thumbs-up/thumbs-down rating could actually add nuance, particularly if a minimum word limit were required when giving a negative review. Amazon has recognized the value of a minimum word limit in reviews, and it's turned out to be quite helpful.
Without a redesign of the FEX rating system, is there a way to encourage more thoughtful feedback?
5 Comments
I'd agree with that. I often see poor ratings of submissions that are in fact quite good, having the impression that people give a "it was not plug and play" kind of rating (i.e. "I have research to accomplish without caring about how I obtain my results, so I have really no time for reading your doc").
Chad Greene
on 24 Jun 2014
John D'Errico
on 16 Dec 2014
Edited: John D'Errico
on 16 Dec 2014
Chad has a good comment. Old reviews may be inaccurate, if they reflect an old release of a tool. I suppose ratings could be made so that old reviews are deprecated in weight after a new release. And some poor reviews may be simply wrong, just as good reviews are also often wrong. A simple repair for this is MORE reviews.
Sean de Wolski
on 16 Dec 2014
Which almost makes the revision history more important than reviews so maybe it should be first.
I rarely use FEX, I don't know why really other than that I am very fussy and rarely find code that matches up to the standards I want (it may have higher standards by an unbiased metric, but just not be in a style I appreciate - e.g. I like OOP code, fully validated code, intuitive variable/function names, etc, etc), but I have downloaded a few nice small utility functions from there.
I don't imagine I would ever upload something though for the same reasons, I write code in a style that suits me and the people I work with and if I did upload something it would be in that "take it or leave it" vein.
All that said, my main point is I think the opposite of moving to a binary voting system would be more useful. Determine some broad categories - e.g. documentation, validation, ease of use, functionality,...(those are just off the top of my head, there would be many better ones) and people leaving a rating have to leave a score in each category. That avoids the interpretation and different wording problems that a free-for-all text box has.
It's a similar problem to Matlab Answers tags - people use words rather than a pre-defined list so there are hundreds of different interpretations of essentially the same thing.
If, on my rare sojourns to the FEX, I see a 1-star submission I do usually pass over it. If it were rated in categories though maybe I would decide actually on this occasion I don't care if it is badly documented because it has good functionality and no known bugs. I wouldn't be likely to sift through all the comments trying to locate this kind of information though.
Answers (5)
John D'Errico
on 16 Dec 2014
Edited: John D'Errico
on 16 Dec 2014
2 votes
I just saw this question, so my response is a bit late in coming...
The star rating system is sometimes a problem. For example, I tend to be rather demanding of myself before I'll post something. I want it to be as perfect as I can make it. So I also tend to be rather demanding of those I rate. If you get 5 stars from me, I was truly impressed with what I saw in a submission. At the same time, to get even 4 stars from me is a nice compliment, but it says that I saw something missing in the submission. I might give a submission that has serious (but repairable) flaws in it 2 or 3 stars, depending on the number of flaws. Usually then I offer advice on what needs repair, and often ideas on ways to do that repair. And I do try to follow up when I see a change made.
The problem is that others will not use the same rules as do I. The point is, unless we all use the same definition for star ratings, then they have less meaning.
So maybe ratings would be better based on a like/dislike system. At the same time, there are many factors in a rating. Is the help great, but the tool itself has only limited value? Does a tool do something good, but have terrible help? Is the tool a great idea, but is it poorly implemented?
Karma or reputation based systems also have value. To some extent that currently exists on Answers, where a higher reputation unlocks some useful privileges. An idea is for users with higher rep on the FEX to get somewhat more weight attached to their vote. The problem of course is it is always possible to game any such index.
3 Comments
Chad Greene
on 16 Dec 2014
Sean de Wolski
on 16 Dec 2014
Edited: Sean de Wolski
on 16 Dec 2014
I wouldn't worry about giving a file you liked a good rating ever. But then again, I try to be completely rating agnostic when picking files for pick of the week since "I didn't have the toolbox, know how to add to the path, mex a file, etc." is the source of many bad ratings.
Case in point from today's most recent files:
Chad Greene
on 16 Dec 2014
Get rid of the stars entirely.
These seem to mostly serve as a "I want to give some feedback, but I don't really want to put any effort into it" indicator, for both positive and negative criticism of a submission. Personally I tend to rate submissions over the whole scale from one to five stars, but the fewer stars I give the more I explain the specific reasons of why the submission does not meet the standards I would expect of a submission on FEX. However it seems that some people are not able to cope with any kind of feedback (it is to improve your submission!) and just want to get hurt by any kind of criticism of their precious baby: for as long as there are stars I suspect there will be star-hunters who prize them above all else, even above trying to actually write good code or improve their own MATLAB knowledge.
I recently had the experience of being rated by a user who registered but whose sole activity was to rate four of my submissions with one star and without comment.... ouch! This might not matter to a seasoned-submission, but to a new-ish user of FEX, this can really affect how their work is summarized on their user-page. While the stars might be useful over an extended period of time, for small population sizes their statistics is somewhat meaningless and heavily influenced by any outliers. The stars actually only give a good representation of the value of a submission that has many, many comments with ratings, i.e. submissions that have existed for a significant length of time or that somehow catch the attention of [insert search engine name here].
As has been commented several times already, I also find it much more useful to read what people have to say in their written comments. Not only do they address salient points related to the submission itself, written comments also usually give a clear indication of the users ability and experience with MATLAB, something the star-system fails to do entirely. In my experience, stars without written comments tend to be from much less experienced users, which does result in some submissions (that are clearly second-rate) being given five stars by users who don't really have a good technical reason for doing so. On the other hand, more experienced users understand the value of writing down some points for the author to consider...
So the question becomes "do the star-ratings represent useful information?", and I think the answer is clearly "not really". The easiest way to improve the feedback system would be to remove the stars entirely, and to rely solely on users making written comments, which can then also be judged on their usefulness, both by the submission's author and future browsers of FEX.
4 Comments
Sean de Wolski
on 18 Dec 2014
I agree and further think this is true for most surveys that require a quantitative answer.
On a scale of 1 to 10, how likely are you to get our x again?
Adam
on 18 Dec 2014
I think a combination of the two would be better, but with a more detailed breakdown of categories for stars (as I mentioned in a comment to the question rather than suggesting it as an answer).
A completely free text box for comments is way too open to people's different wording and interpratations. People who are going to take time to give useful feedback probably already do, people who currently just leave a 1 star rating and no feedback would probably either not bother (no bad thing!) or just add a pointless comment because they have to (e.g. like many people do with 'Tags' here on Matlab answers, just putting random rubbish in).
Chad Greene
on 18 Dec 2014
Very well said, Stephen. I really enjoyed reading your comment.
Your experience with one bad apple spoiling your reputation is not uncommon. A few years ago I uploaded a function that quickly accrued two bad ratings without explanation. I completely rewrote the function, but the rating has still not recovered. If I cared greatly about ratings or number of downloads, I know I would have been better off starting a new FEX page for the function.
It is so easy for an anonymous rater to permanently dog contributors' real-world reputations. Take the case that Sean mentioned. I have used ImGRAFT for my research. It would have taken me a year to write a similar program, and I wouldn't have thought of all the clever tricks that Aslak uses to make it work. Despite this, because of one anonymous rater, unacquainted users may be introduced to Aslak's work as being sub par. One doesn't even need to come to the Mathworks site to see it--a web search for ImGRAFT shows the one-star rating, implying that Aslak produces work of the lowest-possible quality. Again, I can attest to the fact that such an assumption is completely inaccurate. Yet, because of one anonymous rater, that's what the public perceives.
Still, some kind of easy-to-quantify endorsement system does seem valuable for sifting through search results on FEX. It does oversimplify great nuance, but without any stars or likes for guidance, users can feel a bit lost. This is particularly true for newbies who may not know how to interpret over-their-head discussions in the FEX comments.
Kelly Kearney
on 18 Dec 2014
Perhaps something as simple as not assigning a star rating to a submission, but instead showing a "not enough data" icon or similar, until a function has accrued a certain number of ratings, could help this early low rating issue. The individual star rating in the comments section could still be visible, but they wouldn't apply to all the other summary lists (and hence, Google) until a threshold is reached.
I had a similar experience, where a 2-star rating was the first one added to a submission of mine, because the function relied on a helper function that the user failed to add to his path. It was perhaps a legitimate complaint, but I immediately updated the description to be really clear about the need to get that function (and later removed the dependency). Still, almost 3 years passed before someone else added a new review. Once other ratings started to overshadow that 2 (average is now up to 4.8), the download rate (and review/comment rate) for that function noticeably increased, so people definitely use the stars to make decisions.
Only allow non-anonymous registered users.
A little more drastic perhaps than you were expecting, but I believe it has a lot going for it.
There is a clear tendency to hide behind Avatars and anonymity on the internet, which has been shown to increase peoples tendency to behave antisocially. The very specialized nature of MATLAB Central (Answers, FEX, etc) means that we avoid most of the flame-wars and trolling that havoc many sites, but I still get the feeling that some users are not quite playing by the rules, and are attempting to game the system. Too many one-star ratings by users who simply disappear again, or "coincidental" five-star ratings all on the same day by one user for all of the submissions of another user. What do they all have in common?: short, anonymous user names.
However most of the users who comment and give useful advice seem to be real people, with names to reflect this (with a few exceptions). Is this a coincidence? During the action of commenting or giving feedback, perhaps being in the position of knowing "my words and thoughts can be used to judge me too" is not such a bad thing. It promotes a little self-reflection and consideration for the tone, content and quality of the feedback. And this really seems to be aligned with the aim of the original question "How can we encourage more thoughtful feedback on FEX submissions?"
James Tursa
on 24 Jun 2014
Edited: James Tursa
on 24 Jun 2014
1 vote
Maybe have a voting system in place for the FEX feedback similar to Answers. I.e., you get FEX feedback points if you provide some text in your rating AND other users vote for your response. Users could accumulate FEX feedback points, or maybe combine them in with the Answers points to get an overall score of some type. Not sure what privileges would go with the FEX feedback points.
4 Comments
Chad Greene
on 24 Jun 2014
John D'Errico
on 17 Dec 2014
Your Answers points could be more than just for Answers rep. They could contribute towards an overall site reputation. So answering questions, posting submissions on the FEX that have many downloads, etc., all could factor in.
Chad Greene
on 18 Dec 2014
Stephen23
on 20 Dec 2014
Using feedback to weight the ratings may ensure that outliers don't get too much prominence, if there are sufficient users prepared to login and take part. This seems to work fairly well for the likes of Amazon: they have huge numbers of site visitors who are prepared to rate and interact with the system in this way. Although it obviously requires a much more complicated rating system, both to implement and to use, I can see that this might lead to more "reliable" submission ratings, in the sense that the effect of outliers might be reduced.
Yet I would argue that rating-feedback is still not useful enough information to justify implementing it.
Most of the suggestions in this thread for improvements to the star rating system are really just to internalize some complicated rules into the ratings, based on (so far in this thread) user reputation, feedback on the ratings, some threshold for a minimum number of ratings, and providing incentives to use the system.
Yet at the end we still just have some stars in a row... Most of the commenters here have stated that when browsing FEX submissions, the written feedback is of far more value to them than the star ratings. So we can improve the ratings to be more "reliable", but the question still remains: exactly what information does this rating contain that would be more useful than written feedback?
Jan
on 16 Dec 2014
1 vote
When a submission in the FileExchange looks interesting to me, I read the comments. The star ratings do not have a remarkable effect to my opinion, when they are not accompanied by a meaningful text. Therefore for me a bare 3-star rating is no problem at the point of downloading.
In opposite to this, the stars matter, when a programmer catches my interest. The number of submissions is not really trustworthy, because some users submit a pile of useless code snippets. The number of downloads is only partially meaningful, because some people seem to run a download bot to get a high ranking in the top 100. But the star ratings are not flawed, as long as one considers the outliers of unmotivated 1 star ratings.
If a user has more than 20 ratings and an average of >4.5 or more than 100 ratings and >4.8, I take the time to check the other submissions and search for gems.
In consequence, for me it is more useful to increase the number of comments with stars than to reduce the number of stars without comments.
1 Comment
Chad Greene
on 17 Dec 2014
Edited: Chad Greene
on 18 Dec 2014
Categories
Find more on Historical Contests in Help Center and File Exchange
on 24 Jun 2014
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
