Topic: Rating system needs a re think

Hi,
     generally I like the way Chumby works but I am not very keen on the rating system - that is where users rate a widget (1 - 5 stars). The key problem is that there is no meaning of rating. It could mean that the widget is clever, has / has not bugs, meets a need for the user.

Ok so I am a bit annoyed - just put up a widget that i spent a few weeks on and some smuck gives it a 1. I ask my self why - does it have bugs - no, does it meet there needs - maybe not so don't use it. At least if someone gives me a 1 they should tell me why.

Sorry i am a bit disgruntled when people get critical about free content. Anyhow back in to positive mode - could this be reviewed - maybe we need a reason for a rating etc or define a criteria for users to categories against.

thanks

Nigel

smile=)=)

Re: Rating system needs a re think

You make a really good point.  I've always had issues with 5-point ranking systems because I'm convinced that most users only vote the extremes (5 if they like it, 1 if not, and rarely anything in between).  And if the voter doesn't leave any comments, then there's no context.  It might be a good idea to break the ratings down and have a 5-star system for different categories (check out the product ratings on BestBuy.com for an example), but that may be too complicated for some users.

If it's any consolation, I pay very little attention to ratings.  I check the description, read comments if available, and try it out myself if the widget interests me.

Re: Rating system needs a re think

While it sometimes rankles me when my widgets get 1 point ratings, it turns out the collective wisdom of the crowd tends to dilute that and the overall consensus isn't too bad.  I would say that objectively "good" stuff seems to rise to the top, while widgets of obvious marginal quality seem to get low ratings.

Looking at the patterns, some people seem to rate on the basis of some vague notion of "usefulness", while others rate on the basis of technical execution, and others seem to rate on the basis of "novelty".  Some widgets get wildly disparate ratings, exposing a sort of "love/hate" that some people have for certain widgets - "Screen Clean", for instance, is generally highly regarded, but some people absolutely hate it - do they hate dogs, or is it just that the widget itself is profoundly useless, or that's it's technically trivial?

I've taken a look at the ratings table, and most people do tend to use the full range - they don't stick to just "1" or "5".  The only thing that's universal is that widget authors give their own widgets "5"s.

We have talked about the rating system quite a bit in house - I guess the problem is that it's hard to get consensus what the various rating categories should be, and in some cases, some would be quite controversial.  I''ve also noticed that in most of the multi-axis rating systems out there, they *always* create a merged rating that they use for sorting.  That would seem to dilute any value that separate ratings might have.

Please keep giving us your thoughts on this issue.

Re: Rating system needs a re think

MSlafka wrote:

You make a really good point.  I've always had issues with 5-point ranking systems because I'm convinced that most users only vote the extremes (5 if they like it, 1 if not, and rarely anything in between).  And if the voter doesn't leave any comments, then there's no context.  It might be a good idea to break the ratings down and have a 5-star system for different categories (check out the product ratings on BestBuy.com for an example), but that may be too complicated for some users.

If it's any consolation, I pay very little attention to ratings.  I check the description, read comments if available, and try it out myself if the widget interests me.

MSlafka, Duane,
     thanks for the feedback

I think the most relevant point out of this is that the rating seems not top be used as a decided in selecting a widget. In hindsight thats my experience - rather I look at all new ones and make a choice.

Feeling better now thanks - I suspect the rating system will evolve over time as necessary and so happy to bury this for now.

cheers

Nigel


smile  smile  smile

Re: Rating system needs a re think

I really, really hope my first "ratings rant" is buried deep-deep-deep in the system roll.  I got schooled pretty good by Duane.  He made me the man I am today. tongue

Your widget is really cool BTW.  Looking forward to more!

Cheers.

Re: Rating system needs a re think

Another problem is the ratings of widgets in different languages.  I would probably rate a Japanese News Widget a 1 because I have no use for it, as I don't speak Japanese.  I think they should really separate widgets into languages or countries.

Re: Rating system needs a re think

tyler124 wrote:

Another problem is the ratings of widgets in different languages.  I would probably rate a Japanese News Widget a 1 because I have no use for it, as I don't speak Japanese.  I think they should really separate widgets into languages or countries.

Yes, we've noticed that - English speakers giving Japanese widgets low ratings simply because they're Japanese.  It's probably best to just ignore a widget that's obviously designed for another country.

We considered putting users from other countries on different servers with entirely parallel sets of widgets, however, it quickly became apparent that many widgets are "universal" - many of the widgets that are popular in the original catalog are also popular in Japan, plus US users might miss a really cool widget.  Rather than segregate at the server level, we hoped that putting up a "Japan" category would be enough for users that aren't interested in Japanese widgets to avoid them.

Re: Rating system needs a re think

cbreeze wrote:

I really, really hope my first "ratings rant" is buried deep-deep-deep in the system roll.  I got schooled pretty good by Duane.  He made me the man I am today. tongue

Your widget is really cool BTW.  Looking forward to more!

Cheers.

Thanks - feeling better now - hope this stimulates some debate. As a side comment Streams widget is very very good - hope you can eventully lever it into the control panel.

cheers

Nigel

  smile  smile  smile

Re: Rating system needs a re think

tyler124 wrote:

Another problem is the ratings of widgets in different languages.  I would probably rate a Japanese News Widget a 1 because I have no use for it, as I don't speak Japanese.  I think they should really separate widgets into languages or countries.

I guess I wouldn't feel right putting a rating on something I didn't understand. smile

I personally like "seeing" the other widgets in there,  it's interesting to see what users in other locales are doing.  But I can certainly see your point.

Cheers.