Effective immediately, we have changed our algorithm for rewarding voters of successful products during eval.
The previous algorithm rewarded influence based on the proportion of votes cast overall to the proportion of votes cast for products that went on to be marked Under Consideration (with larger portions given to ideas that were ultimately selected).
There were two main problems with the former system:
1. Clarity – Due to the complexity of the old system, it was hard for users to tell how much influence they stood to gain from voting on an idea at any given time. It was also a challenge to understand for users unfamiliar with the system.
2. Influence amounts – Many people are casting extremely large numbers of votes (sometimes upwards of 150 per day). Because they cast such a wide net, they often are awarded for ideas that go under consideration. However, because they are penalized for the number of votes they cast overall, their experience percentages tend to be minute to the point of absurdity. This is a bad experience for everyone involved, and we want to encourage people to vote for ideas they like, not to simply hedge their bets by voting for as many ideas as possible.
To that end, we are changing the influence algorithm for voting, effective immediately. As before, at the end of every eval round, two groups will be awarded influence for each winning product: 1) people that voted for that product, 2) people that voted for products that were under consideration, but were ultimately passed on during the course of that round. However, instead of diluting influence on a per user basis, each voter will be given the same amount of voting influence, depending on which of those two groups they belonged to. The catch is that users will be limited to casting 15 votes per day. The vote count will reset each night at Midnight EST. Votes cast prior to this change will stand as is.
The 15 vote limit is a measure to prevent over-voting (voting on as many possible ideas in order to have a better chance at getting influence). As such, there will be no rollovers on unused votes from day to day. We realize that not everyone can log on to Quirky every day, but 15 votes per day is meant as an absolute maximum. We are not trying to encourage people to use every possible vote; we are trying to encourage people to vote for the ideas they like the most.
As with everything at Quirky, this is an ongoing experiment. We may change things in the future based on the results. Possible future changes may include adjusting the amount of daily votes up or down, awarding additional daily votes to certain groups based on merit, and eliminating influence for votes on non-winners.
As always, thanks for sticking with us as we strive to make invention more accessible.
Nathan
(edit: the system resets at Midnight EST, not GMT)
20 Comments
Elizabeth Wingfield | 02/06/2012 3:12 PM
Thanks, Nathan, I will give it a go. Wish it were cumulative during the week, as I like to rank things and weigh it out, but this is a start. It is going to be pretty important for us to have some sort of date stamp on the idea when we vote, so we can keep track in our sorts. Can you do that, or a date sort for us so we don’t get confused? Thanks to the team for getting this done, a beer reward is in order
UNO | 02/06/2012 3:43 PM
This is better than before as long as you considered all angles involved and logically applied the proper multipliers etc. I’d say thanks now but that will only come if I see any change. If not I’ll go straight to my phase 2.
Clinton | 02/06/2012 4:56 PM
Quirky is a bubble up, Digg style system.
Such systems do not work well unless users are voting for the ideas they like…
…which is very different from voting for the ideas they like the most.
So, it’s puzzling to see Nathan use the phrase, “we are trying to encourage people to vote for the ideas they like the most.”
Claudette | 02/06/2012 5:03 PM
Thanks, since I am new to Quirky & haven’t learned the ropes, I do appreciate the openness for change.
Chad Maxey | 02/06/2012 5:44 PM
Thanks
Clinton | 02/06/2012 6:12 PM
Ouch. Maybe the problem of diminishing influence % is not solved at all.
This issue was raised because people felt their % was being cut.
• The community conjectured that it was because of the overvote penalty.
• Nathan agreed.
• So, today Quirky introduces an eval system that has a set limit of 15 votes per day.
• Now that we know that, we can keep from overvoting.
Problem solved. No more cut %.
Except, what if that was never the problem?
What if the problem is a glitch in the system?
What if many people are having their voting % arbitrarily cut like some are having it arbitrarily raised? (If several get 2.5% extra, a lot of somebodys together lose 2.5% too.)
Right now it doesn’t appear to me that this 15 vote limit fixes the issue…
…so we may continue to see a few voters get 1000x more than most until the staff can find and fix the glitch.
Clinton | 02/06/2012 6:34 PM
Update: I think Nathan found the cause of the diminishing voting %. The old iPhone app was still showing a rating function for the submissions, and those who used it to rate were still getting tons of extra influence…out of the rest of our’s influence.
nathan | 02/06/2012 6:40 PM
Just to be clear, the rating influence does not in any way cut into the voting influence, and is only awarded if ratings are present.
Full explanation (quoted from Get Satisfaction)
“Looks like Awesemo went and rated all of the ideas that were under consideration using the iPhone app.
If there are ratings, the influence for the ideator goes from 37.5% (.75 x .5) to 35% (.70 x .5), and the raters get that extra 2.5% (.05 x .5).
Since Awesemo didn’t do anything wrong, instead of revoking the influence, what I am going to do in this situation is increase the overall worth of the ideation rounds for the products in question to 55%. What this means is that the ideators will get their influence back and more (.70 x .55 = 38.5%), and everyone else will get a bump in influence on their votes, considering that not everyone has access to the iPhone app.
In the future, raters using the iPhone will not be awarded influence. That functionality will be removed in the next release.”
We left in the rating influence because the rating functionality was still in the iPhone app; we did not foresee a single person being the benefactor in the way that Awesemo was in this round, but since it worked out that way, we have changed our policy moving forward.
(edited: added quote)
Clinton | 02/06/2012 7:39 PM
Well, I guess that means Q discovered why a few were getting more influence; but perhaps not all, nor why many are feeling like they get much less.
stazy | 02/06/2012 8:02 PM
Glad to see some changes made. I would like to see the 15 roll over for the week as some people don’t get a chance to be on here daily like I can.
I think Clinton is on to something. I never vote for 150 ideas a day (only around 10 if that) and I still get .0001%. That doesn’t make sense. I think there could be a flaw in the system as to how and who gets influence. I’ve been questioning that all along as well as others.
Ways to get the ‘carrot’ need to be clear cut so we can then use checks and balances to make sure the system is working.
mark wisdom | 02/06/2012 9:17 PM
It was nice to be able to vote freely again today:) 15 seemed like enough although a weekly limit would be nice instead of daily. Overall good improvement though.
mark wisdom | 02/06/2012 9:21 PM
Stazy I know I was down to voting for 5-15 a week and still in the .004 to .0007 range hope this helps:)
Brandon Craven | 02/06/2012 9:56 PM
Hmm… sounds like a mess.
Jesse Freeman | 02/06/2012 10:13 PM
Thanks for the change, I think its for the better! Good luck to everyone!
Patricia Bender | 02/07/2012 7:41 AM
I like the idea.
Vicky Laursen | 02/07/2012 7:49 AM
Good Job Team Nathan!
nstlkr | 02/08/2012 10:05 AM
What strikes me as strange here, as the statement they don’t want to encourage us to vote 15 times, but simply limiting us to that as a max. If I’m understanding the explanation of rewards, we now have every incentive to vote 15 times every day. Now influence for voting is awarded solely on selecting products that advance.
As there is no penalty for voting for 15 times a day, and you only get influence when your associated to products that advance (at leat to UC), it’s in your best interest to be associated with as many products as possible. Which in this case is 15 per day. You penalizing yourself if you don’t vote the max times per day.
Stalias | 02/08/2012 10:15 AM
@ Stazy — If I understand correctly (and this is all getting beyond me, I’ll admit), it wasn’t just a question of what you personally were voting for, but how everyone (aggregated) was voting: With so many people casting so many votes, the influence awarded for each individual vote was diluted into nothingness. More pieces of finite pie = smaller pieces.
AliKira | 02/09/2012 4:12 PM
I actually tentatively like this change.
Jesse Freeman | 02/13/2012 1:02 AM
If we do not get a penalty for voting 15 times a day then we can vote up to the top 105 ideas a week which is easy to catch all of the ideas that enter the “UC” category. The problem is that you will have 20,000 to 50,000 people doing this and thus everyone will get 1/1000 of a percent. When Quirky hits 1 million users then what? I still believe one day it will have to a max of influence winners like around 25 winners, if not then in the future we will get 1 cent for every $1,000,000.00 sold and whats the point of that? A candy bar once a year!