I wanted to start a thread where people could ask questions about this HIT (or complain, or talk about how awesome this HIT is, whatever). It also goes by "Behavioral Study of Cognitive Function". I'm a researcher involved in the project.
Below I've put some comments I've made before in regard to previous comments in the Daily HIT threads as a reference. This will hopefully make it easier for new people thinking about this HIT to find information, especially as I roll out new HITs in the future.
Check-In
We have implemented a check-in system where you must complete a separate HIT (linked from the $60 HIT) to proceed. This separate HIT takes no time, and pays nothing. Its only point is to make sure we can track you so that we can communicate with you if needed and pay you at the end should you fail to complete the entire battery.
Communication
This HIT has had some technical issues in the original roll out. While I have dealt with most of them (thanks in part to communication from this site and the turkers working), but others may present themselves. I will always update the Ad at the beginning of the HIT with updates so please read that if you are participating.
Time Estimate
This HIT does actually take 10 hours. This seems to be the main dissapointment among turkers, where people think I am only saying it takes 10 hours, but you can actually do it in 6 hours (or something). This is not the case. It will take around 10 hours.
How do we know that? The 10 hour estimate was originally based on our pilot work, but we have changed the list of tasks a bit so things could change. When last I looked, the people who had run through the tasks so far (20 or so people having gotten to each task), if I average the amount of time taken on each task and sum them I get 5.6 hours. This doesn't include surveys (there are ~20 surveys). I don't have info on how long each survey takes, but if they each take an average of 10 minutes, we'd still only be at 9 hours, and I doubt people spend 10 minutes on average across all surveys (some surveys have 1 question, some have 40).
The time estimate I gave above does not include instruction time for each task. I'd imagine that each task (not surveys) should take at least 2 minutes of instruction time. If that's true, and there are 35 tasks I believe, that's around another hour and ten minutes. So let's guess 7 hours plus survey time.
It is difficult to perfectly determine how long such a long HIT takes, but we believe we've been slightly conservative with our 10 hour estimate. There will be a bit of wiggle room, but most tasks take a set amount of time regardless of how you take them.
Payment and Submission (and bonuses)
I saw a post about finishing the 63 tasks and the HIT suddenly moving to a blank screen and removed from your queue. We didn't handle the submission process the best, so I understand that this can be a bit confusing. Regardless, what's going on is that the HIT is submitted, and most of the time is automatically verified and credited. Once in a while our code screws up (there are some timing issues coordinating with Mturk's site) and I have to go in afterwards to manually approve. Either way, don't worry! Your work is almost certainly submitted.
And I'll just mention this here - the HIT mentions bonuses. The bonuses will take a little bit for us to get to, but we will get to them! I know immediate payment is preferable, but this part takes a little more work. Bonuses normally range from $5-$15, depending on performance. I'll code up the "bonus calculator" and get them out as soon as possible - hopefully no later than a week after the end date for this HIT (next Thursday), but hopefully much earlier!
Will we have more HITs?
Yes (as of 7/26). We will have 4 more rounds of HITs.
Finally, I'd like to say how appreciate I've been of this community (and the broader community like people who post on turkopticon). It is refreshing to see people care about their work and care about their community such that they keep each other informed.
Below I've put some comments I've made before in regard to previous comments in the Daily HIT threads as a reference. This will hopefully make it easier for new people thinking about this HIT to find information, especially as I roll out new HITs in the future.
Check-In
We have implemented a check-in system where you must complete a separate HIT (linked from the $60 HIT) to proceed. This separate HIT takes no time, and pays nothing. Its only point is to make sure we can track you so that we can communicate with you if needed and pay you at the end should you fail to complete the entire battery.
Communication
This HIT has had some technical issues in the original roll out. While I have dealt with most of them (thanks in part to communication from this site and the turkers working), but others may present themselves. I will always update the Ad at the beginning of the HIT with updates so please read that if you are participating.
Time Estimate
This HIT does actually take 10 hours. This seems to be the main dissapointment among turkers, where people think I am only saying it takes 10 hours, but you can actually do it in 6 hours (or something). This is not the case. It will take around 10 hours.
How do we know that? The 10 hour estimate was originally based on our pilot work, but we have changed the list of tasks a bit so things could change. When last I looked, the people who had run through the tasks so far (20 or so people having gotten to each task), if I average the amount of time taken on each task and sum them I get 5.6 hours. This doesn't include surveys (there are ~20 surveys). I don't have info on how long each survey takes, but if they each take an average of 10 minutes, we'd still only be at 9 hours, and I doubt people spend 10 minutes on average across all surveys (some surveys have 1 question, some have 40).
The time estimate I gave above does not include instruction time for each task. I'd imagine that each task (not surveys) should take at least 2 minutes of instruction time. If that's true, and there are 35 tasks I believe, that's around another hour and ten minutes. So let's guess 7 hours plus survey time.
It is difficult to perfectly determine how long such a long HIT takes, but we believe we've been slightly conservative with our 10 hour estimate. There will be a bit of wiggle room, but most tasks take a set amount of time regardless of how you take them.
Payment and Submission (and bonuses)
I saw a post about finishing the 63 tasks and the HIT suddenly moving to a blank screen and removed from your queue. We didn't handle the submission process the best, so I understand that this can be a bit confusing. Regardless, what's going on is that the HIT is submitted, and most of the time is automatically verified and credited. Once in a while our code screws up (there are some timing issues coordinating with Mturk's site) and I have to go in afterwards to manually approve. Either way, don't worry! Your work is almost certainly submitted.
And I'll just mention this here - the HIT mentions bonuses. The bonuses will take a little bit for us to get to, but we will get to them! I know immediate payment is preferable, but this part takes a little more work. Bonuses normally range from $5-$15, depending on performance. I'll code up the "bonus calculator" and get them out as soon as possible - hopefully no later than a week after the end date for this HIT (next Thursday), but hopefully much earlier!
Will we have more HITs?
Yes (as of 7/26). We will have 4 more rounds of HITs.
Finally, I'd like to say how appreciate I've been of this community (and the broader community like people who post on turkopticon). It is refreshing to see people care about their work and care about their community such that they keep each other informed.
Last edited:
