Repeat Workers Angry About Rejection HELP

[EndOfAnEar]

Hi all,

I am doing a large study and it involves me re-publishing a batch of hits for the same study and then I have to prevent workers from taking it over and over again. My institution does not allow for automatic filters of worker IDs so I provide a list of Worker IDs at the beginning of my study to let people see if they have already taken it to prevent later rejection for repeat HITs. I have had complaints from people saying they were not on this list (they were though) and that their MTurk records show no history of them having taken the HIT. My MTurk records show they have taken the HIT in a previous batch and I tell them the date they participated but they still say I'm wrong. I have contacted MTurk to see if something is up but I am not sure what to do, they are threatening to contact my institution about rejecting them for other reasons (e.g., their responses, their demographic, etc.). I feel bad but am I in the wrong?

Any help or feedback would be greatly appreciated. I am new to MTurk but I have done a lot of reading and research to try and be a "good" requester. I feel like I'm failing.

Thanks,

Ear

 

[TSolo315]

Hi all,

I am doing a large study and it involves me re-publishing a batch of hits for the same study and then I have to prevent workers from taking it over and over again. My institution does not allow for automatic filters of worker IDs so I provide a list of Worker IDs at the beginning of my study to let people see if they have already taken it to prevent later rejection for repeat HITs. I have had complaints from people saying they were not on this list (they were though) and that their MTurk records show no history of them having taken the HIT. My MTurk records show they have taken the HIT in a previous batch and I tell them the date they participated but they still say I'm wrong. I have contacted MTurk to see if something is up but I am not sure what to do, they are threatening to contact my institution about rejecting them for other reasons (e.g., their responses, their demographic, etc.). I feel bad but am I in the wrong?

Any help or feedback would be greatly appreciated. I am new to MTurk but I have done a lot of reading and research to try and be a "good" requester. I feel like I'm failing.

Thanks,

Ear

Well, if the list was easily visible in the HIT instructions (ideally within the HIT frame itself) and did indeed include the rejected worker's IDs, then I believe the rejections would be warranted.

What is the reasoning behind not allowing the automatic filtering of IDs? Seems like a common sense way to avoid these headaches with no real downsides.

 

[TQueen]

No doubt, turkers usually get upset about rejections. I've been working for almost two years, and they're just part of the game sometimes. I know if I had any questions about a rejection, whether it was merited or reversed, good communication with the requester usually will ease the salty feelings.

Did you provide a link to some kind of Excel document to show their ID's? If this is how you did this, I would suggest making that very clear. Bold, red color, all caps something telling them CHECK YOUR ID HERE and explaining they would not be able to retake and won't be compensated for their time if they find their ID. Like @TSolo315 said, the best thing to do would be put them all within frame, giving the turker a quick chance to ctrl+f and see if they need to bypass your survey.

If you did something like this, giving them a clear way to check, then yes, your rejections are merited. If you are getting too many rejections from this though, you might need to revisit your methods. One or two turkers ignoring this might happen, but ~20-30 will wreck your TO and could be a result of unclear instructions or a technical error.

Don't let it get you down. Everyone has to learn how to maneuver Mturk. I'd also ask your institution why they don't allow the automatic filters.

 

[jan]

Hi all,

I am doing a large study and it involves me re-publishing a batch of hits for the same study and then I have to prevent workers from taking it over and over again. My institution does not allow for automatic filters of worker IDs so I provide a list of Worker IDs at the beginning of my study to let people see if they have already taken it to prevent later rejection for repeat HITs. I have had complaints from people saying they were not on this list (they were though) and that their MTurk records show no history of them having taken the HIT. My MTurk records show they have taken the HIT in a previous batch and I tell them the date they participated but they still say I'm wrong. I have contacted MTurk to see if something is up but I am not sure what to do, they are threatening to contact my institution about rejecting them for other reasons (e.g., their responses, their demographic, etc.). I feel bad but am I in the wrong?

Any help or feedback would be greatly appreciated. I am new to MTurk but I have done a lot of reading and research to try and be a "good" requester. I feel like I'm failing.

Thanks,

Ear

I think you should make a decision based on the percentage of workers who were able to see your instructions successfully. If 99% of workers completed it fine, checked their ID, and did not repeat - then you know for sure you were clear. If 20% of workers (for instance) did NOT see your instructions - then perhaps there was a problem on your end.

I don't know where you should draw the line (percentage wise), but there is a line - somewhere in all of this. I'd come to a conclusion based on that. If you're worried about bad reviews on turkopticon - you can join and make a comment about how you came to your decision. People will read that, too, and if there are a few negative reviews - your input could mitigate those complaints.

Of course, at the end of the day you can't please everyone and you might need to submit some papers to your professor/IRB. Shoot, I might screenshot their ID on your checklist and send it to them and the IRB (putting them on the CC line) - if I felt that strongly about it. I'd make VERY sure I got support first .

Don't forget to weigh out the financial and time/struggle costs. The truth is, some things in life are not worth haggling over. If the survey was, for instance, .50 and you are out a few dollars (and have to toss that data), well, maybe so be it.

If you did your study in person, you'd still have missing data.......that's kind of the nature of it. Only you can decide where to draw the line. I gave out several hundred surveys (in person!) several years ago - it took hours and hours. Most of the data came in and was great. A few (really a scant few) - well, they 'christmas-treed it.' I was annoyed (it was voluntary - why waste our time....sigh.......) - but toss it we did. Our n was high enough and we had oversampled in order to account for a variety of possible problems.

Good luck, whatever you decide .

 

[EndOfAnEar]

Thank you so much everyone. This has really great. I feel supported. I definitely have 95% or more of people getting the instructions read and finding their ID. They are able to search for the ID immediately after the consent form and they have to click that their ID is not on the list to proceed to the study. I even put on there instructions how to Ctrl / Command - F to find it. I have alerted my IRB just in case, but I will definitely be following up with the institution about the automatic filters. It's something about it being external sourcing. Maybe they'll be able to fix this... Thanks again everyone!