I would love to see some examples.
Well, one off the top of my head is the McDermott experiment. Now, that is a bad example because it is a .25 hit that takes roughly a half hour. So there is a good chance that was partially rated badly because of those workers who have not even attempted it and see that it is paying only 25 cents, yet takes about a half hour to finish. Even though it pays its lump some in bonus.
Unfortunately I haven't written down every instance of this I have seen. But I can tell you I'm not lying and have in fact ran across this on several occasions doing hits myself.
Are you talking about the return warnings? Users cannot review HITs on TV without submitting them, and generally McDermott's avgs are pretty close w/ the bonuses since most users are good about inputting them. Return reviews are incredibly heavily skewed towards newbies who straight up don't know better, with the exception being writing return warnings which are 2x as likely to be from experienced workers who don't find value in a lot of writing prompt type HITs/surveys. That's why TVJS has so many options to filter/turn off certain return warnings because depending on where you are in your Turk journey some will be more/less useful to any individual.
As far as batches though anytime you see this feel free to throw me a DM and I'll investigate (or simply report it on TV). We get thousands of data points every day, and we have some systems that monitor known problems so I can get involved but it isn't all inclusive and there are always new things coming up that I haven't even thought of as being a problem lol.
I'll always do my best to fix the issue, but FWIW I can't follow MTC anymore since the forum switch so please DM me so it triggers an email alert (or just reach out over email, I monitor
hello@turkerview.com all day).
Oh ok. Well then. I don't know. Maybe there is part screwery and partly because of people not starting hits right away? My gut tells me lots of screwery though.
I just have major trust issues I guess. Can you blame me?
The actual truth is most people just aren't good at online work. People turking from 15 year old laptops on McDonalds wifi, users who are dealing with a fussy baby while filling out a survey, folks who type 40wpm trying to do transcription, really the instances are endless and
most (at least when it comes to my investigations on TV) aren't malicious, they're just oddities of situations. The one and only metric that anyone should care about on batches is # of rejections. Surveys, generally, are pretty standardized and as long as you know where you fall in TV's completion time groups you'll find the breakdowns/%s are generally pretty close. Biggest exception being surveys that have multiple completion paths [ie you get writing, someone else doesn't, etc]
I remove at least 1 reviewer/week from the aggregate scores of requesters for things like that to help alleviate the issues with dragging down hourlies when they shouldn't be because they're unrealistically slow. It's rarer for the opposite end, although we do occasionally receive complaints about
some users lol.
Forreal though, the vast majority of folks are good natured & just trying to help as they're able. MTurk draws such a gigantic demographic pool and there is so much turnover there is probably always going to be some friction between various groups/types of workers. Same goes for
most requesters.. The
better razor & all that.
