Copycat "News" on "Mturk Bot Panic"

[Achilles2357]

In the past week there have been multiple "news" reports that psych researchers are in a panic over the possibility that "bots" are doing psych surveys on mturk. They all basically repeat the same stuff, and you can find them on Google News by searching for mturk. So far as I can tell, there is basically no original reporting involved and they all seem to derive from some observations by the guy (or girl, I guess from the name) who posted this (https://www.reddit.com/r/mturk/comments/95yy96/data_contamination/) on reddit, and apparently much the same stuff on Facebook.
tl;dr: Researchers need to PUT IN THE WORK to block bots.
tl;dr: Researchers need to PUT IN THE WORK to eliminate bad data.
Note to researchers: Jeff Bezos doesn't care about you any more than he cares about us. Accept it and life will get better.

 

[TSolo315]

Note to researchers: Jeff Bezos doesn't care about you any more than he cares about us. Accept it and life will get better.

IMO Mturk goes out of their way to exploit researchers who don't know better with how they have set up their fee structure.

I'm honestly surprised survey bots are a problem with how many people seem to have difficulty even setting up an mturk account.

 

[Achilles2357]

IMO Mturk goes out of their way to exploit researchers who don't know better with how they have set up their fee structure.

I'm honestly surprised survey bots are a problem with how many people seem to have difficulty even setting up an mturk account.

I suppose some originally legitimate accounts get "sold". I don't see how the effort to pull this off will yield any major profits for bad actors, assuming researchers are using reasonable controls. There is absolutely no way that anyone has a bot capable of maintaining a 99% doing only surveys. I suppose I could be wrong, but I simply do not think it so. If it is true, it is a sign of utter cluelessness on the part of researchers, since it would not be difficult at all to deal with this, and significantly increase bot identification.
Maybe people trying surveys with bots are just testing their bots for other uses.

 

[TSolo315]

I suppose some originally legitimate accounts get "sold". I don't see how the effort to pull this off will yield any major profits for bad actors, assuming researchers are using reasonable controls. There is absolutely no way that anyone has a bot capable of maintaining a 99% doing only surveys. I suppose I could be wrong, but I simply do not think it so. If it is true, it is a sign of utter cluelessness on the part of researchers, since it would not be difficult at all to deal with this, and significantly increase bot identification.
Maybe people trying surveys with bots are just testing their bots for other uses.

Amazon will also ban those accounts at some point. Buying accounts only to scam surveys before they get banned doesn't seem like it would be profitable. But maybe I'm overestimating how competent Amazon is in this regard.

And bots like this are single-purpose for the most part. A bot created to scam surveys wouldn't be good for much more than scamming surveys. I'm skeptical about the testing angle.

 

[Achilles2357]

Amazon will also ban those accounts at some point. Buying accounts only to scam surveys before they get banned doesn't seem like it would be profitable. But maybe I'm overestimating how competent Amazon is in this regard.

And bots like this are single-purpose for the most part. A bot created to scam surveys wouldn't be good for much more than scamming surveys. I'm skeptical about the testing angle.

I suppose that if there are lot of surveys that don't set up their quals reasonably, and don't have adequate controls that will ensure that a good portion of bad work will be both detected and rejected, and if the requesters fail to report suspicious accounts to amazon, then maybe a scammer could temporarily "earn" money at a better rate than the typical mturker. That might be all it takes to get someone trying this.
As for "testing", I suppose I was thinking of someone simply testing out their ability to make money via illicit means.

I really think this whole issue can be avoided by ignoring the issue of "bots" per se and focusing on the need to exclude the very bottom tier of mturkers who give nonsense answers. (Bots will be excluded as well.) I really suspect that the research community using online surveys has not given this the actual intellectual attention it deserves. (But maybe that is changing since I did a survey the other day that seemed to be nothing but a series of rather different attention check type stuff, so maybe they are they are actually trying to take this seriously.)

I do think bots matter, but not for surveys, so long as you exclude "bad workers". Mturk itself likely needs active strategies for bot detection, but I think academic researchers simply need to focus on eliminating completely unengaged workers. I think part of the problem is that the IRB culture actively eschews the idea that researchers are hiring "workers", whereas mturk is unequivocally a work platform.

 

[XWarrior]

Basing things on the "GPS coordinates" equals bad input to make the grand assumption about bot problems. Those are notoriously inaccurate, and depend on multiple factors that can alter the data. The major type of bot deployed would answer the questions on a page almost instantly, and timer pause after, and repeat. That data can be found in the big Q if there are concerns. There are other ways to suspect/assume bot use and ensure data accuracy/integrity for requestors, GPS is not really a reliable one.