Rep Present Hummingbird Watch

[HummingbirdWatch]

Hello MTurk Crowd community,

Our company is interested in trialing with MTurk. We are hoping to have a trial HIT released later this week.

Here's how it works: We would upload images collected by remote cameras, and the workers would look at the images and click yes/no if the image contained a specific item (similar to validating search result images). Speed and accuracy are paramount. These images would have to be processed immediately after being uploaded, and accurately.

We understand that if we want our images to be processed quickly and accurately, we need to maintain a reputation of honesty and fairness with the worker community.

Any advice on making our time with MTurk successful would be appreciated!

Thanks,

HW

 

[MerryLou]

Making an account here and seeking out info from 'the other side' is a great way to start!

Setting up a qualification test or a small batch of HITs as a kind of preliminary round for workers to earn a qualification from you is a good way to make sure you'll get consistently accurate results from workers.

Looking forward to trying out your HITs!

 

[HummingbirdWatch]

Making an account here and seeking out info from 'the other side' is a great way to start!

Setting up a qualification test or a small batch of HITs as a kind of preliminary round for workers to earn a qualification from you is a good way to make sure you'll get consistently accurate results from workers.

Looking forward to trying out your HITs!

Thanks for your reply MerryLou. We are currently putting together a few preliminary HITs to replicate those we would find in our actual project. Is there a means to 'soft' reject worker responses, where their reputations wouldn't be affected and we could provide feedback?

Realistically, we'll likely have to tweak our HIT delivery (for example, clearer instructions..), and we don't want good quality workers to pay the price for our 'growing pains'.

 

[TSolo315]

Thanks for your reply MerryLou. We are currently putting together a few preliminary HITs to replicate those we would find in our actual project. Is there a means to 'soft' reject worker responses, where their reputations wouldn't be affected and we could provide feedback?

Realistically, we'll likely have to tweak our HIT delivery (for example, clearer instructions..), and we don't want good quality workers to pay the price for our 'growing pains'.

No, unfortunately there is no way to reject a HIT without affecting approval rate. It would be a nice feature to have -- a way for requesters to send a HIT back to a worker so they could fix any mistakes.

 

[MerryLou]

As far as I know, you can also leave a feedback messages on an approved HIT.

If it's feasible, you could post some kind of an example here and them I'm sure you could at least get feedback on the instructions before testing on mturk.

 

[electrolyte]

You may also want to try the Requester Sandbox to post some practice HITs before you post live HITs.

 

[HummingbirdWatch]

No, unfortunately there is no way to reject a HIT without affecting approval rate. It would be a nice feature to have -- a way for requesters to send a HIT back to a worker so they could fix any mistakes.

Thanks for your reply TSolo315. Forgive me if the answer is in another post, but is there perhaps a way to accept the submission that is incorrect, but change the reward to be 0 or 0.01 and provide feedback for the worker? This may sound dangerously close to accepting work without pay, but I'm trying to think of a work around to hurting reputations that don't need to be hurt.

 

[HummingbirdWatch]

As far as I know, you can also leave a feedback messages on an approved HIT.

If it's feasible, you could post some kind of an example here and them I'm sure you could at least get feedback on the instructions before testing on mturk.

Thanks MerryLou for your response. We are hoping to have our first test HIT ready to go by Thursday or Friday, I'll post the HIT instructions on the forum earlier so we can see if there's room for improvement.

 

[HummingbirdWatch]

You may also want to try the Requester Sandbox to post some practice HITs before you post live HITs.

Thanks electrolyte for your reply. We have looked into the Requester Sandbox, and will definitely be testing our HITs that way. We predict that our staff will be prone to bias and miss things that mturk workers will see pretty quick.

 

[TSolo315]

Thanks for your reply TSolo315. Forgive me if the answer is in another post, but is there perhaps a way to accept the submission that is incorrect, but change the reward to be 0 or 0.01 and provide feedback for the worker? This may sound dangerously close to accepting work without pay, but I'm trying to think of a work around to hurting reputations that don't need to be hurt.

Well, you can do it backwards. You can set up your trial hits with .01 as the pay and then send the full amount as a bonus to people who answer correctly. You will have to make this clear to workers when you set up the HIT.

Feedback can be given along with bonuses (people receive emails when they get a bonus.) You can send penny bonuses to people who don't answer correctly with feedback as well.

 

[clickhappier]

Thanks electrolyte for your reply. We have looked into the Requester Sandbox, and will definitely be testing our HITs that way. We predict that our staff will be prone to bias and miss things that mturk workers will see pretty quick.

To clarify, your staff would be the ones to do the testing of your sandbox HITs, as workers would receive no real earnings in the sandbox.

 

[HummingbirdWatch]

To clarify, your staff would be the ones to do the testing of your sandbox HITs, as workers would receive no real earnings in the sandbox.

Apologies, I wasn't very clear with my response. What I meant to say was that we would like to do the initial testing in the Requester Sandbox with our own staff. Once we move on from the sandbox into the trial week, we predict more issues to come to light once our staff are no longer the ones doing the HITs. When this becomes the case, it would be beneficial if there were a mechanism for feedback that wouldn't hurt the worker's reputation (or ours). Thanks for the clarification.

 

[HummingbirdWatch]

Well, you can do it backwards. You can set up your trial hits with .01 as the pay and then send the full amount as a bonus to people who answer correctly. You will have to make this clear to workers when you set up the HIT.

Feedback can be given along with bonuses (people receive emails when they get a bonus.) You can send penny bonuses to people who don't answer correctly with feedback as well.

This method sounds like it could work. Thanks for your input. I've read that dealing with bonuses can be a fairly labor intensive approach for the requester, but it may be the only way to provide feedback during our trial week (without rejecting submissions).

 

[Metallica]

This method sounds like it could work. Thanks for your input. I've read that dealing with bonuses can be a fairly labor intensive approach for the requester, but it may be the only way to provide feedback during our trial week (without rejecting submissions).

I'm not too sure if he's been around lately but you could try to give @GlobalWorkForge a shout. He's pretty experienced with the requester side and has a hand in, or runs multiple different requester accounts.

 

[Randomacts]

This method sounds like it could work. Thanks for your input. I've read that dealing with bonuses can be a fairly labor intensive approach for the requester, but it may be the only way to provide feedback during our trial week (without rejecting submissions).

It can be automated, I'm just not sure how exactly.

You should try and contact the Sticky team https://turkopticon.ucsd.edu/reports?id=A1I5WZRIXY5GS9 they have the whole thing automated although I doubt you could fully automate your process like they do.

 

[HummingbirdWatch]

I'm not too sure if he's been around lately but you could try to give @GlobalWorkForge a shout. He's pretty experienced with the requester side and has a hand in, or runs multiple different requester accounts.

I'll drop @GlobalWorkForge a line and see if he can offer a new requester some advice! Thanks Janthir.

 

[HummingbirdWatch]

It can be automated, I'm just not sure how exactly.

You should try and contact the Sticky team https://turkopticon.ucsd.edu/reports?id=A1I5WZRIXY5GS9 they have the whole thing automated although I doubt you could fully automate your process like they do.

Thanks for the link, I'll have a look at it and see if we can apply it to our process. Automating our workflow will be a priority once we establish something that works in the meantime.

 

[Metallica]

I'll drop @GlobalWorkForge a line and see if he can offer a new requester some advice! Thanks Janthir.

There's also this
https://forums.aws.amazon.com/forum.jspa?forumID=11

 

[Zacharias]

I'm looking forward to working with your hits. A requester that is putting in the work to get things right out of the gate and give feedback to develop their worker pool is a breath of fresh air. Good luck with this project.

 

[HummingbirdWatch]

Hello everyone!

We are working hard to have our HITs ready for the week long trial, and I'm happy to have a schedule update for you. The daily HIT schedule will go as follows (Pacific Time):

1. 6:00 am
2. 10:00 am
3. 2:00 pm
4. 4:30 pm
5. 5:00 pm
6. 5:30 pm
7. 9:00 pm

The first day of the 7 day trial is this Friday (2016-08-12).

Each HIT will consist of 3 assignments, and should take less than a minute to complete.

More information to come! Thanks,

HW