Rep Present Hummingbird Watch

[HummingbirdWatch]

Here's a quick update about todays HITs.

Timer has been adjusted to 15 minutes to allow for the large images to load completely. Thanks and good luck!

 

[Jharkan]

This sounds like it's very interesting, but I'm only at 8000 hits, so I cannot do them.
To be honest, I don't understand what the difference is, that a requester wants the 10000 submissions, instead of the 5000. I do understand wanting people that have consistently done work on mturk, but i very rarely see any opportunity to get more than 15, 20 submissions per day, so 10000 is a long way away.

 

[HummingbirdWatch]

Requiring a worker to have submitted 10000 submissions was a way for us to limit our HITs to those who have the most experience that we could draw on for feedback and guidance. This trial week is our first go at information and data gathering, and doesn't necessarily reflect what future projects will look like.

We may do away with all of Mechanical Turk's Qualifications, and replace it with our own custom one that we assign to workers, or a combination of Qualifications. The criteria for receiving our Qualification is not determined yet, but we may loosen up the 10000 submissions requirement.

We're making a list of workers who have expressed interest in participating in long-term projects with us, and would be happy to include you on that list.

The only requirements we wouldn't be able to bend on are the location Qualifications, and accuracy.

 

[Randomacts]

Requiring a worker to have submitted 10000 submissions was a way for us to limit our HITs to those who have the most experience that we could draw on for feedback and guidance. This trial week is our first go at information and data gathering, and doesn't necessarily reflect what future projects will look like.

We may do away with all of Mechanical Turk's Qualifications, and replace it with our own custom one that we assign to workers, or a combination of Qualifications. The criteria for receiving our Qualification is not determined yet, but we may loosen up the 10000 submissions requirement.

We're making a list of workers who have expressed interest in participating in long-term projects with us, and would be happy to include you on that list.

The only requirements we wouldn't be able to bend on are the location Qualifications, and accuracy.

I'm in the same boat that Jharkan is and around the same amount of hits done and I agree with what he said.

If you are looking for people to do long term work I would love to join in on that as well.

 

[HummingbirdWatch]

I'm in the same boat that Jharkan is and around the same amount of hits done and I agree with what he said.

If you are looking for people to do long term work I would love to join in on that as well.

@Randomacts Thanks for your interest! We'll reach out to you when we have some more details about the Hummingbird Watch Qualification.

 

[HummingbirdWatch]

Hello everyone, just a quick update.

Our HITs now include several smoke examples, as well as some examples of common mistakes.

Thanks for the feedback!

 

[Johnnyrs]

I've been doing some of these and I liked the changes that was made. At first the timer was too short but now it's better. The examples help a lot to better judge what to look for. I'm interested in any future work also.

 

[HummingbirdWatch]

Hello everyone,

Another update: We are no longer offering bonuses to provide feedback. This process has proved to be very time consuming and easy to lose track of which worker has been given a bonus. From now on we'll change the wage to 0.15, and we'll process all the data after the test week.

Just a reminder that we aren't going to reject any submissions for this trial, just do your best to select images that match the examples provided. Thanks!

HW

 

[Fernando]

HummingbirdWatch, there's a reason for this qualification Location is one of AU, CA, NZ, GB, US.?

 

[HummingbirdWatch]

@Fernando, thank you for your reply. Our client has requested we limit our HITs to these particular countries for the trial. Depending on the results, it's not impossible that we'll broaden our scope to include more countries.

 

[clickhappier]

Thanks for adding examples, but I really wish they were on an externally-linked webpage/document/forum-post instead of directly inside the HIT, making it even more massive to load.

Here's the examples from the HIT, now that I got one:

" The following images are examples of smoke we are interested in:

The following images are examples of images that are commonly mistaken as containing smoke:

This image is a low contrast image that appears to contain haze, however it is the result of poor image quality.

This image contains land features that look like smoke, but there is no smoke in this image.

"​

 

[clickhappier]

And wow, now there's 12 images to evaluate in one HIT too? I thought there was like 5 or 6 in the one I had time out on me the other day.

 

[clickhappier]

I just saw a glimpse of one of this evening's HITs, it looked like the timer was now a mere 2 minutes?? Heading in the wrong direction on that, heh...

 

[HummingbirdWatch]

Thanks everyone for the terrific first week Trial! We'll be analyzing our collected data, discussing our received feedback, and begin organizing the next trial. Timeline updates will be posted soon.

 

[HummingbirdWatch]

Hello everyone. We have been analyzing the data collected from the previous trial, and we are very pleased with the results! Thank you for all those who accepted our HITs, we learned a lot from the first trial and are excited to release a second one later this week (Wednesday or Thursday).

Here are some details for the upcoming second trial:

• $0.15 per assignment
• No bonuses
• 5 assignments per HIT
• 15 minute time limit
• Same HIT template as previous trial, including example images.
• Same stock qualifications as the last trial:
  • HIT Approval Rate greater than or equal to 95%
  • Location is one of AU, CA, NZ, GB, US.
• We will be dropping the Number of Hits Accepted Qualification to 5000.
• We will NOT be rejecting any submissions for this trial.
• 49 individual HITs will be released over the 7 day trial.
• HITs will be published at the same times as the previous trial:
  • 6:00 am, 10:00 am, 2:00 pm, 4:30 pm, 5:00 pm, 5:30 pm and 9:00 pm.

The wage will be the same as the first trial, except we will not be giving bonuses. This was a means for us to reach out and provide individual feedback, rather than to reward exemplary performance. This method proved too prone to human error, where some workers would be skipped and others would receive more than 1 bonus per submission. For this next trial we encourage all workers to reach out to us through the forums or direct e-mail if they would like some clarification or feedback, or to provide us with feedback.

We will be setting the time limit to 15 minutes to allow for the large images to load completely in the MTurk stock UI. Once time permits we'll move the images to a separate server to cut down on image loading time.

Also, in the previous HIT we specified that there were different types of smoke we would be looking for. In this trial we want to clarify that we want workers to click on images that contain ANY smoke whatsoever.

After reviewing the data we decided not to go forward with a custom Qualification yet. The same stock qualifications as last time will be used, with the exception of dropping the number of HITs accepted to 5000. We understand that some very interested people may have a hard time accessing our HITs because of this. Our priority is on accuracy and speed, and feel that both these priorities would be better met if we avoided custom qualifications at this time.

No submissions will be rejected, so do your best without fear of damaging your reputation with our HITs.

The HITs will be released as new HITs each time to make organization and data entry easier for us. The naming convention for our HITs will look something like: HBW Trial 2 Day 1 Batch 1. The HITs will be released at the same times as the previous to make the two trials comparable for our future analyses.

Here are some questions that we have:

How do we reach out to workers once we have their worker IDs?

We are interested in finding out where people live that accept our HITs, is this something that can be done with worker IDs?

Is 15 minutes long enough to compensate for the image loading times?

Were the smoke examples provided sufficient or would you benefit from a more robust smoke example image dataset?

Would it be beneficial to you to include examples in the forums as well as the HITs themselves?

Are many workers full time mechanical turkers, or is the vast majority of workers accepting HITs in their spare time?


We appreciate your continued support and feedback as we continue to learn and develop our process. Thanks to all those who have already provided feedback and guidance, and we look forward to hearing more from you!

HBW

 

[Metallica]

@HummingbirdWatch
1. It's been a while since I've used the requester interface, but I remember there being an option to message workers from the worker panel, or whatever it's called. It should be beside each worker ID. edit: I'm still probably wrong here, but when you see the list of worker ID's, try clicking on one of them and see what it does.
2. You can check the IP of the worker who submitted the hit. edit 2: You COULD use a ID to find their relative location by adding it into an Amazon link- if you're interested in that I will find it for you. However, many workers either have that disabled, or do not like the fact requesters can see that much about someone (includes past purchases, relative age, maybe even a picture, as far as I know.)
3. 15 minutes is fine, and given your concerns about wanting the data immediately, there's really no use to raise it - since putting multiple of these hits in one's queue will prove useless.
4. Your examples look good, the decision to clarify on that further is your choice.
5. Personally I would prefer to just have the images in the hit, loading them all up here sucks up all the memory my crappy PC has and seems a little unnecessary.
6. That's though to say. If you hang out across various mturk forums, you will quickly see the large majority of them are full time turkers. However, there are probably even more part time workers out there than full time - its a little hard to get exact numbers on that.

 

[HummingbirdWatch]

@HummingbirdWatch
1. It's been a while since I've used the requester interface, but I remember there being an option to message workers from the worker panel, or whatever it's called. It should be beside each worker ID. edit: I'm still probably wrong here, but when you see the list of worker ID's, try clicking on one of them and see what it does.
2. You can check the IP of the worker who submitted the hit. edit 2: You COULD use a ID to find their relative location by adding it into an Amazon link- if you're interested in that I will find it for you. However, many workers either have that disabled, or do not like the fact requesters can see that much about someone (includes past purchases, relative age, maybe even a picture, as far as I know.)
3. 15 minutes is fine, and given your concerns about wanting the data immediately, there's really no use to raise it - since putting multiple of these hits in one's queue will prove useless.
4. Your examples look good, the decision to clarify on that further is your choice.
5. Personally I would prefer to just have the images in the hit, loading them all up here sucks up all the memory my crappy PC has and seems a little unnecessary.
6. That's though to say. If you hang out across various mturk forums, you will quickly see the large majority of them are full time turkers. However, there are probably even more part time workers out there than full time - its a little hard to get exact numbers on that.

Thanks for the reply @Janthir. We appreciate you taking the time to offer your input.

 

[HummingbirdWatch]

Hello all,

The first hit of our second trial will be released at 6:00 am tomorrow morning.

Thanks and good luck!

HBW

 

[HummingbirdWatch]

Hello all,

The first hit of our second trial will be released at 6:00 am tomorrow morning.

Thanks and good luck!

HBW

Hello all, it appears we're having more technical issues and will be postponing the trial for today. Hopefully we'll have the issue resolved in time to begin the trial tomorrow. Thanks for your patience! Edit: It appears our account is temporarily locked, we're told that the issue should be resolved within 24 hours. We'll let you know when we're up and running again, and when we plan to post the first HIT!

HBW

 

[HummingbirdWatch]

Looks like our account issue has been sorted out, so we'll begin the week-long trial tomorrow morning at 6:00 am. Thanks again for your patience!