Rep Present Surgery Videos Summarization

[LOS]

After doing 3 of the $4.50 hits and 2 of the $2.25 hits on sunday, I tried to do one today. After 1 video, It said I failed an attention check. I really do not think I did, and another worker on this site said they had the same problem. I'll be emailing them again. I actually like these hits, hope it can get worked out.

I am away from my computer right now. Will get back to you soon.

 

[LOS]

We have added another 13 HITs and will be adding the final 12 HITs for this round soon after that.

 

[LOS]

Just an FYI, in case someone missed out on the notifications/update emails.
We have three groups of HITs on MTurk. Depending on your qualification scores you will be eligible for them.

• Scores (1-4): 15 HITs (Group Link)
• Scores (5-10): 29 HITs (Group Link)
• Scores (10+): 6 HITs (Group Link)

For the time being , these are the only HITs we will be making available.

Then it would be time to do some data mining!

 

[LOS]

Scores (5-10): 5 HITs (Group Link) - are remaining with 2-3 assignments each

 

[Empress328]

Are you still doing those hits? I worked for a CHHA in intake for eight years.

 

[LOS]

Are you still doing those hits? I worked for a CHHA in intake for eight years.

We will host more HITs but this will happen only after a month or so. We shall post here about the HITs.
Thanks for your interest.

 

[LOS]

Hello folks, I wanted to get some feedback and opinions about HITs in general. As some of you may know the HITs that we requested are a bit specialized and require a bit of a learning. Looking at the data from our initial HITs, we performed some analyses. Here is a summary:

1. We launched the HITs at a pricing based on our internal testing, which ended up underpaying the workers. After listening to complaints and feedback from the workers, we increased the price without thinking much.
   
2. Workers started participating in the HITs are were typically satisfied and enjoyed the HITs.
   
3. We noticed from the data collected that the HIT completion times went from 30 minutes on an average at 1st HIT to 15 minutes and under for 15th HIT completed. However, we were still paying at the same rate and there were around 60 different HITs.

We understood that there was a definite learning curve for the workers and as more HITs are performed the time taken drops down significantly. The fastest one being 10 minutes only! That was faster than any of our internal testers. We think a fair approach for both the workers and ourselves would be to pay a higher amount for inexperienced workers since they need longer time and should be paid more. However, after 12-15 HITs, workers are able to finish the HITs much faster and will be paid a lower amount per HIT.

Is this something that requesters have explored previously? Is this something workers do not mind? Please let us know.
Thank you.

 

[SquishyFluffkin]

Hello folks, I wanted to get some feedback and opinions about HITs in general. As some of you may know the HITs that we requested are a bit specialized and require a bit of a learning. Looking at the data from our initial HITs, we performed some analyses. Here is a summary:

1. We launched the HITs at a pricing based on our internal testing, which ended up underpaying the workers. After listening to complaints and feedback from the workers, we increased the price without thinking much.
   
2. Workers started participating in the HITs are were typically satisfied and enjoyed the HITs.
   
3. We noticed from the data collected that the HIT completion times went from 30 minutes on an average at 1st HIT to 15 minutes and under for 15th HIT completed. However, we were still paying at the same rate and there were around 60 different HITs.

We understood that there was a definite learning curve for the workers and as more HITs are performed the time taken drops down significantly. The fastest one being 10 minutes only! That was faster than any of our internal testers. We think a fair approach for both the workers and ourselves would be to pay a higher amount for inexperienced workers since they need longer time and should be paid more. However, after 12-15 HITs, workers are able to finish the HITs much faster and will be paid a lower amount per HIT.

Is this something that requesters have explored previously? Is this something workers do not mind? Please let us know.
Thank you.

It's an interesting thought, but I don't know how you would determine a worker's experience level with your hits when assigning them. I think the only way would be to give out qualifications as worker's do more of them, then post hits specifically for those qualification. Anyways I think it's a good idea, though I haven't personally worked any of your hits so I wouldn't be a good judge to determine whether they were underpaid or if the new amount would be fair.

 

[tsatsagt khyaruul]

Hello folks, I wanted to get some feedback and opinions about HITs in general. As some of you may know the HITs that we requested are a bit specialized and require a bit of a learning. Looking at the data from our initial HITs, we performed some analyses. Here is a summary:

1. We launched the HITs at a pricing based on our internal testing, which ended up underpaying the workers. After listening to complaints and feedback from the workers, we increased the price without thinking much.
   
2. Workers started participating in the HITs are were typically satisfied and enjoyed the HITs.
   
3. We noticed from the data collected that the HIT completion times went from 30 minutes on an average at 1st HIT to 15 minutes and under for 15th HIT completed. However, we were still paying at the same rate and there were around 60 different HITs.

We understood that there was a definite learning curve for the workers and as more HITs are performed the time taken drops down significantly. The fastest one being 10 minutes only! That was faster than any of our internal testers. We think a fair approach for both the workers and ourselves would be to pay a higher amount for inexperienced workers since they need longer time and should be paid more. However, after 12-15 HITs, workers are able to finish the HITs much faster and will be paid a lower amount per HIT.

Is this something that requesters have explored previously? Is this something workers do not mind? Please let us know.
Thank you.

I remember doing these HITS in March. They were hard at first but became easier and more enjoyable with familiarity.
The proposal is an interesting idea. I would be okay with a lowering amount/HIT for experienced worker as long as the difficulty remained the same and there are many HITS available. I don't know how much lower you could set the pay as what is going to be acceptable is different for each person.
Also, if you are looking for both speed and accuracy, I would suggest setting different qualification levels and bonuses for performance.

 

[Metallica]

In what world is it acceptable to expect the same worker to give you the same exact results when you decide to just rip the pay out from under them? All you will cause from that is opening scammers up to your hits.

 

[LOS]

It's an interesting thought, but I don't know how you would determine a worker's experience level with your hits when assigning them. I think the only way would be to give out qualifications as worker's do more of them, then post hits specifically for those qualification. Anyways I think it's a good idea, though I haven't personally worked any of your hits so I wouldn't be a good judge to determine whether they were underpaid or if the new amount would be fair.

We are planning on deploying the same strategy as you mentioned. We will wait to get some more input form workers who have performed these HITs. Thanks!

 

[LOS]

I remember doing these HITS in March. They were hard at first but became easier and more enjoyable with familiarity.
The proposal is an interesting idea. I would be okay with a lowering amount/HIT for experienced worker as long as the difficulty remained the same and there are many HITS available. I don't know how much lower you could set the pay as what is going to be acceptable is different for each person.
Also, if you are looking for both speed and accuracy, I would suggest setting different qualification levels and bonuses for performance.

We intend to keep the same amount of difficult and a large number of HITs available (since otherwise it would just be of no use for someone to perform more of these HITs if there aren't a lot more while the pay also gets reduced for experienced workers). Bonuses would definitely be a part of the plan, and we shall aim to follow the guidelines provided for requesters while pricing the HITs for more experienced workers.
Plus, we can also compensate for the lower pay by making more HITs available for experienced workers as compared to lesser experienced ones.
Thanks for your feedback.

 

[LOS]

In what world is it acceptable to expect the same worker to give you the same exact results when you decide to just rip the pay out from under them? All you will cause from that is opening scammers up to your hits.

We understand what you are saying here. That is why we would like to get feedback before we act. We think that by making enough HITs available to more experienced workers, and still following set guidelines for pricing, we will be able to maintain a good balance between time required for each HIT, experience level of the worker with our HITs.
Could you explain what you mean by scammers? (people with multiple accounts trying to get more pay?)

 

[Squatty]

I don't think it is fair to lower the pay for more experienced workers. Think of it from a different prospective, like a brick layer.
An apprentice bricklayer only makes $10 an hour, but it takes him 8 hours to build a wall.
The experienced bricklayers make $20/hour but can do the same wall in 4 hours.

Why would you pay the experienced brick layer less just because he can build the wall faster?
The old saying you get what you pay for will hold true.

 

[Metallica]

We understand what you are saying here. That is why we would like to get feedback before we act. We think that by making enough HITs available to more experienced workers, and still following set guidelines for pricing, we will be able to maintain a good balance between time required for each HIT, experience level of the worker with our HITs.
Could you explain what you mean by scammers? (people with multiple accounts trying to get more pay?)

Scammers as in people who intentionally pass the work up and hope to get paid. It wastes your time, and theirs. I don't mean to sound harsh, but lowering pay for your long term workers is just a terrible thing to do. Think about it; you're working a $80,000 a year job and then one day your boss says 'since you've been doing good work for many years, we're going to cut your pay down to half that.'

As for the multiple accounts type, that's typically illegal as 99% of the time it is identity theft. They'll get their own karma.

Edit: If anything, you should consider paying more to your highest-quality workers. Requester's lowering pay is something we are all too used to on this platform and it would probably result in you losing good workers.

 

[LOS]

@Janthir , @Squatty
Here is why we are thinking of what we proposed (reframed in a better way). We believe the HITs can be completed in 15 minutes or less. However, initially people tend to take a lot more time getting used to the framework. And it can take them up to 25-30 minutes. So we saw workers post on the forums that the HITs are taking a lot more time than what was mentioned and are paying less. Since, we are from academia and we were hoping to get the data from MTurk within certain time constraints, the best thing was to look at completion times and estimate a new (fair) price and we did that. However, then we saw that the completion times went down, so we were in fact over paying (as far as the amount of time spent was on an average). So, we launched just a small set of HITs as a pilot. However, we are going beyond our budget if we continue at the same rate. And, so we decided to be fair we should pay initially higher amounts and then reduce it to lower amounts (which are the actual rates per the time that should be needed).
However, Iwe think that the better strategy now would be to just continue with a fixed amount and hope that workers will continue to perform more HITs and reach the expected time. We can motivate them by sending bonuses initially rather than having higher priced HITs.

 

[Squatty]

@Janthir , @Squatty
Here is why we are thinking of what we proposed (reframed in a better way). We believe the HITs can be completed in 15 minutes or less. However, initially people tend to take a lot more time getting used to the framework. And it can take them up to 25-30 minutes. So we saw workers post on the forums that the HITs are taking a lot more time than what was mentioned and are paying less. Since, we are from academia and we were hoping to get the data from MTurk within certain time constraints, the best thing was to look at completion times and estimate a new (fair) price and we did that. However, then we saw that the completion times went down, so we were in fact over paying (as far as the amount of time spent was on an average). So, we launched just a small set of HITs as a pilot. However, we are going beyond our budget if we continue at the same rate. And, so we decided to be fair we should pay initially higher amounts and then reduce it to lower amounts (which are the actual rates per the time that should be needed).
However, Iwe think that the better strategy now would be to just continue with a fixed amount and hope that workers will continue to perform more HITs and reach the expected time. We can motivate them by sending bonuses initially rather than having higher priced HITs.

Yes I agree. I would price them at the expected time of an experienced worker.
We all understand that tasks take longer when are learning. I would be more than willing to do practice hits at a lower pay rate if I knew I would get better and faster at the task.

 

[Squatty]

Could you please let us know when you post new Hits? Also will you give out more quals? I have never done any of your Hits and now I really would like to.

 

[LOS]

Could you please let us know when you post new Hits? Also will you give out more quals? I have never done any of your Hits and now I really would like to.

Surely, we are still working out on some changes to the survey form and making it better. We will post here once we have new batch of HITs ready. We will open the Qualification for the HITs also alongside for some time to get new workers as well. Thanks for the enthusiasm.