Rep Present Surgery Videos Summarization

[LOS]

Hello folks, by now some of you might have seen these HITs on the MTurk page.
https://www.mturk.com/mturk/preview?groupId=3A5N6ETZEU1R273R6CW4BZ5MOYQJFL

We are researchers from the Computer Science department at the Johns Hopkins University.
We wanted to join in here and make sure I was available for people to send me comments, suggestions, etc.
It is my first time posting HITs on MTurk. Also, this is a pretty neat project that we are working on. We would like to see how we can improve this.
We have already received some feedback from folks who have tried the HIT and are actively trying to incorporate them. We realize that this isn't possible without the Turker community and would like to try our best in our capacity to have a smooth and nice experience for all.

 

[electrolyte]

Welcome!

Title: Qualification - Summarize Surgery RALH (WARNING: HIT contains anatomy/organ images and blood. Worker discretion is advised.) | PANDA Requester: LOS [A2MG9E64Z06RY7] (TO) TO Ratings: 5.00 Communicativity 1.00 Generosity 5.00 Fairness 5.00 Promptness Number of Reviews: 2 (Submit a new TO rating for this requester) Description: This HIT should be completed with success to grant you the qualification needed to complete other HITs hosted by us. Please review the information and take a short test. The survey contains graphical surgical images with blood at times. Time: 45 minutes Hits Available: 1 Reward: $0.50 Qualifications: Total approved HITs is not less than 100, BlockRobHystSummarizer has not been granted, RobHystSummarizer has not been granted, HIT approval rate (%) is not less than 95, Adult Content Qualification is 1

Title: Summarize Surgical Videos, Duration: 30 minutes (WARNING: HIT contains organ images and blood. Worker discretion is advised.) | PANDA Requester: LOS [A2MG9E64Z06RY7] (TO) TO Ratings: 5.00 Communicativity 1.00 Generosity 5.00 Fairness 5.00 Promptness Number of Reviews: 2 (Submit a new TO rating for this requester) Description: The survey consists of observing portions of videos of surgical procedures (with graphical surgical images), and answering multiple choice questions that provide a description of the video. Each survey contains a set of 10 videos with seven multiple choice questisons per video. Time: 1 hour 30 minutes Hits Available: 4 Reward: $2.00 Qualifications: RobHystSummarizer has been granted, Total approved HITs is not less than 100, BlockRobHystSummarizer has not been granted, HIT approval rate (%) is not less than 95, Adult Content Qualification is 1

 

[LOS]

We have published updated HITs taking into account different recommendations from Turkers who participated in the initial few HITs.
We are new at this and from academia, so please do bear with us, since we are constrained in terms of support for this. We will try our best to address any issues that could/might affect your HIT submission.

https://www.mturk.com/mturk/preview?groupId=3Z7ICGQCYAUFMNCPH9TY0Z19E1ATOO
(Please use the preview link since there are repeated HITs and you would end up returning a HIT if you have already performed it previously)

UPDATE: per HIT payment has been increased to $4.5

We have also increased the total time allotted since we realized Turkers would not be solely working on this HIT or they could get disrupted for whatever other reasons.

 

[clickhappier]

@LOS would it be possible for you to describe more of what the HITs are like, to help people get a better idea of how interested they'd be in pursuing the apparently-takes-20min-for-$0.50 qual HIT beforehand? Examples of what the questions are and such.

 

[LOS]

@LOS would it be possible for you to describe more of what the HITs are like, to help people get a better idea of how interested they'd be in pursuing the apparently-takes-20min-for-$0.50 qual HIT beforehand? Examples of what the questions are and such.

Are you suggesting including some text about this on the HIT page, or posting something here?

 

[electrolyte]

Are you suggesting including some text about this on the HIT page, or posting something here?

I'm not @clickhappier but I'd be happy with either or both.

 

[electrolyte]

Export for the new $4.50 version:

Title: Summarize Surgical Videos, Duration: 45 minutes (WARNING: HIT contains organ images and blood. Worker discretion is advised.) | PANDA Requester: LOS [A2MG9E64Z06RY7] (TO) TO Ratings: 5.00 Communicativity 1.00 Generosity 5.00 Fairness 5.00 Promptness Number of Reviews: 2 (Submit a new TO rating for this requester) Description: The survey consists of observing portions of videos of surgical procedures (with graphical surgical images), and answering multiple choice questions that provide a description of the video. Each survey contains a set of 10 videos with seven multiple choice questisons per video. Time: 2 hours 30 minutes Hits Available: 4 Reward: $4.50 Qualifications: RobHystSummarizer has been granted, Total approved HITs is not less than 100, BlockRobHystSummarizer has not been granted, HIT approval rate (%) is not less than 95, Adult Content Qualification is 1

 

[LOS]

@LOS would it be possible for you to describe more of what the HITs are like, to help people get a better idea of how interested they'd be in pursuing the apparently-takes-20min-for-$0.50 qual HIT beforehand? Examples of what the questions are and such.

I have included the following on the front page of the qualification HIT.

About the HITs:
You will be asked to look at a 90 seconds long video that is part of a longer length surgery. Based on the video, you will be asked 6 questions related to:

1. whether you saw a particular object appear in the video - Yes/No,
2. what was the activity being performed e.g. cutting tissue, burning tissue or stitching,
3. what were the robotic tools used for,
4. how was the skill of the surgeon performing the surgery - similar to C-SATS

Additionally, I have mentioned the fact that we pay a bonus of $0.5 to workers who pass the test in addition to granting them the qualification.

 

[LOS]

We have uploaded shorter versions of the previous HITs on MTurk. These contain a total of 6 videos.
Many workers had requested that the total number of videos in a HIT be reduced, and it made quite sense to do so.
Requester page: https://www.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&requesterId=A2MG9E64Z06RY7

If you have already completed ALL the previous HITs then you don't have anything new and we have assigned qualifications to disallow previous completions.
Otherwise please take up whichever of the longer or shorter versions and help our research.

As always we are fortunate to have such an opportunity to get data that can be used towards meaningful research improving the lives of patients, surgeons, other healthcare providers and our community. And, we are open to receive feedback to improve our HITs and make it convenient for the workers while abiding by our research goals.
Please do review us on TO or other forums so we can obtain the required amount of data sooner

 

[Cursed Scroll]

Just an FYI for everyone here:

I did the $4.50 today. It took me 55 mins going very slow. I got coffee twice, and emaiiled the requestor during that time. The more videos I watched the faster I got at going thru the questions. I think I might be able to get down to 35 mins with no distractions. I emailed the requestor 2 times, once after I returned a hit after 1 video, and once after I did the whole 10 videos. Each time they got back to me within 5 mins.

The first time I emailed about having the submit button also being the the same as " go to next video" button. They changed that within 10 mins of my email. The second time was about what to do if the instruments where idle but action was happening behind tissue that you could not see. Again, they replied very quick.

 

[LOS]

Just an FYI for everyone here:

I did the $4.50 today. It took me 55 mins going very slow. I got coffee twice, and emaiiled the requestor during that time. The more videos I watched the faster I got at going thru the questions. I think I might be able to get down to 35 mins with no distractions. I emailed the requestor 2 times, once after I returned a hit after 1 video, and once after I did the whole 10 videos. Each time they got back to me within 5 mins.

The first time I emailed about having the submit button also being the the same as " go to next video" button. They changed that within 10 mins of my email. The second time was about what to do if the instruments where idle but action was happening behind tissue that you could not see. Again, they replied very quick.

Thanks @Cursed Scroll for this input. We think that initially workers will need time to get accustomed to the HIT. This also has a learning curve like everything else. However, we won't know that unless workers perform these HITs and we are seeing similar trends with other workers as well.

 

[LOS]

We have launched a new batch of HITs (15 of them) with 35 more to follow later tonight.
We hope you enjoy doing those and help us with our research.
Please let us know if there are any troubles, glitches, etc. We will try our best to resolve them as quickly as possible.

Requester page:
https://www.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&requesterId=A2MG9E64Z06RY7

Regarding additional qualifications:
We will be monitoring the progress of the HITs and accordingly open new slots for qualification tests. We will try and make sure that we post once they are available here.

 

[Roscoe E Goldchain]

We have launched a new batch of HITs (15 of them) with 35 more to follow later tonight.
We hope you enjoy doing those and help us with our research.
Please let us know if there are any troubles, glitches, etc. We will try our best to resolve them as quickly as possible.

Requester page:
https://www.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&requesterId=A2MG9E64Z06RY7

Regarding additional qualifications:
We will be monitoring the progress of the HITs and accordingly open new slots for qualification tests. We will try and make sure that we post once they are available here.

Hi, this newest batch seems to have some problems

On my second vid, it never loaded. Returned after trying multiple times, Tried again & this time the third vid will not load

 

[Shadowsbane1988]

Hi, this newest batch seems to have some problems

On my second vid, it never loaded. Returned after trying multiple times, Tried again & this time the third vid will not load

I'm having the same issue as Roscoe - the second video just won't load

 

[LOS]

Hi, this newest batch seems to have some problems

On my second vid, it never loaded. Returned after trying multiple times, Tried again & this time the third vid will not load

I am looking into this. Thanks!

 

[LOS]

I'm having the same issue as Roscoe - the second video just won't load

This has been fixed. It was a rookie rookie mistake!

 

[LOS]

Hi, this newest batch seems to have some problems

On my second vid, it never loaded. Returned after trying multiple times, Tried again & this time the third vid will not load

I have resolved this. Please continue. Apologies for this issue.

 

[Roscoe E Goldchain]

Thanks alot

 

[LOS]

The new batch for qualification score 5+ have been published as well - 10 of them for now.

 

[Cursed Scroll]

After doing 3 of the $4.50 hits and 2 of the $2.25 hits on sunday, I tried to do one today. After 1 video, It said I failed an attention check. I really do not think I did, and another worker on this site said they had the same problem. I'll be emailing them again. I actually like these hits, hope it can get worked out.