Rep Present C-SATS - Surgical Video Review

[C-SATS Inc]

Hi everybody, we're C-SATS, and we assess surgical skill through video review. Please read this post and our FAQ to learn more about our HITs. If you don't find an answer to your question, please feel free to ask it below.

A little background about us - we started as a research project to try and measure how good a doctor is at surgery, and quickly learned that the easiest way to do that is simply by watching video. Our research shows that non-surgeons can accurately assess technical skills of a surgeon [1], [2]. That's where you all come in.

If you've ever done a HIT for us before, you'll know our tasks usually consist of watching a ~10 minute video and then answering 6-8 questions about the skill of the surgeon. Sometimes the videos are much shorter, and the pay for that HIT will reflect the difference. These videos are captured from inside the human body, so they can be a bit graphic if you're easily bothered by such things. But what you're watching for isn't regarding the anatomy - it's about how fluidly and confidently the surgeon uses the surgical tools.

The first time you do a new review type for us, you might also be asked to watch a brief training video. You'll only have to do that once for that survey type.

Happy to answer any questions about our HITs and we welcome feedback & suggestions for how to do things better.

Useful Links:

• Frequently Asked Questions
• Search for our HITs

C-SATS

 

[Jaded]

Welcome to mTurkCrowd!

 

[SleepComa]

You guys are loved by the Mturk community

 

[AlphaFemale]

Welcome! Love your hits, looking forward to more.

 

[Roscoe E Goldchain]

hi!

 

[C-SATS Inc]

You guys are loved by the Mturk community

Thanks very much. What would you say the primary reason is for being liked? The HITs are interesting? And what could be done better?

 

[humbleturker]

Thanks very much. What would you say the primary reason is for being liked? The HITs are interesting? And what could be done better?

You're fair, you don't insult us with the pay, and your HITs are unique. Better than rummaging around in the same surveys each day.

 

[Serial Mom]

.

 

[Tigerpants]

Thanks very much. What would you say the primary reason is for being liked? The HITs are interesting? And what could be done better?

I can't speak for everyone, but since I started Mturk about a year ago you have been my favorite requester. I find the content very interesting and loved when you added the procedures and steps.

As far as improvement goes I can't think of much. The directions are clear and the little notes on the scales help a lot.

I especially loved the critical view of safety on the gallbladder HITs last week or so, those where so interesting!

 

[coffeeowl]

Hi everybody, We're C-SATS, and we assess surgical skill through video review. Please read this post and our FAQ to learn more about our HITs. If you don't find an answer to your question, please feel free to ask it below.

A little background about us - we started as a research project to try and measure how good a doctor is at surgery, and quickly learned that the easiest way to do that is simply by watching video. Our research shows that non-surgeons can accurately assess technical skills of a surgeon [1], [2]. That's where you all come in.

If you've ever done a HIT for us before, you'll know our tasks usually consist of watching a ~10 minute video and then answering 6-8 questions about the skill of the surgeon. Sometimes the videos are much shorter, and the pay for that HIT will reflect the difference. These videos are captured from inside the human body, so they can be a bit graphic if you're easily bothered by such things. But what you're watching for isn't regarding the anatomy - it's about how fluidly and confidently the surgeon uses the surgical tools.

The first time you do a new review type for us, you might also be asked to watch a brief training video. You'll only have to do that once for that survey type.

Happy to answer any questions about our HITs and we welcome feedback & suggestions for how to do things better.

Useful Links:

• Frequently Asked Questions
• Search for our HITs

C-SATS

Love your HITs!! Thank you very much for reaching out and for updating your FAQ's. Much appreciated!

 

[A6_Foul_Out]

A little background about us - we started as a research project to try and measure how good a doctor is at surgery, and quickly learned that the easiest way to do that is simply by watching video. Our research shows that non-surgeons can accurately assess technical skills of a surgeon [1], [2].

C-SATS

Fascinating.

Truly.

 

[Yatagarasu]

Oh nice, one of my favorite requesters. Glad to see you here.

 

[Squatty]

Thanks very much. What would you say the primary reason is for being liked? The HITs are interesting? And what could be done better?

I hate to be the negative Nelly.

I do really enjoy your hits. They are interesting and the Gallbladders last week were amazing. I loved the training video, so I knew what I was looking for.

But, the pay really isn't that good. You are paying about 11 cents a minute, which comes to $6.60/hour. This is below minimum wage.
If it is at all possible, could you try to increase the pay to 14 cents a minute?

Thank you again!

 

[humbleturker]

I hate to be the negative Nelly.

I do really enjoy your hits. They are interesting and the Gallbladders last week were amazing. I loved the training video, so I knew what I was looking for.

But, the pay really isn't that good. You are paying about 11 cents a minute, which comes to $6.60/hour. This is below minimum wage.
If it is at all possible, could you try to increase the pay to 14 cents a minute?

Thank you again!

I can coattail off of this because I'm just curious about the altered pay rates. Some videos are, as Squatty mentioned, about 11c/min. That being said, you had amazing batches of gallbladders as well as the 50 cent batch that I believe was posted yesterday or the day before that the pay rate was far better. I have two questions: in cases like this, how do you dictate the increased pay in these situations and also, like Squatty mentioned, is it at all possible to increase the pay a little on videos that are more in tune with the wage relationship that she mentioned?

 

[C-SATS Inc]

I can coattail off of this because I'm just curious about the altered pay rates. Some videos are, as Squatty mentioned, about 11c/min. That being said, you had amazing batches of gallbladders as well as the 50 cent batch that I believe was posted yesterday or the day before that the pay rate was far better. I have two questions: in cases like this, how do you dictate the increased pay in these situations and also, like Squatty mentioned, is it at all possible to increase the pay a little on videos that are more in tune with the wage relationship that she mentioned?

Thanks for the honest feedback. Pay rate is something we are working to get a better handle on.

To respond to both of you at once: obviously, everyone works at different speeds. We try and compute a fair rate that's going to a) allow us to stay in business and b) attract quality crowd workers like yourselves, not scammers. Sometimes, depending on the customer and the kind of video, amount of training, number of questions, etc, your perceived pay rate may jump around. That could be because we over/under estimated how long the HIT would take. That said, your perceived pay rate seems plausible. And yes, we need to ensure we pay a fair pay rate. We may have overestimated the amount of work for the gallbladders HITs and underestimated elsewhere. One of our growing edges needs to be getting to a consistent pay rate / hr.

In an ideal world, I think we move away from a flat pay rate for all workers and start paying repeating workers who do a good job and pay attention during HITs a higher rate. Honestly, MTurk doesn't make that super easy for us, but as we grow we're looking harder into it.

 

[C-SATS Inc]

A question for all of you: what kind of training do you wish we provided? Would it be useful to know more about the anatomy or procedure you're looking at? Are there any parts of our HITs that you think should be better explained?

Interestingly, our research has suggested that teaching workers about anatomy doesn't improve the accuracy of the ratings they give. That might be because the ratings are focused on technical skill, not judgement or decision making.

 

[jessers]

Can you explain what you expect in terms of written responses? At one time I read that workers should comment in every box, but more recently it was being passed around to "only comment when necessary", meaning some HITs could have no writing whatsoever if the only feedback would be something already covered in the questions (e.g. "works slow but confident").

 

[B1llyBlanco]

A question for all of you: what kind of training do you wish we provided? Would it be useful to know more about the anatomy or procedure you're looking at? Are there any parts of our HITs that you think should be better explained?

Interestingly, our research has suggested that teaching workers about anatomy doesn't improve the accuracy of the ratings they give. That might be because the ratings are focused on technical skill, not judgement or decision making.

1.You guys should give some type of guideline as to what constitutes a pass and a fail. For example do they need to be at least all 3's to be passable because since this is a surgeon we are grading, is just average(3 out of 5) good enough? Surgery, at least how I see it is very important and you need to be great at it to be passed so I have been grading on the harsher side. Are these people just students and this is the first in many tests they will take to be qualified to do these procedures or is this the final test and we are saying that if we pass them that we think they are ready to do that actual surgery? I think this would make a difference in the way people rate the pass/fail part at least.

2. I think that just saying which of the two surgeons are better for the training isn't enough. I think you should show two video for each procedure, one that shows a proficient surgeon and shows the things that they are doing well and one that points out all the things that would be wrong or should be marked off. You should make people watch them once and then embed links to the videos so people can re-watch them whenever they need to.

3. What you were saying about making HITs for people who do a good job. You could have us enter our worker ID's on each HIT and take the time to look for workers who do a lot of your HITs and are marking the answers the correct way and then make a Qualification on Mturk for them. It may be time consuming at first, but youd have that worker pool on hand from then on and weed out the people you dont want. Or you could use the Mturk ID's to just track bad workers and give them Qualifications that if they have them, it denies them access to the HITs.

4. As someone mentioned earlier, it's unclear sometimes if you want us to write in the text boxes. Sometimes it's labeled as optional and other times it isnt. I would like clarification on that.

 

[C-SATS Inc]

Can you explain what you expect in terms of written responses? At one time I read that workers should comment in every box, but more recently it was being passed around to "only comment when necessary", meaning some HITs could have no writing whatsoever if the only feedback would be something already covered in the questions (ie "works slow but confident").

Sure thing. Recently, you've probably noticed the "OPTIONAL" tag above most of the comment boxes. Obviously, those are not required, though they are appreciated.

In general, the comments are intended to be used for if/when you have something specific to point out. Maybe you saw the surgeon drop a needle or burn something - that's not something that would get captured in the 1-5 scale. A useful comment like that will get passed directly along to the surgeon themselves for their review! We very much value thoughtful comments like that, but don't expect you to write them all the time. I think eventually we'd implement a bonus system for reviews that leave useful comments. But to be clear, saying "great efficiency overall" when you are giving them a 5 isn't very informative, and you're welcome to not bother saying anything in that case.

 

[laby]

I agree with @B1llyBlanco about some extra clarification about when to pass/fail. There is of course the obvious ones but then there are lot that are more where they got the job done but they look like they could use more practice to get more comfortable with the tools or to get faster, etc. I did a few of the ones where they were moving the ring over the tube, some of them weren't bad but not the best either, I passed them but left comments about where they could use more practice. Was that ok? Should I have just failed them instead?