1.You guys should give some type of guideline as to what constitutes a pass and a fail. For example do they need to be at least all 3's to be passable because since this is a surgeon we are grading, is just average(3 out of 5) good enough? Surgery, at least how I see it is very important and you need to be great at it to be passed so I have been grading on the harsher side. Are these people just students and this is the first in many tests they will take to be qualified to do these procedures or is this the final test and we are saying that if we pass them that we think they are ready to do that actual surgery? I think this would make a difference in the way people rate the pass/fail part at least.
2. I think that just saying which of the two surgeons are better for the training isn't enough. I think you should show two video for each procedure, one that shows a proficient surgeon and shows the things that they are doing well and one that points out all the things that would be wrong or should be marked off. You should make people watch them once and then embed links to the videos so people can re-watch them whenever they need to.
3. What you were saying about making HITs for people who do a good job. You could have us enter our worker ID's on each HIT and take the time to look for workers who do a lot of your HITs and are marking the answers the correct way and then make a Qualification on Mturk for them. It may be time consuming at first, but youd have that worker pool on hand from then on and weed out the people you dont want. Or you could use the Mturk ID's to just track bad workers and give them Qualifications that if they have them, it denies them access to the HITs.
4. As someone mentioned earlier, it's unclear sometimes if you want us to write in the text boxes. Sometimes it's labeled as optional and other times it isnt. I would like clarification on that.
1. As far as the pass/fail thing goes I've always thought to myself, 'would I want this guy working on me or a loved one?' that usually fixes any confusion I might have.
"Are these people just students and this is the first in many tests they will take to be qualified to do these procedures or is this the final test and we are saying that if we pass them that we think they are ready to do that actual surgery?"
Sorta depends, if they're doing the classroom stuff like the blocks and snipping gauze and stuff I sorta think of it like 'do I think they're ready to try the next test, like turkey legs or cadavers?'
2. It's plenty. It's pretty easy to tell who's shaky with tools, when you see them nick an artery, etc. Although switching the video up 1x a month or so would probably be a good idea.
4. I think it's pretty clear, I'm not trying to be abrasive or anything but if it says 'optional' then it's optional, if it says 'required' then it's required.
