Rep Present Question–answer pairs and question answering

[Julian Michael]

The two hits completely contradict each other, but I can't even talk about it without discussing the qualification test on the one task.

Could you please explain the issue? I would like to fix any ambiguities or problems if possible. I also want the qualification test to be reasonable. It is mainly meant to get people to make sure they fully understand the instructions before starting on the task so they don't bring down other people's agreement rates.

 

[VGB]

Hi everyone, I'm new to the forums (and a relatively new requester). I have been posting HITs for writing question–answer pairs about English sentences, as well as answering them. Several people working on my task recommended I check out MTurk Crowd and introduce myself.

Since I can't post links yet, the HIT type IDs for my tasks are 36SUH4ZPJUVEFKCRRRIAVB1LGOZ705 (question–answer writing) and 3XRW87W7OXAP1BEXLSFQFDKLIPNTQ0 (question answering).

I'd be happy to receive any feedback on the task design, etc. I will also post about it in the daily work thread and I'm happy to have any more people join onto the task.. there is a bit of a backlog on the question answering task right now so there are lots of HITs available.

I will not be able to make huge changes to the design any time soon, but feedback would be really valuable for designing future tasks.

Thanks, and nice to meet you all!
Julian

Welcome to the forum!

I've been doing these for the past couple of weeks and have no issues with the explanations, the instructions are quite thorough and go over everything you'd need to know.

From a task design standpoint, if there was a way to put the instructions into a collapsible box/frame it might help. Due to the sheer amount of them, it makes navigating the task a little harder than necessary once you understand how they are done.

Also a slight pay bump to the answering portion would be nice, .09 or .10. Some of the text requires some mental gymnastics to parse, which makes even simple 3-4 question answers take extra time. These are slightly mitigated by the occasional extremely simple snippet that can be done in seconds, though.

Overall they're quite enjoyable, and you've been friendly through the whole process, to boot.

 

[Julian Michael]

Welcome to the forum!

I've been doing these for the past couple of weeks and have no issues with the explanations, the instructions are quite thorough and go over everything you'd need to know.

From a task design standpoint, if there was a way to put the instructions into a collapsible box/frame it might help. Due to the sheer amount of them, it makes navigating the task a little harder than necessary once you understand how they are done.

Also a slight pay bump to the answering portion would be nice, .09 or .10. Some of the text requires some mental gymnastics to parse, which makes even simple 3-4 question answers take extra time. These are slightly mitigated by the occasional extremely simple snippet that can be done in seconds, though.

Overall they're quite enjoyable, and you've been friendly through the whole process, to boot.

Thanks!

That advice is helpful. I tried to calibrate the payment for answering so that the two tasks would go at about equal speed. However, the answering task has slowed down since I added the qualification so I was thinking of increasing it already... ok, I will do that in the next batch (once all the ones currently uploaded are completed).

Re: the instructions, I was a bit worried that making them collapsed by default would lead to more people not bothering to read them, or missing them. But I have a compromise solution: I'll just put them at the bottom.

 

[Jharkan]

Could you please explain the issue? I would like to fix any ambiguities or problems if possible. I also want the qualification test to be reasonable. It is mainly meant to get people to make sure they fully understand the instructions before starting on the task so they don't bring down other people's agreement rates.

well, I guess I'm just stupid,
if other people are able to do the hits

even though the test results I got suggest that you must include redundant sentences,
and then the task instructions themselves say not to include anything redundant.
so when I receive conflicting information like that,
and then see people claim to have no problems,
obviously there's a disconnect,
and I'm probably the common problem.

 

[humbleturker]

I'm just gonna say this is a properly difficult qual test. I'm not complaining, and I did a few of those HITs before they became qualled. It'll just make it more worth it when I eventually pass.

 

[kryss]

But I have a compromise solution: I'll just put them at the bottom.

It might also be worth your time to include some type of text (maybe a different color text to grab attention) at the top of the HIT specifying that there are detailed instructions at the bottom.

 

[WalkingEmphasis]

well, I guess I'm just stupid,
if other people are able to do the hits

even though the test results I got suggest that you must include redundant sentences,
and then the task instructions themselves say not to include anything redundant.
so when I receive conflicting information like that,
and then see people claim to have no problems,
obviously there's a disconnect,
and I'm probably the common problem.

I think you probably misunderstood something. I can't think of where in the task instructions you might have thought it said not to include redundant answers. The good thing is, you can retake the qualification if you still want to.

 

[Jharkan]

I think you probably misunderstood something. I can't think of where in the task instructions you might have thought it said not to include redundant answers. The good thing is, you can retake the qualification if you still want to.

6th requirement

doesn't matter, I redid the test, but can't get higher than 40
obviously, this task isn't for me.


Requirements
This task is best fit for native speakers of English. Your response must be grammatical, fluent English that satisfies the following criteria:

1. Either the question or the answer contains the special word.
2. The question contains at least one word from the sentence.
3. The question is about the meaning of the sentence (and not, for example, the order of the words).
4. The question is answered obviously and explicitly in the sentence.
5. The question is open-ended: yes/no and either/or questions are not allowed.
6. None of your question-answer pairs are redundant with each other, even for different special words.

 

[Julian Michael]

6th requirement

doesn't matter, I redid the test, but can't get higher than 40
obviously, this task isn't for me.
...
6. None of your question-answer pairs are redundant with each other, even for different special words.

What requirement 6 means is that as the question writer, you should not write redundant questions. So, as the question answerer, you should mark when questions are redundant so the question writer doesn't get rewarded for that question and so it counts against the question writer's accuracy.

I'll admit that the test is actually pretty tricky and I have to stop and think when taking it. The particular issues are about 1) how redundancy is defined in the task, and 2) what it means for a sentence to "obviously answer a question" by virtue of its meaning. On the former point, you have to be careful because questions that look similar might not actually count as redundant. On the latter point it's a pretty squishy issue and hard to explicitly write out any instructions without going into linguistics—but I try to illustrate it with a bunch of examples. Even then, there is a fair amount of room for disagreement, which is why I leave room for error on the qual test (and its scoring).

 

[Drwho10]

Hi everyone, I'm new to the forums (and a relatively new requester). I have been posting HITs for writing question–answer pairs about English sentences, as well as answering them. Several people working on my task recommended I check out MTurk Crowd and introduce myself.

Since I can't post links yet, the HIT type IDs for my tasks are 36SUH4ZPJUVEFKCRRRIAVB1LGOZ705 (question–answer writing) and 3XRW87W7OXAP1BEXLSFQFDKLIPNTQ0 (question answering).

I'd be happy to receive any feedback on the task design, etc. I will also post about it in the daily work thread and I'm happy to have any more people join onto the task.. there is a bit of a backlog on the question answering task right now so there are lots of HITs available.

I will not be able to make huge changes to the design any time soon, but feedback would be really valuable for designing future tasks.

Thanks, and nice to meet you all!
Julian

Good to see you in the forum. Your hits have been quite enjoyable and I look forward to doing more of them. The instructions are pretty straight forward and the pay is solid or could use a slight bump. Thanks for being such a communicative requester.

This is Kent M. from the emails by the way

 

[Julian Michael]

This is Kent M. from the emails by the way

Hi again! You were right, coming here was a good choice—I've already gotten some helpful feedback and once the remaining batch of questions is all answered I will kick off the task with improvements based on everyone's suggestions.

 

[Drwho10]

Hi again! You were right, coming here was a good choice—I've already gotten some helpful feedback and once the remaining batch of questions is all answered I will kick off the task with improvements based on everyone's suggestions.

Good to hear. This is one of the best forums in the mturk community I'm sure you'll get even better quality work now.

 

[CatWhiskers]

I think you probably misunderstood something. I can't think of where in the task instructions you might have thought it said not to include redundant answers. The good thing is, you can retake the qualification if you still want to.

When? I took it and got 57 so I must have misunderstood something, but I see no way to retake it.

 

[Julian Michael]

When? I took it and got 57 so I must have misunderstood something, but I see no way to retake it.

It should have a 5 minute retry delay. Just do the same thing you did to take it the first time.

 

[T. Leela]

I've enjoyed these hits, I did the very first batch, and a few of the next, just missed the qual but am going to try again.
Right now actually.
Thank you

 

[T. Leela]

sorry wrong thread :oops:

 

[sryan66611]

Just a heads up... People doing the questions if you change the name around or leave it out and call it "what" or other generalization as per the instructions - In the what since? --> <Invalid>The question writer simply replaced a word with "what" instead of using it to form a proper English question.

I have seen this a lot the past few days and just wanted to let everyone know so they don't lose their qual.