Rep Present Question–answer pairs and question answering

[Julian Michael]

Hi everyone, I'm new to the forums (and a relatively new requester). I have been posting HITs for writing question–answer pairs about English sentences, as well as answering them. Several people working on my task recommended I check out MTurk Crowd and introduce myself.

Since I can't post links yet, the HIT type IDs for my tasks are 36SUH4ZPJUVEFKCRRRIAVB1LGOZ705 (question–answer writing) and 3XRW87W7OXAP1BEXLSFQFDKLIPNTQ0 (question answering).

I'd be happy to receive any feedback on the task design, etc. I will also post about it in the daily work thread and I'm happy to have any more people join onto the task.. there is a bit of a backlog on the question answering task right now so there are lots of HITs available.

I will not be able to make huge changes to the design any time soon, but feedback would be really valuable for designing future tasks.

Thanks, and nice to meet you all!
Julian

 

[electrolyte]

Title: Answer simple questions about a sentence | PANDA Worker: Preview | Accept | Requester Requester: Julian Michael [A3AA2VKV87R6PG] (Contact) TO 1: Not Available TO 2: Not Available Description: Given a sentence and several questions about it, highlight the part of the sentence that answers each question, and mark questions that are invalid or redundant. Time: 10 minutes HITs Available: 346 Reward: $0.08 Qualifications: Question answering test score (%) is not less than 80; Question answering agreement rate (%) is not less than 70; HIT approval rate (%) is not less than 95; Location is US;

Title: Write question-answer pairs about a sentence's meaning | PANDA Worker: Preview | Accept | Requester Requester: Julian Michael [A3AA2VKV87R6PG] (Contact) TO 1: Not Available TO 2: Not Available Description: Given a sentence and some words from that sentence, write questions and answers involving each word. Write more question-answer pairs for increasing bonuses! Time: 10 minutes HITs Available: 145 Reward: $0.20 Qualifications: Question-answer generation accuracy (%) is not less than 75; HIT approval rate (%) is not less than 95; Location is US;

My export doesn't have working TO links, so for reference:
TO 1 - https://turkopticon.ucsd.edu/A3AA2VKV87R6PG
TO 2 - https://turkopticon.info/requesters/A3AA2VKV87R6PG

 

[electrolyte]

Hi! I see a qualification test to get a "Question answering test" score. How are you granting the "Question-answer generation accuracy" qualification?

 

[Julian Michael]

Hi! I see a qualification test to get a "Question answering test" score. How are you granting the "Question-answer generation accuracy" qualification?

It's auto-granted. This seems to be a source of confusion among workers. Should I write that in the task description? I figured this is how most requesters would do it..

 

[Julian Michael]

Maybe there is some nice little acronym like "(AG)" I can put in the qual name to show that it's auto-granted. Is there a common way of doing this?

 

[electrolyte]

It's auto-granted. This seems to be a source of confusion among workers. Should I write that in the task description? I figured this is how most requesters would do it..

Oh, that would probably be helpful. It's not too common to have autogranted quals, so most workers will be either looking for a qualification test or a batch of test HITs, or assuming the requester grants it after reviewing some completed HITs.

I don't think anyone would know what "AG" means but you could put "autogranted qual" in the title.

 

[Julian Michael]

Oh, that would probably be helpful. It's not too common to have autogranted quals, so most workers will be either looking for a qualification test or a batch of test HITs, or assuming the requester grants it after reviewing some completed HITs.

I don't think anyone would know what "AG" means but you could put "autogranted qual" in the title.

Ah I see, that makes sense. Ok, that is super helpful, thanks!

 

[Jharkan]

I thought I had aced the qual test, but scored lower than 30.
what?

 

[Julian Michael]

By the way.. how do I post one of those nicely formatted links to a HIT?

 

[Julian Michael]

I thought I had aced the qual test, but scored lower than 30.
what?

It's designed to get at the issues where the task instructions might contradict your intuition. It's possible that there is some ambiguity in the instructions though. Are there any parts of it that you find confusing?

 

[sryan66611]

By the way.. how do I post one of those nicely formatted links to a HIT?

https://greasyfork.org/en/scripts/911-mturk-great-hit-export

pretty sure its that

 

[sryan66611]

J @Julian Michael ur talking about this right ?/

Title: Answer simple questions about a sentence | PANDA Worker: Preview | Accept | Requester Requester: Julian Michael [A3AA2VKV87R6PG] (Contact) TO: [Pay: 4.80] [Fair: 4.60] [Comm: 5.00] [Fast: 5.00] [Reviews: 6] [ToS: 0] Description: Given a sentence and several questions about it, highlight the part of the sentence that answers each question, and mark questions that are invalid or redundant. Time: 10 minutes HITs Available: 350 Reward: $0.08 Qualifications: Question answering test score (%) GreaterThanOrEqualTo 80; Question answering agreement rate (%) GreaterThanOrEqualTo 70; HIT approval rate (%) GreaterThanOrEqualTo 95; Location EqualTo US;

 

[Jharkan]

It's designed to get at the issues where the task instructions might contradict your intuition. It's possible that there is some ambiguity in the instructions though. Are there any parts of it that you find confusing?

Yea, the questions' answers suggest that redundant questions are not wanted.
Given the answers I submitted, and the score I received, I can only guess that I misinterpreted the present material.

And there is no way to access the other hit that doesn't require the test, that I can see.

 

[Julian Michael]

J @Julian Michael ur talking about this right ?/

Title: Answer simple questions about a sentence | PANDA Worker: Preview | Accept | Requester Requester: Julian Michael [A3AA2VKV87R6PG] (Contact) TO: [Pay: 4.80] [Fair: 4.60] [Comm: 5.00] [Fast: 5.00] [Reviews: 6] [ToS: 0] Description: Given a sentence and several questions about it, highlight the part of the sentence that answers each question, and mark questions that are invalid or redundant. Time: 10 minutes HITs Available: 350 Reward: $0.08 Qualifications: Question answering test score (%) GreaterThanOrEqualTo 80; Question answering agreement rate (%) GreaterThanOrEqualTo 70; HIT approval rate (%) GreaterThanOrEqualTo 95; Location EqualTo US;

Yep, that's it. And thanks for the link!

 

[Julian Michael]

Yea, the questions' answers suggest that redundant questions are not wanted.
Given the answers I submitted, and the score I received, I can only guess that I misinterpreted the present material.

And there is no way to access the other hit that doesn't require the test, that I can see.

Can you not preview the HIT? I thought I set it to allow previewing without qualifications. Also it has an "accuracy" qual requirement which you have to request and which will be auto-granted.

It all worked as expected when I tested it on the sandbox. Unfortunately I can't test it on production because Amazon rejected my worker account registration.

 

[Jharkan]

Can you not preview the HIT? I thought I set it to allow previewing without qualifications. Also it has an "accuracy" qual requirement which you have to request and which will be auto-granted.

It all worked as expected when I tested it on the sandbox. Unfortunately I can't test it on production because Amazon rejected my worker account registration.

oh, er... yea, requesting it works.
Usually I don't hit the request, because nothing ever happens - it tells us that it emails the requester.
the end.
never hear from them again.

 

[sryan66611]

Can you not preview the HIT? I thought I set it to allow previewing without qualifications. Also it has an "accuracy" qual requirement which you have to request and which will be auto-granted.

It all worked as expected when I tested it on the sandbox. Unfortunately I can't test it on production because Amazon rejected my worker account registration.

Everything works as expected I took the qual yesterday and you can preview your hits without having the qual.

 

[electrolyte]

https://greasyfork.org/en/scripts/911-mturk-great-hit-export

pretty sure its that

That's it! Here's the longer explanation. http://www.mturkcrowd.com/threads/how-to-post-and-share-hits-on-mtc.35/

 

[Julian Michael]

oh, er... yea, requesting it works.
Usually I don't hit the request, because nothing ever happens - it tells us that it emails the requester.
the end.
never hear from them again.

Makes sense. I'll add some clarifications in the obvious places and have those on new uploads.

Everything works as expected I took the qual yesterday and you can preview your hits without having the qual.

Ok, good to have that clarified, thanks!

That's it! Here's the longer explanation. http://www.mturkcrowd.com/threads/how-to-post-and-share-hits-on-mtc.35/

And thank you too

 

[Jharkan]

The two hits completely contradict each other, but I can't even talk about it without discussing the qualification test on the one task.