Requester Question to MTurkers on Alternatives to HIT Pay Reduction

Annabel

New Member
Requester
Joined
Oct 13, 2016
Messages
11
Reaction score
37
Points
13
Gender
Female
Hi! My name is Annabel, and I'm a requester on MTurk (http://turkopticon.ucsd.edu/A3HB3KPJ2RZ2K8). I'm on here to ask you guys how I can make my HITs more efficient so I can keep running them. Right now, my funding is becoming a bit limited, but our project still needs a lot more information.

I've decreased the amount of pay for my HITs in the past before, but I'd like to avoid decreasing the amount of pay again, especially since I'd like to keep a good relationship with the people who've been working on my HITs. Each HIT currently pays $0.50. I was advised to lower this amount to $0.20, but I am trying to avoid this.

Right now, my HITs revolve around asking people to compare two sentences with each other, and to select the sentence that is the more important one. (Title: Comparing two statements with each other) The goal of these HITs is to collect enough information to create an app or a website that can automatically read the Terms & Conditions and give people only the important parts.

I do have an idea to address this (change the task so that people compare groups of sentences with each other so we get more data, and slightly increase the base pay), but was wondering whether there were other ways out there.
 
Last edited:

Jaded

The real themildone
Administrator
Joined
Jan 10, 2016
Messages
46,610
Reaction score
123,816
Points
1,414
Age
123
Gender
Female
First, I think it's awesome that you don't want to lower the pay again. I know last time I did your hits (it's been a while since I've seen them,) it was one question per page which caused a bit of loading time. If there were more questions per page (and fewer pages) it might make them quicker so that a pay reduction wouldn't really impact how much one was earning from the hits.

Hopefully, that makes sense.
 

LurkyTurkey

New Member
Contributor
Joined
Jan 13, 2016
Messages
97
Reaction score
149
Points
308
Welcome! I recall working on these HITs before and some of the sentences were equally ambiguous, making it hard to give a useful answer. Clauses would help address that, but it might also be useful to specify what is relevant.

If you want to PM me an example or link to refresh my memory, I can give more specific feedback. I do recall the T&C weren't the typical ones I encounter as a shopper. It seems like they included info about subscriptions and maybe even litigation.
 

chuchotant

I made that batch faaamous
Joined
Jan 22, 2016
Messages
966
Reaction score
2,243
Points
493
Age
46
Hi A @Annabel . I just said to my mom this morning, I miss Annabel. Remember when it was Annabel weekend. funny that you should sign up! Welcome. And thank you so much for thinking of the workers and reaching out to us!

I remember specifically thinking that it would be easier if all of the 10 binary comparisons were on one page (like Jaded @Jaded suggested). Another thing, some comparisons seemed to be made between really inconsequential information (ie "These are the terms and conditions") and really important legal stuff (ie statements about arbitration etc). I'm not sure if you have enough data already, but to group the inconsequential statements with each other so that I am making a choice between in consequential statements and group the heavy stuff together so I am making a choice between heavy stuff.
I felt like I wasnt being really that helpful to you if I chose an obviously more important statement when the only other option was a statement just telling me these were the terms and conditions.
In other words, here was the choice I was making on many of the choices before:

These are the TOC vs Important Legal Stuff

Whereas if you could group them:
These are TOC vs Customer service number is 800-
Important Legal Stuff vs Important Legal Stuff

I hope this makes sense.
 

LurkyTurkey

New Member
Contributor
Joined
Jan 13, 2016
Messages
97
Reaction score
149
Points
308
Hi A @Annabel . I just said to my mom this morning, I miss Annabel. Remember when it was Annabel weekend. funny that you should sign up! Welcome. And thank you so much for thinking of the workers and reaching out to us!

I remember specifically thinking that it would be easier if all of the 10 binary comparisons were on one page (like Jaded @Jaded suggested). Another thing, some comparisons seemed to be made between really inconsequential information (ie "These are the terms and conditions") and really important legal stuff (ie statements about arbitration etc). I'm not sure if you have enough data already, but to group the inconsequential statements with each other so that I am making a choice between in consequential statements and group the heavy stuff together so I am making a choice between heavy stuff.
I felt like I wasnt being really that helpful to you if I chose an obviously more important statement when the only other option was a statement just telling me these were the terms and conditions.
In other words, here was the choice I was making on many of the choices before:

These are the TOC vs Important Legal Stuff

Whereas if you could group them:
These are TOC vs Customer service number is 800-
Important Legal Stuff vs Important Legal Stuff

I hope this makes sense.
Yes. This seems like it would yield more substantial, useful answers and perhaps avoid a pay reduction by generating better results.
 

Annabel

New Member
Requester
Joined
Oct 13, 2016
Messages
11
Reaction score
37
Points
13
Gender
Female
I should also mention, I'll be outputting another batch this weekend (at the same price) to hopefully reach out to all of the MTurkers who usually work on my HITs to ask for their input too. Feel free to just repeat your suggestions there in the box if you decide to do it!

First, I think it's awesome that you don't want to lower the pay again. I know last time I did your hits (it's been a while since I've seen them,) it was one question per page which caused a bit of loading time. If there were more questions per page (and fewer pages) it might make them quicker so that a pay reduction wouldn't really impact how much one was earning from the hits.

Hopefully, that makes sense.
Oh! Good point. I'll list creating just one page full of comparisons as an option.

Welcome! I recall working on these HITs before and some of the sentences were equally ambiguous, making it hard to give a useful answer. Clauses would help address that, but it might also be useful to specify what is relevant.

If you want to PM me an example or link to refresh my memory, I can give more specific feedback. I do recall the T&C weren't the typical ones I encounter as a shopper. It seems like they included info about subscriptions and maybe even litigation.
Thank you!

The link to the website is right here: http://crowdverify.herokuapp.com/
Generating a list of similarly important sentences might be a bit tricky, especially since the project is about figuring out what consumers care about!
 
Last edited:

Annabel

New Member
Requester
Joined
Oct 13, 2016
Messages
11
Reaction score
37
Points
13
Gender
Female
Forgot to respond to the other comments! Also, I'll be outputting another batch this weekend (at the same price) to hopefully reach out to all of the MTurkers who usually work on my HITs to ask for their input too. Feel free to just repeat your suggestions there in the box if you decide to do it!

Hi A @Annabel . I just said to my mom this morning, I miss Annabel. Remember when it was Annabel weekend. funny that you should sign up! Welcome. And thank you so much for thinking of the workers and reaching out to us!

I remember specifically thinking that it would be easier if all of the 10 binary comparisons were on one page (like Jaded @Jaded suggested). Another thing, some comparisons seemed to be made between really inconsequential information (ie "These are the terms and conditions") and really important legal stuff (ie statements about arbitration etc). I'm not sure if you have enough data already, but to group the inconsequential statements with each other so that I am making a choice between in consequential statements and group the heavy stuff together so I am making a choice between heavy stuff.
I felt like I wasnt being really that helpful to you if I chose an obviously more important statement when the only other option was a statement just telling me these were the terms and conditions.
In other words, here was the choice I was making on many of the choices before:

These are the TOC vs Important Legal Stuff

Whereas if you could group them:
These are TOC vs Customer service number is 800-
Important Legal Stuff vs Important Legal Stuff

I hope this makes sense.
What a coincidence! :D One of your admins ( Jaded @Jaded) reached out to me about this site earlier today, and of course I had to sign up!

And thank you for your kind words! I'll see if I can put them all on one page. As for the inconsequential items, as mentioned in response to an earlier comment, that is a bit tricky since we're trying to figure out what people find important with this round of studies. For the obvious cases like 1-800 number, that shouldn't be a problem, but for other stuff, that might not be possible :(

I could go more into detail about why, but I'm not quite sure if anyone wants to read a really long post on it!
 
  • Like
Reactions: Jaded

Jaded

The real themildone
Administrator
Joined
Jan 10, 2016
Messages
46,610
Reaction score
123,816
Points
1,414
Age
123
Gender
Female
I could go more into detail about why, but I'm not quite sure if anyone wants to read a really long post on it!
You'd be surprised!
 

Annabel

New Member
Requester
Joined
Oct 13, 2016
Messages
11
Reaction score
37
Points
13
Gender
Female
You'd be surprised!
Hm. Let me think of a way to shorten the explanation then -

In order to create HITs where the sentences in the comparison are either both unimportant or important, we would need to figure out a way to know how important the sentences are in the first place.

Solution #1: The super long, slow way is for me to read all of the sentences, and then choose which ones to line up in a HIT. (Probably not possible since people could argue that what I think is important isn't actually that important to other people.)

Solution #2: The less slow way is to have something like a competitive ranking system, where sentences that aren't selected as important over other sentences are assigned a lower score. (We would need to code the website differently which might take a bit of time! We are considering using this though)

Solution #3: Create a computer program that can automatically detect what's important or not, and then create HITs based off of how important those sentences are.

There are two main ways of doing this:
1) Sentence matching: This means that we look at how important a sentence is judged to be in one Terms & Conditions policy, and search for all sentences that are the same or close to the same in other Terms & Conditions policies. One potential problem here is that a sentence that is judged as important in one Terms & Conditions may not actually be important in another one. We are trying to address this issue right now actually. Not very easy.
2) Machine-learning algorithms: This means that we use existing algorithms to figure out how important a sentence is. This is more accurate than sentence matching, but needs a lot of data to work (which we are trying to collect).
 

chuchotant

I made that batch faaamous
Joined
Jan 22, 2016
Messages
966
Reaction score
2,243
Points
493
Age
46
Hm. Let me think of a way to shorten the explanation then -

In order to create HITs where the sentences in the comparison are either both unimportant or important, we would need to figure out a way to know how important the sentences are in the first place.

Solution #1: The super long, slow way is for me to read all of the sentences, and then choose which ones to line up in a HIT. (Probably not possible since people could argue that what I think is important isn't actually that important to other people.)

Solution #2: The less slow way is to have something like a competitive ranking system, where sentences that aren't selected as important over other sentences are assigned a lower score. (We would need to code the website differently which might take a bit of time! We are considering using this though)

Solution #3: Create a computer program that can automatically detect what's important or not, and then create HITs based off of how important those sentences are.

There are two main ways of doing this:
1) Sentence matching: This means that we look at how important a sentence is judged to be in one Terms & Conditions policy, and search for all sentences that are the same or close to the same in other Terms & Conditions policies. One potential problem here is that a sentence that is judged as important in one Terms & Conditions may not actually be important in another one. We are trying to address this issue right now actually. Not very easy.
2) Machine-learning algorithms: This means that we use existing algorithms to figure out how important a sentence is. This is more accurate than sentence matching, but needs a lot of data to work (which we are trying to collect).
Hi A @Annabel
I just did your new set of hits and noticed a few changes which are a bit different than the last batch!
Question:
On the solution 2 way: For comparisons that are not deemed equally important (where a choice is actually made) could a question be generated to ask what score on a scale of 1-5 the respondent would give each sentence. Then all the same numbers could be grouped together for comparisons. You would get a variety of what some people view as a 5 vs others, but then the 5's could be paired off against each other (tournament style, lol) etc. Is that what you were thinking? We as workers could add some more value to the comparison making that way.

On the solution 3 type 1 way, that was sort of a question I emailed you (I emailed you because I didnt know that question 2 was mandatory in your hit). I noticed the names of the companies were replaced with "e-commerce" site, but it does make a difference when I consider the type of company (retailer vs content publisher) that the terms apply to.

Hopefully you get some more really good suggestions!
 

Itsmee

Condor
Contributor
Joined
Jan 20, 2016
Messages
2,007
Reaction score
6,395
Points
838
Age
66
Gender
Male
Server would only let me advance to the 2nd set of comparisons before the crash.... Refreshed and refreshed...We will help you figure something out.
 

Annabel

New Member
Requester
Joined
Oct 13, 2016
Messages
11
Reaction score
37
Points
13
Gender
Female
Hi A @Annabel
I just did your new set of hits and noticed a few changes which are a bit different than the last batch!
Question:
On the solution 2 way: For comparisons that are not deemed equally important (where a choice is actually made) could a question be generated to ask what score on a scale of 1-5 the respondent would give each sentence. Then all the same numbers could be grouped together for comparisons. You would get a variety of what some people view as a 5 vs others, but then the 5's could be paired off against each other (tournament style, lol) etc. Is that what you were thinking? We as workers could add some more value to the comparison making that way.

On the solution 3 type 1 way, that was sort of a question I emailed you (I emailed you because I didnt know that question 2 was mandatory in your hit). I noticed the names of the companies were replaced with "e-commerce" site, but it does make a difference when I consider the type of company (retailer vs content publisher) that the terms apply to.

Hopefully you get some more really good suggestions!
Hm, that could be a way to do it! I will list this as an option! Thank you!

Server would only let me advance to the 2nd set of comparisons before the crash.... Refreshed and refreshed...We will help you figure something out.
Thank you! I will try and figure out what the problem is once the batch expires.
 
  • Like
Reactions: Itsmee

chuchotant

I made that batch faaamous
Joined
Jan 22, 2016
Messages
966
Reaction score
2,243
Points
493
Age
46
Server would only let me advance to the 2nd set of comparisons before the crash.... Refreshed and refreshed...We will help you figure something out.
Instead of refreshing when the server crashed, I just exxed out of the screeen and hit the link (in the mturk window) again. It put me right back on the page and at the number of completed comparisons. Hope that helps!
 
  • Like
Reactions: Annabel and Itsmee

Itsmee

Condor
Contributor
Joined
Jan 20, 2016
Messages
2,007
Reaction score
6,395
Points
838
Age
66
Gender
Male
Instead of refreshing when the server crashed, I just exxed out of the screeen and hit the link (in the mturk window) again. It put me right back on the page and at the number of completed comparisons. Hope that helps!
I tried that and it didn't work for me. I was using Firefox...
 

Annabel

New Member
Requester
Joined
Oct 13, 2016
Messages
11
Reaction score
37
Points
13
Gender
Female
I tried that and it didn't work for me. I was using Firefox...
It's probably either one of two things (none of which are your fault):
1) Too many people on at the same time, so I would need to get more servers
2) I coded the website wrong when trying to get it to compare statements from different websites!

I think it's probably the second one, but will make sure once the HIT has been taken by everyone!
 
  • Like
Reactions: chuchotant

Itsmee

Condor
Contributor
Joined
Jan 20, 2016
Messages
2,007
Reaction score
6,395
Points
838
Age
66
Gender
Male
It's probably either one of two things (none of which are your fault):
1) Too many people on at the same time, so I would need to get more servers
2) I coded the website wrong when trying to get it to compare statements from different websites!

I think it's probably the second one, but will make sure once the HIT has been taken by everyone!
Thank you A @Annabel
 
  • Like
Reactions: Annabel

chuchotant

I made that batch faaamous
Joined
Jan 22, 2016
Messages
966
Reaction score
2,243
Points
493
Age
46
Hi A @Annabel ,
Not sure if you know, but when I enter the link to do a new set of 40, a message pops up saying You're done. I know I'm not done, and I go ahead and do my 40, but wanted to alert you that it comes up. thanks!
 

Annabel

New Member
Requester
Joined
Oct 13, 2016
Messages
11
Reaction score
37
Points
13
Gender
Female
Hi A @Annabel ,
Not sure if you know, but when I enter the link to do a new set of 40, a message pops up saying You're done. I know I'm not done, and I go ahead and do my 40, but wanted to alert you that it comes up. thanks!
Thanks for letting me know! Yeah, it's a message that's carried over from the last time you completed the HIT! Someone did reach out to me with a solution for that, but I've been a bit busy lately and haven't been able to use it yet..
 
  • Like
Reactions: chuchotant