Response from GSB about double checking data and adjusting qual scores:
Thank you for the suggestion. I'm currently in the process of doing these checks. The agreement rate is generally quite high, but I think the most recent batch was just more confusing to people since there are many more cases of the data not being found in the documents. I partially expected this since these documents gave me trouble when I tried finding the approximate location of the data with an algorithm that I wrote.
Thanks again for your feedback as it is very valuable to us. We want to find the best and most efficient way to work with a large group of MTurkers so it's nice to get some insight.