So here's what happened with that qualification test mess yesterday:
The idea was to have detailed training and a test that would take half an hour to an hour, and use a qualification to record the score on the test. I felt this was too much to ask for without paying people, so I put a batch of do-nothing HITs behind the qual so people could get paid for them after the test. We only wanted to qualify 25-50 people or so to start and the limited number of HIT assignments was supposed to accomplish this.
The first problem was that you could hit the "get paid for trying" threshold by answering "yes" to every question. There were a lot of these submissions. I updated the test figuring that retaking it wouldn't be a big deal since it wouldn't take nearly as long as the first time. This caused a bunch of confusion about what score was needed, with 4 different numbers floating around.
Second problem: I assumed HIT assignments would be held while people took the test. They aren't. So a lot of people took the test with the promise of a paying HIT behind it which then disappeared. I spent most of the last work day trying to get these people paid.
The third problem, and ultimately the reason we pulled the qualification, is that there's no way to close the test except to delete the qualification and copy everyone's scores to a new one. As some of you discovered, this means it gets deleted while people are in the middle of the test. This is even worse since I have no way to verify who this happened to and I can't just be a money piñata.
We're looking for a small enough pool of qualified workers that I should have just done a normal qualifying batch and processed it manually. Sorry for the frustration and drama it stirred up. For the record, I'm not aware of any answers having been shared.