Worker Viewpoints: Valuable Feedback for Microtask Designers in Crowdsourcing
Ryota Hayashi (University of Tsukuba), Nobuyuki Shimizu, Atsuyuki Morishima (University of Tsukuba)
IWSC 2017 (The International Workshop on Social Computing), 2017/4
- One of the problems a requester faces when crowdsourcing a microtask is that, due to the underspecifie or ambiguous task description, workers may misinterpret the microtask at hand. We call a set of such interpretations worker viewpoints. In this paper, we argue that assisting requesters to gather a worker’s interpretation of the microtask can help in providing useful feedback to designers, who may restate the task description if necessary. In our method, we create a corpus of viewpoints annotated with the types of viewpoints that reflec the logical structure embedded in them. Our experimental results suggest that the logic-oriented annotation is effective in choosing useful viewpoints from a possibly huge set of collected viewpoints, in the sense that removing viewpoints of particular types did not affect the quality of revised task instructions. We also show that the logic-oriented annotation can perform comparably with an entropy-based method, without several workers performing the same task in parallel.
Worker Viewpoints: Valuable Feedback for Microtask Designers in Crowdsourcing（External Site Link）