論 文Papers

WORKSHOP (INTERNATIONAL)

Worker Viewpoints: Valuable Feedback for Microtask Designers in Crowdsourcing

Ryota Hayashi (University of Tsukuba), Nobuyuki Shimizu, Atsuyuki Morishima (University of Tsukuba)

IWSC 2017 (The International Workshop on Social Computing), 2017/4

Category:

クラウドソーシング (Crowdsourcing)

Abstract:
One of the problems a requester faces when crowdsourcing a microtask is that, due to the underspecifie or ambiguous task description, workers may misinterpret the microtask at hand. We call a set of such interpretations worker viewpoints. In this paper, we argue that assisting requesters to gather a worker’s interpretation of the microtask can help in providing useful feedback to designers, who may restate the task description if necessary. In our method, we create a corpus of viewpoints annotated with the types of viewpoints that reflec the logical structure embedded in them. Our experimental results suggest that the logic-oriented annotation is effective in choosing useful viewpoints from a possibly huge set of collected viewpoints, in the sense that removing viewpoints of particular types did not affect the quality of revised task instructions. We also show that the logic-oriented annotation can perform comparably with an entropy-based method, without several workers performing the same task in parallel.
Download:

Worker Viewpoints: Valuable Feedback for Microtask Designers in Crowdsourcing(外部サイト/External Site Link)