This was one of most interesting and challenging datasets I explored. The original dataset was collected by a brilliant team of researchers over at CMU. If you want to see how awesome they are, check out their video describing the data:
Here’s the description provided by Jordan Boyd-Graber:
Machine learning techniques to detect deception in online communications requires training and evaluation data. However, there is a dearth of data either because of uncertain gold labels or privacy concerns; we create a new, large deception-centered dataset in the online game of Diplomacy. We gathered 17,289 messages from 12 games (each of which took over a month) involving 84 players, the majority of which were unique users. This data was collected with a custom-made bot that allowed us to collect messages and annotations. The user pool was created from scratch: we varied participant demographics across gender, age, nationality, and past game experience. Some of our participants included the former president of the Diplomacy players’ association, several top ranked players in the world, a board game shop owner, and scientists. We create machine learning models to detect lies using linguistic, context, and power-dynamic features. Our best model had similar lie detection accuracy to humans. Paper: http://users.umiacs.umd.edu/~jbg//doc…
So yeah, we got this data and began to explore it. We were mainly surprised by how limited the recent literature was, on online cooperation. The headlining papers at CL conferences have focused on detecting multimodal affect using deep learning networks. This challenge — our challenge — of detecting the fine nuances of negotiation in online teams — was last properly touched in the 1990s when semantic frames were still a thing and natural language generation was called “machinery for Artificial Language” 😀 thank you Candace L. Sidner for a fabulous paper on the topic, which really kickstarted our own conceptualization.
Ultimately, we only released a teaser for the Shared Task, but further investigation is ongoing, I promise! There’s a lot to be gauged from how complete strangers please, flatter, and persuade each other to gain ground, wage wars, and forge alliances. Here’s a sample of the kind of things they say: