Chat AIs can role-play humans in surveys and pilot studies

Researchers at the Finnish Center for Artificial Intelligence (FCAI) have utilized large language models (LLMs), specifically GPT-3, to generate AI-generated responses for studying video game player experience in human-computer interaction (HCI) research. These synthetic interviews provide a quick and cost-effective way to gather data and may aid in fast iteration and initial testing of study designs. However, it is important to validate findings based on AI-generated data with real data. The researchers discovered subtle differences in GPT-3 versions that impacted the diversity of AI-generated responses.

Furthermore, the use of LLMs raises concerns about the trustworthiness of crowdsourcing platforms like Amazon's Mechanical Turk (MTurk), as AI-generated responses are difficult to distinguish from real ones. The accessibility of LLMs may incentivize malicious users to employ bots and LLMs for generating high-quality fake responses. While synthetic data may have implications for anonymity and privacy, in HCI and scientific research, it can be useful for initial exploration and piloting of research ideas. However, when real data is needed, it may be necessary to reconsider the use of crowdsourcing platforms like MTurk.