IFiS PAN: seminar "Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization"

We would like to invite you to a new series of seminars organized by the Department of Computational Social Sciences (head: Prof. Artur Pokropek).
PAN–Metrics seminars cover the topics of methodology and measurement in the broadly defined social sciences.

The next meeting will be held online on Monday, May 19th at 4:00 p.m. (16:00) CET.

We are happy to announce that we will host Dr. AJ Alvero (Cornell University, NY, USA), who will give a presentation entitled:
Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization. 

Abstract:
The growing popularity of generative AI tools presents new challenges for data quality in online surveys and experiments. This study examines participants’ use of large language models to answer open-ended survey questions and describes empirical tendencies in human vs LLM-generated text responses. In an original survey of participants recruited from a popular online platform for sourcing social science research subjects, 34% reported using LLMs to help them answer open-ended survey questions. Simulations comparing human-written responses from three pre-ChatGPT studies with LLM-generated text reveal that LLM responses are more homogeneous and positive, particularly when they describe social groups in sensitive questions. These homogenization patterns may mask important underlying social variation in attitudes and beliefs among human subjects, raising concerns about data validity. Our findings shed light on the scope and potential consequences of participants’ LLM use in online research.

Bio:

AJ Alvero is a computational sociologist at the Cornell University Center for Data Science for Enterprise and Society with departmental affiliations in Sociology, Information Science, and Computer Science. Most of his research examines moments of high stakes evaluation, specifically college admissions and parole hearings. In doing so, he addresses questions and topics related to the sociological inquiry of artificial intelligence, culture, language, education, race and ethnicity, and organizational decision making. This work has appeared or is forthcoming in journals such as Science Advances, Poetics, The Oxford Handbook of the Sociology of Machine Learning, Sociological Methods & Research, Journal of Big Data, and other venues. AJ earned his PhD at Stanford University along with an MS in statistics.

The meeting is scheduled for around one hour.

Please contact us to receive the Zoom link (marek.muszynski@ifispan.edu.pl).