Linguistic alignment with an artificial agent: A commentary and re-analysis in the spirit of collaborative Open Science

Encouraged by recent advancements in Open Science, in this manuscript we provide a commentary and a complementary analysis of Cirillo et al.'s (2022) study on conceptual alignment in a joint picture naming task involving a social robot (Cognition, 227, 105213). In their study, Cirillo and collaborators present evidence suggesting automatic alignment by examining response proportions, reflecting adaptation to the lexical choices made by the artificial agent (i.e., providing category names instead of basic names for specific semantic categories). Here, we conducted a complementary analysis using the openly available dataset, employing a multiverse approach and focusing on response times as a more nuanced measure of cognitive processing and automaticity. Our findings indicate that alignment in the Category condition (i.e., when the robot provided a superordinate label) is associated with longer response times and greater variability. When providing the Basic name in the Basic condition, RTs are much shorter and variability is reduced, compatible with the Basic-level advantage phenomenon. Non-alignment to each condition completely reverses the pattern. This suggests that aligning when producing a superordinate label is a strategic and effortful rather than an automatic response mechanism. Furthermore, through comprehensive visual exploration of response proportions across categories and lexical frequency of pictures’ basic labels, we observed category naming alignment primarily emerging in specific categories and mostly for stimuli at low lexical frequency and newly designed pictures not taken from the MultiPic database, thus suggesting a limited generalizability of the effect. These insights were confirmed using leave-one-out robustness checks. In conclusion, our contribution provides complementary evidence in support of strategic rather than automatic responses when aligning with Category labels in the analyzed dataset, with a limited generalizability despite all the balancing procedures the authors carefully implemented in the experimental material. This is likely to reflect individual task strategies rather than genuine alignment. We ultimately advocate open data sharing and collective methodological discussion to foster constructive dialogue and critique, aiming to support early career researchers amidst the uncertainties inherent in scientific inquiry, underscoring the importance of a collaborative Open Science approach.