Description
The rapid advancement of artificial intelligence (AI), particularly large language models (LLMs), has introduced powerful tools for various research domains, including psychological scale development. This study presents a fully automated method to efficiently generate and select high-quality, non-redundant items for psychological assessments using LLMs and network psychometrics. Our approach called, Automatic Item Generation and Validation via Network-Integrated Evaluation (AI-GENIE), reduces reliance on expert intervention by integrating generative AI with the latest network psychometric techniques. The efficacy of AI-GENIE was evaluated through Monte Carlo simulations using the Mixtral, Gemma 2, Llama 3, GPT 3.5, and GPT 4o models to generate item pools that mimic Big Five personality assessment. The results demonstrated improvement in item selection efficiency, with overall average increases of 9.78-17.80 in normalized mutual information in the final item pool across all models. After, each model in AI-GENIE generated a Big Five inventory that was administered to independent, representative samples (N = 1000 each) in the U.S. The empirical results show that the items produced across all models were diverse, theoretically consistent, and structurally stable. Taken together, these findings demonstrate that AI-GENIE is a highly effective tool to automate and streamline scale development and validation processes.
