Exploring the Potential Implications of AI-generated Content in Social Engineering Attacks

Abstract

The evolution of artificial intelligence (AI) and machine learning presents both utility and security implications for our digital interactions. This study focuses on the transformative role of generative AI in social engineering attacks, specifically examining three pillars where it significantly amplifies their impact: advanced targeting and personification, genuine content creation, and automated attack infrastructure. The analysis forms a conceptual model named the generative AI social engineering framework. The research delves into human implications and measures to counter social engineering attacks, blending theoretical analysis with practical insights through case studies. Ethical considerations surrounding AI in malicious activities are discussed, emphasizing the importance of safe AI development, and various articles were reviewed to highlight social engineering attacks as a common threat. Two studies were conducted: a user testing study with 48 participants from diverse occupations and social engineering awareness, and an exploratory study collecting qualitative data from 40 social engineering attack victims. The user testing study revealed universal acceptance of the AI-based tool, irrespective of participants’ occupations. Victim themes included reasons for falling prey to attacks, methods, prevention advice, and detection. The research concludes by highlighting AI-generated content as a key factor fueling social engineering attacks and bridging the gap between AI development and cybersecurity practices, highlighting the need for interdisciplinary approaches to address evolving challenges.