Data, AI systems, and society: Ethical dilemmas in the digital age

Abstract

Collecting and using biometric data in AI systems presents significant ethical, legal, and societal challenges. This paper critically examines the interplay between emergent AI technologies, normative ethics, and governance, drawing from an experiment on emotionally embedded cognition and applications used during the COVID-19 pandemic. Ethical and regulatory frameworks underscore the pivotal role of governance in mitigating risks associated with AI. The European Union’s Artificial Intelligence Act establishes critical boundaries by categorizing AI systems based on risk levels, prohibiting those deemed unacceptable, and imposing strict compliance requirements on high-risk systems, including biometric surveillance tools. The text explores key concerns such as data privacy, consent, and normative regulation, proposing an ethical framework grounded in regulation, digital literacy, and shared responsibility. This paper contributes to ongoing debates on ensuring ethical and equitable AI innovation by situating data governance within broader AI ethics.