Artificial intelligence versus journalists: The quality of automated news and bias by authorship using a Turing test

Authors

Abstract

The integration of Artificial Intelligence (AI) in the media results in the publication of thousands of automated news articles in Spanish every day. This study uses a Turing test to compare the quality of news articles written by professional journalists (from Efe) with those produced by natural language generation (NLG) software (from Narrativa). Based on Sundar’s dimensions (1999) crucial to news perception – credibility, readability and journalistic expertise – , an internationally validated experimental methodology is employed, exploring a novel topic in Spanish: health information. The experiment deliberately varied real and declared authorships – AI and human journalists – to detect potential biases in assessing authorship credibility. A self-administered questionnaire adapted for online surveys was used (N=222), and gender imbalances were minimized to ensure gender equality in the sample (N=128). The study reveals that there are no significant differences between news articles generated by the AI and those written by professional journalists. Both types of news are considered equally credible, though some biases are detected in the evaluation of declared authorship: the AI author is perceived as more believable than the human, while the human journalist is perceived as creating a more lively narrative. The study concludes that it is feasible to produce automated news in Spanish without compromising its quality. In the global media landscape, automated systems employing NLG, machine learning and sophisticated databases successfully advance into new domains such as health information.

Keywords

automated journalism, automated news, artificial intelligence, Turing test, COVID-19

References

ARGUEDAS, A.R. and SIMON, F.M. (2023). Automating Democracy: Generative AI, journalism, and the future of democracy. Oxford: Oxford Internet Institute. https://doi.org/10.5287/ora-e262xv7no

BECKETT C. and YASEEN, M. (2023). Generating Change. A global survey of what news organisations are doing with AI. London: LSE. Retrieved from https://www.journalismai.info/research/2023-generating-change

CALVO-RUBIO, L.-M. and UFARTE-RUIZ, M.-J. (2020). “Percepción de docentes universitarios, estudiantes, responsables de innovación y periodistas sobre el uso de inteligencia artificial en periodismo”. Profesional de la información, 29 (1). https://doi.org/10.3145/epi.2020.ene.09

CARLSON, M. (2015). “The Robotic Reporter”. Digital Journalism, 3 (3), 416-431. https://doi.org/10.1080/21670811.2014.976412

CHANG, Y., WANG, X., WANG, J., WU, Y., YANG, L., ZHU, K., CHEN, H., YI, X., WANG, C., WANG, Y., YE, W., ZHANG, Y., CHANG, Y., YU, P.S., YANG, Q. and XIE, X. (2023). “A Survey on Evaluation of Large Language Models”. ArXiv preprint arXiv:2307.03109v7, 1-26. https://doi.org/10.48550/arXiv.2307.03109

CLERWALL, C. (2014). “Enter the Robot Journalist. Users’ perceptions of automated content”. Journalism Practice, 8 (5), 519-531. https://doi.org/10.1080/17512786.2014.883116

EUROPEAN COMMISSION (2018). A definition of Artificial Intelligence. Brussels: European Commission. Retrieved from https://ec.europa.eu/futurium/en/system/files/ged/ai_hleg_definition_of_ai_18_december_1.pdf

DANZON-CHAMBAUD, S. (2021). “Tow Report. Covering COVID-19 with automated news”. Columbia Journalism Review. Retrieved from https://www.cjr.org/tow_center_reports/covering-covid-automated-news.php

DANZON-CHAMBAUD, S. (2023). Automated news in practice: Changing the journalistic doxa during COVID-19, at the BBC and across media organizations [Doctoral thesis, School of Communications Dublin City University]. Doras. Retrieved from https://doras.dcu.ie/28004/1/DEPOSIT_JANUARY_2023_Danzon-Chambaud_Automated_News_Practice_PhD_Thesis.pdf

DIAKOPOULOS, N. (2019). Automating the News. MA: Harvard University Press.

DOMINGOS, P. (2015). The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. St Ives: Allen Lane.

DÖRR, K.N. (2016). “Mapping the Field of Algorithmic Journalism”. Digital Journalism, 4 (6), 700-722. https://doi.org/10.1080/21670811.2015.1096748

EFRON, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. California: Standford University. https://doi.org/10.1137/1.9781611970319

FANTA, A. (2017). Putting Europe’s Robots on the Map: Automated journalism in news agencies. Oxford: Reuters Institute for the Study of Journalism.

FLANAGIN, A.J. and METZGER, M.J. (2000). “Perceptions of Internet Information Credibility”. Journalism & Mass Communication Quarterly, 77 (3), 515-540. https://doi.org/10.1177/107769900007700304

GRAEFE, A. and BOHLKEN, N. (2020). “Automated journalism: A meta-analysis of readers’ perceptions of human-written in comparison to automated news”. Media and Communication, 8 (3), 50-59. http://dx.doi.org/10.17645/mac.v8i3.3019

GRAEFE, A., HAIM, M., HAARMANN, B. and BROSIUS, H.-B. (2018). “Reader’s perception of computer-generated news: credibility, expertise, and readability”. Journalism, 19 (5), 595-610. https://doi.org/10.1177/1464884916641269

GUISADO-CLAVERO M., ASTIER-PEÑA, M.P., GÓMEZ-BRAVO, R. and ARES-BLANCO, S. (2022). “Open data for monitoring COVID-19 in Spain: descriptive study”. Enfermedades Infecciosas y Microbiología Clínica, 42 (2), 80-87. https://doi.org/10.1016/j.eimc.2022.10.002

HAIM, M. and GRAEFE, A. (2017). “Automated News: Better than expected?”. Digital Journalism, 5 (8), 1044-1059. https://doi.org/10.1080/21670811.2017.1345643

INSTITUTO CERVANTES (2022). El español: una lengua viva. Informe 2022. Retrieved from https://cvc.cervantes.es/lengua/espanol_lengua_viva/pdf/espanol_lengua_viva_2022.pdf

JIA, C. (2020). “Chinese automated journalism: A comparison between expectations and perceived quality”. International Journal of Communication, 14, 2611-2632. Retrieved from https://ijoc.org/index.php/ijoc/article/download/13334/3083

JIA, C. and JOHNSON, T. (2021). “Source Credibility Matters: Does Automated Journalism Inspire Selective Exposure?”. International Journal of Communication, 15, 3760-3781.

JUNG, J., SONG, H., KIM, Y., IM, H. and OH, S. (2017). “Intrusion of software robots into journalism: the public’s and journalists’ perceptions of news written by algorithms and human journalists”. Computers in Human Behavior, 71, 291-298. https://doi.org/10.1016/j.chb.2017.02.022

LATAR, N. (2014). “The robot journalist in the age of social physics: the end of human journalism?”. In: G. EINAV (Ed.). The New World of Transitioned Media. New York: Springer International Publishing, 65-80. https://doi.org/10.1007/978-3-319-09009-2_6

LATAR, N. and NORDFORS, D. (2009). “Digital Identities and Journalism Content: How Artificial Intelligence and Journalism May Co-Develop and Why Society Should Care”. Innovation Journalism, 6 (7).

LECOMPTE, C. (2015). “Automation in the Newsroom”. Nieman Reports, 69 (3), 32-45. Retrieved from http://niemanreports.org/wp-content/uploads/2015/08/NRsummer2015.pdf

LEE, S.M. and KIM, T.Y. (1998). “A news on demand service system based on robot agent”. In: International Conference on Parallel and Distributed Systems Proceedings, 528-532. https://doi.org/10.1109/ICPADS.1998.741128

LERMANN-HENESTROSA, L., GREVING, H., KIMMERLE, J. (2023). “Automated journalism: The effects of AI authorship and evaluative information on the perception of a science journalism article”. Computers in Human Behavior, 138. https://doi.org/10.1016/j.chb.2022.107445

LINDEN, C. and TUULONEN, H. (2019). News Automation. The rewards, risks and realities of ‘machine journalism’. WAN-IFRA. Retrieved from https://wan-ifra.org/insight/report-news-automation-the-rewards-risks-and-realities-of-machine-journalism/

MATSUMOTO, R., NAKAYAMA, H., HARADA, T. and KUNIYOSHI, Y. (2007). “Journalist Robot: Robot System Making News Articles from Real World”. In: 2007 IEEE Intl conf on robotics and automation, 1234-1241. https://doi.org/10.1109/IROS.2007.4399598

MEEHAN, J.R. (1977). “Tale-Spin, an Interactive Programme That Writes Stories”. In: Proceedings of the Fifth International Joint Conference on Artificial Intelligence, 91-98. Retrieved from https://www.ijcai.org/Proceedings/77-1/Papers/013.pdf

MEYER, P. (1973). Precision journalism: A reporter’s introduction to social science methods. Bloomington: Indiana University Press.

MEYER, P. (1988). “Defining and measuring credibility of newspapers: Developing an index”. Journalism & Mass Communication Quarterly, 65 (3), 567-574. https://doi.org/10.1177/107769908806500301

MORAN, R.E. and SHAIKH, S.J. (2022). “Robots in the news and newsrooms: Unpacking meta-journalistic discourse on the use of artificial intelligence in journalism”. Digital Journalism, 10 (10), 1756-1774. https://doi.org/10.1080/21670811.2022.2085129

MORAVEC, V., MACKOVÁ, V., SIDO, J. and EKŠTEIN, K. (2020). “The Robotic Reporter in The Czech News Agency: Automated Journalism and Augmentation in the Newsroom”. Communication Today, 11 (1), 37-52. Retrieved from https://communicationtoday.sk/wp-content/uploads/03_MORAVEC-et-al_CT-1-2020.pdf

MURCIA VERDÚ, F. J., RAMOS ANTÓN, R. and CALVO RUBIO, L. M. (2022). “Comparative analysis of the sports chronicle quality written by artificial intelligence and journalists”. Revista Latina de Comunicación Social, 80, 91-111. https://doi.org/10.4185/RLCS-2022-1553

NAPOLI, P.M. (2014). “Automated Media: An Institutional Theory Perspective on Algorithmic Media Production and Consumption”. Communication Theory, 24 (3), 340-360. https://doi.org/10.1111/comt.12039

PARRA, S. and TORRENS, M. (2018). La inteligencia artificial: el camino hacia la ultrainteligencia. Barcelona: National Geographic.

RUTKIN, A. (2014). “Rise of Robot Reporters: When Software Writes the News”. New Scientist, 221. Retrieved from https://www.newscientist.com/article/dn25273-rise-of-robot-reporters-when-software-writes-the-news/

SANDOVAL-MARTÍN, T. and LA-ROSA, L. (2018). “Big Data as a differentiating sociocultural element of data journalism: the perception of data journalists and experts”. Communication & Society, 31 (4), 193-209. https://doi.org/10.15581/003.31.4.193-209

SANDOVAL-MARTÍN, T. and LA-ROSA, L. (2023). “Investigación sobre la calidad de las noticias automatizadas en la producción científica internacional: metodologías y resultados”. Cuadernos.info, 55, 114-136. https://doi.org/10.7764/cdi.55.54705

SCHWAB, K. (2016). La Cuarta Revolución Industrial (The Fourth Industrial Revolution). Barcelona: Debate.

SEPI (2020). Informe Anual 2020. Sociedad Estatal de Participaciones Industriales. Retrieved from https://www.sepi.es/sites/default/files/2022-03/informe_anual_2020_grupo_sepi_0.pdf

SEPI (2021). Informe Anual 2021. Sociedad Estatal de Participaciones Industriales. Retrieved from https://www.sepi.es/sites/default/files/2022-12/informe_anual_2021_grupo_sepi.pdf

SUNDAR, S. (1999). “Exploring receivers’ criteria for perception of print and online news”. Journalism & Mass Communication Quarterly, 76 (2), 373-386. http://dx.doi.org/10.1177/107769909907600213

SUNDAR, S. (2008). “The MAIN model: A heuristic approach to understanding technology effects on credibility”. In: M.J. METZGER and A.J. FLANAGIN (Eds.). Digital media, youth, and credibility. Cambridge, MA: The MIT Press, 73-100. https://doi.org/10.1162/dmal.9780262562324.073

TANDOC Jr, E.C., YAO, L.J. and WU, S. (2020). “Man vs. Machine? The Impact of Algorithm Authorship on News Credibility”. Digital Journalism, 8 (4), 548-562. https://doi.org/10.1080/21670811.2020.1762102

TEJEDOR-CALVO, S., CERVI, L., PULIDO, C.M. and PÉREZ-TORNERO, J.M. (2021a). “Análisis de la integración de sistemas inteligentes de alertas y automatización de contenidos en cuatro cibermedios”. Estudios sobre el mensaje periodístico, 27 (3), 973-983. https://doi.org/10.5209/esmp.77003

TEJEDOR, S., CERVI, L., TUSA JUMBO, F. and PORTALÉS, M. (2021b). “La información de la pandemia de la Covid-19 en las portadas de los diarios. Estudio comparativo de Italia, Reino Unido, España, Francia, Portugal, Estados Unidos, Rusia y Alemania”. Revista Mexicana de Ciencias Políticas y Sociales, 66 (242), 251-291. https://doi.org/10.22201/fcpys.2448492xe.2021.242.77439

TÚÑEZ-LÓPEZ, M., TOURAL-BRAN, C. and VALDIVIEZO-ABAD, C. (2019). “Automatización, bots y algoritmos en la redacción de noticias. Impacto y calidad del periodismo artificial”. Revista Latina de Comunicación Social, 74, 1411-1433. https://doi.org/10.4185/RLCS-2019-1391

TÚÑEZ-LÓPEZ, J. M., FIEIRAS CEIDE, C. and VAZ-ÁLVAREZ, M. (2021). “Impact of Artificial Intelligence on Journalism: transformations in the company, products, contents and professional profile”. Communication & Society, 34 (1), 177-193. https://doi.org/10.15581/003.34.1.177-193

TURING, A. (1950). “Computing Machinery and Intelligence”. Mind, LIX (236), 433-460. https://doi.org/10.1093/mind/LIX.236.433

UFARTE-RUIZ, M.J. and MANFREDI SÁNCHEZ, J.L. (2019). “Algoritmos y bots aplicados al periodismo. El caso de Narrativa Inteligencia Artificial: estructura, producción y calidad informative”. Doxa Comunicación, 29, 213-233. https://doi.org/10.31921/doxacom.n29a11

VAN-DER-KAA, H. and KRAHMER, E. (2014). “Journalist versus news consumer: The perceived credibility of machine written news”. In: Proceedings of the Computation + Journalism conference, Columbia University, New York, 24–25, October. Retrieved from https://research.tilburguniversity.edu/en/publications/journalist-versus-news-consumer-the-perceived-credibility-of-mach

WADDELL, T.F. (2018). “A Robot Wrote This? How perceived machine authorship affects news credibility”. Digital Journalism, 6 (2), 236-255. https://doi.org/10.1080/21670811.2017.1384319

WADDELL, T.F. (2019). “Can an Algorithm Reduce the Perceived Bias of News? Testing the Effect of Machine Attribution on News Readers’ Evaluations of Bias, Anthropomorphism, and Credibility”. Journalism & Mass Communication Quarterly, 96 (1), 82-100. https://doi.org/10.1177/1077699018815891

WÖLKER, A. and POWELL, T.E. (2021). “Algorithms in the newsroom? News readers’ perceived credibility and selection of automated journalism”. Journalism, 22 (2), 86-103. https://doi.org/10.1177/1464884918757072

ZHENG, Y., ZHONG, B. and YANG, F. (2018). “When algorithms meet journalism: The user perception to automated news in a cross-cultural context”. Computers in Human Behavior, 86, 266-275. https://doi.org/10.1016/j.chb.2018.04.046

WU, Y. (2019). “Is automated journalistic writing less biased? An experimental test of auto-written and human-written news stories”. Journalism Practice, 14 (8), 1008-1028. https://doi.org/10.1080/17512786.2019.1682940

Author Biography

Teresa Sandoval-Martín, Universidad Carlos III de Madrid

Teresa Sandoval-Martín, tenured  professor  of  the  Department  of  Communication  at  the  Universidad Carlos III de Madrid, specialized in journalism and new technologies. Lead researcher of the project Identificación de Sesgos de Género en Inteligencia Artificial (Identification of Gender Biases in Artificial Intelligence), funded by the Spanish State Research Agency. She directs the research group Journalism and Social Analysis: Evolution, Effects and Trends. She has directed five doctoral theses and a European postdoctoral fellowship. She has participated in more than a dozen research projects and 70 publications.

Published

2024-06-28

Downloads

Download data is not yet available.