Artificial intelligence versus journalists: The quality of automated news and bias by authorship using a Turing test
Abstract
The integration of Artificial Intelligence (AI) in the media results in the publication of thousands of automated news articles in Spanish every day. This study uses a Turing test to compare the quality of news articles written by professional journalists (from Efe) with those produced by natural language generation (NLG) software (from Narrativa). Based on Sundar’s dimensions (1999) crucial to news perception – credibility, readability and journalistic expertise – , an internationally validated experimental methodology is employed, exploring a novel topic in Spanish: health information. The experiment deliberately varied real and declared authorships – AI and human journalists – to detect potential biases in assessing authorship credibility. A self-administered questionnaire adapted for online surveys was used (N=222), and gender imbalances were minimized to ensure gender equality in the sample (N=128). The study reveals that there are no significant differences between news articles generated by the AI and those written by professional journalists. Both types of news are considered equally credible, though some biases are detected in the evaluation of declared authorship: the AI author is perceived as more believable than the human, while the human journalist is perceived as creating a more lively narrative. The study concludes that it is feasible to produce automated news in Spanish without compromising its quality. In the global media landscape, automated systems employing NLG, machine learning and sophisticated databases successfully advance into new domains such as health information.
Keywords
automated journalism, automated news, artificial intelligence, Turing test, COVID-19References
ARGUEDAS, A.R. and SIMON, F.M. (2023). Automating Democracy: Generative AI, journalism, and the future of democracy. Oxford: Oxford Internet Institute. https://doi.org/10.5287/ora-e262xv7no
BECKETT C. and YASEEN, M. (2023). Generating Change. A global survey of what news organisations are doing with AI. London: LSE. Retrieved from https://www.journalismai.info/research/2023-generating-change
CALVO-RUBIO, L.-M. and UFARTE-RUIZ, M.-J. (2020). “Percepción de docentes universitarios, estudiantes, responsables de innovación y periodistas sobre el uso de inteligencia artificial en periodismo”. Profesional de la información, 29 (1). https://doi.org/10.3145/epi.2020.ene.09
CARLSON, M. (2015). “The Robotic Reporter”. Digital Journalism, 3 (3), 416-431. https://doi.org/10.1080/21670811.2014.976412
CHANG, Y., WANG, X., WANG, J., WU, Y., YANG, L., ZHU, K., CHEN, H., YI, X., WANG, C., WANG, Y., YE, W., ZHANG, Y., CHANG, Y., YU, P.S., YANG, Q. and XIE, X. (2023). “A Survey on Evaluation of Large Language Models”. ArXiv preprint arXiv:2307.03109v7, 1-26. https://doi.org/10.48550/arXiv.2307.03109
CLERWALL, C. (2014). “Enter the Robot Journalist. Users’ perceptions of automated content”. Journalism Practice, 8 (5), 519-531. https://doi.org/10.1080/17512786.2014.883116
EUROPEAN COMMISSION (2018). A definition of Artificial Intelligence. Brussels: European Commission. Retrieved from https://ec.europa.eu/futurium/en/system/files/ged/ai_hleg_definition_of_ai_18_december_1.pdf
DANZON-CHAMBAUD, S. (2021). “Tow Report. Covering COVID-19 with automated news”. Columbia Journalism Review. Retrieved from https://www.cjr.org/tow_center_reports/covering-covid-automated-news.php
DANZON-CHAMBAUD, S. (2023). Automated news in practice: Changing the journalistic doxa during COVID-19, at the BBC and across media organizations [Doctoral thesis, School of Communications Dublin City University]. Doras. Retrieved from https://doras.dcu.ie/28004/1/DEPOSIT_JANUARY_2023_Danzon-Chambaud_Automated_News_Practice_PhD_Thesis.pdf
DIAKOPOULOS, N. (2019). Automating the News. MA: Harvard University Press.
DOMINGOS, P. (2015). The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. St Ives: Allen Lane.
DÖRR, K.N. (2016). “Mapping the Field of Algorithmic Journalism”. Digital Journalism, 4 (6), 700-722. https://doi.org/10.1080/21670811.2015.1096748
EFRON, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. California: Standford University. https://doi.org/10.1137/1.9781611970319
FANTA, A. (2017). Putting Europe’s Robots on the Map: Automated journalism in news agencies. Oxford: Reuters Institute for the Study of Journalism.
FLANAGIN, A.J. and METZGER, M.J. (2000). “Perceptions of Internet Information Credibility”. Journalism & Mass Communication Quarterly, 77 (3), 515-540. https://doi.org/10.1177/107769900007700304
GRAEFE, A. and BOHLKEN, N. (2020). “Automated journalism: A meta-analysis of readers’ perceptions of human-written in comparison to automated news”. Media and Communication, 8 (3), 50-59. http://dx.doi.org/10.17645/mac.v8i3.3019
GRAEFE, A., HAIM, M., HAARMANN, B. and BROSIUS, H.-B. (2018). “Reader’s perception of computer-generated news: credibility, expertise, and readability”. Journalism, 19 (5), 595-610. https://doi.org/10.1177/1464884916641269
GUISADO-CLAVERO M., ASTIER-PEÑA, M.P., GÓMEZ-BRAVO, R. and ARES-BLANCO, S. (2022). “Open data for monitoring COVID-19 in Spain: descriptive study”. Enfermedades Infecciosas y Microbiología Clínica, 42 (2), 80-87. https://doi.org/10.1016/j.eimc.2022.10.002
HAIM, M. and GRAEFE, A. (2017). “Automated News: Better than expected?”. Digital Journalism, 5 (8), 1044-1059. https://doi.org/10.1080/21670811.2017.1345643
INSTITUTO CERVANTES (2022). El español: una lengua viva. Informe 2022. Retrieved from https://cvc.cervantes.es/lengua/espanol_lengua_viva/pdf/espanol_lengua_viva_2022.pdf
JIA, C. (2020). “Chinese automated journalism: A comparison between expectations and perceived quality”. International Journal of Communication, 14, 2611-2632. Retrieved from https://ijoc.org/index.php/ijoc/article/download/13334/3083
JIA, C. and JOHNSON, T. (2021). “Source Credibility Matters: Does Automated Journalism Inspire Selective Exposure?”. International Journal of Communication, 15, 3760-3781.
JUNG, J., SONG, H., KIM, Y., IM, H. and OH, S. (2017). “Intrusion of software robots into journalism: the public’s and journalists’ perceptions of news written by algorithms and human journalists”. Computers in Human Behavior, 71, 291-298. https://doi.org/10.1016/j.chb.2017.02.022
LATAR, N. (2014). “The robot journalist in the age of social physics: the end of human journalism?”. In: G. EINAV (Ed.). The New World of Transitioned Media. New York: Springer International Publishing, 65-80. https://doi.org/10.1007/978-3-319-09009-2_6
LATAR, N. and NORDFORS, D. (2009). “Digital Identities and Journalism Content: How Artificial Intelligence and Journalism May Co-Develop and Why Society Should Care”. Innovation Journalism, 6 (7).
LECOMPTE, C. (2015). “Automation in the Newsroom”. Nieman Reports, 69 (3), 32-45. Retrieved from http://niemanreports.org/wp-content/uploads/2015/08/NRsummer2015.pdf
LEE, S.M. and KIM, T.Y. (1998). “A news on demand service system based on robot agent”. In: International Conference on Parallel and Distributed Systems Proceedings, 528-532. https://doi.org/10.1109/ICPADS.1998.741128
LERMANN-HENESTROSA, L., GREVING, H., KIMMERLE, J. (2023). “Automated journalism: The effects of AI authorship and evaluative information on the perception of a science journalism article”. Computers in Human Behavior, 138. https://doi.org/10.1016/j.chb.2022.107445
LINDEN, C. and TUULONEN, H. (2019). News Automation. The rewards, risks and realities of ‘machine journalism’. WAN-IFRA. Retrieved from https://wan-ifra.org/insight/report-news-automation-the-rewards-risks-and-realities-of-machine-journalism/
MATSUMOTO, R., NAKAYAMA, H., HARADA, T. and KUNIYOSHI, Y. (2007). “Journalist Robot: Robot System Making News Articles from Real World”. In: 2007 IEEE Intl conf on robotics and automation, 1234-1241. https://doi.org/10.1109/IROS.2007.4399598
MEEHAN, J.R. (1977). “Tale-Spin, an Interactive Programme That Writes Stories”. In: Proceedings of the Fifth International Joint Conference on Artificial Intelligence, 91-98. Retrieved from https://www.ijcai.org/Proceedings/77-1/Papers/013.pdf
MEYER, P. (1973). Precision journalism: A reporter’s introduction to social science methods. Bloomington: Indiana University Press.
MEYER, P. (1988). “Defining and measuring credibility of newspapers: Developing an index”. Journalism & Mass Communication Quarterly, 65 (3), 567-574. https://doi.org/10.1177/107769908806500301
MORAN, R.E. and SHAIKH, S.J. (2022). “Robots in the news and newsrooms: Unpacking meta-journalistic discourse on the use of artificial intelligence in journalism”. Digital Journalism, 10 (10), 1756-1774. https://doi.org/10.1080/21670811.2022.2085129
MORAVEC, V., MACKOVÁ, V., SIDO, J. and EKŠTEIN, K. (2020). “The Robotic Reporter in The Czech News Agency: Automated Journalism and Augmentation in the Newsroom”. Communication Today, 11 (1), 37-52. Retrieved from https://communicationtoday.sk/wp-content/uploads/03_MORAVEC-et-al_CT-1-2020.pdf
MURCIA VERDÚ, F. J., RAMOS ANTÓN, R. and CALVO RUBIO, L. M. (2022). “Comparative analysis of the sports chronicle quality written by artificial intelligence and journalists”. Revista Latina de Comunicación Social, 80, 91-111. https://doi.org/10.4185/RLCS-2022-1553
NAPOLI, P.M. (2014). “Automated Media: An Institutional Theory Perspective on Algorithmic Media Production and Consumption”. Communication Theory, 24 (3), 340-360. https://doi.org/10.1111/comt.12039
PARRA, S. and TORRENS, M. (2018). La inteligencia artificial: el camino hacia la ultrainteligencia. Barcelona: National Geographic.
RUTKIN, A. (2014). “Rise of Robot Reporters: When Software Writes the News”. New Scientist, 221. Retrieved from https://www.newscientist.com/article/dn25273-rise-of-robot-reporters-when-software-writes-the-news/
SANDOVAL-MARTÍN, T. and LA-ROSA, L. (2018). “Big Data as a differentiating sociocultural element of data journalism: the perception of data journalists and experts”. Communication & Society, 31 (4), 193-209. https://doi.org/10.15581/003.31.4.193-209
SANDOVAL-MARTÍN, T. and LA-ROSA, L. (2023). “Investigación sobre la calidad de las noticias automatizadas en la producción científica internacional: metodologías y resultados”. Cuadernos.info, 55, 114-136. https://doi.org/10.7764/cdi.55.54705
SCHWAB, K. (2016). La Cuarta Revolución Industrial (The Fourth Industrial Revolution). Barcelona: Debate.
SEPI (2020). Informe Anual 2020. Sociedad Estatal de Participaciones Industriales. Retrieved from https://www.sepi.es/sites/default/files/2022-03/informe_anual_2020_grupo_sepi_0.pdf
SEPI (2021). Informe Anual 2021. Sociedad Estatal de Participaciones Industriales. Retrieved from https://www.sepi.es/sites/default/files/2022-12/informe_anual_2021_grupo_sepi.pdf
SUNDAR, S. (1999). “Exploring receivers’ criteria for perception of print and online news”. Journalism & Mass Communication Quarterly, 76 (2), 373-386. http://dx.doi.org/10.1177/107769909907600213
SUNDAR, S. (2008). “The MAIN model: A heuristic approach to understanding technology effects on credibility”. In: M.J. METZGER and A.J. FLANAGIN (Eds.). Digital media, youth, and credibility. Cambridge, MA: The MIT Press, 73-100. https://doi.org/10.1162/dmal.9780262562324.073
TANDOC Jr, E.C., YAO, L.J. and WU, S. (2020). “Man vs. Machine? The Impact of Algorithm Authorship on News Credibility”. Digital Journalism, 8 (4), 548-562. https://doi.org/10.1080/21670811.2020.1762102
TEJEDOR-CALVO, S., CERVI, L., PULIDO, C.M. and PÉREZ-TORNERO, J.M. (2021a). “Análisis de la integración de sistemas inteligentes de alertas y automatización de contenidos en cuatro cibermedios”. Estudios sobre el mensaje periodístico, 27 (3), 973-983. https://doi.org/10.5209/esmp.77003
TEJEDOR, S., CERVI, L., TUSA JUMBO, F. and PORTALÉS, M. (2021b). “La información de la pandemia de la Covid-19 en las portadas de los diarios. Estudio comparativo de Italia, Reino Unido, España, Francia, Portugal, Estados Unidos, Rusia y Alemania”. Revista Mexicana de Ciencias Políticas y Sociales, 66 (242), 251-291. https://doi.org/10.22201/fcpys.2448492xe.2021.242.77439
TÚÑEZ-LÓPEZ, M., TOURAL-BRAN, C. and VALDIVIEZO-ABAD, C. (2019). “Automatización, bots y algoritmos en la redacción de noticias. Impacto y calidad del periodismo artificial”. Revista Latina de Comunicación Social, 74, 1411-1433. https://doi.org/10.4185/RLCS-2019-1391
TÚÑEZ-LÓPEZ, J. M., FIEIRAS CEIDE, C. and VAZ-ÁLVAREZ, M. (2021). “Impact of Artificial Intelligence on Journalism: transformations in the company, products, contents and professional profile”. Communication & Society, 34 (1), 177-193. https://doi.org/10.15581/003.34.1.177-193
TURING, A. (1950). “Computing Machinery and Intelligence”. Mind, LIX (236), 433-460. https://doi.org/10.1093/mind/LIX.236.433
UFARTE-RUIZ, M.J. and MANFREDI SÁNCHEZ, J.L. (2019). “Algoritmos y bots aplicados al periodismo. El caso de Narrativa Inteligencia Artificial: estructura, producción y calidad informative”. Doxa Comunicación, 29, 213-233. https://doi.org/10.31921/doxacom.n29a11
VAN-DER-KAA, H. and KRAHMER, E. (2014). “Journalist versus news consumer: The perceived credibility of machine written news”. In: Proceedings of the Computation + Journalism conference, Columbia University, New York, 24–25, October. Retrieved from https://research.tilburguniversity.edu/en/publications/journalist-versus-news-consumer-the-perceived-credibility-of-mach
WADDELL, T.F. (2018). “A Robot Wrote This? How perceived machine authorship affects news credibility”. Digital Journalism, 6 (2), 236-255. https://doi.org/10.1080/21670811.2017.1384319
WADDELL, T.F. (2019). “Can an Algorithm Reduce the Perceived Bias of News? Testing the Effect of Machine Attribution on News Readers’ Evaluations of Bias, Anthropomorphism, and Credibility”. Journalism & Mass Communication Quarterly, 96 (1), 82-100. https://doi.org/10.1177/1077699018815891
WÖLKER, A. and POWELL, T.E. (2021). “Algorithms in the newsroom? News readers’ perceived credibility and selection of automated journalism”. Journalism, 22 (2), 86-103. https://doi.org/10.1177/1464884918757072
ZHENG, Y., ZHONG, B. and YANG, F. (2018). “When algorithms meet journalism: The user perception to automated news in a cross-cultural context”. Computers in Human Behavior, 86, 266-275. https://doi.org/10.1016/j.chb.2018.04.046
WU, Y. (2019). “Is automated journalistic writing less biased? An experimental test of auto-written and human-written news stories”. Journalism Practice, 14 (8), 1008-1028. https://doi.org/10.1080/17512786.2019.1682940
Published
Downloads
Copyright (c) 2024 Leonardo Alberto La-Rosa Barrolleta, Teresa Sandoval-Martín
This work is licensed under a Creative Commons Attribution 4.0 International License.
