digital competence in K-12 . theoretical models , assessment tools and empirical research

over the last years the theme of the digital competence in its different aspects has been object of a growing interest. in a number of official documents and communications, international bodies underlined the significance of this competence for lifelong learning and to participate in the so-called ‘information society’. Within this context, education research has the duty to provide realistic conceptual models coherent with the school’s objectives and which can be put into practice within the school curriculum. in the present paper, we shall introduce a theoretical model, education oriented, to represent this competence and a set of tools to assess it in the school context, i.e., the instant dca (idca) and the situated dca. then we shall focus on idca and on the results of the testing carried out over the last two years in the italian secondary school. Key-words: digital competence; assessment; K-12 school system. Resumen. Competencia digital en educación primaria y secundaria. Modelos teóricos, Herramientas de evaluación e investigación empírica durante los últimos años el tema de la competencia digital en sus diferentes aspectos ha sido objeto de un creciente interés. los organismos internacionales han destacado a importancia de esta competencia para el aprendizaje permanente y la inclusión en la sociedad de la información en una serie de documentos oficiales. dentro de este contexto, la investigación educativa tiene el deber de proporcionar modelos conceptuales coherentes con los objetivos de la escuela y que se puedan poner en práctica dentro del currículum escolar. en el presente trabajo, introducimos un modelo teórico, orientado a la educación, para desarrollar esta competencia y un conjunto de herramientas para evaluarla en el contexto escolar. Palabras clave: competencia digital, evaluación, educación primaria y secundaria. anàlisi 40, 2010 157-171 158 anàlisi 40, 2010 antonio calvani, antonio fini y maria ranieri


Introduction
the theme of digital literacy, or digital competence, has gained ground in the past twenty to thirty years while most of contemporary society's production activities were being computerized, and it is becoming a crucial point in the education debate on in the new millennium.
Why is this theme so important today?at least two main reasons can be pointed out.
on one hand, a reflection on this topic is necessary for a more aware understanding of the reasons and the modalities of using new technologies at school.activities with icts done at school in these past thirty years are still of scarce significance on the cognitive and cultural level.so today, being able to choose significant cognitive activities, which technologies can enhance in a specific way is of vital importance.
on the other hand, we are faced with a socio-cultural process affecting the cognitive structures involved in the acquisition and processing of information and basic knowledge, and therefore with a process of redefining "literacies".Walter ong has admirably shown that changes involving language forms have strong repercussions on cognitive processes and knowledge organization.today we cannot foresee what new cognitive practices will emerge related to changes in technologies of the word.changes related to elementary forms of literacy are definitely closely connected to knowledge forms and models.dealing with digital literacy in schools or in lifelong learning means reflecting on what personal and social structures will be suitable for the construction and organization of knowledge in the coming years.
Within this context, education research has the duty to provide an outline of realistic projects and models coherent with the school's objectives and which can be put into practice within the school curriculum.so how should schools be advised about assessment and the consequent development of digital competence?What tools can be used? in this paper, we will present a digital competence model, based on educational grounds, focusing importance on the critical and cognitive dimension.We shall start with an overview of existing literature and then move on to the outlined model and the developed assessment tools.gilster (1997) first used and defined the term digital literacy, putting emphasis on the ability for critical thinking rather than it skills.in less than fifteen years, definitions have multiplied.some authors point out that digital literacy is the result of a stratified and complex integration of competences, skills and knowledge.for example, tornero states ( 2004): «digital literacy merges capabilities: purely technical aspects, intellectual competences and also competences related to responsible citizenship.they all allow individuals to develop themselves completely in information society» (ibid., p. 31).likewise martin (2005) defines digital literacy as «the awareness, attitude and ability of individuals to appropriately use digital tools and facilities to identify, access, manage, integrate, evaluate, analyze and synthesize digital resources, construct new knowledge, create media expressions, and communicate with others, in the context of specific life situations, in order to enable constructive social action; and to reflect upon this process» (ibid., p.135).gapski articulates digital literacy into two main strands: "instrumental-technological" and "normative media-educational," corresponding respectively to usage/functionality, and pedagogical aims (gapski, 2007).these strands are operationalized into three forms of competence: (a) interpreting messages; (b) choosing messages; and (c) articulating messages.in turn, these competences inform objectives and measures of functional, cognitive and ethical proficiencies.
other research carried out by international organizations deserves to be mentioned.over the last years the concept of it literacy has evolved towards more reflexive rather than technical approaches to icts, as emerged from the ict literacy panel proposed in 2002 by the ets (educational testing service).in the ets panel the concept of ict literacy indicates the ability to use communication technologies and tools to access, manage, integrate, evaluate and create information in order to be able to integrate it successfully into everyday life.
similarly, in the u.s. the international society for technology in education (iste) developed the national educational technology standards (nets-s) and performance indicators for students (international society for technology in education, 2007).according to nets-s, a digitally literate student demonstrates critical and creative thinking and is able to use digital media to work collaboratively and to gather and evaluate information.
at the same time, research on information literacy has been developing.in particular, in 2000 the association of college and research libraries (acrl) promoted new standards for the definition of information literacy, where the capacity to determine the nature and extent of the information needed and the ability to critically evaluate information are indicated as essential components (acrl, 2000, pp. 8-13).
in more recent years, with the introduction of the so-called Web 2.0 and the consequent emphasis on Participatory Culture (Jenkins et al., 2006), the focus on socio-ethical aspects has been further emphasized.researchers question whether, and in what ways the web can develop ethical minds (gardner, 2007), an aspect which has been studied by the new media literacy team at the macarthur foundation.the team pointed out that factors such as identity, ownership, authorship, credibility, participation should be placed at the core of the research (James et al, 2009).also the european commission (ec) has promoted several initiatives over the last ten years in order to encourage the development of digital literacy in the eu member states.a group of experts was made-up to define actions, studies and surveys were carries out, and a set of recommendations were published.in particular, in december 2006 the european parliament (ep) and the council released two recommendations which deserve to be considered.in the Recommendation on Key Competences for Lifelong Learning (2006/962/ ec), a new framework for key competences was outlined and digital competence was included among the competences for lifelong learning.Here digital competence is defined as involving "the confident and critical use of information society technology (ist) for work, leisure and communication. it is underpinned by basic skills in ict: the use of computers to retrieve, assess, store, produce, present and exchange information, and to communicate and participate in collaborative networks via the internet."(european parliament, 2006a, p. l. 394/16).at the same time, the ep published the Recommendation on the protection of minors and human dignity (2006/952/ec), where the following aspects are emphasized: the need for teacher training on media literacy; the inclusion of media literacy in the curriculum to enhance children's capacity of self-protection and promote responsible attitudes among all users (for a more detailed discussion on digital and media literacy in europe see: tornero, paredes, simelio, 2010; celot & tornero, 2008).
lastly, an outstanding study has been commissioned recently by the ec on the issue of media and digital literacy assessment (celot & tornero, 2009). in this study, two main dimensions for media literacy are identified in terms of individual competences and environmental factors.the first category involves the personal capacity of accessing, using and understanding media, and a set of more social skills related to communication and social participation.the second category includes contextual factors (such as media availability or media literacy policies) affecting individuals and related to citizens' rights.
in conclusion, even though each author or organization emphasises different aspects, they all agree that digital literacy or competence is a multidimensional concept entailing a complex integration of technical skills, cognitive (e.g., problem solving, critical thinking skills) and meta-cognitive processes as well as civic engagement and ethical awareness.

ASSESSING digital competence: What Instruments?
When reflecting on how digital competence can be assessed, we come across an amount of rapidly evolving tools and hypotheses.a very well known test in europe is the european computer driving license (ecdl by the ecdl foundation, or the ic3, managed by certiport and mous, supervised by microsoft and aimed at the use of the company's products).the need for a more sophisticated tool able to capture the educational aspects involved in the use of technologies is reflected in the assessment instrument developed by ets.based on the above mentioned work by the international ict literacy panel (2002), ets developed the already quoted icritical thinking (formerly iskills, formerly ict literacy assessment), intended for students from high school (grades 10-12) through college, teachers, employers and working adults.this tool consists of 60 items to be answered in 75 minutes and includes various sections: basic technical skills, short scenarios (dealing with basic e-mail functions), web search (ability to select and evaluate internet research) and simulation tasks (a more complex area dealing with the understanding of experimental models).
another tool which deserves attention is the 2014 national assessment of educational progress (naep) technology and engineering literacy assessment (naep, 2010), promoted in u.s. by the national assessment governing board.this instrument, which is still in a pilot testing phase, will be used in 2014 for K-12 assessment on a national scale.it is based on standards such as the above mentioned nets-s and is subdivided into the three major assessment areas: technology and society, design and systems, and ict.this means that, according to naep, a technological literate student is able "to analyze the relationship between technology and society, has a broad understanding of technology and can solve problems using the engineering design process, and is able to make fluent use of digital technologies and media in creative and innovative ways" (naep, 2010, p. 1-9). in particular, three practices are considered in the assessment, that is, understanding technological principles, developing solutions and achieving goals, and communicating and collaborating.the test will include two types of scenario-based assessment sets, a long one (25 min.to respond) and a short one (12-15 min.to respond), and discrete items.
other tools that can be mentioned are the ict literacy development project by the university of alberta, ca, which is intended for higher education students; the australian national ict literacy assessment program in years 6 and 10, by the australian ministerial council on education, employment, training and youth affairs, which is based on ets standards.
While certifications such as ecdl are mainly focused on mastering basic technical skills, the other instruments such as the ets icritical thinking or the naep technology and engineering literacy assessment shift the emphasis to cognitive processes and socio-ethical issues.

A conceptual model for digital competence
by now there is sufficient consensus on the fact that a notion of digital competence has to shift the emphasis from a purely technical meaning to a more complex concept, in order to be pedagogically meaningful.such a concept should pay more attention to conceptual and critical dimensions and to the capability of understanding the underlying nature of technological phenomena, as well as to the knowledge of the ethical and social implications related to the use of web technologies.
among the various terms in use, we preferred «digital competence», in line with the european recommendation, given that this term is rapidly spreading in the educational language and recognizes a greater importance to the concept itself, putting it on the same level of other core competences.
However, we deem it important to preserve this concept from any possible reductionism.the concept of digital competence we want to pursue is: multidimensional: it implies integration between abilities and skills of cognitive, relational and social nature which make it necessarily a nonlinear concept; complex: it is not completely quantifiable with single tests; some aspects of this competence are difficult to assess, at least in the short term, and may remain concealed, requiring more time and very differentiated contexts before coming to surface; interconnected: it is not independent from other abilities or basic competences with which it overlaps (for instance, reading, problem solving, numeracy, logical, inferential, and metacognitive skills); sensitive to the socio-cultural context: it would not be reasonable to think of a unique model of digital literacy adequate at all times and in all contexts; the meaning of this literacy partly will change also according to the various educational settings (basic training, professional training, lifelong learning, specialized training).
in our case we have defined digital competence as the capability "to explore and face new technological situations in a flexible way, to analyze, select and critically evaluate data and information, to exploit technological potentials in order to represent and solve problems and build shared and collaborative knowledge, while fostering awareness of one's own personal responsibilities and the respect of reciprocal rights/obligations."(calvani, fini & ranieri 2009, p. 161).this definition emphasizes the co-existence of three dimensions and their integration: technological dimension: being able to explore and face problems and new technological contexts in a flexible way; cognitive dimension: being able to read, select, interpret and evaluate data and information taking into account their pertinence and reliability; ethical dimension: being able to interact with other individuals constructively and with a sense of responsibility towards oneself and towards others.
the chart below (figure 1) summarizes the adopted model.starting from the conceptual model outlined above and considering the complexity of the construct under consideration, we developed two diverse types of tests addressed for different age levels, i.e. the idca (instant digital competence assessment) and the situated dca. in this paper, we shall focus on the idca whereas we will only give some information about the situated dca.
unlike the ets icritical thinking tests, a prerequisite for our tests was the knowledge of basic skills, which were therefore excluded from our analysis.as regards the assessment of competences in authentic situations we created more complex tests, i.e. the situated dca, which ask students to face with real problem-solving tasks which cannot be assessed by simple testing.the situated dca consists of a set of 4 x 2 (type and level of complexity change); they take place in a computer laboratory and need two or more students sitting in front of a computer or interacting through computer mediated communication. in the first typology of test (technological exploration) students have to deal with an unknown technological interface and learn how to use and master it.in the second typology (simulation) data must be empirically processed and hypotheses on possible relationships have to be formulated.in the third typology (inquiry) relevant information pertaining to a predefined subject must be critically selected and gathered.lastly, in the fourth typology (collaborative Wiki) students have to draft a document together following criteria of collaborative activity management.the complexity factor arises either from the greater objective difficulty of the required task or from the higher complexity level of the technological and collaborative situation: at the first level the test is performed in pairs in front of a computer, while at the second level in a group interacting via cmc.students are first given the goals to be achieved, then the specific norms to follow during collaborative activities and finally the prerequisites for self-assessment.they will be also judged by third-party observers.
continuing our discussion on idca, subject of this paper, it is conceived as a rapid means of assessment and can be used by entire scholastic institutes or by teachers of single classes, so as to be able to offer a completely automatic user-friendly assessment method.
operatively the three dimensions (technological, cognitive, ethical) represented in figure n. 1 were divided into the following subcategories (see figure n. 2) the technological dimension was limited to three subcategories, two of a practical nature (recognizing interfaces and solving common problems) and a more abstract one (understanding how the underlying technology works).
as regards the cognitive dimension, which is the most significant, we have emphasized on activities such as extracting important data from a text, judging the reliability of information, comparing contrasting information, organizing data in tables and making inferences.such activities represent an arena where traditional literacies and digital competence clearly overlap.
ethical and social themes were divided into three subcategories: safety, respect and awareness of technological inequalities.Box n. 1-exAmple of A mulTiple-ChoiCe QuesTion, sCores AnD feeDBACk iDCA applications in school over the last two years the idca has been administered several times at school in italy and also abroad. in particular, the test has been validated through diverse applications in the italian schools and also translated in english and then adapted in chinese to be administered to a sample of chinese students (li & ranieri, 2010).
besides that, a systematic application has been carried out in italy to assess the digital competence level of italian students entering secondary school.in this paragraph, we shall focus on these applications.

The validation test and the Italy-China Survey
the first version of the tests, designed for adolescents (14-16 year-old high school students), was prepared in a paper questionnaire format.this initial elaborative phase led to the creation of a paper questionnaire, made up of 1. the online tests are available on the www.digitalcompetence.org/moodlewebsite in italian and in english.teachers who are interested can ask for free access passwords for their classes and administer the test to their students autonomously.
87 questions.With the collaboration of some high schools, the questionnaire was administered to several first and second year classes from three different High schools, and supervised by the researchers.this phase ran parallel to a further sounding of opinions by giving the questionnaire to a panel of experts to validate the content.the combined results of these operations, that is, the item analysis on the results of the first phase, the feedback from the collaborating teachers and observing researchers, the comments and suggestions from the expert panel, led to the modification, integration and even elimination of some items.in particular, those items on which the experts had expressed some doubts or where their answer was different from that expected, were eliminated or reviewed.
after this first phase, the chosen and re-examined items were 85 altogether.this group of items was then implemented on a web platform to permit its experimentation on a wider scale and to verify the feasibility of a computerized administration of the tests.
the developed online version was used between february and december 2008. in this first administration of the test a total of 220 cases of first and second class high school students were collected (by 1/6/2009).thanks to this testing a first significant item analysis was made.
the items were later translated into english.following contacts with a university in china2 a shorter version of the test, called "ita-china experimentation", was made.these items were chosen, taking into account both the item analysis carried out earlier and the need to adapt the items to the specific chinese context.a further and more complete item analysis was therefore possible through the experimentation carried out in italy and in china.the reliability of the test was confirmed.in fact, cronbach's alpha coefficient was satisfactory in both the chinese (0.77) and italian (0.79) samples.
this led to the identification of the 35 items used in the third stage of the experimentation, intended for italian high schools and which took place between september 2009 and January 2010.this last stage will be fully presented in the following paragraph.

The wide Italian survey
in the period september 2009-January 2010 the idca test was administered to a sample of students from italian secondary schools with the aim at assessing the "state" of digital competence among italian students.considering the way the questions are formulated and their adaptation to the age, we assumed that "digitally competent" students could generally pass the tests with considerable high scores.We, therefore, imagined a hypothetical threshold of at least 75% correct answers.
the schools involved in the study were randomly chosen proportionately to area and type of school, from the universe of all the italian technical high schools and "licei".total number of respondents was 1056 from 34 schools and the average age of the students was 15.
if we examine the overall scores, the results are on the whole lower than we expected.the average score was in fact 62,5 (dev.st=15,8), which is less than the 75% threshold, and only one fourth of the students reaches the 75% threshold (table 1).as regards the single dimensions and subcategories, an overall picture of the obtained results is provided in the graph below (figure 3): figure 3 -perCenTAges of The AVerAge sCores for eACh inDiCATor as shown in figure 3, there is a clear-cut difference between the data obtained in the first two subcategories and all the others.if we apply the threshold we adopted (75%) the respondents could be considered digitally competent only if we understand this concept as the ability to recognize the most common interface icons, or with the ability to solve a basic technical problem.
if we move to questions which imply a more critical and complex knowledge of technology, several critical issues emerge.for example, 1/3 of the respondents are not able to explain adequately why an email may not reach the recipient or what could be the effects of a virus; about half of the respondents believe that a computer can perfectly translate a literary text from one language to another.
as regards the cognitive subcategories, none of them reach the 75% threshold.in particular, the percentage of correct answers to the items related to activities such as dealing with textual data, hierarchical structuring of information and its synthesis, identifying keywords and interpreting dynamic graphs, is slightly below 70%.
even as regards the ability to critically evaluate information on the internet, the scores go down to about 60% with a particular drop in the interpretation of search engine results: for example, when asking what are the factors influencing the order of results in a search, over half of the respondents seem to be unaware that this depends on the criteria adopted by the search engine.
the subcategory with the worst results is the one relating to the items asking to manipulate, interpret and formulate inferences from data organized in tables or to select the graph representation of an algorithm corresponding to a succession of events.the average reached here was about 43%, with slightly higher scores in the formal logic problems, which include, e.g., the use of boolean operators.more diversified results emerge in the ethical dimension.for example, on one hand, students disapprove cyberbullying, on the other hand, they do not have clear ideas about respect of privacy and personal safety.the most difficult question proved to be the one about the digital divide, which shows the scant awareness of the access problems and technical difficulties that developing countries could have in internet communication.about half of the students state that the quality of communication depends only on the quantity of pictures, audio and video, and do not agree with the opinion that an excessive quantity of multimedia can give rise to some problems.

Conclusion
digital competence stands as an important challenge for the educational systems of the new century.understanding that this concept cannot be reduced to a single component, nor can it be assessed with just one type of test is the real crucial point.the adoption of a flexible and integrated approach is therefore needed, without renouncing to define criteria and methodologies that allow a reasonable comparison of data gathered from the various schools or data gathered at different times from one school.in this perspective, we have developed a set of assessment tools to evaluate digital competence at different levels of complexity and for different students' ages.Here we have focused on the idca and on the applications of this tool in secondary school.in particular, we have presented the validations' steps of the tool and the results of a wide survey carried out in italy in september 2009-January 2010.
a considerable results emerging from the survey is that as the cognitive, critical and logical levels entailed in the test go higher, the scores go lower: adolescents answer correctly questions related to technological activities that require less demanding cognitive processes, but, in line with other already studies (eagleton et al., 2003;ravestein et al., 2007;bennet et al., 2008bennet et al., , Katz & macklin, 2007)), they achieve low results where more complex conceptual aspects come into play, such as those implicit in activities like comparing information, critically evaluating the reliability of a site or deriving inferences from data.
as regards the socio-ethical dimension, the results seem to be conflicting in some degree.in fact, though adolescents judge cyberbullying as objectionable, they are little aware that online behaviour needs to be adequate for their own safety and respectful of privacy and they are completely ignorant about the problems connected to technological inequality and the digital divide.these aspects bring up once again the classical psychological theory of child and adolescent egocentrism (from piaget, 1964, onwards), that is, the difficulty to understand that others could have different points of view.as it were, the internet amplifies youth egocentrism in so far as young people seem to overlook the digital needs of the other people (James et al, 2009).
We believe that school should play a fundamental role in promoting technical and social access to digital media and in supporting a more critical and significant use of technologies.on one hand it should make sure that the same basic technological abilities are acquired by everybody, thus eliminating the still existing disparities due to socio-economic and cultural gaps.on the other hand, it should guarantee the integration of the notions and technological abilities which adolescents could acquire spontaneously through their own practices, within a more articulate cognitive framework, adequately related to other significant competences.this should be accomplished through specific educational interventions aiming at developing a critically, ethically and socially aware personality (buckingham, 2006;Jenkins et al., 2006;calvani, fini & ranieri, 2010).
figure 1 -DigiTAl CompeTenCe frAmeWork

TABle 1 -
DisTriBuTion of oVerAll sCores