Large-scale writing assessment: analysis of the Third Regional Comparative and Explanatory Study (TERCE) writing test
DOI:
https://doi.org/10.18800/educacion.201902.001Keywords:
Writing test, task analysis, rubrics (scoring guides), validity, reliabilityAbstract
The writing test of the Third Comparative and Explanatory Study implemented by the Latin American Laboratory for Assessment of the Quality of Education constitutes one of the few international experiences of large-scale assessment of writing. In order to strengthen these experiences, the theoretical framework adopted in this test, the task applied, the rubric and the coding process are reviewed, combining qualitative and quantitative techniques. The results show problems related to validity and reliability, and allow to formulate recommendations for future international assessments of writing. These include an articulation between the theoretical framework, the objectives and the scope of the assessment, a process of thorough review of the topics and syntax of the prompts, and the use of strategies to obtain evidences of validity and reliability at the different stages of the assessment process.
Downloads
References
ACE (2015). Informe técnico Simce 2013. Santiago de Chile. Recuperado de http://www.agenciaeducacion.cl/wp-content/uploads/2016/03/InformeTecnicoSimce_2013.pdf
Adam, J.-M. (1992). Les textes: types et prototypes. Récit, description, argumentation, explication et dialogue. Paris: Nathan.
AERA, APA y NCME (2014). Standards for Educational & Psychological Testing. Washington, DC.
Atorresi, A. (2005). Construcción y evaluación de consignas para evaluar la escritura como competencia para la vida. Enunciación, 10(1), 4-14.
Atorresi, A. y cols. (2010). Escritura: Un estudio de las habilidades de los estudiantes de América Latina y el Caribe. Santiago: OREALC/Unesco. Recuperado de http://unesdoc.unesco.org/images/0019/001919/191925s.pdf
Bachman, L. (2002). Some reflections on task-based language performance assessment. Language Testing, 19(4), 453-476. https://doi.org/10.1191/0265532202lt240oa
Barton, B. y Peat, J. (2014). Medical Statistics: A Guide to SPSS, Data analysis and Critical Appraisal. New Delhi: Wiley Blackwell.
Barton, D. y Hamilton, M. (2000). Literacy practices. En D. Barton, M. Hamilton y R. Ivanič (eds.), Situated Literacies. Reading and Writing in Context (pp. 7-15). Londres: Routledge.
Bazerman, C. (2017). What do sociocultural studies of writing tell us about learning to write? En C. MacArthur, S. Graham y J. Fitzgerald (eds.), Handbook of Writing Research (pp. 11-23). Nueva York: The Guilford Press.
Beach, R; Newell, G.E. y VanDerHeide, J. (2017). A sociocultural perspective on writing development: Toward an agenda for classroom research on students’ use of social practices. En C. MacArthur, S. Graham y J. Fitzgerald (eds.), Handbook of Writing Research (pp. 88-101). Ney York: The Guilford Press.
Behizadeh, N. (2014). Mitigating the dangers of a single story: Creating large-scale writing assessments aligned with sociocultural theory. Educational Researcher, 43(3), 125-136. https://doi.org/10.3102/0013189X14529604
Bejar, I. I. (2017). A historical survey of research regarding constructed-response formats. En R. E. Bennett y M. von Davier (eds.), Advancing Human Assessment (pp. 565-633). Nueva York: Springer Science. https://doi.org/10.1007/978-3-319-58689-2_18
Bereiter, C. y Scardamalia, M. (1992). Dos modelos explicativos de la composición escrita. Infancia y Aprendizaje, (58), 43-64. https://doi.org/10.1080/02103702.1992.10822332
Bond, T. G. y Fox, C. M. (2007). Applying the Rasch Model. Fundamental Measurement in the Human Sciences. Mahwah, NJ: LEA. https://doi.org/10.4324/9781410614575
Bridgeman, B. (2016). Can a two-question test be reliable and valid for predicting academic outcomes? Educational Measurement: Issues and Practice, 35(4), 21-24. https://doi.org/10.1111/emip.12130
Bridgeman, B.; Trapani, C. y Bivens-Tatum, J. (2011). Comparability of essay question variants. Assessing Writing, 16(4), 237-255. https://doi.org/10.1016/j.asw.2011.06.002
Burga, A. (2006). La unidimensionalidad de un instrumento de medición: perspectiva factorial. Revista de Psicología,24(1), 53-80. Recuperado de http://revistas.pucp.edu.pe/index.php/psicologia/article/view/642
Calfee, R. y Miller, R. (2007). Best practices in writing assessment. En S. Graham, C. MacArthur y J. Fitzgerald (eds.), Handbook of Writing Research (pp. 265-280). Nueva York: The Guilford Press.
Chan, S., Inoue, C. y Taylor, L. (2015). Developing rubrics to assess the reading-into-writing skills: A case study. Assessing Writing, 26, 20-37. https://doi.org/10.1016/j.asw.2015.07.004
Christie, F. y Derewianka, B. (2008). School Discourse. Learning to Write across the Years of Schooling. Londres, Nueva York: Continuum.
Donovan, C. y Smolkin, L. (2017). Children’s understanding of genre and writing development. En C. MacArthur, S. Graham, S. y J. Fitzgerald (eds.), Handbook of Writing Research (pp. 131-143). Nueva York: The Guilford Press.
Eckes, T. (2015). Introduction to Many-Facet Rasch Measurement. Nueva York: Peter Lang Edition.
Ferretti, R. y Fan, Y. (2017). Argumentative writing. En C. MacArthur, S. Graham y J. Fitzgerald (eds.), Handbook of Writing Research (pp. 301-315). Nueva York: The Guilford Press.
Flotts, M. P., Manzi, J., Abarzúa, A., Cayumán, C., Treviño, E., Meyer, A., Jiménez Rodríguez, J. (2016b). Tercer Estudio Regional Comparativo y Explicativo (TERCE). Reporte técnico. Santiago: OREALC/Unesco. Recuperado de http://www.unesco.org/new/es/santiago/education/education-assessment-llece/terce/
Flotts, M. P., Manzi, J., Jiménez, D., Abarzúa, A., Cayuman, C. y García, M.J. (2015). Los aprendizajes en escritura. En Tercer Estudio Regional Comparativo y Explicativo (TERCE). Informe de resultados. Logros de aprendizaje (pp. 95-145). Santiago: OREALC/Unesco. Recuperado de http://www.unesco.org/new/es/santiago/education/education-assessment-llece/terce/
Flotts, M. P., Manzi, J., Lobato, P., Durán, M. I., Díaz, M. P. y Abarzúa, A. (2016a). Aportes para la enseñanza de la escritura. Santiago: OREALC/Unesco. Recuperado de http://www.unesco.org/new/es/santiago/education/education-assessment-llece/terce/
Greenberg, K. (1992). Validity and reliability issues in the direct assessment of writing. Writing Program Administration (16), 7-22. Recuperado de ttps://eric.ed.gov/?id=EJ452781
Huot, B. (2002). (Re) articulating Assessment: Writing Assessment for Teaching and Learning. Logan: Utah State University Press.
INEE (2013). El aprendizaje de la expresión escrita en la educación básica en México. México DF: INEE. Recuperado de http://www.inee.edu.mx/index.php/72-publicaciones/resultados-de-aprendizaje-capitulos/567-el-apren-dizaje-de-la-expresion-escrita-en-la-educacion-basica-en-mexico-sexto-de-primaria-y-tercero-de-secundaria
Johnson, R., Penny, J. y Gordon, B. (2009). Assessing Performance: Designing, Scoring, and Validating Performance Tasks. Nueva York: The Guilford Press.
Jonsson, A., Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review,2(2), 130-144. https://doi.org/10.1016/j.edurev.2007.05.002
Kane, M. (2016). Validation strategies: Delineating and validating proposed interpretations and uses of test scores. En S. Lane, M. Raymond y T. Haladyna (eds.), Handbook of Test Development (pp. 64-80). Nueva York: Routledge.
Li, J. (2014). Examining genre effects on test takers’ summary writing performance. Assessing Writing, 22, 75-90. https://doi.org/10.1016/j.asw.2014.08.003
Linacre, J. M. (2017). Winsteps® (Version 4.0.0) [Computer Software]. Beaverton: Oregon. Recuperado de http://www.winsteps.com/
Lynne, P. (2004). Coming to Terms: A Theory of Writing Assessment. Logan: Utah State University Press. https://doi.org/10.2307/j.ctt46nqsz
Márquez, M. S., Iparraguirre, M. S. y Bengtsson, A. (2015). Un recorrido por el aprendizaje de la escritura desde una perspectiva evolutivo-educativa y semiótica. Interdisciplinaria 32(1), 151-168. https://doi.org/10.16888/interd.2015.32.1.8
Martin, J. R. y Rose, D. (2003). Interpreting social discourse. Working with Discourse: Meaning Beyond the Clause. Londres: Continuum.
Masters, G. N. (2016). Partial credit model. En W. van der Linden (ed.), Handbook of Item Response Theory (pp. 137-154). Boca Raton: Chapman & Hall/CRC.
Mendoza, A. (2018). El uso de Many-Facet Rasch Measurement para examinar la calidad del proceso de corrección de pruebas de desempeño. Revista mexicana de investigación educativa, 23(77), 597-625. Recuperado de http://www.scielo.org.mx/scielo.php?script=sci_arttext&pid=S1405-66662018000200597&lng=es&nrm=iso
Minedu (2016). ¿Qué logran nuestros estudiantes en Escritura? Recuperado de http://umc.minedu.gob.pe/evaluacion-censal-de-estudiantes-ece-2015/
Murillo, F. J. y Martínez-Garrido, C. (2016). La educación en América Latina y el Caribe. Aportes del TERCE y sus reanálisis. REICE, 14(4), 5-8. Recuperado de https://revistas.uam.es/index.php/reice/article/view/6577
Murphy, S. y Blake Yancey, K. (2008). Construct and consequence: Validity in writing assessment. En C. Bazerman (ed.), (2009). Handbook of Research on Writing: History, Society, School, Individual, Text (pp. 448-474). Nueva York-Londres: Taylor & Francis Group.
Real Academia Española (2009). Nueva gramática de la lengua española. Madrid: Espasa.
Ruiz-Funes, M. (2015). Exploring the potential of second/foreign language writing for language learning: The effects of task factors and learner variables. Journal of Second Language Writing, 28, 1-19. https://doi.org/10.1016/j.jslw.2015.02.001
Snow, C. y Uccelli, P. (2009). The challenge of academic language. En D. Olson y N. Torrance (eds.), The Cambridge Handbook of Literacy (pp. 112-133). Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511609664.008
Street, B. (2003). The implications of “New Literacy Studies” for literacy education. En S. Goodman et al. (eds.), Language Literacy and Education: A Reader (pp.77-88). Stoke on Trent: Trentham Books.
Tolchinsky, L. (2017). From text to language and back: The emergence of written language. En C. MacArthur, S. Graham y J. Fitzgerald (eds.), Handbook of Writing Research (pp.144-159). Nueva York-Londres: Guilford.
Unesco y MIDE U.C. (2011). Tercer Estudio Regional Comparativo y Explicativo (TERCE). Manual para la elaboración de ítems. (s/d).
Van den Bergh, H., Rijlaarsdam, G. y van Steendam, E. (2017). Writing process pheory: A functional dynamic approach. En C. MacArthur, S. Graham y J. Fitzgerald (eds.), Handbook of Writing Research (pp. 57-71). Nueva York-Londres: Guilford Press.
Villegas, F. (2016). ¿Qué logran los estudiantes peruanos en escritura? En V. Villarán (Presidencia), V Seminario Nacional de Investigación Educativa. Ayacucho, Perú.
Weigle, S. C. (2002). Assessing Writing. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511732997
Wiggins, G. (1998). Educative Assessment. Designing Assessments to Inform and Improve Student Performance. San Francisco: Jossey-Bass Publishers.
Wind, S. A. y Peterson, M. E. (2017). A systematic review of methods for evaluating rating quality in language assessment. Language Testing, 35(2), 161-192. https://doi.org/10.1177/0265532216686999
Zabaleta, V., Roldán, L. Á. y Centeleghe, M. E. (2016). La evaluación de la escritura a la finalización de la escolaridad primaria según las pruebas TERCE y ONE. REICE, 14(4). https://doi.org/10.15366/reice2016.14.4.008
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2019 Educacion
This work is licensed under a Creative Commons Attribution 4.0 International License.
