Using Document Analysis and Observation in Language Program Evaluation
Document analysis is often conducted to understand contextual information, such as program or instructional context, official/public policies and plans, program updates, curriculum and syllabus theory/design, and so on. Common types of documents often gathered in language program evaluation include course syllabi, instructor/curriculum handbooks, mission, goals, and outcomes statements, and students' enrollment and achievement records. While document analysis can reveal official or stated views of a program, observation techniques can reveal what actually happens in the program. Observation is useful when you are interested in understanding program context, implementation, processes, experiences, and interactions. In language program evaluation, observation is frequently applied to what learners and teachers do in the classroom, but other foci of observation might include language assessment practices, teacher development or induction procedures, learner counseling, and so on. Document analysis and observation can be complementary when identifying gaps between what actually happens in the program and what is formally stated. Below are a handful of useful methodological guides and example applications of document analysis and observation techniques in general and language program evaluation.
Caulley, D. N. (1983). Document analysis in program evaluation. Evaluation and Program Planning, 6 (1), 19-29.
Patton, M. Q. (2002). Qualitative research & evaluation methods. Thousand Oaks, CA: Sage. (see chapter 6 on field work strategies and observation methods)
Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell. (see Chapter 7 on classroom observation)
Wajnryb, R. (1992). Classroom observation tasks: A resource book for language teachers and trainers. New York, NY: Cambridge University Press.
University of Texas, Instructional Assessment Resources provides an overview of document analysis and observation methods and example data analysis frameworks.
University of Wisconsin Extension has an extensive guide on the values, types (structured and unstructured), instrumentation, and data analysis of observation. Example observation guides and recording sheets are useful.
The Center for Disease Control offers pros and cons of observation, when to use observation, and a step-by-step guide for planning an observation.
United States General Accounting Office has a comprehensive guide to content analysis, which is often applied to analyzing written documents.
Content analysis: A methodology for structuring and analyzing written material: archive.gao.gov/d48t13/138426.pdf
Harold Shiffman's (University of Pennsylvania) example language classroom observation form: http://ccat.sas.upenn.edu/~haroldfs/pedagog/evaluation/evaluate.html
Lynch, B. K. (1996). Language program evaluation: Theory and practice. Cambridge: Cambridge University Press.
Ross, S. J. (2005). The impact of assessment method on language proficiency growth. Applied Linguistics, 26(3), 317-342.
Watanabe, Y. (2006). A Needs analysis for a Japanese high school EFL general education curriculum. Second Language Studies, 25(1). Retrieved from http://www.hawaii.edu/sls/uhwpesl/25(1)/Watanabe.pdf
Anderson, J. (1998). Managing and evaluating change: The case of teacher appraisal. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 159-186). London: Longman.
Horwitz, E. K. (1985). Formative evaluation of an experimental foreign-language class. Canadian Modern Language Review, 42(1), 83-90.
Karava-Doukas, K. (1998). Evaluating the implementation of educational innovations: Lessons from the past. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 25-50). London: Longman.
Lynch, B. K. (2000). Evaluating a project-oriented CALL innovation. Computer Assisted Language Learning, 13(4-5), 417-440.
Mitchell, R. (1989). Second language learning: Investigating the classroom context. System, 17(2), 195-210.
Rea-Dickins, P. (2001). Mirror, mirror on the wall: Identifying processes of classroom assessment. Language Testing, 18(4), 429-462.
Slimani, A. (1992). Evaluation of classroom interaction. In J. C. Alderson & A. Beretta (Eds.), Evaluating second language education (pp. 197-221). Cambridge: Cambridge University Press.