resources

Using Document Analysis and Observation in Language Program Evaluation

Document analysis is often conducted to understand contextual information, such as program or instructional context, official/public policies and plans, program updates, curriculum and syllabus theory/design, and so on. Common types of documents often gathered in language program evaluation include course syllabi, instructor/curriculum handbooks, mission, goals, and outcomes statements, and students' enrollment and achievement records. While document analysis can reveal official or stated views of a program, observation techniques can reveal what actually happens in the program. Observation is useful when you are interested in understanding program context, implementation, processes, experiences, and interactions. In language program evaluation, observation is frequently applied to what learners and teachers do in the classroom, but other foci of observation might include language assessment practices, teacher development or induction procedures, learner counseling, and so on. Document analysis and observation can be complementary when identifying gaps between what actually happens in the program and what is formally stated. Below are a handful of useful methodological guides and example applications of document analysis and observation techniques in general and language program evaluation.

 

 

I. How-To Books and Articles

II. Online Advice and Guides

III. Document Analysis and Observation in L2 Program Evaluation Studies

 

 


I. How-To Books and Articles

 

GENERAL

Caulley, D. N. (1983). Document analysis in program evaluation. Evaluation and Program Planning, 6 (1), 19-29.

 

Patton, M. Q. (2002). Qualitative research & evaluation methods. Thousand Oaks, CA: Sage. (see chapter 6 on field work strategies and observation methods)

 

LANGUAGE

Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell. (see Chapter 7 on classroom observation)

 

Wajnryb, R. (1992). Classroom observation tasks: A resource book for language teachers and trainers. New York, NY: Cambridge University Press.

 

 

back to top

 


II. Online Advice and Guides

 

GENERAL

University of Texas, Instructional Assessment Resources provides an overview of document analysis and observation methods and example data analysis frameworks.

 

Document analysis: http://www.utexas.edu/academic/ctl/assessment/iar/programs/plan/method/doc-analysis.php

Observation:

http://www.utexas.edu/academic/ctl/assessment/iar/programs/plan/method/observ.php

 

University of Wisconsin Extension has an extensive guide on the values, types (structured and unstructured), instrumentation, and data analysis of observation. Example observation guides and recording sheets are useful.

Observation: http://learningstore.uwex.edu/Assets/pdfs/G3658-05.pdf

 

The Center for Disease Control offers pros and cons of observation, when to use observation, and a step-by-step guide for planning an observation.

Observation: www.cdc.gov/healthyyouth/evaluation/pdf/brief16.pdf

 

United States General Accounting Office has a comprehensive guide to content analysis, which is often applied to analyzing written documents.

Content analysis: A methodology for structuring and analyzing written material: archive.gao.gov/d48t13/138426.pdf

 

 

LANGUAGE

Harold Shiffman's (University of Pennsylvania) example language classroom observation form: http://ccat.sas.upenn.edu/~haroldfs/pedagog/evaluation/evaluate.html


 

 

back to top


III. Document Analysis and Observation in L2 Program Evaluation Studies

 

Document analysis

Lynch, B. K. (1996). Language program evaluation: Theory and practice. Cambridge: Cambridge University Press.

 

Ross, S. J. (2005). The impact of assessment method on language proficiency growth. Applied Linguistics, 26(3), 317-342.

 

Watanabe, Y. (2006). A Needs analysis for a Japanese high school EFL general education curriculum. Second Language Studies, 25(1). Retrieved from http://www.hawaii.edu/sls/uhwpesl/25(1)/Watanabe.pdf

 

Observation

Anderson, J. (1998). Managing and evaluating change: The case of teacher appraisal. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 159-186). London: Longman.

 

Horwitz, E. K. (1985). Formative evaluation of an experimental foreign-language class. Canadian Modern Language Review, 42(1), 83-90.

 

Karava-Doukas, K. (1998). Evaluating the implementation of educational innovations: Lessons from the past. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 25-50). London: Longman.

 

Lynch, B. K. (2000). Evaluating a project-oriented CALL innovation. Computer Assisted Language Learning, 13(4-5), 417-440.

 

Mitchell, R. (1989). Second language learning: Investigating the classroom context. System, 17(2), 195-210.

 

Rea-Dickins, P. (2001). Mirror, mirror on the wall: Identifying processes of classroom assessment. Language Testing, 18(4), 429-462.

 

Slimani, A. (1992). Evaluation of classroom interaction. In J. C. Alderson & A. Beretta (Eds.), Evaluating second language education (pp. 197-221). Cambridge: Cambridge University Press.

 

back to top

 

©2007 John Norris, Yukiko Watanabe, Marta Gonzalez-Lloret & Hye Ri Joo