resources

Using Interviews and Focus Groups in Language Program Evaluation

I. How-To Books on Interviewing

II. How-to Books on Focus Group Techniques

III. How-to Books on Qualitative Data Analysis

IV. Online Advice on Designing and Analyzing Interviews

V. Online Advice on Focus Group Techniques

VI. Examples of Interviews and Focus Groups in Language Program Evaluation

References marked with an asterisk (*) indicate those sources that we have found particularly helpful.

I. How-To Books on Interviewing

Bourque, L. B., & Fielder, E. P. (2002). The Survey Kit (2nd ed.): Vol. 4. How to Conduct Telephone Surveys. Thousand Oaks, CA: Sage.

Fowler, F. J., & Mangione, T. W. (1990). Standardized survey interviewing: Minimizing interviewer-related error. Newbury Park, CA: Sage.

Frey, J. H., & Oishi, S. M. (1995). How to Conduct Interviews by Telephone and in Person (Survey Kit, Vol. 4). Thousand Oaks, CA: Sage.

Gubrium, J. F., & Holstein, J. A. (Eds.). (2002). Handbook of interview research: Context & method. Thousand Oaks, CA: Sage.

*Kvale, S. (1996). InterViews: An introduction to qualitative research interviewing. Thousand Oaks, CA: Sage.

NOTE: Kvale walks the reader through seven methodological stages of qualitative interview studies: thematizing, designing, interviewing, transcribing, analyzing, verifying, and reporting.

Maso, I., & Wester, F. (Eds.). (1996). The deliberate dialogue: Qualitative perspectives on the interview. Brussels: VUB University Press.

Mishler, E. G. (1986). Research interviewing: Context and narrative. Cambridge, MA: Harvard University Press.

Rubin, H. J., & Rubin, I. (2005). Qualitative interviewing: The art of hearing data (2nd ed.). Thousand Oaks, CA: Sage.

Seidman, I. E. (2006). Interviewing as qualitative research: A guide for researchers in education and the social sciences (3rd ed.). New York: Teachers College Press.

Weiss, R. S. (1995). Learning from Strangers: The Art and Method of Qualitative Interview Studies. New York: Free Press.

Wengraf, T. (2001). Qualitative research interviewing: Biographic narrative and semi-structured methods. London: Sage.

II. How-to Books on Focus Group Techniques

Fern, E. F. (2001). Advanced focus group research. Thousand Oaks, CA: Sage.

*Krueger, R. A., & Casey, M. A. (2000). Focus groups: A practical guide for applied research (3rd ed.).Thousand Oaks, CA: Sage.
NOTE: A non-technical step-by-step guide on how to design focus group studies, select participants, moderate focus group discussion, analyze focus group data, and report results.

Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Newbury Park, CA: Sage.

*Morgan, D. L., & Krueger, R. A. (Eds.). (1997). Focus Group Kit (Focus Group Kit, Vols. 1-6). Thousand Oaks, CA: Sage.
NOTE: This six volume set provides a straightforward blueprint for people conducting focus groups, illustrating a number of key steps involved in conducting focus group research. Below is the list of volumes included in the kit.

(1) Morgan, D. L. (1997). The focus group guidebook (Focus Group Kit, Vol. 1). Thousand Oaks, CA: Sage.
(2) Morgan, D. L., & Scannell, A. U. (1997). Planning focus groups (Focus Group Kit, Vol. 2). Thousand Oaks, CA: Sage.
(3) Krueger, R. A. (1997). Developing questions for focus groups (Focus Group Kit, Vol. 3). Thousand Oaks, CA: Sage.
(4) Krueger, R. A. (1997). Moderating focus groups (Focus Group Kit, Vol. 4). Thousand Oaks, CA: Sage.
(5) Krueger, R. A., & King, J. A. (1997). Involving community members in focus groups (Focus Group Kit, Vol. 5). Thousand Oaks, CA: Sage.
(6) Krueger, R. A. (1997). Analyzing and reporting focus group results (Focus Group Kit, Vol. 6). Thousand Oaks, CA: Sage.

*Stewart, D. W., & Shamdasani, P. N. (1990). Focus groups: Theory and practice. Thousand Oaks, CA: Sage.
NOTE: The book helps the researcher attend to the intra- and inter- personal and environmental variables when designing focus group studies.

Vaughn, S., Schumm, J. S. & Sinagub, J. M. (1996). Focus group interviews in education and psychology. Thousand Oaks, CA: Sage.

III. How-to Books on Qualitative Data Analysis

*Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage.
NOTE: This book provides clear guidelines on the process of thematic analysis, a method to encode and find patterns in qualitative data.

Coffey, A., & Atkinson, P. (1996). Making sense of qualitative data analysis: Complementary strategies. Thousand Oaks, CA: Sage.

Denzin, N. K., & Lincoln, Y. S. (Eds.). (2005). The Sage handbook of qualitative research. Thousand Oaks, CA: Sage.

Erlandson, D. A., Harris, E. L., Skipper, B. L., & Allen, S. D. (1993). Doing naturalistic inquiry. Newbury Park, CA: Sage.

Glesne, C. (2005). Becoming qualitative researchers: An introduction (3rd ed.). New York: Allyn & Bacon.

Maso, I., & Wester, F. (Eds.). (1996). The deliberate dialogue. Brussel: VUB Press.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage.

Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks CA: Sage.

Silverman, D. (2001) Interpreting qualitative data: Methods for analysing talk, text and interaction (2nd ed.). London: Sage.

IV. Online Advice on Designing and Analyzing Interviews

General Topics on Interviewing & Needs assessment decision aid http://www.ceismc.gatech.edu/MM_Tools/analysis.html
Heidler, Reeves, and Brackett describe the advantages and disadvantages of focus groups, interviews, and questionnaires, for the purpose of identifying needs.

Overview of qualitative methods and analytic techniques: Common qualitative methods
http://www.nsf.gov/pubs/1997/nsf97153/chap_3.htm
Mahoney describes and compares the purposes, advantages, and disadvantages of common qualitative methods: observation, in-depth interviews, and focus groups. This site includes sample instruments in the Appendices.

Online interviews
http://www.geog.le.ac.uk/ORM/interviews/intcontents.htm
O'Connor introduces an online module for conducting synchronous interviews (a real time online chat interview).

Designing structured interviews for educational research
http://pareonline.net/getvn.asp?v=5&n=12
This article in Practical Assessment, Research & Evaluation provides a brief introduction to problems and solutions for using interviews in program evaluation.

Developing interviews: Administering interviews
http://oerl.sri.com/module/mod7/m7_p1.html
Colker provides a module on how to set-up, conduct, follow-up, and analyze interviews.

Developing interviews: Preparing an interview protocol
http://oerl.sri.com/module/mod6/m6_p1.html
In conjunction with “Administering Interviews,” Colker offers a module on how to prepare and structure interview questions and interview protocols.

The use of qualitative interviews in evaluation
http://ag.arizona.edu/sfcs/cyfernet/cyfar/Intervu5.htm
Sewell summarizes interview types, the designing of an interview, types of analysis, and ethical issues in interviewing. This site also has a short annotated bibliography which can be useful for learning more about interview methodology.

Evaluate programs: Interviews
http://www.utexas.edu/academic/diia/assessment/iar/programs/plan/method/interview.php
This very useful site from the University of Texas provides a step by step guide to planning and conducting interviews, and transcribing and coding data from interviews. In addition, it suggests ways of using interview findings for evaluative purposes, including how to report.

V. Online Advice on Focus Group Techniques

General Topics on Focus Groups & Focus group fundamentals
http://www.extension.iastate.edu/Publications/PM1969B.pdf
Grudens-Schuck, Lundy-Allen, and Larson summarize the features of focus group methodology for evaluators to use in generating program-related information.

Can you call it a focus group?
http://www.extension.iastate.edu/Publications/PM1969A.pdf
Larson, Grudens-Schuck and Allen compare the use of focus groups with small and large discussion groups when developing and evaluating programs.

What is a survey?
http://www.whatisasurvey.info/
Scheuren illustrates planning of and data collection in a survey study, types of surveys, and concerns about error. Focus group techniques are described in chapter 5.

Use of Focus Groups in Program Evaluation
Using focus groups for evaluation
http://ag.arizona.edu/sfcs/cyfernet/cyfar/focus.htm
Marczak and Sewell describe the effective use of focus groups in program evaluation and outline how they can be used in Jacob’s (1988) five-tiered approach to evaluation.
Jacobs, F. H. (1988). The five-tiered approach to evaluation: Context and implementation. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs, New York: Aldine DeGruyter.

Focus groups
http://www.programevaluation.org/focusgroups.htm
New York State Teacher Centers Evaluation and Strategy Committee introduces focus group questioning strategies and methodological strategies when teacher development centers have limited time and resources available.

VI. Examples of Interviews and Focus Groups in Language Program Evaluation

Using Interviews

Alderson, J. C., & Scott, M. (1992). Insiders, outsiders and participatory evaluation. In J. C. Alderson & A. Beretta (Eds.), Evaluating second language education (pp. 25-57). Cambridge: Cambridge University Press.

Chaudron, C., Doughty, C., Kim, Y., Kong, D., Lee, J., Lee, Y., Long, M. H., Rivers, R., & Urano, K. (2005). A task-based needs analysis of a tertiary Korean as a foreign language program. In M. H. Long (Ed.), Second language needs analysis (pp. 105-124). Cambridge: Cambridge University Press.

Gattullo, F. (2000). Formative assessment in primary (elementary) ELT classes: An Italian case study. Language Testing, 17(2), 278-288.

Horwitz, E. K. (1985). Formative evaluation of an experimental foreign-language class. Canadian Modern Language Review, 42(1), 83-90.

Karava-Doukas, K. (1998). Evaluating the implementation of educational innovations: Lessons from the past. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 25-50). London: Longman.

Kieley, R. (1998). Programme evaluation by teachers: Issues of policy and practice. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 78-104). London: Longman.

Liskin-Gasparro, J. E. (1995). Practical approaches to outcomes assessment: The undergraduate major in foreign

languages and literatures. ADFL Bulletin, 26(2), 21-27.

Long, M. H. (2005). Methodological issues in learner needs analysis. In M. H. Long (Ed.), Second language needs analysis (pp. 19-76). Cambridge: Cambridge University Press.

Lynch, B. K. (1992). Evaluating a program inside and out. In J. C. Alderson & A. Beretta (Eds.), Evaluating second language education (pp. 61-99). Cambridge: Cambridge University Press.

Lynch, B. K. (2000). Evaluating a project-oriented CALL innovation. Computer Assisted Language Learning, 13(4-5), 417-440.

Mitchell, R. (1989). Second language learning: Investigating the classroom context. System, 17(2), 195-210.

Mitchell, R. (1990). Evaluation of second language teaching projects and programmes. Language, Culture, and Curriculum, 3(1), 3-15.

Mitchell, R. (1992). The "independent" evaluation of bilingual primary education: A narrative account. In J. C. Alderson & A. Beretta (Eds.), Evaluating Second Language Education (pp. 100-140). Cambridge: Cambridge University Press.

Pawan, F., & Thomalla, T. G. (2006). Making the invisible visible: A responsive evaluation study of ESL and Spanish language services for immigrants in a small rural county in Indiana. TESOL Quarterly, 39(4), 683-705.

Rea-Dickins, P. (2001). Mirror, mirror on the wall: Identifying processes of classroom assessment. Language Testing, 18(4), 429-462.

Roberts, C. (1998). Language and cultural issues in innovation: The European dimension. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 51-77). London: Longman.

Slimani, A. (1992). Evaluation of classroom interaction. In J. C. Alderson & A. Beretta (Eds.), Evaluating second language education (pp. 197-221). Cambridge: Cambridge University Press.

Snow, M. A., & Brinton, D. M. (1988). Content-based language instruction: Investigating the effectiveness of the adjunct model. TESOL Quarterly, 22(4), 553-574.

Weir, C., & Roberts, J. (1994). Evaluation in ELT.Oxford: Blackwell.

Using focus groups

Dassier, J-L. P., & Powell, W. (2001). Formative foreign language program evaluation: Dare to find out how good you really are. In C. C. Maurice (Ed.), The Odyssey Continues: Dimension 2001. Selected Proceedings of the Joint Conference of the Southern Conference on Language Teaching and the South Carolina Foreign Language Teachers' Association (pp. 91-110). GA: Valdosta State University.

Lynch, B. K. (2000). Evaluating a project-oriented CALL innovation. Computer Assisted
Language Learning, 13(4-5), 417-440.