Multimedia

Checklist: Evaluative criteria for computer-delivered language learning systems
compiled during the Invitational Symposium on Assessing and Advancing Technology Options in Language Learning
(AATOLL) February 1998

General description of the program

Name/version ___________________________

Intended users

age

o children

o high school

o college

o learners with special interests (tourists, businessmen, etc.)

ability level

o beginning

o intermediate

o advanced

Intended use

o self-instruction

o textbook/classroom supplement

o combination of above

Content

o content likely to be of interest to intended users

o instructional units logically sequenced

o lessons build on material presented earlier

Program goals

o goals and objectives of the program clearly described

o program capable of meeting the stated goals

Scope

number of lessons or instructional units _________________________________

equivalence in terms of hours, semesters, years of instruction ______________

Documentation

o discussion of programís goals, design, and contents

o tutorial on programís operation

o demonstration lesson included

o lesson plans for use in a course or guidance for individual use

o research results on effectiveness of the program available

o information about independent reviews of the program

Customization

o choices depending on user level

o users can customize the operation of the program

o variety of options to support different types of learners

Installation/operation

o installation instructions clear

o program easy to install/uninstall

How often does the system crash? _________________________________

Is there a particular action that causes regular crashes? _________________________________

Hardware requirements

type of computer _________________________________

version of operating system _________________________________

memory (RAM) _________________________________

disk space _________________________________

CD-ROM drive/DVD-ROM drive _________________________________

speakers _________________________________

microphone _________________________________

monitor _________________________________

sound card/video card _________________________________

network/browser requirements _________________________________

other _________________________________

Media

o graphics, video and audio help users to concentrate on content

o text easy to read

o audio clear

o colors and graphics crisp and clear

o video runs smoothly

o hypertext links work well

o animation serves a pedagogical purpose

Feedback

o users receive quick feedback to responses

o level of feedback language appropriate for intended users

availability of feedback (e.g., item-by-item, logical content break, end-of-unit or session, learner-controlled) _________________________________

type of feedback is available

o indication as to whether response is correct or incorrect

o invitation to try again

o hint leading to correct response

o explanation as to why a particular response is incorrect

o a "give-up" option

o directions to relevant part of program for review

o intelligent feedback depending on learner response history

Evaluation/tests

o frequency of the tests

o type of scores is made available to users

o explanation of test scores

o cumulative log of scores to track progress

o recommendations for remedial action given

o tests and/or test scores can be printed out

Special features:

o speech recognition

o system requires initial training

What do users need to do to train the system? ___________________________

_________________________________

How frequently does the system recognize a range of native speech? _________

How does the system react to nonnative speech? _________________________

___________________________

___________________________

How extensively is speech recognition/processing utilized in each unit of the program? ___________________________

___________________________

function of speech recognition/processing in this program

o voice navigation

o pronunciation instruction

o speaking practice

Developer/distributor

name ___________________________

street address ___________________________

telephone ___________________________

fax ___________________________

e-mail ___________________________

URL ___________________________

Price

individual ___________________________

institutional ___________________________

o users likely to get their moneyís worth

Checklist for listening software

Listening input

audio

o conditions against which the audio is played authentic

o tempo of the sound track natural

o variety of voices and dialects

o sound track supported by video or graphics

listening passages

o passages authentic

o topics of interest to intended users

o variety of topics

o users can choose among several passages on the same topic

o variety of genres

o passages of an appropriate length for intended users

o vocabulary appropriate for the intended level

o syntax appropriate for the intended level

Listening activities

pre-listening activities

o program adapts to different levels of usersí prior knowledge

o pre-listening activities that activate prior knowledge

listening activities

o listening activities emphasize comprehension of the passage

o listening activities authentic

o variety of listening activities

o users have a choice of listening activities

o activities motivate learners to keep listening

post-listening activities

o post-listening activities based on selected features of passages

o features selected for special attention well chosen

o post-listening activities promote acquisition of vocabulary

Listening strategy instruction

o explanation about the benefits of using listening strategies

o systematic practice in the use of listening strategies

o users can see that certain tasks are more easily accomplished when strategies are used

o choice of listening strategies depending on learning style

o feedback on the effectiveness of strategy use

Listening tools

links to the written version of passage

o the whole passage

o sentence-by-sentence

o phrase-by-phrase

o word-by-word

spoken glosses

o monolingual

o bilingual

o hint-type

written glosses

o monolingual

o bilingual

o hint-type

visual glosses

o images

o graphics

o videos

additional resources available

o online talking dictionary

o online written dictionary

o online reference grammar

o background information

o cultural notes

Listening interface

o ease of navigation

o navigation between audio, activities, glosses, and tools simple

o navigation between screens fast

playback control

o users can control the speed of audio playback

playback can be stopped

o after each phrase

o after each sentence

timing

o enough time to complete activities

archiving

o usersí work can be saved

o usersí work can be printed out

Checklist for reading software

Reading input

How authentic is the appearance of the passage?

o layout

o fonts

o illustrations

o colors

content of the reading passages

o reading passages authentic

o topics of interest to intended users

o variety of topics

o topics not likely to become outdated

o variety of genres

o users can choose among several passages on the same topic

o length appropriate for intended users

o vocabulary appropriate for intended users

o syntax appropriate for intended users

Reading activities

pre-reading activities

o program adapts to different levels of usersí prior knowledge

o pre-reading activities that activate prior knowledge

reading activities

o reading activities emphasize comprehension of the passage

o reading activities authentic

o variety of reading activities

o users have a choice of reading activities

o activities motivate learners to keep reading

post-reading activities

o post-reading activities based on selected features of the passages

o features selected for special attention well chosen

o post-listening activities promote acquisition of vocabulary

Reading strategy instruction

o explanation about the benefits of using reading strategies

o systematic practice in the use of reading strategies

o users can see that certain tasks are more easily accomplished with strategy use

o choice of reading strategies depending on learning style

o feedback on the effectiveness of strategy use

Reading tools

links available to the spoken version of the passage

o the whole passage

o sentence-by-sentence

o phrase-by-phrase

o word-by-word

written glosses available

o monolingual

o bilingual

o hint-type

spoken glosses available

o monolingual

o bilingual

o hint-type

visual glosses available

o images

o graphics

o videos

additional resources available

o topical glossaries

o background information

o cultural information

o online dictionary

o online reference grammar

Reading interface

ease of navigation

o navigation between text, activities, glosses, and tools simple

o navigation between screens fast

appearance of the text

o texts preserve their original format

o texts presented in a variety of fonts and typefaces

o fonts and typefaces attractive and easy to read

o illustrations and graphics aid in text comprehension

glosses

o equal access to different types of glosses

o glossed items marked unobtrusively

o glosses cover up text or make it disappear

o different types of glosses can be customized by users

timing

o enough time to complete activities

archiving

o usersí work can be saved

o usersí work can be printed out

Checklist for speaking software

Speaking input

general qualities of the input

o authenticity

o situational appropriateness

o cultural appropriateness

o interest to intended users

topics

o topics of interest to intended users

o variety of topics

o topics logically sequenced

o users have a choice of topics

vocabulary

o vocabulary appropriate for the intended level

o vocabulary organized into well-defined thematic units

o program focuses usersí attention on specific vocabulary

o vocabulary recycled throughout the program

grammar

o grammar appropriate for the intended level

o presentation of grammar logically sequenced

o program focuses usersí attention on specific structures

o grammar recycled throughout the program

o grammatical explanations clear

Speaking activities

o speaking activities simulate real-world interactions

o activities challenging and fun

o activities progress from mechanical drills to open-ended responses

o activities require integration of linguistic, sociolinguistic, and pragmatic skills

o speaking activities contextualized

o variety of activities

o users have a choice of activities

What is the quality of the speaking prompts? ___________________________

o examples for the activities to be performed

responses allowed

o only one

o more than one

Speaking strategy instruction

o program explains why the use of communication strategies can be helpful

o systematic practice in the use of communication strategies

o users can see that a task is more easily accomplished when a particular communication strategy is used

o choice of communication strategies depending on learning style

o feedback on effectiveness of strategy use

Speaking tools

online tools are available

o reference grammar

o monolingual dictionary

o bilingual dictionary

o talking dictionary

o thesaurus

o sociolinguistic and cultural notes

Speech interface

o speech interface user-friendly

speech recognition (SR)

o pedagogical goals of the program could be achieved without SR

tasks associated with SR

o creating a sentence with audio and/or visual cues

o conversing with the system (conversation changes depending on response)

Feedback from SR

o Is the feedback from SR correlated with how well something was said

o feedback relies only on learnersí own assessment

o users can record and compare their responses with correct models

SR program provides feedback on accuracy

o in responses chosen from a closed set

o in open-ended responses

archiving

o usersí work can be recorded and saved

Checklist for writing software

Input for writing activities

input provided in order to contextualize writing tasks

o written texts

o listening passages

o videos

o input at the appropriate level of difficulty

o variety of input

o users have a choice of type of input

Writing activities

o writing tasks have counterparts in the real world

o tasks at the appropriate level of difficulty

o tasks culturally appropriate

o writing prompts clearly and concisely worded

writing prompts specify

o an audience

o a function

o a motivation for writing

o choice of writing tasks

Writing strategy instruction

o explanation of he benefits of using writing strategies

o program offers systematic practice in the use of writing strategies

o users can see that certain writing tasks are more easily accomplished when strategies are used

o choice of writing strategies depending on learning style

o feedback on the effectiveness of strategy use

Writing tools

macro-aides available

o online templates for brainstorming writing tasks

o online templates for organizing the writing tasks

o online samples for imitation

micro-aides available

o online bilingual dictionary

o online thesaurus

o online grammar reference

editing aides available

o online spell checker

o online grammar checker

Writing interface

o enough time given for task completion

o program can save successive drafts of usersí work

o usersí work can be printed out

o program can keep track of user queries

o program includes a network component for group work or peer editing

Checklist for vocabulary learning software

Input for vocabulary learning

basis for the choice of vocabulary included in the program

o frequency

o association with a particular topic or topics

o association with a particular textbook

vocabulary appropriate for the intended users in terms of

o level of difficulty

o interest

o usefulness

Vocabulary learning activities

o activities challenging and fun

o users have choice of difficulty level

o activities relate new vocabulary to vocabulary previously introduced

o users have a choice of activities

vocabulary learning activities offered

o repetition of the word

o recognition/matching activities pairing the word with its native language equivalent

o pairing the word with its target language synonym

o pairing the word with its target language antonym

o finding the related word in a list

o pairing the word with a picture

o pairing a word with a video

o listening to the word being pronounced and finding it in a list

production activities

o giving the target language equivalent for a native language word

o finding the right word to describe a picture

o finding a target language synonym for a word

o finding a target language antonym for a word

o putting a word in the appropriate semantic group

o producing derivationally related words (e.g., noun from a verb)

o using the word in a sentence

o games

Vocabulary learning strategy instruction

o explanation of different kinds of mnemonic strategies

o explanation of why the use of certain strategies can improve retention of vocabulary

o systematic practice in the use of mnemonic strategies

o users can see that a task is more easily accomplished when a mnemonic strategy is used

o users offered a choice of mnemonic strategies depending on their learning style

o feedback on whether the use of a particular mnemonic strategy was effective or not

Vocabulary learning tools

o vocabulary search engine

glosses provided

o monolingual

o bilingual

o hint-type

o examples of usage

o visual

o audio

o video

Vocabulary learning interface

vocabulary search engine

glosses provided

o monolingual

o bilingual

o hint-type

o examples of usage

o visual

o audio

o video

Checklist for software to teach pronunciation

Pronunciation input

words presented

o in citation form only

o in different contexts

o suprasegmental features included in the program

o intonation included in the program

o native-speaker models include a variety of voices and dialects

Pronunciation activities

o activities well-sequenced

o activities challenging and fun

o activities progress from easier to more complex

o users have control over selection of activities

o opportunities to review the material

tasks associated with the use of speech recognition

o reading one word at a time

o filling in a blank in a sentence

o reading from a selection of several words

o reading one sentence at a time

o reading from a selection of 2-5 sentences

o reading from a selection of 6 or more sentences

o creating a sentence with audio and/or visual cues

Pronunciation interface

speech recognition system reaction to nonnative speech

o recognizes despite mistakes

o asks for repetition

o does not recognize

o recognition system robust to various disfluencies

o recognition system sensitive to ambient noise

feedback recognition system provides

o correlates with how well something was said

o relies only on learnersí own assessment

o users can record and compare their responses with correct models

mistakes in learner responses are marked and displayed

o accuracy of vowels and consonants

o accuracy of speech rhythm and timing

o accuracy of intonation contours


method by which visual pronunciation feedback presented

o speakerís face

o vocal tract

o spectrum information

o speech waveforms

Checklist for software to teach script

Script input

basis for selection of characters

o frequency

o topic association

o textbook association

o information about the history of the writing system

o general description of the writing system and its distinguishing features

Script learning activities

characters (or symbols) practiced

o in isolation

o in combinations

o in meaningful sequences

o from the more simple to the more complex

o characters (or symbols) presented earlier are recycled throughout the program

o variety of activities

o users have control over choice of activities

Script strategy instruction

o program explains what techniques are helpful in learning to write the target language script

o program provides systematic practice in the use of these techniques

Script learning tools

o characters (or symbols) linked to their pronunciation

links to glosses

o monolingual

o bilingual

o images

o video

Script learning interface

o program shows stroke order of the characters or symbols

o animation of how characters or symbols are written

o search engine

National Foreign Language Resource Center, University of Hawai`i

©1998 Second Language Teaching & Curriculum Center, University of Hawai`i