Direct & Indirect Measures

Direct & Indirect Measures

1. Chapter Four: Setting Expectations & Preparing to Select Measures

   • Intentions for Learning
     • "Assessment necessarily begins before, not after, the learning experience"
     • Learning Goals- used to express intended results in general terms & consist of broad learning topics (clear communication)
       • Often included in institution mission statement
     • Learning Objectives- describe expected learning & behavior in precise terms
       • Often for a specific academic or cocurricular program
     • Student Learning Outcomes (SLO)- clear ways to describe how students demonstrate mastery of program
     • Bloom's Taxonomy
       • Six levels: knowledge, comprehension, application, analysis, synthesis, & evaluation
         • each level associated with a behavior
         • Revision in 2001; added a top level, creating
           • Also created 4 levels of knowledge: Factual, conceptual, procedural, metacognitive
       • Cognitive Objective - thinking skills vs. Affective Objectives- attitudes and values
         • Psychomotor Skills- recognize different levels of skill development
   • Developing Statements of Expectations
     • Use words such as identify, solve, construct
     • making sure the expectations are detailed and specific
     • statement content can be found in current reports, syllabi, instructional materials, catalogue copy, etc.
     • Curriculum Map - a grid that shows all the program courses as row headings and expected outcomes as column headings
     • Matrices are helpful in facilitating assessment work
   • Selecting Methods and Approaches
     • Validity and Reliability
       • Reliability- can produce consistent responses over time
         • Internal Consistency - how well items are related to each other
         • Interrater Reliability- degree of agreement among raters
       • Validity- how accurately a method measures what it is intended to measure
         • Construct- related validity- congruence between the meaning of the underlying construct and items on the test or survey
         • Criterion- related validity includes predictive validity
         • Content- related validity- refers to overall match between content of instrument and content of curriculum or other domains of interest
     • Include timeliness, cost benefit comparison, and student motivation
       • Is it taking away time from other important projects?
       • "...amount gained compared to the amount spent"
       • elicit student cooperation
     • Assessment Questions
       • asking what evidence could provide the desired results
   • Designing Instruments
     • Recognize uniqueness of designing instruments
     • Enlisting help from campus experts
     • Enhancing instrument reliability and validity
       • test blueprint
   • Determining Approaches for Implementation
     • research other strategies
       • cross sectional approach - comparing different groups of students at a common point of time
       • longitudinal design- collecting info from same set of students over a period of time
     • Identify eligible participants
       • sampling and sampling size
         • random sampling- every person has same chance
         • stratified sampling- used to ensure adequate number of people in some subgroups
         • convenience sample- the best that they can in locating participants and their ability to participate

2. Chapter Five: Using Direct Measures

   • Performance Assessment - process of using student activities or products rather than tests or surveys
     • Authentic assessment- demonstrate learning directly related to nature of discipline in which students are engaged
     • collect evidence about the performances of their students
       • examples are essays, research papers, problem sets, oral exam
     • signature assignment- assignment that most effectively displays the outcomes of a course have been achieved
   • Rubrics
     • Holistic scoring- giving performance a single overall score
     • analytic rubric- rate separately the various important characteristics of a performance
     • VALUE Rubrics- Valid Assessment of Learning in Undergraduate Education
     • Issues with Rubrics
       • tie faculty to overly simplified views of the constructs they are assessing
       • E-Portfolio
   • Objective Tests- allow students to demonstrate knowledge and abilities
     • multiple choice, true or false, and matching questions
     • standardized objective tests (i.e. licensure exams)
   • E-Portfolio- digital collections of student work including artifacts, reflective commentary, evidence linked to institutional outcomes

3. Chapter Six: Using Indirect Assessment Methods

   • Surveying- administering a set of predetermined questions to collect information from individuals
     • could ask satisfaction with aspects of their experience
     • need to know student needs for various support services
     • HEDS- Higher Education Data Sharing Consortium
     • Likert - Scale- a # corresponds to a positive or negative answer
     • Importance and preparations ratings
   • Questionnaire Administration
     • Cover Letter and Invitation
     • Anonymous Vs. Confidential Questionnaires
     • Encouraging Responses
     • Seeking Help
   • National Surveys for Assessment
     • Cooperative Institutional Research Program
     • National Survey of Student Engagement
   • Focus Groups provide opportunity to listen to student voices, explore issues, and obtain insights
   • Additional Indirect Methods
     • Interviews
       • appreciative inquiry
     • Written Materials
       • Documents & Records
   • Quantitative vs. Qualitative
     • Qualitative - open ended interviews, observations of activities
     • Quantitative- emphasis on numbers, measurement, experimental design, statistical analysis.
   • Classroom Assessment Techniques