Please enable JavaScript.
Coggle requires JavaScript to display documents.
Evaluation - Coggle Diagram
Evaluation
Validity and Reliability of Assessment Instruments
Content Validity
Characteristic
Measures if items align with curriculum and objectives.
Ensures comprehensive and proportional content coverage.
Essential in language teaching: must assess listening, speaking, reading, writing.
Criterion Validity
Characteristic
Assesses if results correlate with external standards (e.g., TOEFL).
Indicates test’s predictive power and alignment with recognized benchmarks.
High correlation = strong validity and credibility of internal assessments.
Construct Validity
characteristic
Measures if test accurately reflects theoretical concepts.
Aligns with underlying models (e.g., grammar proficiency).
Validated through theoretical alignment, factor analysis, comparison with similar instruments.
Ensures meaningful interpretation of results.
Inter-Rater Reliability
Characteristic
Evaluates consistency among different assessors.
Improved through
Statistical analysis (e.g., correlation, Cohen’s Kappa).
Use of detailed rubrics.
Assessor training.
Ensures fairness and objectivity in grading.
Test-Retest Reliability
Tests score stability over time.
Same group retakes the test after a period; results are compared.
High correlation = reliable and stable tool.
Important for tracking language skill development.
Internal Consistency
Checks if all items in a test measure the same concept.
Measured with Cronbach’s alpha (acceptable ≥ 0.70).
Example: reading comprehension items must consistently assess that skill.
Confirms the test is coherent and reliable.
Norms and Criteria for Assessment
Establishment of Norms
Involves setting performance standards based on:
Assessment data analysis
Reference to external standards (e.g., CEFR)
Ensures alignment with educational goals and realistic expectations
Requires continuous review and updates
Absolute Norms
Based on fixed criteria
Independent of peer performance
Example: 80% = pass mark
Relative Norms
Based on peer comparison
Performance depends on ranking within a group
Example: top 20% of the class
Use depends on assessment purpose and context
Application of Rubrics
Rubrics = structured tools with criteria + performance levels
Benefits
Ensure objective, transparent grading
Provide specific feedback
Reduce bias among multiple evaluators
Authentic Assessment
Evaluates real-world application of knowledge
Promotes
Engagement
Higher-order thinking (e.g., analysis, problem-solving)
Holistic evaluation of competencies
Formative vs. Summative Assessment
Formative (Assessment for Learning)
Ongoing, informal (e.g., quizzes, feedback, peer review)
Aims to guide learning
Summative (Assessment of Learning)
Final, formal (e.g., exams, final projects)
Evaluates achievement of objectives
Design of Assessment Instruments
Types of Assessment Instruments
Written tests: Evaluate theory, grammar, reading (objective, limited in practical skills).
Questionnaires: Gather attitudes and perceptions (subjective, depends on honesty).
Rubrics: Criteria-based evaluation (ideal for complex tasks, require careful design).
Portfolios: Show learning progress (time-consuming, foster reflection).
Projects: Assess real-world skills (integrative, resource-intensive).
Conclusion: Use a mix of instruments for holistic evaluation.
Structure and Format
Organize: Group similar questions, give clear instructions.
Sequence: Start with easy items, progress to complex.
Design: Use readable fonts, spacing, and visuals.
Consistency: Uniform numbering, headings, layout.
Time management: Suggest timing per section.
Selection of Measurement Scales
Numeric scales: Objective scoring, easy to analyze.
Descriptive scales: Qualitative feedback (e.g., Excellent–Poor).
Likert scales: Measure attitudes/opinions (Strongly Agree–Strongly Disagree).
Combine scales based on skill/competency assessed.
Calibration and Pilot Testing
Calibration: Review for clarity, bias, alignment with outcomes.
Pilot testing: Trial with small group; simulate real conditions.
Analysis: Evaluate item performance, reliability (e.g., Cronbach’s alpha).
Feedback: Collect from students and teachers.
Revise: Based on data and feedback; re-test if needed.
Ethical Use of Assessment: Evaluate or Grade
Privacy and Confidentiality
Confidentiality: Ensures data is only accessible to authorized individuals.
Privacy: Protects students' personal and assessment data
Equity and Justice
Equity: Equal opportunity for all learners regardless of background.
Justice: Fair and unbiased assessments.
Approaches
Use standardized, bias-free criteria
Design culturally inclusive assessments
Address language, ability, and background differences
Apply diverse assessment types (projects, portfolios)
Constructive Feedback
Feedback should be
Specific: Focused on clear, observable aspects of performance.
Timely: Given soon after the task.
Balanced: Highlight strengths as well as areas for improvement.
Actionable: Offers steps or resources for improvement.
Reflective: Encourages self-assessment and growth.
Responsible Use of Results
Use results
Ethically and fairly to support learning
For evidence-based decisions
To offer support, not punishment
Protect
Privacy: Limit access to assessment data
Equity: Avoid misuse that could reinforce discrimination
Promote
Transparency in how results are used
Accountability in decision-making processes
Technology in Assessment
Online Assessment Platforms
Digital tools for designing and administering assessments.
Features include
Customizable templates, various question types.
Remote access on multiple devices.
Automated grading and analytics.
Immediate feedback (text, video, audio).
Security measures: proctoring, randomization, anti-cheating tools.
Data Analysis Applications
Software for interpreting assessment results.
Key functions
Data import and organization
Statistical analysis (e.g., mean, regression)
Visualization (charts, dashboards)
Reporting for individuals/groups
Filtering and segmentation (e.g., by demographics)
Gamification and Simulations
Gamification: Adds points, badges, levels, and leaderboards to assessments.
Simulations: Real-world scenarios for experiential learning.
Benefits
Increases motivation and engagement
Encourages active participation
Supports diverse learning styles
Automated Responses
Use of AI to automatically grade objective questions (e.g., MCQs).
Advantages
Efficiency in grading
Consistency and fairness
Immediate feedback for learners
Detailed analytics for educators
Design of Assessment Plans
Identification of Learning Objectives
First step in assessment planning: define clear, specific, and measurable goals.
Objectives should
Be action-oriented (e.g., “analyze,” “create”).
Align with instructional content and broader curriculum goals.
Help in designing relevant assessment tools and providing targeted feedback.
Selection of Assessment Methods
Choose methods that match learning objectives and content.
Types of assessments
Formative: ongoing checks (quizzes, peer reviews).
Summative: final evaluations (exams, projects).
Diagnostic: pre-instruction analysis.
Normative: comparison against peers or standards.
Continuous Feedback
Formative feedback helps learners improve as they learn.
Should be
Timely and specific.
Constructive: identify strengths and areas for growth.
Supportive: encourage a growth mindset.
Assessment Plan Evaluation
Reflect on how well the plan achieved learning objectives.
Evaluate
Validity (does it measure the intended outcome?).
Reliability (are results consistent?).
Student feedback and performance trends.
Use findings to
Revise assessment strategies.
Improve future implementations.