Please enable JavaScript.
Coggle requires JavaScript to display documents.
Chapter 12: Evaluating Teaching and Learning at a Distance (Evaluation and…
Chapter 12: Evaluating Teaching and Learning at a Distance
Evaluation and Distance Education - Five Steps
Level 1 - Reactions (Did they like it?)
: At this level of evaluation, participants are asked what they did and didn't like about training that they were involved in. This can be done once or multiple times throughout a course.
Comes in the form of checklists, open ended questions, and responses to statements.
Level 2 - Learning:
This part of the evaluation process tries to find out what the participants actually learning during the course/training. This also identifies if new skills were gained since participating in the course/traIning.
Comes in the form of tests/assessments of some kind. These assessments can be done in teams, by themselves, or through pre and post-testing.
Level 3 - Transfer:
This is where the evaporators try to determine if the course information/things that the participants learned are being transferred to the workplace. Are the participants using their new skills in the way that they should? While timing is imperative, it can often be problematic.
QUESTION
: Could this be done in the form of observations?
Level 4 - Results
: This is where the success of the training/course is measured. This is looked at through improved quality, lower costs, and for businesses.
The level of evaluation looks at learning motivation, educational achievement, and enrollment in additional courses.
Level 5 - Return on Investment:
This is where training results are converted to monetary values and the values are then weighed against the cost of the course itself to decide if there is a return of investment.
This is very time consuming, needs a skilled evaluation team, and can be criticized because it looks at what has happened and not what could happen.
Evaluation and the Open University
: The Open University merged two evaluation strategies, a traditional strategy, and an eclectic strategy. Six evaluation categories can be collected about distance education.
Measures of Activity
: "These measures are counts of the numbers of events, people, and objects." (Simonson, 2015, p. 310)
Measures of Efficiency
: "Measures of efficiency are closely related to measures of activity, and often administrative records can be the source of efficiency information." (Simonson, 2015, p. 310)
Measures of Outcomes:
"Measures of adequate learning are usually considered the most important measures of outcomes of distance education activities." (Simonson, 2015, p. 310)
Measures of Program Aids
: "Some distance teaching programs specify their aims in terms of what and whom they intend to teach, and evaluation information is collected to establish the extent to which these aims were met." (Simonson, 2015, p. 310)
Measures of Policy
: "Evaluation in the policy area often takes the form of market research. Surveys of prospective students and employers can be used to determine the demand for distance education activities." (Simonson, 2015, p. 311)
Measures of Organizations
: "This category of evaluation requires on-site visits, interviews, and sometimes the use of journals by key organization leaders." (Simonson, 2015, p. 311)
Quality Scorecard and Quality Matters
Scorecard
: measures institutional support, course structure, course development and design, teaching and learning, engagement, faculty and student support, and assessment
Quality Matters
: provides a rubric for evaluating courses. On the rubric are assessments for course overview, objectives, assessment and measurement, materials, interaction and engagement, support for the learner, and accessibility.
The AEIOU Approach:
provides important questions that need to be answered in order to have an effective evaluation.
Component 1 - Accountability Questions
: These are typically answered with a yes or a no and based on a specific activity.
"Were the appropriate number of class sessions held?" (Simonson, 2015, p. 312)
"How many students were enrolled?" (Simonson, 2015, p. 312)
"How many copies of program materials were produced, and how many were distributed?" (Simonson, 2015, p. 312)
Component 2 - Effectiveness
: This component focuses on the learners attitude and knowledge.
"Were the in-service participants satisfied with their distance education instruction?" (Simonson, 2015, p. 312)
"Did the students learn what they were suppose to learn?" (Simonson, 2015, p. 312)
"Did the teachers feel adequately prepared to teach distance learners?" (Simonson, 2015, p. 312)
Component 3 - Impact
: These can be difficult to identify. Longitudinal data is an important element for this measurement.
"Did students register for additional distance education courses?" (Simonson, 2015, p. 313)
"Has the use of the distance education system increased?" (Simonson, 2015, p. 313)
"Have policies and procedures related to the use of the distance education system been developed or changed?" (Simonson, 2015, p. 313)
Component 4 - Organizational Context
: This component identifies factor's that could contribute to the courses abilities or inabilities to conduct activities.
"What factors made it difficult to implement the project or to successfully complete the course?" (Simonson, 2015, p. 314)
"What contributed most of the success or failure of the program, course, project, or the students in the course?" (Simonson, 2015, p. 314)
"What should be done differently to improve things and make the course more effective?" (Simonson, 2015, p. 314)
Component 5 - Unanticipated Consequences
: Gives good information about why some projects worked and why some didn't work.
"Have relationships between collaborators or students changed in way not expected?" (Simonson, 2015, p. 314)
"Have related, complementary projects been developed?" (Simonson, 2015, p. 314)
"Were unexpected linkages developed between groups or participants?" (Simonson, 2015, p. 314)
"Was the distance education system used in unanticipated ways?" (Simonson, 2015, p. 314)
"Did the distance education system have an impact on student learning other than that expected?" (Simonson, 2015, p. 314)
Program Evaluation: Examples
South Dakota
: Connecting the Schools Questionnaire (CSQ) evaluation uses participant observations, site visits, interviews, and focus groups. Questions were asked during this evaluation. 1) What did the educators think were biggest benefits? 2) What was keeping learners from becoming involved in distance education? 3) What were the superintendent's perceptions of the impact that distance education made? 4) Where there any impediments to distance education?
Iowa
: The state of Iowa decided to implement distance education throughout classrooms in the state. The AEIOU model was used to gain information about technical training, curriculum revisions, staff development, and school reconstruction.
Student Evaluation of Distance Education Courses
Instructor Evaluations should focus on six different components: teaching, learning, community, instructors, student, implementation, and technology use.
Evaluations that are done on a course and or instructors should be used as a learning tool. These evaluations should allow for improvement and revisions to course and instruction. This is true whether the course is done through distance instruction or classroom instruction.
Reflection
: Evaluations are a critical part of any instruction or course. Evaluations allow for instructors to reflect on the materials and activities that were used and done throughout the course. By obtaining information like engagement, student involvement, retention of course information, and the instructors comfort level with materials taught among many other, this allows the for a course or instructor to improve what they are doing and the learning experiences that are taking place.