What makes a good question? How do you write effective answer distractors? Is feedback really used on exams? What tools can be used to engage students with questions and answers? These questions and more were addressed in this week's Teach Talk Webinar, presented by Katrina Fogelson and Sarah Prosory.
What Makes a Good Question?
We need better questions to go beyond simply assessing a student's understanding. As instructors, we should be checking a student's ability to apply the material in a meaningful way.
Starting with Bloom's
Hopefully you’ve learned about objectives by now, but in brief: learning objectives help answer the question, "what do I want students to be able to do upon completion of this lesson/topic/module?" Learning objectives are one of the foundational pieces to designing and developing a learning experience. Everything in the course should tie back to them, especially assessments.So, why do we need to think about objectives when we are writing assessment questions? The questions in the assessment are how we determine if the student has met the learning objective! We start with Bloom’s Taxonomy because it provides the verbs and the level for the questions we want to create. The levels range from the simple recall of facts, to the ability to use information to create new knowledge. Each level of this hierarchy builds upon the lower levels.
What Level is Your Assessment?
First, make sure any assessment ties to your objectives. Then, what level are you assessing? Think of how students can achieve that level. They may need some lower level questions to get them to the higher level ones.Most often our questions that are multiple choice fall to the lower levels of Bloom’s. Starting at the bottom with Remember- take a look at the verbs provided, which can help begin to craft an effective question. It is possible to create ‘recall’ (remember/understand) questions, up to the more challenging ‘analytical’ (apply/analyze) questions, with multiple choice answers (Riggs, et al, 2020). For example, you could offer two different scenarios and then have students analyze each one for a set of components, ultimately choosing the correct multiple choice answer.
Effective questions in the higher levels of Bloom’s can be done with providing problem-based questions, and then for example, asking students to critique or justify the solution. The higher levels will most likely be short-answer or essay, so as to allow for students to compose a response.
When we write questions, they can fall under three different levels: basic, intermediate, and advanced. Most questions are written at the basic level. Here are the different levels of questions and what they entail.
Basic questions have a simple and common design: they ask the learner to identify the one answer that possesses a particular characteristic. This design can be used for any subject or difficulty level—from introductory to complex or technical—but they all have one thing in common: they are basic questions because they fall at the bottom level of Bloom’s Taxonomy.
Intermediate questions go beyond asking learners to identify which answer possesses an attribute or characteristic. There are many approaches, but they all ask learners to consider, and distinguish between, multiple attributes or characteristics to arrive at an answer.
Advanced questions move up the cognitive ladder of Bloom's Taxonomy by challenging learners to apply what they know when they are answering the question. As a result, they are typically presented as questions related to scenarios and procedures. Examples include: think scenarios, case studies, or application of principles and/or formulas.
This tool is good because it doesn’t involve students leaving Zoom to get into another technology tool. The poll should be added in advance of the Zoom session, and the poll can only be created by the Host of the meeting. Zoom polls offer two answer formats. Either single choice or multiple choice, meaning participants can select multiple answers (not the same as the “multiple choice” terminology we are used to). Check out the ASU Teaching Toolkit: Poll Using Zoom for more tips.
Effective questions in the higher levels of Bloom’s can be done with providing problem-based questions, and then for example, asking students to critique or justify the solution. The higher levels will most likely be short-answer or essay, so as to allow for students to compose a response.
Writing the Questions
Intermediate questions go beyond asking learners to identify which answer possesses an attribute or characteristic. There are many approaches, but they all ask learners to consider, and distinguish between, multiple attributes or characteristics to arrive at an answer.
Advanced questions move up the cognitive ladder of Bloom's Taxonomy by challenging learners to apply what they know when they are answering the question. As a result, they are typically presented as questions related to scenarios and procedures. Examples include: think scenarios, case studies, or application of principles and/or formulas.
Question Banks & Groups
A great way to incorporate different question levels into your quizzes or exams on Canvas, is to use Question Banks. We recommend creating question banks for each level, and then using Question Groups so you can pull a specific amount of questions from these banks. The key is to make sure that all the questions in the bank are the same level, as you don’t want to create disadvantages to students. Using Question Groups also limits opportunities for cheating, if the banks have enough questions to pull from, it shuffles the questions so students will not see the same exams.
How to Create Good Answer Distractors & Why
It is very important to make the incorrect answers plausible for the content that is being covered, as incorrect answers also serve to teach the learner about the material. Here are a few guidelines to follow when creating distractors:- Must be relatable to course material
- Keep foils approximately the same length as the correct answer, if possible
- Do not use "All of the Above" or "None of the Above"
- Keep sentence structure the same, if possible. (If correct answer choice starts with a verb, continue that with other options)
Best practices for writing incorrect answers
Finding effective distractors can be challenging. Effective learning depends on answer choices that:- Are plausible or "competitive"—not obviously wrong;
- Are explainable beyond "this answer is wrong"—learning happens in feedback and details in the explanation;
- Do not suggest why the answer might be right or wrong.
- Only one answer can possibly be correct.
- Other items can be explained by how they vary or differ from the correct answer.
- All answer choices are related and relevant.
Providing Feedback
By giving good, in-depth feedback when students most need it, you can help students continuously engage with the material and improve performance on assessments. Instructors should provide feedback that guides students toward deeper learning and should inspire them to succeed.
Feedback, as we know, involves more than circling errors on a student's work with a red pen. Feedback should acknowledge student efforts while providing opportunities for improvement and fueling a growth mindset. Watch this short video describing what effective feedback is, how it can be used, and the potential impact on student learning.
Feedback is an ongoing process of goal-setting, gathering evidence, providing instruction, and improving performance. There are three things to consider when looking at feedback:
Feedback, as we know, involves more than circling errors on a student's work with a red pen. Feedback should acknowledge student efforts while providing opportunities for improvement and fueling a growth mindset. Watch this short video describing what effective feedback is, how it can be used, and the potential impact on student learning.
Feedback is an ongoing process of goal-setting, gathering evidence, providing instruction, and improving performance. There are three things to consider when looking at feedback:
- Where is the learner going?
- Where is the learner right now?
- How does the learner get there?
Engage Students with Q & A
Students are engaged with your course content as they consider the questions and answers, reflect upon what they’ve learned, determine what they need to learn, and even study with peers with the creation of their own questions and answers. There are many types of technology to use for this, and we will briefly cover each one.Zoom Polling
Zoom has polling embedded, but it doesn’t often get used. If you are not using iClicker, you should consider adding brief polls to your Zoom sessions. There are limitations (no grading, no short answer), but the goal with this technology tool is to allow for those pauses to check for understanding. And the key here are the answer distractors! If they are too easy, is it a true reflection of the accurate answer chosen?This tool is good because it doesn’t involve students leaving Zoom to get into another technology tool. The poll should be added in advance of the Zoom session, and the poll can only be created by the Host of the meeting. Zoom polls offer two answer formats. Either single choice or multiple choice, meaning participants can select multiple answers (not the same as the “multiple choice” terminology we are used to). Check out the ASU Teaching Toolkit: Poll Using Zoom for more tips.
iClicker
Our newest student response system is iClicker. Once you set this up for your class and tell students you will be using it, iClicker can be a powerful tool to engage students with your course content. On class day, you’ll open the desktop software “iClicker Cloud,” choose your course, and click the play button to begin polling. iClicker takes a picture of your screen, so you’ll want to embed the Q & A in your presentation slides. You can use PowerPoint or Google Slides.iClicker offers 4 types of question formats:
StudyMate is a tool that can be used within Canvas, for creating practice quizzes or flashcards. When students come up with the questions and answer distractors, they are actively participating with the course content! With practice, they are studying, helping their peers study, and providing instructors with questions and answers that could be used in future practice quizzes or exams. As students create “projects” in StudyMate, an instructor or a TA can review the questions/answers and award bonus points for creating these practice quizzes or flash cards.
The best part, this is already available in Canvas at ASU! No need to create an account! There is an app for students to use. And video tutorials within Canvas to help you get started. One limitation I saw was that you cannot add feedback to the answers. So if they select incorrectly it doesn’t provide guidance.
Post author:Sarah Prosory is an Instructional Designer within the School of Life Sciences at Arizona State University. She has worked in higher education for 10 years, supporting faculty in law, engineering, and biological sciences. Her experience includes assisting faculty with in-person, blended, and hybrid courses, as well as making the leap to fully online courses. She provides training to faculty and teaching assistants on how to use educational technologies, and shares best practices in course design to improve the student experience.
- Multiple Choice: most common type, only allows for 5 answer choices, can also use for true/false.
- Short Answer: 140 character limit.
- Numeric: Answers accepted are only numbers.
- Target: Often called "hotspot" this question type allows for students to click an area on the image you are displaying.
Canvas Practice Quiz
We do recommend offering practice quizzes in Canvas, so students can practice the format and types of questions that are similar to what they will expect on exam day. A Practice Quiz in Canvas does not get graded, and will not be in the gradebook. Good practice is to allow unlimited attempts, and definitely provide feedback on the answer responses.
StudyMate- What if Students Wrote the Questions?
Up to this point we’ve primarily talked about questions and answers that the instructor creates for students. But what if students created the questions and answers?! Research provides evidence that question or problem authoring by students leads to better exam or even course success (Riggs, et al, 2020).StudyMate is a tool that can be used within Canvas, for creating practice quizzes or flashcards. When students come up with the questions and answer distractors, they are actively participating with the course content! With practice, they are studying, helping their peers study, and providing instructors with questions and answers that could be used in future practice quizzes or exams. As students create “projects” in StudyMate, an instructor or a TA can review the questions/answers and award bonus points for creating these practice quizzes or flash cards.
The best part, this is already available in Canvas at ASU! No need to create an account! There is an app for students to use. And video tutorials within Canvas to help you get started. One limitation I saw was that you cannot add feedback to the answers. So if they select incorrectly it doesn’t provide guidance.
Apply What You've Learned
So, what's your next step? Reviewing exam questions? Attending a training? Implementing a tool for engaging students with the course content? Share in the comments below, or email us at tic @ asu.edu! Reach out if you'd like support with any of the above.
Join us next month on October 13th at noon, for the topic: "Cultural Lens Approach to Online Teaching and Learning: Small Manageable Strategies and Big Returns," with Courtney Plotts, Ph.D.
Webinar Resources
References
- ASU Sync Assessment Workshop: Self-Enroll Code
- ASU Teach Online: Assessments
- ASU Teaching Toolkits: Brame, C., (2013) Writing good multiple choice test questions. Retrieved September 8, 2020 from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/.
- Canvas Quizzes
- iClicker: How to Run a Polling Session
- Riggs, C. D., Kang, S., & Rennie, O. (2020). Positive Impact of Multiple-Choice Question Authoring and Regular Quiz Participation on Student Learning. CBE—Life Sciences Education, 19(2). doi:10.1187/cbe.19-09-0189
- StudyMate (requires ASURITE login)
- Writing Good Multiple Choice Test Questions, Vanderbilt University Center for Teaching
- Zoom Help Guide- Polling for Meetings
Comments
Post a Comment