Last time, we talked about the different steps involved in the process of making a test, all the way from planning to moderating and trialing your test and then moderating your results. Today we are going to be looking at specifically different types of tests and questions. So this is a lot of things you are already familiar with, such as multiple choice questions and closed questions, but we are going to be looking at the different aspects of them and how to make good test questions in a variety of ways, in a variety of methods. But first we are going to be looking at multiple choice, so this is a little funny, like rolling the dice, it's a bit up in the air. It is pretty easy to guess right on the multiple chioce test. We can't always be sure about the results. If students do well on the multiple choice test, we cannot be sure they are actually good at the skill, because multiple choice tests are indirect. So as you can see on the screen, we have got six different characteristics of multiple choice items. See if you can get those answers right.
Multiple choice items are only tests of recognition of
That means I don't have to produce something, I only have to recognize the right answer. So they only test recognition knowledge. Second, they are guessable like we saw in the previous picture. rolling the dice. It is easy to guess the right answer with certain techniques, without testing the skill. Three, restrictive of what can be tested. You can't test everything with a multiple choice question, or you can't do it well. Four, it is difficult to write a good So for a test taker, multiple choice tests are wonderful, and they are so easy, they love them, but for the test maker. Whew! It can be a nightmare. It is difficult for a multiple choice test maker to make items with good distractors. If you have any experience with test writing, you know what I say is true. If you don't have experience, wait until next week, because you are going to get experience. and it can be quite difficult at times. Difficult to write good items. Five, it can be harmful to teaching. Cause negative backwash; we don't want negative backwash. Sixth, it is easy to cheat on multiple choice tests. All you have to do it see a letter, or see the bubble, very easy to cheat on. And something that is worse than multiple choice is true/false. So, you should never use true/false if you don't have to. If we have to use multiple choice items, we want to make sure at least they're good. So let's look at some ways to improve multiple choice questions. Try to test only one language point at a time. If you test more than one element of language in a multiple choice question, and students get it wrong, we don't know which point the students did not understand. So, for multiple choice items, especially for a diagnostic test to define the problem, you should only test one language point at a time. Also, the options (the distractors, like A, B, C, D) they should be of similar length. You don;t want a short A and a long B,short C, long D. It doesn't look good. And if they are similar length it is harder to guess the right answer. We also want to rephrase words from the text. So try not to write test items which have the exact answer. Third is it is like alanguage in the tesxt we want to paraphrase, to use words with similar meaning to the test. Also, giving more choices. Most multiple choice questions have four options, but if you can make it five, that is even better. So on a multiple choice test with four options, A, B, C, D, your chance of guessing right is 25%. If you add E, that's five options, then suddenly that drops to 20%, better than 25. We also need to make solid questions, that, um, as I said before are plausible. Make sure that all the distractors are believable. It is very difficult to write believable, plausible, ___ items that are of a similar length, but if you are going to write multiple choice that is what you are goint to have to do. Here we have got a little cartoon for you. Brian's guide to studying. It says "exam techniques: when in doubt choose C". I am sure you have heard that before. I heard that when I was a high school student. If not C, then try "photosynthesis." Maybe it is a high school biology class. This is just to illustrate that sometimes students try to learn how to be good test takers, rather than learning the skills . . .
...that they need, that we are trying to test. So for example, many students who are really good at getting high scores on English tests, aren't actually good speakers of English. Or maybe they can't write English very well, they can't communicate very well. We don't want to have that situation. We don't want to make it easy for students to guess the right answers. That's called using a "test-wise" technique. So something like "guess C", that's a test-wise technique. Or, uh, "look for the long option". If there are three short and one long, choose the long one! That's a test-wise technique. If students can get the answer right with just a test-wise technique, that's going to decrease your test reliability. That's one more reason that we want to write believable distractors. So if we have four choices and they are all believable then there's a 25% chance of guessing correctly. If D is completely unbelievable, then they have a 33% chance of guessing correctly. That's not good.
So now we are going to look at an example test question. And I want you to figure out what is wrong with the test item. Question: "Why was Sam upset?" A: It was Tuesday. B: He is moody. C: He lost his homework. D: He didn't do his homework so his mom said that he could not go outside and play with his friends. Okay, so you didn't see the text. You don't know what the context is for this question. But you can probably guess that the right answer is D. It's so much longer than the others, almost every student is going to guess D must be the right answer! It's so detailed. So this is an example of a test-wise technique and we do not want to allow students to use them, So we should try to make our distractors similar.
So let's look at two examples here. I've got the same question, uhm, modified, so instead of the very long answer now I've changed D. "He couldn't play with his friends". So this is much shorter, so that's much better. Because now the options are of similar length, they're still not great, but it's better. So you've got two options. A, the first option, is like "dun-de-dun-de-dun-de-duh" and then the second one is like this "dun-dun-dun-dun". This is better. The second one is better. It just looks better if you write your distractors from smaller to largest, so try to keep that in mind.
But even with this second example, even though D is shortened compared to the original, it's still about twice as long as A. But I can't think of a way to shorten it any more. So what can I do? I can lengthen A, B, and C. So let's look at a new example. So now I have made "He is moody" into "He is a moody person." "It was Tuesday", "He is upset on Tuesdays". "He lost his homework", "He had lost his homework". So just by adding a few words but still keeping grammatical rules, I can make the options longer and now students will be less likely to choose D. So if possible, we want to shorten the distractors, but if we can't shorten them anymore, we've got to lengthen the other ones so that they will be standardized or similar to each other.
Let's look at another test question, you can see what's wrong with it. This question says "Which of the following is not true?" A: John went to the park. B: John is really a woman. C: Mark is good at science. D: Mark is not John's friend. Now this one might be harder for you to find the error on. But look at the test question and then look at option D. Both of them are asking "not". So which of the following is "not" true? Mark is "not" John's friend. Uh, to figure out the answer to this I have to use some complicated language in my head. So if it's not true that he is not his friend then it IS true that he IS his fr-- Is that right? So this is a tricky thing. It's like, it's more a test of understanding instructions or double negatives. And you know that is different in Korean than in English. So that's too confusing. So you don't want to have "not" in the question stem or the question AND in the answer choices. You should only have it in one place, not in both. So, instead of saying "What is not true? Mark is not John's friend". You can have "What is not true? Mark is John's friend" or change one of the other answer choices. But having "not" and "not" is too confusing, it decreases test reliability because students will often get this kind of question wrong because they don't understand what you're asking. Or they can't think that way. "Not not" double negatives, too confusing.
( Stopped a little after 10:59 )
Let's look at another test question. On this one we have a picture of a family tree. And there are statements about the family, and you're going to find the one that is true. And there's a problem with this one, so I'm gonna let you have a minute, look at the question, see if you can figure out the problem. You're gonna have to actually try to figure out the answer for this one. Look at the family tree.
Okay, if you've had a chance to check the distractors, hopefully you have found that answer B is correct. Bo is Jill's brother. That's right. Unfortunately, that is not the ONLY correct item. D is also true. Bob and Mae are siblings. So if I wrote this test question, I'd probably only put that B is the answer. But if there's more than one answer and I'm not counting both answers correct, that's gonna create big problems for scoring reliability and scoring validity. So it's going to affect both. So make sure that there is only one right answer to your question, especially if it's a multiple choice question.
Let's look at another test question. And find out what's wrong with it. This one says, "Choose the meaning of the underlined word in the paragraph." Well, the good thing is it's contextualized. We know that there's a paragraph and the word is in the paragraph. That's at least something good. To find the problem with this one, we have to look at the distractors.
A: Excited
B: Thrilled
C: Agitated, and
D: Enthused
Okay, we don't know what word is underlined in the paragraph because we can't see the paragraph. But we can figure out the correct answer.
How can we figure out the correct answer? 3 out of 4 of these choices have similar meanings. "Agitated" is the one that is different. So "excited" and "thrilled" and "enthused"... those all have positive meanings. "Agitated" has a negative meaning. It is the only one that is different, so that one has to be the answer. So having distractors with similar or same meanings is very bad for test reliability because students can figure out they're not the right answer, even without seeing the actual paragraph or even knowing what the underlined word is.
Let's take a look at another example. Find something wrong with it. "Fill in the blank. Marge is feeling ____ because her mother is coming to visit her. Elated - Exciting - Worried - Nervous".
So this one has a problem with the distractors. See if you can figure out the problem. Look at A, C, and D. What kind of words are they? "Elated, worried, nervous"... they're all adverbs. B is an adjective. It is the only one that is different, so then students can easily guess the answer must be B, "Marge is feeling exciting".
Is that true? No. So then students would get it wrong by a testwise technique. So before, I gave you an example where students were using testwise techniques to get it right. Now, they're gonna use the same testwise technique looking for the different one and get it wrong. But we don't want the students to get their answer wrong because of a testwise technique. We only want them to get it wrong if they can't solve the actual question.
<ended transcription at 16:05>
So, testwise techniques can be used for good or for bad, but we don't want them used at all. So, all four answer choices need to be similar. So in this case, A, C, and D are adverbs B is an adjective. So, now we have to think: what is the other problem? Even without knowing the text, B is not grammatically correct. "Marge is feeling____" only an adverb can fill in that blank. So, I hope the answer is not "exciting" because it's not even grammatical. And if we're trying to test comprehension -- not grammar -- all of the answer choices should be grammatical for reliability purposes. So, if you're gonna have adverbs, make sure they are all adverbs. Unless you are testing for grammar, make sure all of your option choices are grammatical. Next, we have another test question. Read the passage, then answer the questions. See what's wrong with this one? Which of the following is true about the earth?
A) It is the only planet
B) people can live on it
C) it has water and trees
D) people cannot live on it
Ok? What's the first problem you see? there are a few different problems but what's the obvious problem? Look at B and D. Look at those options. So B and D are opposites. B says people can live on it. D says people cannot live on it. If we have two that are opposites, we know only one of those can be true. So, if B and D are opposites, I know the answer cannot be A or C. So, now my chance of guessing right is 50%. That's not good. So, as I said before, that's not the only problem. So now let's look at, uh, another part of the problem. Think about it again. It's the only planet, people can live on it, it has water and trees, people cannot live on it. What's the other problem? This is testing general knowledge. There's a reading passage but you do not have to read it to guess the right answer. You know people can live on it. What is true about the earth? It's not the only planet. Uh, we know it has water and trees. And people can live on it. So here we have two problems: one, it's testing general knowledge -- not comprehension; and two, there are two correct answers: B people can live on it and C it has water and trees. Yes. so this is the- a big problem um, we don't wanna have two distractors that are opposites. We don't want to have a question -- any question -- that is meant to test comprehension, but really only tests general knowledge. Students shouldn't already know the answer without reading the passage if it's a reading question. If it's a listening question, students should only be able to get the right answer by listening and comprehending the listening.
Also, we don't wanna have questions where there's more than one answer because that's gonna cause problems for scoring reliability and scoring validity. Let's look at another example here. So, in theory, there was a dialog. And then you're going to choose the best answer. So look at these four answer choices -- the distractors -- and find the problem.
Ok? Hopefully you've had time to read over them.
A) Tom is Harry's older brother
B) Harry likes to fly kites in fall
C) Tom is learning how to read
D) the boys goes to school together
Ok, well, without even reading the answer choices, I can find one thing that's good. If you'd look at them, A, B, C, and D are of similar length. And they're all sentences. Right? So, they look similar. That's one positive aspect. But, there's a problem with choice D, "the boys goes to school". It's not grammatical. You want to make sure that all of your answer choices are grammatical. Since D is not grammatical -- even if it's true -- some students will not choose D because they know that it is not grammatically possible. So, again, this really decreases the reliability of your test. And, if we're not testing, grammar, make sure your grammar is correct because you want to always be providing good input for your students. Authentic input, grammatically correct input, input that they can use to increase their own language skills.
Ok. Let's consider true/false questions. I've mentioned these a little bit before. So on this one, what's the chance of guessing correctly? Example: John is Mary's brother. True or False?
Well, in a regular multiple choice question, if four options are there, you have a 25% chance of guessing correctly. If there are only two options, it's 50/50. So, it's too easy to guess this one correctly. Sometimes, true or false is much easier to write than a multiple choice question. Before I mentioned, when you're writing a multiple choice item, there's so many things to consider: we have to make similar length of distracters, we wanna make sure they're grammatical, we want to make sure they're believable, they should all sound logical, they should use a similar structure. We can't have -- you know -- three adjectives and one adverb. We want to have them as consistent as possible. That's really difficult. It takes a long time. Sometimes you're gonna wanna write true/false questions instead. But, we have the problem of 50/50. But there is a way to overcome this. Let's look at the example: John is Mary's brother. True or false? Harry is Mike's cousin. True or false? So, here, I've got one question but two parts. So, I have a 50% chance of guessing John is Mary's brother. And I've got a 50% chance of guessing Harry is Mike's cousin. Now, in case you don't remember how to -- you know -- calculate percentages, I've done it on the screen for you. So, if you have one half times one half then suddenly you get one fourth. So now there's only a 25% chance of guessing correctly. 'Cause it could be true-true, true-false or false-true, false-false. So in reality, this is still a multiple choice question with four possibilities. So if you have to use multiple choice -- um -- or if you have to use true/false, make it into this kind of form, where it's really multiple choice, by having two different statements as one test item. And then you're going to decrease your odds of guessing down to 25%, so that's better than 50.
Next, we're going to look at short answer questions. So, on the screen I've got three examples of short answer questions. What day comes after Tuesday? What is John's relation to Timmy? and why is it dangerous to vacation in Mexico? So, you should suppose that there's probably some text you would read or some listening text you would hear before you answered these. So, you can see that, uh, even without the text we can predict that the answers are pretty clear. What day comes after Tuesday? Wednsday. There's a unique answer. There is only one right answer. So, that is a good aspect of a well-written short answer question. There should be a unique answer. Two, what is John's relation to Timmy? Well, I don't know. But, maybe it's his brother or his friend or his uncle. But there is going to be only one right answer.
<25:21>
That's good. Three, why is it dangerous to vacation in Mexico? Well, probably something like, um, if- if you're a foreigner, you're likely to be kidnapped. And, maybe they're gonna ask your family for a lot of money. So, I don't know, it depends on the text. But, with a short answer, we want to make sure it can be answered in -- you know -- one to ten words. So, even though it's short answer, we want to give them a limit for purposes of reliability. And the best kind of short answer questions will have only one unique answer. So comparing short answer to multiple choice, short answer does have some advantages. We'll look at three. One, there's less impact of guessing. On a multiple choice, remember, we're only testing recognition knowledge -- what I can recognize. I don't have to produce something. But, with short answer, I have to produce something. I might be able to easily find the answer, but I'm still having to produce. There's less likelihood that I'm going to guess correctly on short answer. Two, for the test maker, so much easier! You don't have to write distractors. Writing distractors for multiple choice is really difficult. You can get good at it over time. Um, it's much easier for me now than it was eight years ago, for example. But, it still takes a lot of time. And I still sometimes make mistakes. So, of course, I have my colleagues moderate my exams for me to find errors or things that I don't predict. And every time I do that, I'm surprised by the feedback because it's always something I didn't expect. So, multiple choice are difficult to write because of the distractors. Short answer is an advantage over that. You don't have to write distractors. Three, cheating is more difficult on short answer questions. Ok? but, compared to multiple choice, short answer do have some disadvantages as well. So, lets look at those. We have to think what skill are we really testing. What kind of test do we use short answer for? Usually for reading or for listening. If we only wanna test reading, but we give a short answer question, students have to write an answer. So then we have a problem with construct validity. Because my goal is to test reading. But I'm also making students write. So, there's kind of, uh, an issue there. And maybe students are good at reading or good at listening but they're not good at writing. Then they might get the question wrong only because of their writing skills and not because of the skill that we are trying to test. Two, how many items can you have? Well, compared to a multiple choice test, you can't have as many. If you had 50 multiple choice questions, you might be only to- able to have maybe -- eh -- twenty or so short answer. Maybe even less depending on how long the answers are. Because it takes a lot less time to circle A than it does to write a five word answer. Ok? Three, how much time for scoring? Scoring, short answer takes much longer. Ok? so multiple choice takes much longer to write. But, short answer takes much more time to grade, ok? So, you have to decide which end do you want to spend more time on, the beginning or the end. Of course we know short answer is, um, a bit more authentic, so you should probably lean towards than instead of multiple choice. Fourth problem, how reliable are short answer questions? If there's only one right answer -- like, what day comes after Tuesday? Wednesday -- there is only one right answer. That is the goal. But what's the problem? We have problems with spelling. What if the student spells the word wrong? If we can still tell that they knew the right answer but just didn't spell it correctly, we should not take points off for spelling if it's not a spelling test. But, the problem is sometimes they're gonna misspell the word so much that we don't know if they really understood what they read or what they heard. So, there are problems with scoring in terms of reliability. Because we're only trying to test a receptive skill, usually, like reading or listening. But we're grading them on what they produce in writing. Or maybe through speaking. So, there's a reliability issue because we're depending on some other skill in the test that we're not trying to test for that skill. So, um, it can be very difficult. And we're gonna talk more about that when we talk about each of the four skills individually.
Alright, next, we're gonna have short answer. So we had multiple choice, and then short answer, and now we're gonna talk about how to write the short answer questions or write short answer items. We want to make them specific. They shouldn't be confusing. Our goal is not to confuse students. It should be clear what students are supposed to do. Also, give them the length of answer in words -- this is really important for reliability. So, you wanna tell them give your answer within ten words, or give your answer within five words. Or give a one word answer. Remember to compare students, we need to have similar- uh, similar results from students. Unique answer. We want to have a unique answer. Your goal should be to have only one right answer. If there's more than one right answer to a short answer question, it's gonna be really difficult when it comes time to scoring because you can't compare different students if they write different answers. Ok? so, multiple choice, short answer, and gap filling.
So, on your screen, I've got some examples of gap filling. So in a gap fill question, you have an entire sentence with some words blanked out. Students have to write in the answer. So it's a little different than short answer. One, Seoul is between blank and blank. Two, if you want to eat the best blank, you should go to Chunju <unintelligible>. Three, multiple choice questions are harder to answer than short answer because blank. Now there's a right answer to this one but each of you might have different words to answer number three. I would say multiple choice items are harder to write than short answer because, um, it's hard to write believable distractors. But you might say its hard to write distractors of similar length. But your classmate might say its hard to write them- it's hard to write distractors that are grammatical. So there are many different possible answers to this question.
<32:46>
So number three is an example of gap fill where there is not a unique answer. So that could cause some issues in grading or issues in scoring reliability.
One way to help solve the problem of multiple possible answers is to give students a hint. You've noticed I do this a lot in this class. Sometimes on the quizzes if there's a blank, I'll give the first word, the first letter, like R: it's a hint for reliability or V: hint for validity. So if there's more than one possible answer to a question, to a gap fill question, and you want to limit the responses, make sure that students give you the word that you're thinking of. You can give the first letter, and that's a hint for students.
Gap fill are more objective than short answer, but it still requires some judgment because there could be problems with spelling or bad handwriting. But they are a little bit more reliable than short-answer questions. There's a disadvantage to gap fill and that's that, um, if you have more than one gap, like if you have a grammatical question like Tom didn't go the store on Wednesday, he blank blank'ed to the store the day before. Now students have a hint that it's he had gone. If there's only blank, they might say went. So if you're testing something where they could be a one word or two word answer, um, having two blanks is gonna give too much of a hint in some instances.
Ok, so, let's look at another example of a gap-filled question. If you wanna blank, you should go to Chunju<intelligible>. So what's the answer to this question? Earlier I said eat, so you might be thinking, eat. That's a unique answer. But there is more than one right answer. I can see if you wanna try bebop <unintelligible> . If you wanna happy bebop<untelliglbe>. If you wanna enjoy bebop<unintelligble>. So there is actually not one unique answer to this question. So it's not a really good question to ask.
In your workbook on pages 44 to 45, you can find Billy's diary dated Wednesday, April 24th. I really hope that you have already completed these tasks or already looked at these test ideas. So you're gonna have to read the paragraph and then we're going to look at the questions about them. So if you haven't read Billy's diary, please pause for a moment, get out your workbook, pages 44 to 45, read the text, and then we'll talk about the questions together.
So first we're going to look at number one. This is an example of multiple choice. The word stubborn could be best replaced by
A)Kind
B)Obstinate
C)Wise
D)Gentle
We know the answer is B, it has the same meaning as stubborn. But the problem is the distractors are more difficult than the text. But they shouldn't be, they should be easier. Um, this test item is supposed to make students get the right meaning from the context of the story. But stubborn is not a difficult word. Obstinate is a difficult word. So for testing comprehension, we need to put obstinate in the text, not in the answer choices. So let's look at the improved version. The word obstinate could best be replaced by
A)Kind
B)Stubborn
C)Wise
D)Gentle
This one actually tests comprehension of the text. Because most students don't know what obstinate means. But by reading the text, they can figure it out. So this is a much better option. Put obstinate in the text and make that the question you're asking about.
Let's look at the next one, number two. I wanted to have my one ear pierced _____ dad didn't agree to my piercing. The answer to this one is D. Good points about this is that you do need to um, look at the answer choices and you can see that they're all grammatically possible. So that's one good point. However, A and C have the same meaning: since and because. If I know they have the same meaning, I know A and C cannot be the answer. So now my answer choices are narrowed to B and D. So now I have a 50 percent chance of guessing correctly. So make sure your distractors do not have the same meanings.
Number three, what does Billy want?
A)To have long hair
B)To have a crazy idea
C)To keep up with new trends
D)To be as fashionable as dad
The answer is C and this is a very good example of rephrasing. So in the text it says, he wants to be in fashion. But in the question it's rephrased as new trends. So this is really important for testing comprehension. We want to rephrase, don't use the exact words from the text, but use the same meaning.
In number four the writer wants dad to duh-duh-duh-duh-duh his idea to want your piercing. Ok. THe hint is given that it's ten letters long. So this is something we can for reliability. We can give the first letter, or we can say how many letters are in the word. However if you go back to the article, understand is the only word in the whole story that has ten letters. So now we're testing a test-wise technique. We're not actually testing comprehension. All they have to do is count ten and then find one word that has ten letters. It's the only word with ten letters. So now it's not a good question.
And five, why is ear piercing so important to Billy? He wants to be in fashion like all the other boys. It could have another answer, though. Maybe all the other guys? But maybe it's limited to two possible answers, so it's not bad. It's a little bit rephrased from the text so, eh, this one's okay.
Number six, this is the last one. What fashion did uh, dad follow when he was a teenager? He had long hair. So this is a rewording. The wording of teenager in the question. Whereas in the text, it said, when he was my age. So you know, that's a little bit better than just saying, um, when dad was my age. It makes it harder to just find the answer. So this is helping more to test actual comprehension.
Okay, for next time you need to review your statement of the problem and writing specs, writing items, moderating, trailing, remember in week 8 we're going to have an intensive session where you're going to put all that into practice. Uh, for next week, you're also going to read chapter 11. It's about reading and don't forget to do the workbook. Several good exercises in there. They're going to help you understand. And I will see you on the discussion board.
<stopped at 19:53>