Hello and welcome to the seventh week of testing. Last week we learned about the steps for making a good test and we learned about different types of questions that you can make. We also learned some tips for how to make those questions good. And today we are going to move on and finally start the practical side of things with reading.
But before we could start it on reading I wanna make sure you have a solid foundation about the right steps for making a test so we're gonna have a short quiz to review what we learned from last week. There are going to be nine (9) questions so I'm going to give you 6 minutes to complete this and then we're gonna go over the answers together. Please do take a moment to get out your pen and your paper, actually write down your questions so that this can be a good form of formative assessment for you.
Ok, let's go over these questions together.
Number 1-Which comes first, moderating or trialing
I really hope you got this right, it's moderating.
So after you make your test questions, you're going to show them to some colleges. Usually at least 2 colleges to see if there are any problems, if there are grammatical issues, if there might be more than one right answer, if the answer they gave was not the one that you expected. So that is the process of moderation.
Trialing is when we try our test out. So trial, try it out. Try it out on real students. Umm..we try to find native speakers of a similar population and if we can't find that we're going to find umm...well we want to find native speakers in..;
Korean students if we're giving tests to Korean students, we'd also want to find Korean students similar to our own students, but if that's not possible then we'd also like to trial the tests on some teachers, but they'd need to be different teachers than the ones who moderated it. So if people who moderate it need to being seeing the test for the first time and then you would trial it on different people who have also seen the test for a first time, after you have modified your exam from the feedback you get in the process of moderation.
Number 2- Who should moderate your test?
As I said, at least two colleges. So get at least two pairs of fresh eyes to look at your test and look for errors.
<2:33>
Three. Who should trial your tests? Again you want to find native speakers who are of a similar population to the students you are going to be testing and it's also recommended to get students similar to your population so probably Korean students of a similar age. But if that's not possible at least get some other teachers to trial your test.
Four. What are some examples of operations? There are so many listed in the book. Last week I gave you umm about four different sets of pages in your book where you can find lists of operations. But some are giving an opinion, asking for directions, ah expressing thanks, expressing a complaint, offering advice, ordering a pizza on the phone or greeting people so these are just some of the operations that we are going to be talking about.
Five. For which type of test do you need more specific criterial levels of performance? Of course this is going to be for our productive skills. Ah, whenever we're having subjective tests or tests that are direct. So we wanna have direct tests we wanna have tests that are going to be measured subjectively because direct tests have to be measured subjectively, so direct testing of speaking or direct testing of writing. These tests require more subjective judgment. And so we need better criterial levels of performance or more detailed criterial levels of performance. This is because when we're rating them, we want to be as objective as possible. So even though it's a subjective test, we want to try to be objective when we're scoring. Try to reduce the bias reduce any random error. Umm and increase reliability. So having very specific methods for grading is going to increase your test reliability and scoring reliability.
<4:35>
Number six: which type of question is easier to write? Multiple choice, or short answer? For writing, for the test maker or teacher, short answer is much easier because we don't have to write distractors. Number seven: which type of question is easier to grade? Multiple choice or short answer?
Multiple choice is much easier to grade because hopefully there's only one right answer. On a short answer question, [umm] we have to, well it's going to take more time. But also we have reliability issues. What if a student misspells the word, or uses incorrect grammar, or writes something and we think they know what they were trying to say or we think they had comprehension, but we can't completely tell. These are issues that come up whenever we have to grade short answer questions, where there's some judgement required. It's a bit subjective.
K, number eight: which type of question is probably more reliable? Multiple choice, short answer or a gap fill? Well, reliability's related to consistency and multiple choice only has one answer. That's pretty reliable. So, multiple choice is going to be more reliable. Remember I told you before that we want to have both validity and reliability. But if a test gets more valid, it usually gets less reliable.
So, multiple choice tests are more reliable becuse there's only one answer. But, multiple choice tests are not direct, they're not authentic, so validity goes down. Number eight: which type of question is probably more [uh] I told you reliable is multiple choice. And, nine: which should be more complex? The language of the text, or the language in the test item, the actual question.
This is a very important part to remember, and it's especially important for when we're going into receptive skills. Right now, we're going to be starting reading, and next week we'll be doing listening, so it's really important you understand this. The text should be more complex. So, if you have a reading test, and students have to read a paragraph, and then answer questions, the language in the questions should be simpler than the language in the passage that they are reading.
Because our goal is to test comprehension of the text. Remember that if we have confusing instructions, then that's going to decrease the reliability of our test, because we won't know what we're actually testing. So of course, that means it can also decrease the validity of our test.
If we're trying to test for comprehension of the text, so if we have students read something and then we ask them questions, we wanna find out if they understood what they read. But if we have confusing questions with difficult vocabulary, then we're actually testing their vocabulary skills, not their reading comprehension skills. So if we're not testing the skill or ability that we want to test, that's a problem of construct validity.
Let's go ahead and start talking about reading. So, first of all, simple question: what is reading?
So it's a simple question but it's actually a complex answer. Reading is a complex set of skills. So if I ask you to give me a definition of reading, you would probably have a hard time putting it into words. It's not something that we usually define with words. But, it involves many different things, on a higher level... [umm] well, you're at a higher level, so whenever you read, you're thinking about meaning, and you're maybe interpreting things, but think about when you first learned to read.
When you were a child, and maybe your parents were reading to you, or you were a young student at school, and your teachers were reading to you, trying to show you how to read, what was the first thing they did? I don't know why I remember this, but I do remember that when I was in kindergarten, the first word I learned how to read was "doll". I remember seeing "d" "o" "l" "l" on the page, and I remember my teacher ya know showing me "duh" "duh" "doll", and for some reason that just really stuck out in my mind.
So, when we first learn to read, we're just assigning meaning to symbols. I knew what a doll was, I always played with my doll, so I was very familiar with that word. And then I had to be able to hear the difference in the sounds. "Duh" "all". So, of course when we're learning a language, reading is not the first skill. First we listen, and then we can comprehend what we're hearing. And then usually speaking, and then usually reading, and then writing.
So, I had to learn how to hear the word first, and speak it before I could read it. And I had to understand the idea about different letters having meanings. So, at the basic level, reading is giving meaning to symbols. So first you have to recognize individual letters, and then later we're able to read entire words, and give meaning to entire words. And then of course we can move on up, to entire sentences and then beyond the literal meaning we can assign [umm] maybe metaphorical meanings to texts, that we either hear or that we read.
So the problem with reading, is that we can't always see the measurable behavior. It's unlike [uh] writing or speaking. Because with writing, if a student writes an essay and they give it to you, you have evidence. You can see their work. And with speaking, if I give a direct speaking test, I can actually hear what they're saying. I can even record the conversation, so that I have proof of it, later I can play it again. But with reading, [uh] it's not so easy, it's hard to know exactly what we're testing when we give a reading test, and today we're going to discuss why.
<10:46>
So, let's talk about receptive skills. They are reading and listening, and what is the challenge? They are invisible. So if a student is reading a book, how do I know he's reading the book? When I was a student I remember in seventh/7th grade one of our days each week usually on Fridays was a reading day. So for forty-five/45 minutes we were supposed to just sit there in our seats and read a book Unfortunately I did not like reading but I really loved daydreaming. Every Friday I would bring a book, and I would just stare at the pages. And every now and then, I would turn the page but I was never actually reading. I couldn't, I couldn't concentrate in a classroom of other students and read at the same time. And I just had other things on my mind, I was in middle school. Yeah, I just wasn't interested in reading during class. So, the teacher never knew though, because if you looked at me, it looked like I was reading but we can't tell by looking at someone if they're actually reading. So just observing students reading is not evidence. So how do we know if they are actually reading? There are a few different ways but one way is testing them. But, If you test them you're going to be testing some other skill. How do I prove that I read a book? Two/2 ways,I can either talk to you about the book or you can ask me written questions and I write the answer. So to prove my reading skill, I have to rely on my productive skill. So this is the issue with testing receptive skills. Receptive skills are reading and listening,but whenever we test those we have to, a little bit, test productive skills as well. So the construct validity is always a bit of an issue in receptive skills tests.
What is the construct of reading? We've been talking about construct validity, so what is the construct of reading? This is a little bit more challenging question. So, it depends on what we're reading,and the way we're reading and the reason that we're reading. So the reason we reading will change the construct.
13:12
The type of text what we read that will change the abilities. For example, the ability to read an academic journal is very much different than the ability to read a very simple novel just for fun. So the text or what we are reading can change the context.
So let's look at reasons for reading. Why do we read in every day life? We have several different reasons. Probably your first reason before you were a student in this [?] programme, your first reason probably was to enjoy. For me, that is my second biggest reason, is to enjoy. I first of all have to do an awful lot of reading just for work, but when ever I have the time I read just for fun, just for pleasure, to enjoy.
We also read to understand. So right now that's probably what you are doing the most of, is reading to understand something.
We can also read to get specific or detailed information. Sometimes, especially in school, we have to read to prove that we can read. So as I said before, if we want to show someone evidence of our reading skills we have to rely on productive skills and the only way we can do that is actually reading aloud. Sometimes we have children read books out loud so that we know that they are able to read.
And finally, sometimes we prove that we know English so we need to prove we know English so we read to prove that we know English. Of course this usually happens in school, but it can also happen at home. Maybe if you have mother's that teach you at home or really encourage you and want to make sure that you are learning what you are supposed to learn at school, then your mother might have forced you to read English books at home.
So let's look at the different ways of reading.
There are various skills of reading so some of them are guessing, inferring and also grammar or vocabulary. Of course, we can't read successfully if we don't understand grammar or if we don't understand the words that we are reading. So we might be able to, you know, literally look at the words and hear something in our head or say the words out loud but we won't actually have comprehension of the text, if we can't understand the words or understand how the grammar changes the meaning.
So if we are reading for understanding, that's usually slow and careful reading. So reading for understanding could include skimming or scanning. If you are skimming, you usually read the whole text quickly, looking for an overall meaning or the gist. If we are scanning, we are looking for something very specific. So often times when you are on the internet and you have a question and you just want to find a really fast answer, that's usually scanning. So if you have a problem, you type something into the search bar and then you find some results, you go to the page and then you just quickly scroll through. Ahh and then you find the one thing that you are looking for, that is scanning.
16:25
<16:27>
Alright, let's move on to the way that we read, or how we read. So we have two different questions here. One, does the reader need to read quickly or does the reader need to read precisely?
So for the first question, what do you think this is called? If you need to read quickly, we need to be very efficient. We call this expeditious reading. So expeditious or expeditiously.
Second, does the reader need to read precisely? If you are reading precisely, you are looking for details. This is careful reading.
So which one do you think is taught in Korean schools? If you are like most of my students you would probably say "both". That may be true, and I certainly hope that's true, but in reality I think the bigger focus is on careful reading.
If I look in Korean tests and I go through the reading section, I think it's incredibly difficult because they ask so many detailed questions, so I think the focus, in Korea, is usually on careful reading.
<17:15>
But we want to encourage our students to also read expeditiously and part of reading expeditiously is reading for enjoyment so we want to foster a love of reading for enjoyment because reading actually helps increase the other skills as well.
Let’s take a look at this cartoon. One woman is saying to the other, “I doubt that Hector has read a book in months but he’s texting at an eighth grade level.” So, this teacher is actually probably complaining to another teacher; um, however, texting or reading text is actually a form of reading. There are actually different kinds of reading and that is one form. I’m sure if you’re like most students you probably read about fifty text messages per day.
Okay, let’s look at another cartoon, comic here. It says, “I like subtitles. I can watch the movie, skip the book and not feel guilty.” So, many people think that if you watch subtitled movies instead of reading a book it’s cheating but reading subtitles is a form of reading. If you try to watch a foreign movie and you can’t read quickly enough then you can’t actually follow the movie. So, there is a certain skill required in reading subtitles.
Okay, so now we’re going to look at the different constructs that we talked about earlier. So, first of all, what we read, there are different themes we can read, three different reasons: for academic reading, job related reading and personal reading. So, what are some different academic readings? What are some different texts that fall onto the category of academic reading? Well, of course right now you are in a very academic situation, you have to read “testing for language teachers” that’s an example of academic reading. So, any sort of textbook or if you read journal articles that would be academic reading. Sometimes you have to read things on the internet that are related to academics or essays or editorials or technical reports; these are academic things. Secondly we have job related reading. I’m sure most of you probably have jobs, that’s probably why you’re two-four-one students; so, if you have a job you have lots of things you read at work. You probably read reports and memos and e-mails, maybe faxes, most people don’t fax anymore, but maybe. You could also read financial documents or spreadsheets or powerpoints; these are all things we read for business purposes or for jobs.
<20:20>
And finally personal reading - what do you read for personal purposes? I mentioned earlier text messages. Also we read emails; um, they're different email than the ones for the job, but it's another form of text - emails. And I read novels; whenever I get a chance I read novels, and I love reading magazines. Um, of course where you are that might determine reading. So we also have like newspapers or, uh, recipes. It depends on what your hobby is that's going to affect your personal reading.
Some of these genres overlap. So before I said emails could be included in job-related or personal - um, they could also be in academic. I'm on some mailing lists for academic purposes and maybe once or twice a week I get an academic email. It's telling me about research done in my area of interest: educational technology. So some text can overlap through different genres. So what we call personal, business, or academic will vary according to the following list:
- type
- form
- graphic features
- topics
- style or register
- intended readership (so, who we want to read this)
- length
- readability
- and range of vocabulary and grammar
So text types could be books or handouts are articles. Text form - this is related to the purpose. So "Is this for entertainment? Is it for instruction? Is it for description?" And graphic features - um, this relates to things that we see. So charts or pictures or illustrations; just what it looks like. You know what topics are. Style or register, this is related to the level of formality. "Is it really formal? Is it really informal?" And we've got variations in between.
<22:23>
Intended readership - so who we want to read this, length, readability, and range of vocabulary and grammar. So text types could be books or handouts or articles... Text form - this is related to the purpose. So, is this for entertainment? Is it for instruction? Is it for description? And, graphic features. This relates to things that we see. So, charts, or pictures, or illustrations - just, what it looks like. You know what topics are. Style or register - this is related to the level of formality. Is it really formal? Is it really... informal? And then we've got, you know, variations in between. Intended readership - maybe it's university students. Or maybe the intended readership is young children. Length is measured in number of words, usually, or in, uh... could be pages. And, readability - how difficult is the text? And, finally, range of vocabulary. Some reading material is very technical and requires very specific types of vocabulary.
Okay, let's look at assessing reading for the overall gist.
<22:43>
This is assessing reading for understanding. We are looking for the main idea. So on a test the text where we are looking for the main idea can vary usually from two hundred to two thousand words.
So if we are looking for the main idea, we can ask a variety of different kinds of questions. These are all mentioned in your book — summarizing, reordering, comparing, contrasting, short answer. Um we can have the students identify the main idea in many different ways. Next we usually assess to, um, check if students can usually understand the details. So assessing reading for details.
<23:28>
The text used on assessing details could be the same text as the one where we're checking for the main idea, or it could be a new text. So what techniques can we use? When I say what techniques, I mean what question type? What is the method? So short answer or gap fill, or summary close, or multiple choice, matching, information transfer, and we're gonna go over these question types a little bit later. Sometimes we assess reading to show that you can read. So take a minute to look at the screen, see if you can figure out the answers here.
OK, this tests whether your readers can identify and differentiate key features of words. So you have two different words and you want to see if your students can see that there is a difference. That is one way to show that they can, on a very basic level, that they can read.
Second, this type of assessment is not looking at meaning. It's only looking at whether or not they can read very basically, but not understand.
So how do teachers do this? We have a couple of different ways, I said before we can have them read aloud. If they're looking at a book following along and saying the words out loud, we can be pretty sure that they are actually reading. We can also have them identify different words. So on the screen we have "same," "same," and "some." So we're looking to see if the students can tell the difference in "a" and "o." If it looks different to them, that's a good sign of reading.
And finally we can give a picture with a word, and take one of the letters out. If they can produce the missing letter, that shows that they are able to read.
<25:27>
