Inclusive e-assessment

Video
Transcript
References and further reading

Video

(This recording was created with Explain Everything)

Transcript

This talk is about inclusive e-assessment. It's not something I'm an expert in but it's something I care about. I'm going to draw from my own reading on the subject and my experiences designing e-assessment to give you some ideas for ways you can improve your assessments.

First of all these slides are done in a nice colourful handwritten style which I enjoy, but there are captions on the video version and a transcribed version on the conference website.

This talk could have been called "who are you not designing this for?".

The problem is that it's easy to imagine a student just like you while you're designing any assessment and anybody else who is different because they have particular requirements or just come from a different background, you might have to make some effort to adapt to them.

That's called the deficit model.

Another way of doing things is to think about all of your students before you implement your e-assessment and design for everybody.

The best outcome of inclusive design is that when an issue arises and you need to make an accommodation for a student you can say 'well, this student is already catered for'.

Now, I'm a privately educated straight white British male so many of the things I'm going to talk about I've heard about secondhand, to be honest, so the things I'm going to talk about are from my perspective.

You might do things differently but the idea that you should be thinking about other people than yourself hopefully is transferable.

Three boxes headed 'Subjectivity', 'Accessibility' and 'Integrity'

I've come up with three kinds of barriers to inclusion.

First, subjectivity: these are largely cultural aspects that might make students feel alienated or mean that there's something that they don't know that you assume they know that would stop them engaging with the assessment.

Then, accessibility: this is largely barriers to individuals: reasons a particular person can't access your e-assessment.

And finally, integrity: these are the sort of things that you do to make your job easier but which might make students' lives harder or make them perform worse than they should relative to other students.

Subjectivity - who is the subject of your assessment?

Subjectivity: this is a term I first encountered in Sara Hottinger's book Inventing the mathematician and the idea is it's about the question, "does the student believe that they are the subject of your assessment?" Sara Hottinger's book gives many examples of questions which either support stereotypes or have a very restricted perspective.

For example, questions where a female character is always a helper or a male character is allowed to make mistakes, and Eurocentrism in what kind of maths is covered.

Consider: gender (names, pronouns); ethnicity; culture (references, common knowledge); class (amounts of money, activities); geography; family

Things to consider: first of all gender and ethnicity are really about who you portray in your question prompts. Try to avoid it always being men, try to avoid it always being people from the same part of the world as you.

Something that's not as obvious is names of people. It's quite hard to think of names that aren't from the same sort of background as you and even when you don't have named characters make sure you use a sensible mix of pronouns - just picking at random is a good way to do it.

Culture can be a barrier: if you use references to TV shows or films that students haven't seen then they might miss something. You might think it's fun to set a question about Game of Thrones but if the students don't get the references that might make it a little bit harder for them, or there might be a really good reason why that student doesn't watch Game of Thrones.

Be aware of relying on common knowledge: for example, the fact that there are 11 players on a football team. Don't assume the students know this.

In the UK we have a national speed limit sign which has no numbers on it and whose meaning depends on exactly where you are so don't assume the students will know what that means.

Now these cultural aspects and these knowledge aspects quite often go along with class, so for example a question all about a skiing holiday that begins "Sarah spends X thousand pounds on her skiing holiday..." - I have no idea how much one would spend on a skiing holiday - and that can make students feel alienated, like they're not the same kind of people that you're expecting to be in this class.

Geography: there's a couple of things here. Think about where things typically happen.

Here in Newcastle I've often caught lecturers setting big mistakes happening in Sunderland, which is nearby and which we have a big rivalry with. We have students from Sunderland so I'm sure they don't feel great about reading questions about that.

Make sure that there aren't certain countries that you only ever refer to as holiday destinations or as places that people who need help come from.

And finally family not everybody comes from a classic atomic family of a mum and a dad and kids so try and vary the inter-personal relationships that you portray in your questions and where possible use statistics to make representative choices of all of these aspects.

Accessibility: think about what you're assessing.

Are you assessing whether the student knows how to plot a graph or are you assessing if they can use your plotting tool?

Often when accessibility comes up it's to do with physical disabilities, to do with the way the student interacts with your e-assessment system but you also need to consider non-physical disabilities that might affect how a student processes the information you're giving them or their ability to engage with the content.

Vision: screenreaders; colours and contrast. Text: simple language; clear instructions; captions and audio description. Motor skills: reduce clicks; offer multiple input methods. Be very careful with new interaction types.

So the first thing pretty much everybody thinks of for accessibility is 'can you use a screen reader?' Every system should be usable with a screen reader and that also often means usable with just a keyboard, not a mouse or a touchscreen. So that's well covered.

It's easy to forget about things like colours: never just use colour to convey information. This is something I'm always on top of because I'm very colourblind and contrast is important: people with visual impairments where they need a very strong contrast between colours.

Firefox has a very good accessibility inspector you can use to check that you're coping with that.

In the text of your questions use as simple language as possible when you tell the student what to do or when you give them information: it makes it less likely that they'll misinterpret.

In your instructions be as clear as you possibly can about what you want the student to do without giving away a method that you might want them to recall on their own.

Everybody's bad at reading instructions. And don't rely on conventions because for example don't use the word 'simplify' for an expression because that's very ambiguous.

When you have images make sure you give alt text for them so it's read out by screen readers or assistive technology, when you have videos make sure that there are captions, and when you have audio make sure it's transcribed as well.

Think about what you're requiring in terms of motor skills from the student.

For some peoplejust using a mouse is a massive effort so just clicking on things repeatedly is going to take much longer than other students. And try to offer multiple input methods for any interaction, for example for a drag-and-drop activity allow a mouse of course but make sure a student can also use the keyboard to rearrange the items.

In general be very careful when you have a new interaction type: built in input methods for most systems can be relied upon because the developers have done work on accessibility but things like interactive graphics, embedded tools from outside the system, you need to think about 'can every student use this interaction?'

Integrity - one cheater is less of a problem than 100 students giving up.

Strict rules cause problems - let students change their minds. Instructions should help, not constrain - accept every reasonable answer. Limits add pressue - time limit, submission counts.

Strict rules cause problems. For example if you limit when an assessment is available, are all of your students sure to be free at that time? What if their internet breaks and it's a very restricted time period. If you give longer for students to access something then it's more likely they'd be able to complete it.

You might need to tell the student to enter their answer in a certain way because of a limitation of the system itself. For example you might need them to use a particular letter for their constant of integration - that's your problem not theirs. Certainly tell them what they need to do but if the student enters an answer that would be reasonable make every effort to accept it. Try to minimize the times that you have to give students this sort of instruction.

Limits add pressure. If a student only has a certain amount of time in which to complete an assessment, some people respond better to that than others. If there's a limit on how many times a student can submit an answer that can cause problems, for example if a student just submits an answer by mistake.

Finally, if you want to limit how the student navigates through an exam, for example they can't go back to a question they've already answered, that promotes bad exam technique. Really you should always allow a student to go back and change their answer.

So that's just about it.

Again, I'm not an expert on these sorts of things.

References and further reading

Jisc page on Inclusive assessment.
Creating inclusive assessments, part of a FutureLearn course on inclusive learning and teaching environments
Oxford Brookes University page on inclusive assessment.
Inclusive Mathematics Education: State-of-the-Art Research from Brazil and Germany.
Inventing the Mathematician by Sara N. Hottinger.
The Inclusive STEM & CS summit.
News feature: In South Africa, 'decolonizing' mathematics.
The 'random person' extension for Numbas provides functions to randomly generate names and pronouns for named characters in word problems. Demo.
Jisc page on making assessments accessible.
Numbas accessibility statement.
Learn Desmos: Accessibility. Desmos is world-leading in its accessibility features.
Methods to produce flexible and accessible learning resources in mathematics (some of the information is out of date).
Newcastle's Coursebuilder project can produce accessible versions of LaTeX notes.
Sigma network accessibility special interest group.

Contents

Video

Transcript

References and further reading