Access the guide by clicking the image above
Over the last six months I have been looking in to the Undergraduate Viva. Asking questions such as what are the benefits? What makes a good undergraduate viva? and, How can students be prepared for their undergraduate viva? One of the results of this is a guidance document on how to conduct a viva of this type. It may be of interest to others.
I am aware others are grappling with how to define graduate attributes, so I thought it helpful to share the approach that we took. As part of a whole university curriculum review, and a strategy review, we set about trying to identify what it was that the curriculum should achieve. Essentially we asked, what was our goal? Unless we know this any curriculum initiatives would be tinkering. So we asked a very fundamental question, what should a Harper graduate be? This goes beyond simply asking what they should be able to do, and incorporates a sense of self that is needed to deal with a fast changing external environment and this is needed to be resilient for the future. This idea is underpinned by Ron Barnett’s work on working in super complexity. It’s a huge question but one that we answered, I think, in a creative way.
Resources from the ‘build a graduate’ workshop
We gathered as many staff as were able to attend to join a room with huge pieces of card printed with a giant graduate. In course teams staff were then asked to build a graduate in their discipline. Using the card as a focus for thinking, prioritising, debate and discussion each team built their own graduate. Of course this informed course level thinking before more detailed discussions got underway about course content. Using post it notes to stick on to the graduate allowed rearrangement, re-prioritisation and change as the group discussions evolved. The views in the room were not formed in isolation since colleagues were involved in both student and industry engagement.
After each team had spent several hours identifying what they graduate would look like in a perfect world, we collated all of the words used by all of the teams. These were then collated and put in to a word cloud creator. The commonality in the lists showed itself as the larger words were repeated across different course areas. After some sorting and filtering it became clear that we did have a collective and common vision of what the graduates of the future should be. This exercise became the foundation of the new graduate attributes. The build a graduate exercise was also undertaken by course teams with students and industry contacts. The word cloud produced is shown below.
The word cloud gave students and staff a visual connection to the exercise that we had taken, and a constant reminder that the definition of ‘our’ graduateness was a collective exercise.
A first workshop output on defining graduateness
The final version of the graduate attributes
The headline attributes helped to ground the Learning and Teaching Strategy; they provided clear direction as to what our activity should be pointing to. It provided one of the key cascading ideas for strategy and operational policy.
For the curriculum aspects, once we have the broad terms for what a graduate should be, we interpreted each attribute, skill area of understanding for each level of study. This involves some word-smithery and some external scoping to see how others level their outcomes, but it also required an eye on the future. We ended up with was a breakdown of each of the graduate attributes, and a description of what should be achieved each level in this area. A snapshot of the attributes are offered below.
It’s one thing articulating the graduate attributes and specifying them for each level, it is quite another to deploy them as the beating heart of the real curriculum. The first thing that we did was ask course teams to develop programmes that addressed each area at the correct level. Course level engagement forced deeper conversations about ‘what does digital literacy mean in our context?’ ‘where are the opportunities for global perspectives?’ and this sparked the attributes into life. Each programme then mapped where the attributes were met, but this one way mapping was deemed insufficient, as once it is complete it can, in reality, be committed to a top drawer and dismissed as a paper exercise. So we went a step further and requested that modules were individually mapped against the graduate outcomes. This makes it much clearer to students and staff, what skills the module should address. Through validation and scrutiny each module was checked to ensure it really was enabling the development of these attributes, through its content, pedagogy, assessment or independent activities. The next step is to get student to actually consider their progress against the graduate outcomes in a meaningful, rather than tick-boxy way. I’m sure others have taken different approaches to developing graduate attributes, but this sought to be pragmatic and inclusive.
One of the most frequent conversations I have is around improving feedback, and how technology can help. Increasingly I am trying to encourage a more nuanced discussion about feedback, because deciding on what feedback to give and how to give it is not simply about a choice between one tool or another; the choice should be the result of the individual lecturer’s preference, the context, the type of assessment or activity upon which feedback is being offered, the characteristics of the student or cohort, the aims of the feedback, the tools and technology required, the quality management requirements and no doubt many other factors. Some of the common questions I get are shared below with comments:
Should I use GradeMark for feedback?
Well, it depends a good deal on what you want to achieve. GradeMark has many benefits but in itself it will not make feedback better. To be clear, like any technology it can make a difference but it is not a silver bullet; without meaningful engagement and a commitment to change practice, it will not improve satisfaction with feedback.
GradeMark can help you to achieve consistency in the comments that you offer to students because you can create a bank of common points to annotate the work, and it can enable you to add a greater amount of feed forward signposting advice to students for their common errors, for example if a group are struggling to paraphrase, you could create a comment advising of techniques and pointing to resources that might help and use this many times. GradeMark can help with a sense of fairness too, as marks can be allocated using a rubric. This is entirely optional, and there are of course other ways to employ a rubric. It can help with legibility, as comments are typed; but so too can very clear handwriting and other technologies. It can allow you to save time at certain points in the marking and feedback process, as you can get started on your marking as soon as students hand in rather than delaying until you receive a complete set of papers. It can aid transparency when team marking; you can see how another tutor is marking and feeding back – again this is possible to achieve in other ways, but being able to see each other’s marking in real time can create ongoing dialogue about the way marks are allocated and the way comments are added. If you are really concerned about reading on a screen, this might be a problem; but if you consume news, media, research and other things via a screen, it may be worth laying aside your concerns and giving this a try. All of these benefits though can only be realised if the user is working with the technology and is not simply transferring existing practices in to a digital environment.
Will it save me time?
Yes and no. It’s not that simple. It depends how you use the facilities and what type of feedback you give. You can use as many or as few of the tools within GradeMark as you see fit. You can use the facilities within GradeMark in any combination: Voice over comments, annotations (stock comments or personalised as if marking on paper), you can use a rubric, auto generated scoring from the rubric (or not) and you can use a final summary comment. Each individual needs to look at their set up and then consider what they want to achieve, they should then select the aspects of the tool that work for their situation. Annotations may be better for corrective, structural feedback, or feedback on specific aspects of calculations, but the narrative may be the place to provide feedback on key ideas within the work. If you go in to using GradeMark solely to achieve efficiencies, you will most likely be disappointed upon first usage because there is a set up investment and it takes a large group or multiple iterations to get payback on that initial time spent. In my experience those who use GradeMark may start out seeking efficiency, but end up with a focus on enhancing their feedback within the time constraints available to them. When time is saved by a user, I have seen colleagues simply re-spend this time on making enhancements, particularly to personalise the feedback further.
Ok, so what equipment do I need to be able to use GradeMark? Is it best to use a tablet?
Again it much depends on your work flows and preferences. A desktop computer is my preference as I like lots of screen room and I like to settle in to one spot with unlimited supplies of tea, whenever I mark. Others like to be mobile and the tablet version of GradeMark allows you to effectively download all scripts, mark and feedback and then upload. So unlike the desktop version you don’t need to be connected to the Internet – for those marking on the go, this is a good thing.
I see other people using other technologies for marking, like Dragon Dictate and annotation apps on tablets, are these better than GradeMark?
There is a large toolkit available for assessment and feedback and each has strengths and weaknesses, and each fits differently with personal preferences and context. So Dragon dictate can be used to speak a narrative or extensive comments, it’s not perfect but may help those who struggle with typing; annotation apps allow the conversion of handwriting to text, and they allow comments to be added at the point of need within a script (though GradeMark allows this too). On the downside a manual intervention is needed to to return the feedback to students. Track change can be good for corrective feedback, but it can cause students to look at their work and feel that it wasn’t good enough as it has the electronic equivalent of red pen all over it!
Second markers or external examiners refuse to use the same interface… Then what …?
I’d suggest that you encourage others in the process to use the technology that you have selected. Give them early warning and offer to support the process. A pre-emptive way of dealing with this is to ensure a course wide approach to feedback, so agreeing, as a group, the tools that you will use. This should then be discussed with the external and others at appointment. It’s harder to resist a coordinated approach. Policy change is what is really needed for this, so lobbying might help!!!
But students like handwritten feedback, they find computer based feedback impersonal …
Maybe so, but all students prefer legible feedback and feedback that they can collect without coming back on to campus. Also is it not part of our responsibility as educators to ensure students can work digitally, even with their feedback? Students who tell us that they like handwritten feedback often feel a personal connection between them and the marker, but feedback using technology can be highly personalised. It is simply up to the assessor to use the tools available to achieve levels of personalisation; the tools themselves offer choices to the feedback craftsman. Adding a narrative comment, an audio comment or customising stock comments can all give a personal touch. However if the feedback giver chooses none of these things, then of course the feedback will be depersonalised.
Students say they don’t like electronic feedback…
Some might and the reasons are complex. If we introduce a new feedback method at the end of a students programme, without explanation, irritation is inevitable as we have just added a complication at a critical point. Equally if feedback across a students journey is predominantly paper based, it is no wonder they struggle to remember how to retrieve their digital feedback and so get frustrated. If the feedback is too late to be useful, that will also cause students to prefer old methods. It may be useful to coordinate feedback approaches with others on your course area so the student gets a consistent approach, rather than encountering the occasional exotic technology with no clear rationale. Finally, though, students also need to be trained to do more than receive their feedback. They might file it, return to it, précis it and identify salient points. Good delivery of feedback will never alone be enough. Timeliness and engagement are also key to allowing students to work gain the benefits of their feedback.
Seeing things differently ….
One of the benefits of using technology in feedback is not often spoken about, or written of. When we engage meaningfully with technology in feedback it can change our approach to providing feedback, irrespective of the technology. By (real) example, someone trying annotation software may have a realisation that legibility is a real issue for them and they must prioritise this in future; someone using a rubric may start giving priority to assessment criteria as the need for equity and consistency becomes more sharply placed in focus; someone using stock comments and adding a voice over becomes aware of the need for the personal touch in feedback; and finally, someone using audio becomes aware that the volume of feedback produced may be overwhelming for students to digest and so revise their approach. These realisations live on beyond any particular technology use; so when we think of using technology for feedback, it may be useful to be conscious of the changes that can be brought about to the feedback mindset, and judge success in these terms rather than just mastery of, or persistence with one or another tool.
At the end of another period of working with lecturers from overseas I thought it would be useful to pause for thought and identify lessons from working in the area of transnational staff development. I have just completed my third international staff development summer school, and here is what I found. These points come with the caveat that they are only my learning, and others may, of course, have different views of what works.
Be conscious of your assumptions.
Before meeting academics from other continents it’s easy to let assumptions about what they may or may not be doing in practice, or what they may or may not believe, creep in to your thinking and planning. By example with tutors from China a common belief is that they are always involved in a transmission modes of education; this is simply not my experience and the view relates to outdated assumptions. While some Anglicized techniques may be new to colleagues from other locations my experience tells me we have a shared passion for making learning better, and some overlap already in our methods such as employing a flipped classroom.
Explore biography.
It’s always useful to start any transnational academic staff development by exploring the experiences and biography of the individuals involved. Those involved in supporting transnational staff development can then be agile and responsive to the specific needs of the group. Things that I have tried are very simple: Starting the development programme with a list of questions and concerns that the group would like to address, shared via a post-it-note board, hosting a session with no plan and offering the hot seat format where we simply respond to the questions of the group in an effort to take stock and allow international colleagues to add context to their growing understanding, and walking together to listen more informally to the needs of the group.
Ensure that learning is always two way.
One directional international staff development appears like neo-colonial self-righteousness. We need to employ empathetic methods wherein we offer our own practices, identify some of the limitations of these approaches, and also invite teachers from other locations and nations to do the same. It feels disrespectful to do anything else.
Get the basic accessibility matters right
Take care of the basics to ensure provision is accessible. Particularly it’s important to make sure that delivery is slow enough to allow the digestion of material; if you’re not sure, keep checking with the group. Also, ensure that resources (slides, papers, etc.) are available before the class so that translation apps can be used to familiarise with any tricky words.
Team teach, always.
Team teaching allows a richness which is not possiblealone; I tend to work with people who can offer a very practical take on the theoretical ideas and research that I am exploring. As well as simply providing more experience in the room, teaching in pairs provides an opportunity to model professional differences, which are inevitable in teaching. So by example when working with a colleague in a session about teaching evaluation, my view was that it is okay to identify some of the areas upon which you specifically want feedback from teaching observation, but my colleague’s view was that this might limit the range of feedback and prevent previously unnoticed habits or issues from surfacing. Our nuanced differences were explored publicly as a model of divergent views; this raises the exposure of the group to detailed discussion. Team teaching also provided in the moment opportunities for peer observation and debrief; this should be routinely incorporated through the provision of collegial feedback on what worked well, and what less so.
Use staff development as a vehicle for the host organisation’s staff development.
By keeping international staff development locked within one or two people, there is limited benefit. Encouraging colleagues who may be outside of the usual staff development circuit to come and join in, can, I think, have a considerable impact, in growing capacity and confidence for this type of work. It can also provide experience for aspiring Senior Fellows within the UK Professional Standards Framework. It can also provide challenge and a refreshing set of ideas, for staff who often manage staff development.
Be confident about lessons on technology transferability
Within international staff development programmes we may wish to explore tools for learning and teaching. Things like Twitter, Padlett, Nearpod, Facebook and Kahoot. A first thought may be, well what’s the point as we have access to different apps, and some of our apps may not be appropriate in a different cultural context. However, our experiences are that sharing is still good, even when the tools are not immediately transferable, as we can learn by return about alternative apps. Most importantly, since technology and the way we use it reflects much about the underpinning power, beliefs and values in learning, then exploring technology is a much more valuable experience than just swapping ideas on apps we like.
Hold discussion using first language
Group work with international, especially Chinese staff, is always very productive, but it is incredible demanding and limiting to make that group work happen in a second language (e.g. English). Permitting sense-making in the first language allows rich debate and discussion to evolve, rather than slowing down the pace and adding another cognitive load. This approach loses the opportunity for input to the discussion from the session host, but in my experience this is a process worth paying to enable lively group discussion, and in any case group précis in English can give the headlines.
Lead by self-exposure on the difficult discussions.
Asking questions, or discussing topics, which require some exposure of personal fragilities, can be tricky to get started. Working with my colleague, Jane Headley, we found that by sharing something of ourselves before we ask others to do the same was helpful in creating an open forum. So for example, when discussing technology, I shared some adverse feedback that I had received on my own approach; I then told the story of what happened next, and I identified the decisions I made that lead to a less than perfect learner experience. This lighthearted ‘fessing up’ made others comfortable to share their own critical incidents and learning from it. As well as showing openness, this deconstruction of practice also models reflection in action.
Use many examples and stories
Using real examples of situations, challenges and successes can really aid understanding, though remembering that some political context may be important to explain why one or another decision was taken at the time, e.g. we had funding for this type of work, or we are preparing for TEF.
Encourage journal writing
Finally, by encouraging some reflective writing after each topic or session, international staff can form their own ‘take away’ record of i) what were the key learning points and ii) what next to research, extent or apply aspects of learning? A simplified, structured formal of learning journals can promote consolidation and impact from learning.
This list is no doubt incomplete, so please do add any other points that you might have.
Creative commons image sources:
https://commons.wikimedia.org/wiki/File:Globe_terrestre_Orange_te_Bleu.svg
To help colleagues assess whether they are ‘really good’ Fellows, or instead ‘Senior Fellows’ I have created a three step self assessment exercise. This tries to encourage teaching and learning practitioners to think about the scholarly or evidence based approaches that they use, the values that underpin their own practice and critically the ways in which they exert leadership and influence over others. Please feel free to use it if useful.
In September 2015 I committed to deliver a thirty-credit module, called The Teaching Practitioner, using A flipped classroom pedagogy. The module is the first of two in a PgC Teaching and Supporting Learning in HE; it is associated with Associate Fellowship of the Higher Education Academy.
My motivation for flipping the classroom was three fold:
The pattern of delivery was simply that each week I shared materials to work through, including narrated presentations, videos (commissioned and existent), reading, reflective tasks and then we would gather to discuss. The discussions varied in formality, structure and style as the module progressed. Over the course of the module I learnt a great deal, the key points from my mental list of lessons are shared below.
To do list example (click to view)
Essential to do list: Each week I published what needed to be done in advance of the face-to-face class. Importantly the list split out what was essential and what was optional. Participants reported that this was a helpful organizing distinction and allowed better management of their activity. This is something that I would definitely adopt in future modules of any type to act as a pacesetter. Simple, perhaps obvious, but actively encouraging participants to make choices about the level of engagement they can make is a pragmatic way of supporting work based practitioners who have so many competing demands on their time.
Slides not videos: I experimented with the media format of presentational material (pre-class content). The staple across most weeks was the narrated PowerPoint. I found more editing control by using Audacity to record the audio and then drag and drop in to PowerPoint, compared to recording direct in to PowerPoint. Audacity gave me opportunity to edit out any major interruptions with ease (phone calls, door knocks etc). I included some video lectures of studio production quality however participants found them relatively less engaging, with a preference for visuals and audios mixed in together with the ability to more easily navigate the presentation. I was surprised by this preference, but there is no doubt narrated presentations are easier to create.
Don’t force theory: We took a discursive approach to our face-to-face time (which was usually two hours per week). I provided questions and starters and then tried to guide the discussion. At first the conversation was loose, multi-directional, on and off-topic. I worried that we were not being ‘very level seven’ and the participants shared some of these concerns. However an under the surface, a process of sense making was going on; each person, in their own language and terms, through sharing and reflecting on their own experience got chance to reconceive, affirm and evaluate their practice. The explicit linking to theory was a more private activity, which seemed to occur in response to assessment. It was only obvious that this had taken place at the end of the module as discussion and theory were fused. Perhaps the discussions were a shared liminal space in which we muddled through difficult issues, then we went away to individually reflect and make clear.
A conception of flipped learning as a three stage process
Facilitation skills matter more than online production skills: My role can be linked to all the activities of a facilitator, including:
A discussion summary in progress
As we progressed through the weeks, methods for each of these aspects became more developed e.g. creating graphics for summaries, defining the discussion purpose to keep us mainly on task. One thing I did from time to time was add a summary of the discussion as a resource for reference so that everyone had opportunity to revisit key points. This involved simply using my mobile phone and talking through the diagrams that we had created in class such that everyone had a record. This was not onerous at all if done straight after the session while fresh in memory.
Quick and dirty production process: If the model of delivery is going to be sustainable then resources need to be produced within a realistic time frame. By taking a quick and dirty approach to development, those on the programme see the approach as achievable and replicable; it provides accessible modeled practice. For me there is also a really clear sign in this approach that the value of the learning experience is the interaction and not a resource. To avoid perfectionism I never listened to my own presentations after they were recorded other than for a quick sound check.
Shared endeavor: While new roles were not formally defined, we fell in to a more even relationship. I sensed that we were co-researchers (in to the effectiveness of the pedagogy) and co-learners (about all aspects of the programme). We were facilitators and facilitated, rather than ‘teacher and student’. To reinforce this role equality, I tried to be very open about when I was learning too.
Allow choice about levels of engagement: As grown ups, participants face a simple rational choice about whether to engage or attend; sometime this choice is made in light of personal life and professional workload. In the weeks where individuals had not done the preparation for class, no action was taken or penalty applied. This approach relies on a commitment to engage and the rewards are implicit in the design. It also reflects the idea of running a community of equals. The group dynamic needs to be honest about the need for preparation, but pragmatic when this slips. If the facilitation works well then even those who have not prepared should be invited and able to contribute experience, and hopefully then inspired to retrospectively visit the online class.
A human process not a technical one: Flipped classroom may evoke thoughts about complex online tools and an unfathomable methodology of teaching promoted by centres of e-learning and academic development, but for me the experience of flipped classroom is a fundamentally human process which involves a respect the opportunity to explore individual experience and knowledge. It allows social learning and creates space for the discussion of any issues arising that matter to the group. I hope the language around this practice, and the identity of the learning model as slightly exotic, does not take away from the collegial simplicity, which resonates with traditional seminar based learning.
Support for the flipped approach from participants was demonstrated in three distinct ways: i) the adoption of flipped classroom by some group members ii) protest when classes are not flipped iii) outstanding, highly personalized, deeply connected assignments to demonstrate the culmination of meaningful engagement (though I am a little bias on the last point).
If I had a point nine on my list, it would be to keep faith that the approach will pay off, even when there is angst about its effectiveness. That said, when I saw in the module assessments that we had reached our destination (albeit a fleeting one on the way to the next module) I was very relieved!
Much has been written about the value of lectures: It is not clear whether they will endure, or if instead they are dated and doomed. Either way though I don’t see them going away any time soon. While the lecture has remained core in many universities, we know that lecture content (i.e. higher level knowledge) has gone from being underpinned by privileged information locked down within the academic community, to in many cases nothing more than ‘stuff’ that might be gleaned in a decent web search. How many colleagues Google source at least some of their content (be honest now!)?
The nature of knowledge has changed rapidly, but still the main mode of operation for ‘oh so many’ courses is transmission. When I hear others say ‘why would students come to lectures if there is … no assessment incentive … when there is lecture capture …or, when there are full notes on the virtual learning environment’, I can’t help feeling it’s the wrong question. We don’t want to dupe students in to coming to lectures by denying these beneficial actions just to sustain the status quo; this is a defensive approach, which devalues the time of students and the professionalism of lecturers. Instead it is perhaps better to ask ‘why would students come to class at all?’ What are we offering that is good enough and useful enough for students to want come along and engage?
There are clear synergies between higher education and contemporary trends in music. Commercial music, like knowledge, has gone from a prized product, to being cheap and accessible, and then to being free and abundant. Artists are succeeding by not holding on the old modes of distributing, but by making their product open and available, to whet the appetite of fans who will then travel, pay and commit time to go to the live event; to share an experience and share in something which cannot be consumed by other means. Often the benefits of live events lie in the way they make you feel rather than in a quantifiable outcome.
I am advising colleagues who are new to teaching to ‘find their rockstar’. This doesn’t mean becoming an edutainer; which I take issue with as it trivializes scholarship. It rather means locating the reasons that make the live coming together of people to learn, a valuable and meaningful experience. If the transmission mode is used, then what extra does the live performance add? When I watch a band the value for me is ‘feeling’ the bass, making memories, and being part of an audience. When I watch a speaker, I enjoy the focus, the interaction and the stories, which feel are told for the specific audience.
When asked by a researcher, who is new to teaching, what is my ‘rockstar’ and how do I find it? My answer … you already have it …
‘your unpublished and ongoing research’
‘your empathy with the students you truly appreciate their angst because a similar experience is not so far in your own past’
‘your calm and considered outlook coupled with boundless enthusiasm that makes your content far richer than any download’ (your newness means your passion is undiminished). This can be felt by those around you; it can’t be measured’.
If I think about brilliant lecturers they all have their own attributes that make attendance worthwhile – these are always not displays of radical teaching or ‘fizz-bomb’ personalities; but there is always something more than transmission. Examples include anecdotes, stories, empathy, humour, and enthusiasm, and research. Rather than assume students are disengaged we should make sure when they come to class they have something more than a download… it’s potentially a useful, confidence generating staff development reflection to #FindYourRockstar.
