Course level assessment – nice idea, but what does it really mean?

It is increasingly clear that thinking about curriculum in the unit of ‘the course’ rather than the unit of ‘the module is conducive to cohesive course design. It avoids repetition, ensures the assessment journey makes sense to the student and can make feedback meaningful as one task is designed to link to the next. I have not found much  in the literature on course level assessment; while it is advocated in principle amongst educational development communities, it is perhaps less clear what course level assessment actually looks like.

I can see three possibilities, though there may be more. These conceptions are described as if delivered through the modular frameworks which remain the dominant framework for programmes. Any comments on other approaches would be very welcome.

Type 1: Compound assessment

Imagine two modules being taught on entirely discrete themes. Within them might be learning about terminology, key theories, processes, and calculations. Within the modular framework they may be taught entirely independently. In such a model there is nowhere in the curriculum where these skills can be overtly combined. A third module could be introduced which draws upon learning from module one and module two. Of course in reality it may be five modules drawn upon in a sixth compound module.

By example, a module focused upon business strategy may be taught entirely separately from a module on economics. Under such a scenario students may never get to consider how changes in the economy influence strategy, the associated tactics and the need for responsive planning. It is these compound skills, abilities and levels of professional judgment that the course (not the modules) seek to develop. One way of addressing this limitation is to provide a third module which draws together real business scenarios and concentrates on encouraging students to combine their knowledge. A ‘compound’ module could be based around case studies and real world scenarios, it may be limited in its ‘indicative content’ and leave a degree of openness to draw more flexibly on what is happening in the current external environment. Open modules can be uncomfortable and liberating in equal measure for the tutor, as there is a less familiar script. It might concentrate on the development of professional behaviours rather than additional content.The module might have timetabled slots, or could take the form of a one off exercise, field trip or inquiry. Teaching would be more facilitative rather than content/delivery led.

One of the challenges with such a module is that many tutors may be reluctant to give over credits to what seems to be a content free or light module. Going back to basics though, graduates are necessarily more than empty vessels filled with ‘stuff’. If we look at the course level and identify what we want to produce in our outcomes, and what the aims of our programmes actually are, then the flexible compound module fits well as an opportunity for fusing knowledge and developing competent, confident, early professionals. When knowledge is free and ubiquitous online, acting as a global external hard disk, we need to look at the graduates we build and challenge any view that higher education is primarily about the transfer of what the lecturer knows to the student. Surely the compound skills of researching the unfamiliar, combining knowledge from different areas, and making decisions with incomplete data in a moving environment are much more important. The compound module is an opportunity to facilitate learning which alights with the course level outcomes sought.

This type of course level learning and assessment undoubtedly requires an appreciation of the skills, attitudes, values and behaviours that we wish to foster in students and it needs confidence in the tutor to facilitate rather than transmit.

Type 2: Shared  assessment

The next way that I can conceive a form of course level assessment is more mechanistic. Take two modules (module one and module two, taught separately); to bring about efficiencies, the assessment of each module is undertaken within the same assignment, activity or exam. It may be an exam with two parts one for each module; it may be a presentation which is viewed by two assessors, each reviewing a separate aspect of content or it could be an assignment which has areas of attention clearly marked for each module. The education benefits of this are, in my view, much less obvious than for type 1, nevertheless students may see some links between the parts of modules in taking such an approach. The shared assessment must be designed to make clear which aspect relates to which module or else a student could be penalised or rewarded twice for the same points. Under such an approach it is conceivable to pass one element and fail the other. I remain to be convinced of the real benefits of this approach which feels like surface level ‘joined up-ness’.

Type 3: Combined assessment 

The term combined assessment is used here to describe an approach which assesses two modules through a single meaningful strategy. If there are two fifteen credit modules, one on mathematics for engineers and one on product design, an assessment which uses knowledge from each taught unit can be drawn upon to pass a single assessment – for example via a design and build exercise. The assessment subsumes both modules, the two elements are integrated (in contrast to the shared assessment approach) and there are potential marking efficiencies. Without clear attribution of marks to one or the other module it may be tricky when a student fails; what do they restudy? But presumably a tutor would be able to advise where the limitations of the performance are and which unit would be usefully revisited. In some cases it may be both. In reality this approach may be little different than having a large module with two themes contained within it.

So they were my three ideas for programme level assessment but I am convinced that there are other ways of achieving this in a meaningful way. The suitability of each approach will depend on what the course team want to achieve, but clearly the benefits of the compound assessment approach are very different from a shared or combined strategy.

permalink jigsaw header image courtesy of Yoel Ben-Avraham under Creative Commons

Viva preparation

Having survived my doctoral viva on the EdD programme at the University of LiverpooI, I wanted to share some of my own experiences in the hope that they might be useful to others.

  • To prepare for my viva the first thing I did was take six weeks away from the research and from even thinking about the doctorate. This was an important preparatory step to make myself objective about the thesis when re-engaging; essentially it allowed me to come back with fresh eyes and a clear mind. I questioned whether this was a wise thing to do as some advice says keep  reading around your topic in the gap, but I really valued the break.
  • I then read my thesis back (twice), from cover to cover. As I read I annotated typographical errors. I decided not to berate myself for their presence, since that would be a distraction. Finding these minor typographical and phrasing errors early on was helpful as it removed any sense of thinking that the viva will be the final step (it became clear that I would need to make modifications). Getting this realization over and done with earlier in the process made my expectation management much easier.
  • As I read, I noted areas where I felt I should have said more. Particularly I noted areas where I had said more and then, for editorial reasons, cut back on the detail. In these cases I read through the words and diagrams that were reluctantly cut out of the final draft (I always saved copies of earlier drafts of each chapter). Logically I figured, if I had struggled to cut out certain sections, their eventual absence may be of concern to the examiners. Reacquainting myself with this material was invaluable. By example I had cut out text which explained how the two strands of analysis in my study were synthesized. I had cut out this detail in the editing process, but re-familiarizing with it in the viva preparation allowed me to answer questions on this apparent gap in the thesis.
  • I  used a number of websites to generate common viva questions. I found one from the University of Leicester particularly helpful (see ).
  • Armed with the lists of questions, I generated answers in my mind. I did not write them down, frankly I didn’t think this helped. I sometimes committed two or three questions to mind and mulled them over while driving. This was a useful exercise as I could happily mumble answers to myself in the privacy of my car. Answering two or three at a time was enough to keep concentration. Tackling  more that three questions in one sitting was not particularly productive for me.
  • None of the ‘text book’ questions came up, but without doubt, these generic questions focused my attention and provided very good preparation.
  • I made a conscious effort to talk with others about my research before the viva. This gave me an opportunity to clarify my own understanding and to make the research accessible. Responding to my ten year old son’s question, what is your thesis about? Was actually the most challenging and the most valuable step in this process. he pushed me to be able to explain it in a way that he could understand.
  • Another  useful pre-viva question to consider was ‘which three works most influenced your research?’. Answering this  really forced me to focus on how I had used different influences, in turn this brought further clarity to the themes and ideas within the work.
  • Keeping perspective was very important in getting ready. One side of my brain felt like my doctorate depended on the viva. The other side constantly reminded me that the thesis and viva are, in reality, part of an extended study journey and should not be seen, as with a PhD, as the only product of assessment. Essentially I was two-thirds of the way to success without the project. This was a calming fact.
  • I found it really useful to ask myself ‘what would be the worst questions that could come up?’. Answering this is a real test of knowing your own limitations and those of your research. Sure enough my worst question came up (after all if I recognised this as a weakness in my research, surely others would too!). Having accepted this area as a weakness in advance, I was able to read around the issue and fill the gap. I was therefore comfortable on the day with defending my position, while accepting that some of the things I had learned through revisiting the issue would be usefully incorporated in modifications. This is a long way of recommending viva candidates face up to the areas that you know are weakest, in advance of the viva, and use the new found impetus that this phase of your journey brings to resolve any concerns that might have seemed unfathomable under the pressure to complete for submission.
  • Practically, I used post it notes to separate the chapters of the tome. This was useful for finding my way around the parts quickly when questions were asked. Also I researched the outcomes of the viva, so that I was prepared to hear the judgment and absorb the critical information, rather than get lost in the terminology.
  • Finally, one of the biggest challenges was to manage my own, and my supporters, expectations of completing the viva. While some friends/family/colleagues/strangers on a train congratulated me, with minor modifications outstanding I couldn’t fully celebrate. I had anticipated feeling like the viva was the end of the doctorate, but on the day it was just another milestone (albeit a significant one). This was a massive deflation. I wanted to keep the champagne on ice a little while longer. This was tricky to manage when others saw this as the finish line. In the end I settled on celebrating twice. In your mind be clear whether you feel your doctorate is over after the viva, or when any modifications are in. For me it was the latter.

Action research for higher education practitioners: Booklet

I have formed a short guide to action research particularly to support colleagues in higher education who may be undertaking action research for the first time. This is absolutely not intended to be a substitute for literature but it is offered as a ‘first stop’ for anyone contemplating this methodology. It offers practical ideas and tips and seeks to answer some of the key questions that I understand new action researchers to have. As ever, any feedback, additional inclusions or suggestions for revision would be welcome.


Download the booklet here: Action Research Introductory Resource

Five quick ways to write reflectively

  1. Imagine an audience for your musings. It’s hard to write without an audience. Write like you are talking to someone that you trust and connect with, and to extend your thoughts imagine their probing questions when you hit natural pauses.
  2. Talk, don’t just write. Use voice memos on your phone to capture thoughts in the moment and then write them down when back at base. Some of the most reflective thoughts happen in the car – catch them! This model is effective with adults and children alike.
  3. Use a model … a blank page can be daunting, use a reflective model to provide a writing frame for your reflections. Gibbs is my favourite but there are others too …
  4.  Go beyond describing what happened in an event or situation. Always follow up with the question, so what? (so what …. For me, for my students, for my colleagues, for my CPD needs, for my confidence, for my progression , for my efficiency, for my well-being?*).
  5. Write quickly, naturally and without concern for prose. This is a first layer of reflection. Then  a) develop the text and tidy it up and b) add comments or text boxes to annotate and add further observations on your initial thoughts. Comments or annotations can add major depth compared to a first attempt – ‘when I wrote this, I was thinking …. And I thought this because … but now I have discussed it with my colleague/friend and have revised my original understanding’ or ‘ I can see the choices I made here were limited by ….’. Adding layers to a reflection in this way can be very productive and can help us to question how we see things in the moment.

* Delete and expand!

Other ideas welcome.

5 reasons why giving pass/fail marks, as opposed to percentage grades, might not be a bad idea

Grades1. Grades may be an inhibitor of deeper self-reflection, which is in turn linked to self-regulated learning (White and Fantone 2010). Grade chasing distracts from meaningful learning review (see also Dweck 2010). For real examples of this, some student views visible in the comments here are useful

2. Research shows that performance is neither reduced nor enhanced by pass/fail grading systems (Robins, Fantone et al. 1995). For those worrying about a reduction in standards caused by the removal of grades, don’t!

3. Pass-Fail grades are more conducive to a culture of collaboration, which in turn links to higher levels of student satisfaction (Robins, Fantone et al. 1995; Rohe, Barrier et al. 2006; White and Fantone 2010). The increased collaboration may be especially beneficial as preparation for certain professions which require high levels of cooperative working (as noted in a medical context by Rohe, Barrier et al. 2006).

4. Pass-fail counteracts challenges brought about by grade inflation practices (Jackson 2011).

5. Pass-fail is associated with lower student anxiety and higher levels well being (Rohe, Barrier et al. 2006). That has to be good!

Dweck, C. S. (2010). “Even Geniuses Work Hard.” Educational Leadership 68(1): 16-20.
Jackson, L. J. (2011). “IS MY SCHOOL NEXT?” Student Lawyer 39(8): 30-32.
Robins, L. S., J. C. Fantone, et al. (1995). “The effect of pass/fail grading and weekly quizzes on first-year students’ performances and satisfaction.” Academic Medicine: Journal Of The Association Of American Medical Colleges 70(4): 327-329.
Rohe, D. E., P. A. Barrier, et al. (2006). “The Benefits of Pass-Fail Grading on Stress, Mood, and Group Cohesion in Medical Students.” Mayo Clinic Proceedings 81(11): 1443-1448.
White, C. B. and J. C. Fantone (2010). “Pass-Fail Grading: Laying the Foundation for Self-Regulated Learning.” Advances in Health Sciences Education 15(4): 469-477.

Threshold concepts and troublesome knowledge (a brief introduction)

Troublesome knowledge (Perkins 2006) and threshold concepts (Meyer and Land 2006) help us to understand why learners do not learn at a steady rate. The trajectory of learning is peppered with stormy patches to be navigated. These periods may be associated with emotional turmoil, irreversible changes in perspective and challenges to identity.

Perkins (2006) explains why knowledge may be troublesome (e.g. It may be ritualistic, conceptually challenging or alien). This in turn helps us to understand how students may grapple to assimilate and accommodate information, how they may behave when thresholds are not dealt with (e.g. mimicry) and how they may be supported by teachers (see for example Cousins 2006).

According to Meyer & Land (2006) a liminal state exits for learners between entering and emerging from engagement with a threshold concept. There are parallels here with the emotional turmoil associated with critical thinking (Brookfield 2011). Cousins (2006) suggests that teachers may support students in their liminal state by empathising with them and by tuning into the factors which influence the learning journey of individuals to inform their support approach. Additionally she suggests teachers should permit liminal states and facilitate learners to know that they are not alone in their tumultuous time. Land (2011) advocates meta-learning approaches to equip students to cope with liminality. His recommended approaches include reflective logs, cognitive mapping and diarizing the learning journey (see Land 2011).

The use of recursive approaches (Cousins 2006; Land 2011) and varied learning techniques (Marton and Trigwell 2000; Land 2011) may be useful if learning happens in the manner proposed. Such methods enable the learner to approach a challenge from different angles. Learners will ‘click’ with one method or another or may build up a triangulated picture through the mixed methods to which they are exposed. Methods that are active, perhaps with creative and social dimensions, are deemed appropriate (Meyer and Land 2006; Hill 2010) for assisting learners through troublesome knowledge. These may include discovery learning, inquiry based learning and role-play, discussion, creative writing and experimentation (Perkins 2006). However, Hill (2010) cautions that not all learners always want to be active and prefer transmission modes.

Practitioners need to bring about pragmatic constructivism recognizing the reality of their operating context and traditions and nuances of the discipline when deciding on precise learning activities. The active learning process can be consuming on the part of both learners and teachers, so choices in the learning context should be sustainable and the curriculum not crammed (Cousins 2006).

Research in to threshold concepts has tended to focus on subject areas (e.g. economics, mathematics and research methods) across different levels, including for doctoral research students (see Kiley & Wisker 2009). The focus on subjects makes sense since precise choices of appropriate learning activities may be a product of:
-Subject tradition and nuances
-Types of knowledge under consideration
-The cause of ‘the trouble’
-Learner’s own prior view of the world

In their paper, Kiley & Wisker (2009) looked specifically at thresholds for doctoral students and suggest that there are six specific threshold concepts at this level for research students:

-Knowledge creation
-Analysis and interpretation
-Research paradigm

In a programme with a large social learning element the online discursive dimension may be loaded with it’s own challenges; especially since it encompasses, or at least touches upon, the core threshold concepts as cited above (or others). While academic community is proposed as a solution to negotiating threshold concepts (Wisker et al. 2010) engagement is also, in itself, a challenge.

Teachers may scaffold student progress but the responsibility for working through troublesome learning points ultimately remains with the learner. Bruner (1966 cited in Allen 2005) asserts that instruction should ultimately lead to self sufficiency in learners. Marton and Trigwell (2000) suggest that independence is important for the real world beyond student life.

For a comprehensive resource see Michael Flannegan’s pages:

Allen, K. (2005). “Online learning: constructivism and conversation as an approach to learning.” Innovations in Education & Teaching International 42(3): 247-256.
Brookfield, S. (2011). The Risks of Becoming Critically Reflective, Laureate Education.
Cousin, G. (2006). “An introduction to threshold concepts.” Planet (17): 4-5.
Hill, S. (2010). “Troublesome knowledge: why don’t they understand?” Health Information & Libraries Journal 27(1): 80-83.
Kiley, M. and G. Wisker (2009). “Threshold concepts in research education and evidence of threshold crossing.” Higher Education Research & Development 28(4): 431-441.
Land, R. (2011). Threshold Concepts and Troublesome Knowledge, Laureate
Marton, F. and K. Trigwell (2000). “Variatio Est Mater Studiorum.” Higher Education Research & Development 19(3): 381-395.
Meyer, J. H. F. and R. Land (2006). Threshold Concepts and Troublesome Knowledge: An Introduction. In J. H. F. Meyer & R. Land (Eds.) Overcoming Barriers to Student Understanding: Threshold Concepts and Troublesome Knowledge, 3-18. Routledge: London.
Perkins, D. (2006). Constructivism and troublesome knowledge. In J. H. F. Meyer & R. Land (Eds.) Overcoming Barriers to Student Understanding: Threshold Concepts and Troublesome Knowledge, 33-47. Routledge: London.
Wisker, G., Morris, C., Cheng, M., Masika, R., Warnes, M., Trafford, V., Robinson, G., & Lilly, J. (2010) Doctoral Learning Journeys (draft report). HEA: York

How can work based learners develop effective study strategies?

Success in work-based study requires the development of effective personal study strategies (Lee, 2008) to enhance independence, confidence, ability to manage complex projects or tasks and to maximise the integration of learning in the workplace.

So how can work based learners develop effective study strategies?

1. Make time to understand the expectations of the level of study at which you are working. This can help you align your expectations.

2. Understand that uncertainty is ok. There will be times when disorientation kicks in, but this period of difficulty between meeting new ideas and engaging with them is perfectly natural. It is important for deep learning. A break can help when this kicks in.

3. Get it written, don’t aim to get everything perfect. As a work based learner there is inevitable trade off between getting the job done and perfection. Heed Wellington et al’s. (2005) advice get it written rather than striving to get it absolutely right.

4. Take control of managing time. Make dedicated diary time. When you are not in those slots avoid preoccupation with study and immerse fully in other activities. When slots are short be realistic about what can be achieved (take a task or part thereof, that fits the time available). Be realistic and proactive with planning. Planning should account for the schedule of study but also, for sustainability, wider work and life commitments (see Lee, 2008; Vitae, 2010; Wellington et al., 2005).

5. Negotiate support. Family and friends can be really essential to work based learners. Support will not magically appear. Be proactive in asking for help. Be pro active in explaining the demands of study and negotiate what support each party may helpfully give. In a similar vain it may be important to recognise that supporters too may also require a channel to ‘let off steam’.

6. Negotiate work based support. Colleagues may provide support by respecting study time, actively exchanging ideas in an overlapping community of practice, providing financial support or time and providing access to material and situations to provide an effective work based context for study.

7. Get organised. In the early firefighting period of adjustment to a new course it is easy to neglect steps that will help ongoing ease of study. The use of filing, labelling and referencing lists or data bases can be valuable. An investment at the beginning is hugely valuable.

8. Keep a learning journal. The learning journal can be a place to have a conversation with yourself. It enables you to work through challenges and difficulties and to plan next steps. It can be very levelling especially at times of uncertainty.

9. Use others in the group. Beyond the formal group activities that may form part of a course make time to engage with others on the course to share stories, understanding others are on the same journey can be very comforting.

10. Have goal and break this down in to smaller goals. Modules, assignments even activities. Celebrate each achievement and recognise progress. Map out your milestones.

Personal qualities
Alongside good planning and time management skills are the attributes needed to execute the plan. The role of personal attributes for study cannot be underplayed. Qualities such as persistence, an acceptance of uncertainty as well as a thirst for knowledge with personal determination (or as Lee calls it, a zest) are all concomitant with success.

Each step in the learning journey will look and feel different. It can be valuable to take an active role in meta-learning, the process by which learners develop an awareness of their own learning process (Biggs, 1985) and become expert learners (Ertmer & Newby, 1996). This is an ongoing approach to be adopted throughout the learning journey to help navigate current and future unforeseen challenges.

Biggs, J. B. (1985) The role of meta‐learning in study processes. British Journal of Educational Psychology. 55 pp. 185‐212.

Ertmer, P. A. & Newby, T. J. (1996) The expert learner: strategic, self‐regulated, and reflective’ Instructional Science. 24 pp. 1‐24.

Lee, N. J. (2008). Achieving your professional doctorate. Berkshire: Open University Press.

Wellington, J., Bathmaker, A., Hunt, C., McCulloch, G., & Sikes, P. (2005). Succeeding with your doctorate. London: Sage

Vitae. (2008). Start off a new year with a healthy mind in a healthy body. PGR Tips, (15). Retrieved June 2 2011 from

Professional development on professional development

I usefully spent today at Liverpool John Moores University in a workshop run by Janet Strivens and Rob Ward from the Centre for Recording Achievement, around PDP and eportfolios.

Key points from the session:

PDP is a culture that can be embedded and not an output

The use of constructive alignment can be stifling and at odds with the irregularity and risk taking nature of PD

There is a degree of confusion in practice about PD as output and PD as process.

PD incorporates meta learning, decision making and critical self review. It is an intense activity and should not be reduced only to blocks but should be valued and embedded.

PD helps learners assume responsibility for learning; the knock on effect is that staff are freed to teach and enable and not to micromanage. (there is then a potential efficiency).

PD and e-portfolios are complementary (enhanced reflection, asynchronisity) but can also be the source of tension when the technology is the design driver.

Good PD is intensive to demands upfront design.

Whilst staff and students may resist and in some cases resent PD, once normalised in to systems staff and students can see huge value in developing mutual understanding of learning styles, skills in managing uncertain knowledge and information, skills for the changing world and personal realisations.

Confusion between employment skills and employability skills may be evident in some systems.

PD in a competitive HE Market is hugely important for adding value to an individual; the evidence base is growing to help articulate this case (see CRA).

The language of PD can be interpreted to focus on practice, practise, application, planning and reviewing. It need nit be abstract.

Students showcasing their attributes through portfolios and cvs etc. is only a tiny part of the PD journey, the real value for the student is in the journey of self understanding that ultimately informs such outputs.

Questions remain around what to assess: quality of evidence, ‘academic-ness’ of reflection or individual progress.

Excellent examples of embedded PD are brave and bold!

For more information

UVAC 2010

Liz Warr and I hosted a workshop session on frameworks, wrapper modules and inquiry based learning design at UVAC 2010. The workshop was informed by work undertaken in the last 12-18 months at Harper Adams through the REEDNet project and formally the Aspire CETL.

I think it was fair to say that the most interest in the session was on wrappers and re-scalable modules. To read more about the wrapper idea you can click here or go straight to the paper.

Thanks to those who joined us today, it has certainly helped us move on our thinking and to consolidate some thoughts. Here is the session presentation for reference: