Preliminary findings – Lecturer experiences of choosing and using technology in assessment feedback

Part way through my thesis research I have stood back to ask what is all this data saying? To this end I have produced a pause for thought document about the emerging findings. This is not the finished output, but in creating it I managed to consolidate my thoughts, and in sharing it I hope for any comments that may help refine further analysis or additional data collection.

Please feel free to leave any questions or comments in response.

And thank you to those who have assisted me in setting up the next round of interviews after a recent plea for help.

Preliminary Research Findings available here.

 

Marrying mixed methods and critical realism

As  I edit through my thesis I needed to cut out a short section that I had written about how mixed methods can be highly compatible with critical realism. As my work emerged I did not, in the end, use mixed methods.  This section was hard won and so simply deleting it felt like a waste. Therefore, attached to this post is a short piece about the compatibility of mixed methods and critical realism, in hope that it may be useful to someone else.

Download the short article here: Marrying mixed methods and critical realism

Efficiency, technology and feedback

In considering staff experiences of choosing and using feedback technology, one of the emerging themes has been the differing views on feedback technologies and efficiencies. While the jury is still out on the data and the process is incomplete, my observations are that efficiency can be conceived in different ways in the negotiation of technology. For some efficiency is a primary driver in the decision making process. The search for technology and the refinement of its use is motivated and shaped by the quest for efficiencies. For others efficiencies are a welcome benefit of technology – they are almost an unexpected gift – welcome, but not necessary. Efficiencies also appear to be conceived relatively; rarely are efficiencies discussed without a reference to the relative enhancement gains that can be made through a technology. Wherever there is a time saving there is a tendency to ‘re-spend’ the saved time making still more enhancements to the feedback – adding detail and depth for example. In this way efficiencies become difficult to identify as they are theoretically achievable but in reality they are trumped by the possibility for improvement. Efficiency also seems to be a veto concept for some; it is not a particular concern in the run of practice but is triggered only when a particular technology is likely to encroach other activities or provide an intolerable stress.

Empathetic validity and action research in educational development

As part of my doctoral studies I have recently undertaken an action research project relating to strengthening approaches to feedback practice. Informal reconnaissance led me to believe that feedback practice is very siloed. At the same time in the planning process I encountered a paper by Ball  (2009) who showed that collaborative practitioner centred action research  in itself can bring about the questioning of ones own practice, put simply, discussing the feedback practice of others shines a light on the way that each of us works and we then ask questions of ourselves and review how we might act differently.

Informed by this, my action hunch was that the development of an electronic sharing resource for good practice could be a mechanism for modelling good practice and bringing about transparency. Influenced by Ball, I envisaged that ensuring ownership of this resource by those who would use and populate it could act as a catalyst for critical dialogue around practice.

In seeking exemplary practice to populate the resource it became clear that there were some issues to address first. The project got messy in the way described by Cook (2009).  The intended action therefore was put on hold, and became a second project phase. This was clearly emergent work in progress.

The action research methodology permits these off-piste directions and in my search for good practice to populate the resource, I generated four spin off cycles to explore what is good feedback in this context? How can feedback practice be developed?  What are the barriers to developing good feedback practice? What conditions might be needed for the benefits of practitioner sharing to be realised?

I have take away learning about all of these points but by far the greatest learning from this research has been  the development of empathy with those engaged in the process. I have a much better sense of their experience and in staff development terms this is important for productive ways forward. My data was not vast and my conclusions didn’t add a lot to the already overflowing pool of literature on this topic, but it felt valuable. Trying to justify your research in terms couched in feelings is something that even I, as a self-confessed navel gazer, am not used to doing. In reading around this I was drawn to the work of  Dadds  who described a phenomenon called empathetic validity which refers to “the potential of the research in its processes and outcomes to transform the emotional dispositions of people towards each other, such that more positive feelings are created between them in the form of greater empathy” (2008, p208)

Whether empathy can really be incorporated a project aim I am not clear, I imagine it either happens or it doesn’t, but its benefit for me has trumped any of the the intended consequences of this project.

Ball, E. (2009). A participatory action research study on handwritten annotation feedback and its impact on staff and students. Systemic Practice and Action Research, 22(2), 111-124. doi: 10.1007/s11213-008-9116-6

Cook, T. (2009). The purpose of mess in action research: Building rigour though a messy turn. Educational Action Research, 17(2), 277-291. doi: 10.1080/09650790902914241

Dadds, M. (2008). Empathetic validity in practitioner research. Educational Action Research, 16(2), 279-290. doi: 10.1080/09650790802011973

Apps 2012

Image

As I have progressed through my EdD my ways of working have got a little smarter. There are four apps that have served me well in 2012 for supporting my studies …

1. Reminders (so in the hours where I have too much to do I can remember what they were!)

2. Good Reader – managing my online library downloads and annotating my reading without reams of paper. By far the best reading app I have found (still).

3. Good notes – high levels of functionality, a great jotter and annotator – good for generating diagrams and mapping out thoughts.

4. Splashtop – allows my desktop (including Endnote) to be fully functional from my ipad or phone. Excellent when not wanting to be stuck at my desk.

Two more fab apps (not study related)  for 2012 have been

5. Screenchomp – Jing for the ipad – great for audio visual feedback for students and again this means there is no need to be desk bound.

6. Spelling – my best parenting app! So the kids can input the spelling list for the week and then run the tests until the spellings stick. Very motivational for kids who hate spelling.

Strategy and mission

Having poured over some fairly hard going documentation and policy text books for a few days for assorted reasons I was pleased to stumble upon a refreshing approach to writing mission statements, which I’m sure would work equally well for policy and strategy documents!

OK, so not in a million years will HE documentation ever take this flavour, but wouldn’t it be better if it did!

C-Map

I am asked increasingly about concept mapping software. I have previously favoured iThoughtHD; however, while this is very intuitive it is not so good at enabling inter-label links (something only realised after a little time and intensive usage!). C-Map was recommended to me as an alternative. Though not native to the ipad, it has a greater focus on the links rather than the labels and in turn this helps the author to think about structure, more than the brain dump. It forces the user to clarify: Why is x connected to Y?

“A concept by itself does not provide meaning, but when two concepts are connected using linking words or phrases, they form a meaningful proposition”. (Villalon and Calvo 2011 p18)

C-map is downloadable for Windows and Mac and wonderfully, is free.

Below is my own mind map to demonstrate C-map (though I am confident that there are better examples!!). Click to view.
Lydia's map of learning theory

Villalon, J. and R. A. Calvo (2011). “Concept Maps as Cognitive Visualizations of Writing Assignments.” Journal of Educational Technology & Society 14(3): 16-27.

5 reasons why giving pass/fail marks, as opposed to percentage grades, might not be a bad idea

Grades1. Grades may be an inhibitor of deeper self-reflection, which is in turn linked to self-regulated learning (White and Fantone 2010). Grade chasing distracts from meaningful learning review (see also Dweck 2010). For real examples of this, some student views visible in the comments here are useful http://tinyurl.com/66r3mdu

2. Research shows that performance is neither reduced nor enhanced by pass/fail grading systems (Robins, Fantone et al. 1995). For those worrying about a reduction in standards caused by the removal of grades, don’t!

3. Pass-Fail grades are more conducive to a culture of collaboration, which in turn links to higher levels of student satisfaction (Robins, Fantone et al. 1995; Rohe, Barrier et al. 2006; White and Fantone 2010). The increased collaboration may be especially beneficial as preparation for certain professions which require high levels of cooperative working (as noted in a medical context by Rohe, Barrier et al. 2006).

4. Pass-fail counteracts challenges brought about by grade inflation practices (Jackson 2011).

5. Pass-fail is associated with lower student anxiety and higher levels well being (Rohe, Barrier et al. 2006). That has to be good!

Dweck, C. S. (2010). “Even Geniuses Work Hard.” Educational Leadership 68(1): 16-20.
Jackson, L. J. (2011). “IS MY SCHOOL NEXT?” Student Lawyer 39(8): 30-32.
Robins, L. S., J. C. Fantone, et al. (1995). “The effect of pass/fail grading and weekly quizzes on first-year students’ performances and satisfaction.” Academic Medicine: Journal Of The Association Of American Medical Colleges 70(4): 327-329.
Rohe, D. E., P. A. Barrier, et al. (2006). “The Benefits of Pass-Fail Grading on Stress, Mood, and Group Cohesion in Medical Students.” Mayo Clinic Proceedings 81(11): 1443-1448.
White, C. B. and J. C. Fantone (2010). “Pass-Fail Grading: Laying the Foundation for Self-Regulated Learning.” Advances in Health Sciences Education 15(4): 469-477.

Learning transitions and playing with concept map

In an attempt to unscramble my own thoughts and make them remain in my head longer than the time it takes to type and send I have embarked upon a concept mapping exercise. So taking a reply I made earlier to a forum question, and concept mapping it, it was useful but remarkably difficult. Its tricky to build links and annotate without over-crowding. A fine art and more practice needed. Tips welcome!

So, the question – how do transitions affect new comers to higher education And what might be done to help … ?

Secondary school students have often been engaged in surface learning which has been cultivated by a climate of testing and a grade facing culture (Hussey and Smith 2010). The learning habits and cultures of a secondary school are very different than most HEIs where large class sizes exist (Cook and Leckey 1999) and there is an expectation of autonomy (Hussey and Smith 2010) – a transition is needed to thrive in this new environment (a transition of self and in learning approach). In terms of self-concept students may go from being confident amongst a small group to feeling disorientated by their new place in the bigger order. At the same time as needing to undertake transitions in their approach to learning they may be undergoing great shifts in their personal life from being dependent to being independent as they move locations and away from family. Transitions are essential to learning and may occur on a number of fronts – in knowledge, in learning orientation, in social dimensions (Hussey and Smith 2010) and in epistemologies (Chan, Ho et al. 2011). First year attrition of students is high (Beaty, Gibbs et al. 1997; Cook and Leckey 1999; Hussey and Smith 2010). It seems little wonder given the multi-faceted transitions. Such transitions are more complex for first generation learners who face added challenges.

To facilitate these transitions a number of recommendations emerge from the literature:

· Induction – induction for students should address student expectations such that learners can ‘see’ the transitions ahead (Cook and Leckey 1999)
· Monitoring – so student transitions are not hidden from staff and can be engaged with positively and appropriately. Montitoring ensures teachers are not surprised at the end of a course when transitions have not occurred as anticipated (Hussey and Smith 2010)
· Avoiding a crammed curriculum – to provide space for deep engagement such that time for deep learning is made (Cousin 2006; Hussey and Smith 2010)
· Formative assessment – to enable students to develop an understanding of the expectations and allow them to adapt on their journey ahead of high stakes assessment (Hussey and Smith 2010)
· Progression flexibility – more radically, student programmes could be undertaken over a longer period of time where an individual’s transition path requires, when they are not ready to move on at the speed of the academic calendar (Hussey and Smith 2010)
· Promote study skills for higher education (ideally before arrival) such that students have to tools to adapt (Cook and Leckey 1999)
· A culture of support where disorientation or turmoil is OK (Cousin 2006)
· Ensure student’s make good course choices which hold their engagement (Jansen and Suhre 2010).

While study skills are suggested as being important, their role may be less significant than the process of aligning courses to student requirements (Jansen and Suhre 2010). In ensuring good choices of programme are made, students may develop higher levels of buy-in and therefore be more prepared to undertake active involvement in learning (which, in turn, is particularly helpful to the surface – deep learning transition (Atherton 2011)).

For first generation students there may be additional or exacerbated challenges. Self-concept lies at the heart of many transitions. First generation students may have a different self concept (perhaps in confidence, beliefs and self-worth) than those who have been socialized in to HE by family. The may feel that they do not belong (Mehta, Newbold et al. 2011).

Cultural changes may be exacerbated for first generation students. HE has its own culture and even language. For students who are first generation the newness of this culture will be starker since exposure to the language and rituals of HE may have been nil. Outreach programmes (from HE to school and induction may help).

According to Mehta, Newbold et al (2011) first generation students “enter college less prepared to succeed but also have greater time demands and financial commitments”. The distractions of financial pressures, part time jobs etc may be a challenge for some first generation students especially when they are immersed in so many transitions, and forming new learning habits.

Care must be taken not to over-generalise first-gen students in to one category though, since in itself this category represents great diversity of culture, class and values. For example, first generation students from a work-based background may face different self-concept issues than school leavers (as inferred by Hussey and Smith 2010)

Broadly the notion of personalizing provision and induction to individual student need may be an approach to facilitate transition. However particular attention may need to be offered to financial support, pace (in response to financial and emotional transitions), integration (social) and the management of expectations.

Much better perhaps to see it like this

concept map screen shot (section on transitions)

 

I suspect work on transitions could usefully inform personal development programmes as well as induction.

Atherton, J. S. (2011). “Learning and teaching: Approaches to study: Deep and surface learning.” Retrieved 3 August 2011, from http://www.learningandteaching.info/learning/deepsurf.htm.
Beaty, L., G. Gibbs, et al. (1997). Learning orientations and study contracts. The experience of learning. D. H. F. Marton, & N. Entwistle. Edinburgh, Scotland, Scottish Academic Press.
Chan, N.-M., I. T. Ho, et al. (2011). “Epistemic beliefs and critical thinking of Chinese students.” Learning and Individual Differences 21(1): 67-77.
Cook, A. and J. Leckey (1999). “Do Expectations Meet Reality? A survey of changes in first-year student opinion.” Journal of Further & Higher Education 23(2): 157.
Cousin, G. (2006). “An introduction to threshold concepts.” Planet(no. 17): 4-5.
Hussey, T. and P. Smith (2010). “Transitions in higher education.” Innovations in Education & Teaching International 47(2): 155-164.
Jansen, E. P. W. A. and C. J. M. Suhre (2010). “The effect of secondary school study skills preparation on first-year university achievement.” Educational Studies 36(5): 569-580.
Mehta, S. S., J. J. Newbold, et al. (2011). “WHY DO FIRST-GENERATION STUDENTS FAIL?” College Student Journal 45(1): 20-35.

Threshold concepts and troublesome knowledge (a brief introduction)

Troublesome knowledge (Perkins 2006) and threshold concepts (Meyer and Land 2006) help us to understand why learners do not learn at a steady rate. The trajectory of learning is peppered with stormy patches to be navigated. These periods may be associated with emotional turmoil, irreversible changes in perspective and challenges to identity.

Perkins (2006) explains why knowledge may be troublesome (e.g. It may be ritualistic, conceptually challenging or alien). This in turn helps us to understand how students may grapple to assimilate and accommodate information, how they may behave when thresholds are not dealt with (e.g. mimicry) and how they may be supported by teachers (see for example Cousins 2006).

According to Meyer & Land (2006) a liminal state exits for learners between entering and emerging from engagement with a threshold concept. There are parallels here with the emotional turmoil associated with critical thinking (Brookfield 2011). Cousins (2006) suggests that teachers may support students in their liminal state by empathising with them and by tuning into the factors which influence the learning journey of individuals to inform their support approach. Additionally she suggests teachers should permit liminal states and facilitate learners to know that they are not alone in their tumultuous time. Land (2011) advocates meta-learning approaches to equip students to cope with liminality. His recommended approaches include reflective logs, cognitive mapping and diarizing the learning journey (see Land 2011).

The use of recursive approaches (Cousins 2006; Land 2011) and varied learning techniques (Marton and Trigwell 2000; Land 2011) may be useful if learning happens in the manner proposed. Such methods enable the learner to approach a challenge from different angles. Learners will ‘click’ with one method or another or may build up a triangulated picture through the mixed methods to which they are exposed. Methods that are active, perhaps with creative and social dimensions, are deemed appropriate (Meyer and Land 2006; Hill 2010) for assisting learners through troublesome knowledge. These may include discovery learning, inquiry based learning and role-play, discussion, creative writing and experimentation (Perkins 2006). However, Hill (2010) cautions that not all learners always want to be active and prefer transmission modes.

Practitioners need to bring about pragmatic constructivism recognizing the reality of their operating context and traditions and nuances of the discipline when deciding on precise learning activities. The active learning process can be consuming on the part of both learners and teachers, so choices in the learning context should be sustainable and the curriculum not crammed (Cousins 2006).

Research in to threshold concepts has tended to focus on subject areas (e.g. economics, mathematics and research methods) across different levels, including for doctoral research students (see Kiley & Wisker 2009). The focus on subjects makes sense since precise choices of appropriate learning activities may be a product of:
-Subject tradition and nuances
-Types of knowledge under consideration
-The cause of ‘the trouble’
-Learner’s own prior view of the world

In their paper, Kiley & Wisker (2009) looked specifically at thresholds for doctoral students and suggest that there are six specific threshold concepts at this level for research students:

-Argument
-Theorising
-Framework
-Knowledge creation
-Analysis and interpretation
-Research paradigm

In a programme with a large social learning element the online discursive dimension may be loaded with it’s own challenges; especially since it encompasses, or at least touches upon, the core threshold concepts as cited above (or others). While academic community is proposed as a solution to negotiating threshold concepts (Wisker et al. 2010) engagement is also, in itself, a challenge.

Teachers may scaffold student progress but the responsibility for working through troublesome learning points ultimately remains with the learner. Bruner (1966 cited in Allen 2005) asserts that instruction should ultimately lead to self sufficiency in learners. Marton and Trigwell (2000) suggest that independence is important for the real world beyond student life.

For a comprehensive resource see Michael Flannegan’s pages: http://www.ee.ucl.ac.uk/~mflanaga/thresholds.html

Allen, K. (2005). “Online learning: constructivism and conversation as an approach to learning.” Innovations in Education & Teaching International 42(3): 247-256.
Brookfield, S. (2011). The Risks of Becoming Critically Reflective, Laureate Education.
Cousin, G. (2006). “An introduction to threshold concepts.” Planet (17): 4-5.
Hill, S. (2010). “Troublesome knowledge: why don’t they understand?” Health Information & Libraries Journal 27(1): 80-83.
Kiley, M. and G. Wisker (2009). “Threshold concepts in research education and evidence of threshold crossing.” Higher Education Research & Development 28(4): 431-441.
Land, R. (2011). Threshold Concepts and Troublesome Knowledge, Laureate
Marton, F. and K. Trigwell (2000). “Variatio Est Mater Studiorum.” Higher Education Research & Development 19(3): 381-395.
Meyer, J. H. F. and R. Land (2006). Threshold Concepts and Troublesome Knowledge: An Introduction. In J. H. F. Meyer & R. Land (Eds.) Overcoming Barriers to Student Understanding: Threshold Concepts and Troublesome Knowledge, 3-18. Routledge: London.
Perkins, D. (2006). Constructivism and troublesome knowledge. In J. H. F. Meyer & R. Land (Eds.) Overcoming Barriers to Student Understanding: Threshold Concepts and Troublesome Knowledge, 33-47. Routledge: London.
Wisker, G., Morris, C., Cheng, M., Masika, R., Warnes, M., Trafford, V., Robinson, G., & Lilly, J. (2010) Doctoral Learning Journeys (draft report). HEA: York