• Home
• About
• Login
• Register
• Search
• Current
• Archives
• Announcements
• Pre-2013 Index

Home > Vol 22, No 3 (2016) > Hervey

"Now I understand what you were trying to do, I see that this was the best module I had at University": Student Learning Expectations Reviewed Eight Years Later

Tamara Hervey* and Jamie Wood**[1]

* Jean Monnet Professor of EU Law, School of Law, University of Sheffield

** Principal Lecturer in History, School of History and Heritage, University of Lincoln

Abstract

The article reflects on a small longitudinal case study of a large compulsory module in a pre-1992 red-brick law school, using a narrative discursive method. It suggests that "alternative" methods of university law school teaching, such as Problem Based Learning, are experienced by students at the time as unsatisfactory. After being in graduate employment for several years, the students' experience has entirely changed. At that stage in their development and careers, students understand the relationships between the skills being fostered in such modules, and employment in the legal or other graduate professions. Processes, such as the Teaching Excellence Framework, that rely too strongly on contemporaneous assessments of the quality of student learning, such as the National Student Survey, create strong individual and institutional incentives to avoid such "risky" teaching. This undermines the stated aim of enhancing "employability" through university education.

1. Introduction

"What Robyn likes to do is to deconstruct the texts, to probe the gaps and absences in them, to uncover what they are not saying, to expose their ideological bad faith, to cut a cross-section through the twisted strands of their semiotic codes and literary conventions. What the students want her to do is to give them some basic facts that will enable them to read the novels as simple, straightforward reflections of 'reality', and to write simple, straightforward, exam-passing essays about them." [2]

David Lodge's 1988 humorous observation, that the desires of university students may be at odds with the insights of those who teach them, reminds us that we need not accept the notion that student satisfaction with university learning is an appropriate measure of teaching excellence. Students may "consistently ask for the wrong things". [3] They may not understand the difference between teaching and learning, they may seek "easy" or "unsophisticated" modes of teaching, and they may judge the quality of their learning experiences on that basis.[4]

In this article, we report on a small longitudinal qualitative study of a module that adopted a pedagogy (Problem Based Learning: PBL) that was far from "easy" for the students concerned, and that, at the time (2007-8), set the module apart from others that the students studied contemporaneously. [5] Using a narrative and discursive method, we report on the students' responses to that module at the time they studied it. As these were second year students, we conjecture that their responses to the National Student Survey (NSS), which would have been lodged less than a year later than these responses were captured, are likely to have echoed these views. We assume, therefore, that the students would have reported themselves as fundamentally "unsatisfied" with that module, in the terms of the NSS. We then report the results of follow-up qualitative research conducted some eight years later. The students we were able to contact and who agreed to participate have all been in the workplace for a number of years. Their views of the module, unsurprisingly, [6] are now strikingly different. Indeed, they are the polar opposite to the views they expressed when they were students. We argue, therefore, that the narratives about quality of Higher Education based on "student satisfaction" and "employability" [7] are, at best, misplaced if they rely too heavily on the views of students captured through relatively short-term mechanisms such as the NSS, before students have experienced the world of graduate work. Furthermore, there are troubling conclusions to be drawn about the perverse incentives created by such metrics-based assessments of quality, when considering the effects of such measures at both individual and institutional levels.

2. Context: Problem and Inquiry-based Learning in University law schools

Problem-Based Learning (PBL) and Inquiry-Based Learning (IBL) are student-centred and active forms of learning which engage students, individually and collaboratively, in self-directed research into the subject matter and problems of their academic and professional disciplines. [8] In PBL, students are presented with a problem, to which there is usually a known solution, and are guided through the process of addressing that problem by a facilitator. [9] By contrast, in IBL, students are generally given greater freedom to define for themselves both the questions they will address and the processes by which they will engage with those questions. Although there are variations according to discipline, level, teaching philosophy, the approach of individual academics and students, [10] IBL is essentially question-driven, while PBL is problem-driven. Both seek to develop skills and professional competencies for future employment, through 'real world' engagement,[11] as well as academic disciplinary knowledge and skills. For instance:

"The case scenario, which describes a situation inviting social work intervention and tells a story unfolding over time, requires students to construct their own analytical framework, through addressing definitional and problem-solving questions." [12]

Research across a range of disciplines has shown that the research process of searching, finding, evaluating, using and communicating information that is central to PBL and IBL pedagogies means that it is essential for students to develop their (transferable) competencies in information and digital literacy if they are to engage successfully in either mode of learning. [13] In addition to developing students' skills and knowledge in the discipline, it has been suggested that IBL, PBL, and other research-based pedagogies can be effective in enabling students to cultivate a range of other essential capabilities and dispositions such as independent learning, effective time management, critical thinking, decision-making, an ethical outlook, and citizenship, all of which are essential for employment and life after university. [14] IBL seems to have been particularly effective at enhancing students' ethical and legal awareness across a range of disciplines. [15]

Criticisms of IBL, PBL and other constructivist forms of teaching and learning have sometimes focussed on the perception that they fail to provide students with sufficient guidance and support for their learning. [16] However, the aim of well-designed PBL or IBL pedagogies is not to leave students to fend for themselves, but to provide them with a structured and supportive environment in which they can carry out their work: [17] it is poor teaching that results in students receiving insufficient support and guidance (often termed "scaffolding for learning"), not inquiry-based or problem-based pedagogies as such.

Over recent decades, IBL and PBL have been integrated into a growing range of legal curricula at undergraduate and postgraduate levels. Mackinnon's review of the adoption of PBL in legal education in New Zealand [18] argued that, as well as developing disciplinary knowledge and skills, PBL pedagogies develop students' transferrable skills, including the capacity for reflection, and therefore should be adopted more widely. Legal educators have been active in implementing IBL and PBL approaches in the UK too, although few UK law schools adopt the method across the whole curriculum. [19] IBL and PBL are regularly reported on in The Law Teacher. [20] In the UK, Sheffield Law School's adoption of PBL for some undergraduate modules drew on the broader work undertaken in the University's Centre for Inquiry-Based Learning in the Arts and Social Sciences (CILASS). [21] Sheffield Law students across all undergraduate levels have opportunities to engage in individual and collaborative research projects with the aim of developing information literacy skills, understanding of the research processes involved in legal studies, and subject knowledge. [22]

These and other studies on IBL/PBL in law and related disciplines have demonstrated the potential of such pedagogies to develop students' legal knowledge, as well as transferable and legal professional academic skills. Importantly, PBL seems to be particularly effective in enabling students to improve their ability to think as apprentice practitioners within the discipline. This was certainly one of the key rationales that underpinned the adoption of the pedagogy in the Level 2 undergraduate EU Law module at Sheffield, on which we report in this article. PBL was chosen inter alia as a pedagogical method appropriate to a context where 'body of knowledge' learning (and assessment) had to be blended with skills-based learning (and assessment). The problems around which the module was structured, and which formed the basis of the assessment, captured the body of knowledge, as determined by the legal profession and legal academy, as one of the 'foundations of legal knowledge'. The themes that emerge from our discussions have been identified in other research into IBL and PBL and have broader implications for the evaluation of teaching and learning in Higher Education, which we discuss further below.

3. Method

Data on the student experience of the module was captured in several ways, at the time (in 2007-08), and in 2014-15. In January 2008, the standard School of Law student feedback mechanism (online questionnaires, appendix 1) was supplemented by a feedback questionnaire (appendix 2) focusing on the intended learning outcomes of the module, which was completed by 60 students. During the module, students were asked to give anonymous reflections at the end of a lecture, about what they felt were the best and worst things about the module. These reflections formed the basis of questions discussed with a small focus group (appendix 3), formed of volunteers from the student cohort, and from that group, three students volunteered to co-author a discursive account of the module, in the form of a conversation between the module convenor and the students. In 2014-15, we contacted the three students via social media (LinkedIn and Facebook) and asked them to fill in a short questionnaire on their views on the module now, and give any other feedback they wished to provide. All three gave information. They were also asked if they could recommend any other former students we might approach who might be willing also to fill in the questionnaire.

The data was examined thematically, by the authors, and was compared with previous and contemporaneous qualitative and quantitative student feedback on the module convenors' teaching. The comparison showed a marked decline in student satisfaction compared to other modules, and indeed the module convenor was asked to see the Head of School, to discuss student complaints that had reached him. Of course, one possible explanation is that the module convenor's "conventional" teaching was good, but her teaching within a PBL method was poor. However, the student feedback praised the convenor's communication skills, passion for the subject, and willingness to give the students a great deal of time and attention. Given this, coupled with the convenor's experience, which is recognised both internally[23] and externally,[24] and the fact that the convenor adopted a PBL methodology voluntarily, and in collaboration with PBL experts employed by the University of Sheffield, this seems an unlikely explanation, though it cannot be discounted entirely.

The themes that emerged from the student feedback (apart from the lack of overall satisfaction with the learning experience) were: workload; comfort/discomfort with using lectures to develop skills; and staff-student interaction: in seminars, learning journals, discussion boards and feedback. We discuss each in turn, using the co-authored discursive account described above. [25]

4. Discussion

4.1 Workload in an inquiry-based approach

TKH : The heart of the module is three over-arching problem scenarios. Students were presented with these problems at the start of the module. As they discovered new material through research, reading and note-taking during the module, students were directed to apply their findings to the problems. They received individual feedback on their work in progress through a 'learning journal' in the virtual learning environment. One of the problems formed 50% of the final assessment for the module but the students did not know which one it would be until the final week of the semester. Students then had 48 hours in which to polish the problem on which they were to be assessed and to hand in their work. The other two problems, on which students had worked throughout the semester, were assessed via the final unseen examination, which accounted for the other 50% of the final mark for the module.

The module was structured, in particular the assessed essay which assessed one of the three problems at 48-hours-notice, to ensure that students worked at a consistent and appropriate level throughout the semester. At the start of the module, students were provided with written guidance about the module ethos, the practicalities and problems of teamwork, and the module assessment and how to address it:

"You should work on these questions throughout the semester. As you finish each topic in the module, we encourage you to make notes and add text to your answer for each of these. You may use your Learning Journal to record this work, and to get some feedback from academic staff on how to improve it." (extract from module documentation)

Students 2007 : Although the ideas underpinning this approach - to encourage us to work throughout the semester and to provide multiple opportunities for feedback during the process of addressing the problems - are sound, there was simply too much work for us to cope with. Preparing and taking notes for seminars and student-led colloquia at the same time as completing three separate problem questions, left minimal time for the submission of material to the learning journal. Likewise, the need to prepare three separate essays in response to the problems, any of which could have been assessed, but which were worth 50% of the module assessment, was excessive. This module required far more work than comparable modules with different assessment regimes. Our other modules suffered as a consequence of the effort we put into EU Law. The work for the exam, which was worth 50% of the module assessment, equated to roughly the same amount as was required for modules where the exam forms 100% of the assessment.

TKH : There was obviously a failure here on my part to communicate that preparing for the seminars, preparing the essays and preparing for the examination was essentially the same work. For each topic, the intention was that students would discover, read and assimilate substantive material, and then practise its application to the problems in colloquia groups, which fed into seminars and the essay questions, all of which were meant to reinforce and assess the same body of substantive material. The essays were not additional to the overall workload on the module; they were integral to it, and the preparation of all of them (including the two that were not handed in) was also intended to help get students ready for the examination.

Students 2015 : Every response in 2015 confirmed that the workload associated with the way that this module was structured equipped students effectively for future careers. For instance,

"Now that I work in the legal sector I don't think there was too much work in the module, although at the time I would have disagreed. The amount of work was a good training exercise for the amount of work you have as a professional."

"Types of problems we got were actually more representative of real life workloads."

Some of the 2015 responses show that the hindsight of memory had the effect of changing students' perceptions of how they had felt. We asked, based on the 2007/08 data from the three students who wrote the case study:

"Comments from students who took the module back in 2007/08 included "there was simply too much work for us to cope with"; the work was "excessive"; it was "too/unfairly stressful". What do you think about these statements now? Do you agree with them?"

One respondent in 2015 stated:

"I do not recall having such strong sentiments about it at the time. Yes, it was more work than other courses but it was also more fun and it turned out to be very useful in the long run."

Discussion and reflection : Studies have shown that student perceptions of workload are important to their motivation and engagement in learning. Various factors within the overall educational environment play roles in determining perceived workload, [26] although there is variation across disciplines.[27] Problem- and inquiry-based learning may be especially problematic in this regard due to students' lack of familiarity with expectations in relation to more 'traditional' forms of teaching and learning. [28] Indeed, a meta-analytical survey of inquiry-based learning projects across the Faculty of Arts at the University of Sheffield suggested that some students did perceive an increased workload. Periods of fieldwork, unfamiliar methods of delivery and assessments, as well as the struggle of balancing inquiry-based and other pieces of work were sometimes problematic.[29]

A significant number of students felt that the workload for this module was excessive. Comments on the free text part of the module feedback questionnaires included "there was simply too much work for us to cope with"; the work was "excessive"; it was "too stressful"; and even "unfairly stressful" in comparison with other modules. However, other evidence from the module feedback complicates the picture somewhat. On the standard module feedback form, over half of student respondents agreed or strongly agreed with the statement, "I worked throughout the module". As part of the evaluation of the PBL-approach, an additional feedback form was administered to students and of the 60 students who filled out the supplementary form, nearly 90% agreed that they had worked throughout the module, not just to prepare for end-of-term tests. However, of those 60 students who answered this question (the School does not seek this evidence on its standard feedback form), just over 50% worked at least 10 hours a week, which is the level of work that the School recommends for a module with the credit-weighting of EU Law. 43% worked only 5-9 hours a week on this module. 16% of those 60 students also reported covering less than 50% of the required reading for the module. One student recognised that their work on the module had been somewhat uneven, stating: "I wish I'd worked earlier in the module but my error I suppose and one I've learnt from."

All of this is significant because it suggests that there is a disconnection between the expectations of staff and students about the level of work required on a module and what might be deemed acceptable and unacceptable levels. If one of the main aims of the module was to encourage students to take responsibility for maintaining an appropriate level of work across the entire semester, then it was only partially successful at the time, although it should be noted that a range of studies have demonstrated that students' perceptions of their learning develops through the course of their studies. [30] In this regard, it might be more profitable to see the EU Law module as one (key?) stage in enabling law students to develop independent learning skills and, as important perhaps, dispositions. Contact hours, and other teaching provided by academic staff, for instance through a virtual learning environment, is nowhere near as important to student learning as overall student effort and student time on-task. [31]

There were some constraints on the module design, in terms of the balance that needed to be struck between 'body of knowledge' learning (required by professional accreditation bodies) and skills development. The large cohort size and relative inexperience of the students were further factors that conditioned the design of the module, while the module convener was keen to encourage students to increase the time they spent on a module that 'counts' towards their final degree result. What (and how) the students would like to be taught and what they need to be taught (from the perspective of academic staff and accrediting/professional bodies) may be two quite different things, and this may have fed into students' dissatisfaction with the workloads on the module.

At the time, the students reported that they found the PBL approach a highly stressful experience, mainly due to the workload, but also due to the different learning style required, and the different types of "scaffolding" available as part of the module design. Although the module adopted PBL elements from modules which the students had experienced in their first year, the holistic approach and the workload it entailed do seem to have been new, as the following quotation reveals:

"Initially, the module was hard as it was a break from the modes of teaching that I had experienced in my first year … It was also difficult to adapt to the method of learning where the learning was very much in our own hands."

How can the learning be maintained and the stress levels reduced to manageable proportions? There are some lessons to be learnt here. First, in a module with over 400 students it is never going to be possible to tailor the content and delivery to meet the needs, interests and existing skills and knowledge of all and this seem to be reflected in the conflicting feedback from students. For example, some students felt that they needed a certain level of support, structure and content to be delivered, but that they did not need such intensive instruction in legal skills. But this opinion was by no means universal. The module convener thus has a difficult balancing act to perform. Second, it is important to recognise that the students were not expecting to be taught in this way and reacted negatively when they were presented with a whole-scale PBL approach. Research has suggested that managing expectations can play an important role here. [32] As the module developed over time, and one cohort of students informed the next of what to expect, we can predict that student stress levels would decrease over time.

Comparisons with other modules studied contemporaneously formed part of this perception, but it was also informed by (unrealistic) notions of how many hours of independent study a student on a law degree is expected to undertake. On later reflection, the students felt quite differently. Experiences in the world of work resulted in very different views on what constitutes a "stressful" workload. The module was seen (by implication unlike other modules the students had experienced) as an excellent preparation for the future careers in the legal profession (and by implication elsewhere where workloads are similar). Capturing students' notions of satisfaction with the module at the time would miss this element of their assessment of the module as developing skills seen as valuable for future employment. Indeed, we could go so far as to say that, in retrospect, students who had reported satisfaction with other modules' contributions to the development of their employability skills in 2007/08 might have reported dissatisfaction in 2015.

4.2 Comfort and discomfort with skills-based learning

TKH : The use of the lectures was clearly the point where students and staff differed most in their evaluation and experience of the module. The module included twenty-two lectures. They were not 'standard' lectures, as experienced by students on other law modules and were not intended to form the basis of the outline of the substantive knowledge base of the module (this was given in documentary form), nor were they structured to give a linear account of this module material (also in the module outline, which included an indicative reading list). Instead, the lectures:

• introduced the broad topics for the module,

• covered recent legal developments that were not in the standard textbooks,

• focussed upon the active development of students' core legal skills through the use of practical examples of material that students were studying, especially in tackling problem scenarios,

• and allowed the class as a whole to revise topics as they were completed.

Students 2007 : We were uncomfortable with the lectures on this module. Words such as "stressful" were repeatedly found in the student feedback, both during and at the end of the module. The lectures did not provide the substantive basis that we would normally expect from contact hours. We would have preferred it if the lectures were more like the ones we received in other modules.

"I did not like the lack of content teaching in lectures. […] If lectures teach you the bare bones, seminars don't feel as daunting and I feel I would get more out of them."

"I have no idea why we're having lectures if we're not being taught anything in them ... The concept of a 'self-learning' module is ridiculous."

"I have no idea what I am supposed to know for this module. I don't feel as if I have gained anything from these lectures."

"Please go over your core points you want us to learn … It would be really helpful to have a lecture outline like Contract I."

We think the module design probably over-estimates (some/most?) students' abilities to engage with what is complex material independently and without the support that traditional lectures give. Maybe these approaches would be better for a level 3 module, when students have already understood "the basics" delivered in a more "traditional" style?

"Some of the material was really technical and as much as a student can read and learn the material it is still necessary to receive an explanation which will complete a student's understanding."

Why did you spend so much time in the lectures developing skills we already have? We know how to write essays already!

Students 2015: We now appreciate the ways in which legal knowledge and skills are deployed to solve problems in our current employment. In those contexts (solicitor, paralegal, legal officer at the International Criminal Court), problems present as open-ended and unstructured. It is an important part of being a lawyer to be able to take a set of social facts and distil them into legal problems, before proposing legal solutions.

"The University taught me how to analyse legal problems in front of me, how to apply the legal rules to the facts …"

"The skills needed to read the vast amount of information and understand the legislation were undoubtedly useful for the rest of my degree, masters degree and my professional role."

One of the other benefits was the depth of learning, in the sense that what we learned stayed with us, more than in other modules where we learned for the examination but didn't necessarily remember much afterwards:

"The module was tough as there was a lot of reading involved. It was a complex and difficult module, mostly I think because it felt … different … compared to the other traditional modules. I still remember the general principals of EU Law from the module so it must have made an impression."

Discussion and reflection :

The kinds of comments made in 2007/08 reveal that the students did not understand the planned benefits of the approach taken at the time that they studied the module. The aim was to introduce students to higher (i.e. university) level approaches to learning. The intention was never to provide a set of lecture notes, divided into topics, with a defined set of facts and concepts that can be learned and applied to the examination in response to questions that are obviously focussed on specific topics. This might be the approach experienced by many students in other (non-PBL) modules and at earlier levels of study and this clearly has its proponents and its benefits. [33]

However, the aim of the EU law module was to develop students' skills and abilities to work with legal material; to read in the subject effectively and purposefully; to use their understanding of the law and legal reasoning to solve problems; and to understand, through this experience, that, like "real life" problems, "real life" law does not come neatly packaged into "modules" or "topics". In this regard, acquisition of knowledge was seen as integrated with the development of skills, [34] through practice, not as a separate body of information to be imparted through the lectures. Indeed, one respondent in 2015 showed that she now understood that "body of knowledge" learning was not what was of most value in her University experience, explaining that the skills learned on the module are very useful in her current employment, but that:

"The specific EU knowledge has not been as helpful, but this is because my current role does not require this knowledge."

One of the aims of the module was to enhance skills that will be useful for students' future professional lives (be that in the legal profession, or in other graduate careers). The intention was that subject-specific and generic skills were developed on the module. However, only 35% of the 60 students who answered the supplementary feedback questionnaire in 2007 agreed that the module encouraged them to develop skills and approaches to learning that will be useful for their professional life, although 35% were neutral on this question, perhaps suggesting a widespread ignorance of what skills are useful in professional life.

Some scholars argue that students can best be prepared for the world of work and life after university by engaging with authentic problems and issues[35] in collaboration with their tutors,[36] thereby equipping them with the capacity for what Baxter-Magolda [37] has termed 'self authorship'. The problems through which the EU law module was assessed were similarly open-ended and the range of information that is potentially relevant for them is extremely wide. Much of the material students had encountered at first year level and on other second year modules is also of relevance. The module was therefore designed to enable students to develop their skills in navigating this information and applying it practically rather than on the recital of a more restricted range of material, skills and dispositions. Such a learning experience was designed to prove useful to professional life after university.

The module was structured so that support (scaffolding) was offered by a number of other means beyond the lectures. The support through the feedback given in seminars, the learning journal and the discussion board was targeted on individuals or small groups of students. Admittedly, this is one of the most challenging aspects of the module: how to design it so as to cater for all the students (the strongest and the weakest) in a very large cohort. Of course, no module can achieve this perfectly. In the second year in which the module ran with the IBL design a series of podcasts of more "traditional" lectures (on "the basics") were added. Students could access these in their own time. However, the podcasts were framed so as to make it clear that students did not feel that simply knowing these "basics" would be enough to do well in the assessment, which was, as already noted, an assessment not only of knowledge, but of legal analytical skills.

As for the 2007 students' perception that they "already know how to write essays", unfortunately, although many students feel that this is true and some of the strongest students are very effective at communicating their ideas in writing, most undergraduate students still have a long way to go. [38] Assessments from first year suggest that many students are only part way down the path of learning how to apply the skills of legal analysis necessary for professional practice or how to write analytical essays. This is to be expected and here the module was designed on the basis that it is the responsibility of academic staff to help students to acquire these skills. Although the substantive material for this module is explained in the textbooks, no textbook can teach students these skills; there is no substitute for repeated practice. The module was designed to make use of the available technology in order to develop these skills in lectures, using seminars to follow up and clarify any concerns or problems.

Linked to the issue of expectations discussed above, communication between the module convener and the students, between the different members of the teaching team and within the student cohort was also key to how students responded to the learning experience. Opinions on the module seem to have rapidly become polarised and this was not helpful in resolving the issues that arose as the module progressed. Students did not feel that the fact that the course is compulsory was necessarily a negative factor, because all modules in the first two years of the law degree are prescribed. Many students were enthusiastic about EU Law as a subject but reacted negatively to the way in which it was taught in this instance, feeling that the module structure was overcomplicated and did not function effectively. Importantly, there was also a sense that complaints and suggestions were not listened to sufficiently. Devising avenues for such communication and interaction - like repeatedly reminding students of the availability of the highly effective virtual discussion board, and the individual feed-forward on learning journals - helped to alleviate problems in future iterations. Providing such opportunities for communication and dialogue is an essential part of effective scaffolding of inquiry-based learning pedagogies.[39]

4.3 Collaborative learning and research

TKH : The team that designed the module was aware that the approach and, in particular the use of lecture time in 'non-standard' ways, was likely to increase student anxiety. Significant effort was therefore expended in providing students with support in their learning, especially in providing opportunities for students to receive feedback from academic staff on progress. These included:

• the seminars,

• the learning journal,

• a discussion board in the virtual learning environment,

• and problem-solving meetings between the module convener and students outside regular class time.

Because students (and most of the teaching team) did not know which problem scenario would become a component of the formative assessment for the module, staff were able to give detailed advice on work-in-progress on the problems without the risk of inadvertently "telling students the answer". The module convener monitored the learning journal entries, giving individually tailored feedback (or rather "feed forward") on student work-in-progress, and indicating for each submission what was needed to improve it to the next level. This activity was supported by the rest of the module team, especially at points in the semester when "traffic" rose substantially.

Students 2007 : We appreciate the rationale behind the learning journals. But some of us felt that there was insufficient time to engage with them on a consistent basis throughout the semester. For instance, the three students involved in the writing of this case study did not use the learning journal throughout the semester, although they did use it just before the essay deadline. This was due to the amount of time they felt had to be spent on other work for the module.

TKH : Those students who did use the learning journal consistently through the module reported in the module feedback how much they appreciated it. It was possible to see their work improving over time, which was encouraging. In later iterations of the module more was done to "sell" the learning journal to students. (See further above for discussion of module workload.)

Students 2007 : Students liked the electronic discussion board in the virtual learning environment and used it heavily, to check both factual understanding and to support their analyses. Although the entire module team monitored the discussion board, and contributed where necessary, students generally responded to each other, often with detailed directions on where in the substantive material to look for answers to the problems that had been posed. This meant that students were reliant on each other, rather than solely on members of academic staff, for checking understanding:

"It was often the only way to check whether the reading and work you were doing was relevant."

TKH : The teaching team also judged that the electronic discussion board was useful. It was a great way to reflect the fact that none of us works '9-5' any more. For example, as an academic with a young family, I appreciated being able to be 'available' to students at non-standard times, and without having to have face-to-face meetings. It was also evidence that students were developing skills of group working and peer learning.

Students 2007 : Many students felt that the perception that they were not being given enough basic knowledge, understanding and structure was heightened by the seminars. There was a feeling that there was insufficient time in seminars to discuss the material that each independent colloquia group was presenting; seminar leaders had to rush through topics and often simply tell the students the "right" answer if they were unable to work it out for themselves or had not prepared for class. In some cases this hampered understanding of specific subjects, in other cases of the course as a whole:

"Seminars were too short and based on presentations of groups that left no time for any discussion about topics."

TKH : In response to these issues, seminar times were increased from 60 minutes to 90 minutes in the second iteration of the module. We also moved one topic (which is covered in Public Law, a module that some students take at first year level and others take at the same time as EU Law) into a structured electronic workbook, which students completed in their own time and which incorporated a problem scenario at the end. Detailed online feedback was available for those students who completed the workbook. But, of course, the seminars will only work well if the colloquia groups meet beforehand and prepare effectively and more was done in subsequent iterations of the module to explain this to the students.

Students 2015 : The module was "tough" to begin with and the textbook "difficult to get on with" but also "interesting and challenging", while the seminars were "very involved and problem solving orientated". When asked to reflect on the usefulness of the skills and/or knowledge developed on the module in the world of work, students stressed the importance of the teamwork ("it was truly interactive") and research elements of the module, including the "creativity" of the assignments. One student stated that the module learning encouraged them to engage with the entire research process: "starting with choosing of the topic, conducting in-depth research and analysis and preparing the final product". The student reflected further on the useful of the PBL/IBL approach in the context of their overall studies:

"I think that in a system where there is no final thesis required, it is important to give students an opportunity to conduct proper, full-scaled research at some point of their studies".

Discussion and reflection:

The "team-learning" aspects of the module design had the effect of blurring the lines between "teachers" and "learners" on the module. Particularly through the electronic discussion board, students and staff teaching on the module became in effect a single "team", puzzling out together the answers to the problems that underpinned the module's content and its assessment. Creative solutions to the problems were celebrated, certainly among the staff on the module, who regularly shared with each other some of the excellent draft answers on which they gave feed-forward. These kinds of collaborative working were intended to simulate a post-university work environment, in the legal or other graduate profession. Comparing and contrasting the student feedback from 2007 to that from 2015 shows that appreciating and even enjoying this aspect of the learning experience was not shared by students until much later in their learning and development.

5. Lessons for broader HE contexts

The data on which we reflect in this case study obviously affect the claims we are able to support. In particular, our qualitative data is based on a very small sample size (the 2007 focus group and the 2015 respondents), and the fact that the sample is drawn from just one law school, in a pre-1992 'red-brick' university. Research in other law schools, of different types, could yield different results. Although the three students who had been involved in the original focus group were willing to give their time to reflect on the module some eight years later, none of their peers whom they had suggested did so. Attempts to contact other students in the cohort via social media (Linked-In and Facebook) also failed to yield any further data. It is nearly impossible to follow up with larger scale quantitative data, as students disperse and cannot even be contacted, still less relied upon to respond to further questionnaires about their university learning experiences. Indeed, the literature on student evaluation of teaching which claims that ratings are stable over time either reports on the same teacher with successive cohorts of students; or, occasionally, on alumni ratings typically just one year after graduation.[40] For studies like this, relying on small scale qualitative data is essentially therefore "as good as it gets" in practice. Equally, we should be cautious about presenting students' memories of an experience some eight years after that experience. In general, remembering past events or experiences tends to have the effect of creating a "positive spin" on those events or experiences. [41]

In reaching our relatively modest conclusions, we rely on the following observations. First, the students themselves felt that they were representative of their cohort. Second, there is no reason to suppose they were different from the rest of their cohort. These particular students went on to legal careers, of various sorts, or related graduate employment, as the rest of their cohort did.

The introduction of PBL into all elements of a core module with a large cohort was very ambitious and so it could be argued that this module might stand as an example of attempting to do too much too soon. A more gradual approach might have seen student satisfaction in 2007, and also later, once the gradual introduction of the "new" style of learning was complete. Yet, students taking the module in 2007 got good results, in fact the overall examination grades for the module were similar to previous versions of the module, with slightly more at the first class and fail ends of the curve.

As noted above, one possible explanation for the student feedback in 2007 is a disparity between the module convenor's [42] abilities to provide "conventional" teaching, compared to her abilities within a PBL method. We think this an unlikely explanation, for the reasons we give above and especially as student satisfaction with the teaching provided by the module convenor and the teaching team was consistently high in the years before the introduction of the new approach to learning on the module. Reviews of over 1500[43] and over 2800 [44] published studies, including longitudinal research, showed that student evaluations of a particular university teacher are reliable across courses. The more likely explanation, therefore, is that the students did not appreciate the learning style in 2007. The 2015 data suggests that they did come to appreciate it later. Our study reinforces the intuitive 'conventional wisdom' to the effect that students are not able to appreciate the long-term value of university learning experiences until they have spent several years out of education, in the world of graduate employment. [45] It suggests that the literature on reliability of student evaluations of teaching over time should be adjusted, to reflect relationships between perceived value to employability at different times in a student/graduate's life.

The ratings and student feedback on the module improved somewhat in subsequent years, as modifications in the second and subsequent iterations, such as the introduction of podcasts to provide basic subject information and lectures on current trends in EU Law, were well received by students. These modifications represent something of a compromise on the original strategy and a proactive response to student feedback, thereby providing greater support and "scaffolding" for student engagement. [46] This extra "scaffolding" was more like standard style "lectures" on the core content. Students who were less comfortable with the open-ended and student-led aspects of PBL could access this more "traditional" teaching. On reflection, a more blended, gradual approach to implanting PBL into the EU Law module may have been less traumatic and just as rewarding for the students. But of course, opportunities (and funding) to develop "innovation" are rarely available for such incremental approaches. Certainly the support offered by Sheffield's CILASS would not have been available for an incremental approach, not least because CILASS' funding was constrained to a particular, relatively short, time frame.

PBL and other "new" teaching methods often present the secret of their success as communicating with the students why they are being asked to learn in the way they are being asked to learn. [47] Certainly the module convenor and team made significant efforts to offer such explanations, and to keep multiple lines of communication open with the students. These were reinforced in subsequent iterations of the module. But our case study shows that there is an imbalance between "traditional" and "expected" ("easy") ways of teaching, where students do not perceive a requirement to explain why; and other ways of promoting learning in Higher Education, where they do. Indeed, at least in some contexts, such expectations may extend beyond students, to colleagues and line managers. If a university department were able to adopt a consistent approach to skills-based learning as "normal" learning, students might respond differently. This imbalance of expectations for explanations makes the use of "alternative" pedagogies more labour intensive.

Levels of student satisfaction expressed through feedback on the module did not recover to the levels from before PBL was introduced until the module was taken over by a new convenor who abandoned the PBL approach altogether, retaining only small aspects of the module design, such as asking the students to undertake independent group work before standard style seminars. Introducing an ambitious pedagogical approach, without hope of evidence of student appreciation of the benefits of that approach, until some eight years later, is a highly risky strategy. At an individual level, it would be a particularly precarious approach for an early career academic, whose teaching credentials were subject to probation. Students reporting dissatisfaction in feedback can delay career progression. Even if career progression is not an issue (as in this case), other barriers may arise, for example being required to account to one's Head of School for student complaints, which could reflect badly in an appraisal or even a pay review. Even though by 2015, the students really appreciate what they learned through the module, in the current Higher Education environment, incentives for designing learning in this way are few. The prevailing culture of paying attention to contemporaneous student assessment of their learning experience (through mechanisms such as module and teacher feedback; the NSS) strongly encourages academic staff to adopt teaching strategies that fall well within the comfort zones of the students on their modules,[48] and align with established practice in their institutional settings. Moreover, there are powerful institutional drivers to keeping students "satisfied" at the time of their learning, especially if metrics-based assessments of quality of teaching/student learning (such as the NSS) are taken seriously as a basis for funding decisions, as is proposed under the Teaching Excellence Framework (TEF). [49]

Disincentives to adopting PBL or similar learning approaches apply even where those designing student learning are familiar with the pedagogical literature that suggests such "alternative" strategies are better suited to adult learning in Higher Education settings, and better suited to the employability/skills agenda. More challenging modes of learning will provide more appropriate support for development of the very skills prized by future employers that mechanisms such as the TEF are supposed to improve. The employability agenda implies a distinction between (transferable) skills and (academic) 'body of knowledge'/content. But these are not really distinct in the way that is implied. Skills are learned through engagement with content. This type of learning (where skills are 'built-in' rather than 'bolt-on') is recognised as most appropriate for skills development: [50] the case study we discuss shows how difficult it is to implement that realisation in the contemporary Higher Education context.

A peer review of an earlier version of this article criticized it for expressing a "teacher knows best" view. Of course, in one sense, we reject such a view entirely - the whole point of developing an approach for this module based on IBL/PBL was precisely to embody the idea that students and academic staff are learners. The problems that the students were engaging with on the module were problems to which there were no obvious or straightforward answers - they were like "real world" legal problems. The academic staff on the module team were learning what the answers might be, as we gave feedback/feedforward on the students' work. The collaborative learning on the module is one of the themes that emerged from our analysis.

But it is true that, in another sense, we do want to suggest here that sometimes and about some matters, the teacher does know best - at least at a particular moment in time. That "better knowing" comes from the greater experience of academic staff, compared to that of the undergraduate students we teach (in general - though noting that mature undergraduate students may in fact have considerably greater experience in some things than the academic staff who teach them). It also comes from our scholarship of learning and teaching, our engagement with pedagogical literature in our discipline and beyond. We agree with the recommendation that student evaluation be used only to assess some aspects of teaching, not course design or teaching methods. [51] Our evidence for this assertion comes from the realizations of the students whose views in 2015 we report here. Of course, many academics would be able to recount numerous anecdotal experiences and exchanges that also support these conclusions. The quotation from the David Lodge novel with which we began draws on such experiences for its humorous effect. We have cited above some of the very small number of truly longitudinal studies that have also explored these questions.

Student evaluation of teaching is not new, but its meaning and significance are being reframed in the light of the forthcoming 'Teaching Excellence Framework'. The TEF sits within a suite of mechanisms pursuing governmental agendas focused on what authors such as Collini,[52] in general, and Thornton, [53] in the case of law, see as the 'marketisation', 'privatisation', or 'commodification' of Higher Education. Within this conception of university learning, students are to be developed, and equipped with skills, for the future - in particular, for future employment. The desires of students and their consequent assessments of their learning experiences are informed by their developmental and life stage, rather than by a longer view. If based on the assessments of those students, the TEF will reward short-termism. It will dis-incentivise modes of student learning that are uncomfortable in the present moment, even if they will be valued by the students concerned (and their employers) in the future. Ironically, it will therefore discourage the very commodification of Higher Education that it is supposed to support.

Appendix 1: Standard School of Law module questionnaire 2007-08

Appendix 1:

The 'standard' Sheffield Law School module feedback form in use in 2007/08:

Q1 Your gender.

a. Male

b. Female

Q2 Your level

a. 1

b. 2

c. 3

d. 4

e. Erasmus

Q3 I thought the module was interesting

a. Disagree a lot

b. Disagree a bit

c. Neutral

d. Agree a bit

e. Agree a lot

Q4 There was more work on this module than others with the same credits

a. Disagree a lot

b. Disagree a bit

c. Neutral

d. Agree a bit

e. Agree a lot

Q5 It was hard to get the materials to study for the module.

a. Disagree a lot

b. Disagree a bit

c. Neutral

d. Agree a bit

e. Agree a lot

Q6 I would have prefered another form of assessment for the module

a. Disagree a lot

b. Disagree a bit

c. Neutral

d. Agree a bit

e. Agree a lot

Q7 Lectures: are they helpful as a means of adding to your knowledge of this subject?

a. Yes

b. No

Q8 Lectures: are they helpful as a means of raising your critical awareness of the subject?

a. Yes

b. No

Q9 Tutorials /Seminars: are they helpful as a means of adding to your knowledge of this subject?

a. Yes

b. No

Q10 Tutorials / Seminars: are they helpful as a means of raising your critical awareness of the subject?

a. Yes

b. No

Q11 The MOLE E-learning page enhanced the teaching of this module

a. Disagree a Lot

b. Disagree a bit

c. Neutral

d. agree a bit

e. agree a lot

Q12 Finally make any comments you want (positive and / or negative) about the course.

Appendix 2: Supplementary questionnaire used for this module in 2007/08:

1. Study time: The amount of time I spent studying for the EU Law module was approximately

a. More than 15 hours per week

b. 10 to 15 hours per week

c. 5 to 9 hours per week

d. 2 to 4 hours per week

e. Less than 2 hours per week

2. Reading: Of the 12 chapters in the textbook that covered the material for the EU law module, I read

a. More than 90%

b. 80% to 90%

c. 65% to 79%

d. 50% to 64%

e. Less than 50%

3. Group participation: My group work participation in my colloquia (compared to others in my group) was about

a. More than 90%

b. 80% to 90%

c. 65% to 79%

d. 50% to 64%

e. Less than 50%

4. Attendance: I attended approximately ______% of the class sessions and lectures

a. More than 90%

b. 80% to 90%

c. 65% to 79%

d. 50% to 64%

e. Less than 50%

5. Format: The format of the EU Law module encouraged me to develop skills and approaches to learning that will be useful to me in my professional life.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

6. Skills emphasis: I benefited from the emphasis on reading, writing, and note-taking skills.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

7. Skills learning: I learned useful reading and writing skills on this module.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

8. Collaborative learning skills: I developed my skills as a collaborative learner on this module.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

9. Independent learning skills: This module helped me to develop my independent learning skills.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

10. Confidence: This module has helped me become more confident as a learner.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

11. Evenness of workload: I worked throughout this module, not just to prepare for end-of-term tests.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

12. Responsibility: I feel that the EU Law module encouraged me to take responsibility for my own learning.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

13. Motivating: I found the EU Law module enjoyable and motivating.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

14. Active role: I feel that the format of the EU Law module encouraged me to take an active role in the class.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

15. Learning journal: I used the learning journal on MOLE

a. Yes

b. No

16. Learning journal perception: If you used the learning journal, please answer the following: I found the learning journal a useful mechanism in providing guidance and support for my learning on the module.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

f. Did not use

17. Discussion board: I used the discussion board on MOLE.

a. Yes

b. No

18. Discussion board usefulness: If you used the discussion board, please answer the following: I found the discussion board a useful mechanism in providing guidance and support for my learning on the module.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

f. Did not use

19. Extra note-taking session: I attended the extra note-taking session.

a. Yes

b. No

20. Usefulness of note-taking session: If you attended the note-taking session, please answer the following: I found the note-taking session a useful mechanism in providing guidance and support for my learning on the module.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

f. Did not take

21. Support and guidance: I was provided the support and guidance I needed to carry out the inquiry-based tasks required in the EU Law module.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

22. Results: I am pleased with my results on the EU Law module.

a. Strongly Agree

b. Agree

c. Neutral

d. Disagree

e. Strongly Disagree

23. Suggestions or comments: Please add any suggestions or other thoughts here

Appendix 3: Focus group questions for EU Law students

Friday 16^th November 2007, CILASS 5, Information Commons

• What sorts of work are you doing outside scheduled classes?

  [Reading textbooks; taking notes]

• How much do you do, on average, each week?

• How does this work relate to the colloquia/ seminars/ lectures?

• Why do you think the course is structured in the way that it is?

  • What do you think is the point of the problem questions?

  • Do you find the interactive lectures/ 1 minute papers useful?

  • Why do you think this course involves the use of non-traditional lectures?

• How do you feel that this module relates to Understanding Law and other modules you have done in the past/ are doing now?

• Are there any things that you would like to see in the lectures that aren't in them at the moment (given the parameters that exist)?

  [The lecture schedule is flexible and what you say here can change it]

• Do you feel that the seminars/ colloquia are meeting your needs?

• What do you think about the overarching problems?

• Have any of you been to see (or email or contact via MOLE) the module or seminar leaders about any concerns that you have?