Equality Missteps in the Stern Review and the Proposals for the Second Research Excellence Framework

James Hand*


Periodic assessment of UK Higher Education Institutions' research through the Research Assessment Exercises have included consideration of equality in their operation. The successor Research Excellence Framework (REF) strengthened measures to promote equality and diversity. The Stern review of REF 2014 and subsequent proposals by the funding bodies recognise the need to promote equality and yet some of those proposals may act to increase the scope for inequality.

Keywords: Research Excellence Framework, Equality, Discrimination, Higher Education

1. Introduction

Research by UK Higher Education Institutions has been subject to periodic assessment since 1986. Every three to seven years since then has seen a new assessment exercise take place. [1] The manner and stated purpose of these exercises often subtly changes, with the approach becoming "gradually more transparent, comprehensive and systematic." [2] The stated purpose in 1992 was to produce ratings which the national funding councils would use to help allocate research grants among higher education institutions. [3] By 2001 this became the "main purpose". [4] While this aspect remains of fundamental importance, the 2014 Research Excellence Framework's primary purpose was described more broadly as "to assess the quality of research" to inform allocation of funding but also to provide public accountability for funding and to provide benchmarking information for use within the HE sector and by the public. [5] The impact of these assessments on equality was regularly reviewed from 2006 [6] and played an important part in framing the 2014 Research Excellence Framework (for example, adding "further caution to the use of citations, even at a secondary level, to inform peer review judgements"). [7] However, as the research assessment processes became more sophisticated, the burdens and cost have also arguably increased [8] and, following the 2014 research process, Lord Stern was asked to chair an independent review.

The Stern Review of the Research Excellence Framework (REF) sought to identify "ways forward for the REF that reduce distortions and burdens whilst maintaining and improving incentives for research excellence wherever it may be found". [9] Among the recommendations were: that all research active staff should be returned in the REF; that outputs should be submitted at Unit of Assessment level with a de-coupling of publications (and other outputs) from individuals but with a set average number per FTE (with flexibility for some faculty members to submit more and others less than the average); and that outputs should not be portable. While the review sought to simplify and to reduce not only the cost of the process but also "game playing" and the potential to discriminate, its recommendations - and the subsequent initial proposals from the Higher Education Funding Council for England (HEFCE) - could have the effect of increasing the risk of detriment and differential treatment.

2. Stern and HEFCE Proposals

2.1. All-in?

The requirement that all "research active" staff be included in the exercise was designed to prevent game playing, to simplify the process and to avoid potentially discriminatory selection of some researchers over others. In the sector feedback on the 2014 REF, the selection of staff was "was frequently described as burdensome, divisive, or having divisive potential, and requiring careful management". [10] If all staff were to be submitted then such burdens can clearly be seen to fall away. Part of the burden on Higher Education Institutions was the "the considerable volume of work and the often challenging internal processes involved" in undertaking "the strengthened measures to promote equality and diversity in the REF 2014". [11] While there remained under-selection across a range of characteristics in REF 2014 (which an all-in approach would on the surface address), [12] and despite the noted burden, the strengthened measures were "recognised as having an overwhelmingly positive impact" not just with regard to staff selection but also in "establishing equality and diversity as important considerations in universities' everyday activities". [13] An all-in approach removes the need for such measures with regard to selection but concomitantly removes the wider positive effect of them. This, however, need not be a problem and can be mitigated through a general focus on equality. A greater problem lies in the definition of "research active" if not all lecturing/research staff are deemed to be research active when determining who is "all-in".

The original recommendation from Stern was that "all research active staff should be returned in the REF" (in para. 64) and this was stated to mean "all academic staff who have any significant responsibility to undertake research" (as per para. 65). [14] This led to HEFCE's original proposal that Higher Education Statistics Agency returns should be used to identify the research active full-time equivalent staff and to link those staff to units of assessment. Thus those staff "returned to the HESA Staff Collection with an activity code of 'Academic professional' and an academic employment function of either 'Research only' or 'Teaching and research'" would, subject to filtering out research assistants through a measure of independence, be deemed to be research active. This would have the tremendous advantage of simplicity and inclusivity. However, as the activity codes are currently based on teaching and do not accurately reflect the research work it risks producing a misleading picture if the unit of assessment is linked by the codes at present. Furthermore, during the consultation concern has been raised that this could be highly divisive and lead to staff being moved on to teaching only contracts which could have equality implications unless such contractual game-playing was hindered through disclosure in the environment statement part of the assessment. If some other means than contractual status is used to identify staff who are research active - necessitating an element of selection - then the advantages of simplicity and inclusivity are lost.

2.2. De-coupling

The recommendation to decouple publications from individuals and to focus on the submission at collated unit of assessment level could according to Stern "reduce the current consequences for morale of non-submission… encourage cohesiveness and productivity within the submitting unit" and "reduce the burden of demonstrating individuals' special circumstances". [15] HEFCE, in its consultation document, went further and stated that the decoupling of staff and outputs "would negate the need for arrangements to account for individual staff circumstances and may contribute significantly towards deregulation in the exercise". [16] As staff and outputs would be delinked, there would be flexibility in meeting the target for the unit rather than requiring four pieces per person and this flexibility would replace the need to take account of staff circumstances. HEFCE did, however, seek views on the potential for underrepresentation of research by some groups under such a system and in an initial response the REF Manager noted "It is clear that we will need carefully to explore the potential for disadvantage arising for submitting units with different staff profiles that are required to meet a set number of outputs." [17] This careful consideration is vital as the position mooted by Stern and the consultation, relying on flexibility and removing consideration of special circumstances, is highly problematic.

It is clearly sound to hold (as Stern does) that, when determining the total number of outputs which a unit should submit, there should be a reduction in monitored output to reflect fractional staff (as indeed was the case with the last REF through taking account of individual special circumstances ). However, the move away from other special circumstances could promote discriminatory pressures. While the unit-level approach and the use of flexibility may allow a unit to focus on the highest quality outputs across the unit as a whole, the units' total will be a set amount based on full-time equivalent staff (which will depend on the definition of research active). Unless the definition is highly exclusionary (and possibly even then), the indisputable fact that the unit will be the sum of its parts means that failure to consider individual special circumstances may inherently be discriminatory. The suggestion that flexibility would negate the need to take account of individual circumstances thus contains a dangerous assumption - as it would depend on the characteristics of the institution, or unit within it, at that time: it could help; it could hide; it could damage; it could be highly discriminatory without exceptions to the general rule. It seems quite frankly surprisingly ill-thought through for the Stern Review to imply that flexibility (i.e. others submitting more) is the way that those who miss, for example, half the REF period on maternity/paternity leave must be accounted for (whereas those who are on a 0.5 FTE contract will attract a deduction in the unit's target figure). Those figures are not far-fetched and it is certainly not unforeseeable that a Unit of Assessment with a young age profile may be adversely affected through the cumulative effect of such leaves against a Unit of Assessment with an older age profile. And such adverse effect may lead to pressures, consciously or subconsciously, to discriminate.

The following example expands on the difference in treatment between part-time staff and staff who, for example, have taken parental leave or who have had time off because of a disability. In a 6 year review period someone who works 0.66FTE will have contributed the equivalent of 4 years and the missing two years will not be included in the submitting Unit of Assessment's Full Time Equivalent figure which will be used to calculate the number of papers required for the submitting Unit of Assessment. A woman who has had had two children in that 6 year period may equally have two missing years but those two years will nonetheless be included in the target. Likewise someone who has taken a significant period of sick leave (potentially related to a disability) will none the less count fully in calculating the Unit of Assessment's target. There is a thus an incentive for the submitting Unit of Assessment to prefer to have fractional members of staff (or full-time staff) rather than those likely to take parental leave or sick leave. While the individual may benefit from flexibility, they have contributed to a higher requirement which means that they may see themselves (and others may see them) as a burden for which others have to compensate. Those full-time employees robust in health and unlikely to take familial leave will be particularly prized.

Even if the submitting Unit of Assessment fought against this encouragement to discriminate, the different demographic make-up of some units may see some units having to rely on flexibility more than others and in some units there may not be sufficient slack available to provide the flexibility (if they are, for example, populated particularly by those more likely to need to take parental leave or have a propensity to sick leave). [18] In such a case, one unit is being subjected to detriment because of one or more protected characteristics when compared to another which has the slack or less need for flexibility. While the treatment of units of assessment would not fall within the individual protections under the Equality Act 2010 (unlike detriment to the individual researchers), it would risk contravening the public sector equality duty in section 149(1) of the Act which holds that public authorities must, when exercising functions "have due regard to the need to (a) eliminate discrimination, harassment, victimisation and any other conduct that is prohibited by or under this Act; (b) advance equality of opportunity between persons who share a relevant protected characteristic and persons who do not share it; [and] (c) foster good relations between persons who share a relevant protected characteristic and persons who do not share it."

A solution would be to allow the Full Time Equivalent figure to take account of not only such things as part-time status (which is proposed) but also time lost through maternity leave or illness (which is not proposed but could be a similar amount of time). Allowing submitting Unit of Assessments to thus amend the Full Time Equivalent figure (measured over the census period) and briefly explain why need not be burdensome on the institutions and would remove this differential treatment. While this would add a level of complexity it need not be great as the information should already be available and verifiable and it is required to prevent detriment occurring to, for example, the pregnant and disabled. Furthermore, the concept of changing the target figure has been mooted re filtering out non-independent researchers so it is not unprecedented. [19]

2.3. Non-portability

The recommendation that outputs should not be "portable" could also have discriminatory effect but this may be more justifiable. Previously, academics could take their papers with them when they left an institution. This meant that institutions - particularly rich ones - could buy in researchers toward the end of the assessment period to boost their returns, with such opportunities for "buying" institutions and individual academics leading to a "demonstrable increase in the number of individuals being recruited from other institutions shortly before the census date". [20]  The expansion of the concept of "non-portability" (which previously applied to assessments of impact arising from research) to outputs is intended to ensure that the credit for the work goes to the institution where the output was (as far as can reasonably be determined) demonstrably generated and to reward long-term focus on investment and support and to discourage short-termism and game-playing. There is a clear legitimate aim in linking outputs with the producing institution but some argue that it does disproportionate harm to academics with particular characteristics. All individual academics lose out in not having a clutch of golden eggs they can bring with them to a new institution and it is thus not surprising that, in a synthesis of responses to the call for evidence ahead of the Stern report, while institutions offered "moderate support" to the principle of non-portability it attracted "no support" from individuals. [21] However, while all lose out, those on fixed-term contracts are particularly affected by any such move and it is argued that this could give rise to indirect discrimination.

The Equality Challenge Unit, in their response to the Stern consultation, noted that "81% of academic staff under the age of 25, 71% between the ages of 26-30 and 50.5% between the ages of 31-35 are on fixed term contracts". [22] It is thus clear that people of younger age groups would be put at a particular disadvantage by the practice of non-portability. The position is, however, less clear with regard to the other protected characteristics. While the proportion of women who have fixed-term contracts has, according to Higher Education Statistics Authority figures, fallen substantially from 51% in 2003/04 to 36.6% in 2015/16 (and the disparity with men narrowing from 10.4% to 4%), 43.7% of open-ended contracts were held by women in 2014/15 and while the trend is narrowing there it may take some time to reach equality. The ECU are more cautious still with regard to Race commenting that there may be a negative impact on BME staff. In all such cases, it is possible that non-portability could potentially be justified as being a proportionate means of meeting the legitimate aim of assessing institutions by the work produced there rather than bought in. While the underlying situation needs to be investigated and addressed, Stern's blatant (and possibly inadvertent) incentivisation to discriminate through the definition of FTE is arguably a greater concern.

3. Conclusion

Compared to past exercises, the REF 2014 included strengthened measures to promote equality and diversity and the consultation on the second Research Excellence Framework included expressly listed equality among the three principles by which a future exercise will continue to be governed, stating that the exercise would promote "equality and diversity in all aspects of the assessment". [23] The proposals put forth, however, can be seen to undermine equality to varying extents. These may often be mitigated through other actions and - simplicity has many virtues - and it would be wrong to lay discrimination revealed but not caused by the REF at its door. However, the idea that flexibility within a unit can compensate for special circumstances of the individuals who contribute outputs within the unit could act to not only hide but exacerbate inequality.

*Principal Lecturer, Portsmouth Law School, University of Portsmouth

[1] In 1989, 1992, 1996, 2001, 2008, 2014 with the next scheduled for 2021.

[3] Universities Funding Council, Research Assessment Exercise 1992 Circular 5/9, at [5].

[4] http://www.rae.ac.uk/2001/AboutUs/ ; see also HEFCE (2001) "A Guide to the 2001 Research Assessment Exercise" p.2 ("The purpose of the RAE is not just to enable funding to be allocated selectively…").

[6] HEFCE, Selection of staff for inclusion in RAE2001 (Issues Paper Aug 2006/32) available at < http://webarchive.nationalarchives.gov.uk/20100202100434/http://www.hefce.ac.uk/pubs/hefce/2006/06_32/ >; HEFCE, Selection of staff for inclusion in RAE2008 (Issues Paper Sept 2009/34) available at <http://webarchive.nationalarchives.gov.uk/20100202100434/http://www.hefce.ac.uk/media/hefce1/pubs/hefce/2009/0934/09_34.pdf>.

[7] HEFCE, Research Excellence Framework 2014: Manager's report. (Bristol 2015) available at www.ref.ac.uk/media/ref/content/pub/REF_managers_report.pdf .

[8] Cost estimates are disputed but the Stern Review stated estimated costs had risen from £66 million for 2008 to £246 million for 2014 [4]. This contrasts with 0.8% of the amount allocated in 2001 (House of Commons Science & Technology Committee Second Report 2002) which would amount to around £80 of the sum awarded). The number of outputs hovers around the 200,000 mark. (E.g. 191,150 in 2014, 215,507 in 2008 (in HEFCE (2015) at [185], see note 7 above) and around 200,000 in 1996 (figures drawn from raw RAE data).

[9] UK Department for Business, Energy and Industrial Strategy, Building on Success and Learning from Experience: An Independent Review of the Research Excellence Framework (July 2016) ("Stern Review") available at <https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf

[10] HEFCE, HEFCW, SFC and DELNI. Evaluating the 2014 REF: Feedback from Participating Institutions . (Bristol 2015), at [40] available at < http://www.hefce.ac.uk/media/HEFCE,2014/Content/Research/Review,of,REF/Sector%20feedback%20on%20the%20REF%20(cleared).pdf >.

[11] HEFCE, Selection of staff for inclusion in the REF 2014 ( (Issues Paper Aug 2015/17), at [122] available at < http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/2015/201517/HEFCE2015_17.pdf >.

[12] Ibid, at [124].

[13] Ibid, at [19].

[14] Stern Review, above at note 9.

[15] Stern Review, above at note 9, at [68].

[16] HEFCE, HEFCW, SFC and DELNI,6 Consultation on the second Research Excellence Framework (December 2016/36) at [48] available at < http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/2016/201636/HEFCE2016_36.pdf >.

[17] K. Hackett, "A moment for REFlection" (21 March 2017) http://blog.hefce.ac.uk/2017/03/21/a-moment-for-reflection

[18] A crude look at the at cost centre figures for 2015/16 shows that women make up 53.5% of Medicine, dentistry & health but only 30.4% of the Biological, mathematical & physical sciences (< https://www.hesa.ac.uk/file/6660/download?token=pHMOuUlS > ) and the Unit of Assessments could be even more stark and be affected by institutional factors such as the individual age profile of the departments. Furthermore, during the consultation process it has been suggested that a combination of FTE and Headcount be used which would disadvantage units of assessment with a higher proportion of part-time staff (see e.g. http://wonkhe.com/blogs/equality-and-the-ref/)

[19] HEFCE, HEFCW, SFC and DELNI (2016) Consultation on the second Research Excellence Framework at [43], see note 16 above. Furthermore, the FTE figure could also present a realistic figure (particularly if the eligible-staff-over-a-set-period approach is adopted rather than the census date approach at [47]) regarding staff who join which could solve the ECR problem. If someone joins halfway through the set period it would be sensible to say that they only need to contribute half the outputs and should be counted as such in the FTE figure (rather than relying on other staff to make up for them as Stern suggests).

[20] Stern Review, above at note 9, at [72].

[21] Technopolis, Synthesis of responses submitted to the REF Review Call for Evidence and follow-up interviews , (Brighton 2016) 47, available at https://www.gov.uk/government/publications/research-excellence-framework-review-summary-of-views. Across the 21 options surveyed, individuals were generally the least supportive offering moderate support to only three items (as well as very strong support to the statement that REF has negative consequences).

[22] Equality Challenge Unit, Consultation Response to the Consultation on the second Research Excellence Framework , (London 2017) 9, available at < http://www.ecu.ac.uk/wp-content/uploads/2017/03/ECU-consultation-response-to-the-second-research-excellence-framework-March-2017.pdf >.

[23] HEFCE, HEFCW, SFC and DELNI, Consultation on the second Research Excellence Framework at [12], see note 16 above (the other two principles being equity and transparency).