A Note on Using Vignettes in Socio-Legal Research
Queen’s University Belfast
© 2013 Philip Leith
First published in the Web Journal of Current Legal Issues
Citation: Leith, ‘A Note on Using Vignettes in Socio-Legal Research’, (2013) 19(3) Web JCLI
I report on the use of ‘vignettes’ in an on-line survey where the vignettes (or scenarios) were used to ascertain the beliefs of lawyers regarding the nature of merit. The project was carried out for the Northern Ireland Judicial Appointments Commission (NIJAC) and attempted – at low cost to NIJAC – to reflect the gender differences in attitudes to applying for appointment to Northern Ireland judicial posts. The results of that project appear elsewhere, but in this paper I suggest that vignette based research may be a suitable vehicle for other socio-legal projects since they allow an element of qualitative research from a technique which is usually more directly quantitative.
The use of vignettes in this research was somewhat accidental in that no prior theoretical consideration had been given to their advantages or disadvantages, and they arose more from practical need rather than pre-considered reason. However, having utilised these I suggest that socio-legal researchers might use and consider these as part of other research projects, and that they require some further discussion as to their particular utility within the socio-legal research field.1.The Problem
3.Survey Construction and Plausibility
4.What did we find?
A prior project for NIJAC (1) had carried out a series of interviews and focus groups, requiring quite a significant expenditure of time and effort by the author, and also utilised input from other members of the project team. A second project (2) which would attempt to tease out some of the issues raised by the first project was suggested by NIJAC and we were asked to respond to the tender document. The tender requirements were rather optimistic for the small amount of funding on offer:
“3.1 The overall aim of the research will be
- to re-visit earlier findings regarding the real and perceived obstacles and difficulties experienced, in applying for, and securing judicial appointment, and in particular by women
- in line with the Commission’s statutory responsibilities to establish if NIJAC’s strategies for increasing diversity in judicial appointments in attracting, recruiting and appointing applicants have led to positive change
- to explore the concept of a judicial career
- to re-examine the extent to which, women in particular, consider applying for judicial appointment and the subjective and objective reasons for these career decisions
- to determine a range of initiatives e.g. flexible working, mentoring, and assess their perceived value.”
To fully cover these topics inputting the same level of time and effort (through one-to-one interviews for example) as we had utilised in the first project would have been too substantial project for the funding available, and therefore this approach was not considered suitable by NIJAC in their tender document – rather a survey and some post-survey face to face interviews were required. The team felt obliged for a number of reasons to attempt to try to meet these research aims. These reasons included a colleague who was a Commissioner with NIJAC, (3) a sense that academic research is often viewed as problematic by external agencies and this offered an opportunity to undermine that negativity, but also a feeling that the Law School at Queen’s should use its research skills to support the important aim of equality in the judicial appointment process. Members of the research team had also related interests (such as the legal profession, gender, the judicial task, etc.).
A first meeting with the steering group from NIJAC (this comprised Commissioners – judges and others – and NIJAC staff) indicated they were interested in achieving very expansive results from the survey, and that using traditional survey techniques may not have been an effective manner in which to try to answer their research questions. In the discussion it struck me that a technique I had used in an earlier project might be utilised as the basis of the survey.
As the legal expert in a project investigating whether and how ‘youth views’ on new technology and the problems for young people which arise from social networking etc. might be fed into policy making at the European Parliament, I had created a number of small scenarios which could be used as the basis for discussion by the young people. (4) These were essentially simplified instances of existing legal problems from Information Technology Law which would be used as starting points for discussion. Thus, for something to start discussion on online child pornography the scenario placed the young person close to the accused perpetrator rather than someone ‘other’:
You believe your Dad when he says he wasn't looking at child pornography. Someone - he says - stole his credit card and used it with one of those sites. Unfortunately, someone found out about the police inquiry. Your mates at school have been told not to have anything to do with your family. What can you do?
On copyright infringement:
Your younger brother was told not to continue downloading after your Dad was contacted by the Copyright Unit of the local police. He didn't. Your internet connection has now been disconnected, all domains linked to your address have been shut down, and your address is on a blacklist. You need an internet connection to sell your own music - your songs are doing well and you might break through. But how can you sell if you can't get access to your website?
You have finished off your education and begin to look for a career. You apply for a management traineeship with TopStore. Your application has your email address. TopStore do a Google search of your email address and come across photographs you posted of you in embarrassing positions at drunken parties. You don't get the job. Unfortunately, TopStore are members of a group who share information on potential employees - all the places you want to work in are members. What could you do?
These scenarios had been quickly produced and only slightly modified by the project team, but the feedback from those who used them with groups indicated that they had a reality to the groups which I had not expected – many of the young people involved in the discussions appeared ‘to know someone’ who had experienced similar situations. The power of these scenarios appeared to be that they linked in with respondents’ experience and gave a storyline which the discussants could pick up and discuss – they were not being asked specific questions set in a structured interview (or survey), rather they were being prompted to voice their own beliefs and attitudes rather in the manner of open-ended qualitative interviewing and that these beliefs and attitudes concerned normative and strategic choices which would be in line with live experiences. Of course, the technique of setting up small narratives is hardly new – we frequently do this in teaching or in exam setting, but socio-legal researchers do not usually use this technique in research matters. It was the utility of this in the Huwy project which suggested itself as a technique which might be utilised in an online survey for NIJAC and which popped into my mind during the meeting with the steering group. I suggested to my team colleagues:
“My idea of using vignettes as the core of a survey has been growing on me. The usual problem with surveys is that they are just after quantitative data, but I think that if we have a smallish number of well-considered 'problems' around which the questions were set then we can get a combination of short reply but also ask them 'What would you do' and 'How should this be better organised' kind of questions - so that they are expected to put a sentence or two into their replies. We could ask different questions according to who they are (in terms of solicitor, etc.). Encouraging that might get us quite good qualitative data easily, which could then be used for focus groups. It would also be an example of a 'bottom up' approach where we are going beyond the typical yes/no.”
The eventual way in which we actually used vignettes differed from this early suggestion, but the basic philosophy remained the same: we would try to get attitudes to merit from the survey by creating individuals with ‘back stories’ which were relevant to how they might be assessed for a judicial post.
A common worry of those who fund research is its methodology, and although they may not fully understand the methodological aspects of research, they are concerned that the funded research will not be attacked on a methodological front. Of course, having a solid methodology does not in reality protect one from such attack, since the preferred approach of those who dislike the results is usually to suggest the methodology is flawed (it may be, of course). To persuade NIJAC that the methodology was sound, I looked for prior use in socio-legal research but searching for the use of vignettes in law research did not turn up examples. However, it did become clear that it was a technique which was used in other social science fields as well in fields such as medicine. In medicine it was used to develop skills of clinical assessment in, for example, teaching young doctors or in through career updating, but in social sciences the technique was perceived as being particularly relevant when context and moral judgments are to be assessed. Since our goal was to assess the notion of merit in context, the technique seemed appropriate. Barter and Renold suggest:
"[V]ignettes generally fulfil three main purposes:
- interpretation of actions and occurrences that allows situational context to be explored and influential variables to be elucidated;
- clarification of individual judgements, often in relation to moral dilemmas;
- discussion of sensitive experiences in comparison with the ‘normality’ of the vignette.” (5)
A problem with the approach, as perceived from the perspective of social scientists, is that of the distinction between belief and action – that is, that a respondent may believe that they would act in a certain manner to the scenario, but in reality their action would not match that belief. (6) In our research, we were only interested in attitudes towards whether merit was being rewarded so the particular problem of actual implementation of that attitude did not appear too relevant. After all, our respondents would not be in the position of NIJAC Commissioners having to undertake the action of appointing a judicial applicant. From the literature, it can be seen that this vignette technique can be used at the basic level of providing the starting point for an interview of focus group, but also offers more complex usage. For example, Jenkins et. al, describe the use of vignettes which evolve over stages in the interaction with respondents (longitudinal research). There are problems which arise from such 'developmental' vignettes which are discussed (p185) by Jenkins et. al. such as when the respondent becomes more knowledgeable during the research period and changes their attitude arising from that greater knowledge, but these were not relevant to our proposed usage since the project would only utilise static or fixed interaction. Another technique is to use ‘anchoring vignettes’ when there is social or other diversity of the respondents which might affect the responses to a survey. (7) Here the vignettes allow the researchers to test the attitudinal context of each respondent and thus to statistically modify the responses to take these attitudinal aspects into account.
There are various examples of vignettes being used in online surveys in non-legal fields so for the NIJAc project this was not to be a novel conjunction of techniques. Also, having used both paper based and online surveys in various research projects over the years, I was aware of the differences in each, and it did not appear to be problematic to use an online delivery method here.
There are various definitions of what a vignette actually is, but all note the essential qualities of being plausible, short, and concrete and which elicit perceptions from respondents. There is no need to provide any further definitional description since they constitute a relatively simple construct.
Having decided to use vignettes, the next problem was what kinds of responses could be elicited which would be meaningful and what kind of vignette could encourage those responses? Since our goal was to get attitudes towards merit and whether merit was being rewarded in judicial appointment with particular regard to women applicants, I drew up a number of draft vignettes around individuals with certain qualities, considering applying for a particular position and decided to ask whether for each of these individuals, the respondent felt that they should be/would be rewarded under the current system. Since respondents would provide information on gender and professional background, this would give us the information we required. The question relating to respondent had originally been set to appear as the last question, but was moved to being first – this ensured that those who ‘fell off’ the survey had provided some information (and we used this to indicate who hadn’t answered some of/any of the following questions).
Each scenario attempts to unlock a number of key themes. For example, Helen Black (below) comprised a number of issues: gender, a local NI solicitor but who might not be known widely in Belfast where most 'law happens' and thus might find suitable judicial referees difficult to locate, interested in substantive law (as a lecturer), and working part-time. Two questions were asked and the respondents could answer either or both with Agree/Disagree/Don’t Know:
Helen - in an ideal world - would be successful in achieving a judicial career
Helen - in Northern Ireland today - would be successful in achieving a judicial career
These initial questions were set to test the match between whether the respondent felt that the individual was a good candidate ‘in an ideal world’ or whether there was scepticism about the candidate being successful in Northern Ireland under the NIJAC regime. ‘Merit’ is a subject which is easy to talk about in the abstract but difficult to pin down in any detail, but these scenarios allowed us a way to try to unpick what respondents felt comprised ‘merit’ in terms of judicial posts. This is not obvious: the oft made comparison with European judicial posts shows a different kind of view of merit – the best law students are recruited for judicial school and then, depending upon performance in lower roles, they move up the system. But even in Europe, the vast majority of senior judiciary are male. Since we know that representation is lower, we wanted to know whether there was a perception that women with a very good CV were being disadvantaged at the upper levels of the judicial scale rather than a perception that they were not interested in judicial office.
We invited respondents to optionally provide textual information as to why they answered as they did. The kinds of responses we received were interesting and covered a variety of views, which can be linked to the professional role of the respondent. For example, three differing views on Helen were:
“I think Helen's first hurdle would be her age. 36 is quite young. I do not think that the fact she is based in Derry would necessarily prejudice her chances but unfortunately that she is now part time due to her having children might disadvantage her. If so I think this is very wrong. She seems very well rounded in teaching business/tax law, and working in family and criminal circles. For the reasons above I do not know if she would be successful in an ideal world and my main reason for saying this is her age. Perhaps when she is a little older she would be deemed to be more suitable. For this reason I do not agree that she would be successful in achieving a judicial career in Northern Ireland today.”
“As a working mother I feel that Helen would face an obstacle in achieving judicial office especially in light of the fact that she would wish to undertake this work on a part time basis.”
“But it’s not an ideal world. Helen’s appointment would depend on her written test results? IF shortlisted then on interview ... apparently. Technique in answering questions, the attitude of markers and a confident approach to role play could see her through. She would be as good a candidate as any one else ... if the equality regime and expressions on judicial diversity from NIJAC are accepted. Her sex will not be an disadvantage, probably an advantage ... but a male respondent would say that ????”
This vignette provided 121 textual responses – some longer, some shorter – so is an indication of the quantity of material which can be gleaned from such an approach. Clearly, it is not as effective as interview techniques but it is an efficient way which enabled us to provide a ‘snapshot’ of professional perceptions of the NI judicial appointments process within the limitations of project funding and time.
A further advantage of this technique is that it is relatively simple to anonymize a form of qualitative data and use it in other research which, perhaps, might compare the NI views with those of the rest of the UK.
What can be seen from the existing literature is that plausibility is an important element in the vignette – they must be believable in order to draw out the required interaction from the respondent. To ensure that the vignettes were plausible, a number of draft ideas were given to the NIJAC steering group for comment. (8) There were some minor comments about the vignettes but also suggestions for ones which might be useful. One was dropped completely (the Erasmus student). One vignette which involved the Technology and Construction Court in London was sent to a barrister colleague who regularly appeared there and who suggested corrections, but it was not possible to produce a totally accurate vignette and keep the simplicity of the original. However, since most of our respondents would not have expertise in appearing before that Court, we felt a slightly incorrect vignette would still appear plausible.
The final vignettes are as follows, with reasons why we chose each of these. Helen Black is an example of a potential female drop-out from the profession during their mid to late 30s – a class which is well recognised and with the growing number of women entering the profession, may increase in size.. Further, Helen is based away from Belfast where most of the legal interactions – social and professional occur. Even more, she has children. We asked our respondents whether she should leave the profession and do something else or return.
Helen Black, 36, is a solicitor based in Derry for the last 12 years. Over the past four years she reduced her workload to have children, and is now part-time. She has also taught law part-time at the University of Northern Ireland for the business department. Her work has been primarily family law though some criminal work has always been part of her career, but her intellectual interests (and teaching) have been tax related. The idea of a judicial career appeals, but she is unsure how her skills could be used, or whether she should return to full-time to practice.
Malachy is an example of the barrister who does not succeed in private practice, leaves and becomes successful with his legal expertise in a different field. Is his expertise and success transferable to the bench? Clearly he does not follow the model of the bar to bench, and we wondered what view of his merits our respondents might have.
Malachy Gray qualified as a barrister in 1993. After six years in private practice in Belfast he decided that he preferred a regular income as a civil servant in the legal department of the Department of Organic Farming where he has risen to a senior level with a staff of 30. Much of his work has been negotiation with the European Commission, but he has also been responsible for all litigation involving the department, though he almost never appears in court himself. Malachy has a disabled son and this encouraged him to sit on the Special Educational Needs and Disability Tribunal, a role which he has undertaken for three years and which he has enjoyed. Salary levels of a District Judge (Magistrates Court) are attractive (civil service pay has fallen behind and looks set to remain low) and Malachy wonders how he could best prepare his CV for a possible future role as a District Judge.
In our first project we discovered a number of individuals, usually women, who had created a career comprising a number of part-time roles using their legal expertise. Sally Cobolt was the example we gave our respondents – was this the kind of person they felt should be in judicial roles in Northern Ireland? Could concurrently working in practice and on a fee-paid judicial basis be practicable?
Sally Cobolt, 34, has been a solicitor in private practice since qualification, apart from a short period of three years to look after her twins immediately after pregnancy. Her expertise was employment law and – just before pregnancy – she had been appointed as a fee paid chair of the Industrial Tribunals. During her period away from practice she continued to sit on the Industrial Tribunals. She is now back in harness and not sure where she wants to be in 10 years’ time. Her ideal would be two days in practice working in her area of expertise (rather than on anything which came through the door), with a few fee paid judicial posts providing three or four days’ work per month. She feels comfortable with applying for judicial posts as they become available, rather than fixing her sights on any particular role. Alternatively, she might just look for something law related but part-time.
In Northern Ireland the most problematic issue over appointments has been that of the High Court, which currently is without a female judge. The vignettes which were constructed to test attitudes towards candidates who were not typical of those who have been appointed in the past (that is, a senior male figure from the bar) and comprised a female candidate who had been working in London as a specialist solicitor/advocate, (9) a district judge with good judicial skills who wished to move up the judicial ladder, and a young male high flier who didn’t want to wait around before he moved onto the High Court bench.
Jane Brown QC, 43, wishes to return to Northern Ireland (where she first practised) from London. She has had a successful career in London with a large firm as a solicitor advocate, but has most recently been operating as a sole specialist advisor and advocate in the field of construction law. During her period in London she remained a solicitor in NI though her caseload was infrequent. She has also acted as arbiter in high value commercial property disputes. She has, on a number of occasions, sat as a recorder hearing Technology and Construction Court cases in the Central London County Court. She has written a well-regarded text on commercial litigation in Europe. Her interest is in a judicial position, and she views herself as having the necessary skills to sit on the High Court in Belfast.
Ingrid Rose, 57, has been a District Judge for nine years after a career at the private bar. By common consent she has been very successful and is well regarded. Her caseload in the County Court has been efficiently run and covered everything from mundane family matters to reasonably complex matters of property rights. Several of her written judgments (such as one on intestacy and gay partners) have – though unreported – been discussed in the legal literature. In a number of appeals on points of law her reasoning has been supported by the Court of Appeal. Her ideal judicial post would be on the bench of the High Court.
Roger Blue, 44, has practised at the bar in Northern Ireland for 19 years. He has been building a very successful advocacy practice in civil matters. One legal directory suggests that “he has a brilliant mind and is able to put a winning argument over to both judges and juries in even the least likely to win cases.” He is on the Treasury Counsel panel and has handled complex litigation well. He thinks he might aim for a judicial post in the High Court. He doesn’t want to wait about and wonders whether applying earlier (without yet being QC) would be successful or affect any later chances through being seen as ‘too forward’.
We were also interested in a comparison between each of these candidates, and asked our respondents who should be successful and who was most likely to be successful – that is, which of the candidates were meritorious to the respondent and which they felt was likely to be meritorious to the appointments panel.
The survey which was constructed from these vignettes was put online and we advertised within the legal profession of Northern Ireland that we were seeking responses. We had 212 respondents (not all completed all questions) which was a sufficient response rate of around 7% of the total number of solicitors and barristers in Northern Ireland. (10)
The vignettes were also used as the starting point for a number of focus groups which were arranged after the survey had been collated. The team members who carried out these focus group meetings found them useful as starting points for discussion, but they were not utilised in the same manner as with the survey. We briefly discuss this below.
The project findings are available elsewhere, but it is noteworthy that the technique produced clear findings: most women were sceptical that those they considered meritorious would be considered so by the appointments panel. We also found clear evidence that solicitors were sceptical that anyone from a non-traditional background would be successful in the process. Take, for example, the findings relating to Jane Brown where 70% of solicitors saw her as meritorious and only 35% thought she would succeed:
With respect to gender attitudes, we see a difference between male and female responses, where only 58% saw her as meritorious, with 38% seeing her as being potentially successful in the appointments process:
With the female respondents, 74% saw Jane as meritorious, with 33% seeing her as possibly succeeding in her ambitions.
A difference between survey and interview methods was, of course, that rather than being able to undertake a small number of interviews per week and then carry out transcription and analysis of the recordings, as the researcher I did not need to leave my office, and the online survey system enabled advanced processing of responses.
The idea of utilising this vignette/survey technique was entirely accidental and appealed because it met the practical need of attempting to collect attitudes towards merit in a reasonably complex environment in as low cost a manner as possible. Further research demonstrated that it was a technique which had a history of at least four decades in other fields, (11) and had been reasonably well used in the past two decades. It clearly should not be viewed as a panacea for any research methods problem, but it does appear to be a technique which is worthy of further investigation by the socio-legal research field. The kinds of problems which have arisen in other areas such as belief/action are likely to be more relevant in other areas than in the research outlined here, and also if the more complex use of unfolding vignettes is considered, then these may have relevance, too. The well-developed use of vignettes to assess different responses from different social and other groups may be a helpful statistical technique in extracting more information from surveys than is usually possible.
What is particularly interesting about the technique and which requires further discussion is that it enables the collection of more qualitative data from a technique which usually produces more quantitative information.
There are a host of methodological issues which require consideration - what kinds of questions should be attached to the vignettes; what is the balance between qualitative and quantitative responses which are aimed at; how does it compare with the other tools in the toolbox of the socio-legal researcher, etc. – but overall, the technique as used in this project was relatively simple to put into effect and was reasonably efficient at producing clear results in a complex environment.
Originally proposed examples:
- Jane Brown QC, 43, wishes to return to Northern Ireland (where she first practised) from London. She has had a successful career in London with a large firm as a solicitor advocate, but has recently been operating as a sole specialist advisor and advocate in the field of planning law. During her period in London she remained a solicitor in NI though her caseload was infrequent. She has also acted as arbiter in commercial property disputes. She has, on a small number of occasions, sat as a judge in the Technology and Construction Court in London. She has written a well regarded text on commercial litigation in Europe. Her interest is in a judicial position, and she views herself as having the necessary skills to sit on the High Court in Belfast.
- Helen Black, 34, is a solicitor based in Derry for last 10 years. Over the past four years she reduced her workload to have children, and is now part-time. She has also taught law part-time at the University of Northern Ireland for the business department. Her work has been primarily family law though some criminal work has always been part of her career, but her intellectual interests (and teaching) have been tax related. The idea of a judicial career appeals, but she is unsure how her skills could be used, or whether she should return to practice.
- George Gray qualified as a barrister in 1993. After six years in private practice in Belfast he decided that he preferred a regular income as a civil servant in the legal department of the Department of Organic Farming where he has risen to a senior level with a staff of 30. Much of his work has been negotiation with the European Commission, but he has also been responsible for all litigation involving the department, though he almost never appears in court himself. George has a disabled son and this encouraged him to sit on the Special Educational Needs and Disability Tribunal, a role which he has undertaken for three years and which he has enjoyed. Salary levels of a District Judge (Magistrates Court) are attractive (civil service pay has fallen behind and looks set to remain low) and George wonders how he could best prepare his CV for a possible future role as a District Judge.
- Roger Blue, 33, has practised at the bar in Northern Ireland for 10 years. He has been building a very successful practice in criminal law. One legal directory suggests that “he has a brilliant mind and is able to put a winning argument over to both judges and juries in even the least likely to win cases.” He thinks he might aim for a judicial post in the High Court. He doesn’t want to wait about and wonders whether applying earlier (without being a QC) would be successful or affect any later chances through being seen as ‘too forward’. Application to the High Court formally requires 10 year in NI practice, which Roger has.
- Sally Mustard, 21, has just finished her law degree at the University of NI. She is the year’s best student and won several prizes. She has always – since her earliest days – wanted to be a judge. On an Erasmus exchange programme she discovered that judges on the continent are recruited (to judicial school) from the yearly corpus of top law students. She thinks this is a much better way to do it than in the UK: recruiting successful barristers to the High Court just means that older men from middle class backgrounds will get the best judicial posts.
(1) Propensity to Apply for Judicial Office under the new Northern Ireland Judicial Appointments System: A qualitative study for the Northern Ireland Judicial Appointments Commission October. / Leith, P; Lynch, M; Glennon, L; Dickson, B; Wheeler, S. School of Law, Queen's University, Belfast, 2008. 113 p . Available at http://www.nijac.gov.uk/index/what-we-do/publications/qub_research__full_version__october_2008.pdf
(2) Leith, P & Morison, J, Rewarding Merit in Judicial Appointments?. School of Law, Queen's University, Belfast, 2013. .http://www.nijac.gov.uk/index/what-we-do/publications/qub_final_report_merit_2013.pdf
(3) John Morison, who after his period as a Commissioner ended, joined the research team and acted as co-author for the second report. Other team members were Brice Dickson and Sally Wheeler.
(5) Barter C & Renold C., The Use of Vignettes in Qualitative Research. Social Research Update 25, University of Surrey. Available at http://sru.soc.surrey.ac.uk/SRU25.html
(6) See Jenkins, N et. al. Putting it in context: the use of vignettes in qualitative interviewing, Qualitative Research, 10(2), 175-198, 2010.
(7) Hopkins, Daniel, and Gary King. 2010. Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability. Public Opinion Quarterly: 1-22."After a standard self-assessment question, the survey respondent learns about hypothetical individuals through brief vignettes and is asked to place those individuals on the same response scale. Variation in vignette responses across individuals reveals interpersonal incomparability and enables researchers to use one of several statistical techniques to rescale the respondent's own self-assessment.”
(8) These first examples appear below in the appendix for comparison. The final versions are set out below.
(9) This in itself might be problematic since a solicitor who had been appointed a QC in Northern Ireland had been unable to practice at that level due to opposition from the bar.
(10) There are around 2300 solicitors currently practising in Northern Ireland and some 550 barristers in private practice.
(11) Indeed, it might be seen that it follows the technique of Garfinkel in his investigation of jury behaviour.