Measuring the impact, quality and effectiveness of legal assistance services in a climate of reduced funding and increased government expectations: the Australian experience

Liz Sara Curran; Andrew Crockett

Measuring the impact, quality and effectiveness of legal assistance services in a climate of reduced funding and increased government expectations: the Australian experience

Liz Curran* and Andrew Crockett**

*Senior Lecturer, ANU Legal Workshop (Australia-wide Professional Legal Training (PLT). Assoc. Director, International Centre for the Profession, Education & Regulation in Law (PEARL)

**Previously Chief Executive Officer of LAACT, a Director of the Legal Aid Commission Victoria, and a senior lecturer in law at Monash University

Abstract

This article describes collaborative research undertaken by Dr Liz Curran in Australia for Legal Aid ACT with the objective of defining and measuring the quality, outcomes and effectiveness of legal assistance services in a context where services are being asked to assess their impact on broader client outcomes. The challenge was to find an evaluation design that does not impose a further significant administrative or financial burden on legal service delivery, which emerged as an issue with other studies that were costly, industry determined and which do not measure broader impact and client outcomes. In Australia, consistent funding for research and evaluation has not been available. There are significant existing reporting obligations on legal assistance services (LAS) which are also poorly resourced. This situation contrasts with some overseas jurisdictions. It is largely as a result of this climate that a modest, replicable, low cost and sustainable approach has been developed by the authors for application in Australia. Other organisations are using the study's design (for example, Refugee Legal in Melbourne). In this article, the authors aim to share with other jurisdictions with similarly limited legal assistance budgets an evaluation that not only examines the services delivered but also captures client outcomes and service impacts.

1. Introduction

This article shares the research approach undertaken by Dr Liz Curran in Australia for Legal Aid ACT (LAACT) [1] which aimed to define and measure the quality and effectiveness of service and client outcomes without imposing a significant burden on service delivery (Curran, E. 2012a).

The article discusses the challenges facing LAS in Australia in the context of the demands by governments for different and additional reporting on service outcomes, quality, effectiveness and cost efficiency in a context where, unlike the United Kingdom and Canada [2], the Australian government has been unprepared to provide additional resources for such reporting. Numerous Senate Committee and Statutory Inquiries have, for three decades, lamented the absence of, and refusal to fund, research and evaluation into Legal Assistance Services in Australia. [3] A recent review of LAS conducted for the Commonwealth Attorney-General's Department by The Allen Consulting Group [4] (NPA Review) underscored the lack of research. A more coordinated approach to commissioning of research is needed to strengthen the evidence base to inform where preventative and early intervention activities can promote the best outcomes. Agreed research priorities coupled with specific accountability for development of a national plan for education and information activities, would significantly increase the impact of resources devoted to this area. (2014, p. 10)

This article is not intended to review all the literature on legal professional services measurement as many studies in the United Kingdom were undertaken with a level of funding and resources that have not been available in Australia and were peer based. The authors wanted to move beyond solicitors measuring other solicitors' performance through peer review (Paterson, 2007) and include the views of other stakeholders including clients and community groups. [5] Although informative, many studies examined were not relevant to what the current authors were measuring which goes beyond service provision to include impact.

'Legal assistance services' (LAS) are defined in the National Partnership Agreement on Legal Assistance Services (NPA) [6] as legal aid commissions (LACs), community legal centres (CLCs), Aboriginal and Torres Strait Islander Legal Services (ATSILS) and Family Violence Prevention Legal Services (FVPLS). The NPA is an agreement between the Australian Government and the eight state and territory governments for the provision of Commonwealth funding for LAS in Australia.

In Australia, despite minimal funding support for research and evaluation of the legal assistance sector, there has been an intensified focus on reporting as governments seek greater value from their funding of LAS. The phenomenon of adding further accountability burdens on stretched legal assistance resources is not confined to Australia but is also evident overseas (Smith & Patel 2010; Hansen 2011; Smith 2011; Ryan-Collins et al. 2007; Trude & Gibbs 2010).[7]

2. Rationale for the Approach

In Australia further reporting and evaluation imposes additional administrative burdens on services that are already struggling to meet demand and seeking innovative ways to reach people in need and address the root causes of legal problems (Curran, E. 2013a). The authors, in their research, endeavoured to develop an approach to evaluation that was not resource intensive or expensive to conduct but grounded in the realities of the challenging client bases of LAS in Australia. For this reason, the humanitarian literature was examined as it similarly deals with issues of entrenched poverty and disadvantage. This literature suggests each program being evaluated must be understood to better inform and ensure adaptive learning and management processes, rather than evaluative tools being fixed and removed from the realities of practice. Any approach must be able to adapt, hear and incorporate changing realities and demands (UNDP 2010, p. 10). For this reason, an action participatory approach to the research was adopted by Dr Liz Curran. This will be discussed later in the section 'The LAACT Research'.

The approach to evaluation was designed to be easily replicated. The action research approach gives scope for evaluation tools to be adapted to local settings and different client and community groups. The approach can inform and enable adaptation of service delivery as it encourages reflection on what works and why or why not. It provides a solution where there is no research funding and a need to minimise the record keeping burden on services with limited staff and financial resources. This enables resources to be applied to service delivery and systems reform while generating evaluative data that informs service improvement. Depending on the size of the LAS being evaluated and the number of participating staff and clients, the cost of a study using this methodology, including data analysis, would be as little as AUD $7,000. This estimate is based on project staff time of 35 working days at a salary of AUD $65,000 pa + on costs. Additional costs of an estimated AUD $4,000 would be incurred if external consultants were engaged to oversee the survey. [8] As surveying is done by periodic service 'snapshots', it is not as resource intensive as continuous surveying. The triangulated approach developed uses multiple methods to collect quantitative and qualitative data in order to verify and cross reference results. However, the survey tools can also be used discretely enabling smaller services to further reduce costs by pooling resources and limiting the number of tools used.

While LAS in Australia acknowledge that it is important to report on effectiveness and outcomes such as early intervention and holistic service delivery, a key concern is that this should not divert resources from the delivery of legal services to people most in need. Governments should not impose new layers of reporting unless it promotes accountability, is useful for the services collecting the information, and provides an accurate picture for funders and the public. This view was endorsed in the recent NPA review (conducted by the Allen Consulting Group (ACG). The LAACT research was designed to complement rather than replace the collection of quantitative data already required by government. The LAACT research collected qualitative data on the nature of services and the client experience, and sought ways to evaluate responsiveness to different client needs. It not only evaluated service quality, but also identified the impact of the services on clients' lives so as to inform service practice and improve responsiveness to client need. This requires different measures to those used in other professional services fields which often focus on increasing market demand and the provision of commodities, rather than on preventative measures and the impact of services on complex human experiences. [9]

3. The Australian context

In setting the scene for this Australian research into evaluation, it is important to note that models of legal assistance differ between jurisdictions and this affects the aims and methods of service evaluations. Evaluations and reviews in the United Kingdom for instance, have sought to evaluate professional services in the context of contract management; with targets set by government. The reduction of transaction contracting costs and performance monitoring such as 'quality mark' and 'peer review' often sit within these settings. These approaches, as discussed below, tend to be costly and tend to have as their focus solicitors measuring solicitors without the perspectives on experiences of the service from clients themselves (Paterson, 2007). Unlike the predominantly private practitioner ('judicare') models of service delivery in Britain and New Zealand, Australia has a 'mixed model' where legal assistance services are provided by a blend of salaried and private practitioners. The mixed model resulted from a 'market failure' by private law firms to take on more time-intensive and less profitable legal assistance work, and it was considered critical that service delivery be combined with responsiveness to systemic issues and substantive legal equality (Noone & Tomsen 2006, p. 228). LAS tend to focus service delivery around the needs of the people using the services with the aim of preventing or reducing demand for services (contrary to neo-classical economic market theories which would see the market as wanting to increase demand) and strategically solving problems at their core through legal education, law reform and other innovative strategies.[10] For example, avoiding court altogether through 'Bulk Negotiation' with large corporations, settling client claims and improving poor practices (Curran, E. 2013a, p. 26).

Australia is geographically vast with many communities experiencing disadvantage and complex legal issues compounded by remoteness and isolation. Many legal, health and social services in regional and remote areas of Australia struggle with inadequate staff, poor retention rates and a lack of infrastructure and support. These challenges are compounded by the fact LAS in Australia work in a context of under-resourcing and underfunding (Commonwealth of Australia 2004).[11] The environmental and funding challenges confronting LAS in Australia informed the authors' choice of an action research model of evaluation that would engage, empower and support LAS staff rather than further burden and alienate them. LAS in Australia comprise the following.

Legal aid commissions (LACs) are established in each of the eight Australian states and territories. [12] They are independent statutory bodies funded principally by the Commonwealth and state and territory governments. LACs provide legal information and advice, duty lawyer services at courts and tribunals, and legal representation for people of limited means. They also provide dispute resolution services, deliver community legal education and provide policy and law reform advice to governments based on client case work and trends observed in the justice system. This breadth of services enables LACs to present informed and coherent policy positions on legal aid issues that cannot be replicated in a judicare system due to the fragmented and individualised nature of private legal practice.
Community legal centres (CLCs) are independently operated not-for-profit, community-based organisations that provide legal services including information, advice, legal representation, community legal education and law reform for the public, focusing on the disadvantaged and people with special needs. Some CLCs offer specialist legal services such as child support, credit and debt, environmental law, welfare rights, mental health, disability discrimination, tenancy, immigration and employment. Other CLCs provide services targeted to particular groups such as children and young people, women, older people, refugees, prisoners, and the homeless. Australia has around 200 CLCs which are funded mainly through state, territory, Commonwealth governments and some philanthropy.

Aboriginal and Torres Strait Islander Legal Services (ATSILS) provide culturally competent legal assistance services to Aboriginal and Torres Strait Islander peoples. They are community controlled non-for-profit organisations funded by the Commonwealth to provide legal assistance services in the areas of criminal, family and civil law in addition to undertaking community legal education, prisoner through-care and law reform and advocacy activities. Family Violence Prevention Legal Services (FVPLS) are legal and counselling services for victims of family violence and/or sexual assault who are Aboriginal or Torres Strait Islander peoples, or whose partner or children are Aboriginal or Torres Strait Islander peoples. [13] These services are funded by the Commonwealth Government and are located in 31 rural and remote locations around Australia. [14]

The NPA states that the performance of LAS will be evaluated against the broad goals of sector reform which promote a client-centred focus and include:

comprehensive access to information
seamless referral
improved coordination and targeting of services between legal assistance providers, and
the linking of LAS with other services to ensure 'joined up' service delivery.

The NPA also states that the evaluation of the performance of LACs will focus on how they increase their delivery of 'successful' service outcomes, how they increase and direct their service delivery towards prevention and early intervention services, as well as increasing the efficiency of their operations. Because the NPA itself contains few outcome measures, the research for LAACT was designed to provide a timely contribution to the then upcoming NPA Review by identifying some practical measures of service quality and outcomes. After completing the LAACT research, Dr Liz Curran was commissioned by the Commonwealth Attorney-General's Department (AGD) to conduct a literature review (Curran, E. 2012b) for the purpose of informing the NPA Review.

Apart from its relevance to the NPA Review, the research in 2011- 2012 was also timely because of the increasing calls by governments in Australia and overseas for agencies to report on their effectiveness and service impact. This article will also refer to some follow-up research by Dr Liz Curran since 2011 in the section 'Other applications of the model'. Legal assistance services in Australia are defined more broadly than case work, representation, information and referral, and include law reform and community legal education and interaction with community. They incorporate notions of holistic legal practice, early intervention and prevention and joined-up service between legal and non-legal services. The authors tried to include these in the measurement and avoid replicating other reporting mechanisms already in existence such as cross checks[15] and accreditation. [16]

The LAACT research was not only focused on the quality of the legal service provided, but also on broader service outcomes. These incorporate social justice outcomes in community legal education (now often in a community development setting) and law reform. The NPA aims of social inclusion, joined-up services, holistic approaches to problem solving, and client-centred approaches have recently been supported in findings from two significant and large empirical studies (Coumarelos et al. 2012; Schwarz et al. 2013). The first, the 'Australia-Wide Legal Survey' released in August 2012 was funded by National Legal Aid (NLA)[17] and the Law Foundation of NSW after the Commonwealth Government declined a request by NLA to contribute to the cost of the research. It is the most significant national evidence-based study of legal need ever commissioned in Australia and the largest research of its kind undertaken anywhere in the world.

The NPA review commissioned by the AGD in April 2013 was the most extensive (and probably the most expensive) government review of LAS undertaken in Australia, yet it did not involve the sort of micro-level evaluation of professional services that has occurred in the United Kingdom. The review was largely conducted through stakeholder consultation and LAS written submissions. Shortly before the NPA Review was announced, LAACT had commissioned Dr Liz Curran to evaluate the quality of its services and outcomes for clients. This enabled the research for LAACT to take account of, and inform, the NPA Review by developing a cost-effective way of measuring the 'successful outcomes' referred to in the NPA. Following the LAACT engagement, Dr Liz Curran was retained by the AGD to conduct a literature review which informed the NPA Review. [18]The NPA Review found:

Legal assistance service providers are providing services to disadvantaged Australians that are relevant to the needs of this diverse and complex population. However, there are significant levels of unmet demand for LAS, particularly by the most disadvantaged, across a wide range of areas of law and service categories…Services are, in the main, of appropriate quality. Some quality concerns were attributed to high caseloads, inexperienced staff, ability to attract and retain staff in regional and remote areas and gaps in quality assurance processes…Constraints identified in achieving the NPA objectives related generally to costs of providing LAS, with areas of stress including staffing, changes in demand linked to government policy and provision of services to complex high-need clients, and servicing regional and remote locations. Greater clarity about funding and services to be provided is required. In addition, the various agreements, reporting and funding flows mean that NPA objectives are not always adequately reflected in service delivery, especially for those services not funded under the NPA. This impacts on the provision and management of an integrated LAS sector that is well equipped to meet the legal needs of the most disadvantaged in our community. (2014, p. viii, vix)

Funding to both LACs and CLCs was reduced in December 2013 and May 2014 [19] by the Commonwealth Government although some cuts were reversed in late 2015 due to concerns about victims of family violence. There will still be a funding shortfall in 2017 and some cuts have led to services being reduced. This is despite the recent studies which commented on the impact of chronic under-funding on service effectiveness. The NPA Review's findings were posted on the Allen Consulting Group's website with little fanfare and the government has made no formal response.[20] In 2013 the Commonwealth Government also gave a reference to the Productivity Commission to inquire into access to justice arrangements in Australia. A draft report of the Productivity Commission has questioned further cuts to legal assistance service with statements acknowledging the effectiveness of the LAS systemic work in improving access to justice, this being the very area that the Commonwealth government has recently refused to fund. [21]

4. Pitfalls for legal service evaluation of outcome and quality

The LAACT research commenced with a literature review. This review and Dr Liz Curran's broader literature review for the AGD revealed the challenges presented by measuring service impact, outcomes and effectiveness. In exploring the best approaches to measuring the impact of LAS the literature review identified a number of pitfalls. Chief among these in the humanitarian literature is that before attempting to measure the effectiveness of a service, the nature of the service needs to be thoroughly understood. If inappropriate measures or benchmarks are used they risk setting an agency up to fail. LAS are effective because they are diverse, having adapted their services to suit the communities they serve. Such diversity can be placed at risk if, in an effort to streamline or make measurement consistent, or in a desire to homogenise services to make their administration easier for the bureaucracy, the basis of a service's effectiveness and ability to adapt to client circumstances is undermined.

Dr Liz Curran was also mindful that transplanting an approach to evaluation from a different service or jurisdiction can lead to distortions, given varying legislative and policy contexts and imperatives. It is also important to account for the diversity and complexity of factors that influence client behaviours and affect client outcomes. It is the authors' view that outcomes need to be understood in the context of the realities of clients' lives, otherwise measures become remote and speculative. It was for this reason that the authors adopted an action research approach [22] borrowing from the advice of the humanitarian sector. The World Bank 'Handbook' (Kusek & Rist 2004, p. 58) recommends a participatory approach involving key stakeholders because setting goals in isolation from what is being done and those who do it might lead to a 'lack of ownership' on the part of the main internal and external stakeholders. The World Bank Handbook recommends a participatory and consultative process be used at all stages in identifying goals, objectives, what outcomes look like, and setting outcomes over which the agency has control.

Using an action research approach enabled the project's design to be informed by the insights and experience of those who do the work with clients experiencing significant social problems, and the opportunity to hear from the clients themselves about factors that make a difference for them in navigating the legal system. The complexities involved in resolving the multiple, interconnected legal problems typically experienced by LAS clients were addressed in the research by using a methodology which incorporated quantitative and qualitative data to enable insight into the nature of the work, the client and contexts, rather than placing sole reliance on statistics. Open questions were also utilised in a number of the research instruments to uncover deeper background to what led to the identified outcomes and the service qualities demonstrated in the course of the service's work. This approach is discussed in the next section, 'The LAACT Research'.

The preliminary literature review revealed that many overseas and national studies used what were called 'client satisfaction' surveys. This nomenclature is inappropriate in a legal services context as it suggests a lawyer's role is to ensure client satisfaction. While this might apply to the supply of goods it is less relevant to the role and duties of a lawyer in assisting people with complex issues and giving fearless and independent advice on the client's position at law which, though correct, might not always 'satisfy' a client. Another cautionary note emerging from Dr Liz Curran's research is that care is needed in measuring LAS so as to be mindful of the ethical responsibilities of legal professionals. [23] In fulfilling their duties lawyers may sometimes be placed in a position where acting in a client's best interests conflicts with the fulfilment of other professional duties. Surveys purporting to measure client satisfaction also run the risk of misleading survey participants as to the role of legal professionals as 'officers of the court'.

Legal services outcomes must be defined according to factors that are within a lawyer's control. If an outcome is too broadly expressed, for instance 'legal service secures appropriate housing', this assumes that the lawyer has in their control the provision of sufficient housing stock and an ability to circumvent long and tightly targeted waiting lists. Lawyers may not always be successful in seeking a desired outcome for a client for reasons unconnected with the skill and effort of the lawyer and therefore beyond the lawyer's control. A common example is that the evidence against a client's case may be strong yet the client might not be satisfied when they lose the case. For this reason, in evaluating service outcomes Dr Liz Curran suggests that the solution is for the feedback sought from clients to focus on service characteristics and case events that the lawyer is able to control or influence, rather than on whether a case is 'won' or 'lost'.

LAS work with diverse and usually disadvantaged client groups. Service approaches vary across agencies in order to appropriately target and tailor services to the needs of their client groups. Outcome measures need to reflect this diversity, the need for responsiveness to different client groups, and the differing and often lengthy timeframes within which outcomes can realistically occur. Dr Liz Curran's LAACT research reveals the complex and technical nature of legal assistance work and that even within one legal assistance service specific programmatic responses are required to be delivered differently due to different legislative and policy settings. It also notes that capacity to be effective can be affected by the extent of resources available. [24] Overall, Dr Liz Curran's approach highlights the challenges associated with measuring outcomes and the need to acknowledge the difficult and complex nature of LAS delivery. An important element of accounting for these complexities is designing an evaluation framework that takes a 360 degree approach by looking at LAS from different perspectives, including clients, employees and other stakeholders. (CG, 2012, p.41) Government policy and funding decisions which are based on poorly defined outcomes and performance targets, or indicators which are not relevant to the outcomes they purportedly measure, can compromise the effectiveness of a service or program. This was one of the key concerns of the authors concerning the methodology used in the NPA Review.

Outcomes should be defined according to the role or function of the lawyer, which may be to advocate and tell their client's side of the story or to hold authority to account. To propose as a successful legal service outcome one which, like the housing example, is subject to many factors over which a lawyer can have little or no control, sets up the legal service to fail. It may be that the ultimate objective is to secure housing for a client, but it should not form part of the measurement of that legal service's effectiveness.

5. The LAACT research

The research for Dr Liz Curran's report which forms the basis for this article, was undertaken between August and December 2011, and the research findings were published in late April 2012.[25] As noted above, the research model was developed after reviewing Australian and international research into the measurement of service outcomes (UNDP 2010; Downes 2011; Kusek & Rist 2004, p. 58). Most importantly, the research was informed by discussions about the nature of LAS - their layers, complexities, contradictions and impediments - with the people delivering the services.

5.1 The research process - participatory action research

The research was conducted using a participatory action research approach. Participatory action research has been described as a reflective process of progressive problem solving led by individuals working as part of a 'community of practice' to improve the way they address issues and solve problems (Dick 1999/2011). Participatory research is done by taking action, guided by a professional researcher, in this case Dr Liz Curran, with the aim of improving strategies, practices, and knowledge of the environments of practice, in this case legal practice. This approach uses a cyclic or spiral process which alternates between action and critical reflection and, in the later cycles, continuously refines methods, data and interpretation in the light of the understanding developed in the earlier cycles (O'Brien 1998; McCutcheon & Jung 1990; Dick 1999/2011).

This methodology was adopted because international research in the humanitarian field suggests that research evaluations affecting human beings (such as LAS) should be 'bottom up' (informed by the front-line experience of people using the service and delivering the service) rather than 'top down' (designed by government or civil service who are remote from the experiences of those who use the service and often guided by political imperatives), that is, informed by those who do the work and the experiences of the people using the service being measured (Ebrahim & Rangan 2010; Kusek & Rist 2004). This was not a feature that had been acknowledged in many of the evaluations of legal services identified in the literature review.

The research was undertaken in two phases.

5.2 Phase 1 - Developing service outcomes and their indicators

Phase 1 was informed by literature and research from the legal, humanitarian and social spheres. The research methodology also drew on LAACT's values, goals, operational objectives and strategies described in its strategic plan. These goals and objectives were incorporated into the outcome indicators and questions used in the survey instruments to test whether the goals and objectives were being met in practice.. At the time the LAACT study was commissioned there had been little previous research into how the outcomes or quality of LAS might be measured in a cost-efficient way by community and other legal assistance providers with limited resources. [26] Accordingly, the research commenced as a trial of survey instruments developed by the researcher which were designed to:

obtain qualitative data that could be used for reporting purposes;
establish baseline data against which to measure future progress;^{^[27]}
be easily applied and analysed; and
be affordable, given the lack of resources for legal services evaluations in Australia and the need to avoid diverting resources from service delivery.

The development of the survey instruments was informed by international and domestic research and (most importantly) by discussion of the nature of the services in focus groups and with individual staff providing the services in each practice area. Staff echoed the warnings in international research about the dangers of using measures such as outcomes that were outside their control. Accordingly, in consultation with staff, outcomes were defined in the research to be realistically achievable in the context of the legal services being provided and to take account of the perspectives of service users. Some of the international research (Commission for Social Care Inspection 2006; Kilian 2010) reveals that the 'qualities about which service users are particularly concerned' include: choice; flexibility; information; being like other people; respect and being heard; fairness and no discrimination; cost and value; and safety. Other qualities include: responsiveness; empathy; involvement; accessibility; listening carefully; keeping me up-to-date; discreet atmosphere; explaining things clearly and 'in a way I understood so I knew what to do and what was going to happen'; helpfulness of staff; and provided with relevant information in a timely way. [28]

Outcomes and outcome indicators both indicative of and necessary for quality service provision need to be consistent and realistic, taking into consideration the roles and functions of a LAS with a diverse range of policy and legislative settings. This was a challenge in this research as it needed to be conducted in a way that took this into account.

Using the participatory approach - of finding out what the service does, the context, and the legislative and policy settings that operate for each legal practice area - was key to ensuring that what was measured was within the remit of the service and able to reflect the complexities identified. Other benefits in having research informed by what is actually done, and why it is done in a particular way, means the LAS is in a better position to explain their services and outcomes to the community, funders and other stakeholders. The approach also encourages staff to reflect on the way they provide services and what characterises service quality and effectiveness i.e. reflective practice. By collecting both quantitative and qualitative data from interviews and professional journals with guiding questions the authors were better able to unpick and identify the policy and legislative settings; the impact of complex clients on the services work, and identify what worked and did not work and why - features which the authors consider essential in human service research evaluation such as the provision of legal assistance services. This was not an approach taken in other legal service evaluations which were more limited in their data collection (Paterson, 2007; Paterson & Sherr, 2007; Legal Aid Agency, 2013).

Following the literature review the next step in Phase 1 was a conversation with staff in a workshop format facilitated by Dr Liz Curran. [29] Staff worked in groups guided by a set of questions covering why the service existed, what was important in delivering a positive outcome, what constituted a quality service, and what might happen to people involved in legal events 'but for' the service. The information from these discussions was displayed for a period in the staffroom so staff could comment, add further thoughts and reflect on the responses. Following the staff workshop, focus group discussions were held with each section of LAACT engaged in delivering services to the public, namely the family, criminal and civil law practices, and Client Services (which included reception and the administration of grants of financial assistance for legal representation). Each focus group was provided with a background paper prior to the discussion which outlined relevant research and information specific to the areas of practice, for example, the 'Best Practice Guidelines' for family law work issued by the Law Council of Australia (2012).

The service outcomes to be measured were developed based on the focus group discussions, the literature, LAACT's strategic plan, and the aims of the NPA discussed above. The steps necessary to lead to these outcomes were identified as indicators of a quality service being present. Draft survey instruments and questions designed to measure outcomes across the different practice areas were then prepared and reviewed by the focus groups, and by a former client who provided feedback from a client perspective. Eleven positive service outcomes were identified in this way and the service characteristics which, if present, were likely to lead to those positive outcomes were identified as indicators of a quality service. These are described in Table 1.

Table 1. Identified positive service outcomes and indicators

Outcomes	Indicators and Considerations (qualities demonstrated by outcome)
1. A good client interview.	Holistic, Joined-up, Quality, Problem Identification, Empowerment, Good Practice, Early Intervention, Prevention, Responsiveness, Client Centred, Alternative Dispute Resolution (ADR), Targeting, Expertise.
2. Clients with chaotic lifestyles attend interviews, appointments and court dates.	Early Intervention, Prevention, Empowerment, Client Centred, Holistic, Targeting.
3. As appropriate, custodial sentences are minimised or unsubstantiated charges are dropped.	Rule of Law, Efficiency, Good Practice, Expertise.
4. Clients are better able to plan and organise their legal affairs.	Early Intervention, Prevention, Empowerment, Quality, Good Practice, Client Centred.
5. Improvement in the client's interaction with the legal system.	Early Intervention, Prevention, Empowerment, Client Centred.
6. Consideration of issues before a court or tribunal enhanced because the lawyer asked questions/raised issues and brought the client's story before the court.	Rule of Law, Quality, Voice, Flexibility, Good Practice, Client-Centred, Responsiveness, ADR, Expertise.
7. Client is better able to understand their legal position and the options open to them.	Early Intervention, Prevention, Empowerment, Good Practice, Quality.
8. A process is followed where the client is listened to, respected and given fearless advice of their legal position.	9. Quality, Client Centred.
10. Relationships and trust building with other legal and non-legal support agencies enabling client referral and support.	Early Intervention, Prevention, Holistic, Joined-up, Good Practice, Quality.
11. Holding of authority to account.	Rule of Law, Quality, Voice, Flexibility, Good Practice, Client-Centred, Responsiveness.
12. A holistic service delivered to the client through collaboration, networking, community legal education and joined-up services.	Good Practice, Client Centred, Problem Identification, Collaboration, Prevention, Early Intervention, Holistic, Joined-up.

The challenge in measuring legal services is to ensure that, despite the multifaceted and complex nature of the services, the instruments are relevant, consistent and realistic, and that the data obtained can be used for reporting purposes and to improve service delivery. Taking a participatory approach to the research and engaging staff as research partners meant that they had ownership of, and trust in, the process. In a final de-brief with staff after the first snapshot, staff reported that they had found the collaborative process empowering and enabling. This was significant to Dr Liz Curran as in order to encourage changes and improvements in practice it is better if the environment is one of enabling and ownership rather than a more punitive approach to evaluation. In addition, it became evident that although there were differences in the nature and complexity of services across the various practice areas, themes emerged that were consistent and relevant to quality practice across all practice areas. These themes were consolidated and incorporated into the eleven outcomes in Table 1.

The survey instruments were then tested during a two-week 'snapshot' of LAACT's services in November 2011. The snapshot and its findings are discussed later in this article.

5.3 Phase 2 - Developing a methodology for gathering qualitative and quantitative data

In Phase 2 qualitative research was undertaken in order to reveal the complexity of the services being measured and to help explain the reasons behind the quantitative data that would be collected. Given the already onerous reporting requirements imposed on LAACT and the limited resources available for service measurement and evaluation, it was necessary for the researcher to develop a survey methodology that could be implemented at minimal cost and with minimal disruption to service delivery. Accordingly, it was decided to survey service quality and outcomes by taking periodic service 'snapshots' rather than collecting data on a continuous basis. An initial snapshot of each service area would be conducted over a two-week period to establish the baseline data and identify areas where services could be improved. Further snapshots would then be conducted at intervals as resources allowed in order to monitor the effectiveness of service improvement measures. In order to minimise the impact on everyday operations, each snapshot would be limited to one or two service areas with other services areas surveyed in rotation in subsequent surveys. Sufficient surveys would be conducted in the first year to cover all service areas. The first 'snapshot' in each area would be done as a pilot with any necessary refinement of the survey instruments done after the snapshot findings were released and prior to doing any future snapshots.

The survey model developed by Dr Liz Curran can be adapted across a variety of organisations and applied with minimal resourcing. It is designed to be a sustainable method of capturing data on service quality and outcomes that avoids the administrative and cost burdens of continuous measurement regimes. It was critical that both quantitative and qualitative tools were combined to ensure that the layers of complexity in legal services were uncovered and informed all parts of the research. The methodology was also designed to enable responses to be checked and verified against each other and to enable different stages of LAACT's activities to be examined and measured against quality and outcome indicators. The researcher used a combination of different methods including surveys, questionnaires, interviews, staff journals and focus groups in each service area examined. Open and closed questions were used in the survey instruments, framed in such a way as to unravel the complexity and reveal case studies by asking for examples to be provided.

This involved engaging with service providers, clients and external stakeholders (such as the courts and legal and non- legal referral agencies) during the 'snapshot' period to obtain their respective viewpoints on LAACT's services and to assess relationships and interfaces with clients, courts, other parties, significant networks and community agencies. In this way the research presented a '360 degree' view of services from those involved in and affected by the service's operations. This approach ensured quality service processes focused on and attended to the realities of clients' lives and that they were grounded in the experience of those using the service (Kusek, J.Z. & Rist, R.C. 2004; United Nations Development Program (UNDP), 2010) and not just limited to the view of those providing the service which as noted earlier is a limitation on some of the other measurement methods in the United Kingdom.

5.4 Conducting the Snapshot

A two-week trial research approach examining services in LAACT's family and criminal law practices was conducted using the following instruments and methodologies developed in Phase 2:

Eight lawyer and eight client feedback interviews were conducted by the researcher separately following each lawyer/client interview.
Entries were made in observation logs by seven staff who were not involved in other survey tasks.
A voluntary client feedback questionnaire was handed to all clients after receiving legal advice at LAACT's office.
A telephone survey of clients was attempted following closure of their case files. [30]
An online survey was conducted of in-house lawyers and private lawyers who handle legally assisted cases.
Case studies were collected from open questions in observation logs, focus group discussions, client interviews with the researcher, and the online survey.
Interviews were conducted with stakeholders identified by each practice area and with academics from the Australian National University College of Law, who work with law students participating in LAACT's Youth Law Program and the Legal Aid Clinic advice service.

6. Findings from the LAACT research

LAACT scored highly and consistently on the outcome indicators across the different measurement tools used, indicating that the eleven desired outcomes and characteristics of quality service were present. This finding was verified in discussions with clients and stakeholders. Taking this data, LAACT has been able to identify gaps in services, provide additional support and training to staff and has adapted some of its policies and practices to enable it to be more responsive to client need. The following extracts from the data give a sense of the questions and responses obtained by the researcher. More detailed information about the methodology and outcomes can be found in the research report (Curran, E. 2012a).

6.1 From Client Interviews

The feedback interviews with clients and lawyers following the initial client/lawyer advice interview were used to compare the views of the lawyers and clients about what happened at the interview. This process was based on similar research in the United Kingdom (Moorehead et al. 2006, Buck et al. 2010). Care had to be taken in selecting research from the United Kingdom as there were different policy frameworks affecting the nature of data collection in that jurisdiction. This difficulty is discussed by Smith and Patel in relation to their work (2010, p. 4, 34). The comparison of feedback confirmed a high degree of consistency between lawyer and client responses. The interviews revealed the complexity of the clients' legal issues, as well as other legal and non-legal issues affecting or compounding clients' legal problems.

6.2 Client Feedback Questionnaire after interview

Short questionnaires were handed out by reception staff to all clients attending the office for advice interviews across the entire legal practice during the snapshot period, and were voluntarily completed by clients following interview. The questions (as noted above which were tested by a client before roll out) and summary responses included:

Based on your experience today, if you had another legal problem would you seek assistance:

From LAACT (50%)
Earlier than you did this time (23%)
Know better what to do than last time (50%)
If it happened to a friend/family member, recommend Legal Aid to someone else (27%)
No answer (11%)

The service treated me with understanding and respect.

Strongly Agree (38%); Agree (35%); Don't Agree (0); Strongly Disagree (4%);

No answer (11%).

The service was responsive to my individual circumstances.

Strongly Agree (50%); Agree (38%); Don't Agree (0); Strongly Disagree (4%);

No answer (11%).

6.3 Stakeholder interviews

Seven of the eleven stakeholders approached for feedback about LAACT responded. This was a part of the methodology which reached out to those services that had a record of referring clients to LAACT or who worked as networks in which LAACT was a participant. It was designed to gather information from them about their experience of LAACT and measure its quality and effective service from non-legal service and court perspectives. Responses included:

[Legal Aid] remains a fall back resource for the 'underprivileged' especially given there are so many gaps. For example, when the private lawyers do not show up at court, the LAACT lawyers will step in (they often can't represent a client as there is already an established conflict of interest) and locate the lawyer, make enquiries or get the private lawyer to court with speed. Often what LAACT has to do at court is ad hoc and unexpected and can be stressful. They step in willingly and act as 'officers of the court'.

6.4 Professional staff observation logs

Professional staff were asked to volunteer to write an observation log (effectively a personal reflective journal) that recorded their work experiences for five days during the two-week period. These guiding questions were provided:

What work and interactions your day involved.
The nature of the work and what you did.
Any difficulties you faced.
Any breakthroughs in any cases or other client interaction. Why and how they came about.
Any dilemmas or stresses and how you managed these.
Any other agency engagements - briefly describe these.
Any positive outcomes from your day no matter how small, and how these came about.
Any reflections.

The purpose of the reflective journal (which we termed a 'log' in response to a preference of staff) was to identify the complexity of client matters, any barriers to resolution, the extent of interactions with others in dealing with client matters, and the staff reflections on service outcomes. This would elicit data around the complexities in client work and reveal some of the day to day breakthroughs and barriers that the solicitors face. This was to enable the data to reflect the realities and unravel information that might otherwise be hidden but which are a feature of the client work. The following extracts are from the log of a solicitor working in LAACT's Dispute Resolution Program, a conciliation service specifically designed for resolving family disputes:

Day 2: Family law solicitor spoke to me about a duty client who needed assistance to have her [baby] daughter returned from a non-Hague convention country. I spoke to the client about support services available to her as a Muslim woman and parent and made a warm referral to a local Imam from the Islamic Centre. I had recently met with the Imam to discuss outreach work at the Islamic Centre and speaking with Muslim families about FDR [family dispute resolution] and Australian family law and to invite the Imam to address lawyers and other professionals working in the family law sector about Muslim families. This is an ongoing outreach project with FDR.

Outcome: Mother obtained legal advice and referral to family support service appropriate to her culture and religion. Within four days the family law solicitor had taken instructions, initiated proceedings and obtained orders that resulted in the mother being able to travel to the Middle East and retain care of her child and return to Australia.

Reflection: This was an example of FDR and Family law and Client Services (who assisted the client with her legal aid application form) working together to achieve an outcome for the client. The co-operation resulted in mother and child being reunited in circumstances where the child may have been lost to the mother for many years.

Day 3: This day was largely spent progressing FDR files, conducting intake and assessment, arranging conferences and sending out conference documents to parties and their lawyers. Several phone calls were made to and received from self-represented persons who had received a letter inviting them to participate in FDR. One self-represented mother was the victim of many years of domestic violence and was upset and stressed about my letter. I was able to calm her down and explain to her that there were exceptions to compulsory FDR and that nothing would be arranged without her consent. I talked to her about the need for obtaining legal advice and helped her to apply for legal aid.

Outcome: Information given to a distraught and traumatised person resulted in her better understanding a legal process and some confidence that with legal advice she could retain some control of her matter.

Reflection; The work of the Legal Aid ACT FDR Program not only assists legal aid clients but hundreds of self-represented persons each year with referrals, information and other practical assistance.

These log entries indicate that service outcomes and qualities present included integration, holistic care, seamless referral, targeted service and joined- up service, collaboration and good practice.

Table 2. Outcome and quality indicators/considerations present in log entries

Outcomes	Indicators/Considerations (qualities demonstrated by outcome)
A good client interview.	Holistic, Joined-up, Quality, Problem Identification, Empowerment, Good Practice, Early Intervention, Prevention, Responsiveness, Client Centred, Alternative Dispute Resolution (ADR), Targeting, Expertise.
Clients are better able to plan and organise their legal affairs.	Early Intervention, Prevention, Empowerment, Quality, Good Practice, Client Centred.
Improvement in the client's interaction with the legal system.	Early Intervention, Prevention, Empowerment, Client Centred.
Client is better able to understand their legal position and the options open to them.	Early Intervention, Prevention, Empowerment, Good Practice, Quality.
Relationships and trust building with other legal and non-legal support agencies enabling client referral and support.	Early Intervention, Prevention, Holistic, Joined-up, Good Practice, Quality.
A holistic service delivered to the client through collaboration, networking, community legal education and joined-up services.	Good Practice, Client Centred, Problem Identification, Collaboration, Prevention, Early Intervention, Holistic, Joined-up.

6.5 Online survey

An online Survey Monkey of all LAACT lawyers and private lawyers who handle grants of legal assistance on referral from LAACT were conducted over a two-week period. A total of 45 five responsive were received, 25 of which were from private lawyers (out of 146 private lawyers on the LAACT referral panel) and 20 from in-house lawyers.[31] This initial survey sought preliminary data only and measured levels of knowledge and comfort around government requirements for the NPA since the private professional engaged in the initial survey had at that time limited knowledge of the new NPA requirements. It was therefore important to test their knowledge of and intentions at an early stage of the NPA given it was a new and different way of working.

Figure 1. Survey responses on rating importance of certain actions when representing clients at court or tribunal (Curran, E. 2012a, p. 133).

6.6 Case Studies

Data for case studies was gathered during the research through open questions in the staff observation logs, the practitioner survey, the focus groups and interviews highlighted how dire some of the outcomes for clients could have been 'but for' the intervention of LAACT. As the extract below (one of many collected) reveals, client situations are complex and need a response from lawyers which is responsive, holistic and engages both legal and non-legal services and client centred if it is to have an impact and a positive outcome. Collecting such qualitative data emerged as critical in measuring impact given the complicated nature of client circumstances and the need to be aware of these elements for legal services to be effective. Below is an example from the 23 collected case studies:

One criminal lawyer noted that in late October 2011 he was representing a homeless Aboriginal man with significant mental health problems and intellectual disabilities. The lawyer tried to connect the man with supported accommodation services or accommodation of some kind, because the man was 'at risk' of incarceration if he could not find accommodation. Incarceration was of particular concern in this case given the number of Aboriginal men who die in custody, the rates of prison assaults and this client's particular vulnerability. The lawyer had no success in telephoning services to get assistance. The lawyer stated he had to 'go the extra mile . . . with these clients who have no chance at navigating the system - if we don't help, sometimes no one else will. I had to drive him to the [accommodation] places, as otherwise they would not have taken him'. If the lawyer had not 'gone the extra mile', it is almost certain that this client would have been locked up and refused bail.

This case study indicates that the following service outcomes and qualities were present.

Table 3. Service outcomes and indicators/considerations present in case studies

Outcome	Qualities demonstrated by outcome
Clients with chaotic lifestyles attend interviews, appointments and court dates.	Early Intervention, Prevention, Empowerment, Client Centred, Holistic, Targeting.
Relationships and trust building with other legal and non-legal support agencies enabling client referral and support.	Early Intervention, Prevention, Holistic, Joined-up, Good Practice, Quality.

The case study also highlights the effort and significant support and advocacy interventions that are necessary when clients have so many and various compounding legal and social issues. The lawyer must be persistent and, where issues of trust are at stake, consistent. The example also highlights that clients often face an intransigent and non-responsive system that is difficult to navigate, and that having an articulate advocate in such cases can lead to improved outcomes.

7. Subsequent LAACT snapshots and resulting improvements to practice

Since the conduct of Dr Curran's research in 2011, LAACT has conducted three snapshot surveys of its services and the extent to which the outcomes defined in the research are being achieved. The snapshots were taken over two week periods using the survey instruments developed in the research. These surveys have confirmed research findings in Australia and elsewhere that disadvantaged and vulnerable people often experience multiple legal problems with severe consequences and that unresolved legal issues frequently impact on people's health and wellbeing. [32] For example, 75% of clients in one survey had other legal problems linked to their presenting problem and identified their legal problems as causing, being linked to, or increasing their stress levels.[33] The surveys have also been a rich source of case studies which illustrate the complexity of legal aid work and the difference that timely provision of legal advice and other assistance can make to the resolution of legal problems, with positive flow-on effects to people's health and wellbeing.

While the results of the surveys have been very positive overall, they have also identified aspects of service delivery where there is scope for improvement and other information of value in resource planning and management. As a result of this feedback, LAACT was able to improve the planning and delivery of community legal education; ensure clients understand the advice they are given; improve protocols for referrals between agencies; and develop an online directory of free legal services in the ACT. The value of these surveys lies not only in the positive feedback that affirms what the service is doing well and, in doing so helps sustain staff commitment and morale, but also in the negative feedback that reveals service weaknesses which might otherwise be overlooked and not addressed.

8. Other applications of the model

In 2013 Dr Liz Curran used outcomes and quality indicators developed in the LAACT research for six-monthly snapshot evaluations of the Consumer Action Law Centre's telephone advice line which have occurred routinely since October 2012. The Consumer Action Law Centre in Melbourne is a state-wide specialist community legal service specialising in consumer law. The LAACT survey instruments, outcomes and indicators were readily adaptable to reflect the specialist nature of this service. In September and October 2013, Dr Liz Curran also ran a series of workshops on measuring effectiveness and quality of LAS for staff at Law Clinics Ontario and for Legal Aid Ontario at the request of the Community Advocacy and Legal Centre, Ontario, Canada. [34]

In 2014, Dr Liz Curran was asked by the Advocacy and Rights Centre (ARC) in Loddon Campaspe, a rural region of Victoria, to undertake an evaluation utilising the model with adaptations for a legal-health setting for roll-out over two years. In Loddon Campaspe, a Health Justice Partnership-legal practice has been established by the ARC to reach people who visit a community health service with their legal problems, through an integrated model. The study has adaptations to also include measurement of social and health determinants and impact of the Partnership. [35] Dr Liz Curran has been asked to provide evaluation benchmarks for the Legal Services Board Victoria to inform five recently funded further Health Justice Partnerships in Victoria using the approach. These applications demonstrate that the model can be successfully adapted and replicated in different settings, provided that this is done in consultation with the staff and community using those services and takes account of the organisational context, culture, roles and functions of each service.

As demonstrated by the LAACT research, and particularly the case studies used in this article, the situations in which clients and service providers find themselves are often inherently difficult and can be affected by systemic issues. Yet quality service delivery and appropriate interventions can, on occasion, transform the experiences and lives of clients, deal with the power imbalances, and hold authority to account. Research that communicates the real experiences of clients and the challenges of service providers is critical to ensuring a responsive and inclusive legal system. For this reason, the authors want to share the research methodology in the hope it might be replicated with appropriate adjustments for different communities and jurisdictional contexts in a climate where resources for research and evaluation at a grass roots level are limited or non-existent. The research methodology, by virtue of the fact it is developed and applied in collaboration with service providers, stakeholders and clients, enables reflective practice and drives the motivation to be better, and entrenches an environment of continuous improvement and development which can only enhance the service to clients and community.

It is also hoped that this evaluation design might be considered by governments, and departments of justice and finance, as a further method of ensuring that the experience of clients and service providers of day-to-day operations and the difficulties of navigating the legal system, can also inform policy making and funding decisions. As noted earlier, in Australia, such an approach has been recommended by the NPA Review. Governments and funders aside, however, this evaluation design provides a rich and important source of information for services themselves as they seek to improve and become informed on their effectiveness and impact on clients and the community. Given their increasingly inadequate resources, it is critical that LAS remain focused on effective service delivery. While funders and other stakeholders have a legitimate interest in ensuring that service providers are accountable for the efficient and effective expenditure of public funds, accountability measures must take account of the realities of service delivery and not impose an unreasonable time or cost burden on service agencies.

Acknowledgements

The authors wish to thank Legal Aid ACT and its Board and staff, the Consumer Action Law Centre (Melbourne), the Advocacy and Rights Centre (Bendigo), Michelle Leering (Law Clinics Ontario), Meredith Bramich, Pamela Crockett and Julie McMaster.

References

Allen Consulting Group. (2012) Review of the National Partnership Agreement on Legal Assistance Services, Working paper one: final evaluation framework, November (Melbourne, Australia, The Allen Consulting Group).

Allen Consulting Group. (2013) Review of the National Partnership Agreement on Legal Assistance Services, Working paper two: evaluation of legal assistance services, June (Melbourne, Australia, The Allen Consulting Group). Available at: http://www.acilallen.com.au/cms_files/0NPA%20Review_Working%20paper%202_Evaluation%20of%20LAS.pdf accessed June 2013

Allen Consulting Group. (2014) Review of the National Partnership Agreement on Legal Assistance Services, Final Report, June (Melbourne, Australia, The Allen Consulting Group). Available at: http://www.acilallen.com.au/cms_files/NPA%20Review_LAS%20report_final.pdf , accessed [ 1 July 2014 ].

Australian Broadcasting Commission (ABC). (2013) Funding cuts to cripple Indigenous legal services, PM, radio program, 17 December. Available at: http://www.abc.net.au/pm/content/2013/s3913355.htm accessed 17 December

Australian Financial Review . (2014) Aid suffers as Brandis ignores report, 12 September, p. 32.

Australian Law Reform Commission (ALRC). (1999) Review of the Federal Civil Justice System, Discussion Paper 62. Available at: http://www.alrc.gov.au/dp-62, accessed 11 August 2011.

Australian Law Reform Commission (ALRC). (2000) Managing Justice: A Review of the Federal Civil Justice System, Report No. 89. Available at: http://www.austlii.edu.au/au/other/lawreform/ALRC/1999/89.html, accessed 11 August 2011.

Australian Productivity Commission. (2014) Access to Justice Arrangements: draft report (Overview), (Canberra, Australian Government Productivity Commission), p. 2. Available at: http://www.pc.gov.au/inquiries/completed/access-justice/draft, accessed 12 September 2014.

Barendrecht, M., Mulder, J. & Giesen, I. (2006) How to Measure the Price and Quality of Access to Justice, working paper. Available at: http://dx.doi.org/10.2139/ssrn.949209, accessed 11 August 2011.

Bond C. (2013) Funding cut for Aboriginal and Torres Strait Islander legal services should be reconsidered, opinion, 17 October, Human Rights Law Centre, Melbourne. Available at: http://hrlc.org.au/funding-cut-for-aboriginal-and-torres-strait-islander-legal-services-should-be-reconsidered/ , accessed 17 October 2013.

Buck, A., Balmer, N. & Pleasance, P. (2005) Social exclusion and civil law: experience of civil justice problems among vulnerable groups, Journal of Social Policy and Administration, June, 39(3) pp. 302, 318-320.

Buck, A., Smith, M., Sidaway, J. & Scanlan, L. (2010) Piecing it Together: Exploring One-stop Shop Legal Service Delivery in Community Advice Centres, (London, Legal Services Research Centre, United Kingdom Ministry of Justice). Available at: http://www.justice.gov.uk/publications/research-and-analysis/lsrc, accessed 23 December 2012.

Commonwealth of Australia. (2004) Inquiry into Access to Justice, Chapter 8, Senate Legal and Constitutional Affairs Committee, (Canberra: Australian Government Printing Service).

Council of Australian Governments (COAG). (2010) National Partnership Agreement on Legal Assistance Services, Council of Australian Governments. Available at: https://www.ag.gov.au/LegalSystem/Legalaidprogrammes/Documents/National%20Partnership%20Agreement%20on%20Legal%20Assistance%20Services.pdf , accessed 12 August 2011.

Commission for Social Care Inspection. (2006) Real Voices, Real Choices: The Qualities People Expect from Care Services (Newcastle, UK, Commission for Social Care Inspection).

Community Legal Education Ontario. (2013) PLE Learning Exchange Ontario. Available at: http://www.plelearningexchange.ca/ (accessed 12 December 2013).

Council on Social Action. (2009) Time well-spent: The importance of the one-to- one relationship between advice workers and their clients, (London, Community Links). Available at: http://www.community-links.org/linksuk/wp-content/PDF/COSATWS.pdf, accessed [ 17 August 2012 ].

Coumarelos, C., MacCourt, D., People, J. McDonald, H.M., Wei, Z., Iriana, R. & Ramsey, S. (2012) Access to Justice and Legal Needs: Legal Australia Wide Survey Legal Need in Australia', (Sydney, Law and Justice Foundation of New South Wales). Available at: http://www.lawfoundation.net.au/ljf/site/templates/LAW_AUS/$file/LAW_Survey_Australia.pdf , accessed 10 June 2014.

Curran, E., (2012a) 'We Can See there's Light at the End of the Tunnel Now': Demonstrating and Ensuring Quality Service to Clients, Legal Aid ACT, (Canberra, Legal Aid ACT). Available at: http://www.legalaidact.org.au/pdf/Light_at_the_end_of_the_Tunnel_Legal_Aid_Services_Quality_and_Outcomes.pdf , accessed 14 May 2014.

Curran, E., (2012b) A Literature Review: examining the literature on how to measure the 'successful outcomes': quality, effectiveness and efficiency of Legal Assistance Services , (Canberra, Australian Attorney General's Department). Available at: https://www.ag.gov.au/LegalSystem/Legalaidprogrammes/Documents/Literature%20review%20of%20legal%20assistance%20by%20Dr%20Liz%20Curran.pdf , accessed 30 January 2014.

Curran, E., (2013a) Solving problems - a strategic approach: Examples, processes & strategies, Legal Workshop, Australian National University College of Law, 13 March. Available at: http://www.law.anu.edu.au/sites/all/files/legalworkshop/final_report_solving_legal_problems_curran_calc_13_march_2013.pdf accessed 19 June 2014.

Curran, E., (2013b) Encouraging Good Practice in Measuring Effectiveness in the Legal Service Sector, Legal Workshop Australian National University College of Law, 21 May. Available at: http://www.plelearningexchange.ca/database/solving-problems-strategic-approach-examples-processes-strategies/ accessed 19 June 2014.

Curran, E., (2014) Attorney General George Brandis set to silence CLCs, The Saturday Paper, 6 April. Available at: https://www.thesaturdaypaper.com.au/opinion/topic/2014/06/14/attorney-general-george-brandis-set-silence-clcs/1402668000 , [accessed 14 June 2013].

Currie, A. (2003). The Unmet Need for Criminal Legal Aid: A Summary of Research Results, Legal Aid Research Series, Department of Justice, Canada. Available at: http://www.justice.gc.ca/eng/rp-pr/csj-sjc/ccs-ajc/rr03_la9-rr03_aj9/rr03_la9.pdf , [accessed 12 September 2014] .

Dick, B. (2009/2011) Action Research, Action Learning (updated 8 May 2011). Available at: http://www.aral.com.au/resources/arphome.html, accessed 11 August 2011.

Downes, P. (2011) Measuring Outcomes in Relation to SCP Core Elements, presentation, (Dublin, Educational Disadvantage Centre, St Patricks College). Available at: https://www4.dcu.ie/sites/default/files/edc/pdf/scpnewbjan2011.pdf , accessed 11 August 2011.

Ebrahim, A. & Rangan, V.K. (2010) The Limits of Non Profit Impact: A Contingency Framework for Measuring Social Performance, working paper 10-099, (Boston, Harvard Business School). Available at: http://www.hbs.edu/faculty/Publication%20Files/10-099.pdf, accessed 7 January 2012.

Gaze, E. & Hunter, R. (2009) Access to justice for discrimination complainants: courts and legal representation', University of New South Wales Law Journal, 32, p. 699-724.

Hansen, J. (2011) Changes to Legal Aid in New Zealand, International Legal Aid Group Newsletter, No. 21, 30 September (Wellington, International Legal Aid Group). Available at: http://www.internationallegalaidgroup.org/index.php/newsletter/category/34-2011, accessed 7 January 2012.

Human Rights and Equal Opportunity Commission. (1997) Bringing them Home: Report of the National Inquiry into the Separation of Aboriginal and Torres Strait Islander Children from their Families. Available at: https://www.humanrights.gov.au/publications/bringing-them-home-report-1997 , accessed 19 June 2014.

Idaho Legal Aid Services. (2011) Idaho Legal Aid Client Satisfaction Survey. Available at: http://app.fluidsurveys.com/s/ILAS-ClientSatisfactionSurvey/, accessed 12 August 2011.

Legal Aid Agency. (2013), Independent peer review of legal advice and work: final process paper, April, United Kingdom Government. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/512862/peer-review-process-paper.pdf , accessed [ 12 September 2014]

Kilian, M. (2010) 'To pay or not to pay? that is the question: the impact of individuals on the perception of the legal system', Paper to the International Legal Aid Conference, Cambridge 2010.

Kirby, M. (2004) Law and justice in Australia: room for improvement, Queensland University of Technology Law and Justice Journal, 4(2), p. 1.

Kusek, J.Z. & Rist, R.C. (2004) A Handbook for Development Practitioners: Ten Steps to a Results Based Monitoring and Evaluation System (Washington, DC, the International Bank for Reconstruction and Development/The World Bank).

Law Council of Australia. (2012) Best Practice Guidelines for Lawyers doing Family Law Work, (Canberra, Family Law Council and Family Law Section), 2^nd ftn. Available at: http://www.familylawsection.org.au/resource/BestPracticeGuidelinesv8FINAL.pdf , accessed 14 September 2011.

Law Council of Australia. (2014) Chronically underfunded legal aid commissions suffer further cuts in federal budget, media release, no. 1412, Canberra, 13 May. Available at: http://www.lawcouncil.asn.au/lawcouncil/images/LCA-PDF/mediaReleases/1412_--_Chronically_underfunded_legal_aid_commissions_suffer_further_cuts_in_federal_budget.pdf , accessed 20 May 2014

Legal Aid Australian Capital Territory. (2014) Legal Aid ACT Quality Legal Services: report of survey results. Available at: http://www.legalaidact.org.au/pdf/Legal_Aid_ACT_Quality_Legal_services.pdf , accessed 1 April 2014

Loddon Campaspe Community Legal Centre Advocacy Health Alliance (LCCLCAHA), Advocacy and Rights Centre. (2013) Loddon Campaspe Community Legal Centre, Bendigo, Victoria. Available at: http://lcclc.org.au/programs/advocacy-health-alliance/, accessed 13 December 2013.

McCutcheon, G. & Jung, B. (1990) Alternative perspectives on action research, Theory into Practice, 29(3), pp. 144-51.

Moorhead, R. & Robinson, M. & Matrix Research and Consultancy. (2006) A Trouble Shared: Legal Problems Clusters in Solicitors' and Advice Agencies, DCA Research Series 8/06, (London: Department of Constitutional Affairs). Available at: http://orca.cf.ac.uk/5184/ , accessed 12 August 2011.

Moorhead, R., Sher, A. & Paterson, A. (2003) Contesting professionalism: legal aid and nonlawyers in England and Wales, Law and Society Review, 37(4), pp. 765-808.

Mowles, C., Stacey, R. & Griffin, D. (2008) What contribution can insights from the complexity sciences make to the theory and practice of development management?, Journal of International Development, 20, p. 808.

National Association of Community Legal Centres (NACLC). (2005). National PII Scheme for CLCs, Appendix 1, Professional Indemnity Insurance: Cross Check Questionnaire, July. Available at: http://www.communitylaw.org.au/cb_pages/images/Cross%20Check%20Questionnaire.pdf , accessed September 2014

National Association of Community Legal Centres (NACLC). (2013) $16 million cuts to the CLC Sector misguided and short sighted, media release, 17, December. Available at: http://www.naclc.org.au/cb_pages/files/%2419m%20cuts%20to%20CLCs%20short%20sighted%20and%20misguided%20says%20NACLC%20FINAL.pdf , accessed 21 December 2013.

National Association of Community Legal Centres (NACLC). (2014a) National Association of Community Legal Centres Inc, Sydney, New South Wales. Available at: http://www.naclc.org.au/index.php, accessed 9 May 2014.

National Association of Community Legal Centres (NACLC). (2014b) NACLC strongly supports calls by National FVPLS Forum for reinstatement of direct funding, media release, 20 August. Available at: http://www.medianet.com.au/releases/release-details/?id=808575 , accessed 12 September 2014

National Association of Community Legal Centres (NACLC). (2014c) National Accreditation Scheme, National Association of Community Legal Centres Inc, Sydney, New South Wales. Available at:. http://www.naclc.org.au/cb_pages/accreditation_spp.php, accessed [12 September 2014].

National Family Violence Prevention Services. (2012) National Family Violence Prevention and Legal Services, Collingwood, Victoria. Available at: http://www.nationalfvpls.org/, accessed 20 September 2012.

Noone, M. (2001) The state of Australian legal aid, Federal Law Review, 37, p. 29.

Noone, M. & Digney, K. (2010) "It's Hard to Open up to Strangers" Improving Access to Justice: The Key Features of an Integrated Legal Services Delivery Model, research report, (Melbourne, Australia, Legal Services Board and La Trobe University). Available at: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1799648, accessed [9 May 2014].

Noone, M. & Tomsen. (2006) Lawyers in Conflict: Australian Lawyers and Legal Aid, (New South Wales, The Federation Press).

O'Brien, R. (1998) An Overview of the Methodological Approach of Action Research, (Toronto, Faculty of Information Studies, University of Toronto). Available at: http://www.web.ca/~robrien/papers/arfinal.html, accessed [12 August 2012 ].

Our community. (2013) Community legal centres targeted in Coalition cuts, Funding Centre news, December. Available at: http://www.fundingcentre.com.au/news/2013/12/legal-centres, accessed 1 December 2013

Paterson, A. & Sherr, A. (2007) Peer review and quality assurance- the emergence of peer review in the legal profession, Paper presented at the International Legal Aid Conference, Antwerp, June.

Price Waterhouse Coopers. (2009) Economic value of Legal Aid: analysis in relation to Commonwealth funded matters with a focus on family law, National Legal Aid. Available at: http://www.nationallegalaid.org/assets/General-Policies-and-Papers/Economic-Value-of-Legal-Aid-6-Nov-2009.pdf , accessed 17 December 2013

Ryan-Collins, J, Sanfilippo, L. & Spratt, S. (2007) Unintended Consequences: How the Efficiency Agenda Erodes Local Public Services and a New Public Benefit Model to Restore them, New Economics Foundation. Available at: http://www.i-r-e.org/bdf/docs/a007_a-new-public-benefit-model-for-public-services.pdf , accessed 12 December 2011.

Sackville, R. (2001) Access to justice: towards an integrated approach, Judicial Review, 10(2), p. 221.

Schetzer, L. (2003) Community legal centres: resilience and diversity in the face of a changing policy environment, Alternative Law Journal, 31, p. 159.

Schwarz, M. & Allison, F. & Cunneen, C. (2013) The Civil and Family Law Needs of Indigenous People in Victoria: A Report of the Australian Indigenous Legal Needs Project, (Cairns, James Cook University). Available at: https://www.jcu.edu.au/__data/assets/pdf_file/0011/121889/jcu_131180.pdf , accessed 17 January 2014.

Sherr, A. & Moorhead, R., Paterson, A. (1994) Lawyers: The Quality Agenda - Assessing and developing competence and quality in legal aid- transactional criteria, Vol. 2, (London, H.M. Stationery Office).

Smith, M. & Patel, A. (2010) Using Monitoring Data: Examining Community Legal Advice Centres Delivery, (London, Legal Services Commission, United Kingdom Ministry of Justice). http://webarchive.nationalarchives.gov.uk/20130128112038/http://www.justice.gov.uk/downloads/publications/research-and-analysis/lsrc/2010/CLACMonitoringData.pdf , accessed 7 January 2012.

Smith, R. (2011) Proposals for the Reform of Legal Aid in England and Wales: A Summary, International Legal Aid Group Newsletter, No. 16, November & December (London, International Legal Aid Group). Available at: http://internationallegalaidgroup.org/index.php/newsletter/category/35-2010, accessed 7 January 2012.

Stubbs, J & Assoc with Lux, C. (2011) Review of Public Purpose Funded Projects 2008-2011, (Sydney, Legal Aid New South Wales).

Victoria Legal Aid. (2012) Summary Report, Client Satisfaction. (Melbourne, Victoria Legal Aid). Available at: https://www.legalaid.vic.gov.au/about-us/what-we-do/research-and-analysis/client-satisfaction-surveys , accessed 11 October 2013.

Trude, A. & Gibbs, J. (2010) Review of Quality Issues in Legal Advice: measuring and costing quality in asylum work', Lawyers Defending Human Rights, Refugee and Migrant Justice (London, Information Centre About Asylum and Refugees and the Immigration Advisory Service, City University).

United Nations Development Program (UNDP). (2010) Overview of the United Nation's Development Program's (UNDP) Approach to Measuring Capacity, (New York, Capacity Development Group, Bureau for Development Policy, United Nations Development Program).

Notes

[1] Legal Aid ACT is the Legal Aid Commission of the Australian Capital Territory (LAACT).

[2] Elizabeth Curran considered literature from overseas before embarking on the design. This informed the design, for example, the peer review process in the United Kingdom which assisted in the identifying of some questions and standards. However much of the research had limited relevance in view of the lack of funding for such research in Australia and the cost to replicate it. In addition, the setting and contexts for such research evaluation in the United Kingdom and Canada are very different. Key among this is the commitment of governments over time in those jurisdictions to fund such measurement in the United Kingdom and Canada. In the United Kingdom comparatively, there has been investment over time borne out by the detailed work and trials of peer review that occurred before it was rolled out. As Paterson observed 'peer review is expensive to implement'(Paterson & Sher 2007, p. 22) ( By way of an example in Canada, research has been commissioned over many years by the Department of Justice in Canada even though more recently it is being reduced (see Currie 2003).

Unlike the United Kingdom where peer review is funded by the British Government through the Legal Aid Agency, individual legal services are expected to pay for research and evaluation of their services in Australia. See http://webarchive.nationalarchives.gov.uk/20130128112038/ and https://www.gov.uk/guidance/legal-aid-agency-audits Moorhead et.al (2003); Legal Aid Agency (2013).

One of the few occasions where government has provided funding to review services in Australia has been the 2012 Allen Review which cost taxpayers $0.96 million. This was the first significant spending in four decades on any review of the LAS across Australia. It was largely done by stakeholder consultation and focus groups of the service providers rather than detailed analysis of work undertaken by LAS at a micro level. (See Allen Consulting Group 2014).

[3] For example, see the Commonwealth of Australia (2004). This limitation of resources is also recently acknowledged in the Australian Productivity Commission's report titled 'Access to Justice Arrangements' (2014).

[4] Now known as the ACIL Allen Consulting Group.

[5] Many LAS in Australia use a form of internal peer review in which supervising lawyers periodically review the files of junior lawyers to ensure cases are progressing satisfactorily. Another form of peer review is 'cross-checks which involves the following: community legal centres from one centre visit another centre and examine their files with a check list and then report against the checklist. ………… While these review processes are significantly less costly than using independent peer reviewers they share the same limitations, namely, the absence of client and other stakeholder assessments of service quality and impacts.

[6] The NPA is an agreement between the Commonwealth, state and territory governments for the funding of legal aid commissions and the provision of services by other Commonwealth funded legal assistance services. The NPA commenced in July 2010 and expired on 30 June 2014 with a further NPA negotiated in September 2015 to commence on 1 July 2015 - 30 June 2020. While the objectives of the NPA relate to the provision of legal assistance by all services funded by the Commonwealth (including community legal centres and Aboriginal and Torres Strait Islander legal services) the NPA only makes provision for funding and reporting in relation to services provided by legal aid commissions.

[7] See Council on Social Action (2009).

[8] This is based on the estimated cost of undertaking a follow up 'snapshot survey' in December 2012 using the survey instruments developed in the LAACT research. A breakdown of the 35 working days of project officer time was:

· 10 working days for preparation, including tailoring the survey instruments to the services to be surveyed in this 'snapshot', liaising with staff and focus group discussions

· 15 days for the survey period, including data collection and staff interviews, and

· 10 days data entry, analysis and preparing the survey report.

[9] For example, the Services Marketing Quarterly (formerly the Journal of Professional Services Marketing) and Journal of Retailing which were not particularly relevant to the aims of this evaluation.

[10] Noone and Tomsen have documented the different historical and philosophical underpinnings of the legal assistance sector in Australia, Noone, M. & Tomsen. (2006) Lawyers in Conflict: Australian Lawyers and Legal Aid, (New South Wales, The Federation Press), 50-104; 199-242.

[11] This limitation of resources is also recently acknowledged in Productivity Commission (2014); ALRC (1999, 2000); Gaze & Hunter (2009); Kirby (2004); Noone (2001); Sackville (2001); Schetzer (2003). See also for comparative funding levels for legal aid, Price Waterhouse Coopers (2009, p.11).

[12] New South Wales, Victoria, Queensland, South Australia, Western Australia, Tasmania, the Australian Capital Territory and the Northern Territory.

[13] See Curran, E. (2012a).

[14] See National Family Violence Prevention Legal Services (2012).

[15] See NALC 2005.

[16] See National Accreditation Scheme (NALC 2014c).

[17] National Legal Aid represents the represents the directors of each of the eight state and territory legal aid commissions in Australia.

[18] The Allen Consulting Group, Review of the National Partnership Agreement on Legal Assistance Services: Draft Evaluation Framework Discussion Paper, 7, 10, 12, 27-28, 45, 48.

[19] See related media Law Council of Australia (2014), Our community (2013), Australian Broadcasting Commission (2013), Bond (2013) and NACLC (2014b).

[20] See Australian Financial Review (2014).

[21] See Australian Productivity Commission (2014, p. 22) and Curran, E. (2014)

[22] See O'Brien (1998), McCutcheon & Jung (1990) and Dick (1999/2011).

[23] Under the solicitor's conduct rules in most states and the soon to be introduced Australian Solicitor's Conduct Rules (adopted by Queensland, South Australia, Victoria and NSW) legal practitioners' paramount duty is to the court and to give independent and fearless legal advice. Also Trude and Gibbs in the United Kingdom note that good indicators of quality should also include the professional obligations of lawyers (Trude & Gibbs 2010, p. 8).

[24] In the examinations of previous evaluations examined in Elizabeth Curran's Literature Review issues of under-resourcing of services being a factor in ability to deliver services effectively and comprehensively were noted. See Curran, E. (2012a).

[25] See Legal Aid Australia Capital Territory (2014) for the report including the research methodology, survey data and.

[26] See Barendrecht et al. (2006, p. 21). They examine the significant hurdles for measurement and conclude 'measuring access to justice is a challenge.' Some examples include: Paterson & Sherr (2007); Sherr & Moorhead et al. (1994); Moorhead & Robinson (2006) and Noone & Digney (2010).

[27] A further six-monthly snapshot was conducted in June 2012 using the methodology developed in the research and third snapshot is planned for the end of 2012. Thereafter snapshots of all service areas of LAACT will be conducted at least annually.

[28] The last three qualities were incorporated by Elizabeth Curran based on a conversation with a client as part of the research.

[29] Fifty out of a total of 65 staff attended these workshops.

[30] This was the only unsuccessful instrument: client call back rates were low. Staff attributed this to clients' desire to put their cases behind them and 'move on'.

[31] See report of survey results at Legal Aid Australian Capital Territory (2014).

[32] See, for example, Coumarelos et al. (2012) and Human Rights and Equal Opportunity Commission (1997).

[33] In the June 2012 survey clients were asked whether the problem they spoke to us about had caused them and/or their family any stress and, if so, how the stress affected their family. They were also asked whether as a result of the advice they received the stress been reduced, stayed the same, or increased. Responses included: "Yes, massive, increased blood pressure and my wife has a bad heart. Feel a bit eased now. Huge amounts, adverse effects to mental, physical and emotional well-being. Stayed the same but feeling more empowered - know my options."

[34] Request made by Michelle Leering, Executive Director, Community Advocacy and Legal Centre, Ontario, Canada. For further information see Community Legal Education Ontario (2013) and Curran, E. (2013a; 2013b).

[35] For further information see LCCLCAHA 2013.[35] Elizabeth Curran has run a series workshops in September and October 2013 for Legal Aid Ontario and Law Clinics Ontario and has also advised and evaluated the Consumer Action Law Centre Telephone Advice Line and the Advocacy and Rights Centre in the Loddon Campaspe Region in 2013 and 2014.

Username
Password
Remember me