Navigating measurement challenges in Rule of Law programming

Measuring and reporting on Rule of Law programming is demanding, as discussed at a recent event of the Knowledge Platform Security & Rule of Law on programming for complex development with Rachel Kleinfeld. Yet, as Pamela Kovacs, Research and Learning Officer at IDLO argues, there are tools available for this, and embracing this complexity is worthwhile.

The rule of law matters

The importance of rule of law for development is underscored in the universal 2030 Agenda for Sustainable Development, with the inclusion of Goal 16 and its rule of law focus. Yet the global participatory process to adopt this Agenda placed a spotlight on the insufficient knowledge about how to effectively promote the rule of law and more tangibly, to measure and report on this progress.

Programming is demanding

The recent Platform event on programming for complex environments with Rachel Kleinfeld helpfully reflected further on navigating these issues, drawing from research and capturing the project cycle with an image of a meandering sailboat driven by irregular wind patterns rather than a chugging train on a straight set of rails. Visually, this is perhaps more intuitive than problem-driven iterative adaptation (PDIA), an approach reflecting similar guiding instincts, signalling the need for greater time and space continuums for programming aligned to local realities.

Combined with intersecting foci (bottom-up and top-down) and sequencing (institution and end-user), current rule of law programming must incorporate an array of factors - environmental, legal, social, economic and political - that not only require deep knowledge, but careful and deep thinking in response to an array of what have been elsewhere described as critical impediments to programming. 

But it is so important

When conceptualized as strengthening laws, building institutions, and empowering people, the rule of law intrinsically matters for sustainable development. As a concept of justice and as both an outcome and means of development, making rule of law programming accessible for donors (and their constituents) faced with smaller aid budgets and an array of pressing needs is necessary to ensure and prioritize funding.

The drive to enhance aid effectiveness means donors and development organizations must manage for results and show and continuously improve the effectiveness of their work.

Increasingly, this translates to pressure to move away from rigid thinking, focused on activities and outputs, and to put more emphasis on development and testing of strategies that lead to change at the (intermediate) outcome level of results.

Even when it is complex

Unpacking this apparent contradiction is complex – achieving results requires greater flexibility and agility, but achieving aid effectiveness requires greater scrutiny via monitoring and reporting. And, as noted, measuring itself can be political.

It is measurable in meaningful ways

In 2014, IDLO released a report, Doing Justice, which canvassed measurement challenges related to rule of law interventions. The Report noted that such challenges are not unique to the rule of law sector, and are often exacerbated in transitional and post-conflict settings. The Report also signalled lessons learned from the Millennium Development Goals (MDGs) and the need to bring forward evidence on rule of law measurement approaches and considerations, demonstrating a variety of approaches to indicators (which can include objective change, structural, basket, and/or composite indicators).

Both quantitative and qualitative approaches are needed to measure and document institutional capacity, objective change and public perceptions of change in order to ensure meaningful measurement and a complete picture of progress in any particular area of the rule of law. A core lesson of the MDGs was the perverse nature of outcomes that can arise without due diligence.

There are tools – and they have to be used in the right way

 In practice, this due diligence is exercised through the tools and processes available to develop, implement, adjust, and implement, all while monitoring and reporting (bi-weekly, monthly, quarterly, and annually!) on programs or interventions.

Tools and processes such as logic models, log frames, intervention logics, PDIA, and theories of change provide a compass to set direction, but then must respond to implementation with all of its navigational challenges. This means anticipating and adjusting for changes in complex environments - questioning what is and is not leading to positive change? Under what conditions? And how can such positive change be catalyzed further and sustained by journey’s end?

Embracing this complexity is worthwhile

Generating answers to these questions is, in considerable part, a matter of engaging with and learning from practice to develop and test concrete theories of how change occurs in specific circumstances. Capturing these lessons for use in program design or as ‘building blocks’ for evidence-based theories of change completes a circle of doing, measuring, assessing, and learning.

Like many development organizations, IDLO is committed to this type of learning and through a Lessons Learned Program, annually draws lessons from its projects, with the aim to learn from program implementation and the types of interventions relied on to foster change, tackling challenges and sharing good practices both internally as well as with the broader rule of law community. An overarching lesson is that the how of rule of law promotion must be infused with equity and substantive justice objectives to achieve long-term success.

Knowledge sharing events which ask hard questions and bring together perspectives and research are vital, as rule of law programming remains a value-laden global journey, not a destination.