Capturing success: is it time to rethink our results frameworks?

The challenge

Security and rule of law programming is not about digging wells or building schools. And immediate beneficiaries are often not children or the poorest of the poor. This makes demonstrating success and communicating tangible results difficult. When it comes to, for example, strengthening informal justice systems, restoring community trust in post-conflict settings, or facilitating political party dialogue in emerging democracies, there are no quick-impact stories, compelling pictures, or impressive numbers that can convince the public that development euros have been well-spent. At the same time, pressure from Western parliaments for measurable results that are linked to pre-defined objectives with baselines, targets and standard indicators that can be aggregated to overarching policy goals, is on the rise. The recently adopted motions in Dutch parliament on these issues are a case in point.

Adaptive programming

A complicating factor is that programs in the area of security and rule of law almost by definition take place in complex and dynamic environments that are difficult to understand. This means change processes are not known in advance and unlikely to follow a predictable path. Such programs are only successful if:

• they use a relatively long time-horizon that allows for analysis to drive program design;
• a monitoring practice is developed that can lead to changes of interventions and budget allocations along the way;
• interventions are designed to simultaneously address a concrete problem and promote learning about what sort of change is appropriate.

So, if we want to be able to capture success in programs that can change interventions to remain relevant when context changes, we also need flexible and context-specific results frameworks that allow plenty of room for adjustment of results and for qualitative measurement approaches. This implies that we need to let go of standard frameworks with linear change models and pre-determined quantitative indicators of change and targets, and embrace an adaptive programming approach.

In the sub-field of democracy and governance programming, similar discussions have been held between a range of donors and implementing partners. This happened recently in Brussels[1]  and in London[2]. So the topic of flexible results frameworks and adaptive programming is hot.

The Dutch Strategic Partnerships: an inspiring example

The Dutch Ministry of Foreign Affairs’ Strategic Partnerships (SP) for Dialogue and Dissent is an exemplary funding instrument that encourages adaptive programming and flexible results monitoring. NIMD and the Association of European Parliamentarians with Africa (AWEPA) have started such an SP called “A Conducive Environment for Effective Policy Influencing: the Role of Political Parties and Parliaments”. The program focusses on political parties, parliaments, and parliamentarians and the essential role they play in shaping an enabling environment for policy change, and in making sure that lobby and advocacy efforts by civil society fall on fertile ground.

For this SP, a broad Theory of Change guides country programming. The SP gives a five-year time-frame for program implementation, with explicit acceptance of change needed along the way, and allowing considerable room for budget adjustments to allow for these changes. No overarching indicator frameworks were made mandatory or set in advance. On the basis of this, and on the basis of the donor’s trust in the track record of SP partners, program documents were developed around pre-formulated outcomes only. A flexible monitoring system was set up using a menu of intermediate indicators that are contextualized and adjusted based on annual program context changes and learning.

Showing impact

But of course a flexible monitoring system comes with its own challenges. Firstly, aggregation of results across countries and across programs is not always possible, or doesn’t make sense, as indicators may be very country-context specific, or lose their relevance if interventions are adapted. This is not always desirable from a donor’s perspective, who needs to have a grip on how “things come together” across a portfolio of implementers. Secondly, qualitative results do not lend themselves well to tracking by tools such as the International Aid Transparency Initiative (IATI), that is aimed to create more transparency in aid flows and their results, and which is increasingly made mandatory for implementers. IATI activity files are organized around pre-set indicators, and built around the logic of inputs (budgets) being converted into quantifiable results with set targets, that are regularly updated. This is problematic, as it pre-supposes a linear change process that does not fit well with the monitoring of adaptive programs, where activities and their objectives most likely will change along the way.

Acknowledgement by donors that security and rule of law programs, including democracy and governance programs, by their nature need to be adaptive, and do not deliver a “quick win” is fundamental for these programs to flourish. It is important to assess how the flexible monitoring systems that are needed for these programs can best be aligned with the accountability frameworks used by donors and their parliaments, if at all. At the same time, to ensure the continued support for these programs in Western parliaments and with the public, implementers need to reach out and convey compelling stories about final beneficiaries and significant impacts that can be linked back to the interests of Western publics. Only if this outreach becomes an urgent and collective effort by both implementers and donors are we likely to convince the public of the added value of security and rule of law programming that takes place far from their backyards. 

Nic van der Jagt is Planning, Monitoring and Evaluation (PME) Coordinator at the Netherlands Institute for Multiparty Democracy (NIMD) in The Hague.

The recent series of lunch-meetings and an interactive brainstorm organized by the Knowledge Platform Security & Rule of Law and their colleagues from the Netherlands Institute for Multiparty Democracy (NIMD), PAX, Global Partnership for the Prevention of Armed Conflict (GPPAC), Cordaid and the Ministry of Foreign Affairs, included a discussion on implications of adaptive programming practice for monitoring and learning.