Download Concept Note
(MS Word 545kb)
Evaluating the performance of public policy is considered a fundamental ingredient to foster accountability, good governance, and improve programme effectiveness. Efforts to build and sustain effective evaluation systems face challenges of institutional design, political dynamics, technical capacity and resistance to change.
Successive UN General Assembly resolutions including the Quadrennial Comprehensive Policy Review of operational activities of the UN for development and UNDP Executive Board decisions have encouraged the UN development system and UNDP in particular to support national evaluation capacities1
The UNDP Evaluation Office (EO) organized two international conferences on National Evaluation Capacities, one in Casablanca, Morocco, in cooperation with the Moroccan National Observatory for Human Development in 2009, and another in Johannesburg, South Africa in cooperation with the Public Service Commission in 2011.
The UNDP Evaluation Office, in partnership with key players in evaluation will hold the 3rd International Conference on National Evaluation Capacities in Sao Paulo, Brazil on September 29 – October 2nd 2013.
1UNDP Executive Board DP/2011/3 Evaluation Policy of UNDP. New York, January 2011.
To enhance the understanding and appreciation of evaluation as a powerful tool of public accountability and learning in the countries, the conference has the following objectives:
- To share experiences, challenges and solutions from countries with national monitoring and evaluation (M&E) systems, including those that may be considering creating one and have important experiences with other types of evaluation efforts;
- To identify lessons and constraints in implementing national M&E systems and the use of evaluation; and
- To identify opportunities for technical cooperation in strengthening institutional capacity for national M&E systems under the umbrella of South-South and triangular cooperation.
The III conference on National Evaluation Capacities will continue to provide a platform to share experiences and expand on solutions to address common challenges related to the establishment of national monitoring and evaluation systems of public policies and initiatives. Recognizing that monitoring and evaluations are closely related, the conference will concentrate more on evaluation than on monitoring.
This year’s conference will have an emphasis on three interconnected aspects of establishing national M&E systems, namely, how to ensure the independence, credibility and use of evaluations. As the third exercise of this nature, the conference aims to learn from the experience gained by UNDP Evaluation Office of organizing two previous conferences in 2009 and 2011 and establish better follow up mechanisms for continuity of dialogues, but most importantly the continuity of networks, cooperation and of the agreements reached. Due to the participatory nature of the conference, the results will help improve international standards of evaluation in its application to the evaluation of public programmes, policies and initiatives.
The conference will be organized around three topics: independence, credibility and use.
Evaluating the performance of public policy can help deepen democracy by creating the conditions for holding governments accountable for their performance and increasing transparency. Evaluation of public action is embedded into the political, economic and social context of a particular country and needs to take the history and national context into account. Countries have different institutional configurations for assessing the performance of public actions and evaluation can be one of them. Given the complexity of the institutionalization of monitoring and evaluation systems, it is not surprising that there are a number of challenges identified at various stages of the process.
For evaluations to meet the requirements of independence, it generally implies freedom from political influence and/or organizational pressure. According to the norms and standards set out by the United Nations Evaluation Group (UNEG) the evaluation is said to be independent when it is "objective, free from undue influence, and has full authority to submit reports directly to appropriate levels of decision-making”. This definition also applies that the management must not impose restrictions on the scope, content, comments and recommendations of evaluation reports, and, in order to avoid conflict of interest, evaluators must not be directly involved in policy-setting, design, implementation or management of the subject of the evaluation, adding that the evaluators should have full access to all relevant information required for the evaluation.2
The conference will address the challenges faced by governments when establishing monitoring and evaluation systems that are considered independent. It will answer question such as, who is responsible for evaluation of public programmes and policies in the country. Who commissions the evaluations? Is the evaluation function located independently from other management functions? How is the conflict of interests or undue pressure over evaluators avoided? Can the evaluations be submitted to the appropriate level of decision-making? The independence of the evaluation function and its location in the public administration can trigger interesting discussions in the conference.
The conference will also address the complementary role that could be played by self-evaluation and independent evaluation. How are they interrelated? Are mechanisms in place for “checks and balances” between spheres of the state, for instance between executive and legislative branches or between the executive and the judiciary? How do citizens participate in the process?
2UNEG Norms and Standards for Evaluation in the UN System. New York 2005. This definition of evaluation independence is in line with the one provided by the Development Assistance Committee (DAC) of the OECD in the Glossary of Key Terms in Evaluation and Results Based Management, 2002.
The credibility of evaluation usually depends on the expertise and independence of the evaluators, the degree of transparency of the evaluation process, and the quality of evaluation outputs. Credibility requires that evaluations should report successes as well as failures. Beneficiaries of programmes or policies should, ideally, participate in evaluations in order to promote credibility and commitment3.
The conference will address the challenges faced by governments when ensuring the credibility of evaluation systems. It will answer question such as, are evaluations designed, planned and implemented following international quality standards in evaluation? Are the evaluations explicit in the use of methodologies for data collection, analysis and interpretation? Do evaluators have the competency to conduct evaluations? Do they have access to all the required information to conduct the evaluations? How do the systems ensure the quality of evaluations? Are evaluations conducted in an ethical way? Is data integrity respected or is there manipulation of quantitative and qualitative data to influence findings and conclusions? Do evaluations consider issues linked to human rights, gender equality, ethnicity and cultural behavior? How is credibility in evaluation processes enhanced and ensured?
The conference will also address the complementary role between monitoring and evaluation. While recognizing that a large amount of resources are dedicated to monitoring the performance of public programmes policies or initiatives, the evaluation will concentrate on the evaluation function. There is a need for a strong link between monitoring and evaluation. However it is important to mark the distinction because only evaluation can question the assumptions behind public action. In that sense, evaluation can question if the State is doing things right and also if it is doing the right things, something that a monitoring exercise is not able to assess.
3For indicators of credibility look at UNEG Norms 2.5, 3.1, 4.2, 5.2, 8.1, 8.2, 9.1-9.3, 10.1 and 10.2 www.uneval.org/documentdownload?doc_id=21&file_id=562
For evaluation to have an impact on decision-making, evaluation findings must be perceived as relevant and useful and be presented in a clear and concise way. They should fully reflect the different interests and needs of different stakeholders. Importantly, ensuring the utility of evaluations is only partly under the control of the evaluation function. It is also a responsibility of decision-makers within the State and societal structures in commissioning, receiving and using evaluations.
The conference will address the challenges faced by societies for better use of evaluation systems but will go one step ahead of the 2011 NEC conference (that focused on the use of evaluation) by, on the one hand, assessing the country cases in which use of evaluation has moved significantly ahead, and, on the other hand, portraying documented cases of linkages between use of evaluation and independence and credibility in conduction evaluations.
The conference will address the challenges faced by societies for better use of evaluation systems. It will answer questions such as: Are there incentives to motivate potential users to demand and/or to use the evaluations as input for their own decision-making processes? For example, minister councils where they must give accounts, and so on. In your civil society, is the public demanding information from evaluation commissioners? Which stakeholders seem to be most interested in demanding and using the evaluations and why? Which should demand it but don’t, and why? Are the reports publicly available? What happens after an evaluation is conducted?
The conference will also address linkages between evaluation and other processes such as planning and budgeting. Is the availability of the evaluation synchronized to the government cycles and decision making processes? For example, the budgetary and the planning cycles at the country and agency levels. Are evaluations linked to planning processes? What is the timing for evaluations? Are evaluations linked to budgetary allocations? How is accountability ensured? What can be learned from evaluations?
Are the evaluations being demanded and used? Are evaluations being used for accountability or marketing purposes, without consideration for learning or formative intents? How relevant and pertinent do/have primary and secondary users find/found the evaluation, given their own needs for data to inform their work? Does using evaluations produce changes in policy actions? Does/did the use of evaluations by users produce real changes related to the evaluation purposes? e.g. in policy planning and/or budget planning and allocation, corrections or adjustments in programs and services, higher levels of accountability. Are approaches considered in the evaluations that would call attention to use of the evaluations’ findings in an inclusive and comprehensive way for decision making, appropriately considering gender equality and human rights?
The conference will adopt the form of a dialogue. Each topic will be discussed separately and in sequence by all the participants. Three discussion papers will be commissioned to evaluation experts as input for the dialogue, one on independence of the evaluation function, another on credibility of evaluations and a last one on use of evaluation to inform decision-making. Each paper will pose questions to the participants to start a dialogue. The papers can identify sub-themes for its treatment in smaller groups within a topic. The authors of the papers will be invited as key note speakers. A dialogue seeks to collectively arrive to answers to the questions. It is not the same as a debate in which the participants are trying to prove the validity of their propositions. A dialogue is a common search for meaning as ways of addressing common challenges. Participants will then be invited to share their national experience in addressing the challenges under analysis. Professional facilitators will be hired to orient the dialogues towards identifying answers to the questions.
The conference will have a space for open exchange of experiences in the way of posters or stands, therefore opening opportunities for direct interaction in addition to the more structured dialogues. Each poster or stand will be preceded by the presentation of extended abstracts (2-4pages) in which the participants will describe the role played by the institutions in addressing one of the conference topics. EO will issue a call for abstracts with all the details required for presentation and participation. The abstracts will be shared in advance and will constitute a critical input for the final publication.
The conference will facilitate the direct exchange of experiences by participants by organizing trade tables or direct matching exercises between supply and demand. It will work with a specific methodology to facilitate such exchanges based on the experience of UNDP Bureau for Development Policy and the Bureau for Latin America and the Caribbean.
The Evaluation Office of UNDP has identified that the challenges faced in establishing national M&E systems of public policies are not exclusive to developing countries. In that sense it will invite representatives of national institutions responsible of commissioning, conducting and using evaluations of public policies from programme and non programme countries. This approach will facilitate the exchange of experiences and potential learning from each other, but a special focus this year to stimulate continuation of dialogues and potential technical cooperation, commitments and agreements.
- The representatives of national institutions commissioning evaluations will deal mainly with the topic of the independence of the evaluation.
- The representatives of institutions conducting evaluations will be invited to contribute to the dialogue on the credibility of evaluations.
- The representatives responsible for using evaluations in the national decision-making process will be invited to contribute to the dialogue on utility of evaluations.
The Evaluation Office will extend invitations to a number of institutions to reach a critical mass for discussion. It will seek to keep a regional balance in the representation, although it recognizes that the conference is in Brazil and a larger representation of North America, Latin America and the Caribbean is expected.Representatives of international cooperation will participate in the dialogues and as a follow up to the conference will identify ways to support South-South and triangular cooperation, in the form of specific knowledge exchange among participants or technical cooperation.
The expected deliverable of the conference will be a publication that would ideally present the state-of-the-art in each of the three topics. The publication will be based on the abstracts prepared in advance of the conference and on the transcription of the dialogues and exchanges of experiences that will take place during the conference. There will be an explicit request to reference all the practices and experiences shared in the conference.
EO will put additional efforts to develop a structure for following up on bilateral and trilateral exchange of experiences in the way of cooperation programmes. For that it will follow methodologies that will foster engagement of different participants in dialogues aimed at promoting mutual collaboration and leading to south –south cooperation in the area of evaluation. EO will address this effort in partnership with other development partners.