Integrated approaches are key to ensuring the poorest in least developed countries aren’t left behind

By Karina Weinstein, Program Director, FXB International

Photo Credits: FXB International

A recent analysis presented by the UN Conference on Trade and Development (UNCTAD) to its member States on February 5, 2018 revealed that the world’s poorest countries (Least Developed Countries, LDCs) are not on track to achieve goals set out in the 2030 Agenda for Sustainable Development (UN News Centre). The imperative behind the Sustainable Development Goals (SDGs) is equity, ensuring that no one is left behind. Yet the gap between the 47 countries labeled as Least Developed Countries (LDCs) and other developing countries is widening. What can be done?

Pictured in the UNCTAD article is Olivia Nankindu, a mother of two in Kampala District, Uganda. She was living in abject poverty on her husband’s income from hawking plastic bags. Unable to afford meals or pay rent, Olivia’s family participated in FXB’s integrated Community Development FXBVillage program. Provided with integrated support in nutrition, housing, health, education, and livelihoods, Olivia gained confidence, skills and financial capacity. Olivia not only improved her family’s nutrition, children’s educational attainment, and sanitation and hygiene, but also gained hope and confidence that her poverty is not immutable. In 2013, she enrolled in an apprenticeship program in hairdressing and salon management, and landed a job in a hair salon, earning UGX 200,000 per month (approximately $56 USD), which is about four times the amount of her husband’s income before the start of the program. Now, Olivia and her husband are planning to start their own hair salon business. “I now work harder, follow my desires, goals and my principle is to never give up easily,” Olivia says. She is one of over 84,000 people who were able to escape extreme poverty through FXB’s integrated programs.

Integration is key to achieving the SDGs and bridging the gap between the LDCs and other developing countries. We cannot afford to leave anyone behind. As an international non-governmental organization currently working in four of the LDCs (Burundi, Uganda, Rwanda and Myanmar), FXB is committed to inclusion and integration. We do that by focusing on the ultra-poor, not only through the FXBVillage program, but also through our health, education, and women’s and children’s protection programs. FXB advocates for integrated approaches to community development through our participation in Locus, a coalition of organizations dedicated to advancing evidence-based solutions to global development challenges that are integrated, driven by local communities, and based on shared measures. Together, we are amplifying the evidence that integration works and that women like Olivia cannot be left behind.

    Photo Credits: FXB International

ICCE TIG WEEK: Using M&E data to evaluate the impacts of an integrated and holistic international development model

By: Ashweeta Patnaik, Ray Marshall Center 

http://aea365.org/blog/icce-tig-week-using-me-data-to-evaluate-the-impacts-of-an-integrated-and-holistic-international-development-model-by-ashweeta-patnaik/

Hello, I’m Ashweeta Patnaik and I work at the Ray Marshall Center (RMC) at The University of Texas in Austin. RMC has partnered with Nuru International (Nuru) to use Monitoring and Evaluation (M&E) data to evaluate the impacts of Nuru’s integrated development model. Here, I share some lessons learned.

Nuru is a social venture committed to ending extreme poverty in remote, rural areas in Africa. Nuru equips local leaders with tools and knowledge to lead their communities out of extreme poverty by integrating impact programs that address four areas of need: hunger; inability to cope with financial shocks; preventable disease and death; and, lack of access to quality education for children. Nuru’s M&E team collects data routinely to measure progress and drive data based decision making.

Lessons Learned:

1. Establish a study design to measure program impact early – ideally, prior to program implementation.

Nuru has a culture where M&E is considered necessary for decision making. Nuru’s M&E team had carefully designed a robust panel study prior to program implementation. Carefully selected treatment and comparison households were surveyed using common instruments at multiple points across time. As a result, when RMC became involved at a much later stage of program implementation, we had access to high quality data and a research design that allowed us to effectively measure program impacts.

2. When modifying survey instruments, be mindful that new or revised indicators should capture the overall program outcomes and impacts you are trying to measure.

Nuru surveyed treatment and comparison households with the same instruments at multiple time points. However, in some program areas, changes made to the components of the instrument from one time-point to the next led to challenges in constructing comparable indicators, affecting our ability to estimate program impact in these areas.

3. Monitor and ensure quality control in data entry, either by using a customized database or by imposing rigid controls in Excel.

Nuru’s M&E data was collected in the field and later entered into Excel spreadsheets. In some cases, the use of Excel led to inconsistences in data entry that posed challenges when using the data to analyze program impact.

4. When utilizing an integrated development model, be mindful that your evaluation design also captures poverty in a holistic way.

In addition to capturing data to measure the impact of each program, Nuru was also mindful about capturing composite programmatic impact on poverty. At the start of program implementation, Nuru elected to use the Multidimensional Poverty Index (MPI). MPI was measured at multiple time points for both treatment and comparison households using custom built MPI assessments. This allowed RMC to measure the impact of Nuru’s integrated development model on poverty.

 

Are we learning enough about integrated programs through impact evaluations?

By: Tessa Ahner- McHaffie, FHI360

https://blog.f1000.com/2017/11/28/learning-enough-integrated-programs-impact-evaluations/

Integrated development raises a lot of questions. Tessa Ahner-McHaffie and colleagues from FHI 360 tried to answer some of these in a Systematic Review published on Gates Open Research and currently being peer reviewed. In this guest blog, she describes the work that her and colleagues have done in exploring the evidence around this area of development work. 

As someone working on ‘integration’ at FHI 360, a large non-profit dedicated to improving lives in lasting ways by advancing integrated, locally driven solutions, I often get asked ‘does integration work?’ To answer questions like ‘where does working in multiple sectors add value?’ or ‘what does the evidence say about multi-sector integration in this context?’ we first have to determine what evidence is available, and what research questions are being asked and answered. So, my team and I set out to do that by conducting a systematic review to find impact evaluations on integrated strategies, and see how those evaluations were being designed.

Gathering the evidence

[pullquote]Integration is an umbrella phrase that can describe thousands of different cross-sector approaches — from health and microfinance, to nutrition and education, to conservation and livelihoods.[/pullquote]

Conducting a systematic review on something as broad (and often fluid) as integrated development was, from the beginning, difficult. Conducting keyword searches for “integration” – which we defined as any program working across one of the core development sectors agriculture and food security, economic development, education, environment, governance, health, nutrition, and water, sanitation and hygiene (WASH) – was nearly impossible, since people do not consistently use this phrase in titles or abstracts.

So, we needed to come at it from a different direction. We decided to start from the entire realm of impact evaluations, and manually screened all of the 4,000+ entries in 3ie’s Impact Evaluation Repository, looking for which studies involved an integrated program.

The result? We found a lot more evaluations than we originally expected: 601 peer-reviewed impact evaluations on integrated, multi-sector projects. These studies crossed every sector and region. Due to the variability, we were unable to conduct a meta-analysis on the effectiveness of the projects studied. We were, however, able to find some interesting trends on what is getting studied (and what is not).

What the evidence told us

We found that integrated, multi-sector programs are being evaluated, both rigorously and often. Most report positive findings, although we need to consider publication bias, so we can say little beyond that these programs work sometimes. Yet, the majority of these studies did not consider integration as a component of the program to be evaluated, or an objective to monitor or measure in and of itself.

Only 38 had a partial or full factorial Randomized Controlled Trial (RCT) design, which would be the most straightforward way to look at the possible amplification of outcomes of multi-sector integration, although it exponentially adds to the time, resources, and complexity of the evaluation.

Importantly, there are other possible ways that integration can add value to a program beyond the final quantitative results: by improving the reach of the project to more vulnerable people, improving the satisfaction of the staff or community utilizing the project, creating cost efficiencies, in addition to others. Yet, although 60 assessments used a qualitative component, and 43 used a costing component, most of those were measuring some other angle of the program, and not deliberately unpacking the effect of integration.

[pullquote]Very few of these evaluations consider if this integration is adding to (or detracting from) the aims of the project.[/pullquote]

Moving forward, we know that a large proportion of those programs being studied combine multiple sectors in one package. We need to be more deliberate in our attempt to learn about the added value of combining the interventions as a component of outcome evaluations. Doing so does not necessarily require a complicated or expensive design each time – it can mean creatively using mixed methods and other strategies to explore the targeted effects of integration.

Why publish on Gates Open Research?

One reason we were excited to publish on Gates Open Research was the speed at which we would be able to start a dialogue about what we found. Having an open, thoughtful conversation through peer review that can be viewed by readers is appealing, and we are excited to be a part of it. We hope that funders, researchers, and program designers will use the vast set of evaluations we have gathered to dig into the sectors or models of their interest, use those to further analyze effectiveness, and inform new ideas, decisions, and learning in the future.

Gates Open Research provides all Gates Foundation-funded researchers with a place to rapidly publish any results they think are worth sharing. All articles benefit from immediate publication, transparent refereeing and the inclusion of all source data. If you are a grantee of the Gates Foundation you can find more information about how to publish on Gates Open Research here.

Creating New Tools for the Job

By: Lydia Cardona, Conservation International

Calapa seedling © Benjamin Drummond

Headlines about scarce natural resources driving or causing conflicts are not hard to find. Threats of war over the control and use of natural resources – such as oil, timber and water- become more imminent as rapid, unsustainable development and climate change increase the pressure on renewable and non-renewable resources. According to the United Nations Environment Programme, more than 40% of internal conflicts over the last 60 years are linked to the exploitation of natural resources.  However, conflict is not the only side of the intersection between peace and the environment. 

Collaboration around natural resource management can become a catalyst for peace, meaning that the arrow that links these issues points in both directions. Yes, conservation can contribute to conflict, but it can also support peacebuilding. In 2016, the United Nations General Assembly and Security Council unanimously passed joint resolutions on “Sustaining Peace”, an agenda focused on promoting conditions for peace across societies in a way that is preventative of conflict rather than a response to it. 

Throughout the history of our organization, Conservation International (CI) has worked to build cooperative institutions and improve policies and practices around natural resource governance. As a conservation organization that puts human well-being at the center of its work, CI is committed to proactively—rather than reactively—addressing potential conflict within its work from an integrated, locally-led perspective. 

In response to this, CI’s Policy Center for Environment and Peace is pleased to announce the publication of its Environmental Peacebuilding Training Manual, which focuses on increasing awareness, knowledge and skills on identifying drivers or root causes of conflict, analyzing conflict dimensions, fostering conflict sensitive program design and promoting consensus building among conservation practitioners and organizations working in areas affected by conflict or where conservation efforts could potentially impact conflict.  

This manual seeks to broaden practitioners’ awareness of these relationships and to provide specific tools for integrating environmental peacebuilding. Specific modules on stakeholder engagement, conflict analysis, conflict-sensitive programming, and collaborative consensus-building encompass a series of approaches, tools, and skills that are relevant to promoting peace and sustainability across CI’s global efforts. 

CI works in many fragile contexts, and our experience underscores the importance of this integrated approach to conservation and peace and the need for these tools. By creating a dialogue among stakeholders, encouraging local engagement, and helping communities with natural resource management, CI promotes peace in communities in Peru, Timor Leste, the Philippines, Bolivia and numerous other countries. Our facilitation of international dialogues and agreements has enabled the establishment of the Coral Triangle Initiative and the Cordillera del Condor peace park between Ecuador and Peru, which have minimized conflict and led to more coordinated dispute resolution. The manual showcases case studies which ground the relevance and adaptability of these approaches in CI’s local experiences around the world.  

Putting tools that make these links in the hands of staff and partners who are implementing at the local level is critical to supporting local decision-making and amplifying the voices of those impacted by and benefiting from conservation actions.  This is ultimately where the bridge between high-level agendas—both Sustainable Development Goals (SDGS) and Sustaining Peace—to local level dynamics is apparent and critical through CI’s work.

Environmental peacebuilding is an approach that can be transformational in addressing some of the human security challenges that climate change, population growth and resource degradation threaten to exacerbate.  CI is eager to share these tools and resources with our partners under the belief that, ultimately, collaboration and the broadening of perspectives around complex problems will provide the greatest benefit for both people and nature.

Our new Environmental Peacebuilding Training Manual can be accessed on CI’s Environmental Peacebuilding website. For questions or comments, please contact Janet Edmond, Senior Director of Peace and Development Partnerships at CI, at jedmond@conservation.org 

Creating an Integrated Nutrition Game-Plan: Technical Brief and Recommendations from Lao PDR

By: Deirdre McMahon, Global Nutrition Advisor, SNV 

Malnutrition isn’t the result of a simple cause-and-effect algorithm. It can’t be boiled down into an if-then statement. It’s the result of a complex web of underlying factors and interrelated causes. And in response to that complexity, reducing malnutrition requires convergent action from many different sectors and stakeholders. 

Based on the need for integrated action to improve nutrition, SNV applies a convergent approach to nutrition programming. For example in Lao PDR’s remote and ethnically diverse upland farming communities, SNV, with support from the Swiss Agency for Development and Cooperation (SDC) and in partnership with Agrisud International, works with local Government partners to improve family nutrition through the Enhancing Nutrition of Upland Farming Families (ENUFF) project. ENUFF brings together nutrition-sensitive agriculture, gender equity, positive social and behavior change (SBC), and water, sanitation, and hygiene (WASH) into one multisectoral nutrition program. 

ENUFF has gained valuable insights on realizing effective convergence of sectors and implementation across district, community, and household levels to improve nutrition. Drawing on this experience, the project has published ‘Converging for improved nutrition in Lao PDR’, a technical brief that provides key insights into the district level coordination and implementation of a multisectoral nutrition program and  defines strategies for improving nutrition through integrated programming. ‘Converging for improved nutrition in Lao PDR’ is available to download as a Full Brief or as a Summary. 

The brief details four strategic avenues to improve family nutrition in the Laotian uplands through integrated programming that ENUFF has identified based on the project’s implementation and findings from its baseline study: 

 1. Engage in nutrition and WASH social and behavior change communication (SBCC), which involves both women and men, especially in relation to children’s feeding and diets, dietary quality and diversity, improved hygienic practices, and use of improved sanitation facilities.  

2. Provide technical and organizational support to farms and small businesses to enhance local capacities to diversify food crop and small livestock production, and to process, store, market and generate income from food crop value chains. 

3. Promote, support, and engage in context-sensitive yet integrated initiatives, such as nutrition-sensitive home gardens and value chains, which cut across the health, agricultural, environmental, and business sectors and account for the multidimensionality of livelihoods and nutritional issues.

4. Support nutrition coordination committees, including recruiting the right coordinators, establishing mechanisms for accountability, decentralizing national level actions to the village level, and strengthening capacity at all levels including a process for learning and not just reporting. 

The need for improved nutrition in Lao PDR, particularly among women and children, is critical. Although the country has achieved impressive economic growth in recent years, its further progress is impeded by high malnutrition rates. The 2015 Lao Child Anthropometry Assessment Survey (LCAAS) reported that 36.5%, of children under five as  were stunted, with child malnutrition showing strong inequalities across regions and groups, particularly among communities living in remote upland areas where ENUFF is implemented. 

The underlying causes of this malnutrition are heavily influenced both by external factors, including food systems, the sanitation context, livelihood opportunities, and income generation, and by household dynamics and behaviors related to food production and purchase, care practices, and hygiene. Therefore, improving nutrition requires integrated action both among sectors (nutrition, health, education, agriculture, livelihoods, and women’s empowerment), and among governance and social strata (activities at the district, community, and household levels).  

By enacting these four strategies, ENUFF aims to improve households’ physical and economic access to nutritious foods through diversification of production and augmented livelihoods, while fostering inclusive, sustainable food systems and hygienic environments. The increased availability of nutritious foods and hygienic infrastructure will be accompanied with enhanced counselling at the household level of proper dietary needs, especially for children, men’s role in nutrition, proper hygiene practices, and the use of income increases for improved health and nutrition. These activities are coordinated and aligned across different sectors, and from the district to the household level. 

The desired key outcomes of this convergence of sectors and governance strata is to empower households with a set of options, decisions, and actions that together will lead to improved nutrition, in particular for women and children. When designing and implementing nutrition programming, the actions across sectors may or may not be joint, but at a minimum should be strategic and coordinated. By bringing sectors together in an integrated approach, we can address the complex and interrelated factors that comprise the root causes of malnutrition, and not just treat its symptoms.  

You can find more information about SNV and our work in nutrition on our website: www.snv.org  

One Thing at a Time Doesn’t Work for Women

By: Sia Nowrojee, Program Director, 3D Program for Girls and Women

Photo Credit: Sia Nowrojee, September 2017, Pune District, Maharashtra State, India

“There is no thing as a single-issue struggle because we do not live single-issue lives.”                     -Audre Lorde, activist and poet

Early in my career, I worked in a women’s health clinic as a counselor, supporting clients as they made and acted on their reproductive choices. It was a good way to learn how medical and psychosocial services combine to provide an overall experience of quality of care. It was an even better way to learn about the impact of gender on the decisions that women make.

Appointments often took hours, during which the women lamented the lost time they could have used to meet their many responsibilities: child care, working, attending classes, and more. They came from a variety of backgrounds – young and old, poor and wealthy, working in the home and outside the home – and they all had to make health and life decisions rooted in complex realities.

Those women have stayed with me through my career, infusing my work in gender and development with the understanding that life is complicated and multi-layered. Choices are made, sometimes with support and information, sometimes alone, with little to go on. Often you have little control over the decisions that affect you most. One decision is impacted by – and impacts – others. Most of us know this firsthand, as we juggle work schedules with those of our partners and children, and balance personal needs with professional and financial concerns.

I was reminded of this at a numeracy training I attended, organized by the waste pickers’ cooperative SWaCH, in Pune City, India. SWaCH members, mostly women, were learning to better manage their finances. The adult educator skillfully wove together math concepts with messages about dignity and agency, reinforcing that the women, who are family breadwinners, brought seasoned math skills they could build on, despite not being formally educated. For these busy women, an afternoon training session was as much a chance to get together as it was an opportunity to learn something new. The women were well served because that educator understood that numeracy was just one component of what they needed.

This reality was echoed outside Pune City, where I met with village women leaders organized by the NGO MASUM. Engaged in the political life of their community, these women respond to domestic violence, address health concerns, advocate for better bus service so their daughters can go to school, and run savings groups, among other things. On top of this, they tend to fields, maintain impeccable homes and the village temple, and are raising daughters with aspirations and sons committed to gender equality. Doing one thing at a time, individually, doesn’t make sense – safety, health, education and economic assets are all priorities, linked to their wellbeing and survival, and that of their families.

Women often organize this way because government and development programs fail to, requiring them to make impossible choices and overcome numerous barriers to access vital services and basic entitlements. Women are busy, negotiating the world around them and the decisions they face because they are women. To better serve women, and to achieve their own objectives, government and development programs need to accommodate those realities. Through our partnerships across government, civil society and the private sector, this is what the 3D Program for Girls and Women seeks to achieve.

Accelerating evidence-based south-south collaboration to reach the extreme poor

By: Emily Coppel and Isabel Whisson, BRAC

Careful adaptation of proven programs in new contexts is key to achieving SDG 1: End poverty in all its forms everywhere

In January of 2017, five officials from the Government of Kenya, The Boma Project, and CARE International traveled to Bangladesh. Their aim? To understand how BRAC, one of the world’s largest NGOs, headquartered in Dhaka, ‘graduated’ 1.7 million of its very poorest households out of destitution into those with sustainable livelihoods. This trip was part of a larger initiative, funded by the International Fund for Agricultural Development, whereby BRAC is providing technical assistance to Kenya’s government and its two partners on how to develop and roll out more effective, targeted social protection services for its most severely marginalized citizens.

BRAC’s Graduation approach has been implemented widely in the years since Science published seminal research with evidence demonstrating its effectiveness. Evaluating the approach in eight countries, researchers found that participants increased their savings, consumption, and assets one year after the program ended. The approach itself is complex, neither a laundry list of services, nor a formulaic implementation plan, that can be stamped across various regions. As the Kenyan delegation soon came to understand, the adaptation – in both the delivery of services and the services themselves – is much more nuanced.  

What is Graduation? 

Graduation programs have several different components. First, a rigorous targeting and market analysis exercise to identify a particular vulnerable population. This process also identifies participants who exhibit the potential to undertake and expand an income-generating activity or small business. Another component is livelihood development, which can be in the form of cash or a loan for a stipulated purpose, livestock or tools of trade to start a business, linkage to employment, or any productive income-generating activity. This is accompanied by technical training to assist the participant in growing her business. Wraparound services are incorporated, and could include a combination of healthcare, food stipends, financial literacy training, financial services, and savings to provide integrated support to the participant. Mentorship or coaching throughout the program is key to supporting the participant with social assistance, life skills training and in all aspects of the program for its duration. Finally, Graduation criteria must be in place, which consist of specific indicators that illustrate the participant’s economic, social, and personal progression, for example, sending their children to school. 

Graduation Approach
Copyright BRAC/Amplifier Strategies

BRAC launched its Ultra Poor Graduation Initiative to provide technical assistance and advisory services to governments, NGOs, and multilaterals interested in designing effective Graduation programs for extremely poor and marginalized populations. This initiative fosters South-South collaboration, whereby a strong evidence-backed program in Bangladesh can serve as an approach for other regions. 

BRAC’s Initiative assists partners to develop programs that are carefully targeted and time-bound, with sequenced interventions that promote livelihoods, financial inclusion, food security, basic health, social capital, and, most importantly, self-empowerment. Together, they contribute to uplifting households from extreme poverty. Critical to the success of Graduation programming is the fine-tuned contextualization of the approach in each environment.  

Adaptations in Kenya, Lesotho, and the Philippines are underway and these partnerships are already offering insight on how to improve services for the extreme poor living in drastically different contexts. 

Livelihoods and market access: When giving cash works better 

Historically, BRAC has been a strong proponent of providing in-kind assets, like goats, cows, or goods to sell, to help women start their businesses, rather than cash. But in Kenya, early indications show a different approach could be as effective. In Bangladesh, the women served through BRAC’s Graduation programming, have limited interaction with markets. For example, if given a cash transfer to buy and re-sell a cow, a program participant would not know the market price of a healthy cow, and risks being overcharged. In Kenya, women have better understanding of markets and could navigate these challenges with less support. As a result, they may be able to effectively take on a cash transfer to launch their enterprise. For implementers, this cuts down on staff time to purchase assets on behalf of participants.  

Group vs. Individual livelihoods 

Another divergence from BRAC’s traditional design is operating group livelihoods packages in one of the Kenya pilot sites, by The Boma Project who are experienced with this approach. In the past, Bangladesh pilots, group businesses have not been successful, with some members feeling that others freeload off their hard work. But in Kenya, BOMA’s participants combine their efforts to create an enterprise, which also enables them to support one another with family and domestic responsibilities. When one member is working, another is available to assist with childcare for the group. An evaluation comparing individual with group livelihoods will generate more insight into when one approach may be more effective than the other. 

‘It may work here, but it can’t work there’ 

While skillful adaptation often means making tweaks to a model, it also requires knowing what to keep. When BRAC’s NGO partners from Kenya observed BRAC’s participatory process for identifying the most vulnerable in a community, they expected it would not work in Kenya. Because of the remoteness of the pilot sites they thought community members would not show up to participate in the exercise.  After testing it in Kenya earlier this year in January, BRAC found that community members were much more receptive and involved than the government officials had predicted. This method proved more effective and operationally efficient than those they previously used to identify the poorest members of a community.   

In Rangpur, Bangladesh, Haimanti, Anwara, and Aheja attend a final workshop for BRAC’s Targeting the Ultra Poor graduation program in which the instructor taught savings management. They will graduate from the program this December.

Achieving the SDGs will require sharing successful anti-poverty solutions, drawing on lessons learned across contexts and fostering South-South collaboration. To do this effectively, implementers and program designers will need to understand why each program component exists, what purpose it serves, and how the root causes of poverty vary across contexts. In the next several years, BRAC’s Ultra Poor Graduation Initiative team will continue to work with others to understand the elements of a successful approach, design adaptations, and, through careful iteration, ensure these Graduation programs take root and successfully grow into effective anti-poverty solutions.  

When the Evaluation Plan Doesn’t Reflect the Context

By: Aga Khan Foundation

When a partnership offers an opportunity to improve an important value chain

Aga Khan Foundation in Mozambique started the MozaCaju project with USDA funding in late 2013/early 2014, as a subcontractor to TechnoServe. The 3-year project aimed to improve the value chain for cashew production and marketing in Mozambique, providing training and support for cashew producers in growing techniques, harvest and post-harvest improvements, and connections to processing and marketing.

The evaluation plan outlined in the contract was ambitious: It called for baseline, midline and endline evaluations all conducted during the 3-year span, at the tail end of annual cashew sales campaigns. Furthermore, the evaluation design stipulated a more rigorous quasi-experimental approach, with designation of control and beneficiary groups for comparison and attribution of outcomes to project activities. Such an intensive investment in evaluation of a relatively short project was higher than what many non-US donors require in Mozambique and was new to AKF.

We accepted the donor’s plan, however, and the project got underway. Our team had good experience with farmers’ groups in the project area; the cashew project allowed us to leverage that experience with farmers to introduce techniques that would help them improve their production. It appeared to be a promising relationship for improving value chain connections. And in fact, farmer peer-to-peer adoption of the techniques introduced by the project spread faster than we expected.

Project profiling of beneficiaries took time, as did the rollout of project activities and the design of tailored trainings to target smallholder farmers, mid-sized farms, as well as larger and even commercial farm operations. Because there was no random allocation of farmers into beneficiary and non-beneficiary groups at the beginning of the project, there was no scope for a truly experimental research design. However, we were still expected to ensure that the project midline and endline evaluations were done on the basis of a beneficiary vs. control comparison, requiring us to try and ‘match’ non-beneficiaries to our beneficiary sample on characteristics such as household size, number of productive cashew trees and whether farmers sold individually or in a group.

Because our intervention targeted specific administrative posts within a district, our non-beneficiaries were identified in adjacent administrative posts where the project had not targeted its activities.

In theory, this approach could have worked. In reality, it led to problems later.

AKF across its many offices has been grappling with the challenges of Monitoring and Evaluation, creating tools across the organization for better assessment of our impact against key indicators. This case provided useful basis for examination of those tools and shared measurements.

When the donor’s research methods pose a challenge

From the start of MozaCaju, the AKF project team faced difficulties implementing the quasi-experimental methods for evaluation required by the donor. The M&E team lacked experience with this kind of survey design and the program staff and prime grantee decided to accept this design prior to AKF hiring an M&E Director and project M&E Assistant. Additionally, the project planning documents did not provide for training the project team in quasi-experimental research methods. External consultants were hired to oversee the overall project studies, while AKF was responsible for the specific farmer survey. Upon reflection, we determined that while the evaluation design may be appropriate for certain conditions, few AKF teams have the manpower, available time or capacity for implementing it or adjusting such designs in the field.

Compounding the technical challenge of the design was a communications challenge: As a sub on the project, the AKF team didn’t have a direct relationship with the donor and so we couldn’t revisit the evaluation plan directly with the donor. That discussion might have averted later problems.

As noted above, the baseline survey did not make a distinction between control or intervention groups because the survey took place before beneficiary selection. At the time when we chose control and intervention groups for the midline and endline evaluations, we began to note a few problems. First was the fact that our project targeted a high proportion of the cashew farmers in the project districts, leaving a control population that was relatively small from which to draw a sample and which contained a lot of outliers (e.g. low or high production farmers with little interest in being part of the project). This made ‘matching’ rather difficult.

But we did the fieldwork, analyzed the data and found that, in some areas, the control group that we were able to create was dramatically outperforming our beneficiaries. We took a number of steps to try and figure out why this was the case, including re-verifying some of the data in the field, but we were still left with this finding. We eventually realized that when establishing a control group, we forgot to ask non-beneficiaries two very basic questions: 1) have you received any information or training from other farmers on good agricultural practices; and 2) are there are other cashew interventions in this area?

What we found was as: a) there is a strong network between cashew farmers and a number of our beneficiaries were communicating the production techniques they had learned in our project to farmers who were not part of the project (‘contamination’ and ‘spillover’); and b) more important, another NGO was implementing a smaller project focused on some of the same interventions as us, but also on the distribution of chemical pesticides, which we were not. This project also included some of our farmers – which we didn’t realize because farmers thought they were the same project – but was heavily targeting some of the administrative posts where we weren’t working. Chemical pesticide turns out to be absolutely key to improving per tree productivity and this is what we concluded was likely explaining better results in our control group.

These two complicating factors – spillover and the existence of complementary interventions – made it extremely difficult later to isolate factors between the control and intervention groups.

As a result, the midline and endline report analysis showed the control group outperforming the intervention. The project team assessed the results and realized that survey participant selection had failed to reckon with those key contexts.

What lessons went on to inform other programming?

Lessons Learned

One lesson from this experience, in terms of M&E planning, is to recognize the capacities and limitations of your project team at the project’s outset, before accepting donor-driven evaluation procedures and designs. Where there is a relationship and a communication channel for adjusting the evaluation plan given a project’s duration or other tools available, pursue that conversation.

A second lesson is to account for the wider context of the project area or study area to determine whether a specific research design is truly feasible. Context will largely determine the extent to which a quasi-experimental design will be able to capture the project impact or whether alternative approaches would be preferable. For example, spillover effects between beneficiary and control groups can muddy the analysis by bringing into question the degree to which a true control group, isolated from project interventions, can be identified. External factors – such as overlapping and comparable government or other NGO projects in the same sector but implemented in geographies where a control sample is intended to be selected – can also make a quasi-experimental approach of questionable value. As an alternative, mixed method approaches that combine rigorous quantitative data collection among project beneficiaries with good qualitative research to understand the stories behind the numbers, may be a better approach to understanding the impact of project activities.

More broadly for an international NGO is the lesson that project teams need better tools for M&E. This lesson reinforced the need for AKF-wide standards for shared measurements, and for support to ensure teams have the capacity to measure the key indicators. AKF has been conducting a global exercise in M&E shared measurements for this purpose, developing tools for indicators at the global and country level, and standards for benchmarks.

Conclusion

Documenting program impact is important but that documentation is only as good as the methods adopted and the skills of the people doing the evaluations. The methods for monitoring and evaluation of programs must be adapted to the program environment and team capacities. Where a project starts with a donor-driven evaluation plan that is not aligned with team capacities, these must be addressed early, with some combination of: renewed discussion with the donor to adjust the evaluation tools to the project setting and context; reinforcement of team capacity for the resulting adjustment; and continued sharing of M&E measurement experience, tools and standards.

 

Photo Credit: TechnoServe/Antonio Filippi de Castro

Show me the evidence: Cultivating knowledge on governance and food security

By: Tessa Ahner-McHaffie, FHI 360

https://researchforevidence.fhi360.org/show-me-the-evidence-cultivating-knowledge-on-governance-and-food-security

I recently participated in a salon on integrating governance and food security work to enhance development outcomes. Convened by the LOCUS coalition and FHI 360, the salon gathered experts in evaluation, governance and food security to review challenges and best practices for generating evidence and knowledge. A post-salon discussion recorded with Annette Brown and Joseph Sany speaks to the gaps in evidence and the need to more accurately measure how governance principles influence food security outcomes.

I came out of the salon conversation thinking that while there was a hunger for evidence, there are still large gaps and significant differences within the literature on things as basic as definitions. That being said, I wanted to dig a bit more into what evidence was actually out there and think about what needs to be done to move this budding evidence base forward. In this post, I highlight three pieces of interesting research that contribute to the evidence base on governance and food security integration, and then propose a few suggestions on how to grow that knowledge base.

What evidence currently exists?

Numerous studies propose conceptual relationships between food security and governance, but there are a few rigorous pieces of evidence that demonstrate governance work has supported positive food security outcomes. I provide you with three examples here.

  1. Mercy Corps has an excellent report – Pathways from Peace to Resilience – that examines a natural resource management and peacebuilding program in the Mandera triangle of East Africa and a program including health and governance components in Uganda. Both programs build dispute resolution capacity and platforms with actors and institutions that are involved in conflict management within and between communities, and utilize both quantitative and qualitative methods. The study found that across study sites, strong conflict management capacities in institutions and communities contributed to greater food security for households that experienced shocks. This evaluation is excellent because it not only measures specific food security outcomes within a governance program (which is rare), but it also attempts to identify enabling factors between different programs that could facilitate the same kinds of outcomes in other governance programs.
  2. Another interesting report from the International Institute for Environment and Development showcases how farming is affected by governance and policy using case studies from Bolivia, Brazil, India, Kenya, Pakistan, Senegal, South Africa and Thailand. The case studies were undertaken to identify sustainable agricultural practices, and then to identify the policy and non-policy causes of those practices. The report argues that public policies had little involvement in many areas of success; instead local civil society and human capital proved the drivers of agricultural success in many places. In Senegal, for example, although the state has largely withdrawn from participating in local agriculture since liberalization in 1994, it has created enabling agricultural policy and legal environment for NGOs, producer organizations, and rural development institutions. This has allowed these organizations to fill the coordination and advocacy roles that the government does not provide and to support more beneficial policies for small-holder farmers.Additionally, other case studies (in Bolivia, India and Pakistan) show that building social capital through NGOs or producer organizations, or through local decentralization of natural resources management, supports the power of communities to directly advocate for their rights and interests to achieve agricultural outcomes. Policies identified through these case studies that supported improved agricultural outcomes – like improving natural resource management, improving technology, improving market access – all were achieved by strengthening farmers’ bargaining power.
  3. Finally, a study on land tenure and food security provides a framework to understand the link between governance and food security. Maxwell and Wiebe present the links between land and food as a linear relationship of resources, production, income, consumption and nutrition status. Using this as a jumping off point, they examine the land tenure literature. Among other topics, the authors present multiple sources showing a quantitative link between access to food and access to land in an agrarian economy. Using a study from Thailand, the authors show that increased tenure security leads to increased demand for land improvements from farmers and increased formal credit supply because farmers have access to collateral (land). Both effects resulted in short- and long-term investment in agriculture and production.There are also ripple effects to bad policymaking that can be felt when households experience shocks. A study in Ethiopia documents that during food crises, insecure land tenure causes people to delay moving in search of food assistance. This reluctance may have increased the death toll in Ethiopia during the 1984-1985 famine. Maxwell and Wiebe utilize this and other evidence to propose a more circular, dynamic relationship between food security and land tenure, and offer suggestions for researchers and policymakers given the dynamic relationship. This paper is valuable as it serves as a literature overview for land tenure and food security, as well as detailing conceptual relationships that can be tested and measured.

There is not enough evidence to come to any kind of conclusion on what works and, further, there does not appear to be a cohesive direction for growing evidence and knowledge. Recognizing different research cultures, different definitions for food security and governance, and the limited number of existing studies, I find it difficult to envision how this evidence base will move forward.

However, in a world where conflict is becoming more common, and governance and food security are inextricably tied up in the resilience of communities to survive both human and climatic shocks, I think we need to understand the relationship better, and understand how integrating programs can better improve outcomes in both sectors.

How do we grow the evidence base?

Considering how other multi-sector groups have built a knowledge base in their respective areas, I propose a few first steps to expand the evidence base for governance and food security.

  1. Clarify common terms and theoretical framing.As noted in the transdisciplinary toolkit for integration methods and FHI 360’s resource package for integrated development, agreeing on common terms and theoretical frameworks is one of the first steps to conducting research that is useful across disciplines. In the case of governance and food security, multiple evidence reviews have noted the diverse concepts and frameworks used throughout the literature. Narrowing that field could go a long way to directing future research.
  2. Establish what level of evidence is necessary.Much of the evidence around governance and food security is conceptual, rather than empirical. The existing empirical evidence largely lacks the rigor present within other sectors. But is that level of rigor needed or possible in this space? If there is a consensus on what is needed, then it will be easier for the community to plan for what is necessary and move the conversation forward.
  3. Develop a targeted research agenda.Big questions were asked during the LOCUS salon including, “When is governance a necessary condition, and when is it a sufficient condition?” and “What motivates farmers to get involved and engaged in local governance?” Developing key questions can focus researchers in different locations on the same topics to generate enough evidence in targeted areas to produce broader findings.
  4. Improve formative research.Improving formative research can provide a short-term win with long-term benefits. Using multi-criteria assessment, root-cause analysis, and other methods to assess food security or governance from a multi-sector perspective from the beginning will affect what outcomes are reported. By being more deliberative about what we are measuring early in the project, we can use the data from projects to better illustrate where governance is contributing to development outcomes and evaluate that more rigorously in the future.
  5. Explore non-traditional evaluation methods.Using alternative evaluation methods can also help us better understand how food security and governance are related, and where investing in governance for food security is more necessary. Methods like contribution analysis or complexity-aware monitoring can provide results that are not available through more traditional means.
  6. Form integrated teams and procedures.Engaging multi-sector stakeholders, ensuring a place for multi-sector perspectives on a team, and inserting decision points to ask about integration have been successful in other multi-sector spaces (documented here in a nutrition-sensitive agriculture program). Ensuring that there is space for those kinds of discussions could ensure that other perspectives are included in the evaluation process.

These steps can close the gaps and expand the evidence base for governance and food security. What do others think is necessary? How can we continue to move this conversation forward? What other evidence in governance and food security do you find compelling?

Photo caption: Northern Thai farmers work under the Royal Initiative Project, which promotes alternative crops in Thailand as a means to generate income.
Photo credit: © 2016 Hansa Tangmanpoowadol, Courtesy of Photoshare

Integration and Community Leadership Needs More Than Just Funding Changes

Screen Shot 2018-02-07 at 5.57.12 PMThe nature of our world is multi-nodal and, with technology, today’s means of accountability, collaboration, communication and fulfillment of responsibilities are evolving to reflect integration. International development needs to catch up – quickly. Locus, Pact, Church World Service, and The Hunger Project – all members of the international Movement for Community-led Development – addressed this in a comprehensive discussion at the United Nations during the 56th Commission on Social Development.

In looking at respective research and successes from integration, panelists and attendees identified a shared belief that integrated, community-led development is the effective and most dignified approach needed to achieve sustainable development for all. What ensued in discussion around means toward implementation was not typical banter about shifting funding streams and enabling conducive policy environments. Yes, of course that came up. But, robust discourse focused mostly on how development professionals should carry out their work by helping community members live authentically as thee leaders in our collaborative work toward the SDGs.

“When people tell me they will build capacity of communities, I say, ‘Who told you the community doesn’t have capacity?'” MacBain Mkandawire, Executive Director, Youth Net and Counseling, revealed the most obvious “secret” to development professionals: communities already have the capacity [and thirst] to incur their sustainable development. Our work should aim to compliment work already happening at the local level through collaboration with community leaders, not competition or programmatic control over them.

“People say, let us give voice to the voiceless. Well, they are not voiceless. We just haven’t talked to them.” Oyebisi Ohuseyi, Executive Director of Nigeria’s Network for NGOs, revealed the second “secret”: community members already behold opinions, grievances, solutions and priorities. In order to foster true ownership and agency, community partners should be the ones prioritizing which development issues matter most to them. And inarguably, communities have the deepest knowledge of their context and can offer the best insights on [most appropriate] ways forward.

Mkandawire added “Before we go to donors, we should be asking ourselves what do we need to change [in our work as development professionals]?” He was acknowledging that there are multiple layers of power and privilege toward realizing community-led development.

Therefore, the development community and its many stakeholders are called to move from the less helpful “outside expert-driven” tendencies and donor pandering toward budgeting and programs with a bedrock of fostering community agency and expertise. Ultimately, this means we need a new kind of capacity development professional: one able to convene, befriend, facilitate, energize, accompany, co-learn, and co-create with community members “with [sincere] reverence and respect.”

The discussion also brought attention to unrealistic time constraints and reporting windows that strap capacity, ultimately hindering sustainable change. Even incremental progress is valuable and worthwhile, as was described by a woman from and working in Nepal. She detailed an extensive self-managed cooperative thriving with over 1,000 members—but mostly after 18 years of incremental progress.

In reflecting on the discussion, the panel moderator, Ellie Price, Coordinator of Locus Coalition, noted “It is easy to get bogged down in the technicalities of our work, or the limitations imposed by global power structures.” This makes it near impossible to represent and act on views or experiences other than your own.

The event’s discussion shed action-oriented light. Members of the Movement for Community-led Development are dedicated to mobilizing and collaborating with local community leaders. Bringing government ownership to community-led processes is the Movement’s current priority to achieve sustainable development.

Rattling top-down power structures among stakeholders and influencing strongly devolved political systems will not only garner community leadership, but also community ownership and due dignity as people – rightfully so – steer their own development process.