Monitoring and Evaluation

“All human beings are born free and equal in dignity and rights.”
– Article I of the Universal Declaration of Human Rights, 1948

 

ONETRACK International is committed to the fulfillment of this basic human right embodied in the Universal Declaration of Human Rights that should extend to men,women,boys and girls without discrimination. It is the first organization in Cameroon placing orphaned children in the household of surviving relatives while ensuring that they receive an education and proper healthcare within a secure family environment in order to bring hope and opportunity to a large population of the country’s orphans.                                                                                                                                         

As ONETRACK International has now expanded its model to reducing child institutionalization to five countries- Greece, Liberia, Honduras, Zimbabwe and The Gambia, we hope that this Monitoring and Evaluation guide would be useful in aiding the respective organisations in assessing the success of our model.

 

CHAPTER 1: INTRODUCTION

Chapter 1 of this handbook provides an overview of the issues of the orphan crisis in Cameroon that has sparked the creation of the In-Home sponsorship program by CameroonONE and to be promulgated by ONETrack International to the rest of the world. It also further elaborates how it is organised, the objectives of this handbook and how it should be used. 

 

(1) ONETRACK INTERNATIONAL

ONETrack International represents a subsidiary of CameroonONE, a non-governmental organisation (NGO) based in Cameroon, that places orphaned children into the care of surviving relatives while ensuring that they receive an education and proper healthcare within a secure family environment. Following the program created by CameroonONE, called the In-Home sponsorship program, ONETrack International intends to expand its successful orphan-care projects implemented in Cameroon to the rest of the world. The main objective of ONETrack International is to address the worldwide orphan crisis and reduce institutionalization of orphaned children with the implementation of our innovate In-Home sponsorship program that removes children from the orphanage system in order to place them in the homes of surviving relatives where they can remain in their village of birth. 

ONETrack International aspires to uphold globally-recognized international laws and standards that protect an orphaned child’s rights at both the community and global levels by supporting and advising organizations that work with parentless and vulnerable children around the world. The In-Home sponsorship program is currently already operating in Greece, Liberia, The Gambia, and Zimbabwe and seeks to expand its operation further into Asia and Latin America by 2019.  

 

(2) ISSUES, GOALS AND OBJECTIVES

Overcrowding, insufficient funding, inconsistent caregiving, limited community support and even sexual abuse are more often than not associated with large-scale instit utionalized care. CameroonONE was initially established to address the over million orphaned children with few resources dedicated to their protection, wellbeing and housing. A number of these orphans are left to fend for themselves or end up falling victims to exploitation. Studies have also shown that intellectual, emotional and physical development of children are stunted when they are institutionalized. These are obstacles that hinders a child to be effectively cultivated and nurtured to be an invaluable member of society. Overcoming these setbacks by providing material, financial and household provisions to relatives who are responsible for raising children under their care allows them to ultimately provide suitable education, healthcare and security to them. 

Therefore, the main objective of ONETrack International’s In-Home sponsorship program is to continue the work of CameroonONE throughout the globe by maximising the welfare of orphaned children around the world and placing them in the care of existing family members which is a better alternative than institutionalisation. The short term and long term goals of CameroonONE can be extended to that of ONETrack International and can be seen in Table 1 below. The effectiveness of this program is measured via the indicators mentioned in the subsequent chapters. 

Outcomes of ONETrack International’s In House Sponsorship program

Short Term

Long Term

Access to clean drinking water as a child Better health situation as an adult
Higher number of years attending school Find better jobs as adults
Complete Vaccination Stronger immune system as an adult
Access to rice, cooking oil etc. Better health situation as an adult
Lower population inside orphanages Maybe develop orphanages
Access to health care Informed on health care value as an adult. 

Table 1. Short term and long term outcomes of CameroonONE

 

(3) CHILDREN’S RIGHTS

Numerous International adoption laws are in place to ensure that the welfare of  children are prioritised. The purpose of such laws are to uphold and attain the fundamental goals of advancing the rights of a child. The United Nations Declaration on Social and Legal Principles relating to the Protection and Welfare of Children was adopted by the General Assembly on the 3rd December 1986 and was drafted for the “Conscious of the need to proclaim universal principles to be taken into account in cases where procedures are instituted relating to foster placement or adoption of a child, either nationally or internationally”.

As stated in Article 4 of the Declaration:

“When care by the child’s own parents is unavailable or inappropriate, care by relatives of the child’s parents, by another substitute – foster or adoptive – family or, if necessary, by an appropriate institution should be considered”.

Taking into account this declaration, ONETrack International leverages on the fact that children should be allowed to be in the care of their family members and relatives first before adoption is considered. 

 

(4) ORGANIZATION OF THIS HANDBOOK

This Handbook seeks to be a guide that can be used to assess whether the In-Home sponsorship, that was first implemented in Cameroon as a solution to alleviating the problems associated with institutionalized orphans, can be used in other countries to address similar problems there as well as problems specific to those countries. The  planning, monitoring and evaluation framework aims to provide guidance to the management staff of various organisations who would like to adopt the In-Home sponsorship programme on the key methods and mechanisms to be utilised so that the main output of the programme can be achieved and measured via the selected indicators. The methods developed and indicators selected also reflects the intentions of the management staff of ONETrack International as well as CameroonONE and invariably, informs stakeholders of these organisations on how they have been conducting their work thus far. This Handbook would, therefore, inform the programme implementer on what needs to be done, the appropriate timeline to be used and personnel who needs to be involved. 

 

(4.1) WHAT DOES THIS HANDBOOK DO?

The objectives of this Handbook includes the following:

  • To inform the reviewer of the objectives of the In-Home sponsorship programme and the processes utilised when implementing it in the context of an orphanage. 
  • Providing the basics of the planning and monitoring processes in the context of the In-Home sponsorship program implemented in orphanages. I.e. Providing a comprehensive framework to monitor the programme by clearly defining indicators, baselines and goals. 
  • Providing the basics of the evaluation processes in the context of the In-Home sponsorship program implemented in orphanages. I.e. Utilising outputs of the programme to assess its success and therefore providing transparency and accountability of the respective organisations.  
  • To improve the quality of monitoring the success of the In-Home sponsorship program and the evaluation of its success in other countries with their own unique contexts of the orphan crisis there and ultimately, provide accountability to the organisations implementing this program. 

(4.2) WHO IS THE HANDBOOK FOR?

This Handbook can be used by: 

  • Staff of orphanages and other departments that manage programmes , such as: 
    • Planning and monitoring programmes implemented on the national, regional and global level
    • Managing the commissioning process of evaluations
  • Managers of orphanages who oversee and assure the quality of planning, monitoring and evaluation processes and products, and use monitoring and evaluation for decision making.
  • Stakeholders and partners, such as local governments, development partners, and beneficiaries, who are involved in the planning, monitoring and evaluation processes.
  • The Executive Board of the respective orphanages, which oversees and supports the activities of them, ensuring that they remain responsive to the evolving needs of the beneficiaries.
  • Independent evaluators who need to understand guiding principles, standards and processes for evaluation within the context of orphanages. 
  • Members of the national, regional and global development and evaluation community

 

(4.3) HOW SHOULD THE HANDBOOK BE USED?

This Handbook should be used as a reference throughout the lifecycle of the programme. I.e. beginning from the implementation of the programme. This Handbook can be used to monitor and evaluate the results and the management of the programme. It should be used to record and track the results as the programme progresses to guide the organisations to tweak their processes along the way and record it to ensure that they are on track to achieving the intended results.

 

CHAPTER 2: SCOPE & PURPOSE OF M&E

When drafting a monitoring and evaluation guide, it is essential to know  the main objectives of the program otherwise you would not know what you are basing the assessment of the program against. This chapter aims to identify the key components to consider when you begin drafting your M&E guide. 

 

Review of logframe

A logframe or logic model is a developed at the beginning of the implementation of ONETrack International’s In House Sponsorship program. It is designed to ensure that the expectations and objectives of the program are aligned across stakeholders and program managers. This would be further explained in Chapter 4. 

 

Identify expectations

The expectations and needs of internal and external stakeholders should be ascertained to ensure understanding, ownership and the use of data collected is in line across the board. Therefore, a clear understanding of stakeholders’ priorities and how they are affected by the program’s implementation is essential. Besides their priorities, their constraints should also be clarified so that the goals stated can address both these priorities and constraints. These objectives should also be modified according to the local context so that these goals are feasible and therefore the program is credible and would be accepted in the respective countries. 

Examples of key stakeholders and informational needs:

  • Communities (beneficiaries)
  • Donors
  • Project/programme management
  • Project/programme
  • Partners (bilateral or local)
  • Government and local authorities 

When identifying the information from stakeholders, the following can be considered:

  • Who the key stakeholders are
  • 4 Ws ( who, what, where, when and how)
  • Role or function they expect or are required to have in the M&E system

 

Scope of major M&E events and functions

The scope of major M&E events and functions refers to the scale and complexity of the monitoring and evaluation of the program. The M&E process can range from one that is relatively simple and being reliant on internal resources and capacities to one that is highly complex with differing activities and requiring expertise and resources. There are factors that affect the complexity of the M&E processes which are (but not limited to) the following: 

  • Number of outcomes 
  • Type of outcomes
  • Scale of the intended impact
  • Geographic Scale (i.e. Accessibility to the program areas)
  • Demographic scale (i.e. Specific target populations and their accessibility)
  • Time frame
  • Available human resources and budget.

In order to determine the overall complexity of your M&E system, you should identify key activities that you would like to have throughout the entire process to give an overview of the system and any additional components like further plans for funding, additional technical expertise required that you would like to include along the way.

 

CHAPTER 3: M&E DEVELOPMENT  PROCESS

This chapter aims to introduce the monitoring and evaluation processes involved and to differentiate the processes and definitions of monitoring from evaluation.

(1) MONITORING AND EVALUATION COMPONENTS

There are twelve main components of the entire monitoring and evaluation system that are required for it to function efficiently and effective:

  • Organisational structures with M&E functions. 
  • Human capacity for M&E
  • Partnerships to plan, coordinate and manage the M&E system
  • M&E frameworks
  • M&E workplan and costs
  • Advocacy, communication and culture for M&E
  • Routine program monitoring
  • Surveys and surveillance
  • National and sub-national databases
  • Supportive supervision and data auditing
  • Evaluation and research
  • Data dissemination and use

 

(2) MONITORING

Definition of monitoring

Monitoring is a continuous process of information gathering (quantitative and qualitative data) in order to track the progress of the implementation of the project by using indicators to identify whether the goals of the project have been achieved. This allows the programme manager to establish if the programme is moving towards achieving the main objectives. By utilising the data collected, a project’s strengths and weakness can be identified and therefore the limitations that arises out of the implementation of the project can be resolved as the project develops. The next section describes the type of monitoring that can be utilised to track the progress of the project. 

 

        (2.1) TYPES OF MONITORING

There are five main types of monitoring methods that can be included in the M&E process:

Process monitoring/ physical progress monitoring: 

This involves field visits to the location where the project is being implemented. A joint development of a checklist of outputs, inputs and activities (to be identified during these visits)  by the management/stakeholders and project staff can be used by the M&E staff during these visits. The checklist can be used to determine if the project is close to the attainment of its goals. After which, any gaps identified can be flagged to the management/ stakeholders and the project staff to ensure that they are addressed.

 

Assumption monitoring :

As laid out in the Logical Framework section, assumptions are stated to ensure that factors outside of the control of the project are taken into account. Therefore their influences on the project should be considered towards the development of the project since these assumptions can determine the project’s successes or failures. Assumption monitoring measures these external factors that are not within the control of the project and may be contributing factors in explaining why a project has succeeded or failed. For instance, one of the long term goals of ONETrack International is to ensure that the orphans who have transitioned to be under the care of their extended family members are able to have access to education that would eventually lead them to attaining tertiary education. However, this may not happen due to factors such as the interest of the child and family problems that are out of the scope of ONETrack International’s capacity. 

 

Financial Monitoring:

Financial monitoring has several uses. One of the most obvious reasons is to ensure that the project stays within the planned budget set out during the planning stage of the project. It is also important for the organisation to be financially accountable and to therefore, maintain transparency to stakeholders regarding the use of these funds. Lastly, financial efficiency can be measured to track which activities are under resourced or over resourced and therefore to perhaps channel these resources to other activities where they may be needed. 

 

Impact Monitoring: 

This type of monitoring involves the continual assessment of the impacts produced from the   project to the respective target populations. As mentioned previously, the path to achieving the long term goals of the ONETRACK International is dependent upon the systematic attainment of short term goals. The task of impact monitoring is applicable primarily to those long term goals as there is a need to measure whether the intended impacts of the projects have been produced and is on the way to achieving the main long term goals. These impacts can be monitored through the established  impact indicators. It should also be noted that both positive and negative impacts should be taken into consideration to also identify impacts that were not identified and therefore not intended to be achieved prior to the project initiation. 

To establish the monitoring process, three main elements are used

  • Project inputs 
  • Performance
  • Progress.

 

         (2.2) MONITORING CRITERIA

A widely accepted set of monitoring criteria adopted in the M&E procedures of various agencies is the SMART criteria. SMART is used to set the goals and objectives of the project and these are used as a principle guide for indicator formulation for an effective results based monitoring. This section aims to set out the components in this criteria. As each component can be interpreted and defined differently by various organisations according to the context of the project they are implementing, the components stated here represents that of which ONETRACK International is utilising in the monitoring process of our In House Sponsorship programme: 

 

Specific:

The indicator must be formulated to capture the quantitative and qualitative information that is necessary to reflect whether a specific objective has been met and not any other objective. This is so that it can be ascertained if the changes are happening are due to the implementation of the project or otherwise. Hence, it is important that the objectives of the project are set out clearly at the start of the project. Stakeholders and project managers should also have a common understanding of what these indicators are supposed to capture.  

 

Measurable: 

Firstly, the indicator must be feasible to quantify and the data should be gathered in a pragmatic manner for the indicators to measure changes that can be verified objectively. Secondly, these indicators must also be measurable and formulated in such a way that two different people would measure them in the same way irrespective of the time when the data is collected to ensure consistency across contexts. If qualitative data is being collated for the purpose of indicator formulation, then the definition of the qualitative data must be clearly defined so that there are no different interpretations or misinterpretations of the meaning of the indicator. Lastly, the sensitivity of these indicators should be taken into consideration so that they would be able to measure any tweaks in the actions of the programme that would affect the results. 

 

Achievable and Attainable: 

In order for the indicators to be deemed achievable, the target outcome should accurately specify the component that is to be measured by the indicator and therefore, these outcomes must be realistic enough to be achieved. 

Additionally, the indicator should be able to attribute any changes measured by the indicator to the implementation of the project itself. Hence, these changes should be anticipated prior to the implementation of the project so that they can be identified upon its implementation. 

 

Relevant and Realistic: 

The indicators must be realistic in a sense that the way the data is collected must be practical, reflect the expectations of parties involved and utilise sources that are already available to the project manager. If the costs or skills associated with the data collection are obstacles, then those indicators may not be accurate if the information cannot even be obtained. This may have a knock on effect on other criteria in the logical pathway as they may be compromised due to the inability of the indicator to measure what it is supposed to measure. We may refer to Figure 1 for an example of this happening. If information that is required to be measured by the indicators is difficult to obtain, the output would not be able to be measured accurately and therefore we would not be able to ascertain if the short term goal has been achieved. This is important because, as mentioned in the previous section, there is a multi step process where the short term goals have to be achieved first before the long term goals can be obtainable. This follows then that the indicators should relate to those long term goals for it to be meaningful and show that the short term outputs have a related impact. 

This leads to the criteria that the indicators must be relevant. For an indicator to be relevant, it must be able to establish a relationship between what the indicator measures and the theories that led to the development of the short and long term outcomes. For instance, one of the short term goals listed in Table 3 is the attainment of children attending school for a higher number of years whereas the long term goal of this is to see more adults obtaining better jobs. ‘Better’ jobs in this case can mean high-skilled labour. A suitable indicator for this could be to measure the number of years the children have attended school or received education. The theory behind this is that the longer children are in school, the more likely that they would be educated up till the tertiary level which enables them to fulfill high skilled labour jobs that may require a tertiary degree as a prerequisite. This is opposed to them not holding a qualification certificate and they would most likely only be capable of working in a low skilled job. 

 

Time Bound: 

The time factor needs to be considered in the formulation of indicators because it may be considered a constraint. Such scenarios include but are not limited to: 

The time spent on data collection while efficiently using the resources available to do so must be reflected in these indicators.  The time lag incurred between the actual output and expected output needs to be considered. A time frame indicating when the indicators would measure the output.

 

(3) EVALUATION

Definition of evaluation

According to UNICEF, evaluation can be defined as a process utilised to determine the relevance, effectiveness, efficiency and impact of actions taken to attain specific goals in a systematic and objective way. It is therefore a tool that is used to improve both current activities and future planning, programming and decision-making. One can think of it as a tool for improving current and future processes in the programme implemented. 

 

(3.1) EVALUATION CRITERIA

Based on the definition of evaluation, we can then evaluate the success of the project by first stating the criteria used to construct the evaluation. Criteria that could be used to guide the appraisal of the programme includes, but are not limited to

  • Relevance: The value of this programme should be evaluated with regards to whether it addresses the needs of the stakeholders (i.e. the orphans involved, the organisations that have implemented the programme) and whether the UN Millenium goals and various UN Conventions have been met during the implementation of the programme. For instance, for an organisation that has chosen to adopt the programme, are the needs of the orphans in line with the values of the organisation after implementing the programme and are these in line with the international and local regulation requirements with regards to child adoption laws and child rights?
  • Efficiency: Has the programme achieved the intended results while utilising resources available in the most economical way possible?
  • Effectiveness: Was the programme output satisfactory as compared to the expectations set?
  • Impact: The various outcomes, both negative or positive, should be analysed. Other perspectives such as the economic, social, political effects on individuals, communities, societies and the impact on the national level should be taken into consideration as well.  
  • Sustainability: What is the likelihood of this programme continuing its operations independently in the various organisations once support from ONETRACK International has been withdrawn? Can this likelihood be measured? Additionally, would this programme be emulated in other organisations in the region?

 

(3.2) CONTENT IN AN EVALUATION REPORT

An evaluation report should contain the following:

  • Findings: What are the facts( quantitative and qualitative) that could be ascertained from the outcome of this programme?
  • Conclusions: A general statement that relates to the findings section. Causal relationships can be stated here as well. 
  • Recommendations: Suggestions on how to improve the processes in the future and to be more thorough and specific, to also include suggestions of improvement in particular situations with specific contexts and circumstances. 
  • Lessons learned: Lessons can be drawn from the conclusion section which is generally stated to include the lessons that would have a wider impact on the community, society, nation, region and perhaps even the international community. 

 

(3.3)  TYPES OF EVALUATION

There are two broad categories of evaluation which are elaborated below: formative and summative evaluation. 

 

Formative Evaluation:

Formative evaluation is almost similar to the monitoring segment because it is implemented during the program’s development in order to establish the direction of the program and identify methods that can be improved to achieve the program’s outcome. Some difficulties that can be experienced from this evaluation includes transforming the outcomes of this evaluation into innovative solutions and therefore improvements to the programme. There are two segments that can compromise formative evaluation and they are elaborated below:

 

Needs Assessment:

Is used to determine the needs of the organisation by identifying the any gaps that hinders the programme from achieving its goal. These gaps includes but is not limited to those in knowledge, practices or skills. Firstly, the alignment of expectations of both the organisation and program outcomes should be laid out. 

 

Process Evaluation:

Process evaluation is used to determine and measure the direct outputs (i.e. outreach, number of children successfully enrolled in school etc.) of the program implemented and whether the intended outcomes were achieved. By utilising the outcomes, improvements with regards to the processes of implementation can be ascertained. Such evaluation can be conducted throughout the lifetime of the program

 

Summative Evaluation

The Summative evaluation encompasses two types of evaluation: outcome evaluation and impact evaluation. These evaluations should be conducted once the program is implemented and is useful to know if the goals of the program are being achieved. One can therefore decide, based on the achievements of the program up while it is still being implemented, whether it should still continue its expansion or to be discontinued. The outcomes of these evaluations can thus be used to justify the expansion or discontinuation of the program. 

 

Outcome valuation:

Outcome evaluation measures the effects of the program on the target population by assessing how much progress has been made to address the problems that the program seeks to address due to the implementation of the program. The outcomes to be measured should be changes observed in the short term and medium term that results from the implementation of the program in terms of changes in perhaps attitude towards child institutionalisation etc. The outcome component from the logic model ( as seen in chapter 2 ) can be used to design this part of the evaluation

 

Impact Evaluation:

Impact evaluation measures how the program affects long term outcomes, whether these effects are intended or not. These impacts should encompass the overall effects on the community, organisation, society and environment. In order to conduct the impact evaluation, there is a need to establish what was the situation before the program was implemented. I.e. comparing the differences in outcomes with a group of children where the program was not implemented against a group of children where the program was implemented. Ultimately, through impact evaluation, objectives of lesson-learning and accountability can be realised.  

 

(3.4)  PURPOSE OF EVALUATION

Evaluation is usually conducted towards the end of the program or even during the midpoint of the program’s progress. Therefore, evaluation can be used to :

  • Improve the efficiency and effectiveness of the strategies outlined to be implemented in the program. 
  • Allowing the coordination between different target groups to be made transparent in order to ensure that the tasks of these respective groups are completed and whether it fits the overall agenda to accomplish the goals laid out. 
  • To allow accountability to be made possible regarding the target group’s contribution. 
  • To provide information on whether the strategies outlined are sustainable in the long run and therefore, the program can be sustained in the future. 

In order to allow these objectives to be realised, it is important to ensure that stakeholders are well informed to make the appropriate choices, greater team work with partners, ensure that commitments or goals have been met (short and long terms), honor your team’s work and show relevant stakeholders within the community why they should continue to remain invested in the vision and goals of this program

 

Items Monitoring Evaluation
Purpose
Main action
Focus
Frequency
Undertaken by
Reporting to
Information sources

Table 5. Comparisons between Monitoring and Evaluation

 

CHAPTER 4: REFERENCE MODELS

This chapter aims to introduce the logic model and theory of change that can be used to evaluate the processes and outcomes of the programme. There are further elaborations on the methodology for both models. 

(1) LOGIC MODEL

The logic model should be developed at the beginning of the implementation of ONETrack International’s In House Sponsorship program so that all stakeholders, partners and program managers have their understanding in line with regards to the goals, expectations and processes of the program and the way it is conducted throughout its lifecycle. It is thus a reference point for those involved in the programme implementation. This model is meant to show the processes involved in the specific pathway taken during the programme in order to achieve its objectives. Therefore, this model aims to identify the issues of this programme to improve it and to omit items that are no longer relevant as the programme progresses to reach its objectives. The dynamic relationship between resources, activities and goals are then reflected in the model. 

The components of the model includes the input, output, expected outcomes, assumptions, means of verification, indicators and other external factors involved during the lifecycle of the program along with illustrating the assumptions, rationales and purposes of this programme. These would be discussed in greater detail below. A description of the main problem at hand and the target audience of the program should also be mentioned. It should also be noted that the logic model should be amended as the program progresses along instead of being done at the end of it to avoid missing out important information. 

 

(1.1) INPUTS

It is essential to include the resources required for the development of the program to budget the necessary resources before its commencement. These resources can include, but are not limited to, the contributions, staff needed and investments that are needed to go into the program. There could already be resources that are already available to the organisation and/or resources that are yet to be acquired. These can be categorised into a ‘wish list’ of items and can include intangible items such as connections that need to be built with the relevant people in the community, region and country. 

INPUTS

                    Necessities                                                     Wish List

  Table 2: Inputs required for project development

 

(1.2) OUTPUTS

The outputs produced by the implementation of the programme include the activities, services, events  and products that reach the program’s target audience which, in this case, are children placed in orphanages. Outputs list the end product of the organisation’s programme and they represent a sign of progress to achieve the programme’s goals. Unlike outcomes, outputs are not utilised to measure the value of the impact or value of the programme to the beneficiaires. With regards to ONETrack International’s In House Sponsorship program, outputs can include the number of children who have transitioned from the care of orphanages into the care of their families and/or relatives. They can also include the number of children who have received medical care and education. 

 

(1.3) OUTCOMES  

The outcomes of the programme represent the benefits received by the beneficiaries due to the outputs of the programme and serves as indicators of the success of the processes adopted by the programme. It is essential to differentiate between the outputs and outcomes components of the logic model. Therefore, while the outputs focuses on how the goals of the programme are being achieved, the outcomes would focus on what the programme is trying to achieve. In the case of ONETrack International’s In House Sponsorship program, the output of more children transitioning from orphanages into the care of their families and relatives has resulted in a short term outcome of lower population of children inside orphanages. 

Outcomes can be categorised into short, mid and long term goals as seen in Table 3 below. The inputs and outputs described should systematically lead to the achievement of the Short term and mid term outcomes while the long term outcomes should ultimately lead to the solution to the overall problem. In order to determine if the long term outcomes has led to a meaningful and better situation as compared to the situation before the implementation of the programme, a baseline should be established on the targeted areas before the commencement of the programme i.e. how many children in orphanages had access to clean drinking water before the implementation of the programme. 

Outcomes of ONETrack International’s In House Sponsorship program

Short Term

Long Term

Access to clean drinking water as a child Better health situation as an adult
Higher number of years attending school Find better jobs as adults
Complete Vaccination Stronger immune system as an adult
Access to rice, cooking oil etc. Better health situation as an adult
Lower population inside orphanages Develop orphanages
Access to health care Informed on health care value as an adult. 

Table 3: Short term and long term outcomes of ONETrack International’s In House Sponsorship program  

 

(1.4) ASSUMPTIONS

The aim of assumptions being made is to ensure that the relevant elements are already in place before rolling out the programme. Assumptions are usually identified before the inputs are listed and they are important to allow for identification of factors that are outside of the control of the programme if it fails.

The assumptions made must be made on the condition that they are:

  • Outside of the project’s control and they must exist or take place for the programme to be successful. 
  • These conditions may be  actions of certain groups, or Project stakeholders, 
  • Certain economic or social conditions, such as the absence of conflict, 
  • Political conditions, such as stability  
  • Conditions of climate 

These assumptions required when implementing ONETrack International’s In House Sponsorship program can be, but are not limited to, the following:

  • Orphans are not receiving the basic health care, educational and nutritional requirements in orphanages. 
  • Children in orphanages are not meeting the national average number of years that children receive education in the country.
  • That all children in participating orphanages have relatives and extended family members who wants to take care of them.  

 

(1.5) EXTERNAL INFLUENCING FACTORS

External influencing factors are factors that are outside the control of the project manager but may have an impact on the programme’s progress. Such factors can include, but are not limited to, the following:

  • The behaviour of the children who are transitioned from the participating orphanages to the care of their families. 
  • Willingness of families and extended relatives to take care of children from the orphanages despite the financial assistance from the ONETrack International’s In House Sponsorship program.
  • Possibility of the orphans being adopted by other people within the country. 

 

(1.6) INDICATORS

Indicators serve as data to be collected to be used as evidence to measure the effectiveness of the outcomes of the programme by determining whether the performance standards set for the programme have been reached. They can measure any intangible changes in situations or groups and tangible products produced from the activities. The data collected has to be compared to the baseline data to measure the changes experienced due to the implementation of the programme. Indicators need to be based on the below factors:

  • Independent: they measure only the objective, purpose or result to which they are linked 
  • Factual: they are based on factual measurement
  • Plausible: it must be believable that they are measuring the change attributed to the project  
  • Objectively verifiable: we can verify whether they have been achieved

 

INDICATORS TO ASSESS OUTCOME OF PROGRAMME

Average number/ratio/basic indicator in the region (baseline)

Average
number/ ratio/ basic indicator for orphans in the region

Average
number/ ratio/ basic indicator for children in the in-home project of CameroonONE 

Years attending school Secondary education 57.3% (male)- 33% female.

Primary education: 70% nationwide

? ?
Child mortality/ dying from preventable causes 21/1000 under five ( one of the five highest ratios in the world) ? ?
Monthly expenses for
a child
48% under poverty line (2010 estimation by world bank) $7 per month based on CameroonONE info ~$25 per month
Vaccination status 80.8% nationwide-primary vaccination  100%
Access to
safe drinking water
30.7% nation wide (UNICEF report 2012) ? 100%
HIV positive 11.8 nationwide ages
15-49
UN 2013 report
Over 15% (classified)

Table 4: Table showing the indicators used by CameroonONE to assess the impact of their output.

 

The data in Table 4 illustrates the indicators utilised to assess the outcomes of CameroonONE’s In House Sponsorship program and therefore, the impact that it has on the community.  I.e. If the average orphan in the region attends four years of school and the children supported by CameroonONE receive eight years of school in average this is not only an output it is also a short-term impact.

(1.7) MEANS OF VERIFICATION

The means of verification (MoV) is used to describe how the indicators will be measured. The MoV is therefore, required to be created after the indicators are selected as it is a source of data to be used as the “proof” for the indicators chosen. I.e. the data selected can be documents or statistics to show that the standard set by the indicators has been achieved.

Logic Graphic

 

(2) THEORY OF CHANGE

The Theory of Change (ToC) represents the various pathways that can be taken to address the causes of the problems outlined (i.e. institutionalised children), identify solutions and guide the programme manager to make decisions as the programme progresses in order to achieve the objectives of ONETrack International’s In House Sponsorship programme. These can include pathways that are not even directly related to the In House Sponsorship programme. The ToC is meant to be a flexible instrument, as opposed to the rigid logic framework, to be used to plan and monitor the measures taken during the programme,  taking into account the conditions of uncertainty of specific contexts. It also takes into account the underlying assumptions and risks that can be revisited as the programme unfolds to ensure that the programme is on the right path of achieving the desired goals. However, its function is similar to that of the logic model in the sense that they are both required to provide accountability and transparency to stakeholders involved in the processes. 

(2.1) METHODOLOGY

In order to develop the ToC, its methodology should be outlined to understand the components required to create it. This includes:

Desired Change: 

The change that the programme aims to reach should represent a convergence of relationships, conditions and results that the programme manager would like to achieve due to the action of the programme in the current and/or future contexts. The kind of change that we would like to see in the orphan sector would influence the specific sector that the programme manager would be focusing on when implementing the programme. I.e. the Temporal, relational, structural, geographic, social, cultural, economic, political, institutional dimensions. Therefore, it is likely that ONETrack International’s In House Sponsorship programme would focus on the social, cultural and institutional aspect of society to ensure that the culture of the community is retained by ensuring that the orphans are transitioned into the care of their families. It also addresses the social aspect by ensuring that these orphans have their health care and educational needs met. Lastly, besides just the changes that needs to be identified and the sectors to be addressed, the mindsets and interests of the programme managers who formulates the ToC should also be taken into consideration to rectify any biases that could occur.

Actors Involved:

There are numerous actors who hold several roles in society that adds to the dimension of difficulty in the attainment of the solution to the problem at hand. These stakeholders may also involved in the processes we seek to make and they may be affected by these changes or who are already part of the reality that we wish to influence. Playing numerous roles concurrently results in them having overlapping interests. These conflicting interests should be identified in order to identify the respective individuals or groups who have the ability to block or accelerate in the change processes. Therefore, possible strategies should be created to allow greater access of the project to those from the latter group.  By formulating these strategies, we can then study how to collaborate with those from who can hasten the processes to reach the ultimate goals of the programme and to neutralise those who would put up some resistance to it. This therefore, contributes to the complexity of how effective, cooperative and comprehensive the processes of change will be when achieving the desired change. 

Underlying Assumptions: 

Assumptions should be able to connect the outcomes with the conditions, that the program will be dependent on, that has to be identified prior to the implementation of the programme and establish the risks involved when outlining them. In other words, the assumptions laid out should be relevant to the achievement of the outcomes while accounting for other factors that are out of our control. These assumptions should also be able to explain why these outcomes would be able to lead to the desired change we want out of this programme. The flexibility of the ToC is such that we are able to revisit these assumptions and modify them when they are no longer realistic or reasonable in achieving our targets. If these assumptions are not defined clearly, it would be difficult to determine what factors the success of the programme would rely on. 

Pathway to Change: 

A pathway of change is a sequence of outcomes that takes places over the span of the lifetime of the programme while also accounting for the assumptions that would directly or indirectly affect the progress of the programme in order to achieve the ultimate goals. The diagram (Figure 1) of the ToC would illustrate this relationship between actions, conditions and outcomes as well as how the different outcomes are interrelated. The diagram illustrates the long term goal at the top of the pathway map and the subsequent outcomes from each layer that must be attained in order to reach the ultimate long term goal. Therefore, the attainment of the desired goal is dependent on the fulfilment of the other outcomes. Since there are multiple pathways that can be taken, we should trod on the pathway that allows us to take the most efficient and effective route to reach the end. As the ToC involves a non-linear approach to attaining the ultimate goal, it should be noted that the different stages of outcomes may or may not occur along the timeline of the implementation of the programme. They may not even be relevant to the programme overtime. Therefore, the components of the pathway of change that includes the outcomes, assumptions and indicators should be revisited to be reviewed if they are still relevant to the attainment of the ultimate goal and if the outcomes have even been achieved. These revisions can include the addition of new outcomes, assumptions and indicators. It should be noted that the fulfillment of the outcomes may or may not change over time and they should be revisited from time to time to ensure that they are still relevant to attain the ultimate goal. Thus, relevant adjustments should be made to the outcomes and assumptions as the programme progresses that may emerge due to the complex interactions between these three elements and the contribution of the stakeholders. Therefore, we must avoid assuming that all of the assumptions and outcomes that have been listed out will definitely unfold along with the implementation of the programme that was initially envisioned by the programme manager. To avoid overlooking certain conditions, it is important to a timeline on when these elements are to be reviewed and to cross check them against stakeholders with different perspectives so that all angles of the M&E process of the programme are covered. 

Change Indicators: 

The indicators of change in the ToC framework are not the same as the output indicators used in the Logical Framework.  These indicators of change should be formulated in a way that would allow us to understand the context at that moment and the effects that we can gain from our interventions in that context. In this way, the indicators would put us in a position that would allow us to understand how the changes we have made are happening(or not) and what are our contributions to that change. Therefore, each outcome at every subsequent level has their own respective indicators of change. The relative importance of the effects of the outcomes allows us to prioritise the indicators of change accordingly to track and monitor the relevance of these outcomes. The aim of these indicators is to ensure that the actions we implement have the desired effects produced from the outcomes obtained in order to know if the interventions are done not for the sake of implementing them. When constructing these indicators, we should consider the degree to which our actions are contributing to the outcomes and ultimate goal. 

It is only appropriate that when considering the above components, validation of them against evidence and the opinions of the internal and external stakeholders are taken into account. This is also to ensure that the assumptions are conceivable and analysis conducted are reasonable. Ultimately, the ToC must be aligned with the end goals and principles established of ONETrack International’s In House Sponsorship program to ensure that the various pathways forged will lead to our final destination.

Theory of change graphic

 

CHAPTER 5: DEVELOPING INDICATORS

Indicators measure the project impacts, outcomes, outputs and inputs due to the implementation of the project to assess the progress of the programme and compares the outcomes of the programme with a baseline so that the success of the programme can be ascertained. It enables the establishment of the relationships between these factors to identify gaps in the project’s progress that may hinder the attainment of the project’s objectives

        (1) CORE INDICATORS

Core indicators are used to understand the inputs and outcomes of a program. 

Quantitative indicators:

Quantitative indicators measure the outputs of the program that can be easily defined, documented,  counted and compared. They comprise of numbers and facts that can be verified. 

  • Output indicators: Output indicators would measure whether the goals of the program are attained and adds more details on the program. However, output indicators mainly provide us with the information on whether the planning of the program has been executed but they are not able to measure the effects of these outputs. They can also fall under process indicators which measures the programmes activities and outputs.
  • Outcome indicators: These indicators refer to the measurement of the ‘results’ of the outcome of the program or rather the impact of the of the program. These indicators would be useful in determining the reasons for implementing the program in the first place and why certain actions were conducted, or not. 
  • Input indicators: Although it may be obvious that indicators should be tracked, it is important to determine if the amount of inputs that are utilised are also used at the right time. 
  • Cost indicators:

 

Qualitative indicators:

These indicators are useful in assessing the success of the program where it is not easily quantifiable. These factors that are not easily quantifiable are used to answer the whys of different situations and the different contexts. This may, however, be useful because the implementation of the program involves the changing of the children’s lives and it may be insufficient to capture these changes via only numerical means. This is especially useful in measuring the impact and evaluating the long term effects of the program from the perspective of different groups of people and age groups i.e. evaluating women’s rights and children’s rights with respect to the empowerment and development. For example, the future prospects that children feel by being given an opportunity to further their education. Additionally, these qualitative indicators would be significant in identifying any constraints to the implementation of the program and any hindrances to the success of the program which may not be apparent initially.  

     (2) PROGRAMME INDICATORS

Programme indicators are designed to identify and summarise information collected regarding the programme. 

Independent indicators:

Independent indicators are sets of indicators that are not formally linked. They can be collected by staff and even though the results of those indicators used at the programme level may depend on activities carried out at project level( lower level), the sets of indicators collected are not formally linked. 

Framing or basket indicators

Basket indicators are summaries of specific indicators that are captured and summarised at the project level. These indicators are used to identify broad areas. 

Aggregated indicators:

Aggregate indicators are constructed to measure the progress towards the goal of the programme and are not specific indicators but rather indicators used to define more specific indicators.

E.g. The percentage of children who attend school after enrolling in the ONETRACK  programme. 

For instance, if the goal of the programme was for 100% of children in an orphanage that adopted the ONETRACK programme to be graduates, the above percentage could inform us whether there were any behavioural changes of the children towards pursuing a higher education due to the implementation of the programme or not. Therefore, other more specific indicators would be needed.

Translated indicators:

Independent indicators that were used to measure different factors that could be ‘translated’ into a common indicator as the programme progresses. For instance, the number of children who are enrolled in degree programmes and the number of children who are enrolled in masters programmes could be categorised under a common indicator, the number of children who received tertiary education.

 

CHAPTER 6: DATA COLLECTION

Data collection is required to conduct the monitoring and evaluation process. However, when collecting the data, there may be a conflict of interest or ethical considerations that may arise and has to be considered. The table below lists some of these considerations but they are not exhaustive. 

Data Collection Tools Potential ethical issues/ considerations
Stakeholder workshops, meetings and interviews  ▪ Dealing with sensitive issues that may be raised by stakeholders
▪ Dealing with disclosures
▪ Privacy and confidentiality of data collected
Participant Before / After surveys  ▪ Implications if respondents are not literate, or do not feel they have been able to adequately grasp concepts
▪ Privacy and confidentiality of data collected
Tracking Formats Follow Up Surveys ▪ Respecting rights of people or children in this case to drop out of the survey or not be tracked
▪ Being aware and sensitive to people getting tired of being asked questions by projects (respondent fatigue) in data collection
▪ Privacy and confidentiality of data collected 
Direct Observation  ▪ Identification of the observer and their role and intended use of observations
▪ Seeking informed consent

When implementing this data collection methods, the following questions should be posed:

  • Would the data collection methods you are using (e.g. qualitative or quantitative tools or both) be sufficient to capture the diverse opinions of all participants regardless of their comfort levels with sharing personal information?
  • As the participants would be children, have the literacy and level of comprehension been considered in the design of your collection methods? You may want to consider alternative methods or a combination of them when collecting the data to address these possible obstacles. 
  • Is there a need to consider how to encourage the children/orphans to express their insights especially in a context where it may not be conventional or comfortable for them to do so? And if so, is there a possibility that you would need to pose the same question in a different way? 
  • Have you considered if the questions you would be asking have been modified to suit the specific group of people you are asking?

 

CHAPTER 7: REPORTING

Finally, constructing the M&E report in a clear and succinct way is critical to ensure that the overall impact of the program are properly conveyed and understood by stakeholders. The tone of the report is usually formal but stakeholders or donors may require the organisation to use mandatory report templates or formats which can include a financial summary and results of the program. The report would usually consist of an introduction, body and conclusion. When drafting the report, the following should be addressed:

  • Who has requested the report? 
  • Why have they asked for a report? 
  • What do they need to know? 
  • How will the report be used? 
  • Who is/are my audience or audiences?
  • What type of report is needed? (e.g. internal report, external donor report, proposal?) 
  • How long does your report need to be? 
  • What is required in the report? 
  • What is the problem/question to be solved? 
  • What is the aim of the report? 
  • What key points or issues need to be addressed?
  • What information do you need to collect, provide and analyse? 

Subsequently, after considering the above, you may commence your report. Basic rules of report writing are as follows:

  • Level of reporting: If the report is an annual report, a broad overview of the impact and outcomes should be stated instead of being too detailed.
  • Structure: Ensure that you are clear on how you would like to display the content of the M&E report
  • Layout: As this is a formal document, Font sizes 11 or 12 should be used. Visuals to illustrate the outcomes of the program and they should be explained.

 

CHAPTER 8: DISSEMINATION PLAN

This last component of the M&E plan describes how and to whom the data obtained from the plan will be disseminated and more importantly, the purpose of the data collected. 

Questions to consider when deciding how the data should be utilised:

  • How will the data derived from the M&E plan be sued to update internal and external stakeholders of the progress of the program?
  • How will this data be used to assist internal stakeholders to tweak existing processes to ensure that the program is on course to being successful?
  • Ultimately, how would this data be useful in pushing the periphery of existing similar programs and therefore achieving the intended goals?

The following steps can be formulated when disseminating updates on the program:

  • Monthly data review to make necessary changes to the existing plan and develop future ones. 
  • Organise quarterly meetings with stakeholders and donors to evaluate existing data and the progress of the program.
  • Digital dissemination of weekly or monthly progress of the program to internal and external stakeholders such that these updates become a routine that would lead ultimately lead to the outputs that the program aims to achieve.

 

References

https://www.newstatesman.com/world/2017/03/world-must-wake-dangers-orphanages

https://www.globalgiving.org/pfil/10203/projdoc.pdf

https://www.thestar.com.my/news/world/2006/02/18/orphanages-stunt-growth-foster-care-better–study/

http://www.un.org/documents/ga/res/41/a41r085.htm

http://www.un.org/documents/ga/res/41/a41r085.htm

http://www.unaids.org/sites/default/files/sub_landing/files/2_MERG_Strengthening_Tool_12_Components_ME_System.pdf

https://evaluateblog.wordpress.com/2013/05/03/the-12-key-components-of-me-systems/

https://www.unicef.org/violencestudy/pdf/CP%20Manual%20-%20Stage%206.pdf

https://evaluateblog.wordpress.com/2013/07/02/types-of-monitoring-in-monitoring-and-evaluation-me/

http://www.mnestudies.com/monitoring/what-monitoring

https://evaluateblog.wordpress.com/2013/07/02/types-of-monitoring-in-monitoring-and-evaluation-me/

https://www.unicef.org/violencestudy/pdf/CP%20Manual%20-%20Stage%206.pdf

https://www.linkedin.com/pulse/20141022071803-18927814-a-good-start-with-s-m-a-r-t-indicators/

https://www.unitar.org/sites/default/files/uploads/pprs/monitoring-and-evaluation_revised_april_2017.pdf

http://library.cphs.chula.ac.th/Ebooks/ReproductiveHealth/A%20UNICEF%20Guide%20for%20Monitoring%20and%20Evaluation_Making%20a%20Difference.pdf

http://library.cphs.chula.ac.th/Ebooks/ReproductiveHealth/A%20UNICEF%20Guide%20for%20Monitoring%20and%20Evaluation_Making%20a%20Difference.pdf

https://www.unicef.org/evaluation/files/ME_PPP_Manual_2005_013006.pdf

https://www.unicef.org/evaluation/files/ME_PPP_Manual_2005_013006.pdf

https://cyfar.org/different-types-evaluation

https://www.cdc.gov/healthcommunication/pdf/evaluationplanning.pdf

https://aceproject.org/ace-en/topics/ve/veh/veh02

https://www.cdc.gov/healthcommunication/pdf/evaluationplanning.pdf

https://www.cdc.gov/healthcommunication/pdf/evaluationplanning.pdf

https://www.cdc.gov/std/program/pupestd/types%20of%20evaluation.pdf

https://blog.socialcops.com/academy/7-types-of-evaluation/

https://www.cdc.gov/healthcommunication/pdf/evaluationplanning.pdf

https://www.cdc.gov/std/program/pupestd/types%20of%20evaluation.pdf

http://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf

http://masstapp.edc.org/prevention-planning/step-5-evaluation

http://eeas.europa.eu/archives/delegations/ethiopia/documents/eu_ethiopia/ressources/m_e_manual_en.pdf

http://eeas.europa.eu/archives/delegations/ethiopia/documents/eu_ethiopia/ressources/m_e_manual_en.pdf

https://undg.org/wp-content/uploads/2017/06/UNDG-UNDAF-Companion-Pieces-7-Theory-of-Change.pdf

http://www.democraticdialoguenetwork.org/app/documents/view/en/1811

http://siteresources.worldbank.org/BRAZILINPOREXTN/Resources/3817166-1185895645304/4044168-1186409169154/24pub_br217.pdf

http://www.infodev.org/infodev-files/resource/InfodevDocuments_286.pdf

http://www.genderevaluation.net/gem/en/gem_tool/step4a2.htm

http://www.emro.who.int/child-health/research-and-evaluation/indicators/All-Pages.html

https://www.cdc.gov/eval/indicators/index.htm

http://www.genderevaluation.net/gem/en/gem_tool/step4a2.htm

https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Programme-Indicators-9.pdf

https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Programme-Indicators-9.pdf

https://www.devex.com/news/indicators-logframe-and-m-and-e-system-78031

https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Programme-Indicators-9.pdf

https://pacificwomen.org/wp-content/uploads/2018/03/Monitoring-and-Evaluation-Toolkit-November-2017.pdf

https://pacificwomen.org/wp-content/uploads/2018/03/Monitoring-and-Evaluation-Toolkit-November-2017.pdf

https://www.thecompassforsbc.org/how-to-guides/how-develop-monitoring-and-evaluation-plan

About the author