A project commissioned by the Marine Management Organisation and published in February 2016 aimed to review the Monitoring and Evaluation framework presented in the MMO’s East marine plans Implementation and Monitoring Plan to assess whether it is fit for purpose and can provide a framework that can be utilised for subsequent marine plans. Where necessary the report has made recommendations for improvements which were subsequently taken into consideration in the development of the South Marine Plan Approach to Monitoring.
Questions this practice may help answer
- How can a monitoring and evaluation framework be assessed?
- What review criteria can be used to assess a monitoring and evaluation framework?
Implementation Context
The aim was to review the M&E framework to assess whether it is fit for purpose and could be utilised for subsequent marine plans, and where necessary make recommendations for improvements. A key area in this regard was the counterfactual1 and how this might be used for marine plan impact evaluation. The project research was undertaken in two parts.
Part A undertook a detailed review of the M&E framework, testing it against a set of six criteria and providing key recommendations which could improve the performance of the framework against each criterion. The criteria, identified by the MMO, were; relevance, completeness, robustness, evidence and data, governance and resources and wider applicability.
Part B undertook a short literature review of the potential approaches for defining counterfactuals and provided conclusions on their relevance, the implications of this for the overall evaluation approach, and how counterfactuals might be better incorporated into the current framework. Counterfactuals are a commonly used approach for thinking about causation in impact evaluation. In essence they try to address the question “What would have been the situation of the project area or population had the policy intervention not taken place?”
Aspects / Objectives
The primary aim of this project was to review the M&E framework presented in the MMO’s East marine plans IMP to assess whether it is fit for purpose and provides a framework that can be utilised for subsequent marine plans, and where necessary make recommendations for improvements. A key area in this regard was the counterfactual. The project specifically reviews the options for counterfactuals, considered their appropriateness and provided guidance on how counterfactuals may be utilised in a marine plan evaluation.
By establishing a robust M&E framework at this stage, the analytical structures, data needs and processes can be considered well in advance of both undertaking the evaluations and developing other marine plans. This has two distinct benefits. Firstly it enables forward planning for the evaluation, helping to ensure that the information will be available to allow the preferred evaluation methodologies to be employed. Secondly, it enables a consistent framework to be utilised across all marine plans, aiding efficiencies in evaluation processes and improving the extent to which marine plan effects can be contrasted and aggregated – thus helping to satisfy the requirements of the Marine and Coastal Access Act (2009).
Method
The study consisted of two parts. Part A undertook a structured review of the East marine plan M&E framework by exploring a defined set of research questions, set out under six review criteria, which respond to the project objectives and incorporate best practice thinking on M&E frameworks. The approach sequentially addressed each of these research questions to understand where there may be weaknesses and how these may be addressed in order to ensure that East marine plans can be successfully reported on in 2017. The review criteria, as defined by the MMO, were:
- Relevance: how well the M&E approach represents its stated requirements, the requirements in The Act and expectations of stakeholders.
- Completeness: whether the M&E framework and its underlying logic models cover all of the relevant objectives and issues in a way that will enable the M&E questions to be explored.
- Robustness: the degree to which the M&E framework is able to adequately capture and reliably measure the main effects of the marine plans are the certainty that can be placed on the outputs expected.
- Evidence and data: the evidence and data needed to implement the framework, and the adequacy of the systems in place to collect them.
- Governance and resources: the sufficiency of the governance arrangements and resources available for M&E.
- Wider applicability: whether the framework can be applied more widely across the other marine plans and to the effects on the High Level Marine Objectives (HLMO) objectives.
Part B reviewed possible approaches for baselines and counterfactuals and provided conclusions on their relevance for marine plan evaluation. It considered the implications for the overall evaluation approach of the counterfactual options and provided recommendations on how counterfactuals might be enhanced within the current M&E framework. Drawing on experience gained by delivering this project and the challenges encountered, as well as broader Defra evaluation needs, a short article has been developed, focussed on developing and reviewing M&E frameworks and plans. The article, intended for internal use, is not included in this report. The study has benefited from ongoing feedback and meetings with the project Steering Group, which consisted of MMO and Defra staff, on key interim and draft outputs.
Main Outputs / Results
The main output of the study is the report “Review of the Marine Planning Monitoring and Evaluation Framework and Development of Baselines” which can be accessed here.
The report concluded that the approach taken satisfies the basic M&E requirements of the MCAA and as such can be used as a framework to draft IMPs for future plan areas, however in order to fully satisfy the requirements of the MCAA a number of issues should be addressed.
The primary recommendation from the report is that the components of the logic chains, (i.e. whether your input and activities will provide the desired outputs and outcomes) should be re-specified to provide an understanding of how plan policies affect the higher order indicators and provide a basis for articulating the contribution to the achievement of plan and high level marine objectives. It is expected that the logic chain should seek to include some of the intermediate outcomes that plan policies are trying to express, it is here that it is expected marine plan policies have the most effect. Describing the intermediate outcomes will also serve to reduce the reliance on assumptions within the logic chain by reducing the logical jump between steps.
The M&E framework does not fully satisfy the relevance criterion as it omits methodologies to enable economic evaluation.
Where policy signals are weak and nonexperimental counterfactual approaches are employed, the importance of explicitly accounting for external factors is heightened. A poor understanding of what these external factors are, how significant they are and how they have changed over the period under evaluation will limit the extent to which conclusions can draw on causality and attribution. Effort should be made up front within the M&E framework to identify relevant external factors and potential indicators so that baseline and time series data can be collected and used to help interpret observed effects.
Transferability
The study is specific to the UK although the methodology to review a monitoring and evaluation framework could be applied elsewhere.
Responsible Entities
Marine Management Organisation
Costs / Funding Source
Marine Management Organisation
Contact person
Marine Management Organisation Evidence Team