Workplace Health and Safety Strategy Evaluation Framework
7. Impact/Outcome Evaluation
Purpose of impact/outcome evaluation: evaluation activity is directed at determining the positive or negative intermediate or longer-term impacts and outcomes of a programme. The evaluation looks at the impacts the programme has made and whether the programme has achieved the outcomes it is seeking i.e. healthy people in safe and productive workplaces. It is also an opportunity to look at the three intermediate outcomes.
This element of the evaluation will look at impacts and outcomes of the Strategy. It will provide an overall assessment of how the Strategy has worked. Outcome indicators will be developed and refined in the first three years of the Strategy for measuring progress against. These will include indicators already developed such as the Valid Injury Outcome Indicators for the New Zealand Injury Prevention Strategy as well as other workplace injury statistics from Statistics New Zealand.
In the case of this Strategy, the indicators developed will refer to the situation of workplace health and safety in New Zealand. When determining the success of the Strategy as an intervention, stakeholder interviews will be important as they are the key influencers in their sectors. The Evaluation Framework has Stakeholder interviews every three years.
In determining the impact of the Strategy, evaluation will focus on the intermediate objectives. At this stage, the idea would be:
- Industry leadership and community engagement. Stakeholder interviews would assess these aspects of the Strategy by interviewing key individuals in both sectors
- Preventive workplace cultures. A workplace monitoring survey will be used to assess changes in the workplace
- Government leadership and practices. Stakeholder interviews and meta-evaluation will be used to determine any changes.
The outcome of this exercise will be:
- Information on whether the Strategy contributed to outcomes
- Information on the impact of intermediate outcomes/impacts
- A path forward for further work in workplace health and safety
- Detailed information on what worked and what did not work in the WHSS. This information can be used in refining any further work undertaken in this area
Key questions to be considered in the evaluation
- Is the Strategy reaching its intended target? Could be individuals targeted in particular workplaces or specified targets within the Strategy.
- Do individuals have a better understanding of what the Strategy is trying to achieve?
- What has been the impact of deliverables in the Action Plan?
- What are the consequences - intended and unintended?
- How has Strategy contributed to workplace health and safety outcomes?
- What impact did particular programmes have in helping to achieve outcomes?
- What has worked and not worked in the Strategy?
- What else could be done to improve workplace health and safety based in the information we have gathered?
|Process evaluation projects|
|Evaluation activity||Timeline||Method||Issues and ideas|
|Determining indicators and measurement for outcomes (Vision and three outcomes)Regular reporting on indicators once the Strategy has been in place for a certain amount of time - perhaps every two years. Publication of potential indicators for feedback on what stakeholders think of them.||
Identify and develop all indicators by June 2006.
Test and trial indicators by June 2007
Report and publication of indicators in June 2008
Determining outcome measures
Projects that feed into the development of indicators are outlined below:
|Measurement of indicators against outcomes||From July 2008 onwards, report on outcomes every three years||Measurement of outcome indicators
There are three outcome indicators (potentially four with exposure) to be measured. At the three-year stage, there should be agreement on what is to be measured, then three yearly reports following this.
|Regular Stakeholder interviews||June 2007 and every three years after||Stakeholder interviews
Stakeholder interviews are undertaken to determine the success of the WHSS as an intervention/lever in improving workplace health and safety. The interviews are also an avenue to test impact the impact of intermediate outcomes for government, industry and community. They will be used to evaluate government leadership and practices and industry leadership and community engagement. This part of the evaluation is closely linked to the survey of workplaces below.
|Monitoring of workplaces to track changes||
Three yearly - starting 2008/09 year.
Would be used for reporting along with Outcome indicators.
|Survey of workplaces to assess change
The survey will track changes in different workplace to measure the intermediate outcome of preventive workplace cultures. My current thinking is that we would develop a stratified random sample to ensure link between workplaces and outcome variables. For example, we might determine that we need to talk to different sized enterprises in certain industries and certain regions. This would be based on the type of coverage we hope the WHSS will achieve. The process would be as follows:· Development of RFP - 05/06 financial year;· Contracting an organisation to develop and pilot survey in consultation with DoL - completed by June 07. This will also be used as baseline information.· Survey implemented after June 2007 to enable feedback into first report on Strategy in 2008.· Survey then completed at regular intervals to be determined. At this stage, the suggestion is every three years.
This document sets out an evaluation framework for the Workplace Health and Safety Strategy. The next step will be to develop detailed plans for all the projects outlined in the framework. The information will include:
- Objectives of the project;
- Research objectives;
- Key questions;
- Method; and
- Reporting strategy.
The above categories are important in developing any Request for Proposal documentation. Each project will, therefore, be developed to a stage where basic information is provided.
It is intended that these project plans will provide a reference point for development of evaluation throughout the Strategy.
It is important to consider the challenges the evaluation will face at the beginning of the process. This will help people involved in the Strategy to look at how these might be addressed. The table below assesses the challenges and how the Evaluation Framework aims to deal with them.
|Evaluation challenge||Description of Challenge||Framework response|
|Complexity of the Strategy||The Strategy is a dynamic entity that is focused on a range of areas with a number of organisations involved.||Naturalistic inquiry will be used to ensure that the dynamic nature of the Strategy is evaluated.|
|Lack of baseline data in some areas||The Strategy is aimed at improving outcomes in areas where there is no or little baseline data||A Stocktake will establish what data there is and we will work from that point on improving data.|
|Lack of control group||Strategy is being implemented nationwide. There will be no treatment group and control group to make comparisons with.||Outcomes - looking at how the Strategy contributed to workplace health and safety outcomes. Specify that Strategy is one of the things contributing; Intermediate outcomes - stakeholder interviews will be used to look at government, community and industry leadership and workplace survey tool will be used to look at preventive workplace cultures. In developing survey, the first year will be used as counterfactual/control for workplaces.|
|Complexity of outcomes.||In some areas, there will be data that can be used to measure outcomes. However, other areas such as occupational diseases may prove a problem.||Research is being carried out to look at measurement at the moment. International experience in this area shows that health and exposure will be difficult to measure.|
|Different types of workplaces||The Strategy includes initiatives for different workplaces but measurement of change within these will be difficult.||Using a stratified random sampling frame will ensure that we get a good representation of workplaces in the survey.|
This framework sets out a number of projects for evaluation of the WHSS. It is intended to provide evaluation information at a number of different levels, in the workplace, at a national level and at a project level. The combined projects should provide the Government with detail about the effectiveness of the WHSS.
The final product will be a detailed evaluation framework. The framework will include information about each project and how it fits within the framework. This document will then be placed on the WHSS internet site for easy access.
or return to the Workplace Health and Safety Strategy homepage