Workplace Health and Safety Strategy Evaluation Framework

7. Impact/Outcome Evaluation

Purpose of impact/outcome evaluation: evaluation activity is directed at determining the positive or negative intermediate or longer-term impacts and outcomes of a programme. The evaluation looks at the impacts the programme has made and whether the programme has achieved the outcomes it is seeking i.e. healthy people in safe and productive workplaces. It is also an opportunity to look at the three intermediate outcomes.

This element of the evaluation will look at impacts and outcomes of the Strategy. It will provide an overall assessment of how the Strategy has worked. Outcome indicators will be developed and refined in the first three years of the Strategy for measuring progress against. These will include indicators already developed such as the Valid Injury Outcome Indicators for the New Zealand Injury Prevention Strategy as well as other workplace injury statistics from Statistics New Zealand.

In the case of this Strategy, the indicators developed will refer to the situation of workplace health and safety in New Zealand. When determining the success of the Strategy as an intervention, stakeholder interviews will be important as they are the key influencers in their sectors. The Evaluation Framework has Stakeholder interviews every three years.

In determining the impact of the Strategy, evaluation will focus on the intermediate objectives. At this stage, the idea would be:

Industry leadership and community engagement. Stakeholder interviews would assess these aspects of the Strategy by interviewing key individuals in both sectors
Preventive workplace cultures. A workplace monitoring survey will be used to assess changes in the workplace
Government leadership and practices. Stakeholder interviews and meta-evaluation will be used to determine any changes.

The outcome of this exercise will be:

Information on whether the Strategy contributed to outcomes
Information on the impact of intermediate outcomes/impacts
A path forward for further work in workplace health and safety
Detailed information on what worked and what did not work in the WHSS. This information can be used in refining any further work undertaken in this area

Key questions to be considered in the evaluation

Is the Strategy reaching its intended target? Could be individuals targeted in particular workplaces or specified targets within the Strategy.
Do individuals have a better understanding of what the Strategy is trying to achieve?
What has been the impact of deliverables in the Action Plan?
What are the consequences - intended and unintended?
How has Strategy contributed to workplace health and safety outcomes?
What impact did particular programmes have in helping to achieve outcomes?
What has worked and not worked in the Strategy?
What else could be done to improve workplace health and safety based in the information we have gathered?

Process evaluation projects
Evaluation activity	Timeline	Method	Issues and ideas
Outcome focus
Determining indicators and measurement for outcomes (Vision and three outcomes)Regular reporting on indicators once the Strategy has been in place for a certain amount of time - perhaps every two years. Publication of potential indicators for feedback on what stakeholders think of them.	Identify and develop all indicators by June 2006. Test and trial indicators by June 2007 Report and publication of indicators in June 2008	Determining outcome measures Identify indicators for measurement of outcomes. This includes, injury, disease, exposure and productivity indicators. These indicators will be used to measure the current state of workplace health and safety in relation to the Outcomes in the Strategy. Prior to the indicators being confirmed, there needs to be work undertaken in identifying potential indicators, their reliability and validity. Indicator work is set out below: Productivity - Colin Cryer’s piece of work on productivity indicators. The result of this work will help us determine indicators to measure productivity. Injury - Regular reporting by the Injury Information Manager at Statistics NZ as well as Valid Injury Outcome Indicators for New Zealand Injury Prevention Strategy (NZIPS). Health - developing indicators based on information obtained from NOHSAC. Exposure - developing indicators based on information obtained from NOHSAC. Projects that feed into the development of indicators are outlined below: NOHSAC report looking at surveillance systems for workplace injury and disease. This report will provide valuable information on where the gaps are and what types of information we will be able to collect for health and hazard exposure. NOHSAC Stocktake. This may provide useful information which we will be able use as a baseline. NOHSAC report looking at the cost of injury and illness. This report will help provide productivity information in terms of cost of incidents.	More work needs to be done on outcome indicators in most areas; In the injury area, there are links with work for NZIPS and Stats NZ - workplace injury information will be published. As indicators are finalised, we can place them on the website. It will important to also show the process of developing indicators and rationale for them. The development of indicators is crucial to the outcome of the project. Indicators provide an effective tracking device for the public. If more accurate ways of measuring health are developed, this may lead to an increase in health-related statistics as professionals become aware and report disease in the workplace more accurately. Both productivity and health may require some qualitative tracking before quantitative measures can be developed and piloted. It may be some time before measures can be achieved in these areas.
Measurement of indicators against outcomes	From July 2008 onwards, report on outcomes every three years	Measurement of outcome indicators There are three outcome indicators (potentially four with exposure) to be measured. At the three-year stage, there should be agreement on what is to be measured, then three yearly reports following this.	This part of the evaluation helps us look at the state of workplace health and safety in NZ at a national level by measuring against agreed indicators. The previous three years will have seen the development of indicators of some description to allow for measurement of some type. Linking with other work will be important and using statistical information from the Injury Information Manager and Valid Injury Outcome Indicator project will help.
Impact focus
Regular Stakeholder interviews	June 2007 and every three years after	Stakeholder interviews Stakeholder interviews are undertaken to determine the success of the WHSS as an intervention/lever in improving workplace health and safety. The interviews are also an avenue to test impact the impact of intermediate outcomes for government, industry and community. They will be used to evaluate government leadership and practices and industry leadership and community engagement. This part of the evaluation is closely linked to the survey of workplaces below.	Helps monitor and track change through talking to key influencers in workplace health and safety. Able to look at whether there is better co-ordination among key actors in area. As stakeholders are the intermediaries, they will also have good idea of whether Strategy is impacting on wider change.
Monitoring of workplaces to track changes	Three yearly - starting 2008/09 year. Would be used for reporting along with Outcome indicators.	Survey of workplaces to assess change The survey will track changes in different workplace to measure the intermediate outcome of preventive workplace cultures. My current thinking is that we would develop a stratified random sample to ensure link between workplaces and outcome variables. For example, we might determine that we need to talk to different sized enterprises in certain industries and certain regions. This would be based on the type of coverage we hope the WHSS will achieve. The process would be as follows:· Development of RFP - 05/06 financial year;· Contracting an organisation to develop and pilot survey in consultation with DoL - completed by June 07. This will also be used as baseline information.· Survey implemented after June 2007 to enable feedback into first report on Strategy in 2008.· Survey then completed at regular intervals to be determined. At this stage, the suggestion is every three years.	Development of instrument could take place over first few years of Strategy with pilot undertaken in the second year. This would mean that the survey could go out in the third year. This is a good way to assess changes in workplaces. A stratified random sample ensures a wide representation across all variables. Could be linked with other information provided through Valid Injury Outcome Indicators and Injury Information Manager. Taking the Strategy to an enterprise level helps to personalise the experiences of these groups and provides tangible information to use. Linking statistics with the stories of what is happening will be valuable to include on the website.

8. Next steps

This document sets out an evaluation framework for the Workplace Health and Safety Strategy. The next step will be to develop detailed plans for all the projects outlined in the framework. The information will include:

Objectives of the project;
Research objectives;
Key questions;
Scope;
Method; and
Reporting strategy.

The above categories are important in developing any Request for Proposal documentation. Each project will, therefore, be developed to a stage where basic information is provided.

It is intended that these project plans will provide a reference point for development of evaluation throughout the Strategy.

9. Evaluation Challenges

It is important to consider the challenges the evaluation will face at the beginning of the process. This will help people involved in the Strategy to look at how these might be addressed. The table below assesses the challenges and how the Evaluation Framework aims to deal with them.

Evaluation challenge	Description of Challenge	Framework response
Complexity of the Strategy	The Strategy is a dynamic entity that is focused on a range of areas with a number of organisations involved.	Naturalistic inquiry will be used to ensure that the dynamic nature of the Strategy is evaluated.
Lack of baseline data in some areas	The Strategy is aimed at improving outcomes in areas where there is no or little baseline data	A Stocktake will establish what data there is and we will work from that point on improving data.
Lack of control group	Strategy is being implemented nationwide. There will be no treatment group and control group to make comparisons with.	Outcomes - looking at how the Strategy contributed to workplace health and safety outcomes. Specify that Strategy is one of the things contributing; Intermediate outcomes - stakeholder interviews will be used to look at government, community and industry leadership and workplace survey tool will be used to look at preventive workplace cultures. In developing survey, the first year will be used as counterfactual/control for workplaces.
Complexity of outcomes.	In some areas, there will be data that can be used to measure outcomes. However, other areas such as occupational diseases may prove a problem.	Research is being carried out to look at measurement at the moment. International experience in this area shows that health and exposure will be difficult to measure.
Different types of workplaces	The Strategy includes initiatives for different workplaces but measurement of change within these will be difficult.	Using a stratified random sampling frame will ensure that we get a good representation of workplaces in the survey.

10. Conclusion

This framework sets out a number of projects for evaluation of the WHSS. It is intended to provide evaluation information at a number of different levels, in the workplace, at a national level and at a project level. The combined projects should provide the Government with detail about the effectiveness of the WHSS.

The final product will be a detailed evaluation framework. The framework will include information about each project and how it fits within the framework. This document will then be placed on the WHSS internet site for easy access.

Contents | << Previous Page

or return to the Workplace Health and Safety Strategy homepage