Workplace Health and Safety Strategy Evaluation Framework

5. Implementation/Formative Evaluation

Purpose of evaluation: evaluation activity designed to improve the design, development, formation and implementation of a given strategy, policy or programme. It is a disciplined approach to ensuring that a programme is well developed.

This evaluation will document the process of development and implementation of the Strategy. It is an opportunity to critically assess the decisions that are being made and can provide regular formal feedback to those involved in working on the WHSS. The implementation evaluation is also an opportunity for determining how outcomes in the Strategy will be evaluated. It will provide key information on methods for evaluation and monitoring.

There are key reporting requirements already set out within the Strategy. These will be built into monitoring. This includes the first formal review of the Strategy after three years. Regular reports outlining progress on the Action Plan will also be built into the evaluation framework. Key projects include:

Key informant interviews with those involved in development and launch of WHSS
Stakeholder interviews
Stocktake of activities
Formal evaluation after three years that will incorporate and report on all evaluation activities in the Framework

The outcome of this exercise will be:

An outline of how development and implementation of the Strategy occurred
A presentation of stakeholders perceptions of the Strategy’s impact or usefulness at the beginning of the process
An articulation of realistic outcomes for evaluation of the Strategy

Key questions to be considered in the evaluation

Is this a good framework for workplace health and safety at a national level?
Will the Strategy meet the needs of those it is targeting?
Will the Strategy increase ownership of health and safety?
How is the Strategy being used?
How will we determine if Outcomes are achieved?
What can we do better?

The table over the page sets out the key projects that will be undertaken as part of the Implementation Evaluation.

Implementation evaluation projects
Evaluation activity and outcome	Timeline	Method	Ideas
Key informant interviews: Evaluation of development and launch of the WHSS	By 30 June 2006	Key informant interviews Interviews with those involved in the development of the Strategy and its launch. This would only require a representative sample of those involved and includes members of the agency and stakeholder groups, internal DOL officials, those who responded in public consultation and the Minister. Development of Chronology of events including analysis of actions within Chronology.	Would help to provide information for learning environment for strategy development in the future. An opportunity to critically assess the process to date. Would help develop questions to ask as part of stakeholders interviews. Will help to map impact/outcomes of the Strategy. Will help to identify stakeholders for Stakeholder evaluation.
Stakeholder interviews Report presented outlining key findings on how strategy was implemented. This could be posted on the website.	By 30 June 2007	Stakeholder interviews This is an opportunity to talk to the people for whom the Strategy is intended. The evaluation activity allows for information to be gathered on whether the intended audience is using the Strategy and its impact on them. There are a number of ways that evaluation can be undertaken and cost/time constraints will help determine the most appropriate method: paper-based, email, face-to-face or focus group.	This aspect provides information on impact of Strategy in terms of objectives for government leadership and industry/community engagement. This is an ideal opportunity to get feedback from key stakeholders on their opinions. This also shows the Strategy is an iterative instrument. Look at who the Strategy is intended for and whether it has reached them, are they using the strategy and if so, how? Could provide excellent resources for website in terms of advertising the Strategy and what it intends to achieve. If those who are interviewed are happy for quotes to be used, could provide links to their organisations and what they are doing.
Stocktake of activities Baseline information will be used to help determine where gaps are in information.	This will be done by NOHSAC with report produced by 30 June 2006.	Literature review and systems audit Relate back to the ILO promotional framework - review of the national OSH situation. This summarises the existing OSH situation, including national data on occupational accidents and diseases, high-risk industries and occupations and the description of national OSH systems and capacity. National OSH profiles also facilitate a systematic review of the improvements in national OSH systems and programmes.	In terms of monitoring impacts and outcomes, this information may be a key measurement tool to provide a baseline for analysis. Information could be provided in future years on whether there is an increase in information and resources, and activity in the OSH sector. This relates to regular monitoring in the process evaluation. It will be important to link the two.
Formal evaluation of the WHSS after three years	By June 2008	Formal evaluation of project will include the following components: Highlights of regular six-monthly reports; Outcome measurement information; Stakeholder interview undertaken in 2007; and Workplace monitoring survey undertaken beginning of 2008.	The formal evaluation aims to look at where the Strategy is through combining all research and evaluation undertaken to date. It will provide a stocktake of current workplace health and safety based on research/evaluation activity. It is an opportunity to take stock of what has happened over the last three years and give those involved a chance to reflect on what has happened. The first report on outcomes should be completed, impact of the strategy should have been researched in workplaces (workplace monitoring), industry/community (stakeholder interviews) and the government sector (stakeholder interviews and regular reporting, meta evaluation).

6. Process Evaluation

Purpose of process evaluation: activity directed at describing or documenting what actually happened in the course of the programme. It can be crucial for communicating best practice to others who want to replicate elements of the programme.

This element of the evaluation looks at what is happening with the Strategy. It is important in informing stakeholders what is happening in the programme, what is working and what is not working. It also provides an opportunity to regularly report on activities in the Implementation Plan. Key projects include:

Regular reporting on the Action Plan
Meta-evaluation of evaluation activities in the Action Plan
Information from the Stakeholder/key informant interviews will be helpful in documenting what happened

The outcome of this exercise will be:

Information on tasks undertaken in the Strategy and how they occurred
A presentation of stakeholders perceptions of the Strategy’s impact or usefulness
An articulation of realistic outcomes for evaluation of the Strategy

Key questions to be considered in the evaluation

Overview of how the Strategy has been implemented throughout its lifecycle
What initiatives are being undertaken in response to the Strategy?
Are deliverables in the Action Plan happening and how are they working?
What can we do better?
What evaluation is occurring in the Strategy?

The table over the page sets out the key projects that will be undertaken as part of the Process Evaluation.

Process Evaluation Projects
Evaluation activity and outcome	Timeline	Method	Ideas
Regular reporting on activities in Action Plan	Every six months, starting December 2005	Monitoring of activities Development of a reporting method that allows government agencies and NGOs to report on progress within the Action Plan. This will be straightforward and is merely a reporting mechanism to ensure projects are on track. Key information to be provided might include: Project name; agency involved, stage of project; project outcomes.	Provides basic information about how the Strategy is proceeding. Fulfils reporting requirements for government agencies. Provides backbone of information about what is happening in the WHSS.
Monitoring of awareness/reach of Strategy documents	Included in six-monthly reporting	Monitoring Strategy reach Website visits by public Number of agencies/stakeholders involved in Action Plan Activities undertaken to publicise the Strategy e.g. conferences, talks, presentations given by government agencies.	Helps look at who the Strategy is reaching. Provides information on those who are interested in the Strategy - could we have survey for those who have accessed through website.
Are activities in Action Plan achieving their change?	As per the Action Plan	Evaluating particular deliverables This differs to the evaluation activity above, as it looks in more detail at some deliverables within the Action Plan. The WHSS project team is working with agencies involved who will determine which are most appropriate to evaluate. Findings will be included as part of six monthly reporting.	There are a large number of different projects within the Action Plan, how many have evaluation attached to them? Will need to work with agencies involved to determine which deliverables will be evaluated. Include questions to agencies about intended evaluation of projects - this would then be included in six monthly report.

Contents | << Previous Page | Next Page >>

or return to the Workplace Health and Safety Strategy homepage