The Snyder evaluation process ¹ 

 

This is a resource file which supports the regular public program "areol" (action research and evaluation on line) offered twice a year beginning in mid-February and mid-July.  For details email Bob Dick  bdick@scu.edu.au  or  bd@uq.net.au

...  in which a fairly detailed description is given of the Snyder evaluation process, a process which combines process evaluation for understanding, outcome evaluation for improvement, and short-cycle evaluation for ongoing improvement

 

Contents

• Abstract
   
• Introduction
   
• An overview of the process
   
• The process in more detail
   
• Orientation
   
• Process evaluation
   
• Outcome evaluation
   
• Short-cycle evaluation
   
• Notes
   
  

 

Abstract

To understand, to improve, to continue improving...  these are the goals addressed by the evaluation process described in this document.  The vehicles for doing this are a systems model of a project or organisation, and a three-stage process of process evaluation, outcome evaluation, and short-cycle evaluation.  Each stage of the process builds upon the understanding developed in the prior stage.

The method is primarily qualitative in its approach, with some quantitative measures when they are appropriate.  This balance can be changed.  A participative approach is also typically used.  The evaluator then becomes a facilitator who guides those involved through the process.  It can be done by an independent evaluator if this is preferred.

The step-by-step description is accompanied by a rationale which explains the purpose of each of the steps. 

 

In this document I describe a process by which a project (or work team or program or organisation or other social system) can become a "self-improving system".  The process is built upon a "systems" evaluation model devised by Wes Snyder.  The main elements of the model can be diagrammed as follows...

resources -> activities -> immediate effects -> targets -> ideals

Resources are consumed by activities which produce immediate effects in the pursuit of targets which are intended to contribute to eventual ideals.  The ideals provide the criteria by which the other elements are evaluated prior to improvement.

The operation of a project is analysed in these terms -- by identifying resources, activities, immediate effects, targets and ideals, and the way they interact.  The information provided by this analysis is then used...

• to understand how the project operates, and so to improve its operation;
   
• to understand how well the project operates, and so to communicate this to funding bodies, directors, and others;
   
• to build in processes for ongoing monitoring, and so to continue to improve project functioning.

In short, the overall process allows the functioning of the project to be understood and improved and demonstrated, and for the improvement to be ongoing.  This is for the benefit of the project team, its clientele, and others with an interest in it.

In the description which follows, I focus on the end goal of the process as ongoing improvement.  I also assume that it is done participatively.  This is done by involving at least the members of the project team, and preferably other people with a stake in the project.  In effect, the process allows you to set up a qualitative equivalent of total quality management.

What follows is a project improvement method based on an evaluation model.  The focus is on highly&endash;participative project-team planning to improve the functioning of the project, both initially and ongoing

This is only one of several ways in which such a project improvement system might be implemented.  However, it is planned to address the issues which most often threaten the value of such a system.  It is also applicable to a wide variety of types of project, or other work, or other social settings.  It can also be as easily applied in non-work settings, and can also be carried out by an independent evaluator if this is required. 

 

An overview of the process

There are three main phases to a Snyder evaluation.  Each provides a different form of evaluation, and each of the later phases builds on the previous phases.

Process evaluation helps project team members to understand the process.  The team members come to understand the links between the elements of the model.  They learn how resources and activities contribute to targets and ideals while producing desirable and undesirable immediate effects.

Outcome evaluation enables outcomes to be assessed.  From the process evaluation, team members understand the processes they use.  Building on this understanding they are able to identify realistic and valid indicators of their achievements.  The indicators can be used to assess the overall achievement of the project team, or to enable short-cycle evaluation to be done.  The outcome evaluation also provides a check on the process evaluation.

Short-cycle evaluation uses the indicators from the outcome evaluation to develop effective feedback.  In other words, the short-cycle evaluation component enables the project to become a self-improving system in which regular and accurate feedback allows continuous improvement.

The three phases:

         1                2                  3
      process    - >>    outcome    - >>    short-cycle 
    evaluation        evaluation        evaluation
 
    analysing       assessing the    developing a self- 
   the process       achievements     improving system

The description below may appear complicated; in practice it is simple.  Each of the three phases has a simple purpose and theme.  Each builds on the previous phase or phases.

(There are also variations to the process.  In the interests of some brevity, I won't describe them here except very briefly as options.

A.  Process evaluation: Analysing the process

The aim of the process evaluation is for participants to understand how their activities contribute to their goals.  This understanding is achieved by defining the elements (ideals, targets, etc.), and examining how they are linked together.  That is all.  The process evaluation also leads to changes in the way the project is conducted.

1    Develop ideals

2    Define targets

3    Compare ideals and targets

4    Define activities and immediate effects

5    Compare targets and immediate effects

6    Define resources

7    Compare activities and resources

8    Plan new or changed activities

B.  Outcome evaluation: evaluating the process

Outcome evaluation uses the understanding arising from the process evaluation to identify performance indicators.  This is done by identifying those indicators (mostly at the level of resources and immediate effects) which best indicate achievement of the targets and ideals.  The indicators can also be used to demonstrate the effectiveness of the project, for instance to funding bodies.

  9    Note assessable targets

10    Note assessable immediate effects

11    Note assessable activities

12    Note assessable resources

13    Develop monitoring activities

14    Evaluate achievement

C.  Short cycle evaluation: Develop a self-improving project

The short-cycle component uses the indicators to set up effective feedback, so that the project (or whatever it is) can become a self-improving system.

15    Identify evaluation criteria

16    Identify evaluation information

17    Identify sources of information

18    Create information systems

19    Review process and outcome evaluations

20    Create review mechanisms

 

The process in more detail

The remainder of this paper describes in more detail each step of the process shown briefly above.  Each step includes an explanation of its purpose, and something about why it is included.

Where appropriate, the description also draws your attention to any other issues to be considered, or to points at which a more detailed procedure may be required.

The process is described step by step.  The comments follow each step, enclosed in double angle brackets and indented. 

 

Orientation

The orientation consists of preparatory activities.  It serves the purpose of preparing for the evaluation to follow, so that it is a constructive activity.

Set up a working party, which may consist of the project team, or may also include other "stakeholders" from inside or outside the organisation.

<< It is often desirable to form a working party consisting of the team members and representatives of the other people who depend on the team, or on whom the team depends.  Alternatively, it may sometimes be enough if the team is careful to take the views of these other people into account.  >>

If the project team is small enough, full participation is recommended, preferably with the addition of some external people who have an important stake in the team's activities or achievements.

Relationship-building

Even in an intact work team it is not safe to assume that there are good and open relationships between the different people.  Any activity is appropriate if it uses self-disclosure to help people relate to each other as people, and not merely in their usual work relationships.

Goal setting

As part of this, the process to be used can be summarised, and agreement reached on its use and its intended purpose.

The goals are not those of the team in their normal work, but the goals of the evaluation process.  The goals of the team are determined later.

<< People are more accepting of processes which they have previously agreed to >>

In particular, be clear about the uses other people will make of it.

<< Denied information about the motives of other people, participants are quick to fear the worst.  The truth is almost always safer >>

Climate setting

The purpose of climate setting is to reach agreement on the style of interaction to be used during the process.  This may be done, for example, by agreeing on a set of groundrules which people undertake to observe. 

 

Process evaluation:
analysing the team process

This is the process evaluation phase of the process.  In it, people work backwards from ideals to resources.  At each step, they define two adjacent elements and compare them.

Options:

If an intact team is unable to agree on the project goals, instead work outwards from activities to ideals, then refine the ideals, then work back to activities again.

In a beginning team which has no previous process to analyse, begin as described below by defining the ideals.  Derive the targets and activities from the ideals.  Then identify the resources and the immediate effects (intended and unintended) from the activities.

You will recall that the process evaluation component is an analysis of the links between the elements of the systems model.  It works by comparing adjacent elements of the model, looking for mismatches.  It pays particular attention to "orphans" -- items which are not represented in adjacent elements. 

This analysis enables the team members (and other participants) to understand the process by which activities and resources are transformed into targets and ideals.

 

1    Develop ideals

This uses a process known as search, ² where people develop a description of an ideal future.  It has a number of sub-steps.

<< As a process, search offers a number of advantages...

<< By asking people to project a future ideal, it generates more agreement than other goal-setting procedures -- people are often much more agreed about ends than about means

<< Search also allows the values that people hold as important to be expressed indirectly.  If you ask them directly, you are more likely to get their "espoused values": those they think they hold or should hold.

<< Search usually also further enhances the relationship-building which took place previously >>

a.    Participants develop tentative ideals

<< This develops a first approximation to project ideals, to be refined in the following steps >>

b.    If only the project team is participating, other stakeholders are defined

<< The team may otherwise neglect the needs of the wider organisation (if they are part of one) or its clientele >>

c.    The needs of other stakeholders are defined

<< The purpose of this step is to encourage the participants to take the needs of everyone with a legitimate stake in the project into account.  The more limited the range of participants, the greater the importance of this step >>

At the very least, if the project is within an organisation, the goals of the organisation as a whole are taken into account.  So are the goals of the level above the team.  If appropriate, so are the team's interdependencies with other teams at the same level, or with other stakeholders outside the project team

d.    The ideals are refined to take account of the legitimate needs of the stakeholders

<< If stakeholders are defined first, the creativity of the team may be reduced.  This design (tentative ideals which are then refined) provides creativity without ignoring the environmental realities >>

e.    The elements of the ideals are arranged in order of priority by using some system of multiple voting ³

<< The important ideals provide the criteria against which project performance can be evaluated.  In this way the team is able to check that its activities and resource use are directed towards the most important outcomes >>

2    Define targets

Note: it is important that the targets are defined at first without taking the ideals into account, as far as possible.  If there are already defined targets, these can be used.

List the targets which the project team actually pursues at present (from documentation, if it exists).  Identify their present priorities, if known.

<< Although the ideals provide the reason for the project's existence, they cannot be evaluated.  Targets are part of what allows progress towards the ideals to be monitored >>

3     Compare ideals and targets

This and similar steps constitute a long-cycle process evaluation.

a.    Consider each target in turn, in order of priority.  Identify what ideals, if any, the target helps to achieve.

<< This can be regarded as an application of the 80/20 rule: are the most time- and resource-consuming targets the most important targets as assessed by the ideals they address?  >>

b.    Focus now on the ideals which do not seem to be addressed sufficiently by the targets.  This indicates either that the ideals are faulty, or that there are missing targets: amend them accordingly.

<< Provided the targets and ideals were independently defined, a close correspondence between them is a sign that the targets are appropriate >>

c.    Focus now on the targets which do not seem to address the ideals.  This indicates either that the target is not needed, or that some parts of the ideals are missing: amend them accordingly

<< The reasoning is similar to that for the previous sub-step >>

4    Define activities and immediate effects

As before, it is important that activities and immediate effects are defined directly, not by reference to the ideals or the targets.

For some purposes, activities and immediate effects can be separately defined, immediate effects first.  ⁴  In general, however, a project evaluation will be easier if both are defined at the one time (particularly if there are no team "clients" present to identify the immediate effects).

a.    List the project activities.  This is most easily done by asking individuals to do it, and then pooling their lists.

Begin with activities.

<< Team members are often blind to many of the effects their activities produce, though "serving the project's clients" (inside or outside the organisation) is almost certainly an important part of the ideals.  Activities, on the other hand, are usually easy to identify >>

b.    Use multiple voting to arrange the activities in order of their consumption of time and other resources

<< Enabling a further application of the 80/20 rule.  You wish to know if the most resource-consuming activities are those which contribute most to the important ideals >>

c.    For each activity (beginning with the most resource-consuming) list the important immediate effects

<< As mentioned above, most project teams find it hard to identify immediate effects directly.  They can, however, identify the activities.  Having done so, they can then identify the immediate effects of each activity with relative ease >>

Make a special effort to identify all important immediate effects, both desired and undesired, both intended and unintended.

<< The immediate effects often provide many of the most useful and important performance indicators, developed in the later phases >>

5    Compare immediate effects and targets

Resource-consuming activities have immediate effects.  In an effective team, these immediate effects are part of the necessary steps on the way to achievement of the ideals.

This step resumes the long-cycle process evaluation.

a.    Consider each of the activities in turn, beginning with the most resource-consuming of them

<< Continues the application of the 80/20 rule.  You want to know the value of the most resource-consuming activities >>

b.    Identify what targets, if any, each immediate effect helps to achieve

<< The 80/20 rule: are the most time- and resource-consuming targets the most important targets?  This is assessed by the parts of the ideals which they address >>

c.    Focus now on the targets (and associated ideals) which do not seem to be addressed sufficiently by the immediate effects.  This indicates either that the target is faulty, or that there are missing activities: amend them accordingly.

<< Is there a close correspondence between the immediate effects and the targets?  This indicates that the associated resource- consuming activities do contribute towards the target, and hence towards the ideals >>

d.    Focus now on the activities whose immediate effects do not seem to address the targets and ideals.  This indicates either that the activity is not needed, that some target is missing, or that there are some unintended effects: amend activities or targets accordingly.

<< The reasoning is similar to that for the previous sub-step >>

e.    Focus on the unintended effects.  If any of these are harmful, manage them in some way.

<< It may otherwise happen that the intended effects of the important activities are undermined by the unintended effects >>

6    Define resources

Resources consist of anything which is a cost to the team or the organisation.  They include money, materials, services, information, and the like -- in fact they are the inward interdependencies from other teams or from the environment of the organisation

Options:

Derive the resources from the activities.  This is usually the preferred option, as otherwise the non-dollar resources are usually difficult to define.  The sub-steps for this option are described below as steps a and b.

or    Define the resources independently, without consideration of the activities which consume them.  If this option is followed, the sub-steps are similar to those for the other parts of the process evaluation.

a.    Consider the activities in order of importance.  For each of the important activities, define the resources which it consumes.

<< The resources collectively define the costs (in dollar and other forms) of what the team does >>

List resources (such as materials which have an obvious cost), and also resources (such as skills and human energy and the like) where the costs are less tangible.

<< Intangible resources tend to be overlooked unless special attention is given to them.  (This is a particular difficulty when the resources are defined directly: the alternative option for this step) >>

b.    Order the resources in terms of their costs to the organisation and team (include non-dollar costs too)

<< Anything which reduce costs without reducing outcomes is an improvement.  Non-dollar costs are important too (and tend to produce long-term dollar costs or benefits) >>

7    Compare resources and activities

Resources, as already mentioned, are costs.  Activities which contribute to the ideals produce benefits.  In an effective team, the most costly resources are consumed by those activities which do most to achieve the most important ideals.

Options:

When the resources are derived from the activities, check that the most expensive resources are consumed by the most important activities.  This corresponds to the first option immediately above, and is described in detail below.

or    When the activities are defined independently, compare activities and resources as for the other comparisons in this phase.  That is, identify which resources are consumed by which activities; then examine those resources which do not seem to be required, and those activities which are over-resourced or under-resourced in terms of their importance.

This step resumes the long-cycle process evaluation.  Only the first option is described.

a.    Consider each resource in turn, in order of cost.  Re-order the resources in terms of cost.

<< The most costly resources warrant the most attention >>

b.    Focus now on the activities which are resource expensive but do not contribute strongly to the ideals.  Any reduction in the resources they consume, or the costs of those resources, is probably worth achieving.

<< This is to identify the resources which could be better used by being directed towards more important activities >>

c.    Focus now on the expensive resources (expensive in dollars or other costs).  Identify any where resource substitution might lower the costs.

<< The ideal, in this and sub-step "b", is that the most costly resources are consumed by activities which contribute strongly to the ideals.  If this is not true, a change in resource allocation is probably worthwhile >>

8    Plan new or changed team activities

In this step, the desirable changes identified in previous steps are gathered and checked.  An implementation plan for achieving them is developed.

a.    Gather together and list the changes identified in steps 3b, 3c, 5c, 5d, 5e, 7b, and 7c.  Initially, list them in this order (beginning with those from step 3b)

<< To plan improvements, you first identify them and then work out how they can be implemented.  This sub-step prepares for the implementation >>

b.    Select the order of importance of the identified changes -- which will do most to improve the achievement of the team ideals with the same or reduced costs (including psychological costs)? 

<< You may have identified more potential improvements than are worth pursuing.  This ensures that you give priority to those changes which are most worth implementing >>

c.    Develop implementation plans which specify who will do what by when.  You may find both event track, ⁵ job redesign ⁶ and role negotiation ⁷ useful here.  ⁸

This is a substantial step, for which only a cursory description is given here

<< Unless plans specify who will do what by when, they may not be implemented.  Event track does this, and at the same time builds in flexibility and monitoring.  Job redesign and role negotiation ensure that the changes are planned and implemented for maximum satisfaction within the work team >>

Note: Before beginning implementation, it is important to negotiate any changes with anyone else who may be affected

<< What appears an improvement from the perspective of the team may not be, when the views of other organisational members or clients are taken into account >>

d.    Rate the ideals for how well you think they have been achieved.  This can most easily be done by asking team members (and others, if present) to provide individual estimates and then collate them.

<< This may encourage team members to revise the priority of the changes previously decided.  It also provides a base-line against which the later outcome evaluation can be compared.  This acts as a further check on the adequacy of the process evaluation >>

The foregoing steps conclude the phase of process analysis.  The links revealed in the analysis are now used to identify which resources, activities and outcomes can be evaluated.  This evaluation follows... 

 

Outcome evaluation:
evaluating the team process

This phase allows the project team (and other participants) to make use of the process understanding developed in the first phase.  This is done by identifying performance indicators which can monitor progress towards the ideals and targets on a moment-by-moment and day-by-day basis.  The indicators can be used in any of three ways: demonstrating project effectiveness to others; redesigning the project as a self-improving system (in phase 3); and doing a self-evaluation of the project as a check on the process analysis in phase 1.

During the steps in this major stage, the important ideals are "tracked back" though the analysis to the point where they are evaluable, either qualitatively or quantitatively.  Items which are evaluable only qualitatively are further tracked back until they are quantitatively evaluable, if possible.  The analysis begins with targets, as the ideals cannot be directly evaluated.

As far as possible, indicators are identified among resources and immediate effects; this constrains the project team members less than specifying activities as indicators.  The aim is to finish with a package of indicators which adequately sample the ideals, and adequately sample resource use, intended immediate effects, and unintended immediate effects.

Option:

Instead of separating out phases 1 and 2, as is done here, it is possible to merge them.  So, for example, the ideals and targets can be defined and compared (part of phase 1), and evaluable targets can then be chosen (part of phase 2); and so on, working back from ideals towards resources.  When the project is small and time is limited, this alternative may be preferable.

9    Note assessable targets

a.    Identify those ideals which are most important

<< These form the basis of the ongoing evaluation.  If they are poorly chosen, the whole evaluation is called into question >>

b.    Mark those targets which most contribute towards the chosen ideals.  Decide if in the future they are...

• not evaluable
   
• evaluable if base measures are established
   
• evaluable qualitatively without base measures
   
• evaluable quantitatively without base measures

  << The intention is to carry out evaluation as rigorously as possible, but also as close to the ideals as possible so that the gap between ideals and measure is small.  ⁹ If you use performance indicators which are at a large remove from the ideals, people may find ways of achieving the indicator without necessarily moving towards the ideals.  The ideal is an indicator which is very specific and easy to use, and which relates clearly and directly to an important part of the ideals >>

c.    Develop plans for setting up the present base measures which will allow later evaluation of the targets

<< Without present base measures you may have no way of interpreting later measures of some of the targets >>

Further steps in this sub-section track the unevaluable and qualitatively-evaluable items back to the earlier items which relate to them.  A common pattern is that something is quantitatively evaluable as a resource or activity, where it is at some distance from the ideals, even where it is only qualitatively evaluable as an immediate effect or target.

In short, find qualitative and quantitative indicators among earlier elements for the important ideals (which are unevaluable)

10    Note assessable immediate effects

a.    Identify those immediate effects which contribute to important but unevaluable targets

<< This continues the process of tracking back through the model >>

b.    Identify also those immediate effects which are most unintended and undesirable

<< Unintended immediate effects are often overlooked unless you direct attention to them.  Yet often the overall achievement of a project depends as much on avoiding unwanted outcomes as achieving wanted outcomes >>

c.    If the immediate effects from "a" and "b" are evaluable, decide how they will be evaluated (preferably by the person who can make most direct use of the information).

<< Finding evaluable immediate effects which can act as an indicator (a "proxy") for unevaluable targets >>

A combination of wanted and unwanted immediate effects is most often appropriate.

<< To encourage maximising the wanted outcomes while controlling the unwanted outcomes >>

d.    Identify those immediate effects which contribute to important targets which are qualitatively evaluable only

<< Tracking back >>

e If they are quantitatively evaluable, decide how they will be evaluated (preferably by the person who can make most direct use of the information).

<< Again, finding measurable immediate effects which can act as a proxy measure for targets which are only qualitatively evaluable >>

11    Note assessable activities

Activities are often evaluable even when outcomes are not.  In such instances, multiple activities are a safer proxy than single activities.

Note.  As mentioned previously, the use of activities as indicators may unnecessarily constrain project team members.  Provided they achieve the outcomes within the resource constraints, it is usually better to let them decide for themselves how to do this.  (However, sometimes indicators at the activity level are required to sample the ideals adequately)

a.    Identify those activities which contribute to important but unevaluable targets

<< Tracking back >>

b.    If they are evaluable, decide how they will be evaluated (preferably by the person who can make most direct use of the information)

<< Proxy indicators for unevaluable targets >>

c.    Identify those activities which contribute to important targets and immediate effects which are only qualitatively evaluable

<< Tracking back >>

d.    If they are quantitatively evaluable, decide how they will be evaluated (preferably by the person who can make most direct use of the information)

<< Again, finding proxy indicators >>

12    Note assessable resources

Resources are usually evaluable, even when it is only qualitative (for instance through feedback from those who have to provide them).  Multiple resources are a safer proxy than single resources, thought costly resources may be worth evaluating whether they contribute substantially to the ideals or not.

a.    Identify those resources which contribute to important but unevaluable targets, immediate effects and activities

<< Tracking back >>

b.    If they are evaluable, decide how they will be evaluated (preferably by the person who can make most direct use of the information).

<< Proxy indicators for unevaluable targets >>

c.    Identify those activities which contribute to important targets, immediate effects or activities which are only qualitatively evaluable

<< Tracking back >>

d.    If they are quantitatively evaluable, decide how they will be evaluated (preferably by the person who can make most direct use of the information).

<< Again, finding proxy indicators >>

e.    Note those resources which are costly and whose use was reduced in the earlier analysis; decide how they will be evaluated (preferably by the person making the most direct use of them).

<< The material and psychological costs of resource use are often overlooked unless they are specifically evaluated.  This is especially true of resources which are a cost to someone else, not to the user >>

13    Implement outcome evaluation

In this step, the evaluation methods identified in previous steps are gathered and checked.  An implementation plan for them is developed in the following phase (the short-cycle evaluation).

a.    Gather together and list the changes identified in steps 9c, 10b, 10d, 11b, 11d, 12b, 12d and 12e.  Initially, list them in this order (beginning with those from step 9c).

<< To implement the planned improvements, you first identify them and then work out how that can be done >>

b.    Select the order of importance of the identified changes -- which will do most to allow the team to evaluate its performance?

<< This sub-step ensures that you give priority to the evaluation which is most worthwhile >>

c.    Check that the chosen indicators meet the following criteria...

<< A final check that the indicators as far as possible are not biased >>

• they sample the important ideals adequately.

  << The whole purpose of the exercise >>

• there are multiple indicators for each of the important ideals and targets.

  << Single indicators are too easily changed by events which have little to do with the ideal that is being measured >>

• indicators adequately sample resource use, intended immediate effects, and unintended immediate effects.

  << As mentioned earlier, indicators based on activities can be unnecessarily constraining.  Intended immediate effects are positive indicators; resource use and unintended immediate effects are negative indicators.  Unless there are both positive and negative indicators, people may overlook benefits or costs >>

• preferably a mix of qualitative and quantitative indicators are used for each of the important ideals and targets.

  << Qualitative indicators tend to be close to the ideals, and thus more relevant.  However, they are harder to use.  Quantitative indicators by themselves are too easily fudged, but add ease of use to the greater relevance of qualitative indicators >>

And, if the short-cycle evaluation is to be done...

• the indicators can be collected regularly, frequently, easily, and at low cost.

  << You don't want to expend more money and energy on the indicators than you save through your monitoring >>

• the indicators are preferably collected by the person who uses them.

  << The monitoring is more likely to persist, and is less threatening to the project team member most affected >>

14    Use the indicators to evaluate achievement

In this step, the indicators are used to complete the outcome evaluation.  The substeps in this instance are arranged as three options.  They correspond to the three purposes of the outcome evaluation previously identified: demonstrating project effectiveness to others; redesigning the project as a self-improving system (in phase 3); and doing a self-evaluation of the project as a check on the process analysis in phase 1.

a.    Demonstrating project effectiveness to others ¹⁰

For each of the important ideals in turn, collate the relevant indicators.  Then...

• relate them to the relevant targets;
   
• interpret them: what do they indicate about the relevance of the targets and their achievement;
   
• support your interpretations by reference to the indicators;
   
• prepare the report.

The end goal of this part of the activity is a report which demonstrates the extent to which the project:

• has appropriate targets,
   
• is achieving the targets, and
   
• is implementing systems to ensure that achievement is likely to increase further

b.    Redesigning the project as a self-improving system

This forms phase 3 of the process, and is detailed there.

<< Uses the indicators from this phase to set up a monitoring system >>

c.    Check the process analysis from phase 1

First complete step (a) above.  Use it to arrange the ideals in the order in they are being successfully achieved.

<< Step (a) provides an assessment of achievement of the ideals >>

Then compare this result to that from step 8(d) in phase 1.

<< The comparison of two partly-independent rankings may allow some new insights >>

If there is a serious mismatch, this may indicate that there are missing pieces from the process evaluation, or that the indicators are not adequate.

If so, return to the appropriate steps and repeat the evaluation from there.

The foregoing steps conclude the long-cycle outcomes evaluation.  You don't yet implement this; instead, you feed the indicators you have identified into the system for short-cycle evaluation... 

 

Short-cycle evaluation:
develop a self-improving project

The purpose of this phase is to use the indicators, or some of them, as feedback on an ongoing basis.  To this end, ways are designed of providing feedback regularly to those project team members who can use it to improve project effectiveness.

You can discontinue the process at the previous step, having completed a process and outcome evaluation.  Provided it is done with the project team members, already they understand their work better, and have improved performance.  The outcome evaluation by now has provided you with the information you need to gauge the effectiveness of the project (to the extent that you can even do this).  Even if it was not participative, you have assembled enough information to allow a once-only evaluation, and thus a once-only improvement.

The real gains, however, come from continuing improvement.  If you also conduct the next phase, you build ongoing evaluation into the project's very operation.  It is the following steps which offer the greatest long-term improvement and satisfaction.

15    Identify evaluation criteria

Decide which of the items for evaluation (from step 13b above) allow you to improve team and individual performance most appropriately.  You don't want the evaluation itself demanding more resources than it is worth.  In doing this, focus on the most sensitive and most readily available indicators of performance.

<< As at step 9b you are trying to select indicators which give the best balance between evaluability and a valid indication of performance.  At the time, in this instance, you are also seeking a balance between the costs of evaluation (through ready availability), and the pay-offs it offers (through sensitivity, or closeness to important ideals) >>

Check that the chosen indicators are an adequate sample of the important ideals.

16    Identify evaluation information

Identify the information which allows each indicator to be monitored.

<< Information is the means by which the people affected stay informed about the indicator >>

17    Identify sources of information

Decide from whom or from where the information for evaluation is available.

<< That is, decide where you can get the information >>

18    Create information systems

Set up a mechanism to generate and forward the required information to the person requiring it.

<< Unless there is a permanent mechanism in place, the information may not continue to be made available >>

As far as possible, check that the mechanism is simple, and provides information directly to the person who can use it to continue to improve performance.

<< The people who will profit from the information are most likely to put effort into producing it.  They are most likely to use it when they access it directly, not via someone else (who may use it selectively or punitively) >>

19    Review the process and outcome evaluations

In developing performance indicators you may have uncovered information which leads you to change your mind about the earlier analysis.  At this step, you check is this is so.  If it is, you make the necessary adjustments to the results of the process and outcome evaluations.

Do the links between the elements (from the process evaluation) still appear to be well understood in the light of your development of a short-cycle system?  If not, adjust the results of the process evaluation.

<< In having to choose indicators which can be evaluated, people often become aware of other indicators, or of reasons to suspect the operation of the indicators.  This step allows this to be corrected >>

Do the indicators formerly identified, and their use in the outcome evaluation, still seem appropriate?  If not, adjust the results of the outcome evaluation.

<< As above >>

20    Create review mechanisms

This is to set up a means of checking that the indicators and systems remain appropriate, or are adjusted as required by changes in the situation.

Agree regular review times at which the team will...

 

• check that the information is still an appropriate indicator and, unless the information is generated by the receiver, will...
   
  • allow the receiver(s) of the information to notify the sender(s) of its adequacy
     
  • allow the sender(s) of the information to notify the receiver(s) of the cost.

  << Without review mechanisms the value of the evaluation system will slowly decline for such reasons as...

  <<  o    the indicator has become a less appropriate proxy, either because the ideals have changed, or methods have done so;

  <<  o    the sender of the information does not put the same effort into providing it;

  <<  o    the user of the information expects more than it is worth the sender to provide >>

It is useful if the review is coordinated with any other review emerging (for example) from other monitoring systems such as those at step 8c.  It is sometimes useful if the reviews coincide with reviews of the targets

 

Notes

1. The systems model around which this process was developed, and much of the process evaluation phase, were developed by Wes Snyder.  I have also profited from discussions with a range of people using the Snyder, including Tim Dalmau, Bob Williams, Adelle Bish, and many of the people enrolled in my postgraduate subject in evaluation.  [ back ]
    
2. For a more detailed description of search in workbook format, see Bob Dick (1989), Search, Chapel Hill: Interchange.  Search is a development of the Centre for Continuing Education at Australian National University; see Merrilyn Emery (1981), Searching: for new directions, in new ways, for new times, Canberra: Centre for Continuing Education.  [ back ]
    
3. Each person is asked to choose several items.  There is then a much greater likelihood that some items will have the approval of almost everyone.  You can achieve the same effect by giving everyone a number of votes to spread around the items.  To increase agreement further, ask people not to vote for items which they themselves contributed.  [ back ]
    
4. This is most likely to be true for non-participative evaluation by an independent evaluator; it is sometimes true for participative evaluation when it is large-scale evaluation, or where there are stakeholders present who can identify the immediate effects.  Team members are often surprisingly unaware of the immediate effects of their actions.  [ back ]
    
5. Event track is an action planning process described in "Helping groups to be effective" (Interchange, Brisbane, 1991).  Its steps may be summarised as follows.  An initial phase develops a rough action plan: (1) identify goals; (2) identify possible actions; (3) choose the most important of these actions; (4) arrange these actions in sequence.  A second phase refines this rough action plan by carrying out these steps for each action in turn: (5) identify other actions required to complete the step; (6) check the assumptions built into the plan, including assumptions about coordination and resources; (7) identify potential problems, adding extra steps to prevent them, and deal with them if they still occur.  [ back ]
    
6. Job redesign is a process by which a work team analyses its work performance and its satisfaction, and devises alternative work methods which improve either satisfaction or performance without reducing the other.  [ back ]
    
7. In role negotiation, team members first identify what they would like to have from one another, both materially and psychologically.  They then negotiate ways of increasing the satisfaction or performance of each other without at the same time decreasing the satisfaction or performance of other people inside or outside the team.  [ back ]
    
8. Event track is most suitable for single but complex changes which require substantial planning.  Job redesign will often be found appropriate for addressing a number of changes at the one time, especially in "blue-collar" jobs where the work is mostly physical and technical.  Role negotiation will often achieve the same outcome in other types of job.  [ back ]
    
9. In statistical terms, you are striving for indicators which have both validity (measure what they are supposed to) and reliability (are not so vague or fuzzy that they are inherently inaccurate).  [ back ]
    
10. My own view is that outcome evaluation is seldom possible.  In effect, it seeks to answer the question: Is this program worthwhile?  or Did this project succeed?  The indicators can be used for participative monitoring, but this doesn't make them effective for determining the overall success of a project except in a fairly superficial way.  In any event, in most instances the success or otherwise of a project isn't entirely within the control of the participants.  Outcome evaluations, therefore, are often political actions in which project teams attempt to persuade funding agencies that the project deserves continuing funding.  [ back ]

    _____

 

Copyright (c) Bob Dick 1990-2006.  This document may be copied if it is not included in documents sold at a profit, and this and the following notice are included.

This document can be cited as follows:

Dick, B.  (2006) The Snyder evaluation process [On line].  Available at
http://www.uqconnect.net/action_research/arp/snyder.html

 

Maintained by Bob Dick; this version 3.05w last revised 20060516