Gah! Evaluation!!!

It might as well be a diagnosis of the plague!

Most of us associate evaluation with our annual employee performance review. No one likes the way performance evaluation is generally done. Not the manager. Not the employee. It’s probably the most dreaded, feared, heart-wrenching, ulcer-causing activity done in the business world.

So why would we want to put our entire program under that kind of microscope? Could anything be more repulsive?

I’ve really only encountered two types of people when it comes to evaluation: those that avoid it like the plague, and those that are so passionate about it that they will shout about its virtues from the rooftops.

Can you guess which I am?

Okay, so I am one that would shout it from the rooftops. Why, might you ask?

The simplest explanation is, I like to know where I stand, and what people are thinking. I like to know what I am doing really well, and what I need to work on in order to be the best “me” I can be. These concepts transcend my work and personal life.

For those like me, we can get geeked out pretty quickly when we start talking data, outcomes, process, budgets, the benefits of measurements, and what can be measured versus what should be measured. I am certain my pupils just dilated and my heart skipped a beat or two writing that sentence, and I didn’t even go into the difference between an output and an outcome!

For those whom evaluation is the plague, you may have just experienced sweaty palms, a blood pressure spike, hives breaking out across every limb, and may even feel the start of a migraine coming on just from reading that sentence!

So for those of you who think evaluation is like the plaque, know this – it really doesn’t have to be that bad. Breathe deeply and repeat that sentence again if you need to.  It doesn’t have to, and frankly shouldn’t be that bad.

I know, I come from a very special place when I say that, but…really…as business professionals (yes, nonprofits, universities, and governments are businesses too!), and I’d stretch to even say as humans, we need to know if what we are doing is working. Otherwise, why are we even doing it (other than job security)?  Don’t you want to know if your program is as good as you think it is?

 

 

In an environment of “proving your worth” evaluation can be the best prescription: it can show that what you do is worthwhile and you can also show that you identified what was wrong and made efforts to correct course. That’s pretty admirable.

So, next time you hear the word “evaluation” take a deep breath and start the process: 1) plan the evaluation, 2) conduct the evaluation, 3) determine the results of the evaluation, and 4) create and implement a game plan for addressing the results. Yes, this is an over-simplified view of the process, but it’s the framework that can get you to a better place mentally to face the project.

Also remember, that you don’t go though evaluation alone. It’s an all-hands-on-deck type of project. Even better, the best evaluations are those done by neutral third parties (hence you don’t do the bulk of the work!).

So again, take that deep breath. Now, here’s a brief overview of what you can expect when conducting an evaluation project.

What can you expect?

In general, an evaluation project will include the following components.

A literature review:

What are the industry standards for the program?

Research pertinent to your industry and particular program/activity is used to inform the final evaluation design and the outcomes evaluation indicators.

An evaluation design:

What is the focus of the evaluation and how will it be conducted?

The design will be finalized as part of the proposed evaluation project. All outcomes, indicators, and the methodology most appropriate for your organization and target program are reviewed or created. During the evaluation design, interview instruments, process evaluation objectives, outcome indicators with operational definitions, and project charter will be created using appropriate industry frameworks for evaluation (e.g. CDC Framework for Program Evaluation in Public Health and Program Evaluation Standards). The SNF Writing Solutions methodology also includes the Lean Six-Sigma DMAIC model.

Stakeholder interviews:

What do people think about the program?

Interviews with key stakeholders (leadership, employees, management, board members, possible clients, and funders) are conducted to assess program impetus, implementation, and interaction experience. The stakeholder pool and interview schedule is coordinated with the evaluation champion in your organization. These interviews will inform the remainder of the evaluation schedule and serve as a progress gate for any evaluation design modifications before moving forward.

Data Collection:

What data do we have versus what we need?

Information. Excel Sheets. Databases. Calendars. Websites. Marketing literature. Budgets. Tally sheets. Meeting notes. The list goes on.

Depending on the focus of your evaluation, different types of data are gathered to feed the evaluation process. Some will be readily available and some will need to be culled from various sources. With the evaluation design completed, the data needed to compete the evaluation has already been determined and must be pulled together. The SNF Writing Solutions team includes data-gathering and data-entry experts that aid organizations through this process of gathering data and even creating new data sets.

Process Evaluation:

How well is the program executed in relation to the intended model?

The process evaluation assesses the extent to which the target program/department/ service is delivered as planned as well as the facilitators and barriers to implementation. The process evaluation will clarify how, why, and for whom the program works and which components are most/least effective. Through the process evaluation, an understanding of the opportunities for quality improvement and/or corrective feedback is established as well as determining the feasibility for replication and/or expansion of the program operations (remember, funders love programs that are replicable). Consideration of the reach of the program, the intervention dose delivered and the dose received, as well as the implementation and model fidelity within the context of the program environment will be included in the evaluation design.

The process evaluation with SNF Writing Solutions will generally include a Lean Six Sigma review of the program with a process documentation review, analysis of client and organizational service expectations, observation of all organization practices (with regard to HIPAA or other legal requirements as appropriate), documentation of current state, and statistical analysis of current state against service expectations.

Cost Evaluation:

How are costs balanced with benefits?

A cost evaluation looks at the impact of program to the cost of the program/service for the target population or stakeholder (can be organization as a whole). Methodologies for the cost evaluation can vary. The SNF Writing Solutions evaluation includes a review of dollars spent versus income (or funds available in the case of a grant-funded program) and population served and a cost-benefit analysis (CBA) of providing services. As part of the cost evaluation, the return on investment (ROI) is generally calculated for the program. In reviewing the financials and CBA for the program, the feasibility for replication and/or expansion of the program in achieving outcomes can be considered. A high cost program can still be a sound investment for a funder if the results are impressive and long-lasting.

Outcomes Evaluation:

Are we meeting the needs of our customer?

When thinking of evaluating a program, this is the evaluation that most think of: assessing the impact of a program on the societal improvements sought based on existing or determined short-term and long-term activities/services targeted in the evaluation. The literature review and stakeholder interviews will be used for qualitative analysis of the program. SNF Writing Solutions includes text theme determination. Quantitative data analysis of service provision versus expected outcomes will include a review of all outputs and outcome indicators from the program using a correlational analysis. The outcomes evaluation can include the feasibility for replication and/or expansion of the program in meeting societal needs. If needed, a baseline for indicators can be retroactively established from organizational data prior to the implementation of the project/program.

Final Conclusions:

What did we learn?

The final deliverable of a project is largely determined by the client-side champion of the evaluation project. A report will generally include the interpretation of all evaluations with conclusions regarding the program. The report will include statements regarding the merit, worth, and significance of the program/service and interpretations of the findings against the standards identified in the literature review, evaluation design, and stakeholder interviews. Recommendations for specific actions to take into consideration for improvement, replication and/or expansion of the program will also be included in the report.

At the conclusion of the evaluation, SNF Writing Solutions shares the methodology and results with organizational stakeholders, and, at the request of the organization, will assist in presenting to funders, staff, or other groups, and can prepare the findings for dissemination as a potential white paper, journal article, or for conference presentations as opportunities are available.

Implementation:

What will we do with what we learned?

The evaluation process is only as good as the follow-through on the findings. Many things will prove to be working and should be left alone – don’t fix what isn’t broken. That leaves those items that weren’t quite what you were expecting, or had not gone planned. Take a good look at those items and figure out the root cause (this is why SNF Writing Solutions uses the Lean Six Sigma approach – you will already know root causes) and develop potential fixes for the root cause. Create an action plan to implement the change (if a big change, you may want assistance with change management) and monitor the effects of the changes you implement. This ongoing measurement and evaluation can be an extension of your relationship with the evaluator. As you see improvements, acknowledge and celebrate them!

Evaluation is not a one-and-done experience. Those organizations that are successful are always measuring (the right things) and evolving to meet the demands of their customers within the scope of the organizational expertise and mission. This is accomplished with rigorous and continuous evaluation of the organization and individual programs. SNF Writing Solutions generally includes post-evaluation follow-up and implementation support as part of any evaluation project.

Hopefully, with better overall evaluation experiences, those employee performance evaluations are also a little less stressful for you – everyone will know, on a regular basis, where everyone stands!

Here’s to the evaluation plague – turn it into your passion!

With Grantitude,

Stacy N. Fitzsimmons, MBA is the Owner of SNF Writing Solutions, LLC a business services consultancy.