0 comments on “Evaluation: Your Plague or Passion?”

Evaluation: Your Plague or Passion?

Gah! Evaluation!!!

It might as well be a diagnosis of the plague!

Most of us associate evaluation with our annual employee performance review. No one likes the way performance evaluation is generally done. Not the manager. Not the employee. It’s probably the most dreaded, feared, heart-wrenching, ulcer-causing activity done in the business world.

So why would we want to put our entire program under that kind of microscope? Could anything be more repulsive?

I’ve really only encountered two types of people when it comes to evaluation: those that avoid it like the plague, and those that are so passionate about it that they will shout about its virtues from the rooftops.

Can you guess which I am?

Okay, so I am one that would shout it from the rooftops. Why, might you ask?

The simplest explanation is, I like to know where I stand, and what people are thinking. I like to know what I am doing really well, and what I need to work on in order to be the best “me” I can be. These concepts transcend my work and personal life.

For those like me, we can get geeked out pretty quickly when we start talking data, outcomes, process, budgets, the benefits of measurements, and what can be measured versus what should be measured. I am certain my pupils just dilated and my heart skipped a beat or two writing that sentence, and I didn’t even go into the difference between an output and an outcome!

For those whom evaluation is the plague, you may have just experienced sweaty palms, a blood pressure spike, hives breaking out across every limb, and may even feel the start of a migraine coming on just from reading that sentence!

So for those of you who think evaluation is like the plaque, know this – it really doesn’t have to be that bad. Breathe deeply and repeat that sentence again if you need to.  It doesn’t have to, and frankly shouldn’t be that bad.

I know, I come from a very special place when I say that, but…really…as business professionals (yes, nonprofits, universities, and governments are businesses too!), and I’d stretch to even say as humans, we need to know if what we are doing is working. Otherwise, why are we even doing it (other than job security)?  Don’t you want to know if your program is as good as you think it is?

 

 

In an environment of “proving your worth” evaluation can be the best prescription: it can show that what you do is worthwhile and you can also show that you identified what was wrong and made efforts to correct course. That’s pretty admirable.

So, next time you hear the word “evaluation” take a deep breath and start the process: 1) plan the evaluation, 2) conduct the evaluation, 3) determine the results of the evaluation, and 4) create and implement a game plan for addressing the results. Yes, this is an over-simplified view of the process, but it’s the framework that can get you to a better place mentally to face the project.

Also remember, that you don’t go though evaluation alone. It’s an all-hands-on-deck type of project. Even better, the best evaluations are those done by neutral third parties (hence you don’t do the bulk of the work!).

So again, take that deep breath. Now, here’s a brief overview of what you can expect when conducting an evaluation project.

What can you expect?

In general, an evaluation project will include the following components.

A literature review:

What are the industry standards for the program?

Research pertinent to your industry and particular program/activity is used to inform the final evaluation design and the outcomes evaluation indicators.

An evaluation design:

What is the focus of the evaluation and how will it be conducted?

The design will be finalized as part of the proposed evaluation project. All outcomes, indicators, and the methodology most appropriate for your organization and target program are reviewed or created. During the evaluation design, interview instruments, process evaluation objectives, outcome indicators with operational definitions, and project charter will be created using appropriate industry frameworks for evaluation (e.g. CDC Framework for Program Evaluation in Public Health and Program Evaluation Standards). The SNF Writing Solutions methodology also includes the Lean Six-Sigma DMAIC model.

Stakeholder interviews:

What do people think about the program?

Interviews with key stakeholders (leadership, employees, management, board members, possible clients, and funders) are conducted to assess program impetus, implementation, and interaction experience. The stakeholder pool and interview schedule is coordinated with the evaluation champion in your organization. These interviews will inform the remainder of the evaluation schedule and serve as a progress gate for any evaluation design modifications before moving forward.

Data Collection:

What data do we have versus what we need?

Information. Excel Sheets. Databases. Calendars. Websites. Marketing literature. Budgets. Tally sheets. Meeting notes. The list goes on.

Depending on the focus of your evaluation, different types of data are gathered to feed the evaluation process. Some will be readily available and some will need to be culled from various sources. With the evaluation design completed, the data needed to compete the evaluation has already been determined and must be pulled together. The SNF Writing Solutions team includes data-gathering and data-entry experts that aid organizations through this process of gathering data and even creating new data sets.

Process Evaluation:

How well is the program executed in relation to the intended model?

The process evaluation assesses the extent to which the target program/department/ service is delivered as planned as well as the facilitators and barriers to implementation. The process evaluation will clarify how, why, and for whom the program works and which components are most/least effective. Through the process evaluation, an understanding of the opportunities for quality improvement and/or corrective feedback is established as well as determining the feasibility for replication and/or expansion of the program operations (remember, funders love programs that are replicable). Consideration of the reach of the program, the intervention dose delivered and the dose received, as well as the implementation and model fidelity within the context of the program environment will be included in the evaluation design.

The process evaluation with SNF Writing Solutions will generally include a Lean Six Sigma review of the program with a process documentation review, analysis of client and organizational service expectations, observation of all organization practices (with regard to HIPAA or other legal requirements as appropriate), documentation of current state, and statistical analysis of current state against service expectations.

Cost Evaluation:

How are costs balanced with benefits?

A cost evaluation looks at the impact of program to the cost of the program/service for the target population or stakeholder (can be organization as a whole). Methodologies for the cost evaluation can vary. The SNF Writing Solutions evaluation includes a review of dollars spent versus income (or funds available in the case of a grant-funded program) and population served and a cost-benefit analysis (CBA) of providing services. As part of the cost evaluation, the return on investment (ROI) is generally calculated for the program. In reviewing the financials and CBA for the program, the feasibility for replication and/or expansion of the program in achieving outcomes can be considered. A high cost program can still be a sound investment for a funder if the results are impressive and long-lasting.

Outcomes Evaluation:

Are we meeting the needs of our customer?

When thinking of evaluating a program, this is the evaluation that most think of: assessing the impact of a program on the societal improvements sought based on existing or determined short-term and long-term activities/services targeted in the evaluation. The literature review and stakeholder interviews will be used for qualitative analysis of the program. SNF Writing Solutions includes text theme determination. Quantitative data analysis of service provision versus expected outcomes will include a review of all outputs and outcome indicators from the program using a correlational analysis. The outcomes evaluation can include the feasibility for replication and/or expansion of the program in meeting societal needs. If needed, a baseline for indicators can be retroactively established from organizational data prior to the implementation of the project/program.

Final Conclusions:

What did we learn?

The final deliverable of a project is largely determined by the client-side champion of the evaluation project. A report will generally include the interpretation of all evaluations with conclusions regarding the program. The report will include statements regarding the merit, worth, and significance of the program/service and interpretations of the findings against the standards identified in the literature review, evaluation design, and stakeholder interviews. Recommendations for specific actions to take into consideration for improvement, replication and/or expansion of the program will also be included in the report.

At the conclusion of the evaluation, SNF Writing Solutions shares the methodology and results with organizational stakeholders, and, at the request of the organization, will assist in presenting to funders, staff, or other groups, and can prepare the findings for dissemination as a potential white paper, journal article, or for conference presentations as opportunities are available.

Implementation:

What will we do with what we learned?

The evaluation process is only as good as the follow-through on the findings. Many things will prove to be working and should be left alone – don’t fix what isn’t broken. That leaves those items that weren’t quite what you were expecting, or had not gone planned. Take a good look at those items and figure out the root cause (this is why SNF Writing Solutions uses the Lean Six Sigma approach – you will already know root causes) and develop potential fixes for the root cause. Create an action plan to implement the change (if a big change, you may want assistance with change management) and monitor the effects of the changes you implement. This ongoing measurement and evaluation can be an extension of your relationship with the evaluator. As you see improvements, acknowledge and celebrate them!

Evaluation is not a one-and-done experience. Those organizations that are successful are always measuring (the right things) and evolving to meet the demands of their customers within the scope of the organizational expertise and mission. This is accomplished with rigorous and continuous evaluation of the organization and individual programs. SNF Writing Solutions generally includes post-evaluation follow-up and implementation support as part of any evaluation project.

Hopefully, with better overall evaluation experiences, those employee performance evaluations are also a little less stressful for you – everyone will know, on a regular basis, where everyone stands!

Here’s to the evaluation plague – turn it into your passion!

With Grantitude,

stacy sig jpg

Stacy N. Fitzsimmons, MBA is the Owner of SNF Writing Solutions, LLC a business services consultancy.

0 comments on “5 Easy Steps to a Better Budget Proposal”

5 Easy Steps to a Better Budget Proposal

It’s that scary beast: the budget! It’s the part of the application that the program implementers and the innovative visionaries often dread. It’s where details are important and you have to remember the dreaded math.

Here’s some advice from the budget trenches to help you along the way.

1) Read the Guidance

The guidance is the most important tool you will have at your disposal for preparing a winning proposal. It contains specific information that will outline how you should address the funder’s expectations of the budget contents and in some cases a specific format may be required. The guidance may require that matching funds are incorporated into the budget or that specific breakdowns for items like travel are provided.

In some cases, you may find clues (or even direct language) as to what they want or don’t want you to spend the money on. Use these as your guide in developing your budget outline.

2) Determine Allowable Costs

Allowable costs are the items and activities (to include Direct and Indirect Cost) that the funder will consider as an acceptable cost in delivering the grant objectives. All cost should be reasonable and necessary. In the federal grant space, the acceptable costs are outlined in the 2 CFR 200 “Supercircular.” There is usually also a section in the NOFA/FOA/RFP that indicates what the particular program will or won’t fund. Read the content behind the links. If you are unsure, check with your legal team or a grant professional that has federal funding expertise.

On the foundation side, each funder may have their own restrictions on what they will or won’t fund. Be sure to research their giving guidelines before creating the application budget.

When in doubt, ask the funder. If you have a specific line item you are questioning, call the funder’s representative and talk to them about that specific item’s eligibility as part of the program budget. On the federal side, there is always a budget person listed for questions. They won’t bite, give them a call!

3) Do the math

Computer programs and templates are great tools but, not perfect. Take time to check the calculations before you submit. Be sure to use current fringe, travel and indirect rates. If you really struggle in this area, you may want to seek outside assistance for the budget from your accounting staff or a grant professional experienced in grant budgeting and managing grant funds post-award.

Nothing is worse as a reviewer than having a budget that doesn’t add up. It’s hard to trust real dollars to someone you aren’t confident can do basic math!

4) Consider Equipment vs Supplies

Equipment is defined as an item of property that has an acquisition cost of $5,000 or more (unless the organization has established lower levels) and an expected service life of more than one year. If the item does not meet both criteria, then it should be listed in Supplies section. Be sure to talk with your accounting division to determine the threshold established for equipment acquisition. If your threshold is less than that of the federal government, you will need to explain the difference in your budget narrative.

5) Does Budget = Narrative?

Cross-check the project description, narrative and budget to ensure all items discussed do not conflict with each other and that the numbers are consistent throughout. Many times, as the budget comes together numbers change. Go back through all documents and adjust numbers so that all match. It is really important that your top-level budget matches the quotes and subcontractor/subawardee budgets (yes, quotes and sub-budgets are very helpful in getting to the “numbers” part!). This is especially important when it comes to number served and cost per person calculations. Readers will pick up on differences and questions what the real number is supposed to be.

The budget does not have to be the scary monster hiding under your desk. Take your time, develop it as you do the narrative, and, when in doubt, seek assistance.

May the numbers always be in your favor!

With Grantitude,

stacy sig jpg

Stacy N. Fitzsimmons, MBA is the Owner of SNF Writing Solutions, LLC a business services consultancy. This post was written with the substantial contribution from Leighe Disbro, SNF Writing Solutions Associate.

0 comments on “New Year’s: Blah Blah Blah?”

New Year’s: Blah Blah Blah?

Happy New Year!

It’s a new year and I’m a process and planning person, so that means I’m supposed to be the one asking:

  • What are your goals for the new year?
  • Have you reviewed your long-range plan?
  • Do you have measurable objectives established for you and your team?

But. I’m. Not.

Why?

They should already be in place from last year!

Goal-setting and planning are often thought of as stand-alone events that happen once a year with great flourish and fanfare, more often than not at an offsite location where the leadership is quarantined off from the masses to establish the direction of the organization for the next 365 days, or perhaps longer.

While I am a proponent of leadership planning retreats, (after all if you aren’t at the office you aren’t distracted by operations), and having a strategic plan, the annual goal-setting exercise, when isolated, does not cultivate the fertile ground from which prosperity can grow; cultivation is an ongoing, year-around process.

Don’t get me wrong, if you are setting goals and reviewing plans and establishing performance metrics, I’m very happy! Let’s just take it a step further and establish the goal-reviewing, plan tweaking, measurement, and course overhauls that will be needed throughout the year.

Depending on the nature of your business, this may need to be done as often as monthly or can be done quarterly. Any less than quarterly and things can get away from you before you know it. Pick a day of the week that you generally can block a couple of hours, and put a reoccurring (monthly? every two months? quarterly?) event on your calendar now. Set your availability as busy. Treat the time as you would a mandatory meeting with your boss/client. When the time comes, here’s your agenda:

  1. Spend part of the time reviewing data supporting your goals and progress toward your benchmarks. Are you under-performing? Right on track? Are you blowing it out of the water?
  2. Determine what actions need to be taken regarding your performance so far. Do you need to determine why you are where you are? Do you need to establish more realistic (more or less challenging) goals based on true capacity?
  3. Next, review the performance data and possible actions with your team. Discuss trends they are experiencing, clear paths and obstacles they are encountering. Allow honesty and be open to the discussion. Just because you have given this some thought already, keep open ears, you may find that the boots on the ground are walking a different path than you perceive.
  4. Review needed actions and formalize any plan/goal/measure changes and add them to your planning documents.
  5. Continue to gather data and repeat on your next scheduled review.

This doesn’t have to take very long – an hour or two max, if you keep up on data gathering and do the review routinely. With a regular goal and performance measurement planning/reviewing, you see your progress and see your needs year-around and are prepared no matter what date the calendar says.

In business, as in life, the best resolution is to not have to make any resolutions – because you already have a plan and are working it! So, I won’t be asking about your resolutions, let’s bypass the rhetoric and blah, blah, blah that we typically spew forth this time of year and instead establish or continue our routine of evaluation and planning.

I will, however, wish you a Happy New Year! May you be prosperous in all your endeavors.

With Grantitude,

stacy sig jpg

 

0 comments on “Creating a Grants Office from Scratch (Part 6): Resources and Launch”

Creating a Grants Office from Scratch (Part 6): Resources and Launch

Here we are at the last post of this grants office series. I wish you great success as you ramp up your grants office with key stakeholders.

Last week we talked about establishing grant goals for the office and the organization as a whole. This week we’ll focus on technological and human resources and the launch of your grants office.

Let’s get to it.

Determine Resources Needed

Once the duties of the office are clearly stated, determine what human and technology resources are needed to achieve the goals of the office.

Human Resources

Staffing is never an easy process, especially in government! Approach the human resources element knowing the personnel regulations and guidelines (and budget) for your organization.

Look at the scope and goals and ask:

  • How many job functions are there?
  • How can they be grouped?
  • How many people does it take?
  • How many people can we get by with?
  • What skills do people need?

Create job descriptions that reflect the functions you have identified and the requisite skills. There are many job descriptions online for grant-related positions. Do a search as part of your research and determine what elements you need and what others are seeking as qualifications. This was a major part of my reorganization experience—determining what others were doing and how I could adapt to my specific situation. Enlist your HR office early and often as you develop the job descriptions through onboarding.

Grant Management Technology

There are all sorts of software programs and online applications that can be used to manage the grant lifecycle. Some offices use Excel sheets, Access databases, or full-blown grant management systems. In order to determine what you need, consider the following:

  • How will you track grant applications?
  • How will you track grant awards?
  • How will you maintain reporting requirements (deadlines and content)?
  • What dollar amount will you be managing?
  • What volume of applications and awards will you be managing?
  • What are the reporting expectations of organizational leaders?

Once you determine the scope of technology needs, do your research to figure out the best resource for you. Some vendors are scale-able; others are not. It’s important to have a tracking system that meets your needs.

The Launch

Way to go! All of the people and processes are in place. All systems are go.

As you have come along this journey, I’m sure you have already been doing some grant work as well (unless you outsourced or delayed). It’s a tiring road, but well worth it when grant proposals are going out and new awards are coming in.

Revisit the strategic plan goals and reasons for starting the office frequently—weekly at the start, then pare it down. Keeping focus on the “why we are here” is definitely needed in the first few months, which are the most rough. Once everything is operating smoothly, track your progress toward goals and determine how well processes are working. Tweaks will be needed as you work through the first few grants. Don’t be afraid to make changes that make the process less cumbersome, have fewer steps, or simply make more sense. While in the planning phase, it’s somewhat easy to see how things should work, but when things get going, it may look and feel different. That’s okay! Just reevaluate and go back through the process—it will be shorter and easier on each cycle.

Originally published on the eCivis Blog August 10, 2014