Evaluation 2017: From Learning to Action

View Printable Version

An” Inconvenient Truth” for Evaluators: Credibility-based Programming for Small Non-Profits can be Optimal

Session Number: 2890
Track: Nonprofit and Foundations
Session Type: Panel
Tags: arts and culture, Child Wellbeing, children's health, Cost Benefits, Cost Effectiveness, evidence, evidence based design, evidence synthesis, health care, monitoring & evaluation, nonprofits, stakeholder involvement approaches
Session Chair: Dale Miller Hill [Senior Consultant - Independent]
Presenter 1: Amy R. Sokal [Executive Director - ArtWorks, The Naomi Cohain Foundation]
Presenter 2: Dale Miller Hill [Senior Consultant - Independent]
Presenter 3: Ria Hawks, non-member guest [Member, Advisory Board, ArtWorks Naomi Cohain Foundation - ArtWorks Naomi Cohain Foundation ]
Session Facilitator: Dale Miller Hill [Senior Consultant - Independent]
Time: Nov 09, 2017 (08:00 AM - 09:00 AM)
Room: Tyler

Abstract 1 Title: High Aspirations but Tough Choices for Small Non-profits: The Case of ArtWorks
Presentation Abstract 1:

Ms. Sokal sought an independent assessment of the evidence available on Artworks’ activities’ impact, and the implications for improving programming and seeking better evidence for the future.   Through Princeton University’s ARC Innovator program, she collaborated with an independent consultant to work toward this goal in 2016/17. ArtWorks, established in 2002, partners with 50 health facilities to help pediatric patients find comfort and outlets for creative expression through guided art and music activities. ArtWorks’ growth and impact to date was achieved by high leveraging of complementary program inputs with health facility staff programming and funding. ArtWorks is small both in core program staff (3) and annual program budget ($400,000), but benefits from  some in-kind contributions. The annual imperative to raise enough money for each year’s programming coupled with a desire to consolidate and standardize approaches, leaves little room for other change management initiatives, despite the high aspirations.    

Abstract 2 Title: An Evaluator's Dilemma: High Quality, Tailor-made Evidence is Always Better, but What if it is not Cost-effective?
Presentation Abstract 2:

Ms. Hill concluded early that ArtWorks’ laudable aim to improve monitoring and evaluation (M&E) was fueled by an unrealistic growth-oriented strategic plan, which needed scaling back.  Still, she began as usual-- program theory, literature search and review of existing data, replacing the strategic plan’s vague indicators with alternatives.  However, from  a recent partner survey, she saw that ArtWorks’ credibility depends not  on rigorous evidence, but on its:  a) appealing mission; b) reputation for customized help and responsiveness; and c) kids, parents, and hospital staff's testimonials of  smiles, encouragement, and decreased need for pain meds. At the same time, clearly, funds and staff time were short and other change management was going on. Thus came the “inconvenient truth”—despite the evaluator’s hard work, using a traditional approach, the benefit/cost ratio of rolling out an improved plan at this time was negative, and continued reliance on “credibility-based programming” was optimal.

Presentation Abstract 3:

Ria Hawks, Child Life Specialist and Member of the Advisory Board of ArtWorks, will elaborate on ArtWorks  "credibility-based programming"  from the perspective of  hands-on professional health care staff.  She will review the proposed measures of impact that came out of Ms. Hill's interviews and comment on the difficulty of measuring such impacts in a  hospital setting. She will then give examples of the qualitative evidence and testimonials that have proven practical and cost-effective for ArtWorks to present as tangible signs of results to  clients, Board members and donors. Finally, she will give examples from  Princeton ARC's  literature search which she and other child life specialists are reviewing to validate quality and generalizability to the ArtWorks context. A Birds of a Feather session at lunch (Session 2911) will provide a further opportunity for the presenters and audience to share information from the literature on these topics. 

Theme: Learning What Works and Why
Audience Level: All Audiences

Session Abstract (150 words): 

Traditionally, program managers and evaluators seek “evidence/results-based programming”– a buzz word meaning rigorous evidence on results must guide programming.  This session will present the  evolution of thinking that led to the “inconvenient truth” agreed by an evaluator and manager of a small non-profit-- that  roll-out of a full M&E plan is  not presently affordable, or practical. In fact, the current “credibility-based programming” – a phrase coined by the evaluator—has satisfied donors and implementing partners for14 years and is a viable interim strategy. The “credibility” is not automatic, but rests on: a) an appealing mission (helping kids heal); b) evidence in the literature; and c) mutual trust between non-profit and partner staff and earnest supervision, allowing responsive programming based on observation and feedback. The lesson of this session is that, even after producing a program theory, literature search, data analysis, and logframe, the benefit/cost analysis was the trump card.

For questions or concerns about your event registration, please contact registration@eval.org or 202-367-1173.

For questions about your account, membership status, or help logging in, please contact info@eval.org.

Cancellation Policy: Refunds less a $50 fee will be granted for requests received in writing prior to 11:59 PM EDT October 16, 2017. Email cancellation requests to registration@eval.org. Fax request to (202) 367-2173. All refunds are processed after the meeting. After October 16, 2017 all sales are final. For Evaluation 2017, international attendees and presenters who encounter complications due to the international travel environment will have up to 30 days after the event to request a refund and submit appropriate documentation. No administrative fee will apply for the international requests. The $50 fee will be waived for registrants who planned to travel into the US and experienced international travel issues.