1	Charity Accountants’ Conference 2005 USING PREDICTIVE TECHNIQUES TO HELP COPE WITH CHANGE Mary O’Callaghan, Z/Yen Limited 19 September 2005
2	Agenda Background The tools and techniques The experiments: Fundraising Grant Making Raw findings Visualisation and interpretation Next steps
3	Background - Z/Yen UK’s leading risk/reward management firm Most experience in: not-for-profit, technology, finance and business-to-business services Clients such as Barnardo’s, NSPCC, The National Trust, The Shaftesbury Society, Cancer Research UK, British Red Cross Society, Action for Blind People, BEN, The British Heart Foundation, Marine Stewardship Council, The Children’s Society Projects in strategy, intelligence, fundraising, governance, risk management, finance, IT Some Highlights – British Computer Society Award 2004/2005 for PropheZy and VizZy, DTI Smart Award 2003, DTI Foresight Challenge Award of £1.9M for The Financial £aboratory, Investment Banking CCC’s, IT for the Not-for-Profit Sector, Clean Business Cuisine
4	Background – Fundraising Research Joint Research between Z/Yen and National Fundraising Charity Can predictive techniques improve the targeting of fundraising campaigns leading to cost savings? Today – sneak preview of initial results
5	Background – Grant Making Research Joint Research - Z/Yen Limited & CASS Business School Centre for Charity Effectiveness Can predictive techniques inform the decisions of grant-making bodies and thus improve their effectiveness? Participants: New Opportunities Fund and City Parochial Foundation Authors of forthcoming papers Ian Harris & Michael Mainelli (Z/Yen) Peter Grant & Jenny Harrow (CASS CCE) Today – work in progress
6	Tools: PropheZy Multi-dimensional correlation (a.k.a Support Vector Machine maths) Helps to spot patterns and identify anomalies in data Classification and prediction PropheZy can be used to make predictive applications using readily available data
7	Tools: Where to PropheZy? Commercial applications include: Television audience and time predictions Managing trade execution rates (with London Stock Exchange) Reducing failed trades for major investment banks Price-your-audit tool and studies Potential charity applications: Identify best fundraising techniques, bid approaches or cross-sales Set targets for donor revenue, profitability, bid success, satisfaction “Fill in the blanks” on donor data Increase success rates among targets, reduce churn Improve assessment of grant applications
8	Tools: IT Architecture
9	PropheZy Model – “Cast”
10	PropheZy Predict - “Forecast”
11	Fundraising Experiment (1) Test PropheZy ability to improve fundraising approach, e.g., Are some people being mailed who are unlikely to ever respond? Are there defining characteristics for those who will respond? Are we failing to mail people who should be targeted? Work with National Fundraising Charity’s database Including ~ 250,000 records of active warm donors in last 3 to 5 years, excluding committed givers
12	Fundraising Experiment (2) Training Set (cast) included: Gender, Postcode, Contact history, Date of Birth, Campaigns Mailed, response to campaigns Test set (forecast): aim to predict responses to September 2004 campaign Iterative approach, refining and developing questions, allowing PropheZy to cope with charity hit rate
13	Initial Findings Series of tests based on responses to 3 to 15 mailings PropheZy can identify response rates with up to 90% accuracy Most accurate for Yes/No response, more difficult with different response levels Best results by excluding those with no contact history
14	A bit more detail Test includes history of one year’s mailings (excluding those with no history) Comparing “Zero responses” with “Responses” Of the predicted responses: 61% were correctly identified – have the characteristics of givers 37% had not responded – potentially have the characteristics of givers 5% had not been mailed – Charity did not believe they had characteristics of givers, but PropheZy does
15	VizZy – Anomaly/Pattern Detection In Fundraising
16	VizZy – Anomaly/Pattern Detection In Fundraising
17	VizZy – Anomaly/Pattern Detection In Fundraising
18	Fundraising Interpretation (1) PropheZy identifies three groups for further investigation: Farmland (61%): targets who give, and may have the characteristics of committed givers Rocky Soil (37%): targets who did not respond, but worth investigating to find if they would ever respond Hunting Ground (5%): targets who were not mailed but have the characteristics of those who should respond
19	Potential Implications Farmland: up to 5,000 additional committed givers Rocky Soil: An additional 2,300 responses could increase donations by £30,000 OR provide savings by not mailing them Hunting Ground: at least 150 additional donors, previously considered “cold”
20	Fundraising Next Steps Tests show groups worth investigation: Farmland: consider approaching actively as committed givers, saving money on direct mail, and increasing lifetime value Rocky Soil: potential to increase income or make significant savings following further research to find why they don’t respond (e.g., returns) Hunting Ground: begin active fundraising relationship Further research underway, e.g., Seasonality Predicting value of donations
21	Where Could Predictive Analysis Be Useful to Fundraising Charities? Reduced costs – printing, postage etc Improved relationships – more effective targeting Potential for growth in committed givers Enhanced data quality “Fill in the gaps”
22	Grant Making Experiment (1) Three post hoc evaluation questions: Did the funded work mostly achieve its objectives? Was it the right decision to have funded? Did you fund again / would you recommend funding again? Look at all readily accessible applications data stored for each fund
23	Grant Making Experiment (2) City Parochial Foundation all grants awarded in 2000 50 “training set” Remainder (c150) “test set” New Opportunities Fund 3 programmes – Community Access to Lifelong Learning, Digitisation & Healthy Living Centres 150 “training set” (50 from each) Remainder (c1500) “test set”
24	Grant Making Experiment (3) Where PropheZy flags up anomalies (i.e. “no answers”), test whether that result concurs with those of evaluators City Parochial insufficient no answers to enable prediction on that volume of data – asked evaluators to rank projects A, B or C instead New Opportunities Fund did generate sufficient no answers to spot 30 potential anomalies on the test set for one fund
25	Grant Making Raw Findings (1) Even with the ABC ranking rather than yes/no answers, PropheZy couldn’t find anomalies in the City Parochial Data Possibly down to sample size and the relatively limited amount of data stored for each grant at City Parochial Hands-on evaluation on application c/w process-based application methods
26	Grant Making Raw Findings (2) Of the 30 potential anomalies PropheZy found at nof, 50% were confirmed to indeed be problematic grants The training population, a random sample from the whole population, had 18% anomalies (26/141) A 50% “hit rate” is statistically significant at 99.9% confidence, on this sample size These results are very encouraging
27	VizZy – Anomaly Detection In Grant Applications
28	VizZy – Anomaly Detection In Grant Applications
29	VizZy – Anomaly Detection In Grant Applications
30	Grant Making Interpretation Where sample size is large enough and procedural methods are used for most applications, predictive methods could flag up potential problem grants in advance of funding As predictive tools of this kind learn from additional training data, the hit rate could improve to well above 50%:50% NOT a substitute for other methods – should not base decisions on this alone
31	Grantmaker Participants’ Feedback Initial scepticism – is grant making an art, or a science? Issues with data – concern about subjective responses Concerns about timescales – what is correct timescale for evaluating a piece of work? Emphasise that techniques cannot replace the human element
32	Where Could Predictive Analysis Be Useful to Grantmakers? Public grantmakers: Encouraged to demonstrate accountability High volume grants with human resources at premium: Process based evaluation of applications Organisational rather than project investment: where investments aim to meet programme aims ‘Social return on investment’: developing venture philanthropy
33	Real-time Charities Coping with floods of rapidly changing information Static MIS not good enough Seeking robust, general-purpose tools suitable for many datasets Seeking quantitative ways of assessing outcomes Moving from analytics to action
34	Next steps Needs further work on visualisation (outputs as well as inputs) Further refinement of data and tests with other organisations Keen to do further work with other charities