LSO Discovery is the London Symphony Orchestra’s world-leading music education and community programme, and celebrates its 25-year anniversary this year. Sam Duffy recently completed her PhD at Queen Mary University of London, and is now based in the Cognitive Science Research Group and the Music Cognition Lab. The Creativeworks London Researcher in Residence scheme enabled Sam to examine the practice of impact evaluation in music organisations and bring best evaluation practice into the heart of ongoing planning and fundraising at LSO Discovery.
During 2015, LSO Discovery celebrates a quarter of a century of providing inspiring musical experiences to people of all ages and backgrounds, and support and training for emerging young instrumentalists, composers and conductors. The Researcher in Residence Scheme provided a perfectly timed opportunity to reflect on previous projects and programmes, and determine best practice impact evaluation and future evaluation needs. Being able to evidence and communicate impact allows an organisation to support quality, highlight areas for improvement, inform future strategy and planning, and advocate for its work and for the arts more broadly (Arts Impact Assessment Project, 2013). The residency had three main aims: to understand current impact evaluation practice at LSO Discovery, to review examples of impact evaluation elsewhere in the cultural sector and to propose a framework for future evaluation.
As a qualitative researcher, with experience of working with music organisations, I had background knowledge and experience of relevant research methodologies and techniques, but had not worked specifically in the area of impact evaluation before. This presented a challenge, as I had to get up to speed very quickly. However this also represented an opportunity, the organisation felt that because I didn’t come from a conventional arts background, I would bring an interesting perspective to the project.
Semi-structured interviews were carried out with each of the LSO Discovery Project Managers and key people from other LSO departments such as Finance, Development, Marketing, and the LSO Managing Director, Kathryn McDowell. A wide variety of evaluation materials were gathered and reviewed including questionnaires, reports produced with external evaluators, online survey results and marketing materials. In parallel, impact evaluation was researched in the literature, and through publicly available information on how other organisations approach this area.
The projects delivered by Discovery are amazingly diverse in scale, activity and scope and involve a wide range of different groups of people. For example, free Friday Lunchtime Concerts encourage anyone, with any level of musical experience, to discover more about music and the instruments of the orchestra, whilst LSO On Track introduces young instrumentalists to live performance through a partnership with the Music Services of ten East London boroughs. 946 workshops and 147 Discovery concerts were delivered by Discovery in 2014/15[1] alone. Indeed one of the challenges of the Residency, as a researcher, was to understand such a variety of wide-ranging projects and programmes in such a short time. This also represented one of the challenges for impact evaluation at Discovery: how to provide a framework that has relevance for each activity, and can be easily applied by each of the Project Managers, when they implement such different activities?
Interviews revealed a passionate and professional Project Manager team, dedicated to delivering projects through Discovery that make a real difference to the community and growth of young music talent. Delivering a high volume of ongoing, and often overlapping, activities requires planning, organisation, and hard work. Making time for evaluation whilst maintaining a high standard and volume of programmes is a challenge for many organisations.
“Good impact measurement is not easy. Many charities and funders do not know what or how to measure, and many struggle to find the resources to measure their impact.” (Rickey, Lumley, & Ógáin, 2011)
“…organisations had many different initiatives taking place concurrently. With the focus being on delivery there was insufficient remaining capacity for impact assessment.”[2]
Therefore it was important that any recommendations represented sustainable change; they had to be practical and easy to integrate into a busy organisation. For example, integrating evaluation with project management from the very beginning of an activity makes it more likely to be completed and aligned with the aims of the work. In this way evaluation becomes part of the day-to-day and is less likely to be postponed when the demands of delivering projects are at their most challenging (Rickey et al., 2011).
As well as a wide range of diverse activities to consider, there are also different types of impact, and they require different types of evaluation. The potential timescale for impact includes the period before the event, during the event itself, immediately afterwards and even some time in the future after the event; extended impact can encompass all impacts that result from a specific cultural experience over the remainder of the participant’s lifetime (Carnwath & Brown, 2014, p. 16).
“The social impacts of the arts are those effects which are sustained beyond actual arts experiences, and have resonance with the life activities and processes of individuals.” (Reeves, 2002)
Arguably, it is these long-term impacts that are most important but they are also challenging to capture, requiring a conscious effort to plan for this reflection after the event, when participants may be difficult to make contact with and the delivery team has already moved on to the next thing. Producing an annual evaluation plan can help to ensure a proportion of long-term evaluation is carried out, and that project planning includes maintaining contact with participants.
Findings and recommendations from the interviews were shared during a participatory workshop, using anonymised statements from interviews to provoke further reflection and new understanding. Despite the diversity in programmes delivered by each of the Project Managers, they found value in discussing each other’s evaluation methods, experiences and challenges, and a central repository was created containing the internal evaluation materials gathered during the course of the residency. In addition, another repository was created containing useful external resources from the research component of the project, such as toolkits for impact evaluation, online databases and examples of materials from other organisations. Both repositories are owned by the Project Management team and will continue to grow and remain relevant through the addition of new materials over time. The workshop was also an opportunity to share how much other LSO departments, such as Marketing and Development, valued access to materials providing evidence of impact and documenting the outcomes of programmes. There are now plans for these to be made more accessible across the organisation.
An important outcome of the residency has been the raised profile and increased understanding of evaluation across the Project Manager team. The interviews did not just provide data for analysis, they also enabled the Project Managers to make time to reflect on their current evaluation practice. Creating space for reflection encouraged creative ideas to be shared and documented, and many of the recommendations came directly from the team. As a research student coming to the end of four years working on a PhD thesis, being able to apply the research skills I have developed to a new challenge has given me the confidence to develop as an independent researcher. It was also immensely satisfying to be able to work with such a passionate organisation and help them to celebrate their work.
Bibliography
Arts Impact Assessment Project (2013) Arts Inform managed by Catherine Rose’s Office. (2013). Retrieved February 18, 2015, from http://www.arts-impact-measurement.co.uk/
Carnwath, J. D., & Brown, A. S. (2014). Understanding the value and impacts of cultural experiences. London.
Reeves, M. (2002). Measuring the economic and social impact of the arts : a review. London.
Rickey, B., Lumley, T., & Ógáin, E. N. (2011). A Journey to Greater Impact: Six charities that learned to measure better. New Philanthropy Capital.
[1] http://lso.co.uk/lso-discovery/about-lso-discovery
[2] Reported by organisations participating in the Arts Impact Assessment Project (Arts Inform managed by Catherine Rose’s Office, 2013) http://www.arts-impact-measurement.co.uk/#/barriers/4580717978
