The Questionnaire Conundrum: Unraveling the Complexities of Audience Data

When did “audience questionnaires” become such a loaded term in the creative sector?

Having sat on the boards of multiple arts organisations and serving the creative sector through Zealous, I’ve witnessed firsthand how reporting requirements consistently bubble to the surface during conversations, usually alongside passionate emotions.

Capturing data is undeniably vital. It shapes our programming, informs our funders, and, ultimately, determines our impact. Yet, with increasingly stretched resources, it is a time-consuming and expensive endeavour for organisations. It’s also worth exploring whether these methods capture the essence of who walks through our doors or if they merely scratch the surface.

In this article, we’ll delve into one specific method – audience questionnaires. We’ll unpack their purpose, explore their limitations, and explore alternative solutions. We aim to strike that balance between meaningful data capture and realistic effort. So, grab your clipboard (or perhaps hide from it), and let’s get started.

Beyond tick boxes – The Purpose of Audience Questionnaires

Collecting data on the audiences gracing our premises is vital in understanding who we serve and if we are progressing in our mission. How can we measure our impact without appropriate metrics?

These insights allow funders to invest their funds strategically to fulfil their ambitions. For example: empowering the Arts Council to “support England’s diversity fully in those they support and the culture they produce”. Which ACE have brilliantly shared publicly through the Culture and Places Data Explorer (you should take a look, it’s ace! Sorry :D).

The data is core in allowing us all to make better decisions and provide culture to as diverse a set of people as possible. However, data can only reflect our humanity and capture progress with a delicate balance between numbers and personal stories.

This article is NOT about the importance of capturing data. There is no question there, data is vital for us to progress and serve our audiences better (if you’re not collecting the right data and referring back to it, you will fail to progress).

This article is solely about Questionnaires, which are just one of the many tools available to perform that function.

Navigating Questionnaire Methodologies

Let’s be honest: When did you last fill out a questionnaire? Do you cross the road when you see someone with a clipboard trying to catch your attention? And if you have filled one out recently, is it only because you understand how vital they are for the organisation you are supporting because you, yourself, work with them?

All data collection methods have some form of bias – questionnaires have a few key issues that can lead to huge discrepancies when aggregated into the over-arching dataset.

The Numbers Game: Sample Size Matters

Sample size is paramount to reflect your audience correctly in the data collected from questionnaires. Calculating how many surveys to complete for the data to be a good reflection of your audience (sample size determination) depends on the total size of your audience and the margin of error you are willing to put up with.

For example, if you had 2,000 members coming through your doors in a month. To be 95% sure your surveys reflected them, you would need 322 people to respond to your survey. That means around 1 in 6 people would need to fill out a survey for it to be meaningful.

Honestly, you are doing well if you manage to survey 5% of the audience coming through your doors – the reality is probably that this won’t happen.

The Inclusivity Paradox: When Good Intentions Create Blind Spots

ACE cares everyone has access to the arts, and creative organisations care about giving people a fair chance accessing the arts. Those are valuable causes. There is something incredibly rewarding about empowering people from all walks of life.

Being inclusive is incredibly difficult – one person’s needs may actively get in the way of another. You’re not going to get someone who is uncomfortable speaking to someone to fill out a survey in person, but going digital will discount everyone living in digital poverty and older generations struggling with technology.

Audience questionnaires are time-intensive, extremely personal (asking a young person multiple questions about their sexuality in public surrounded by their friends isn’t great, even inappropriate for the younger audience members) and complex to fill out (picking from a list of options read out to you is tiring). Furthermore, they are expensive to run in person (but may not be completed online).

All of these factors mean it is unlikely the following segments will fill them out in person:

Missing these audiences from our answers will automatically skew data towards those happy to fill them out. The likely reason why you are getting more responses from time-rich older individuals who feel a little lonely and would love to chat.

Beyond Demographics: The Unseen Gaps in Our Questioning

As of today, the required questions on the Illuminate questionnaire focus on age, sex, location, ethnicity, and economic status. These are important and strike a delicate balance between how much information we ask for and how to ensure we can measure whether these specific demographics are served adequately. Getting the right questions is hard, and there is a careful balance between reducing questions to the bare minimum without losing the value of the information gathered.

The questions we ask directly shine a light in one direction and create “blind spots” in others.

If we want the arts to serve everyone, we have to contend with the reality that, at present, most people coming through our doors will have liberal values.

What about the others? Those we might not always agree with?

How can we build a fairer, more empathetic society if those with a different take on the world are excluded from the conversation?

The Human Factor: Unconscious Bias in Data Collection

Who do you think you would approach with a questionnaire in person? Would you favour a sweet little old lady smiling away at the work on the wall or a middle-aged man twice your size who looks like he’s had a bad day?

It would be a natural human reaction to select the less intimidating option. However, this human reaction will lead to unconscious bias in your data.

But there is a far bigger problem. What if you’re an organisation that has to serve more people of a specific demographic, and your funding depends on you growing your impact on that demographic – wouldn’t it be tempting to focus your surveys on that demographic?

Charting a New Course: Rethinking Our Approach to Audience Data

Our public funders are doing a great job to the best of their abilities. Just like us, they have requirements for their funding and are facing massive changes in how the world works around them with restricted funds.

I would loathe to be the one adding another problem to the mix without at least trying to think about potential solutions which would work for all of us.

I am mindful that this article is already long and will keep you in anticipation for part 2, where I will offer solutions to the above.

About the author