Every piece of user research we do seeks to understand something about the people who use a DfE service or are impacted by a policy. We call that group a 'population'. When we conduct user research, we need to find research participants. These are called the 'sample.'
In qualitative, agile user research we don't research with statistically representative samples. However, the sample we recruit must still reflect different user types in the population. Otherwise, we might miss important experiences and needs from some groups. This could mean that DfE services don't meet the needs of all users.
When recruiting research participants, we need to be aware of biases in our approaches, such as:
In a recent children's social care project, we used a series of strategies to mitigate these potential sample biases.
The Early Career Framework
The goal of the Early Career Framework (ECF) is to provide enhanced induction support for people entering the social work profession.
One of the populations we researched with were newly qualified social workers. We were particularly interested in their experience of completing the Assessed and Supported Year in Employment.
Although thousands of newly qualified social workers join the workforce every year, they are busy and recruiting them for research is challenging. This meant we were at risk of gaps in our sample. Therefore, missing important insights or creating non-representative findings.
Two examples of potential sample bias in our recruitment were:
- Only recruiting newly qualified social workers from local authorities with a Social Work Learning Academy.
- Not recruiting individuals from the range of entry routes into social work.
How we avoided sampling bias
Multiple recruitment methods
We used multiple recruitment methods to find people from across our population.
We first used lists of individuals who had already agreed to be contacted for research. However, the fact that these people were interested in taking part, is an example of self-selection bias, so we didn't rely solely on this method.
We contacted related organisations and asked them to distribute information about our project in their own newsletters. This included:
- local authorities
- interest groups
- related charities
- education institutions
To address specific gaps in our sample, we also used a specialist research recruitment agency. This was particularly useful when it came to the geographical spread of participants and their entry into social work.
To avoid pre-screening bias, in our pre-screener surveys we described responsibilities that the individuals we wanted to speak to should have, as opposed to their role titles.
Phased research and tracking demographics
Conducting multiple small research rounds allowed us to identify gaps and target our recruitment. We tracked the demographics of participants across research rounds. That gave us a clear picture of any gaps.
For example, we identified early on that we needed to recruit more participants with access needs. We reached out to the British Association of Social Workers and their Neurodivergent Social Workers Special Interest Group. We asked them to share information about our project with their network.
We used a free mapping tool to visualise the geographical spread of local authorities where participants worked. This allowed us to quickly see which locations we should prioritise in our outreach.
Snowball sampling
We asked participants to share information about our project with peers in the organisations they worked for. This was highly effective in getting more individuals to participate.
Whilst this was a useful method, we didn't rely on it. This is because it risked causing overrepresentation of certain locations, or of types of organisations.
Triangulating our findings
We contextualised and validated our findings using existing evidence and research. We worked closely with related projects, especially Develop your career in child and family social work, another digital service in DfE.
We regularly revisited the longitudinal study of child and family social workers to make sure we kept aware of trends and changes in the social work workforce.
Caveating our findings
Finally, we always acknowledged any research sample limitations when presenting findings to colleagues and stakeholders.
In our reports and research playbacks, we caveated our findings by comparing our sample to the overall demographics of our population. In one example, we were able to highlight that our findings may differ for non-local authority-based organisations. Therefore, in future rounds of research we knew we should focus more on recruitment from those types of organisations.
Conclusion
Sample bias in research recruitment is difficult to completely avoid. It is something that needs to be worked on continuously. However, by acknowledging that it exists, planning our recruitment activity carefully, and caveating our findings to be clear about biases that may exist, we feel we will give our service the best chance of meeting the user needs of all our users.
We would like to take this opportunity to acknowledge and thank all social workers and other participants. We appreciate them taking time and effort to share their experiences with us as part of this project.