Most of us have heard of ‘big data’. But there is also the vast wealth of data which is collected but never properly exploited. Techies call it ‘dark data’ because, like mushrooms, it never properly sees the light of day even though it is collected and stored – and local authorities have volumes of it.
To explore this topic, PA Consulting and The MJ recently conducted a survey of council senior managers to find out if they are making the best out of their data.
Dark data is the data held for a single purpose, which is acquired through various computer network operations, but not fully used beyond its original capture. As a result, potential value is left unexploited.
Councils already use data to make key decisions, and the introduction of the General Data Protection Regulations (GDPR) mean there is a better understanding of what data is held by organisations. While in some cases GDPR has been seen as yet another regulation, it has also made councils more data-aware; the next step is how to use this data strategically to improve services.
Just under half of the survey respondents said data was consistently used to provide evidence for strategic decisions and that their key decision-makers were data-literate. Fifty-four per cent of the respondents stated that data was only used occasionally.
There was a more mixed picture on data analysis. Most commonly (38%) analysis was based on a wide variety of internal and external data including data from other public sector partners such as police and the NHS, with just under a quarter supplementing their data with other nationally available local authority data. A quarter used datasets from within specific service areas.
Respondents were less confident about data quality. Just 38% said it was ‘very reliable’, 54% said it was only ‘somewhat reliable’ and 8% even admitted it was ‘not at all reliable’. Reasons ranged from lack of capacity to maintain data sets (more on that to come) and variations caused by base data coming from different sources.
Interestingly, 77% of respondents agreed that investment in data tools and technology would improve decision-making. A similar majority said it would also provide more accurate evidence for strategic decisions while an overwhelming 93% said it would improve transparency. Asked whether such investment would equally improve public perception of their council, respondents were more circumspect, with almost 38% saying it would not and the same proportion having no view.
Respondents were much more positive about their record of sharing data with other areas of local government and the public sector (such as the police or the NHS), with 84% saying they do. However, lack of capacity to combine datasets and exploit their potential is an issue. Just under half (46%) said they disagreed with the statement ‘my service area has sufficient access to internal resources’ while 38% agreed. Nor does there appear to be much likelihood of this changing, since only 15% are planning to expand their resources, either internally or externally, while over half have no plans and the rest are unsure.
As one respondent commented: ‘There is a great deal of misunderstanding about data use’. Another added: ‘Almost all of our current challenges can be informed by data insight, especially if combined with customer insight.’
One principle is important to respond to these findings – start with the decision you would like to evidence, rather than the data itself. Forget whether it is ‘big data’ or small data’. It’s not the size or format of the dataset that matters but how it analysed.
The key is to ask yourself about the hunches you have always wanted to prove or disprove, or the gut-based decisions that need to become evidence-based. In that way appropriate dark data analytics can be applied and the full potential of data to support better outcomes for residents can be uncovered.
