Companies often start with internal data when thinking about analytics and insight. However, companies should not overlook the value of mining third-party and external data.
Information about your operations such as sales transactions or operational performance can help you to make educated predictions about the future. You can also use external data to understand your competitors and how market dynamics, consumer behavior patterns, or weather trends can affect your performance. If you want to take advantage of the data and analytics that can transform your life, I believe it is important to have a good understanding of both.
Machine learning and artificial intelligence (AI), fueled by data are rapidly becoming a transformative force in many markets and industries. Not every company has the resources to create massive amounts of internal, proprietary data from millions of customers. External data is just as valuable and can be easily accessed by anyone.
Many of the models businesses used to forecast demand and change were unable to keep up with the rapid changes in behavior during the Covid-19 pandemic. Their internal data was no longer of any use. Companies often discovered that their external data was crucial in building models to predict how people would respond to changing situations. The internet search traffic data was especially valuable because it allowed for tracking the spread of the virus, predicting the severity of behavior changes, and understanding people’s priorities in a changing world.
External datasets can be made public – many governments, for example data.gov, make large amounts of information accessible through portals like data.gov.uk. They may also be kept privately and made available at no cost (e.g. Google’s basic search or trends data services). Experian and Nielsen provide data on marketing and demographics from many sources. Niche providers also exist that offer niche datasets that are valuable to many industries.
One US glass manufacturer wanted to diversify its revenue streams and discovered that it could predict the locations where window repairs would be most needed by analysing publicly available crime data. It was able to rapidly build a new, profitable business unit that provides emergency repairs by streamlining its supply chain. Finance and credit card companies use credit reference agencies’ data to evaluate the risk of lending to customers. This is a practice that has been widespread across the industry. Real estate companies also use public databases to determine the value of houses that they sell, buy, or lease.
The role of external data in the transformation power of the ” twin” should be highlighted. This is a simulation of a company, product, or process that can be used as a tool to predict the impact of different variables on its performance in real life. Although the twin model is usually built with internal data, it can also be used to simulate the world in which the twin lives. Goodyear, for example, creates simulated versions its tires by using data from its manufacturing process. The company then creates simulated environments using external data about the road surface structure and conditions, weather data, and other data to predict the tire performance.
There is no free lunch and working with external data can be difficult, even though it is provided at no cost. The first is that you may become too dependent on the data provider if you don’t have control over the methods of collecting the data. This could lead to you becoming too dependent on them, or make it impossible for you to change its operating procedures. This could cause problems if you have used resources to create analytics tools around these services and suddenly they aren’t.
There are also technical problems. You need to ensure that your data can be easily correlated with and merged with other datasets from different providers. Combining two or more completely different datasets can often provide the most valuable insights. Data engineering or cleansing is often required to bring it all together.
Keep in mind, however, that you might need multiple data sources. These could include satellite imagery, meteorological data, and anonymized customer data. This means you will have to establish and maintain relationships with many data suppliers. Compliance issues arise because you must ensure that all data purchased has been legally and ethically processed. Compliance and regulation surrounding the use of data are increasing in importance. If your suppliers don’t comply, you could face a costly penalty.
External data can be very rewarding if your company is able to put together strategies and plans to manage it all. This means that your data and analytics strategy no longer revolves around you but about creating awareness of the environment and ecosystems in which your business operates. This can help you streamline your business processes and create new ones.
My second edition of Data Strategy – How To Profit from a World of Big Data Analytics, Artificial Intelligence and Unstructured Data – covers the use of different data types, such as external, structured, and real-time.