Analyzing and mining the retail market as a complex system

Supermarket transaction records, an early form of "big data" emerged in the 1980's with the wide diffusion of barcodes for product identification at the cashier, produced in the 1990's the birth of data mining, i.e., the search for local patterns at the micro level, involving the segmentation, separation and profiling of diverse groups of consumers to the purpose of understanding and predicting their behaviour within the retail market. The first part of this talk provides a succinct overview of some paradigmatic projects along this line. Next, we introduce a complex system perspective into retail market analysis, where each customer can be seen as an independent agent. As a consequence of this choice, the global behaviour of the retail market naturally emerges, enabling a novel description of its properties, complementary to the local pattern approach. Such task demands for a data-driven empirical framework. We leverage a unique transaction database, recording the micro-purchases of a million customers observed for several years in the stores of a national supermarket chain. We show the emergence of the fundamental pattern of this complex system, connecting the products’ volumes of sales with the customers’ volumes of purchases. This pattern has a number of applications. By enabling us to evaluate the sophistication of needs that a customer has and a product satisfies, this pattern has been applied to the task of uncovering the hierarchy of needs of the customers, providing a hint about what is the next product a customer could be interested in buying and predicting in which shop she is likely to go to buy it. We conclude by observing how large scale retail transaction records have a potential in describing socio-economic phenomena at a microscopic scale, which has been only partly explored to date.