Big Data

This essay from the IT4nonIT series takes you on a practical and accessible journey into the world of Big Data without drowning in technical details.
In early 2016, my company offered me the opportunity to take on the responsibility of structuring and growing the Big Data architectures and infrastructure department for my company. With the support of an international team of fantastic Sysadmins and Cloudera admins, we managed to build a functional solution that, for years, according to what the Cloudera salespeople told us, was one of the largest Cloudera/Big Data architectures on-premises in Italy and Europe.
Looking back over these past 9 years, I must admit that the greatest challenges were not purely "technical," such as Kerberization of environments, interfacing with a variety of BI and ML tools, or the administration of real-time services with millions of queries per day. The most complex aspect of my job was interfacing with my managers and "business" clients to help them understand what Big Data really meant: explaining that it wasn't just a way to obtain "cheap storage" for their solutions, that a distributed environment also meant the opportunity to optimize the performance of their services, that Big Data wasn't a place to run 1 GB of data for monthly reporting but to manage access to tens and tens of terabytes of information analytically and efficiently. In other words, that Big Data wasn't just a solution for "...everything that crashes Excel."
In this essay, I have attempted to gather the main concepts that have led from managing a 100 MB database to administering more than 30 Data Product for over 12PB of storage and make them accessible to users with limited knowledge of IT complexities.