BIG DATA

BIG DATA

1. Key definitions

   • An information asset characterized by high volume, high velocity, and high variety, requiring cost-effective and innovative processing methods to enhance decision-making.
     • Compiled definitions: Massive amounts of data that are difficult to manage with common tools; tools and processes for creating, manipulating, and managing large datasets and their storage.
       • The 3 classic Vs: Volume — quantity of data, now in terabytes/zettabytes. Variety — images, video, structured text, social media. Velocity — algorithmic processing in milliseconds.
         • 4th V — Value: Gives meaning to "decision-making" and gives rise to data science, supported by Business Intelligence.
           • 5th V — Veracity: Added to verify the authenticity of each data point before analysis (variability and veracity, alongside volume, velocity, and variety).

2. Origin of Big Data

   • Early precedents: The use of large data repositories for decision-making is not new—examples include the Library of Alexandria (300 BC) and resource records in the Roman and Ottoman empires.
     • "Big data" was coined by John R. Mashey in the 1990s; it gained popularity and became a buzzword.
       • The 2003 "Boom": Google reveals documents on how it manages its data—the Google File System (GFS/DFS) and MapReduce, a parallel processing model.
         • Technological consolidation 2006: Apache Hadoop is born (open source, improves MapReduce). 2008: Facebook creates Hive on top of Hadoop, and HBase appears, inspired by Google's BigTable.
           • Surge in interest: According to Google Trends, global searches for "Big Data" show a marked surge starting in 2011.
             • Three-stage evolution. Stage 1 (1970–2000): structured databases (RDBMS, OLTP/OLAP). Stage 2 (2000–2010): unstructured web content. Stage 3 (2010–present): sensor and mobile device data (IoT).

3. Motivations for Big Data

   • Information society: People and organizations that learn to group and manage the information they obtain. Significant advantage in commercial, logistical, and administrative decisions..
     • Improving decision-making: It enables public and private organizations to read and analyze the behaviors and affinities of their stakeholder groups in order to design strategies.
       • Logistics benefits, efficient asset control (fleet, inventory), distribution optimization, demand segmentation, and increased customer loyalty.
         • State-led initiative in Colombia: The CAOBA Project connects companies, academia, and the government; MinTIC promotes applied research, training, and scholarships for data scientists.
           • Better public policies enable results-informed budgets and precise insight into public opinion within hours.

4. IImplications of Big Data

   • New scenarios are emerging. Areas previously unimagined will open up: real-world purchasing data and the analysis of human behavior via video cameras.
     • Non-discrimination: The aim is to ensure that information managed on the web does not discriminate based on race, gender, income, or location—verified through algorithms.
       • Multidimensional impact: social/ethical (awareness, participation, equality), economic (innovation, employment, new business models), legal (privacy, intellectual property), and political (surveillance, decision-making, risk of abuse).
         • Challenges and risks: high storage costs, predominantly unstructured data, privacy risks, difficulty with legal compliance, and a lack of expert personnel.