BIG DATA_2

BIG DATA_2

1. Definition

   • Big data  is a tools or traditional data processing applications is the term for a collection of data sets  so large and complex that it becomes difficult to process using on-hand database management.The challenges include capture, curation, storage,[3]  search, sharing, transfer, analysis,[4]  and visualization.

2. Purpose

   • Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead "massively parallel software running on tens, hundreds, or even thousands of servers"

3. Where it is Used

   • Business Intelligence uses descriptive statistics  with data with high information density to measure things, detect trends etc.; Big data uses inductive statistics  and concepts from nonlinear system identification  [24]  to infer laws (regressions, nonlinear relationships, and causal effects) from large data sets [25]  to reveal relationships, dependencies, and to perform predictions of outcomes and behaviors.[24] [26] 

4. Big science

   • The Large Hadron Collider  experiments represent about 150 million sensors delivering data 40 million times per second. There are nearly 600 million collisions per second. After filtering and refraining from recording more than 99.999% of these streams, there are 100 collisions of interest per second

5. Science and research

   • When the Sloan Digital Sky Survey  (SDSS) began collecting astronomical data in 2000, it amassed more in its first few weeks than all data collected in the history of astronomy. Continuing at a rate of about 200 GB per night, SDSS has amassed more than 140 terabytes of information. When the Large Synoptic Survey Telescope , successor to SDSS, comes online in 2016 it is anticipated to acquire that amount of data every five days.[5]  Decoding the human genome  originally took 10 years to process, now it can be achieved in less than a week : the DNA sequencers have divided the sequencing cost by 10,000 in the last ten years, which is 100 times faster than the reduction in cost predicted by Moore's Law .[30]  Computational social science — Tobias Preis  et al. used Google Trends  data to demonstrate that Internet users from countries with a higher per capita gross domestic product (GDP) are more likely to search for information about the future than information about the past. The findings suggest there may be a link between online behaviour and real-world economic indicators.[31] [32] [33]  The authors of the study examined Google queries logs made by Internet users in 45 different countries in 2010 and calculated the ratio of the volume of searches for the coming year (‘2011’) to the volume of searches for the previous year (‘2009’), which they call the ‘future orientation index ’.[34]  They compared the future orientation index to the per capita GDP of each country and found a strong tendency for countries in which Google users enquire more about the future to exhibit a higher GDP. The results hint that there may potentially be a relationship between the economic success of a country and the information-seeking behavior of its citizens captured in big data. The  NASA  Center for Climate Simulation (NCCS) stores 32 petabytes of climate observations and simulations on the Discover supercomputing cluster.[35]  Tobias Preis  and his colleagues Helen Susannah Moat  and H. Eugene Stanley  introduced a method to identify online precursors for stock market moves, using trading strategies based on search volume data provided by Google Trends.[36]  Their analysis of Google  search volume for 98 terms of varying financial relevance, published in Scientific Reports ,[37]  suggests that increases in search volume for financially relevant search terms tend to precede large losses in financial markets

6. Big data execution

   • Researcher Danah Boyd  has raised concerns about the use of big data in science  neglecting principles such as choosing a representative sample  by being too concerned about actually handling the huge amounts of data.[94]  This approach may lead to results bias  in one way or another. Integration across heterogeneous data resources — some that might be considered "big data" and others not — presents formidable logistical as well as analytical challenges, but many researchers argue that such integrations are likely to represent the most promising new frontiers in science

7. Government

   • In 2012, the Obama administration  announced the Big Data Research and Development Initiative, which explored how big data could be used to address important problems faced by the government.[46]  The initiative was composed of 84 different big data programs spread across six departments.[47]  Big data analysis played a large role in Barack Obama 's successful 2012 re-election campaign .[48]  The United States Federal Government  owns six of the ten most powerful supercomputers in the world.[49]  The Utah Data Center  is a data center currently being constructed by the United States  National Security Agency . When finished, the facility will be able to handle a large amount of information collected by the NSA over the Internet. The exact amount of storage space is unknown, but more recent sources claim it will be on the order of a few Exabytes

8. Market

   • "Big Data" has increased the demand of information management specialists in that Software AG , Oracle Corporation , IBM , Microsoft , SAP , EMC , HP  and Dell  have spent more than $15 billion on software firms only specializing in data management and analytics. In 2010, this industry on its own was worth more than $100 billion and was growing at almost 10 percent a year: about twice as fast as the software business as a whole.

9. Architecture

   • In 2004, Google published a paper on a process called MapReduce  that used such an architecture. MapReduce framework provides a parallel processing model and associated implementation to process huge amount of data. With MapReduce, queries are split and distributed across parallel nodes and processed in parallel (the Map step). The results are then gathered and delivered (the Reduce step). The framework was incredibly successful,[62]  so others wanted to replicate the algorithm. Therefore, an implementation of MapReduce framework was adopted by an Apache open source project named Hadoop