Flashcards
Big Data- Pig and Hive Week 8
Cards in this deck (23)
- 1Problems with mapreduce?it does not directly support complex N-step data flows (e.g. filter, threshold), which often
arise in practice.
Map-Reduce also lacks explicit support for combined processing of multiple data sets
(e.g., joins and other data matching operations), a crucial aspect of knowledge discovery.
Lastly,
frequently-needed data manipulation primitives like filtering, aggregation and top-k thresholding must be
coded manually, which is hard to reuse - 2What is Pig?- language for executing parallel data flow
- combination of sql and mapreduce - 3Benefits of Pig?- easy to use as can convert complex multiple data transformations in sequence of easy to uderstand data flows which are easy to understand and manage and maintain
-creates optimization opportunities as the the encoding of task permits system to optimize the pig execution so user can worry about semantics rather than efficiency.
-user can create special purpose user functions providing EXTENSIBILITY
- accepts any type of data - 4Ways to interact with pig are?interactive shell - use grunt mode in shell
embedded mode - use java program to invoke pig using method invocation
- batch mode - using scripts to run many pig statements - 5What are different data models in pig ?atom, tuple, bag, map
- 6Types of bags?inner and outer
An example of a outer bag
(1,2,3)
(4,2,1)
(8,3,4)
(4,3,3)
An example of an inner bag
(1,{(1,2,3)})
(4,{(4,2,1),(4,3,3)})
(8,{(8,3,4)})
where the first field is type int, and the second field is an inner bag. An inner bag is enclosed in curly
brackets { }. - 7Does field by position in Pig starts with $0 or $1?$0
- 8Expressions in Pig are..
- 9Explain Load statement in PigLoads data onto pig onto sequece of tuples ie bag. syntax :
x = Load 'data' [Using funciton] [AS schema]
default deserializer function is Pig storage and it assumes data to be a collection of tuples. without schema fields will be unnamed and will have BYTEARRAY type.
eg. A = LOAD 'student.txt' USING PigStorage() AS (name:chararray, age:int,
gpa:float); - 10FOREACH command expl,FOREACH is a basic command used to apply some processing to every tuple of a data set (generating
data transformations).
Note that FOREACH…GENERATE block is used with a relation (outer bag) while Nested
FOREACH…GENERATE block is used with a inner bag.
eg
X = FOREACH A GENERATE name,$2; - 11FLATTEN command in PIgchanges the structure of tuples and bags, which un-nests tuples as well as bags.
It operates on bags by extracting the fields of the tuples in the
bag, and making the fields of the tuple being output by GENERATE, thus removing one level of nesting - 12FILTER commandX = FILTER A BY f3==3;
DUMP X;
The result filters out tuples in realtion A whose third field equals to 3: - 13GROUP commdcreates group as first field with same data type as group key and the other field is the bag with content having the group field value
- 14COGROUP and JOINit combines tuples based on particular field and is similar to join except its results are in forms of group, bag1from1strelation,bag2from2ndrelation
JOIN IS just COGROUP but FLATTENED
- 15STORE commandstores variable onto HDFS using default or specified serializer
STORE x into 'sss' [USING asasd()] - 16ILLUSTRATE and EXPLAINILLUSTRATE displays a step-by-step execution of a sequence of statements, and EXPLAIN Displays
execution plans. - 17Why the need for hive?From -- for querying for business analysis by analysts using hadoop, difficulty using hadoop due to user's unfamiliarity with map reduce low level language
-- writing custom code in it hard to maintain and resuse
- need to improve the query capabilities
of Hadoop with an expressive query - 18What is Hive?-open source DW querying solution
-- built on hadoop
-- facilitates querying and managing large datasets in cost effective manner
- consists of SQL like querying language called HIVEQL
-- compiled by HIVE
-- converted into MR jobs and run on hadoop - 19What is hive data model consisting of?Tables
,
Partitions
,
buckets - 20Whats use of partition?partition is used to increase querying speed because partitioning by certain column creates partition under subdirectory of that table directory in which only partitions are created for every different value of that specified column. this enables faster querying which uses condition based on that column as it scans only a small subset of the whole table.
eg.
CREATE TABLE test_part(c1 string, c2 int) PARTITIONED BY (ds string);
A partition exists for every distinct value of ds specified by the user.
INSERT OVERWRITE TABLE test_part PARTITION(ds='2009-01-01') SELECT * FROM
another_table;
The partition is populated with data from table another_table (SELECT * FROM another_table)
where the
value of ds is 2009-01-01.
3 HADOOP ECOSYSTEM – PIG AND HIVE
52
ALTER TABLE test_part ADD PARTITION(ds='2009-02-02'); - 21What are buckets?Similar to partition in that its used to cluster data.
USED FOR SAMPLING - ie querying a small sample of data for testing and debugging
Unlike partitions, field values are stored
in buckets based on a hash function
At the time the table is created,
the user can specify the number of buckets needed and the column on which to bucket the data.
bucket number is determined by the expression hash_function(bucketing_column) mod
num_buckets.
eg.
CREATE TABLE user_info (user_id BIGINT, firstname STRING, lastname STRING)
PARTITIONED BY(ds STRING) CLUSTERED BY(user_id) INTO 25 BUCKETS;
eg.
SELECT COUNT(*) FROM user_info TABLESAMPLE (bucket 2 out of 25 on user_id); - 22how to enable bucket? is bucket enabled by default?No, not enabled by default.
set hive.enforce.bucketing = true - 23—
Information
- Description
- Flashcards on Big Data- Pig and Hive Week 8, created by Varun m on 17/06/2017.
- Created by
- Varun m
- Views
- 8
- Rating
More on Hadoop Architecture and Ecosystem