Flashcards
Data mining
Flashcards do conjunto (52)
- 1why is data mining important?► enterprises are under strong competition, so they need to make use of their collected and stored data
► we have big data but poor knowledge
► decision making using intuitive is not working anymore, we need to make decisions with knowledge
► we are ready for data mining: there are available data, data warehouse building tools, hardware system, Data mining tools - 2functionality of data mining - associationcorrelation and causality:
e.g.1: association between frequently purchased items: (beer & diapers)
e.g.2: associate between features (age, income and purchase) - 3functionality of data mining - classificationpredefine a set of classes using a training set based on a set of features to assign a new object to one of the classes after examining the features
e.g.classify customers based on the response of sales campaign as one of good respondent or bad respondent - 4functionality of data mining - predictionhistorical data (training examples) is used to build a model, which tells us future behavior or value, given an input.
e.g. predicting of life time value (LTV) of customers - 5functionality of data mining - estimationestimate a value for some unknown continuous variable such as income and probability
- 6functionality of data mining - clustergroup data to form new classes e.g homogeneous sub-population of customers
maximize intra-class similarity & minizing inter-class similarity - 7functionality of data mining - outlier analysisoutlier is a data object that does not comply with the general behavior of the data. This is useful in fraud detection, rare events analysis
- 8applications of data mining - biomedical and DNA►similarity search and comparison among DNA sequences
►identification of co-occuring gene sequence
►linking genes to different stages of diseases development - 9application of data mining - financial data► loan payment prediction
► customer credit policy analysis
► detection of money laundering and other financial crimes
► investment analysis - 10application of data mining - retail industry► analysis of sales, customers, products and region
► analysis of effectiveness of sales campaign
► customer retention (customer loyalty)
►purchase recommendation
►cross reference of items - 11application of data mining - telecommmunication industry►fraudulent or unusual pattern analysis
►find usage patterns for a set of communication services by customer group, by month, etc. - 12application of data mining - marketing►consumer behavior after promotion
►reduce cost of mailing by targeting a set of consumer likely to buy a new cell phone product
►market basket analysis, cross-selling, up-selling
►market segmentation - 13application of data mining - health►side effects of drug
►treatment effectiveness - 14application of data mining - deviation or anomaly detection►detect fraudulent credit card transaction
►detect fraudulent insurance claims
►detect intrusion of computer network - 15application of data mining - web mining►web content mining
►web structure mining
►web usage mining - 16application of data mining - text mining►identifying facts, opinions, requests, trends, trouble spots or positive voice, negative voice, intent-to-buy, intent-to-leave, etc.
►from unstructured data such as email, bog posts, online customer survey, news articles, survey responses, web-based feedback forum, call center notes, tweets, etc. - 17DM Techniques - Association rule Basic Concepts►what products were often purchased together? => we want to find association rules in the form of X->Y which means that customers who buy all items in X also buy all items in Y
►support probability: P(X U Y) the percentage of customers who bought items in X and Y
►confidence probability: P(Y|X) the percentage of customers who bought items in Y among the customers who bought items in X
►confidence(A->C) = P(A|C) = support(A->C)/ support(A) = P(A U C)/ P(A) - 18DM Techniques - Association rule frequent item set►item sets is (X U Y), a frequent item set is an item set which satisfies the minimum support. we need to find the item sets before finding association rules X->Y
►an association rule x->y is general, which means that items in X and items in Y should appear frequently in the transaction data
►given a transaction database, the goal is to find the complete set of frequent item sets - 19DM Techniques - Association rule finding frequent item sets►it would be hard to examine all possible combinations of frequent item sets
►we can use the priori property: all subsets of a frequent item set are also frequent item sets, e.g. if {beer, diaper, nuts} is frequent, so is {beer, diaper} - 20DM Techniques - Association rule finding frequent item sets generate candidates►generate candidates using two steps: self-joining Lk and pruning
►example: Let L3={abc, abd, acd, ace, bcd}
self-joining: L3*L3, e.g. abcd from abc and abd, acde from acd and ace
pruning: acde is removed because ade is not in L3,
in the end, C={abcd}
►know how to find C1, L1, C2, L2, C3, L3 from a transaction database - 21DM Techniques - Association rule finding frequent item sets lift option►not all association rules are useful, sometimes we need to provide the min support and min confidence
►an association rule is only useful if the lift value is greater than 1: lift(A->B)= P(A U B)/ P(A)P(B)= confidence(A->B)/P(B) - 22DM Techniques - Association rule multilevel association rules►in case of sparse data, it is difficult to find strong AR, use the taxonomy of items to find multiple-level AR
►examples:X,Y->Z
age(20-29), income(>50k)->buy(smartTV)
P1(X,Y)^P2(X,W)->buys(X,"educational s/w") where P1 and P2 are predicates, X is customer, Y and W are specific values - 23DM Techniques - Association rule negative association rulesA->Not B
birds->fly
birds^penguin -> Not fly - 24DM Techniques - Clustering Definition►cluster analysis is grouping a set of data objects into clusters
►a cluster is a collection of similar data objects
►clustering is unsupervised classification (no predefined classes, no target variable)
►typical applications are:
a stand-alone tool to get insight into data distribution, as a preprocessing step for other mining algorithm - 25DM Techniques - Clustering what is good clustering?►a good clustering method will produce high quality clusters with high intra-cluster similarity and low inter-cluster similarity
►smallest DBIndex
DB(U)=1/c Sum {max(intra-cluster distances)/(inter-cluster distance)}
►quality of a clustering result depends on the similarity measure used by the method
►This can be an initial analytical tool to understand data
►identified clusters can be used for other analyses such as prediction (e.g using logistics regression) or classification (e.g. each case is classified into the identified clusters such as using discriminant analysis, decision trees) - 26DM Techniques - Clustering cluster analysis vs factor analysis►clustering=segment the data table horizontally (=grouping cases)
►factoring=segment the data table vertically (=grouping variables to reduce the number of variables to be analyzed) e.g. using principal component analysis (PCA) - 27DM Techniques - Clustering examples of clustering►customer segmentation: groups thousands of customers into major types: bargain hunters, man on a mission, impulse shopper, weary parents, dual income no kids
►location segmentation: set locations of stores based on demographics: low-end budget grocery stores, small boutique grocery stores, or large full-service supermarkets - 28DM Techniques - Clustering clustering method - K means clustering definition►the analyst picks K number of clusters,
►the clusters are formed based on the "Euclidean distance" which is a measure of distance from each record to the cluster center, and from each cluster center to each other
►the center are based on the means = average/ mean value of all data points in the cluster - 29DM Techniques - Clustering clustering method - K means clustering k mean algorithm1. we select K seeds from the data set,
2. allocate each record to one of the K seeds, which is most similar to the record.
3. compute the centroids of the K groups of records,
4. repeat step 2 and 3 until the centroids do not change - 30DM Techniques - Clustering clustering method - K means clustering k mean clustering vs k-medoids clustering►k-means clustering is weak to the presence of outliers.
►k-medoids take central object as the centroid of each group. this method is robust because medoid is less influenced by outliers than k-means. this method works well when datasets are small. When the dataset is large, it takes several samples and get sets of clusters for each sample to select the best set out of them - 31DM Techniques - Clustering clustering method - K means clustering Limitations of K-means clustering► need to standardize data to get rid of the difference in scale
► have to select the number of clusters (K)
►all variables are assumed to be equal initially
► The more variables, the greater the computational complexity - this is the curse of dimensionality. The method won't run in a reasonable time if you have too many variables - 32DM Techniques - Clustering clustering method - K means clustering interpret & naming of clusters►give each cluster a unique name to differentiate them.
►decide how to use the results (e.g. using clusters of customers, we can do decision tree analysis and find the profile of the profitable customers) - 33DM Techniques - Clustering clustering method - Hierarchical clustering definition►use distance matrix as clustering criteria. this method does not require the number of clusters k as an input but need a termination condition
►unlike k means clustering, for this method, the number of clusters unspecified a priori
►clusters created in levels/ there are sets of clusters at each level.
►this is a useful method to find nested clusters
►agglomerative (bottom-up approach): initially each item in its own cluster, iteratively cluster are merged together
► and divisive (top-down approach): initially all items are in one cluster, large clusters are successively divided - 34DM Techniques - Clustering clustering method - Hierarchical clustering agglomerative method algorithm►build an n-by-n similarity matrix
►find the smallest value in the matrix and merge the two rows into one and adjust the matrix to a n-1byn-1 matrix
►repeat step 2 until we have only one cluster - 35DM Techniques - Clustering clustering method - Hierarchical clustering three ways of computing distance between clusters►single linkage: distance between closest members
►complete linkage: distance between the most distant members
►comparison of centroids: distance between the centroids of clusters - 36DM Techniques - Clustering clustering method - Hierarchical clustering dendrogram►a tree data structure which illustrates hierarchical clustering techniques
►each level shows clusters for that level
- leaves are individual clusters
- root is one cluster
►a cluster at level i is the union of its children at level i+1
►this is a useful method of finding nested clusters - 37DM Techniques - classification definition►given a dataset which contains many independent variables and one dependent variable
►the goal is to build a classification model which classifies a new instance
►steps: 1. construct a model with a training dataset, 2. test the model with a test dataset, 3. apply the model to a set of new instances - 38DM Techniques - classification how to build decision tree?►select an attribute to split on: often pick one that can produce children nodes whose average purity is best
►purity of a node can be computed based on the expected amount of information that would be needed to classify a given instance in the node, with the formula = -sum(pi)log(pi) where pi is the probability of class Ci in the node
►we choose the attribute with the smallest -sum(pi)log(pi) - 39DM Techniques - classification examples of decision trees►classify individual cases into one of predefined groups (classes) based on given information of individual cases
►classification of data into two predefined groups: reject class (success=N): customers with trouble paying the loan, and accept class (success=Y): customers have no trouble paying the loan
►know how to interpret decision tree - 40DM Techniques - classification how to build decision treestep 1: tree splitting - building the tree using training data
step 2: pruning - prune back using testing data, to prevent overfitting - 41DM Techniques - classification principles of decision tree►data is pre-classified, pre-defined, the values of target variable are defined
►splits are done at one variable at the time
►at each step, we determine the best split (ordering of splitting variables) which increase the purity within individual new group the most (reduce the uncertainty the most)
►when a node contains only one class, no further splitting is necessary - 42DM Techniques - classification measure of impurity►Gini, entropy, and classification errors - the higher the value, the greater the impurity of the node
- 43DM Techniques - classification when to stop splitting/ pruning?►look at error rate: error rates for training data set will continuously decrease, error rates for test data will decrease, stop, and increase => stop when error rate for testing data is the smallest
►look at the variance reduction(AIC, BIC, 1-R^2) R^2 is the proportion of uncertainty that is explained by the model, therefore 1-R^2 is the uncertainty => split to increase R^2,
stop when the R^2 of the testing set start to decrease. - 44DM Techniques - classification analysis of decision tree►serve as a decision support tool that uses a tree-like graph of decisions and their possible consequences
►supervised data mining for classification and prediction
►know how to interpret decision tree can be interpreted - 45DM Techniques - classification post pruning for decision tree►find combined error rate of the children nodes: SUM (error of node i * number of classified items at node i/ total number of classified items at parent node)
►if the combined error rate of the children nodes is greater than that of the parent node =>> the sub tree is replaced by the parent node - 46DM Techniques - classification bayesian theoremP(H|X)=P(X|H)P(H)/P(X)
know what how to interpret the equation - 47DM Techniques - classification naive bayesian classifier►assuming attributes are independent
P(X|Ci)=P(X1|Ci)P(X2|Ci)P(X3|ci)...P(Xn|Ci)
►compute P(Ci|X)=P(X|Ci)*P(Ci)/P(X) (we dont need to compute P(X)
►assign X to Ci where P(Ci|X) has the greatest value
►know how to work with numerical example,
►what to do if P(Xi|Ci)=0 or if Xi follows some distribution - 48DM Techniques - classification advantages and disadvantages of naive bayesian classifier►advantages: easy to implement, good results obtained in most of the cases.
►disadvantages: assuming class conditional independence=> loss of accuracy because practically, dependency exists among variables=> dependencies among these cannot be modeled by Naive Bayesian Classifier - 49DM Techniques - classification bayesian belief network►allows a subset of variables to be conditionally dependent
►is a graphical model of causal relationship which represents the dependency among the variables and it gives a specification of joint probability distribution
►each node (variable) has a set of values, each with certain probability for the value
►to calculate any entry in the joint probability distribution, we need conditional probability tables
►know how to calculate based on examples - 50DM Techniques - genetic algorithm survival of the fitness►biological process: natural selection and propagation of genetic materials of parents to offsprings result in adaptable, specialized species and individuals
►parent: potential solution to the problem
offspring:new solution created from potential solution
natural selection:selection of a potential solution
propagation of of genetic materials: cross-over, mutation
survival of the fittest:a new solution which is used to generate another new solution - 51DM Techniques - genetic algorithm1.initialize a random population of chromosomes
2. evaluate the fitness value of each chromosome
3. while a good solution is not found, repeat these to get new chromosomes:
- apply cross over operation to two selected chromosomes
- apply mutation operation to a selected chromosome
- evaluate the fitness value of each new chromosome - 52what is OODA? What could WD do with BI OLAP/ dashboard?Observe, Orient, Decide, Action loop translates into 4 requirements for the vigilant information systems:
-capabilities to observe/ monitoring: identify KPI in real time, integrate information from various source to capture events of markets/ biz processes
-capabilities to orient/analytics: provide graphical dashboards, report, slice and dice data, send signals/ trends to identify opportunities and threats
-capabilities to decide: analytics for asking what if questions, time series comparison, descriptive statistics
-capabilities to make action/ new competitive action: architecture to communicating decisions quickly to other stakeholders to take actions
Informações
- Descrição
- FlashCards sobre Data mining, criado por Tien Tran em 21-11-2016.
- Criado por
- Tien Tran · nov 2016
- Atualizado
- 23 nov 2016
- Visualizações
- 4
- Avaliação