Data Scientist

Data Scientist

1. Data Munupilation

   • Data Manipulations
     • Group Operations
       • R: Apply Family
         • apply(X, MARGIN, FUN, ...)
           • X    an array, including a matrix.      MARGIN a vector giving the subscripts which the function will be applied over. E.g., for a matrix 1 indicates rows, 2 indicates columns, c(1, 2) indicates rows and columns. Where X has named dimnames, it can be a character vector selecting dimension names. FUN the function to be applied: see ‘Details’. In the case of functions like +, %*%, etc., the function name must be backquoted or quoted.    
           • theMatrix <-matrix(1:9, nrow =3) > theMatrix [,1] [,2] [,3] [1,] 1 4 7 [2,] 2 5 8 [3,] 3 6 9 > apply(theMatrix, 1, sum)[1] 12 15 18  > apply(theMatrix, 2, sum)[1] 6 15 24
           • Alternatively: > rowSums(theMatrix) [1] 12 15 18 > colSums(theMatrix) [1] 6 15 24
           • Handling Missing Value: > theMatrix[2,1] <- NA > theMatrix [,1] [,2] [,3] [1,] 1 4 7 [2,] NA 5 8 [3,] 3 6 9 > apply(theMatrix, 1, sum) [1] 12 NA 18 > apply(theMatrix, 1, sum, na.rm = TRUE) [1] 12 13 18  > rowSums(theMatrix) [1] 12 NA 18 > rowSums(theMatrix, na.rm = TRUE) [1] 12 13 18
         • lapply(X, FUN, ...) : return as a list
           • > theList <-list(A= matrix(1:9, 3), B = 1:5, C = matrix(1:4, 2), D=2) > theList $A [,1] [,2] [,3] [1,] 1 4 7 [2,] 2 5 8 [3,] 3 6 9 $B [1] 1 2 3 4 5 $C [,1] [,2] [1,] 1 3 [2,] 2 4 $D [1] 2 > lapply(theList, sum) $A [1] 45 $B [1] 15 $C [1] 10 $D [1] 2
         • sapply(X, FUN, ...): return as a vector
           • > sapply(theList, sum) A B C D 45 15 10 2 
         • mapply(FUN, ...,)
           • mapply is a multivariate version of sapply. mapply applies FUN to the first elements of each ... argument, the second elements, the third elements, and so on. Arguments are recycled if necessary.    
           • > firstlist <-list(A = matrix(1:16, 4), B = matrix(1:16,2),c = 1:5) > secondlist <-list(A = matrix(1:16, 4), B = matrix(1:16,8),c = 15:1) > mapply(identical, firstlist, secondlist) A B c TRUE FALSE FALSE  > simpleFunc <- function(x,y) {} > simpleFunc <- function(x,y) + { + NROW(x) +NROW(y) + } > mapply(simpleFunc, firstlist, secondlist) A B c 8 10 20 > 
       • Python: Pandas Package
         • groupby
           • >>> df = DataFrame({'key1' : ['a', 'a', 'b', 'b', 'a'], 'key2' : ['one', 'two', 'one', 'two', 'one'], 'data1' : np.random.randn(5), 'data2' : np.random.randn(5)}) >>> df data1 data2 key1 key2 0 -0.879349 -0.398744 a one 1 0.028745 0.475802 a two 2 0.279248 0.743438 b one 3 -2.875181 -0.956999 b two 4 -0.968092 -0.223995 a one
           • >>> grouped = df['data1'].groupby(df['key1']) >>> grouped <pandas.core.groupby.SeriesGroupBy object at 0x104f191d0> >>> print grouped <pandas.core.groupby.SeriesGroupBy object at 0x104f191d0> >>> grouped.mean() key1 a -0.606232 b -1.297967 Name: data1, dtype: float64
   • Data Reshaping
   • ManipulatingStrings

2. Predictive Modeling

3. Statistical Analysis

4. Visulazition

5. SQL/Data Base

   • Select <item> from <source>