SPARK

1. Действия
   1. Set
      1. Intersection(otherSet)
      2. union(otherSet)
      3. cartesian(otherSet)
   2. Functional
      1. filter(func)
      2. map(func)
   3. distinct
2. Трансформации
   1. saveAsTextFile(path)
   2. array
      1. collect()
      2. take(n)
      3. count
      4. drop(n)
   3. reduce(function)

      Anotações:

      • функция должна быть коммутативной и ассоциативной

1. SparkContext
   1. pyspark.sql.SparkSession (sparkContext)
   2. pyspark.sql.SparkSession (sparkContext)

1. pyspark.sql

   Anotações:

   • http://spark.apache.org/docs/latest/api/python/pyspark.sql.html
   1. functions
      1. udf
2. pyspark.streaming
3. pyspark.ml
4. pyspark.mllib

   Anotações:

   • https://habrahabr.ru/company/mlclass/blog/251471/
   1. linalg
      1. Vectors
         1. dense
         2. sparse
   2. stat
      1. Statistics
         1. colStats
            1. mean
            2. numNonzeros
            3. variance
         2. corr
   3. feature
      1. StandardScaler

         Anotações:

         • scaler = StandardScaler(withMean=True, withStd=True).fit(features) scaler.transform (features.map(lambda x:x.toArray()))
   4. classification
      1. LogisticRegressionWithSGD
      2. RidgeRegressionWithSGD
      3. NaiveBayes
   5. tree
      1. DecisionTree
      2. RandomForest
   6. clustering
      1. KMeans
   7. recommendation
      1. ALS

         Anotações:

         • Коллаборативная фильтрация

Anotações:

• https://0x0fff.com/spark-architecture-shuffle/