SPARK

1. Действия
   1. Set
      1. Intersection(otherSet)
      2. union(otherSet)
      3. cartesian(otherSet)
   2. Functional
      1. filter(func)
      2. map(func)
   3. distinct
2. Трансформации
   1. saveAsTextFile(path)
   2. array
      1. collect()
      2. take(n)
      3. count
      4. drop(n)
   3. reduce(function)

      Nota:

      • функция должна быть коммутативной и ассоциативной

1. SparkContext
   1. pyspark.sql.SparkSession (sparkContext)
   2. pyspark.sql.SparkSession (sparkContext)

1. pyspark.sql

   Nota:

   • http://spark.apache.org/docs/latest/api/python/pyspark.sql.html
   1. functions
      1. udf
2. pyspark.streaming
3. pyspark.ml
4. pyspark.mllib

   Nota:

   • https://habrahabr.ru/company/mlclass/blog/251471/
   1. linalg
      1. Vectors
         1. dense
         2. sparse
   2. stat
      1. Statistics
         1. colStats
            1. mean
            2. numNonzeros
            3. variance
         2. corr
   3. feature
      1. StandardScaler

         Nota:

         • scaler = StandardScaler(withMean=True, withStd=True).fit(features) scaler.transform (features.map(lambda x:x.toArray()))
   4. classification
      1. LogisticRegressionWithSGD
      2. RidgeRegressionWithSGD
      3. NaiveBayes
   5. tree
      1. DecisionTree
      2. RandomForest
   6. clustering
      1. KMeans
   7. recommendation
      1. ALS

         Nota:

         • Коллаборативная фильтрация

Nota:

• https://0x0fff.com/spark-architecture-shuffle/