SPARK

1. Действия
   1. Set
      1. Intersection(otherSet)
      2. union(otherSet)
      3. cartesian(otherSet)
   2. Functional
      1. filter(func)
      2. map(func)
   3. distinct
2. Трансформации
   1. saveAsTextFile(path)
   2. array
      1. collect()
      2. take(n)
      3. count
      4. drop(n)
   3. reduce(function)

      Annotations:

      • функция должна быть коммутативной и ассоциативной

1. SparkContext
   1. pyspark.sql.SparkSession (sparkContext)
   2. pyspark.sql.SparkSession (sparkContext)

1. pyspark.sql

   Annotations:

   • http://spark.apache.org/docs/latest/api/python/pyspark.sql.html
   1. functions
      1. udf
2. pyspark.streaming
3. pyspark.ml
4. pyspark.mllib

   Annotations:

   • https://habrahabr.ru/company/mlclass/blog/251471/
   1. linalg
      1. Vectors
         1. dense
         2. sparse
   2. stat
      1. Statistics
         1. colStats
            1. mean
            2. numNonzeros
            3. variance
         2. corr
   3. feature
      1. StandardScaler

         Annotations:

         • scaler = StandardScaler(withMean=True, withStd=True).fit(features) scaler.transform (features.map(lambda x:x.toArray()))
   4. classification
      1. LogisticRegressionWithSGD
      2. RidgeRegressionWithSGD
      3. NaiveBayes
   5. tree
      1. DecisionTree
      2. RandomForest
   6. clustering
      1. KMeans
   7. recommendation
      1. ALS

         Annotations:

         • Коллаборативная фильтрация

Annotations:

• https://0x0fff.com/spark-architecture-shuffle/