RDD-based programming

RDD-based programming

1. SparkContext

   • The "connection" of the driver to the cluster is based on the Spark Context object e.g. SparkConf conf  = new SparkConf().setAppName("Applicationname");   JavaSparkContext sc = new JavaSparkContext(conf);

2. RDD

   • A Spark RDD is an immutable distributed collection of objects, which allows parallelizing the code. Each RDD is split in partitions.
   • RDD ceation
     • By loading an external dataset
       • from files
         • e.g. JavaRDD<String> lines = sc.textFile(inputFile): can set the number of partitions: JavaRDD<String> lines = sc.textFile(inputFile, 5);
     • By parallelizing a local collection created in the driver. e.g a java collection
       • based on the "parallelize(List<T> list)"
         • // create a local Java list List&lt;String&gt; inputList = Arrays.asList("First ele"...); // Build an RDD of Strings from the local list JavaRDD&lt;String&gt; distList = sc.parallelize(inputList); // number of partitions are set automatically by the spark if no parameter of numPartitions specifies in parallelize method
   • Save RDD
     • saveAsTextFile(String outputPath)
   • RDD operations
     • Transformations
     • Actions

Images and files in the map

• 
• Scala and spark (Embed)