DataFrame

DataFrame

1. Extends RDD & contains Row objects

2. can run SQL queries on SparkSession object: df = spark.sql("SELECT * FROM permcd")

   • df.show() to display rows or query results

3. has a schema = leading to more efficient storage

4. to create it, we use Spark Session object

   • can be created from CSV, JSON
   • to make the Dataset available to SQL within the current context: df.createOrReplaceTempView("someTableName")

5. Transformations = select, distinct, groupBy, orderBy, sum, filter, limit

   • Actions (=Lazy Execution) = show, count, collect, save, printSchema

6. since Spark 2.0 it's preferred data API