Spark learning

Page 1

Strongly typed language and week typed:-

The difference between a strongly typed language and a weakly typed language is basically that a strongly typed language checks the type of a variable before performing an operation on it, whereas a weakly typed language does not. In addition, strongly typed languages require explicit casts, while weakly typed languages perform implicit casts.

Datasets

https://databricks.com/blog/2016/01/04/introducing-apache-spark-datasets.html

Spark Datasets, an extension of the DataFrame API that provides a type-safe, object-oriented programming interface

Like DataFrames, Datasets take advantage of Spark’s Catalyst optimizer by exposing expressions and data fields to a query planner. Beyond Catalyst’s optimizer, Datasets also leverage Tungsten’s fast in-memory encoding. They extend these benefits with compile-time type safety—meaning production applications can be checked for errors before they are ran—and they also allow direct operations over user-defined classes

A Dataset is a strongly-typed, immutable collection of objects that are mapped to a relational schema. At the core of the Dataset API is a new concept called an encoder, which is responsible for converting between JVM objects and tabular representation

getting the same performance using RDDs would require users to manually consider how to express the computation in a way that parallelizes optimally.

Another benefit of this new Dataset API is the reduction in memory usage. Since Spark understands the structure of data in Datasets, it can create a more optimal layout in memory when caching Datasets.However, since Dataset encoders provide more information to Spark about the data being stored, the cached representation can be optimized to use 4.5x less space.

Encoders are highly optimized and use runtime code generation to build custom bytecode for serialization and deserialization. As a result, they can operate significantly faster than Java or Kryo serialization.

Seamless Support for Semi-Structured Data- The power of encoders goes beyond performance. They also serve as a powerful bridge between semi-structured formats (e.g. JSON) and type-safe languages like Java and Scala.

A Single API for Java and Scala-Another goal to the Dataset API is to provide a single interface that is usable in both Scala and Java

===================================================

Page 2