Q1)To Get Temporary variable from spark-shell or Check for variable in memory with $intp.
Ans:
$intp.definedTerms.map(dT => s"${dT.toTermName}: ${$intp.typeOfTerm(dT.toTermName.toString)}").filter(x => x.contains("()org.apache.spark.rdd.RDD")).foreach(println)
Q2)To get Lineage of a RDD.
Ans:
use toDebugString function on RDD.
Q3)To Know Wide or Narrow Dependencies of a RDD.
Ans:
use dependencies function on RDD.



No comments:
Post a Comment