Single Node Kafka Cluster in Windows 10 system.

1)Installer JDK1.8

2)Go to https://kafka.apache.org/downloads and download
Scala 2.11 - kafka_2.11-2.0.0.tgz and extract using 7-zip

3)Zookeeper configuration:
Create a directory "C:\kafka_2.11-2.0.0\zookeper_data"
Go to kafka_2.11-2.0.0\config\zookeeper.properties
dataDir=C:\kafka_2.11-2.0.0\zookeper_data

4)Kafka Server properties configuration:
Create a Kafka log directory "C:\kafka_2.11-2.0.0\kafka_log"
log.dirs=C:\kafka_2.11-2.0.0\kfaka-logs
Few more properties need to be added in same properties file.
offsets.topic.num.partitions=1
offsets.topic.replication.factor=1
min.insync.replicas=1
default.replication.factor=1

5)Running & Testing of Kafka with following steps.
Set the Environment variable to 'path' variable with "C:\kafka_2.11-2.0.0\bin\windows".
For Windows system we need to go to windows directory and execute the batch file.
Step1: Start zookeeper ->
C:\kafka_2.11-2.0.0\bin\windows>zookeeper-server-start.bat C:\kafka_2.11-2.0.0\config\zookeeper.properties
Step2:Start Kafka server ->
C:\kafka_2.11-2.0.0\bin\windows>kafka-server-start.bat C:\kafka_2.11-2.0.0\config\server.properties
Step3:Check number of Kafaka Node ->
C:\kafka_2.11-2.0.0\bin\windows>zookeeper-shell.bat localhost:2181 ls /brokers/ids
Step4:Create a Kafka Topic ->
C:\kafka_2.11-2.0.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Step5:Create a Topic to send message(Producer) ->
C:\kafka_2.11-2.0.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic test
>Hello Suresh!
>kafka is amazing
>Terminate batch job (Y/N)? y
Step5:Create a Consumer to send message(Consumer) ->
C:\kafka_2.11-2.0.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
Hello Suresh!
kafa is amazing

ctrl+C to close the command prompt.

How to Fix issue while running Apache Spark on Windows?

Error: 18/06/13 1:04:11 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394) at org.apache.hadoop.util.Shell.(Shell.java:387) at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2327) at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:365) at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:105) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:228) at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:963) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:91)

Solution:
(Note:You Should have the rights of Administrator.)
Step1:
Download winutils.exe binary from https://github.com/steveloughran/winutils or http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe repository.
Save winutils.exe binary to a directory of your choice, e.g. c:\winutils\bin.
Step2:
Using command prompt to set the Environment variable As mentioned below.
set HADOOP_HOME=c:\winutils
Set PATH environment variable to include %HADOOP_HOME%\bin as follows:
set PATH=%HADOOP_HOME%\bin;%PATH%
Or
same can be done by right click on PC->properties ->Advance System Setting -> Environment Variable.
Step3:
Restart the Eclipse or Intellij IDE

Anonymous functions

Q1)What is Anonymous functions in Scala?
Ans:An anonymous function is an expression that evaluates to a function;
the function is defined without giving it a name.
Example:
(x:Int) => x*x*x

Higher-order function

Q1)what is higher-order function in Scala?
Ans:Functions which take other functions as parameters or return them as results are called higher-order functions.
Exmple:
Normal function
def square(x: Int): Int = x * x
High-order function:
def sumSquares(a: Int, b: Int): Int = if (a > b) 0 else square(a) + sumSquares(a + 1, b)

Tail Recursion in Scala

Q1)What is tail-recursion and why in this scenario Scala is Better then Java?
Ans:
If the last action of a function is a call to another (possibly the same)
function, only a single stack frame is needed for both functions. Such calls are called
“tail calls”. In principle, tail calls can always re-use the stack frame of the calling
function. However, some run-time environments (such as the Java VM) lack the
primitives to make stack frame re-use for tail calls efficient. A production quality
Scala implementation is therefore only required to re-use the stack frame of a di-
rectly tail-recursive function whose last action is a call to itself. Other tail calls might
be optimized also, but one should not rely on this across implementations.
Example:
def gcd(a: Int, b: Int): Int = if (b == 0) a else gcd(b, a % b)
gcd(14, 21)
→if (21 == 0) 14 else gcd(21, 14 % 21)
→if (false) 14 else gcd(21, 14 % 21)
→gcd(21, 14 % 21)
→gcd(21, 14)
→if (14 == 0) 21 else gcd(14, 21 % 14)
→ → gcd(14, 21 % 14)
→gcd(14, 7)
→if (7 == 0) 14 else gcd(7, 14 % 7)
→ → gcd(7, 14 % 7)
→gcd(7, 0)
→if (0 == 0) 7 else gcd(0, 7 % 0)
→ → 7

Call-by-Value and Call-by-name in Scala

Q1)What is Call-by-Value and Call-by-name in Scala?
Ans:
Define One Function
def sumOfSquares(x: Double, y: Double) = square(x) + square(y)
Call-by-value
Call-by-value has the advantage that it avoids repeated evaluation of arguments.
Call-by-value is usually more efficient than call-by-name.
example:
sumOfSquares(5, 3+3)
→ sumOfSquares(5, 6)
→ square(5) + square(6)
→ 5 * 5 + square(6)
→ 25 + square(6)
→ 25 + 6 * 6
→ 25 + 36
→ 61
Call-by-name
Call-by-name has the advantage that it avoids evaluation of arguments when the
parameter is not used at all by the function.
example:
sumOfSquares(5, 3+3)
→ square(5) + square(3+3)
→ 5 * 5 + square(3+3)
→ 25 + square(3+3)
→ 25 + (3+3) * (3+3)
→ 25 + 6 * (3+3)
→ 25 + 6 * 6
→ 25 + 36
→ 61
Note:
Scala uses call-by-value by default, but it switches to call-by-name evaluation if the
parameter type is preceded by =>

Small step towards learning Scala

Q1)To get Out put as mentioned below pattern.
Code Snippet:
(1 to 5).map("*"*_).foreach(println(_))
OutPut:
*
**
***
****
*****
Q2)To get all the even number between two number i.e 1 to 10
Code Snippet:
(1 to 10).filter(_%2==0).foreach(println(_))
Output:
2
4
6
8
10
Q3)One way to find factorial of a number?
Code Snippet:
(1 to 5).reduceLeft(_*_)
or def factorial(n: Int):Int = if(n==0) 1 else n*factorial(n-1)
Output:
120
Q4)Split the string_> sort the string by length,Print in sorted length all word?
input: var s= "Scala supports both object-oriented and functional"
Code Snippet:
s.split(" ").sortWith(_.length<_.length).foreach(println(_))
<--> OutPut:
and
both
Scala
supports
functional
object-oriented
Q5)How to call function inside function,let's suppose we pass one list and one max value and get all the value less then max value from the list? Input:
List:List(9,5,1,4,7,3,8)
Max Vaue:8
Code Snippet:
object Nest extends App{
| def fun1(l:List[Int],max:Int)={
| def fun2(m:List[Int]):List[Int]=
| if(m.isEmpty) m
| else if(m.head | else fun2(m.tail)
| fun2(l)
| }
| println(fun1(List(9,5,1,4,7,3,8),5))
| }
Invocation:<-->
Nest.fun1(List(9,5,1,4,7,3,8),8)
Output:
List[Int] = List(5, 1, 4, 7, 3)