Tag Archives: Spark
Run pyspark on your windows machine
1) Download Spark lib on your local machine and decompress the archive. Then set SPARK_HOME and HADOOP_HOME env variables to point to this decompressed folder location – For example: C:\Users\some_user\PycharmProjects\spark-2.4.4-bin-hadoop2.7 Also lookup the winutils executable online and you need to put it in the spark bin folder. 2) Install Java JDK if you do not […]
0 
Setup Apache Spark in JetBrains IntelliJ (Scala Version)
Please refer this post to see how to setup JetBrains IntelliJ IDE and Scala on your machine. 1) Download latest version of Apache Spark from http://spark.apache.org/downloads.html 2) Unpack the tarball to /opt/spark folder 3) Launch IntelliJ, create a new Scala project and choose SBT as the Build tool. 4) Once IntelliJ IDE is up and all dependencies […]
Login