Setup Apache Spark in JetBrains IntelliJ (Scala Version)

Please refer thisĀ post to see how to setup JetBrains IntelliJ IDE and Scala on your machine.

1) Download latest version of Apache Spark fromĀ http://spark.apache.org/downloads.html

2) Unpack the tarball to /opt/spark folder

3) Launch IntelliJ, create a new Scala project and choose SBT as the Build tool.

4) Once IntelliJ IDE is up and all dependencies have been resolved, you need to import Apache Spark’s jar libraries. Spark is written in Scala and you’ll have use the version of Scala with which your version of Spark was originally built. To figure that, just inspect /opt/spark/jars folder.



So from the above, it seems my version of Spark was built using Scala v2.11.

5) In IntelliJ, click on File > Project Structure, then ‘modules’ on the left and finally select ‘Dependencies’ tab on the right. Click on the green ‘+’ sign, browse to /opt/spark/jars folder and import all the jars under here in your project. I believe only a subset of jars (mainly hadoop, scala and spark) from here will actually be needed for running Spark but I haven’t figured which one is or isn’t needed at the moment.



6) Now switch to the ‘Libraries’ on the left and on the right, replace paths to existing scala compiler and library with jar paths from /opt/spark/jars folder.



7) Now hit OK to go back to main IDE, create a new Scala file under src/main/scala folder and type in the following code in the new file.



8) Save the file, press ALT + SHIFT + F10 to run the script and view the output.

leave your comment