How to install Spark in Mac os x 10.7 or later and Working through Jupyter Notebook:
Step1:-
Download Spark from Spark website.
a) Choose a Spark release (I prefer 1.6.0)
b) Choose a package type: (this installation prefers "Pre-built for Hadoop 2.6 and later")
c) Choose a download type: (Direct Download)
d) Download Spark: spark-1.6.0-bin-hadoop2.6.tgz
Make sure you have java installed on your machine.
Go to your home directory. (You can use the command in red)
cd ~
Unzip the folder in your home directory using the following command.
tar -zxvf spark-1.6.0-bin-hadoop2.6.tgz
(in version spark-1.6.0,you choose whatever version you like)
Use the following command to see that you have a .bash_profile
ls -a
Next, we will edit our .bash_profile so we can open a spark notebook in any directory.
sudo nano .bash_profile
Don't remove anything in your .bash_profile. Only add
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
In alis, instead of snotebook you can write any name as you like.Its doesn matter.
After done all this setting,save and exit (cntrl+x,cntrl+o) from your bash frofile.
Clear and close the terminal.
Reopen the terminal again and type snotebook(your prefer name).
It will open browser based Jupyter notebook with Spark install.
Output of PySpark Command:-
Step1:-
Download Spark from Spark website.
a) Choose a Spark release (I prefer 1.6.0)
b) Choose a package type: (this installation prefers "Pre-built for Hadoop 2.6 and later")
c) Choose a download type: (Direct Download)
d) Download Spark: spark-1.6.0-bin-hadoop2.6.tgz
- (you can click on this or go to http://spark.apache.org/downloads.html to choose your own Spark Version.
Make sure you have java installed on your machine.
Go to your home directory. (You can use the command in red)
cd ~
Unzip the folder in your home directory using the following command.
tar -zxvf spark-1.6.0-bin-hadoop2.6.tgz
(in version spark-1.6.0,you choose whatever version you like)
Use the following command to see that you have a .bash_profile
ls -a
Next, we will edit our .bash_profile so we can open a spark notebook in any directory.
sudo nano .bash_profile
Don't remove anything in your .bash_profile. Only add
setting path for spark :-
export SPARK_PATH=~/spark-1.6.0-bin-hadoop2.6
export PYSPARK_DRIVER_PYTHON="jupyter" export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'
In alis, instead of snotebook you can write any name as you like.Its doesn matter.
After done all this setting,save and exit (cntrl+x,cntrl+o) from your bash frofile.
Clear and close the terminal.
Reopen the terminal again and type snotebook(your prefer name).
It will open browser based Jupyter notebook with Spark install.
Output of PySpark Command:-


