How To Install Apache Spark on Ubuntu

September 16, 2020 Liquid Web

Apache Spark is a distributed open-source, general-purpose framework for clustered computing. It is designed with computational speed in mind, from machine learning to stream processing to complex SQL queries. It can easily process and distribute work on large datasets across multiple computers.

Further, it employs in-memory cluster computing to increase the applications’ speed by reducing the need to write to disk. Spark provides APIs for multiple programming languages including Python, R, and Scala. These APIs abstract away the lower-level work that might otherwise be required to handle big data.

In this video, we demonstrate how to install Apache Spark on Ubuntu 18.04.

Commands used:
apt update -y
apt install default-jdk -y
wget https://downloads.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz
tar -xvzf spark-*
mv spark-3.0.1-bin-hadoop2.7/ /opt/spark
echo "export SPARK_HOME=/opt/spark" >> ~/.profile
echo "export PATH=$PATH:/opt/spark/bin:/opt/spark/sbin" >>~/.profile
echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.profile
source ~/.profile
start-master.sh
ssh -L 8080:localhost:8080 root@ubuntu1804.awesome.com
start-slave.sh spark://ubuntu1804.awesome.com:7077
spark-shell
pyspark
stop-slave.sh
stop-master.sh

Video by: Justin Palmer

About the Author

Liquid Web

Liquid Web powers online content, commerce, and potential for SMB entrepreneurs and the designers, developers and digital agencies who create for them. Sign up for our newsletter to stay up to date with our latest content.

More Content by Liquid Web
Previous Video
How to Configure Windows Remote Desktop Users Group
How to Configure Windows Remote Desktop Users Group

In this video, we demonstrate how to configure the windows remote desktop users group.

Next Video
How to Solve the VSFTPD 500 OOPS Error
How to Solve the VSFTPD 500 OOPS Error

In this video, we demonstrate how to solve the error: 500 OOPS: vsftpd: refusing to run with writable root ...

Learn What Qualities to Look for in Your Next Host in Our New Guide!

Download Now