You probably have heard about it, wherever there is a talk about big data the name eventually comes up. In layman’s words Apache Spark is a large-scale data processing engine. Apache Spark provides various APIs for services to perform big data processing on it’s engine. PySpark is the Python API, exposing Spark programming model to Python applications. In my previous blog post, I talked about how set it up on Windows in my previous post. This time, we shall do it on Red Hat Enterprise Linux 8 or 7. You can follow along with free AWS EC2 instance, your hypervisor (VirtualBox, VMWare, Hyper-V, etc.) or a container on almost any Linux distribution. Commands we discuss below might slightly change from one distribution to the next.
Like most of my blog posts, my objective is to write a comprehensive post on real world end to end configuration, rather than talking about just one step. On Red Hat 7, I ran into a problem. I solved this problem without having to solve it.
The Author
Environment
- Linux (I am using Red Hat Enterprise Linux 8 and 7)
- Java
- Hadoop
- Spark
- Anaconda or pip based virtual python environment
I have broken out the process into steps. Please feel free to skip a section as you deem appropriate.
Table of Contents
- EPEL Repository
- Update Linux software
- Set Password
- Install GUI and set it as default
- Set up VNC and Remote Desktop Protocol
- Tip: If your remote desktop connection terminates as soon as you login
- Version Compatibility Warning
- Install Java
- Spark
- Hadoop
- Python
- Environment variables for HadoopSpark & PySpark
EPEL Repository
Extra Packages for Enterprise Linux, EPEL for short. It includes but isn’t limited to Red Hat Enterprise Linux (RHEL), CentOS, Scientific Linux (SL), Oracle Linux (OL), etc. In other words, EPEL is open source and free community supported project by Fedora and thrives to be a reliable source for up to date packages. Needless to say, it provides yum and dnf packages. If you want to know more, please check out their official wiki page and blog post from Red Hat. We would download the rpm using wget and, install it.
How to determine RHEL version?
// option 1
$ cat /etc/redhat-release
// option 2
$ cat /etc/os-release
// option 3 (RHEL 7.x or newer)
$ hostnamectlDepending on the version you use, please see the wiki page to properly configure it.
// RHEL/CentOS 8 64 bit
$ wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
// RHEL/CentOS 7 64 bit
$ wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
// RHEL/CentOS 6 64 bit
$ wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
// RHEL/CentOS 6 32 bit
$ wget http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpmFor example, you could install the rpm package after downloading or could also try doing it all in a single step.
// install the downloaded rpm file above
$ sudo rpm -ivh epel-release-6-8.noarch.rpm
// single step alternative
$ yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpmTo verify, let’s quickly run repolist command which would list currently active repositories.
$ yum repolist | grep epelIf you need it, EPEL configuration file should be located at /etc/yum.repos.d/epel.repo. If you have followed the steps you should have yum config manager installed on your machine at this point. Should you chose you can use the following commands to disable or enable EPEL, respectively.
$ sudo yum-config-manager --disable epel
$ yum-config-manager --enable epelUpdate Linux software
If you are using AWS EC2 or equivalent, chances are you have bare bones base image and you are using SSH to log into your Linux. Let’s start by updating the base image. Depending on the number of packages, your machine configuration and internet speed – this might take a few minutes. The last command would install gcc, flex, autoconf, etc. which we would need to install fastparquet using pip, esp. if you you on RHEL 7.x.
$ sudo yum clean all
$ sudo yum -y update
$ sudo yum groupinstall "Development tools"
$ sudo yum install gcc
$ sudo yum install python3-develSet Password
If you do not know your current username, you can find out by executing whoami command. Now to set password:
$ sudo passwd <user_name>To set root password, you can use the following command:
$ sudo su
# passwdInstall GUI and set it as default
If you are not planning to use GUI on this machine, you can skip this step. It would save you time, internet data and storage space.
// download gui components
$ sudo yum groupinstall -y "Server with GUI"
// set gui as default
$ sudo systemctl set-default graphical.target
$ sudo systemctl defaultSet up VNC and Remote Desktop Protocol
In this step we install VNC server and Remote Desktop Protocol. If you would rather stay with SSH, skip to the next step.
$ sudo yum install -y xrdp tigervnc-serverNow, we can set up SELINUX security as below.
chcon --type=bin_t /usr/sbin/xrdp
chcon --type=bin_t /usr/sbin/xrdp-sesmanWe are now ready to start, enable and check the status of xrdp.service
$ sudo systemctl start xrdp.service
$ sudo systemctl status xrdp.service
$ sudo systemctl enable xrdp.serviceThe default port for using the Remote Desktop Protocol is 3389. This port should be open thru firewall to make it accessible by RDP. To ensure we actually can login, we have to do two things:
- Check the port using
ssornetstat - Add firewall rule to “open” the port
If you want to make this port accessible through the internet, not just your local network then, you want to enable port forwarding. It is uncommon and outside the context of this post. When doing so, please use extra caution.
$ netstat -antpTo add this port permanently to firewall, we want to check if firewall service is running. If it isn’t, we want to kick-start it.
$ sudo systemctl status firewalld.service
// if not running
$ sudo systemctl enable firewalld.service
$ sudo systemctl start firewalld.serviceAdd port and then reboot your machine. When the machine boots up again, you should be able to use Remote Desktop.
$ sudo firewall-cmd --permanent --add-port=3389/tcp
$ sudo firewall-cmd --reload
// reboot
$ sudo rebootTip: If your remote desktop connection terminates as soon as you login …
If you are not facing any issues please feel free to quickly skim through this paragraph.
Subscribe to get access
Read more of this content when you subscribe today.
$ cat ~/.xsession-errorsIn my case, on RHEL 7 when using Anaconda, removing/commenting out the Anaconda block from ~/.bashrc solved the issue. I removed Anaconda and switched to pip based virtual environment since I needed remote desktop.
Version Compatibility Warning
A lot of questions end up on the forums with the root cause of version mismatch. Please do the due diligence to make sure Java, Spark, Hadoop and Python versions are compatible with each other. For example if you are using Spark 2.4.4, then install OpenJDK 1.8.0 instead 11 and Python 3.6.8 instead newer version.
Install Java
You may want to install either of the following. If you need OpenJDK for different platforms download no-cost long-term supported distribution of JDK.
// OpenJDK 11
$ sudo dnf install java-11-openjdk-devel
// OpenJDK 1.8.0 - Option 1
$ sudo yum install java-1.8.0-openjdk-devel.x86_64
// OpenJDK 1.8.0 - Option 2
$ sudo dnf install java-1.8.0-openjdk-develSpark
Download Spark
$ cd /opt
$ sudo wget https://archive.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
$ sudo tar -xf spark-2.4.4-bin-hadoop2.7.tgz
$ sudo ln -s /opt/spark-2.4.4-bin-hadoop2.7 /opt/sparkConfigure Spark Master and Slave services
$ sudo su
# useradd spark
# chown -R spark:spark /opt/spark*Create a file using $ sudo touch /etc/systemd/system/spark-master.service with the following content.
[Unit]
Description=Apache Spark Master
After=network.target
[Service]
Type=forking
User=spark
Group=spark
ExecStart=/opt/spark/sbin/start-master.sh
ExecStop=/opt/spark/sbin/stop-master.sh
[Install]
WantedBy=multi-user.targetTo configure Spark slave service create a file using $ sudo touch /etc/systemd/system/spark-slave.service and add the following content. Replace IP-ADDRESS-or-hostname with IP address or the hostname of your machine.
[Unit]
Description=Apache Spark Slave
After=network.target
[Service]
Type=forking
User=spark
Group=spark
ExecStart=/opt/spark/sbin/start-slave.sh spark://IP-ADDRESS-or-hostname:7077
ExecStop=/opt/spark/sbin/stop-slave.sh
[Install]
WantedBy=multi-user.targetLoad newly created Spark service files
$ sudo systemctl daemon-reloadStart Spark Service
// master service
$ sudo systemctl start spark-master.service
$ sudo systemctl status spark-master.service
// slave service
$ sudo systemctl start spark-slave.service
$ sudo systemctl status spark-slave.serviceHadoop
$ cd /opt
$ sudo wget https://archive.apache.org/dist/hadoop/common/hadoop-2.10.1/hadoop-2.10.1.tar.gz
$ sudo tar -xf hadoop-2.10.1.tar.gz
$ sudo ln -s /opt/hadoop-2.10.1 /opt/hadoopPython
Spark comes with Python and most Linux distributions come with Python. So, why would you need Anaconda or pip virtual environment? Well you don’t, until you do. 😀
Option 1) Anaconda or Miniconda
Miniconda is a minimal installer for conda. Like Anaconda, Miniconda is a software package that includes the conda package manager and Python and its dependencies, but does not include any other packages. Once conda is installed by installing either Anaconda or Miniconda, other software packages may be installed directly from the command line with conda install. Let’s first download Anaconda. I am using a version which is right for me, but take a moment to check the anaconda website and the pages linked below, if you need to better understand the commands I have used below.
- Installing on Linux — Anaconda documentation
- Hashes for all files — Anaconda documentation
- Managing environments — Anaconda documentation
Download and install conda
$ cd /opt
$ sudo wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.shTo ensure the integrity of the file we downloaded, we should verify the SHA256 checksum with the hash Anaconda has published.
$ sha256sum /opt/Anaconda3-2020.11-Linux-x86_64.shIf the hash matches we can start installation and follow the on screen instructions.
$ sh /opt/Anaconda3-2020.11-Linux-x86_64.shCreate or import conda virtual environment
When you are ready, let’s create conda virtual environment. Either you can import the file below or you can create one on your own. I would call this environment dev376, you can call it whatever you like. If you want to quickly import an environment, run create a file by running $ touch ~/dev376.yml and add the following content to it.
name: dev376
channels:
- defaults
dependencies:
- numpy
- jupyter
- fastparquet
- autopep8
- pandas
- snakeviz
- python=3.7.6
- pyspark
- pylint
prefix: /home/<user_name>/anaconda3/envs/dev376Now to automatically create an environment with just these packages above, we would execute the following command:
$ cd ~
$ conda create env --no-default-packages --file dev376.ymlTip: If you are getting SSL related error
In some cases, based on your network settings, at work or home conda might have trouble downloading the metadata and packages from the internet. Temporarily disable SSL check and when you are done, you should probably reset to default value.
$ conda config --set ssl_verify falseOption 2) pip virtual environment
If you skipped the section Update Linux software in the beginning, check if you have gcc installed. pip and wheel might need it and your setup would likely fail without it. For pip, we would first install only the packages we need to set up virtual environment.
$ pip install virtualenv virtualenvwrapperNow is the time to find the version of Python you want to use. Most probably you have multiple versions of Python available to you. To find this, execute the following command and settle on a path you want to use. FWIW, I am going to use /usr/bin/python3.6.
$ whereis pythonWe would now append the following to ~/.bashrc file:
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.6
export WORKON_HOME=/var/home/<user_name>/Envs
export VIRTUALENVWRAPPER_SCRIPT=/var/home/<user_name>/.local/bin/virtualenvwrapper.sh
source /var/home/<user_name>/.local/bin/virtualenvwrapper.shAfter you save and close the file, we would need to load these newly added variables. To do so, run the following command:
$ source ~/.bashrcWe are now ready to create virtual environment dev368.
$ mkvirtualenv -p /usr/bin/python3.6 dev368
$ workon dev368Be a good citizen and upgrade your pip first.
$ /var/home/<user_name>/Envs/dev368/bin/python -m pip install --upgrade pipI am providing you ~/requirements.txt file which you can use to bootstrap your virtual python environment or you certainly can create your own as you please.
pip install -r ~/requirements.txtFinally, if you want to activate this virtual environment every time you start your bash from PuTTY or GNOME Terminal, add the following line to ~/.bashrc file:
workon dev368Environment variables for Hadoop, Spark & PySpark
Add a few more variables to your ~/.bashrc file so that it looks like following:
export HADOOP_HOME=/opt/hadoop
export SPARK_HOME=/opt/spark
export PYSPARK_PYTHON=/usr/bin/python3.6
export PATH=$PATH:$SPARK_HOME/bin:$HADOOP_HOME/bin
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.7-src.zip
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.6
export WORKON_HOME=/var/home/<user_name>/Envs
export VIRTUALENVWRAPPER_SCRIPT=/var/home/<user_name>/.local/bin/virtualenvwrapper.sh
source /var/home/<user_name>/.local/bin/virtualenvwrapper.sh
workon dev368Save and close the file. Take a deep breath and check if Spark shell works:
$ spark-shellAnd if that runs without any problem, then get out of it by pressing Ctrl + D and test PySpark:
$ pysparkMake sure that spark master and slave services are running, start them if they are not and in your browser navigate to Spark UI!
Just realized it is May 4, 2021. Awesome!
May the 4th be with you!
How did you generate the table of content (ToC), bro?
LikeLike