Feed Buzzard
  • General
  • Tech
  • World Tech
  • World Tech Code
  • Wearable Tech
  • Pokemon
  • About Us
    • Terms & Conditions
    • Privacy Policy
  • Contact
No Result
View All Result
  • General
  • Tech
  • World Tech
  • World Tech Code
  • Wearable Tech
  • Pokemon
  • About Us
    • Terms & Conditions
    • Privacy Policy
  • Contact
No Result
View All Result
Feed Buzzard
No Result
View All Result
Home World Tech

How to build your first Raspberry Pi cluster? –

Gordon James by Gordon James
March 28, 2021
in World Tech
0
0
SHARES
0
VIEWS
Share on FacebookShare on Twitter

Do you have two or more Crimson P’s at home?
Would you like to try to collect them in a group?
If so, you’ve come to the right place. When I bought my second Raspberry Pi, I immediately wanted to build a cluster.

You can create a cluster on your Raspberry Pi by installing the same operating system, applications and libraries on all the nodes. Only the MPICH application is needed to execute the commands on all nodes. There is also a library in Python to enhance the functions : MPI4PY.

Since this topic can be difficult for beginners, I’ll start with a brief introduction to clusters in general.
I will then explain what I did and how you can do the same on your end.

Group representation

What is a cluster?

Essentially, a cluster is a group of computers in a unit.
The goal is to get them to work together to improve overall performance.
All computers in the cluster work on the same task, reducing the time it takes to complete the task.

Do not confuse computer clustering with load balancing.
In a load balancing architecture, each computer works separately to reduce the load on the master node.
In a cluster, we use all the power of the cluster to perform the task in parallel.

data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2020/12/1608398820_992_How-to-Easily-Install-Apps-on-Raspberry-Pi-OS-5.jpg />

Raspberry Pi
Course Take it to the next level.
I’m here to help you get started with your Raspberry Pi and learn all the necessary skills in the right order.

Examples of clusters

Computer clusters first appeared in the 1960s and are still in use today (along with the first networking efforts).

alt=hatch width=150 height=150 data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2021/03/How-to-build-your-first-Raspberry-Pi-cluster–.jpg-.jpg />

The first commercial computer cluster in history is Arcnet (see the picture on the left).
His goal was to connect groups of Datapoint 2200 computers.
That’s pretty old in the history of computers :).

IBM’s ORNL Summit is, at the time of writing, the largest supercomputer in the world.
With over 2 million processor cores and 3,000 tons of RAM still growing, it will be hard to compete.
Here’s an illustration if you want to know what it looks like: alt=summary computer cluster ibm width=900 height=465 data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2021/03/1616729233_769_How-to-build-your-first-Raspberry-Pi-cluster–.jpg-.jpg data-ez= />summary from IBM (source: ornl.gov)

Raspberry Pi application

Let’s get back to our more realistic measurements with an adaptation of this definition on the Raspberry Pi.

As you know, the Raspberry Pi is not very powerful, but it is cheap.
It is therefore the ideal device for creating a cluster.
We can make the Raspberry Pi work faster on 4 devices, not just one, and at a reasonable price.

In this tutorial, I will show you how to build your first Raspberry Pi cluster.
You can build a cluster with two nodes for operation and add more nodes later if needed.

Prepare your Raspberry Pi cluster

Making a plan

It is always good to think about what you are building.
I do this exercise for you, with two Raspberry Pi :

  • Raspberry Pi 4B 4G: the main hub that will control everything.
  • A Raspberry 3B+: second node, for better overall performance

Since the preparation phase can be quite long (especially if you use a lot of knots), I will only prepare the 4B.
Then I copy the SD card to another card, so the Raspberry Pi 3B+ is almost ready without me having to do all the basic work on the 3B+.
Finally, for the two Raspberry Pi devices, there are more steps to connect them and run the first script.

If you need to add more than one node, repeat the same process for each node.

Condition

To follow this tutorial, you will need :

  • 2 or more Raspberry Pi’s (any model, but I recommend the Raspberry Pi 4B)
  • 2 or more SD cards (see my recommended products page if you need them)
  • Affordable 5-port Gigabit switch to connect all Pi’s together
  • Power cable, or power supply with 2 or more connectors
  • Network cable for each Pi (wireless is possible, but not optimal).
  • Optional: if you’re serious about this project, this cluster box can be handy for stacking the Raspberry Pi and avoiding a huge mess.
    The enclosure optimizes your wiring, keeps everything clean, and ensures proper cooling of assemblies. I highly recommend this case if you rotate your cluster often.

And as for the software, I’ll explain everything in the next sections.

Message: It does not matter if you are using SD cards of different sizes, but you should set the wizard to the smallest SD card.
Otherwise you will have a problem flashing a 64G image on a 16G SD card :).

Prepare master

The first step in my scenario is to install it on one Raspberry Pi, and then duplicate it on the other.
Start with your powerful Raspberry Pi.

data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2020/12/1607451018_397_How-to-Install-VMWare-ESXi-on-a-Raspberry-Pi-Step.jpg />

Learn the Raspberry Pi in 30 days
Download the ebook.
Discover the secrets of the Raspberry Pi in a 30-day challenge.

Basic installation

As with most projects, we start with the installation of Raspin.
Download Raspbian Lite from the Raspberry Pi Foundation website: Link here.
Raspbian Desktop is fine, but we don’t need a GUI for this project.

Install Raspbian and boot for the first time (if you don’t know how to install Raspbian on your Raspberry Pi, follow my guide and come back later).

You must then perform the following additional steps:

  • Change some parameters with Raspi-config :
    • Start Raspy configuration:
      sudo raspy-config
    • Change the password for the pi user under System Options > Password.
      We’re going to enable SSH, and it’s not a good idea to use the default password when starting SSH.
    • Enable SSH under Interface options > SSH.
    • You can change the host name under System Options > Host Name.
      Pick something obvious, like the host.
  • Update your system:
    • As always, you should start any project with a modern system to avoid problems.
      Update the source code from the repository:
      sudo apt update
    • Update all packages: update
      sudo apt
  • Reboot to apply all changes:
    reboot sud

The basic installation is now complete, we can move on to the software specific to this project.

MPICH installation

What is MPICH?

MPICH is the most important tool to start a cluster.
MPICH is a free implementation of the MPI standard.
MPI stands for Message Passing Interface and is used to control parallel computer architectures.

In short, this is what will allow us to run the script on multiple Raspberry Pi’s at once.

MPICH installation

We are now ready to begin the MPICH installation process.
If you need the latest version, you can download MPICH from the official website and compile from source, but it is also available in the Raspberry Pi OS repository.

Here is the easiest way to install it:
sudo apt install mpich.

When everything is ready, check that everything is working properly.
To do this, run the following command:
mpiexec -n 1 Date

When you receive the current date from the master, MPI installation has been completed.

Creating a simple Python script

Ok, now we will create a simple Python script to test it with MPI.

  • Go to your home directory and create a script:
    cd /home/pi
    nano test.py
  • Add this line (or another line):
    printout(Hi)
  • Make sure your script works directly with Python:
    python test.py
    If you saved my script, it should say ‘hello’.
  • Next, test it by running it on 4 threads with MPI:
    mpiexec -n 4 python test.py
    As you can see, Hello should now appear four times, so we can also run the Python script four times using all available CPU cores.

That’s good, but we’re not using a cluster yet, it’s just a way to run the script on multiple threads.

MPI4PY Installation

What is MPI4PY?

To continue with our cluster, we need a library that we can use in the script. The purpose of this library is to provide communication between all nodes so that our programs run efficiently.

On the Raspberry Pi, MPI can be used directly in Fortran and C scripts.
However, since the Raspberry Pi uses Python, we will add Python functions to our cluster.

To do this, we need to use the : MPI4PY.

Condition MPI4PY

The installation process of MPI4PY is easy as it is available with pip (Python package manager).
However, before you do anything, you need to install some packages from the Raspberry Pi operating system:
sudo apt install python-pip python-dev libopenmpi-dev

That’s it, continue with the installation process.

MPI4PY Installation

We can now install the MPI4PY library using pip:
sudo pip install mpi4py
This may take more or less time depending on your Raspberry Pi model. Be patient.

If everything works properly, the master is ready.
MPI can now run Python scripts and we can start preparing the node.

Duplicate master

The next step is to duplicate the master SD card to other cards, one for each node.
To do this, we create an image of the SD card and flash it to other cards.

If you are trying to do this with only two nodes, it may be faster to repeat the same process as on the master. In that case, you can skip this section.

Creating an image

To run on Windows, you need software like. B. Win32DiskImager.
Click on the link, download and install the software on your computer:

  • Start the program.
    alt=win32diskimager width=482 height=342 data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2021/03/1616729234_928_How-to-build-your-first-Raspberry-Pi-cluster–.jpg-.jpg data-ez= />
  • In the Image File field, select a temporary directory and a file name, for example. B. cluster_master.img.
  • Then select the device letter corresponding to the SD card.
  • Finally, click the Play button to start creating the image.
    This process took about 15 minutes on my computer.
  • Then remove the SD master card and keep it in a safe place.

On Linux it should look like this:
sudo dd if=/dev/sdb > cluster_master.img
You need to make sure that /dev/sdb is your SD card. You can easily find help on this command if needed (or use man dd to see all the options).

Create an SD card for node

Once the image is ready, you need to create an SD card for each node in your cluster:

  • Insert a new SD card into the computer.
  • In Win32 Disk Imager, select an image file name and device letter.
  • Click the Write button to create the same SD card.

If you want, you can use Etcher for this.
I usually use Etcher, but we’re already in Win32 Disk Imager, so that’s the same thing.
Reminder: You should use a larger SD card than the first one. alt=cluster width flashing image=297 height=300 data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2021/03/1616729234_138_How-to-build-your-first-Raspberry-Pi-cluster–.jpg-.jpg data-ez= />.

For Linux and macOS users, you can use the dd command again if you don’t want to install Etcher.

At the end of this step, you will have an SD card for each node you want to use.
All SD cards contain the same image from the previously created master.

Junction configuration

Launch of all Raspberry Pi

  • Insert an SD card into each Raspberry Pi you want to use.
  • Throw them all.

If you want to use Wi-Fi for one or more nodes, there is an additional step.
In my case, for example, I have a Raspberry Pi Zero and it was easier for me to connect it to a Wi-Fi network.

  • Connect the screen and keyboard to the Raspberry Pi on which you want to use Wi-Fi.
  • Use Raspi-config to configure Wi-Fi:
    • Use the following command:
      sudo raspi-config
    • Go to System Preferences > Wireless LAN.
    • Follow the wizard to select the network (country, SSID and password).

Search for all IP addresses

Once all the Raspberry Pi’s are running and connected to the network, we need to get all the IP addresses for further use:

  • Return to the master node (directly or via SSH).
  • Install NMAP:
    sudo apt install nmap
    nmap is a free network discovery program (see the website here).
    We use it to find all IP addresses.
  • Use this command to find all devices on your network whose hostname contains the master.
    At this point, all Raspberry Pi’s have the same hostname :
    nmap -sP 192.168.1.* | grep master
    Change the subnet if you are using a different subnet.
  • You should get this result:
    alt=nmap scan cluster ip width=595 height=50 data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2021/03/1616729235_722_How-to-build-your-first-Raspberry-Pi-cluster–.jpg-.jpg data-ez= />
  • I now know the IP of the second node: 192.168.1.18

You should now have the IPs of all nodes.
If you do not know the master, you should use this command:
sudo ifconfig

You will get something like this: alt=ifconfig width=796 height=182 data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2021/02/How-to-Shut-Down-Raspberry-Pi-via-SSH-And-a.png data-ez= />

The IP address is on the second line after the inet keyword (in this screenshot 192.168.1.200).
The final step is to mark these IP addresses in a text file on your main computer:

  • Create a new file in your home directory:
    cd /home/pi
    nano nodes_ips
  • In this file, add the IP of the node in each line (and only the IP).
    For example:
    192.168.1.15
    192.168.
    192.168.1.17
    192.168.1.18
  • That’s it for this part.

Changing host names

We will now change the hostname on the new node so that each node has a different hostname:

  • Connect from the master node to the first node via SSH:
    ssh pi@192.168.1.18
    Answer with yes and enter the password pi.
  • Go to the Raspy configuration:
    • Use this command to invoke the tool:
      sudo raspi-config
    • Go to System Preferences > Hostname.
    • Set a new hostname for this node, for example. B. Node1.
  • Leave the Raspy configuration and exit this junction:
    exit

Repeat these steps for each node you want to add to the cluster.

SSH key exchange

The final step is to allow the wizard to connect to any host via SSH without a password.
To do this, you need to create an SSH key on the master and distribute it to all nodes to activate it.

  • Create an SSH key in the wizard with:
    ssh-keygen -t rsa
    Accept the default values (default path and no password).
  • This tool creates two keys in the /home/pi/.ssh directory:
    • Your private key, keep it here.
    • id_rsa.pub : public key to send to the peers you want to access without a password.
  • Forward the public key to all nodes:
    scp /home/pi/.ssh/id_rsa.pub pi@192.168.1.18:/home/pi/master.pub
    Do this for each node you wish to use.
  • Then go to each node and add the key to the authorized_keys file.
    This file contains all hosts that are authorized to access SSH without a password:
    ssh pi@192.168.1.18
    cat master.pub >> .ssh/authorized_keys
    exit
    Do this for each host.
    If the directory does not exist, just create it:
    mkdir .ssh
  • You should now be able to connect each node without a password.
    You can try:
    ssh pi@192.168.1.18

That’s it, your cluster is ready. Now let’s see.

Use of cluster

The cluster is now available and we will use MPI to execute commands on each node simultaneously.
As we have seen, you can use MPI to execute basic commands and scripts on a cluster.

Basic operation

First, we can try running the same command on each node.
Preferably something that doesn’t return the same :).

For example:
mpiexec -hostfile nodes_ips -n 8 hostname

nodes_ips is the file we created earlier that contains all the IP addresses.
And the hostname is the command we want to run on each node.
8 is the number of threads to run, in this case change it to the number of cores available in your cluster (the Raspberry Pi 4B and 3B+ have 4 cores each, so I’m testing with 8). alt=width=673 height=212 data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2021/03/1616729236_121_How-to-build-your-first-Raspberry-Pi-cluster–.jpg-.jpg data-ez= />

This will give you a line for each node in the cluster, with the names of all the nodes.

Python Script

Test Script

If you followed this tutorial completely, you should already have a test.py script in your home folder.
You can test if it can be run on each node with the same command:
mpiexec -hostfile nodes_ips -n 8 python test.py

This will show Bonjour twice, once for each node.
We’re not using MPI4PY yet, but we’ll get to that in a minute.

New scenario

Remember that after cloning all SD cards, you must have new scripts on all nodes.
MPI simulates the execution of the script on each node, but it does not copy the code automatically.

To do this, briefly proceed as follows:

  • Create a script on the main node.
  • Make sure it works as intended.
  • Then transfer this script to all nodes with scp:
    scp /home/pi/myscript.py pi@192.168.1.18:/home/pi/
    It is important to have the same script with the same path on each node.
  • Then you can run your script as described above with MPI.

Continue with Python

As I said before, we didn’t just add MPI4PY to run simple Python scripts 4 times instead of once.
MPI4PY is a Python library that you can include in your scripts to use some functions in your cluster.

Here’s a quick example:

#!/usr/bin/env python

from mpi4py import MPI

Switch = MPI.COMM_WORLD
Grade = Switch

if rank == 0 :
data = {‘a’:1, ‘b’:2, ‘c’:3}
otherwise :
data = None

data = comm.bcast(data, root=0)
print ‘rank’,rank,data

The purpose of this script is to send data from one thread or node to all others.
In this script, the data is defined only for the first thread of the master (rank 0).
We then synchronize this data with all running instances using a broadcast function (Comm.bcast).

Here is the run command to try this:
mpirun.openmpi -np

-mashinefile nodes_ips python mpi4py.py

When this script is run, all nodes and rows display the same message: alt=width script pythoncluster=867 height=84 data-ezsrc=http://server.digimetriq.com/wp-content/uploads/2021/03/1616729237_550_How-to-build-your-first-Raspberry-Pi-cluster–.jpg-.jpg data-ez= />.

This is just an example to show that you can add other functions to your Python script to take advantage of your cluster.
I’m not an expert on the subject.
More information can be found here.

Related topics

Can I add more nodes to my cluster now? You can always add nodes to an existing cluster (that’s how it’s done with supercomputers). Just create a new SD card, follow the node configuration steps for the new node and add the new IP address to the nodes_ips file.

IP addresses change every day, what can I do? Yeah, that’s a problem. I didn’t do this step for the test, but if you want to keep your cluster, you should do it. Depending on your network, you can either set redundancy on your DHCP server (so that each Pi always gets the same IP at startup). You can also manually set a static IP address in your network configuration (I’ll explain how at the end of this article).

In reality, what is the purpose of a cluster? What interested me about this guide was the technology and installation process. Not the options that are available in this cluster right now. That’s another topic and I can’t fit everything into one article. If you want to go further, you can find other projects about clusters on Hackaday

Supplement

Now you know how to build your Raspberry Pi cluster of two nodes to infinity :).

I really enjoyed writing this guide for you.
It’s interesting to get an idea of how supercomputers work.
And the technology seems to be stable, as I had no problem making the cluster (and that’s a rarity in the computer world ^^). I hope you enjoy it too.

By the way, you can read my article on what a realistic use of a Raspberry Pi cluster might be here.

If you have any questions or would like to share your experiences, please leave a comment below.
I’d love to know what you’re doing after these first steps into the world of supercomputers…..

frequently asked questions

What is the importance of a Raspberry Pi cluster ?

What is the importance of a Raspberry Pi cluster ? The purpose of a Raspberry Pi cluster is to combine the power of multiple devices into a single stack. Clustering is nice and can also be useful for applications that need a lot of CPU cores but not a lot of speed.

How powerful is the Raspberry Pi cluster?

The maximum load is 4000 watts. Other supercomputers consume between 10 and 25 megawatts of power. A close-up of the BitScope racks. A supercomputer based on a Raspberry Pi will be much slower than a real supercomputer, but the architecture of the system is comparable to more expensive systems.

How do I boot up a Raspberry Pi for the first time?

It’s time to get started with a Raspberry Pi. Insert the SD card into the bottom of the Raspberry Pi and make sure it is aligned properly; it would be a bad idea to break the Pi before turning it on! Connect it to the power supply, monitor, keyboard and mouse. The Raspberry Pi boots up and takes you to the NOOBS screen.

Related Tags:

raspberry pi cluster case,raspberry pi cluster kit,raspberry pi cluster web server,how to build a raspberry pi,building a 4 node raspberry pi cluster,raspberry pi cluster as desktop,Feedback,Privacy settings,How Search works,Raspberry Pi 4 Model B,Raspberry Pi,Raspberry Pi 3 Model B RASPB…,Raspberry Pi 3 Model B+,See more,raspberry pi kubernetes cluster performance,raspberry pi 4

Total
0
Shares
Share 0
Tweet 0
Pin it 0
Share 0
Gordon James

Gordon James

James Gordon is a content manager for the website Feedbuzzard. He loves spending time in nature, and his favorite pastime is watching dogs play. He also enjoys watching sunsets, as the colors are always so soothing to him. James loves learning about new technology, and he is excited to be working on a website that covers this topic.

Related Posts

Blockchain Conference in Poland Showcases New Solutions to Problems in Enterprise Logistics
World Tech

Blockchain Conference in Poland Showcases New Solutions to Problems in Enterprise Logistics

December 15, 2022
World Tech

Rockstar Games Brings GTA to The Metaverse

October 20, 2022
World Tech

Ways Technology Is Changing The World Of Education

September 21, 2022
Next Post

ReactJS vs Angular vs Vue Which One Is Better For Learning

Download Android Multi Tools v1.02b [100% Working Guide]

Download, Install & Use Intellect: Create a Better You on PC (Windows & Mac)

No Result
View All Result

Recommended

3 Types of Facial Implants You Should Know About

3 Types of Facial Implants You Should Know About

12 hours ago
Myths and Misconceptions About Joint Pain

Myths and Misconceptions About Joint Pain

4 days ago
The Spider Veins Facts that Every Individual Ought to Know

The Spider Veins Facts that Every Individual Ought to Know

4 days ago

5 Benefits of Telehealth That You Should Understand

4 days ago

Categories

  • Fitness Trackers
  • General
  • Latest
  • Pokemon
  • Tech
  • Technology and Computing
  • Wearable Tech
  • World Tech
  • World Tech Code

Recent Posts

  • 3 Types of Facial Implants You Should Know About February 2, 2023
  • Myths and Misconceptions About Joint Pain January 30, 2023
  • The Spider Veins Facts that Every Individual Ought to Know January 30, 2023
  • 5 Benefits of Telehealth That You Should Understand January 30, 2023

Categories

  • Fitness Trackers
  • General
  • Latest
  • Pokemon
  • Tech
  • Technology and Computing
  • Wearable Tech
  • World Tech
  • World Tech Code

© 2022 FeedBuzzard.com

No Result
View All Result
  • General
  • Tech
  • World Tech
  • World Tech Code
  • Wearable Tech
  • Pokemon
  • About Us
    • Terms & Conditions
    • Privacy Policy
  • Contact

© 2023 JNews - Premium WordPress news & magazine theme by Jegtheme.

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
Cookie SettingsAccept All
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT