Amazon Web Services

Building a robot with computer vision and speech

Posted on

In this two-part series, I show you how I built a robot using a Raspberry Pi, the CamJam EduKit #3, and Amazon Web Services.

First, let’s take a look at a demo that describes the robot’s capabilities:

The robot works in two modes:

  1. ‘Remote Control’ mode: The robot’s navigation can be controlled remotely through a web-based console, and it can be made to “speak” out what it “sees”.
  2. Chatbot control: I am currently working on this. Control the robot using your own voice and text based chat. This will be featured in part 2 of this post when I’m ready.

In this post, I explain the physical build of the robot as well as preparation of the operating system and software.

Components required

Before we begin, let’s take a look at what components went into my robot:

Raspberry Pi 3 Model B

Available here and on Amazon.com

The lower-priced Raspberry Pi Zero W could also work but it will be complicated to connect the speakers as the Pi Zero does not have an audio jack.

CamJam EduKit #3 – Robotics

I found this kit extremely useful. It has (nearly) all the components you need to get started building a robot instead of having to figure out everything by yourself. The EduKit #3 includes:

  • 2 x DC motors 
  • 1 x DC motor controller board
  • 2 x wheels
  • 1 x ball castor (‘front wheel’ of the robot)
  • 1 x small breadboard
  • 1 x battery box for 4 AA batteries to drive the motor
  • 1 x ultrasonic distance sensor
  • A line follower sensor (not required for this project)
  • Resistors and jumper cables

You can buy one at thePiHut

Power bank – small size

Batteries

You probably already have one of these laying around. This is used to power your Raspberry Pi. I used an old Nokia power bank.

You will also need four (4) AA batteries.

Powered speaker with 3.5mm jack, and an audio Cable

Your robot will speak through this. A rechargeable/powered speaker will be needed. Try to get a light one so that you do not add stress to the motors. I used a Nokia Bluetooth speaker that I had laying around.

Raspberry Pi Camera Module

Your robot will see using this. 

You can buy one here or on Amazon.com

microUSB cable

To connect your power bank to your Raspberry Pi. I recommend a very short cable, with right angle connectors to save space inside the chassis. Speaking of which…

A chassis – a plastic or cardboard box, OR access to a 3D Printer.

You’ll need a chassis for your robot. Print one or use the cardboard box that came with the CamJam EduKit (this is what I did).  Use your imagination!

Amazon Web Services (AWS) account.

Create one at aws.amazon.com if you don’t already have one. We’ll be using Amazon S3, Amazon Polly, and Amazon Rekognition for this project.

Assembling your robot

I’ve put together a short video on the components that went into the robot. This is to supplement the already detailed documentation available on the CamJam website.

In addition to the CamJam components, you’ll need to connect a Raspberry Pi camera module, mount the camera on a ZeroView camera mount minus the Pi Zero (or other mounting arrangement), and connect a speaker to the 3.5mm headphone jack.

Configuring the Raspberry Pi

Now that you have finished putting together your robot, it’s time to prepare your Raspberry Pi. Below are some of the things you’ll need to do. You’ll find steps for these in the documentation for Raspberry Pi – I’ve added links.

  1. Configure the Raspberry Pi with a static IP on your wireless network. [documentation] You’ll need internet access to download updates and some of the packages.
  2. Enable SSH (recommended for convenience) [documentation]
  3. Install the latest updates. [documentation]
  4. Install Python 3 [documentation]. The code provided is not compatible with Python 2.x.
  5. You may want to change the default hostname and change the default password.
  6. Download and install webbot on your Pi from GitHub. We’ll be modifying code from this project.
  7. Install pygame on your Raspberry Pi [documentation]
  8. Enable the camera interface on your Raspberry Pi. (Tip: type sudo raspi-config)
  9. While you’re there, change the audio configuration of the Pi so that it plays through the speakers you’ve connected to the 3.5mm socket, instead of the HDMI port. [documentation]
  10. You can also set the time and timezone in raspi-config. (Tip: look under Localization/Localisation options)
  11. Sign up for an Amazon Web Services (AWS) account if you don’t already have one.
  12. Install the AWS Command Line Interface (CLI) on the Raspberry Pi. [documentation]
  13. Install boto3. That’s the Amazon Web Services SDK for Python.
  14. On the machine that you’re using to SSH into the Raspberry Pi, install an SCP client (like WinSCP) if you don’t already have it installed. This is not a required step, but it will make it much easier for you to edit code on your PC and have it synchronized with your Pi during testing.

Configuring Amazon Web Services (AWS)

Here is a diagram that describes the process at a high level.

Creating and configuring an S3 bucket
  1. Create an S3 bucket. In my example, I am calling it rekorobot.
  2. Since this is meant to be an experiment, you could configure the bucket permissions to allow public read. Important: Note that pictures taken by the robot will be publicly accessible if you use this method, so do not store sensitive content. It is recommended to further secure the bucket for more specific read access but this requires some changes beyond the scope of this post. Public write is not required for this project and should NOT be enabled. 
  3. Optional: Create a lifecycle rule to delete all objects in the bucket after one day. This ensures that pictures uploaded by the robot are deleted every day and you don’t incur unnecessary charges for images that are no longer needed.
Configuring access

1. Create an IAM user (say robotuser) with the following AWS Managed Policies attached: AmazonRekognitionFullAccess, AmazonPollyFullAccess. Note the Access Key ID and Secret Access Key. You’re going to need this later.

2. Create a Managed Policy that allows the user access to put and get objects on the rekorobot bucket you just created. [documentation]

2. On the Raspberry Pi, configure the AWS CLI using the aws configure command. Enter the Access Key ID and the Secret Access Key for the robotuser IAM user. 

Configuring Amazon Rekognition

There isn’t much to configure here. Once you’ve allowed access to Amazon Rekognition for robotuser, your code can call the Amazon Rekognition APIs right away using the boto3 SDK!

Configuring Amazon Polly

There isn’t much to configure here. Once you’ve allowed access to Amazon Polly for robotuser, your code can call the Amazon Rekognition APIs right away!

However, you’ll need Polly to generate a speech file for you to copy to the robot so it can say something like “I am currently not able to identify any objects” in case nothing was “seen” or recognized. To do this, simply type the text and choose Download MP3. You can customize the voice. I chose Salli.

Save this file to the /webbot folder on your Raspberry Pi. Name the file as notfound_Salli.mp3

Follow the same steps to create another file named robotready_Salli.mp3 that can play a message like “Hello. Robot is now ready” when the robot starts up. Copy this file to the /webbot folder.

Writing and modifying code

You will need to modify the index.html file in the /webbot/public folder for cosmetic changes that add the additional functionality we need. Take a look at the modified index.html below:

Here’s how the modified interface looks like, when accessed from my phone:

Modified Webbot Interface

Next, you’ll need to modify the webbot.py code using your favorite text editor so that it does the extra bits:

1. Take a picture programmatically using the Pi Camera and the raspistill utility when a button is clicked on in the web interface provided by webbot.

2. Upload the image from the camera to S3.

3. Have the image analyzed by Amazon Rekognition and obtain the response.

4. Send the response text to Amazon Polly and obtain the speech response.

5. Play out the speech through the speaker using pygame. 

6. Some basic error handling.

I like to use WinSCP and NotePad++ on my PC for editing code on the Raspberry Pi. Below is the modified code. I have provided commented code so things are clear.

Bonus content

In case you’re wondering what “the view from the robot” was like when I recorded the first video above, here they are:

I hope you find this helpful. If you do, be sure to post a comment below. Have fun!

RaspberryPi

Building a motion sensor for Philips Hue lights using Raspberry Pi Zero

Posted on

In this post, we discuss how you can use a Raspberry Pi Zero W to build a motion sensor to control your Philips Hue lights at home. There is a lot of sample code for motion sensors out there on the Internet, however I believe this one would be more practical for actual use at home.

For instance, the code includes:

  • The capability to prevent the motion sensor from turning on lights at certain times of the day (for example during daytime or at bedtime in a bedroom).
  • You can adjust how long the Hue lights in your room should remain on once motion is detected.
  • The code runs in an endless loop. Use crontab to make sure this code starts running each time the pi reboots.
  • The code also prints logging information for troubleshooting.

Here’s what you’ll need:

  1. Raspberry Pi Zero W with NOOBS installed: Although you can use other models of Raspberry Pi with this code, I used a Raspberry Pi Zero W, which is low cost, comes with adequate processing power for a motion sensor and built-in WiFi and Bluetooth.
  2. PIR Motion Sensor module
  3. Suitable case for Raspberry Pi Zero W (I use Pibow)
  4. GPIO Header, jumper cables, power supply, etc.

Wiring up your Raspberry Pi Zero W

If you are familiar connecting a PIR motion sensor module to your Raspberry Pi Zero W, you can skip this step.

Raspberry Pi Zero W - connecting a PIR motion sensor  PIR motion sensor module - connections 

The PIR motion sensor module has three pins – VCC, OUT and GND. Make connections as follows using jumper cables:

VCC –> GPIO Pin 2 (5V power) – black wire in picture

OUT –> GPIO Pin 16 – white wire in picture

GND –> GPIO Pin 6 (ground) – gray wire in picture

The PIR motion sensor module also has two potentiometers labeled TX (time delay) and SX (sensitivity). You will need to experiment these settings with a small Philips screwdriver till you get desired results. Sensitivity (SX) determines how much motion is required for the sensor to conclude that there has been motion. The time delay (TX) determines how long the PIR will maintain a “High” (ON) output after detecting motion.

PIR motion sensor module - adjusting sensitivity and time
The two potentiometers – TX and SX (orange)

Preparing your Raspberry Pi Zero W and writing the python script

Prepare the Raspberry Pi Zero, by installing NOOBS, configuring your WiFi, and enabling SSH. You will also need to install python. For information around this, read the documentation for Raspberry Pi.

Next, install the phue python library on your Raspberry Pi by typing the following command:

[cc lang=”bash”]pip install phue[/cc]

Use the below code to create a python file, say motionsensor.py.

You can use an editor such as nano, for creating the python script.

[cc lang=”bash”]nano motionsensor.py[/cc]

You can test your code and see if it works by executing the python script from the command line:

[cc lang=”bash”]python motionsensor.py[/cc]

The first time you execute this code, you’ll need to push the button on your Philips Hue Bridge to authorize the application to access and send commands to the bridge.

Putty session

Configuring crontab to automatically run the code

Once you’ve satisfactorily tested the code, it’s time to edit crontab so that the motion sensor code executes each time the Raspberry Pi is restarted. The code itself runs in an endless loop so it continues running till the Raspberry Pi device is shutdown or restarted.

First, you’ll need to create a bash script that runs the application. Let’s call this file launcher.sh. You can edit the file using a text editor like nano.

[cc lang=”bash”]nano launcher.sh[/cc]

In the launcher.sh file, type the commands you’ll need to execute to run the python file. Save the file.

[cc lang=”bash”]
cd /
cd /home/pi/motionsensor
python motionsensor.py
cd /
[/cc]

You’ll also need to make the file executable with the following command:

[cc lang=”bash”]chmod +x launcher.sh[/cc]

Now you’re ready to edit the crontab, and configure the launcher.sh bash script to run at reboot. Use the following commands to edit your crontab:

[cc lang=”bash”]sudo crontab -e[/cc]

You can also pipe the screen output from the python code to a log file named after the date and time so that you can maintain log files for troubleshooting. I have also added a command to delete the older log files on a daily basis.

Below is an example of how the crontab file will look like after the required edits (see the last two lines).

For more information on crontab, see the documentation.