Remotely control the robot using a chatbot, serverless compute and IoT – Part 2

Update: RekogRobot now has a 3D printed chassis! I use a graspinghand mount for the camera module.

In part 1 of this two-post series, I showed you how I built a robot powered by Raspberry Pi that moves about on its wheels, looks at objects and speaks what it sees, using Amazon Rekognition, Amazon S3 and Amazon Polly services from Amazon Web Services (AWS). If you haven’t already read that post I encourage you to go back and have a read. This post builds additional capabilities to that robot – letting you control the robot’s movement and functionality using your own voice or text using an Amazon Lex chatbot. Lex is the same technology behind Amazon Alexa that powers your Amazon Echo devices.



Let’s take a look at the high level design of how this looks like (and recap on some basics):

Robot architecture diagram

An introduction to the main components

Conversational Bot

The user interacts with a conversational bot powered by Amazon Lex. Amazon Lex provides advanced deep learning functionalities of automatic speech recognition (ASR) for converting speech to text, and natural language understanding (NLU) to recognize the intent of the text, to enable you to build applications with highly engaging user experiences and lifelike conversational interactions. Amazon Lex brings the same deep learning technologies that power Amazon Alexa to any developer, enabling you to quickly and easily build sophisticated, natural language, conversational bots (“chatbots”). Our Lex bot ‘RekogRobot‘ is configured to understand two intents – Move and SeeObject.

Internet of Things

In addition, the robot is configured as an IoT (Internet of Things) device on AWS IoT. This allows the user to securely and reliably issue commands to remotely control the robot.

If the user intends the robot to Move, the Lex bot calls an AWS Lambda function named MoveRobot. Based on the direction of travel the user commanded, the MoveRobot function updates the device’s IoT shadow using AWS IoT with the command to move, along with the direction of travel. Similary, if the user had intended the robot to see objects, the Lex bot calls another Lambda function named SeeRobot, which updates the device’s IoT shadow using AWS IoT with the command to SEE. I’ve also written a Python 3 script that runs on the robot that uses MQTT to connect to the device’s IoT shadow on AWS and look for commands (such as ‘see’ or ‘move’). If a command is found, it is executed immediately and the status reported back to the shadow.

Serverless functions

AWS Lambda lets you run code without provisioning or managing servers (serverless computing). You pay only for the compute time you consume – there is no charge when your code is not running. In this project, we just upload our python code and Lambda takes care of everything required to run and scale our code with high availability. Lambda supports multiple other languages and you can include libraries as well. In our project, MoveRobot and SeeRobot are both Lambda functions written in Python 3.6.

I have made all the code available as a package for download so you can start building right away.

When you extract the package you will find three folders:

  • lambda_functions: This contains the code that you will go into your Lambda functions. You will find two subdirectories – MoveRobot and SeeRobot, each of these is a separate Lambda function. You can put the files for each function into separate zip archives and import the code in the AWS Lambda console.
  • raspberry_pi: This folder contains the python3 code that needs to run on your raspberry pi. This folder also include the MP3 files with synthesized speech for robot status messages like “Turning left”, “Robot is recognizing objects”, etc.
  • lex_bot: This folder contains a json export of the Lex bot I’ve used. You can use this file or build the bot from scratch using the instructions below.

Create the IAM role and configure access

  1. Create a new IAM role. Let us call it Lambda_IoT_role.
  2. For services that will use this role, choose Lambda.
  3. Attach the following policies to this role: AWSLambdaBasicExecutionRole, AWSIoTDataAccess

IAM role for Lambda

Configure AWS IoT

  1. From the AWS Management Console, go to Services, and open AWS IoT Core.
  2. Go to Things, and click Register a thing.AWS IoT - Register a thing
  3. Choose Create a single thing.AWS IoT - Create a single thing
  4. Give your thing (your robot) a name and click Next. We’re calling it Rekogrobot in this example.
  5. Click Create Certificate. This is used to securely authenticate your robot when it communicates with AWS IoT.
  6. Download the IoT device certificate, the public key, private key, and also the root CA certificate. It is important to download all these files now as the private and public keys will not be able to retrieve once you leave this page. The root CA certificate may open text in a new browser tab. If this happens simply copy the entire block of text and save it in a file called root_ca.pem. After downloading, click the Activate button.Download IoT certificate
  7. Next, go to Secure > Policies and click Create a Policy.
  8. Give the policy a name. I’ll call it RobotPolicy. In the Action box enter iot:* and in the Resource ARN, enter *, and choose Allow as the EffectNote that this allows the robot to perform all iot actions unrestricted for the purpose of this experiment. In the real world, you will have to restrict this policy further.IoT Policy
  9. Next, you need to attach the policy you just created to the thing certificate. To do this, go to Secure > Certificates, right-click on the robot’s certificate and choose Attach policy.Attach IoT Policy to Thing Certificate
  10. Select the RobotPolicy policy and choose Attach.
  11. Go to Manage > Things, and choose Rekogrobot. Click Interact in the sidebar. Make a note of the HTTPS endpoint for updating the thing shadow, as well as the various MQTT topics. You’re going to need these in the next section.

Create the Lambda functions MoveRobot and SeeRobot

You will need to create two Lambda functions – MoveRobot and SeeRobot.

MoveRobot: Called by the Lex bot that you will create when it determines the intent of the user is to make the robot move. The Lex bot will have a slot named ‘Direction’ which will contain the intended direction of travel for the robot (Left, Right, Forward, Backward, Stop). The MoveRobot Lambda function will update the IoT device shadow for the robot so the shadow listener code on the Raspberry Pi can read the updates and perform actions.

  1. On the AWS Management Console, navigate to AWS Lambda. Click Create Function.
  2. Choose Author from scratch. Enter the name as MoveRobot. Runtime as Python 3.6. Choose the IAM role you created in the first section of this post. Click Create Function.
  3. This Lambda function does not require any triggers. It will be invoked by Lex.
  4. The MoveRobot and SeeRobot Lambda functions each consists of two files: and In the Designer view, either copy and paste the code from the gist provided in the next section below into two new files or copy the contents of /lambda_functions/MoveRobot/ folder into a zip file and upload the zip file directly into the Lambda console (Choose Upload .ZIP file in the drop-down under Code Entry Type in the Function code section). Here is how the Function code should look like when you’re done.
  5. Scroll further down and set the environment variables. AWS_IOT_MQTT_HOST should contain your thing shadow HTTPS endpoint that you made note of after configuring AWS IoT.
  6. Repeat steps 2 to 5 for the SeeRobot function. The code is available in /lambda_functions/SeeRobot/ folder or can be copied from the gist below.

Explanation of the code

For convenience, I have commented the code with details in the gist below. Feel free to post a comment if something isn’t clear.



You’ve now created two Lambda functions MoveRobot and SeeRobot. Let’s see how we can tie this up together with Lex.

Create a Lex bot

    1. Open the Amazon Lex on and AWS Management Console. Choose to create a Custom bot.
    2. Give your Lex bot a name. I’ve called mine Rekogrobot.
    3. Create two intents: Move and SeeObject.
    4. For the intent Move:
      • Create some sample utterances like the below:
      • Create a slot named Direction.
      • Create a slot type named RobotDirections. Populate the Slot type with the following values:
      • Configure the Fulfillment with a Lambda Function and choose the MoveRobot function. Choose Latest under Version or alias.
    5. For the intent SeeObject:
      • Create some suitable utterances.
      • Configure the Fulfillment with a Lambda Function and choose the MoveRobot function. Choose Latest under Version or alias.

While configuring your Lex bot, if you come across the below error, you may need to edit your Lambda function policy.

You can manually edit the function policy using the AWS Command-line interface (CLI) by using the following commands:

C:\Users\shijaza>aws lambda add-permission --region eu-west-1 --function-name MoveRobot --statement-id 1 --principal --action lambda:InvokeFunction --source-arn arn:aws:lex:eu-west-1:1xxxxxxxx:intent:Move:*
"Statement": "{\"Sid\":\"3\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:eu-west-1:1xxxxxxxx:function:MoveRobot\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:lex:eu-west-1:1xxxxxxxx:intent:Move:*\"}}}"

C:\Users\shijaza>aws lambda add-permission --region eu-west-1 --function-name SeeRobot --statement-id 1 --principal --action lambda:InvokeFunction --source-arn arn:aws:lex:eu-west-1:1xxxxxxxx:intent:SeeObject:*
"Statement": "{\"Sid\":\"1\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:eu-west-1:1xxxxxxxx:function:SeeRobot\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:lex:eu-west-1:1xxxxxxxx:intent:SeeObject:*\"}}}"

Preparing your robot on the Raspberry Pi

These steps assume you’ve successfully built the robot as per the instructions in Part 1.

You will need python3 to run the supplied code. You would already have python3 installed on your Raspberry Pi if you completed Part 1. Just a reminder – you can install python3 by typing sudo apt-get install python3

You also need to install the following packages:

The following packages are also required, but you’d already have these installed if you completed the steps in part 1 of this series.

To install a python3 package, make sure to use pip3. Example: pip3 install paho-mqtt

Copy the credentials to the ‘thing’ (the Raspberry Pi)

Create a directory where you want to run the Python script on the Raspberry Pi. Copy the thing certificate, private key file, public key file and root CA certificate .pem file that you obtained when you created your thing on the AWS IoT console.

Setting up the environment variables

Type the following commands to set the OS environment variables on the Raspberry Pi.

You are specifying the path/file names of the thing certificate, the private key file, the public key file, the root CA pem file, the thing shadow https endpoint (MQTT host), the default port numbers, the thing name and client ID that you configured in the AWS IoT console.

export AWS_IOTCERTIFICATE_FILENAME="e94de4a864-certificate.pem.crt"
export AWS_PRIVATE_KEY_FILENAME="e94de4a864-private.pem.key"
export AWS_PUBLIC_KEY_FILENAME="e94de4a864-public.pem.key"
export AWS_IOT_ROOT_CA_FILENAME="root_ca.pem"
export AWS_IOT_MQTT_PORT=8883
export AWS_IOT_MQTT_CLIENT_ID="Rekogrobot"
export AWS_IOT_THING_NAME="Rekogrobot"
Copying the script to the Raspberry Pi

Copy the in the /raspberry_pi folder to the same folder your Raspberry Pi and execute it by typing python3

I hope you enjoyed this post. Please share your feedback in the comments section below!