In this tutorial we will learn how to build an IoT application using the Google Cloud Platform. The application will get sensor data from the Raspberry Pi and send it to the cloud. This data will be stored in BigQuery and plotted in a graph.
Contents
- Hardware Requirements
- Initial Setup
- Creating A Project
- Enable Required APIs
- Setup IoT core Service
- Download API credential JSON file
- Setup Google BigQuery
- Setting up cloud storage bucket
- Setting Google Dataflow
- Setting up the Raspberry Pi
- Running the Code
- Visualize the data
1. Hardware Requirements
- Raspberry pi 3
- HC-SR04 (or any sensor)
- 330 Ω and 470 Ω resistors
- Breadboard
- Jumper cables
NOTE :- you can use any sensor you like and send any data to the cloud. Here we will be using the HC-SR04 distance sensor as an example.
2. Initial Setup
- You will have to first create a google account or use an existing google account.
- Create your google cloud account. (This process requires your credit card details to verify that you are not a robot).
Note : You can use the free trial as well to follow this tutorial.
3. Creating A Project
A project organizes all your Google Cloud resources. A project consists of a set of users; a set of APIs; and billing, authentication, and monitoring settings for those APIs. Let’s begin.
- Go to the manage resources page on your Google cloud console and select “Create Project” ( https://console.cloud.google.com/cloud-resource-manager)
- Enter a name for the project, it can be anything that you wish, and click “Create“.
4. Enable Required APIs
- We need to make sure that we have enabled the following APIs :
- Dataflow API
- Cloud Pub/Sub API
- Cloud IoT API
- Find “APIs and service” in the hamburger menu present on the top left. Hover over it and click “Dashboard”.
- Next click on “Enable APIs and Services”.
- In the search bar type “Dataflow API”.
- Select the result and you should have a page like this.
- Click on “Enable“.
- Repeat steps 4 to 6 for the other two APIs mentioned.
5. Setup IoT core Service
- In the search bar present at the top of the page search for “IoT core” and select it.
- Once it opens the IoT core page click on the “Create Registry” button.
- Enter a registry ID and the region that is closest to you geographically.
- Create a new Cloud Pub/Sub topic. For this, click on the drop down menu and choose “Create A Topic”.
- Give it a topic ID that you like, I am going to name it “my-topic” and click “Create Topic”.
- Let’s finish the creation of registry by clicking “Create”.
6. Download API credential JSON file
- Now we have to get the credentials to authenticate our API requests. https://console.cloud.google.com/apis/credentials/serviceaccountkey
Note : make sure you have the correct project selected.
- Under service account select “New service account” and then the rest of the options will open up. Enter a name that you wish and choose your role as owner and keep the Key type as JSON and click create.
- Now send this JSON file to the Raspberry Pi and store it safely, as it gives access to your cloud account.
7. Setup Google BigQuery
We are going to use Google’s BigQuery service as our database to store all the data that our sensor captures.
- Now search for “BigQuery” in the search bar or in the menu on the left side, under the “Big Data” category and select it.
- Once the page opens up you will see a button labeled “Create Dataset”, Click it.
- Now you’ll have to name your new data set. I have named it “DistanceSensorDataset”. You should also choose your geographic location in the “Data Location” drop down.
- Click on the “Create Table” button. All the data that you generate from the sensor will be stored here.
- In this page, select “Empty table” in the “Create table from” drop down. Enter a table name, it can be anything that you wish.
- Click on “Add field”. Enter a field name and the data type that you wish to store in that field. You can add as many field names as you wish.
8. Setting up cloud storage bucket
- Find and select “Storage” in either the hamburger menu on the left side or search for it in the search bar.
- Now create a new bucket, for this click on “Create Bucket”.
- Now lets create the bucket and name it . I will call it “iotedu-tutorial-bucket” and click continue .
- Keep the location at “multi-region” and leave the rest of the options at the default state and click continue.
- Finally click on the “Create” button.
9. Setting up Google Dataflow
- Let’s now find google Dataflow in the hamburger menu on the left side or in the search bar on top.
- On the menu at the top click on the “Create Job From Template” button.
- Now enter a name for this job that we are creating. Let’s call it “send-to-bq”. Then choose the region that is closest to you geographically. Also, choose “Cloud Pub/Sub Topic to BigQuery” under Cloud Dataflow template. After which, more options will appear.
- Under required parameters set the “topic” in the following format. projects/<project id>/topics/<topic name>
- Under the Big Query output table enter the table in the following format. <project_id>:<dataset_name>.<table_name>
- Finally enter the temporary location in the following format. (Remember the “bucket” that we had created in an earlier step ?) gs://<bucket_name>/tmp/
- Click on the show optional parameters and set the max workers as 2 and also set the machine type to “n1-standard-1”.
- Leave the rest of the options as it is and click “run job”.
- You should now have a running job that moves data from your Pub/Sub topic to your BigQuery table.
Note: Remember to remove all spaces that precede or succeed any of the data that you enter.
10. Setting up the Raspberry Pi
- Let’s first make all the connections required to run the distance sensor according to the diagram below(or any sensor of your choice).
- Create a directory called “cloud” and then “cd” into it.
mkdir cloudcd cloud- Now let’s write a python function to return the data from the distance sensor and save it as “sensordata.py”.
import RPi.GPIO as GPIO
import time
GPIO.setmode(GPIO.BCM)
GPIO_TRIGGER = 18
GPIO_ECHO = 24
GPIO.setwarnings(False)
GPIO.setup(GPIO_TRIGGER, GPIO.OUT)
GPIO.setup(GPIO_ECHO, GPIO.IN)
def FindDistance():
GPIO.output(GPIO_TRIGGER, True)
time.sleep(0.00001)
GPIO.output(GPIO_TRIGGER, False)
StartTime = time.time()
StopTime = time.time()
while GPIO.input(GPIO_ECHO) == 0:
StartTime = time.time()
while GPIO.input(GPIO_ECHO) == 1:
StopTime = time.time()
TimeElapsed = StopTime - StartTime
distance = (TimeElapsed * 34300) / 2
return distance
- Now let’s write the code to send messages to the Pub/Sub topic. Name the file as “sendmsg.py”. Enter your project id as the value of “project_id” variable (remember to enter project id, not the project name but they can be the same).
from google.cloud import pubsub_v1
import datetime
import json
import sensordata
import time
project_id = "<your prject id>" # enter your project id here
topic_name = "<topic name>" # enter the name of the topic that you created
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_name)
futures = dict()
def get_callback(f, data):
def callback(f):
try:
# print(f.result())
futures.pop(data)
except:
print("Please handle {} for {}.".format(f.exception(), data))
return callback
while True:
time.sleep(3)
distance = round(float(sensordata.FindDistance()),2)
timenow = float(time.time())
# timenow = datetime.datetime.now()
data = {"time":timenow, "distance" : distance}
print(data)
# When you publish a message, the client returns a future.
future = publisher.publish(
topic_path, data=(json.dumps(data)).encode("utf-8")) # data must be a bytestring.
# Publish failures shall be handled in the callback function.
future.add_done_callback(get_callback(future, data))
time.sleep(5)
# Wait for all the publish futures to resolve before exiting.
while futures:
time.sleep(5)
print("Published message with error handler.")
- Do you see the dictionary stored in the variable called data ? You can change this dictionary to store whatever data you wish to send to Big Query. However, Remember that the key of the dictionary should match with the name of the field in the Big Query table.
- Now create a virtual environment using the command :
virtualenv venv- activate the virtual environment with the command.
source venv/bin/activate- now lets install our required pip packages.
pip install google-cloud-pubsub
pip install Rpi.GPIO- Remember the credentials JSON file that you downloaded from the GCP website? Move this JSON file to the directory named “cloud”.
- At the moment your working directory (cloud) should look something like this in your Raspberry Pi.
11. Running the Code
- Now you have to set your JSON file as the value of an environment variable called “GOOGLE_APPLICATION_CREDENTIALS” you can do it with the following command :
export GOOGLE_APPLICATION_CREDENTIALS="(Path to JSON file)"Remember this environment variable will only last for a single session. You will have to repeat this step for every session.
- now you can run your python script from the terminal with :
python3 sendmsg.py- Your output should look something like this.
- Your BigQuery table should look something like this :
12. Visualize the data
To visualize the data graphically we will use another google product called data studio. It is an analytics tool that perfectly suits our need to graph sensor data. It also seamlessly links with our BigQuery database. Let’s begin.
- Open the data studio website with the following link : https://datastudio.google.com
- Click on the “Blank Report” button.
- Now click on “Create New Data Source” button in the bottom right corner of your screen.
- Hover your mouse over the big query option and then click “Select”.
- Select the correct project, dataset and table and finally click on the blue “Connect” button on the top right side of the screen.
- We will change the data type of the time field to “Date Hour Minute” and the default aggregation of distance to “average” and click “Add To Report“.
- Then click “Insert” on the top menu and select “Time Series“. We get a chart like this :
- If you look carefully you will notice that the graph is plotting the record count (serial number) instead of the distance so lets choose the “distance” field in the metric category (instead of Record count).
- Once we do that we will get a graph like this :
- You can also change your option from average distance to max, min or many others.
- Here are a few examples :
- Max :
- Min:
There is a lot that you can do with Data Studio, spend some time exploring the different types of graphs and other data types, as this will help you learn more.
Hopefully this tutorial gave a clear insight on how to use pub/sub, BigQuery and Data Studio in harmony with your raspberry pi to easily visualize your sensor data.
