IOT, DynamoDB, Data Generator and You.

IOT Data Generator

Facebooktwittergoogle_plusredditpinterestlinkedintumblr

Simulate IOT Devices with a Data Generator

There are many times when you will want to simulate the data being generated from your IOT device in the field.
This post is the 2nd in a series focused on IOT Data Generation. I’ve spend time lately in DynamoDB and I have to
say its a great product and an easy way to connect an IOT Device by using a micro-services architecture like Docker
and a simple Python Service -or- MQTT. Later this month, I’ll show you how to build a Docker container with Python
and my IOT Data Generator. This is a great way to scale and simulate 1000’s of devices. Please check back.

Top 5 Reasons to use a Data Generator

1) You don’t have the device yet or have it installed.
My device is on back-order, Its not out yet, “How do I install this?”

2) You are still building the connecting architecture (i.e. Plumbing).
You want to see something working. One of the first things we do in Architecture is a end-to-end test.

3) The field device is not producing the desired or expected results.
You’ll get this allot when working with a new device. Understanding capacity, metrics, config..etc.

4) You want to control the way another device reacts to the data coming from an Up Stream Device.
A bit more on this, Depending on your use-case; you may have several IOT devices that work together
in Concert. Some will be up-stream of process and other down-stream.

Example: You have a Device that measures position and another that measures speed. These devices
participate in a feedback loop and you need to simulate a event state (decision point).  i.e. Self-Driving cars
collect allot of data. See Google SDC.

5) You are building an Analytics or Machine Learning Solution around Data Anomaly.
For me, This is a big one. We have to build predictive models based on IOT device data. There are
many times when we simulate device state and the device in the field is no where near that state;
So, We have no choice but to simulate.  FYI, Then we work back-wards to create an environment that
will trigger the desired state. This should be the Anomaly State. More on that in a later post.

 

Generate Machine – Person Data to DynamoDB

I’ve recently updated my Github repository: iot_kafka_datagen.
There are three new scripts in the bin directory: sensor_ddb.py, dgen_person_ddb.py, dgen_mach_ddb.py

There are simple instructions on how to run the code in the README. Summary, You will need to have an AWS account
if you want to create a DynamoDB Endpoint. You may also down-load the stand-alone DynamoDB to run locally. You’ll
need python and a good editor.

Note:  You need to edit the scripts and set to your DynamoDB endpoint.  i.e. [ line 9 in sensor_ddb.py].

dynamodb = boto3.resource('dynamodb', region_name='us-west-2', endpoint_url="http://localhost:8000")

Then, You’ll have to run the table build script to create the DDB Table.

host:bin user$ python sensor_ddb.py
Table status: ACTIVE -- This means the table was created.

You can use this AWS DynamoDB Tutorial  to learn how to validate and create a table manually.

Then, Run either  dgen_person_ddb.py  -or- dgen_mach_ddb.py to start generating data. Watch Out, Its Fast!

host:bin user$ python dgen_person_ddb.py 
{"timestamp": "2016-06-03T18:04:25", "timezone": "-0500", "millis": "067", "sensor": "sensor107", "senstype": "currentRespiration", "metric": 18.0}
{"timestamp": "2016-06-03T18:04:25", "timezone": "-0500", "millis": "109", "sensor": "sensor105", "senstype": "currentBodyTemp", "metric": 99.86220647721184}
{"timestamp": "2016-06-03T18:04:25", "timezone": "-0500", "millis": "140", "sensor": "sensor106", "senstype": "currentHeartRate", "metric": 77.72789228047705}
{"timestamp": "2016-06-03T18:04:25", "timezone": "-0500", "millis": "175", "sensor": "sensor106", "senstype": "currentHeartRate", "metric": 78.15125230465665}
{"timestamp": "2016-06-03T18:04:25", "timezone": "-0500", "millis": "217", "sensor": "sensor106", "senstype": "currentHeartRate", "metric": 75.88084481873287}
{"timestamp": "2016-06-03T18:04:25", "timezone": "-0500", "millis": "332", "sensor": "sensor105", "senstype": "currentBodyTemp", "metric": 98.6}
...

 

Parting Thoughts…

Note: The /bin folder [of my Git Repository] has scripts for generating data to kafka, redis and now DynamoDB. For the most part, They are simple and easy to follow. They should be built Object Oriented, I’ll make sure to clean them up moving forward.

Continuing on the AWS Trek; We like the scalability of DDB, rapid prototyping, speed. We’ll be exploring the next steps of importing IOT ddb data into AWS Redshift.  You’ll see us switching back and forth between Cloud and On-Premise in our solutions. Our reasoning is there are allot of folks using Hybrid – Architecture and it nice to see it both ways.

If you have questions about your IOT, EDW, AWS or Analytics project, Please Email Us – We’ll will respond ASAP.
Thank you, – Gus Segura

Please Contact US or Subscribe to our Blog if you found this interesting or would like more information.

Subscribe : Blueskymetrics Blog

* indicates required,  Managed By Mail-chimp – Please check your Spam Folder and Confirm Subscription.