So, You’re getting ready to build the next best IOT Edge Analytics platform but you don’t want to shell out the “big bucks” for a whole lot of IOT sensors. Easy – Build a data generator based on your model and simulate predictable trends.
The following is a framework of a project for you. You should be able to use the sample code on my Github and the following data modeling tool to get up and running in a very short while.
Step 1) Build a Data Model.
Just start building. Its most likely going to change but start down the path and mature the model as your understanding changes. What? We’re building a data model for NoSQL. Yes, If you want to ingest data and do some analytics at some point; you need to start with a model. Here is a Sample of a Pressure Sensor datagram. NoSQL and SQL models change. Lessons learned say: keep things simple or at least manageable.
Step 2) Build a Data Generator
I like to use the right tool for the right job. I know at some point I want to scale my data generator. So, I started with Python. Use a language you’re familiar with and don’t get bogged down in format and/or syntax. Just get it built and start generating data on the command line.
Here is a link to my Github repository (warn: its a work in progress). Take a look in the bin folder and start with dgen_a.py. Note: There are some features there.. A Sin() function to make the data dynamic in a predictable way and a filter to stop negative numbers. There is a timer to control the data rate.
Step 3) Push the Data into a Stream.
If you plan to build analytics on this data then you may want to push this to a stream like Kafka, Redis, Spark Streaming.. etc. There’s some remarked out code and other artifacts for Kafka. In my next post I will show you how to push this to Redis and build and scale on docker. Again, I like Python because I’m able to pull my repositories into a Docker container rather quickly.
Predictable data will mean allot when you start testing a new sensor out in the field. If you’re able to fix all the “plumbing” issues that are sure to arise then you can focus on things like placement, mounting, calibration and much, much more…
If you have questions about your IOT, EDW, AWS or Analytics project, Please Email Us – We’ll will respond ASAP.
Thank you, – Gus Segura
Please Contact US or Subscribe to our Blog if you found this interesting or would like more information.