Study on IoT systems design

Jhayashree Sudha Ramakrishnan
Mind Map by Jhayashree Sudha Ramakrishnan, updated more than 1 year ago More Less
Tomasz Cieplak
Created by Tomasz Cieplak over 3 years ago
Jhayashree Sudha Ramakrishnan
Copied by Jhayashree Sudha Ramakrishnan almost 3 years ago
4
0

Description

Complex knowlage used to design and develope an analytical system for streams analysis

Resource summary

Study on IoT systems design

Annotations:

  • Get prerequiments for the system design. 
  • Kurs: A developer's guide to Exploring and Visualizing IoT Datahttps://www.coursera.org/learn/exploring-visualizing-iot-data
  • Artykuł: IoT 101 – Everything You Need to Know to Start Your IoT Project http://www.datasciencecentral.com/profiles/blogs/iot-101-everything-you-need-to-know-to-start-your-iot-project
  • Koniecznie!!! https://github.com/onurakpolat/awesome-bigdata#data-visualization
  • IoT architecture on the edge http://www.iotcentral.io/blog/the-iot-architecture-at-the-edge?context=category-Platforms
  • Artykuły do projektu związanego z IoT http://www.datasciencecentral.com/profiles/blogs/16-great-iot-articles-published-in-2016
1 Types of data analysis
1.1 Real Time Analysis
1.1.1 Stream Processing
1.1.1.1 Apache Storm
1.1.1.2 Apache Spark Streams

Annotations:

  • Mastering Apache Spark 2 https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/
1.1.1.3 Apache Samza
1.1.1.4 MOA Framework

Annotations:

  • http://moa.cms.waikato.ac.nz/
1.2 Batch Analysis
1.2.1 Apache Hadoop
2 Database Systems
2.1 DBMS

Annotations:

  • Database management system
2.1.1 Apache HBase
2.1.2 Apache CouchDB

Annotations:

  • http://db-engines.com/en/system/Cloudant%3BCouchDB%3BCouchbase
2.1.2.1 IBM Cloudant NoSql
2.1.2.2 Couchbase
2.2 TSDB

Annotations:

  • (Time Series DataBases)
2.2.1 OpenTSDB

Annotations:

  • Data store using HBase
  • Brain Monitoring with Kafka, OpenTSDB, and Grafana http://www.kdnuggets.com/2016/08/brain-monitoring-kafka-opentsb-grafana.html/2
3 File Systems
3.1 HDFS

Annotations:

  • Hadoop File System
4 Messaging Queue Systems
4.1 Apache Kafka

Annotations:

  • Kafka for beginners https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html
  • Znaczenia Kafka dla IoT https://thenewstack.io/apache-kafka-cornerstone-iot-data-platform/
5 Programming Languages
5.1 R Lang
5.1.1 R Open

Annotations:

  • Data visualisation with Shiny https://www.ibm.com/developerworks/community/files/basic/anonymous/api/library/1b6a2624-dc86-4856-b4ed-cdda6bfdecda/document/d26301ed-3155-4e21-ba3d-4936e0bd45cb/media
5.1.2 MS R Server

Annotations:

  • Płatny
5.1.3 Data Science with R

Annotations:

  • Dyplr tricks https://www.r-bloggers.com/lesser-known-dplyr-tricks/
5.1.3.1 Data Import
5.1.3.1.1 Amelia - uzupełnianie danych TS
5.1.3.1.2 tidyr - porządkowanie danych
5.1.3.1.3 readr - ułatwia import danych
5.1.3.1.4 tibble - tab. struktura danych
5.2 Python
5.2.1 Interactive Parallel Computing in Python

Annotations:

  • Interactive Parallel Computing in Python https://ipyparallel.readthedocs.io/ https://github.com/ipython/ipyparallel
5.2.2 Clustering Alghoritms

Annotations:

  • http://www.kdnuggets.com/2017/03/k-means-clustering-algorithms-intro-python.html?utm_content=buffer4b146&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer
5.2.3 Biblioteki - Python
5.2.3.1 EIT
5.2.3.1.1 pyEIT

Annotations:

  • https://github.com/liubenyuan/pyEIT
5.2.3.1.1.1 Calls Diagram

Annotations:

  • C:\Anaconda3\Scripts>python pycallgraph graphviz -o main.png -- D:\tomas\Python\pyEIT\demo\demo_dynamic_greit.py
  • http://pycallgraph.slowchop.com/en/master/guide/intro.html
  • Get readoff of Windows error - http://delimitry.blogspot.com/2015/03/solving-problems-with-pycallgraph-under.html
5.2.3.1.1.2
5.2.3.1.2 Rpi.EIT

Annotations:

  • https://github.com/agungdwiprasetyo/RPi.EIT-algorithms
5.2.3.2 Siatki - Mesh
5.2.3.2.1 MeshPy

Annotations:

  • Site https://mathema.tician.de/software/meshpy/
5.2.3.2.2 Python Triangle

Annotations:

  • Implementacja w Python biblioteki Trianle http://dzhelil.info/triangle/#
  • Biblioteka Triangle: http://www.cs.cmu.edu/~quake/triangle.html
5.2.3.2.3 PyDistMesh

Annotations:

  • https://pypi.python.org/pypi/PyDistMesh
5.2.3.3 Zagad. proste i odwrotne
5.2.3.3.1 pyGimli

Annotations:

  • http://www.pygimli.org/index.html
  • Anaconda install: https://anaconda.org/gimli/pygimli
5.2.3.3.2 Operator Discretization Library

Annotations:

  • https://odlgroup.github.io/odl/getting_started/about_odl.html
5.2.3.3.3 Algorithmic Differentiation in Python

Annotations:

  • http://pythonhosted.org/algopy/
5.2.3.3.4 Linear Inverse Problem
5.2.3.3.4.1 linvpy

Annotations:

  • https://github.com/LCAV/linvpy
5.2.3.3.4.2 pinvprob

Annotations:

  • https://github.com/HajimeKawahara/pinvprob
5.2.3.3.4.3 hIPPYlib

Annotations:

  • https://hippylib.github.io/tutorials/2_PoissonDeterministic/
5.2.3.4 Math Lib.
5.2.3.4.1 Numpy

Annotations:

  • Kurs Numpy http://www.python-course.eu/numpy.php
5.2.3.4.2 SciPy

Annotations:

  • Wykaz oprogramowania naukowego https://www.scipy.org/topical-software.html
  • SciPy dokumentacja https://www.scipy.org/scipylib/index.html
5.2.3.4.2.1 Voronoi Diagrams

Annotations:

  • http://zderadicka.eu/voronoi-diagrams/
  • http://www-astro.physics.ox.ac.uk/~mxc/software/
  • http://dzhelil.info/triangle/examples.html
5.2.3.5 ERT
5.2.3.5.1 BERT

Annotations:

  • Repo https://gitlab.com/resistivity-net/bert/tree/master
5.2.3.5.2 E4D

Annotations:

  • https://e4d.pnnl.gov/Documents/E4D_User_Guide.pdf
5.2.3.6 MISC
5.2.3.6.1 TomograPy

Annotations:

  • Strona projektu http://nbarbey.github.io/TomograPy/
  • Artykuł: https://arxiv.org/pdf/1103.5904.pdf
5.2.3.6.2 Medical Data Manipulation

Annotations:

  • https://idoimaging.com/programs?q%5Bdisplay_function%5D=&q%5Bfor_audience%5D=&q%5Bfunction%5D=&q%5Bheader_function%5D=&q%5Binterface%5D=&q%5Blanguage%5D=19&q%5Bname_cont%5D=&q%5Bnetwork_function%5D=&q%5Bother_function%5D=&q%5Bplatform%5D=&q%5Bprogramming_function%5D=&q%5Bread_format%5D=&q%5Bspeciality%5D=&q%5Bsummary_cont%5D=&q%5Bwrite_format%5D=
5.2.3.7 Data Flow
5.2.3.7.1 Apache AirFlow

Annotations:

  • Tylko Linux http://airflow.incubator.apache.org/start.html
5.2.4 Intel Python

Annotations:

  • Installation: https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda
5.3 Scala

Annotations:

  • Development and deployment of Spark applications with Scala, Eclipse, and sbt – Part 1: Installation & configuration http://www.nodalpoint.com/development-and-deployment-of-spark-applications-with-scala-eclipse-and-sbt-part-1-installation-configuration/
  • Visualizing Big Data with Spark and Scala http://blog.ibmjstart.net/2016/07/18/visualizing-big-data-spark-scala/
5.3.1 SBT
5.3.2 Scala IDE

Annotations:

  • http://scala-ide.org/download/sdk.html
5.4 Java
5.5 Flow Base Programming

Annotations:

  • Used in IBM Streams, Apache NiFi
  • http://www.jpaulmorrison.com/fbp/
  • Apache Beam (Pyton SDK - https://github.com/apache/beam/tree/master/sdks/python#install-virtualenv) Google Cloud DataFlow - środowisko uruchomieniowe dla Apache Beamhttps://cloud.google.com/dataflow/  
  • Opis modelu programistycznego https://beam.apache.org/documentation/programming-guide/   
5.6 Programmer IDE
5.6.1 Eclipse
5.6.1.1 Maven
5.6.2 Zeppelin Notebook

Annotations:

  •  Zbiór przykładów analizy danych do Zeppelina https://github.com/hortonworks-gallery/zeppelin-notebooks   
5.6.3 Jupiter Notebook

Annotations:

  • Data export to HTML https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/
  • Visualisation PixieDust: Magic for Your Python Notebook https://developer.ibm.com/clouddataservices/2016/10/11/pixiedust-magic-for-python-notebook/
5.6.4 R Studio
5.6.5 Anaconda
6 Implementation
6.1 VM

Annotations:

  • Virtualization
6.1.1 Hyper-V
6.1.2 Docker

Annotations:

  • Wdrożenie Zeppelin https://github.com/dylanmei/docker-zeppelin
  • Wdrożenie Apache Spark https://github.com/gettyimages/docker-spark
6.1.3 VMware
6.1.4 VirtualBox
6.2 Cloud Computing
6.2.1 Azure HDInsight
6.2.2 IBM Bluemix IoT

Annotations:

  • Koniecznie!!! Spark Streaming + Watson IoT Platform Integration https://developer.ibm.com/recipes/tutorials/spark-streaming-ibm-watson-iot-platform-integration/
  • Koniecznie: Engage Machine Learning for detecting anomalous behaviors of things https://developer.ibm.com/recipes/tutorials/engage-machine-learning-for-detecting-anomalous-behaviors-of-things/
  • Koniecznie: Timeseries Data Analysis of IoT events by using Jupyter Notebook  https://developer.ibm.com/recipes/tutorials/timeseries-data-analysis-of-iot-events-by-using-jupyter-notebook/
  • Koniecznie!!! Integrating Watson IoT Platform with Message Hub and Apache Spark  https://developer.ibm.com/recipes/tutorials/integrating-watson-iot-platform-with-message-hubkafka/
  • Koniecznie!!! Integracja DB i Spark https://developer.ibm.com/clouddataservices/docs/ibm-data-science-experience/docs/
  • Cleanse and visualize location data using Spark and Bluemix https://www.ibm.com/developerworks/library/ba-1610location-data-spark-bluemix-trs/index.html
  • Apache Spark - using notebooks https://console.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index-gentopic3.html#genTopProcId4
  • !!! SparkR notebook with Cloudant https://github.com/charles2588/bluemixsparknotebooks/blob/master/R/SparkR_Cloudant_Connection.ipynb
6.2.3 AWS
6.2.4 Databricks
6.2.5 Google Cloud Platform

Annotations:

  • Real-Time Stream Processing for IoT: https://cloud.google.com/solutions/architecture/real-time-stream-processing-iot
  • Cena na 1 Rok GPU in Google: https://cloud.google.com/products/calculator/#id=99152826-9096-43b2-a100-836c36ef4d50
6.3 Bare Metal
7 Integrated Platforms
7.1 Hortonworks
7.1.1 Data Platform
7.1.2 Data Flow
7.2 Cloudera

Annotations:

  • Cloudera StreamSets https://vimeo.com/147671735
7.3 IBM Open Platform
8 Data Integration
8.1 Apache NiFi
9 Messaging Protocols
9.1 MQTT

Annotations:

  • !!! iot-device-samples https://github.com/ibm-messaging/iot-device-samples
10 Programming Methods
10.1 Img. Reconstr. Algorithms
10.1.1 Tensor-based

Annotations:

  • Artykuł: http://iopscience.iop.org/article/10.1088/1361-6501/aa58a3/meta
10.1.2 Simulated Annealing Particle Swarm Optimization
10.1.3 Landweber iterative method
10.1.4 Linear back projection (LBP)
10.1.5 Singular value decomposition

Annotations:

  • Implementacja Python: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.svd.html
10.1.6 Conjugate Gradient method
10.1.7 Maximum Likelihood Estimation Method

Annotations:

  • https://github.com/clarehchao/Spark3DImageReconstruction
10.1.8 Inverse problems

Annotations:

  • Introduction to inverse problems (Spring 2014) http://cc.oulu.fi/~smaenpaa/inv_pk/inv_pk14_eng.pdf
10.1.8.1 Tikhonov regularization method

Annotations:

  • Implementacja Python https://pypi.python.org/pypi/InverseProblem/1.0
10.1.8.2 Statistical and Computational InverseProblems

Annotations:

  • https://www.researchgate.net/profile/Ian_Langmore/publication/268250507_Statistical_and_Computational_Inverse_Problems_and_a_little_uncertainty_quantification/links/5654822708aeafc2aabbd811/Statistical-and-Computational-Inverse-Problems-and-a-little-uncertainty-quantification.pdf?origin=publication_detail&ev=pub_int_prw_xdl&msrp=iG4X9V_68LOGbkxNrcZEP4Az6qj7tC0tBRcUQyjL8QYqQT4QoLmsF2FgkN6bhYKWGD5hIwTkn5ZA4aHZ8A8286ns9G-BWrYTAsuNSMv5CNk.bYd_LZtB1khHvYwW5OyIFeUTFjZCQqyRJayOmj_N5AXLMsdZGfQieOSGqKg4N2g4hxqe2aS8SGqs7RTj0lO2mw.wfseq3OvlHxPmYqbEhxOxL040D2b4044fCPINZ0nDdsCbRE9GxeAhwoYCzzzOf64AwlcmWuBBIbKFhDQjoECqQ.jNjix2pbRzy-BWR2vabpp4TbUNEHGKEiBIwCb6q11EYbYkRGdAy-ev4pUz1krhlO9uaouT9_yrSr7F-4hJheJQ
10.1.8.3 Lectures

Annotations:

  • Lecture1: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture1 Lecture2: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture2 Lecture3&4: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture3 Lecture5: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture5
10.2 Machine Intelligence
10.2.1 Neural Networks
10.2.2 Machine Learning

Annotations:

  • A Visual Introduction to Machine Learning http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
10.2.2.1 Apache Spark MLlib
10.2.3 Deep Learning

Annotations:

  • Deeplearning4j on Spark https://deeplearning4j.org/
  • TensorFlow http://www.tensorflow.org
  • An Overview of Python Deep Learning Frameworks http://www.kdnuggets.com/2017/02/python-deep-learning-frameworks-overview.html
  • TensorFlow on Sparkhttps://github.com/yahoo/TensorFlowOnSpark
10.2.3.1 Google TensorFlow

Annotations:

  • Porównanie TensorFlow i CNTK https://esciencegroup.com/2016/02/08/tensorflow-meets-microsofts-cntk/
10.2.3.1.1 TensorBoard: Visualizing Learning

Annotations:

  • (C:\Anaconda3\envs\TensorFlow) C:\Users\tom\Documents>python -m tensorflow.tensorboard --logdir=./ localhost:6006
10.2.3.2 Microsoft Cognitive Toolkit (CNTK)

Annotations:

  • Strona projektu: https://www.microsoft.com/en-us/research/product/cognitive-toolkit/
  • Github: https://github.com/Microsoft/CNTK
  • Wiki: https://github.com/Microsoft/CNTK/wiki
11 Approaches to reconstruction acceleration

Annotations:

  • Artykuł: Review of Parallel Computing Techniques for Computed Tomography Image Reconstruction http://www.imaging.sbes.vt.edu/BIDLib/CT/NiLiWang_ParaCompTech.pdf
  • Rozprawa Doktorska dot. CT: http://ir.uiowa.edu/cgi/viewcontent.cgi?article=2330&context=etd
11.1 Parallel Computing
11.1.1 Distribiuted Computing
11.1.1.1 Apache Spark
11.1.1.1.1 SparkR Desc.

Annotations:

  • Opis biblioteki SparkR https://spark.apache.org/docs/2.1.0/api/R/index.html
11.1.1.1.2 DataFrame
11.1.1.1.2.1 Operations

Annotations:

  • Operacje na obiekcie SparkDataFrame https://www.analyticsvidhya.com/blog/2016/10/spark-dataframe-and-operations/
11.1.2 Local PC Clusters
11.2 GPU

Annotations:

  • Artykuł: P.Kapusta http://annals-csis.org/Volume_8/pliks/344.pdf
11.3 Improved Algorithms
12 Data Raporting
12.1 Web App

Annotations:

  • https://www.slideshare.net/BenLaird/real-time-data-viz-with-spark-streaming-kafka-and-d3js
12.1.1 D3.js

Annotations:

  • https://github.com/d3
12.1.1.1 DC.js

Annotations:

  • https://dc-js.github.io/dc.js/
12.1.2 Socket.io

Annotations:

  • https://github.com/socketio/socket.io
12.1.3 R Shiny

Annotations:

  • Tutorial: new ver. http://shiny.rstudio.com/tutorial/ old http://rstudio.github.io/shiny/tutorial/
12.1.4 Rickshaw TS graphs.

Annotations:

  • Rickshaw is a JavaScript toolkit for creating interactive time series graphs. http://code.shutterstock.com/rickshaw/
12.1.4.1 RealTime Visualization IBM

Annotations:

  • https://developer.ibm.com/recipes/tutorials/visualizing-your-data/
13 Medical Data Analysis
13.1 MIRAN Project

Annotations:

  • Projekt analizy danych medycznych za pomocą Python http://ne-scientific.com/miran/ http://ne-scientific.com/miran/
Show full summary Hide full summary

Similar

Study on IoT systems design
Nikola Tomoff
Python Quiz
karljmurphy
Think Python
tsilvo2001
Basic Python - Print Formatting
Rebecca Noel
What is Python?
Daniel Ingram
What is Python?
Leena Varghese
Python
54671
Know your Python!
educ8ict
Basic Python - Strings
Rebecca Noel
Python
Kirstie Wu
Basic Python - Lists
Rebecca Noel