Mapa mental

Study on IoT systems design

Tomasz Cieplak IoT Project Inglês 5,0Avaliação média 5,0 de 5 (1 avaliação)

Conteúdo do mapa

Get prerequiments for the system design. 
Kurs: A developer's guide to Exploring and Visualizing IoT Datahttps://www.coursera.org/learn/exploring-visualizing-iot-data
Artykuł: IoT 101 – Everything You Need to Know to Start Your IoT Project http://www.datasciencecentral.com/profiles/blogs/iot-101-everything-you-need-to-know-to-start-your-iot-project
Koniecznie!!! https://github.com/onurakpolat/awesome-bigdata#data-visualization
IoT architecture on the edge http://www.iotcentral.io/blog/the-iot-architecture-at-the-edge?context=category-Platforms
Artykuły do projektu związanego z IoT http://www.datasciencecentral.com/profiles/blogs/16-great-iot-articles-published-in-2016

Types of data analysis
- Real Time Analysis
  - Stream Processing
    - Apache Storm
    - Apache Spark Streams
      - Mastering Apache Spark 2 https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/
    - Apache Samza
    - MOA Framework
      - http://moa.cms.waikato.ac.nz/
- Batch Analysis
  - Apache Hadoop
Database Systems
- DBMS
  - Database management system
  - Apache HBase
  - Apache CouchDB
    - http://db-engines.com/en/system/Cloudant%3BCouchDB%3BCouchbase
    - IBM Cloudant NoSql
    - Couchbase
- TSDB
  - (Time Series DataBases)
  - OpenTSDB
    - Data store using HBase
    - Brain Monitoring with Kafka, OpenTSDB, and Grafana http://www.kdnuggets.com/2016/08/brain-monitoring-kafka-opentsb-grafana.html/2
File Systems
- HDFS
  - Hadoop File System
Messaging Queue Systems
- Apache Kafka
  - Kafka for beginners https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html
  - Znaczenia Kafka dla IoT https://thenewstack.io/apache-kafka-cornerstone-iot-data-platform/
Programming Languages
- R Lang
  - R Open
    - Data visualisation with Shiny https://www.ibm.com/developerworks/community/files/basic/anonymous/api/library/1b6a2624-dc86-4856-b4ed-cdda6bfdecda/document/d26301ed-3155-4e21-ba3d-4936e0bd45cb/media
  - MS R Server
    - Płatny
  - Data Science with R
    - Dyplr tricks https://www.r-bloggers.com/lesser-known-dplyr-tricks/
    - Data Import
      - Amelia - uzupełnianie danych TS
      - tidyr - porządkowanie danych
      - readr - ułatwia import danych
      - tibble - tab. struktura danych
- Python
  - Interactive Parallel Computing in Python
    - Interactive Parallel Computing in Python https://ipyparallel.readthedocs.io/ https://github.com/ipython/ipyparallel
  - Clustering Alghoritms
    - http://www.kdnuggets.com/2017/03/k-means-clustering-algorithms-intro-python.html?utm_content=buffer4b146&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer
  - Biblioteki - Python
    - EIT
      - pyEIT
        https://github.com/liubenyuan/pyEIT
        Calls Diagram
        C:\Anaconda3\Scripts>python pycallgraph graphviz -o main.png -- D:\tomas\Python\pyEIT\demo\demo_dynamic_greit.py
        http://pycallgraph.slowchop.com/en/master/guide/intro.html
        Get readoff of Windows error - http://delimitry.blogspot.com/2015/03/solving-problems-with-pycallgraph-under.html
      - Rpi.EIT
        https://github.com/agungdwiprasetyo/RPi.EIT-algorithms
    - Siatki - Mesh
      - MeshPy
        Site https://mathema.tician.de/software/meshpy/
      - Python Triangle
        Implementacja w Python biblioteki Trianle http://dzhelil.info/triangle/#
        Biblioteka Triangle: http://www.cs.cmu.edu/~quake/triangle.html
      - PyDistMesh
        https://pypi.python.org/pypi/PyDistMesh
    - Zagad. proste i odwrotne
      - pyGimli
        http://www.pygimli.org/index.html
        Anaconda install: https://anaconda.org/gimli/pygimli
      - Operator Discretization Library
        https://odlgroup.github.io/odl/getting_started/about_odl.html
      - Algorithmic Differentiation in Python
        http://pythonhosted.org/algopy/
      - Linear Inverse Problem
        linvpy
        https://github.com/LCAV/linvpy
        pinvprob
        https://github.com/HajimeKawahara/pinvprob
        hIPPYlib
        https://hippylib.github.io/tutorials/2_PoissonDeterministic/
    - Math Lib.
      - Numpy
        Kurs Numpy http://www.python-course.eu/numpy.php
      - SciPy
        Wykaz oprogramowania naukowego https://www.scipy.org/topical-software.html
        SciPy dokumentacja https://www.scipy.org/scipylib/index.html
        Voronoi Diagrams
        http://zderadicka.eu/voronoi-diagrams/
        http://www-astro.physics.ox.ac.uk/~mxc/software/
        http://dzhelil.info/triangle/examples.html
    - ERT
      - BERT
        Repo https://gitlab.com/resistivity-net/bert/tree/master
      - E4D
        https://e4d.pnnl.gov/Documents/E4D_User_Guide.pdf
    - MISC
      - TomograPy
        Strona projektu http://nbarbey.github.io/TomograPy/
        Artykuł: https://arxiv.org/pdf/1103.5904.pdf
      - Medical Data Manipulation
        https://idoimaging.com/programs?q%5Bdisplay_function%5D=&q%5Bfor_audience%5D=&q%5Bfunction%5D=&q%5Bheader_function%5D=&q%5Binterface%5D=&q%5Blanguage%5D=19&q%5Bname_cont%5D=&q%5Bnetwork_function%5D=&q%5Bother_function%5D=&q%5Bplatform%5D=&q%5Bprogramming_function%5D=&q%5Bread_format%5D=&q%5Bspeciality%5D=&q%5Bsummary_cont%5D=&q%5Bwrite_format%5D=
    - Data Flow
      - Apache AirFlow
        Tylko Linux http://airflow.incubator.apache.org/start.html
    - Visualization
      - VTK
        http://www.vtk.org/Wiki/VTK/Examples/Python
      - Matplotlib
  - Intel Python
    - Installation: https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda
  - Python (x,y)
    - https://python-xy.github.io/
- Scala
  - Development and deployment of Spark applications with Scala, Eclipse, and sbt – Part 1: Installation & configuration http://www.nodalpoint.com/development-and-deployment-of-spark-applications-with-scala-eclipse-and-sbt-part-1-installation-configuration/
  - Visualizing Big Data with Spark and Scala http://blog.ibmjstart.net/2016/07/18/visualizing-big-data-spark-scala/
  - SBT
  - Scala IDE
    - http://scala-ide.org/download/sdk.html
- Java
- Flow Base Programming
  - Used in IBM Streams, Apache NiFi
  - http://www.jpaulmorrison.com/fbp/
  - Apache Beam (Pyton SDK - https://github.com/apache/beam/tree/master/sdks/python#install-virtualenv) Google Cloud DataFlow - środowisko uruchomieniowe dla Apache Beamhttps://cloud.google.com/dataflow/  
  - Opis modelu programistycznego https://beam.apache.org/documentation/programming-guide/   
- Programmer IDE
  - Eclipse
    - Maven
  - Zeppelin Notebook
    -  Zbiór przykładów analizy danych do Zeppelina https://github.com/hortonworks-gallery/zeppelin-notebooks   
  - Jupiter Notebook
    - Data export to HTML https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/
    - Visualisation PixieDust: Magic for Your Python Notebook https://developer.ibm.com/clouddataservices/2016/10/11/pixiedust-magic-for-python-notebook/
    - http://notebooks.azure.com
    - https://nbviewer.jupyter.org/
  - R Studio
  - Anaconda
  - VisualStudio PTVS
    - https://github.com/Microsoft/PTVS/
Implementation
- VM
  - Virtualization
  - Hyper-V
  - Docker
    - Wdrożenie Zeppelin https://github.com/dylanmei/docker-zeppelin
    - Wdrożenie Apache Spark https://github.com/gettyimages/docker-spark
  - VMware
  - VirtualBox
- Cloud Computing
  - Azure HDInsight
  - IBM Bluemix IoT
    - Koniecznie!!! Spark Streaming + Watson IoT Platform Integration https://developer.ibm.com/recipes/tutorials/spark-streaming-ibm-watson-iot-platform-integration/
    - Koniecznie: Engage Machine Learning for detecting anomalous behaviors of things https://developer.ibm.com/recipes/tutorials/engage-machine-learning-for-detecting-anomalous-behaviors-of-things/
    - Koniecznie: Timeseries Data Analysis of IoT events by using Jupyter Notebook  https://developer.ibm.com/recipes/tutorials/timeseries-data-analysis-of-iot-events-by-using-jupyter-notebook/
    - Koniecznie!!! Integrating Watson IoT Platform with Message Hub and Apache Spark  https://developer.ibm.com/recipes/tutorials/integrating-watson-iot-platform-with-message-hubkafka/
    - Koniecznie!!! Integracja DB i Spark https://developer.ibm.com/clouddataservices/docs/ibm-data-science-experience/docs/
    - Cleanse and visualize location data using Spark and Bluemix https://www.ibm.com/developerworks/library/ba-1610location-data-spark-bluemix-trs/index.html
    - Apache Spark - using notebooks https://console.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index-gentopic3.html#genTopProcId4
    - !!! SparkR notebook with Cloudant https://github.com/charles2588/bluemixsparknotebooks/blob/master/R/SparkR_Cloudant_Connection.ipynb
  - AWS
  - Databricks
  - Google Cloud Platform
    - Real-Time Stream Processing for IoT: https://cloud.google.com/solutions/architecture/real-time-stream-processing-iot
    - Cena na 1 Rok GPU in Google: https://cloud.google.com/products/calculator/#id=99152826-9096-43b2-a100-836c36ef4d50
- Bare Metal
Integrated Platforms
- Hortonworks
  - Data Platform
  - Data Flow
- Cloudera
  - Cloudera StreamSets https://vimeo.com/147671735
- IBM Open Platform
Data Integration
- Apache NiFi
Messaging Protocols
- MQTT
  - !!! iot-device-samples https://github.com/ibm-messaging/iot-device-samples
Programming Methods
- Img. Reconstr. Algorithms
  - Tensor-based
    - Artykuł: http://iopscience.iop.org/article/10.1088/1361-6501/aa58a3/meta
  - Simulated Annealing Particle Swarm Optimization
  - Landweber iterative method
  - Linear back projection (LBP)
  - Singular value decomposition
    - Implementacja Python: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.svd.html
  - Conjugate Gradient method
  - Maximum Likelihood Estimation Method
    - https://github.com/clarehchao/Spark3DImageReconstruction
  - Inverse problems
    - Introduction to inverse problems (Spring 2014) http://cc.oulu.fi/~smaenpaa/inv_pk/inv_pk14_eng.pdf
    - Tikhonov regularization method
      - Implementacja Python https://pypi.python.org/pypi/InverseProblem/1.0
    - Statistical and Computational InverseProblems
      - https://www.researchgate.net/profile/Ian_Langmore/publication/268250507_Statistical_and_Computational_Inverse_Problems_and_a_little_uncertainty_quantification/links/5654822708aeafc2aabbd811/Statistical-and-Computational-Inverse-Problems-and-a-little-uncertainty-quantification.pdf?origin=publication_detail&ev=pub_int_prw_xdl&msrp=iG4X9V_68LOGbkxNrcZEP4Az6qj7tC0tBRcUQyjL8QYqQT4QoLmsF2FgkN6bhYKWGD5hIwTkn5ZA4aHZ8A8286ns9G-BWrYTAsuNSMv5CNk.bYd_LZtB1khHvYwW5OyIFeUTFjZCQqyRJayOmj_N5AXLMsdZGfQieOSGqKg4N2g4hxqe2aS8SGqs7RTj0lO2mw.wfseq3OvlHxPmYqbEhxOxL040D2b4044fCPINZ0nDdsCbRE9GxeAhwoYCzzzOf64AwlcmWuBBIbKFhDQjoECqQ.jNjix2pbRzy-BWR2vabpp4TbUNEHGKEiBIwCb6q11EYbYkRGdAy-ev4pUz1krhlO9uaouT9_yrSr7F-4hJheJQ
    - Lectures
      - Lecture1: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture1 Lecture2: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture2 Lecture3&4: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture3 Lecture5: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture5
- Machine Intelligence
  - Neural Networks
  - Machine Learning
    - A Visual Introduction to Machine Learning http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
    - Apache Spark MLlib
  - Deep Learning
    - Deeplearning4j on Spark https://deeplearning4j.org/
    - TensorFlow http://www.tensorflow.org
    - An Overview of Python Deep Learning Frameworks http://www.kdnuggets.com/2017/02/python-deep-learning-frameworks-overview.html
    - TensorFlow on Sparkhttps://github.com/yahoo/TensorFlowOnSpark
    - Google TensorFlow
      - Porównanie TensorFlow i CNTK https://esciencegroup.com/2016/02/08/tensorflow-meets-microsofts-cntk/
      - TensorBoard: Visualizing Learning
        (C:\Anaconda3\envs\TensorFlow) C:\Users\tom\Documents>python -m tensorflow.tensorboard --logdir=./ localhost:6006
    - Microsoft Cognitive Toolkit (CNTK)
      - Strona projektu: https://www.microsoft.com/en-us/research/product/cognitive-toolkit/
      - Github: https://github.com/Microsoft/CNTK
      - Wiki: https://github.com/Microsoft/CNTK/wiki
Approaches to reconstruction acceleration
- Artykuł: Review of Parallel Computing Techniques for Computed Tomography Image Reconstruction http://www.imaging.sbes.vt.edu/BIDLib/CT/NiLiWang_ParaCompTech.pdf
- Rozprawa Doktorska dot. CT: http://ir.uiowa.edu/cgi/viewcontent.cgi?article=2330&context=etd
- Parallel Computing
  - Distribiuted Computing
    - Apache Spark
      - SparkR Desc.
        Opis biblioteki SparkR https://spark.apache.org/docs/2.1.0/api/R/index.html
      - DataFrame
        Operations
        Operacje na obiekcie SparkDataFrame https://www.analyticsvidhya.com/blog/2016/10/spark-dataframe-and-operations/
  - Local PC Clusters
- GPU
  - Artykuł: P.Kapusta http://annals-csis.org/Volume_8/pliks/344.pdf
- Improved Algorithms
Data Visualization
- Web App
  - https://www.slideshare.net/BenLaird/real-time-data-viz-with-spark-streaming-kafka-and-d3js
  - D3.js
    - https://github.com/d3
    - DC.js
      - https://dc-js.github.io/dc.js/
  - Socket.io
    - https://github.com/socketio/socket.io
  - R Shiny
    - Tutorial: new ver. http://shiny.rstudio.com/tutorial/ old http://rstudio.github.io/shiny/tutorial/
  - Rickshaw TS graphs.
    - Rickshaw is a JavaScript toolkit for creating interactive time series graphs. http://code.shutterstock.com/rickshaw/
    - RealTime Visualization IBM
      - https://developer.ibm.com/recipes/tutorials/visualizing-your-data/
  - Bokeh
    - https://bokeh.pydata.org/en/latest/
  - MPLD3
    - http://mpld3.github.io/
Medical Data Analysis
- MIRAN Project
  - Projekt analizy danych medycznych za pomocą Python http://ne-scientific.com/miran/ http://ne-scientific.com/miran/
Microservices
- https://youtu.be/-zsKY9p_5R8?t=34m2s https://github.com/miguelgrinberg/microflack_admin
- https://youtu.be/nrzLdMWTRMM
- https://12factor.net/
- Loadbalancer
  - Windows
    - Nginx
      - https://nginx.org/en/download.html
  - Linux
    - HAProxy
      - http://www.haproxy.org/
- Design patterns
  - API Gateway
    - http://microservices.io/patterns/apigateway.html
  - Service Registry
    - http://microservices.io/patterns/service-registry.html
- Service Registry
  - Consul
    - https://www.consul.io/intro/index.html
  - etcd
    - https://github.com/coreos/etcd
- Containers
  - Docker
- Service-to-Service Communication
- Logging
  - Logspout
    - https://github.com/gliderlabs/logspout

Informações

Descrição: Complex knowlage used to design and develope an analytical system for streams analysis
Criado por: Tomasz Cieplak
Visualizações: 208
Tags: iot bigdata apache hadoop hortonworks data platform hortonworks data flow docker virtualization streams processing real time data processing apache hbase tsdb apache nifi apache ambari scala python
Avaliação: Avaliação média 5,0 de 5 (1 avaliação)

Study on IoT systems design

Conteúdo do mapa

Informações

Fale Conosco

O que você acha do novo visual?