GoConqr hat ein neues Design! Sag uns, wie es dir gefällt.

Kostenlos registrieren

Mindmap

Study on IoT systems design

Tomasz Cieplak IoT Project Englisch 5,0Durchschnittliche Bewertung 5,0 von 5 (1 Bewertung)

Inhalt der Mindmap

Study on IoT systems design

  • Get prerequiments for the system design. 
  • Kurs: A developer's guide to Exploring and Visualizing IoT Datahttps://www.coursera.org/learn/exploring-visualizing-iot-data
  • Artykuł: IoT 101 – Everything You Need to Know to Start Your IoT Project http://www.datasciencecentral.com/profiles/blogs/iot-101-everything-you-need-to-know-to-start-your-iot-project
  • Koniecznie!!! https://github.com/onurakpolat/awesome-bigdata#data-visualization
  • IoT architecture on the edge http://www.iotcentral.io/blog/the-iot-architecture-at-the-edge?context=category-Platforms
  • Artykuły do projektu związanego z IoT http://www.datasciencecentral.com/profiles/blogs/16-great-iot-articles-published-in-2016

  1. Types of data analysis

    • Real Time Analysis
      • Stream Processing
        • Apache Storm
        • Apache Spark Streams
          • Mastering Apache Spark 2 https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/
        • Apache Samza
        • MOA Framework
          • http://moa.cms.waikato.ac.nz/
    • Batch Analysis
      • Apache Hadoop

  2. Database Systems

    • DBMS
      • Database management system
      • Apache HBase
      • Apache CouchDB
        • http://db-engines.com/en/system/Cloudant%3BCouchDB%3BCouchbase
        • IBM Cloudant NoSql
        • Couchbase
    • TSDB
      • (Time Series DataBases)
      • OpenTSDB
        • Data store using HBase
        • Brain Monitoring with Kafka, OpenTSDB, and Grafana http://www.kdnuggets.com/2016/08/brain-monitoring-kafka-opentsb-grafana.html/2

  3. File Systems

    • HDFS
      • Hadoop File System

  4. Messaging Queue Systems

    • Apache Kafka
      • Kafka for beginners https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html
      • Znaczenia Kafka dla IoT https://thenewstack.io/apache-kafka-cornerstone-iot-data-platform/

  5. Programming Languages

    • R Lang
      • R Open
        • Data visualisation with Shiny https://www.ibm.com/developerworks/community/files/basic/anonymous/api/library/1b6a2624-dc86-4856-b4ed-cdda6bfdecda/document/d26301ed-3155-4e21-ba3d-4936e0bd45cb/media
      • MS R Server
        • Płatny
      • Data Science with R
        • Dyplr tricks https://www.r-bloggers.com/lesser-known-dplyr-tricks/
        • Data Import
          • Amelia - uzupełnianie danych TS
          • tidyr - porządkowanie danych
          • readr - ułatwia import danych
          • tibble - tab. struktura danych
    • Python
      • Interactive Parallel Computing in Python
        • Interactive Parallel Computing in Python https://ipyparallel.readthedocs.io/ https://github.com/ipython/ipyparallel
      • Clustering Alghoritms
        • http://www.kdnuggets.com/2017/03/k-means-clustering-algorithms-intro-python.html?utm_content=buffer4b146&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer
      • Biblioteki - Python
        • EIT
          • pyEIT
            • https://github.com/liubenyuan/pyEIT
            • Calls Diagram
              • C:\Anaconda3\Scripts>python pycallgraph graphviz -o main.png -- D:\tomas\Python\pyEIT\demo\demo_dynamic_greit.py
              • http://pycallgraph.slowchop.com/en/master/guide/intro.html
              • Get readoff of Windows error - http://delimitry.blogspot.com/2015/03/solving-problems-with-pycallgraph-under.html
          • Rpi.EIT
            • https://github.com/agungdwiprasetyo/RPi.EIT-algorithms
        • Siatki - Mesh
          • MeshPy
            • Site https://mathema.tician.de/software/meshpy/
          • Python Triangle
            • Implementacja w Python biblioteki Trianle http://dzhelil.info/triangle/#
            • Biblioteka Triangle: http://www.cs.cmu.edu/~quake/triangle.html
          • PyDistMesh
            • https://pypi.python.org/pypi/PyDistMesh
        • Zagad. proste i odwrotne
          • pyGimli
            • http://www.pygimli.org/index.html
            • Anaconda install: https://anaconda.org/gimli/pygimli
          • Operator Discretization Library
            • https://odlgroup.github.io/odl/getting_started/about_odl.html
          • Algorithmic Differentiation in Python
            • http://pythonhosted.org/algopy/
          • Linear Inverse Problem
            • linvpy
              • https://github.com/LCAV/linvpy
            • pinvprob
              • https://github.com/HajimeKawahara/pinvprob
            • hIPPYlib
              • https://hippylib.github.io/tutorials/2_PoissonDeterministic/
        • Math Lib.
          • Numpy
            • Kurs Numpy http://www.python-course.eu/numpy.php
          • SciPy
            • Wykaz oprogramowania naukowego https://www.scipy.org/topical-software.html
            • SciPy dokumentacja https://www.scipy.org/scipylib/index.html
            • Voronoi Diagrams
              • http://zderadicka.eu/voronoi-diagrams/
              • http://www-astro.physics.ox.ac.uk/~mxc/software/
              • http://dzhelil.info/triangle/examples.html
        • ERT
          • BERT
            • Repo https://gitlab.com/resistivity-net/bert/tree/master
          • E4D
            • https://e4d.pnnl.gov/Documents/E4D_User_Guide.pdf
        • MISC
          • TomograPy
            • Strona projektu http://nbarbey.github.io/TomograPy/
            • Artykuł: https://arxiv.org/pdf/1103.5904.pdf
          • Medical Data Manipulation
            • https://idoimaging.com/programs?q%5Bdisplay_function%5D=&q%5Bfor_audience%5D=&q%5Bfunction%5D=&q%5Bheader_function%5D=&q%5Binterface%5D=&q%5Blanguage%5D=19&q%5Bname_cont%5D=&q%5Bnetwork_function%5D=&q%5Bother_function%5D=&q%5Bplatform%5D=&q%5Bprogramming_function%5D=&q%5Bread_format%5D=&q%5Bspeciality%5D=&q%5Bsummary_cont%5D=&q%5Bwrite_format%5D=
        • Data Flow
          • Apache AirFlow
            • Tylko Linux http://airflow.incubator.apache.org/start.html
        • Visualization
          • VTK
            • http://www.vtk.org/Wiki/VTK/Examples/Python
          • Matplotlib
      • Intel Python
        • Installation: https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda
      • Python (x,y)
        • https://python-xy.github.io/
    • Scala
      • Development and deployment of Spark applications with Scala, Eclipse, and sbt – Part 1: Installation & configuration http://www.nodalpoint.com/development-and-deployment-of-spark-applications-with-scala-eclipse-and-sbt-part-1-installation-configuration/
      • Visualizing Big Data with Spark and Scala http://blog.ibmjstart.net/2016/07/18/visualizing-big-data-spark-scala/
      • SBT
      • Scala IDE
        • http://scala-ide.org/download/sdk.html
    • Java
    • Flow Base Programming
      • Used in IBM Streams, Apache NiFi
      • http://www.jpaulmorrison.com/fbp/
      • Apache Beam (Pyton SDK - https://github.com/apache/beam/tree/master/sdks/python#install-virtualenv) Google Cloud DataFlow - środowisko uruchomieniowe dla Apache Beamhttps://cloud.google.com/dataflow/  
      • Opis modelu programistycznego https://beam.apache.org/documentation/programming-guide/   
    • Programmer IDE
      • Eclipse
        • Maven
      • Zeppelin Notebook
        •  Zbiór przykładów analizy danych do Zeppelina https://github.com/hortonworks-gallery/zeppelin-notebooks   
      • Jupiter Notebook
        • Data export to HTML https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/
        • Visualisation PixieDust: Magic for Your Python Notebook https://developer.ibm.com/clouddataservices/2016/10/11/pixiedust-magic-for-python-notebook/
        • http://notebooks.azure.com
        • https://nbviewer.jupyter.org/
      • R Studio
      • Anaconda
      • VisualStudio PTVS
        • https://github.com/Microsoft/PTVS/

  6. Implementation

    • VM
      • Virtualization
      • Hyper-V
      • Docker
        • Wdrożenie Zeppelin https://github.com/dylanmei/docker-zeppelin
        • Wdrożenie Apache Spark https://github.com/gettyimages/docker-spark
      • VMware
      • VirtualBox
    • Cloud Computing
      • Azure HDInsight
      • IBM Bluemix IoT
        • Koniecznie!!! Spark Streaming + Watson IoT Platform Integration https://developer.ibm.com/recipes/tutorials/spark-streaming-ibm-watson-iot-platform-integration/
        • Koniecznie: Engage Machine Learning for detecting anomalous behaviors of things https://developer.ibm.com/recipes/tutorials/engage-machine-learning-for-detecting-anomalous-behaviors-of-things/
        • Koniecznie: Timeseries Data Analysis of IoT events by using Jupyter Notebook  https://developer.ibm.com/recipes/tutorials/timeseries-data-analysis-of-iot-events-by-using-jupyter-notebook/
        • Koniecznie!!! Integrating Watson IoT Platform with Message Hub and Apache Spark  https://developer.ibm.com/recipes/tutorials/integrating-watson-iot-platform-with-message-hubkafka/
        • Koniecznie!!! Integracja DB i Spark https://developer.ibm.com/clouddataservices/docs/ibm-data-science-experience/docs/
        • Cleanse and visualize location data using Spark and Bluemix https://www.ibm.com/developerworks/library/ba-1610location-data-spark-bluemix-trs/index.html
        • Apache Spark - using notebooks https://console.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index-gentopic3.html#genTopProcId4
        • !!! SparkR notebook with Cloudant https://github.com/charles2588/bluemixsparknotebooks/blob/master/R/SparkR_Cloudant_Connection.ipynb
      • AWS
      • Databricks
      • Google Cloud Platform
        • Real-Time Stream Processing for IoT: https://cloud.google.com/solutions/architecture/real-time-stream-processing-iot
        • Cena na 1 Rok GPU in Google: https://cloud.google.com/products/calculator/#id=99152826-9096-43b2-a100-836c36ef4d50
    • Bare Metal

  7. Integrated Platforms

    • Hortonworks
      • Data Platform
      • Data Flow
    • Cloudera
      • Cloudera StreamSets https://vimeo.com/147671735
    • IBM Open Platform

  8. Data Integration

    • Apache NiFi

  9. Messaging Protocols

    • MQTT
      • !!! iot-device-samples https://github.com/ibm-messaging/iot-device-samples

  10. Programming Methods

    • Img. Reconstr. Algorithms
      • Tensor-based
        • Artykuł: http://iopscience.iop.org/article/10.1088/1361-6501/aa58a3/meta
      • Simulated Annealing Particle Swarm Optimization
      • Landweber iterative method
      • Linear back projection (LBP)
      • Singular value decomposition
        • Implementacja Python: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.svd.html
      • Conjugate Gradient method
      • Maximum Likelihood Estimation Method
        • https://github.com/clarehchao/Spark3DImageReconstruction
      • Inverse problems
        • Introduction to inverse problems (Spring 2014) http://cc.oulu.fi/~smaenpaa/inv_pk/inv_pk14_eng.pdf
        • Tikhonov regularization method
          • Implementacja Python https://pypi.python.org/pypi/InverseProblem/1.0
        • Statistical and Computational InverseProblems
          • https://www.researchgate.net/profile/Ian_Langmore/publication/268250507_Statistical_and_Computational_Inverse_Problems_and_a_little_uncertainty_quantification/links/5654822708aeafc2aabbd811/Statistical-and-Computational-Inverse-Problems-and-a-little-uncertainty-quantification.pdf?origin=publication_detail&ev=pub_int_prw_xdl&msrp=iG4X9V_68LOGbkxNrcZEP4Az6qj7tC0tBRcUQyjL8QYqQT4QoLmsF2FgkN6bhYKWGD5hIwTkn5ZA4aHZ8A8286ns9G-BWrYTAsuNSMv5CNk.bYd_LZtB1khHvYwW5OyIFeUTFjZCQqyRJayOmj_N5AXLMsdZGfQieOSGqKg4N2g4hxqe2aS8SGqs7RTj0lO2mw.wfseq3OvlHxPmYqbEhxOxL040D2b4044fCPINZ0nDdsCbRE9GxeAhwoYCzzzOf64AwlcmWuBBIbKFhDQjoECqQ.jNjix2pbRzy-BWR2vabpp4TbUNEHGKEiBIwCb6q11EYbYkRGdAy-ev4pUz1krhlO9uaouT9_yrSr7F-4hJheJQ
        • Lectures
          • Lecture1: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture1 Lecture2: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture2 Lecture3&4: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture3 Lecture5: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture5
    • Machine Intelligence
      • Neural Networks
      • Machine Learning
        • A Visual Introduction to Machine Learning http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
        • Apache Spark MLlib
      • Deep Learning
        • Deeplearning4j on Spark https://deeplearning4j.org/
        • TensorFlow http://www.tensorflow.org
        • An Overview of Python Deep Learning Frameworks http://www.kdnuggets.com/2017/02/python-deep-learning-frameworks-overview.html
        • TensorFlow on Sparkhttps://github.com/yahoo/TensorFlowOnSpark
        • Google TensorFlow
          • Porównanie TensorFlow i CNTK https://esciencegroup.com/2016/02/08/tensorflow-meets-microsofts-cntk/
          • TensorBoard: Visualizing Learning
            • (C:\Anaconda3\envs\TensorFlow) C:\Users\tom\Documents>python -m tensorflow.tensorboard --logdir=./ localhost:6006
        • Microsoft Cognitive Toolkit (CNTK)
          • Strona projektu: https://www.microsoft.com/en-us/research/product/cognitive-toolkit/
          • Github: https://github.com/Microsoft/CNTK
          • Wiki: https://github.com/Microsoft/CNTK/wiki

  11. Approaches to reconstruction acceleration

    • Artykuł: Review of Parallel Computing Techniques for Computed Tomography Image Reconstruction http://www.imaging.sbes.vt.edu/BIDLib/CT/NiLiWang_ParaCompTech.pdf
    • Rozprawa Doktorska dot. CT: http://ir.uiowa.edu/cgi/viewcontent.cgi?article=2330&context=etd
    • Parallel Computing
      • Distribiuted Computing
        • Apache Spark
          • SparkR Desc.
            • Opis biblioteki SparkR https://spark.apache.org/docs/2.1.0/api/R/index.html
          • DataFrame
            • Operations
              • Operacje na obiekcie SparkDataFrame https://www.analyticsvidhya.com/blog/2016/10/spark-dataframe-and-operations/
      • Local PC Clusters
    • GPU
      • Artykuł: P.Kapusta http://annals-csis.org/Volume_8/pliks/344.pdf
    • Improved Algorithms

  12. Data Visualization

    • Web App
      • https://www.slideshare.net/BenLaird/real-time-data-viz-with-spark-streaming-kafka-and-d3js
      • D3.js
        • https://github.com/d3
        • DC.js
          • https://dc-js.github.io/dc.js/
      • Socket.io
        • https://github.com/socketio/socket.io
      • R Shiny
        • Tutorial: new ver. http://shiny.rstudio.com/tutorial/ old http://rstudio.github.io/shiny/tutorial/
      • Rickshaw TS graphs.
        • Rickshaw is a JavaScript toolkit for creating interactive time series graphs. http://code.shutterstock.com/rickshaw/
        • RealTime Visualization IBM
          • https://developer.ibm.com/recipes/tutorials/visualizing-your-data/
      • Bokeh
        • https://bokeh.pydata.org/en/latest/
      • MPLD3
        • http://mpld3.github.io/

  13. Medical Data Analysis

    • MIRAN Project
      • Projekt analizy danych medycznych za pomocą Python http://ne-scientific.com/miran/ http://ne-scientific.com/miran/

  14. Microservices

    • https://youtu.be/-zsKY9p_5R8?t=34m2s https://github.com/miguelgrinberg/microflack_admin
    • https://youtu.be/nrzLdMWTRMM
    • https://12factor.net/
    • Loadbalancer
      • Windows
        • Nginx
          • https://nginx.org/en/download.html
      • Linux
        • HAProxy
          • http://www.haproxy.org/
    • Design patterns
      • API Gateway
        • http://microservices.io/patterns/apigateway.html
      • Service Registry
        • http://microservices.io/patterns/service-registry.html
    • Service Registry
      • Consul
        • https://www.consul.io/intro/index.html
      • etcd
        • https://github.com/coreos/etcd
    • Containers
      • Docker
    • Service-to-Service Communication
    • Logging
      • Logspout
        • https://github.com/gliderlabs/logspout

Informationen

Beschreibung
Complex knowlage used to design and develope an analytical system for streams analysis
Erstellt von
Tomasz Cieplak
Aufrufe
208
Bewertung
Durchschnittliche Bewertung 5,0 von 5 (1 Bewertung)

Feedback

Worum geht es?

Wie gefällt dir das neue Design?

Dein Eindruck