TensorFlow

TensorFlow

1. Import

   • import tensorflow as tf
   • train-tech batching
     • Placeholders
       • x = tf.placeholder(dtype, shape=None, name=None) Shape - список с измерениями, None если меняется.
       • https://www.tensorflow.org/versions/r0.10/api_docs/python/io_ops.html#placeholder
       • Constants
       • Variables
         • Матрица коофициентов, которые мы меняем для подстройки
         • W = tf.Variable(tf.zeros([784, 10])) b = tf.Variable(tf.zeros([10]))
         • Initialize
           • tf.initialize_all_variables()
           • Zeros
             • tf.Variable(tf.zeros(shape))
           • Random
             • tf.Variable(tf.random_normal(shape, stddev=0.01))
           • truncated normal
             • tf.truncated_normal(shape, stddev=0.1) Нормальное распределение, но всё, что больше 2 стандартных отклонений обрезано
           • Constant
             • tf.constant(0.1, shape=shape)
         • Виды слоёв
           • Функция активации
             • https://en.wikipedia.org/wiki/Activation_function
             • Loss function
               • http://stats.stackexchange.com/questions/154879/a-list-of-cost-functions-used-in-neural-networks-alongside-applications
               • cross_entropy
                 • cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1])) где y_ - правильные ответы, а y - то, что выдаёт
                 • tf.nn.softmax_cross_entropy_with_logits(py_x, Y)
               • Quadratic cost
               • Exponentional cost
               • Hellinger distance
               • Kullback–Leibler divergence
               • Itakura–Saito distance
               • Train Step
                 • tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
                 • Окончания
                   • one-hot vectors
                     • Вектор, заполненый нулями, и одна единица
                     • https://en.wikipedia.org/wiki/One-hot
                   • Softmax
                     • https://en.wikipedia.org/wiki/Softmax_function
                     • tf.nn.softmax(tf.matmul(x, W) + b)
                   • Session
                     • sess = tf.Session() sess.run(init)
                     • Train
                       • for i in range(1000):      sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
                       • Test
                         • correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels}))
                     • Interactive
                       • sess = tf.InteractiveSession()
                   • Argmax
                     • tf.argmax(input, dimension, name=None)
                 • Gradient Descent
                   • http://sebastianruder.com/optimizing-gradient-descent/
                   • Adam Optimizer
                     • https://www.quora.com/Can-you-explain-basic-intuition-behind-ADAM-a-method-for-stochastic-optimization
                   • Nesterov accelerated gradient
             • ReLU
               • tf.nn.relu(features, name=None)
               • https://en.wikipedia.org/wiki/Rectifier_(neural_networks)
               • f(x) = max(0, x)
             • Sigmoid
               • tf.sigmoid(x, name=None)
               • (-∞, ∞) -> (0, 1)
             • Tahn
               • tf.tanh(x, name=None)
               • (-∞, ∞) -> (-1, 1)
               • https://en.wikipedia.org/wiki/Hyperbolic_function
           • Convolution
             • tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
             • https://www.tensorflow.org/versions/master/api_docs/python/nn.html#conv2d
             • W[x1,x2,x3,x4]
               • 4-мерный тензор
               • https://www.quora.com/What-do-channels-refer-to-in-a-convolutional-neural-network
               • x1,x2 - размар ядра свёртки
               • x3 - каналы
                 • например, цвета
               • x4 - фичи
                 • кол-во ядер свёртки
             • Strides
               • на сколько смещается окно при проходе
           • Pooling
             • tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
             • Max
             • Average
           • Densely Connected
             • y = tf.matmul(tf.matmul(x, W1), W2)...
             • Matmul
               • Умножение матриц  X[a, b]xY[b, c] = Z[a, c]
               • tf.matmul(a, b, transpose_a=False, transpose_b=False, a_is_sparse=False, b_is_sparse=False, name=None)
             • Dropout
               • keep_prob = tf.placeholder("float") h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)
               • С вероятностью keep_prob, некоторые нейроны исключаются из процесса во время тренировки
               • http://stackoverflow.com/questions/35545798/keep-prob-in-tensorflow-mnist-tutorial
           • LSTMCell
             • https://www.tensorflow.org/versions/r0.10/tutorials/recurrent/index.html
             • State
             • Output
           • Normalisation
             • http://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn

2. TensorBoard: Visualizing Learning

   • https://www.tensorflow.org/versions/r0.8/how_tos/graph_viz/index.html
   • tensorboard --logdir=log_simple_graph

3. Типы NN

   • RNN
     • tf.nn.rnn_cell.RNNCell(num) num - число выходов
     • lstm
       • http://monik.in/a-noobs-guide-to-implementing-rnn-lstm-using-tensorflow/
     • seq2seq
   • Convolutional
     • https://en.wikipedia.org/wiki/Convolutional_neural_network
     • First Convolutional Layer -> Pooling -> Second Convolutional Layer -> Densely Connected Layer -> Pooling -> Densely Connected Layer -> Densely Connected Layer -> Softmax
     • http://cs231n.github.io/convolutional-networks
   • Многослойный перцептрон
     • Densely Connected Layer -> Densely Connected Layer -> Densely Connected Layer ->...

4. Applications

   • Image recognition
     • Face detection
   • NLP
     • Emotion analys