lineCode
diff --git a/‎docs/modules/activation.rst‎
Lines changed: 17 additions & 3 deletions b/‎docs/modules/activation.rst‎
Lines changed: 17 additions & 3 deletions
diff --git a/‎docs/modules/files.rst‎
Lines changed: 20 additions & 0 deletions b/‎docs/modules/files.rst‎
Lines changed: 20 additions & 0 deletions
diff --git a/‎docs/modules/layers.rst‎
Lines changed: 185 additions & 0 deletions b/‎docs/modules/layers.rst‎
Lines changed: 185 additions & 0 deletions
diff --git a/‎tensorlayer/__pycache__/__init__.cpython-34.pyc‎
898 Bytes b/‎tensorlayer/__pycache__/__init__.cpython-34.pyc‎
898 Bytes
diff --git a/‎tensorlayer/__pycache__/activation.cpython-34.pyc‎
1002 Bytes b/‎tensorlayer/__pycache__/activation.cpython-34.pyc‎
1002 Bytes
diff --git a/‎tensorlayer/__pycache__/cost.cpython-34.pyc‎
10.4 KB b/‎tensorlayer/__pycache__/cost.cpython-34.pyc‎
10.4 KB
diff --git a/‎tensorlayer/__pycache__/files.cpython-34.pyc‎
25.2 KB b/‎tensorlayer/__pycache__/files.cpython-34.pyc‎
25.2 KB
diff --git a/‎tensorlayer/__pycache__/iterate.cpython-34.pyc‎
6.3 KB b/‎tensorlayer/__pycache__/iterate.cpython-34.pyc‎
6.3 KB
diff --git a/‎tensorlayer/__pycache__/layers.cpython-34.pyc‎
63.9 KB b/‎tensorlayer/__pycache__/layers.cpython-34.pyc‎
63.9 KB
diff --git a/‎tensorlayer/__pycache__/nlp.cpython-34.pyc‎
23.3 KB b/‎tensorlayer/__pycache__/nlp.cpython-34.pyc‎
23.3 KB
@@ -4,7 +4,23 @@ API - Activations
 To make TensorLayer simple, we minimize the number of activation functions as much as
 we can. So we encourage you to use TensorFlow's function. TensorFlow provides
 ``tf.nn.relu``, ``tf.nn.relu6``, ``tf.nn.elu``, ``tf.nn.softplus``,
-``tf.nn.softsign`` and so on, see `TensorFlow API <https://www.tensorflow.org/versions/master/api_docs/index.html>`_.
+``tf.nn.softsign`` and so on. More TensorFlow official activation functions can be found
+`here <https://www.tensorflow.org/versions/master/api_docs/python/nn.html#activation-functions>`_.
+
+
+Creating custom layers
+------------------------
+
+To implement a custom activation function in TensorLayer is very easy.
+
+The following is an example implementation of an activation that multiplies its input by 2.
+For more complex activation, TensorFlow API will be required.
+
+.. code-block:: python
+
+  def double_activation(x):
+      return x * 2
+
 
 
 .. automodule:: tensorlayer.activation
@@ -14,8 +30,6 @@ we can. So we encourage you to use TensorFlow's function. TensorFlow provides
    identity
    ramp
 
-More TensorFlow official activation functions can be found
-`here <https://www.tensorflow.org/versions/master/api_docs/python/nn.html#activation-functions>`_.
 
 Activation functions
 ---------------------
 
@@ -2,6 +2,26 @@ API - Load, Save Model and Data
 ===================================
 
 Load benchmark dataset, save and restore model, save and load variables.
+TensorFlow provides ``.ckpt`` file format to save and restore the models, while
+we suggest to use standard python file format ``.npz`` to save models for the
+sake of cross-platform.
+
+
+.. code-block:: python
+
+  # save model as .ckpt
+  saver = tf.train.Saver()
+  save_path = saver.save(sess, "model.ckpt")
+  # restore model from .ckpt
+  saver = tf.train.Saver()
+  saver.restore(sess, "model.ckpt")
+
+  # save model as .npz
+  tl.files.save_npz(network.all_params , name='model.npz')
+  # restore model from .npz
+  load_params = tl.files.load_npz(path='', name='model.npz')
+  tl.files.assign_params(sess, load_params, network)
+
 
 
 .. automodule:: tensorlayer.files
 
@@ -7,6 +7,191 @@ For example, we do not provide layer for local response normalization, we sugges
 you to apply ``tf.nn.lrn`` on ``Layer.outputs``.
 More functions can be found in `TensorFlow API <https://www.tensorflow.org/versions/master/api_docs/index.html>`_
 
+
+Understand layer
+-----------------
+
+All TensorLayer layers have a number of properties in common:
+
+ - ``layer.outputs`` : Tensor, the outputs of current layer.
+ - ``layer.all_params`` : a list of Tensor, all network variables in order.
+ - ``layer.all_layers`` : a list of Tensor, all network outputs in order.
+ - ``layer.all_drop`` : a dictionary of {placeholder : float}, all keeping probabilities of noise layer.
+
+All TensorLayer layers have a number of methods in common:
+
+ - ``layer.print_params()`` : print the network variables information in order (after ``sess.run(tf.initialize_all_variables())``). alternatively, print all variables by ``tl.layers.print_all_variables()``.
+ - ``layer.print_layers()`` : print the network layers information in order.
+ - ``layer.count_params()`` : print the number of parameters in the network.
+
+
+
+The initialization of a network is done by input layer, then we can stacked layers
+as follow, then a network is a ``Layer`` class.
+The most important properties of a network are ``network.all_params``, ``network.all_layers`` and ``network.all_drop``.
+The ``all_params`` is a list which store all pointers of all network parameters in order,
+the following script define a 3 layer network, then ``all_params = [W1, b1, W2, b2, W_out, b_out]``.
+The ``all_layers`` is a list which store all pointers of the outputs of all layers,
+in the following network, ``all_layers = [dropout(?, 784), relu(?, 800), dropout(?, 800), relu(?, 800), dropout(?, 800)], identity(?, 10)]``
+where ``?`` reflects any batch size. You can print the layer information and parameters information by
+using ``network.print_layers()`` and ``network.print_params()``.
+To count the number of parameters in a network, run ``network.count_params()``.
+
+
+
+.. code-block:: python
+
+  sess = tf.InteractiveSession()
+
+  x = tf.placeholder(tf.float32, shape=[None, 784], name='x')
+  y_ = tf.placeholder(tf.int64, shape=[None, ], name='y_')
+
+  network = tl.layers.InputLayer(x, name='input_layer')
+  network = tl.layers.DropoutLayer(network, keep=0.8, name='drop1')
+  network = tl.layers.DenseLayer(network, n_units=800,
+                                  act = tf.nn.relu, name='relu1')
+  network = tl.layers.DropoutLayer(network, keep=0.5, name='drop2')
+  network = tl.layers.DenseLayer(network, n_units=800,
+                                  act = tf.nn.relu, name='relu2')
+  network = tl.layers.DropoutLayer(network, keep=0.5, name='drop3')
+  network = tl.layers.DenseLayer(network, n_units=10,
+                                  act = tl.activation.identity,
+                                  name='output_layer')
+
+  y = network.outputs
+  y_op = tf.argmax(tf.nn.softmax(y), 1)
+
+  cost = tl.cost.cross_entropy(y, y_)
+
+  train_params = network.all_params
+
+  train_op = tf.train.AdamOptimizer(learning_rate, beta1=0.9, beta2=0.999,
+                              epsilon=1e-08, use_locking=False).minimize(cost, var_list = train_params)
+
+  sess.run(tf.initialize_all_variables())
+
+  network.print_params()
+  network.print_layers()
+
+In addition, ``network.all_drop`` is a dictionary which stores the keeping probabilities of all
+noise layer. In the above network, they are the keeping probabilities of dropout layers.
+
+So for training, enable all dropout layers as follow.
+
+.. code-block:: python
+
+  feed_dict = {x: X_train_a, y_: y_train_a}
+  feed_dict.update( network.all_drop )
+  loss, _ = sess.run([cost, train_op], feed_dict=feed_dict)
+  feed_dict.update( network.all_drop )
+
+For evaluating and testing, disable all dropout layers as follow.
+
+.. code-block:: python
+
+  feed_dict = {x: X_val, y_: y_val}
+  feed_dict.update(dp_dict)
+  print("   val loss: %f" % sess.run(cost, feed_dict=feed_dict))
+  print("   val acc: %f" % np.mean(y_val ==
+                          sess.run(y_op, feed_dict=feed_dict)))
+
+For more details, please read the MNIST examples.
+
+Creating custom layers
+------------------------
+
+Understand Dense layer
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before creating your own TensorLayer layer, let's have a look at Dense layer.
+It creates a weights matrix and biases vector if not exists, then implement
+the output expression.
+At the end, as a layer with parameter, we also need to append the parameters into ``all_params``.
+
+
+.. code-block:: python
+
+  class DenseLayer(Layer):
+      """
+      The :class:`DenseLayer` class is a fully connected layer.
+
+      Parameters
+      ----------
+      layer : a :class:`Layer` instance
+          The `Layer` class feeding into this layer.
+      n_units : int
+          The number of units of the layer.
+      act : activation function
+          The function that is applied to the layer activations.
+      W_init : weights initializer
+          The initializer for initializing the weight matrix.
+      b_init : biases initializer
+          The initializer for initializing the bias vector.
+      W_init_args : dictionary
+          The arguments for the weights tf.get_variable.
+      b_init_args : dictionary
+          The arguments for the biases tf.get_variable.
+      name : a string or None
+          An optional name to attach to this layer.
+
+      def __init__(
+          self,
+          layer = None,
+          n_units = 100,
+          act = tf.nn.relu,
+          W_init = tf.truncated_normal_initializer(stddev=0.1),
+          b_init = tf.constant_initializer(value=0.0),
+          W_init_args = {},
+          b_init_args = {},
+          name ='dense_layer',
+      ):
+          Layer.__init__(self, name=name)
+          self.inputs = layer.outputs
+          if self.inputs.get_shape().ndims != 2:
+              raise Exception("The input dimension must be rank 2")
+          n_in = int(self.inputs._shape[-1])
+          self.n_units = n_units
+          print("  tensorlayer:Instantiate DenseLayer %s: %d, %s" % (self.name, self.n_units, act))
+          with tf.variable_scope(name) as vs:
+              W = tf.get_variable(name='W', shape=(n_in, n_units), initializer=W_init, **W_init_args )
+              b = tf.get_variable(name='b', shape=(n_units), initializer=b_init, **b_init_args )
+          self.outputs = act(tf.matmul(self.inputs, W) + b)
+
+          # Hint : list(), dict() is pass by value (shallow).
+          self.all_layers = list(layer.all_layers)
+          self.all_params = list(layer.all_params)
+          self.all_drop = dict(layer.all_drop)
+          self.all_layers.extend( [self.outputs] )
+          self.all_params.extend( [W, b] )
+
+
+A simple layer
+^^^^^^^^^^^^^^^
+
+To implement a custom layer in TensorLayer, you will have to write a Python class
+that subclasses Layer and implement the ``outputs`` expression.
+
+The following is an example implementation of a layer that multiplies its input by 2:
+
+.. code-block:: python
+
+  class DoubleLayer(Layer):
+      def __init__(
+          self,
+          layer = None,
+          name ='dense_layer',
+      ):
+          Layer.__init__(self, name=name)
+          self.inputs = layer.outputs
+          self.outputs = self.inputs * 2
+
+          self.all_layers = list(layer.all_layers)
+          self.all_params = list(layer.all_params)
+          self.all_drop = dict(layer.all_drop)
+          self.all_layers.extend( [self.outputs] )
+
+
+
 .. automodule:: tensorlayer.layers
 
 .. autosummary::