Please enable JavaScript.

Coggle requires JavaScript to display documents.

NASNet (:snowflake:idea (:heavy_dollar_sign:利用这个search space…

- - - - :fire:keep the heights and widths of the images
    - - :fire:reduce the heights and widths by a factor of two
      - :dagger_knife:stride of 2
- - - - :warning:applying NAS directly to the large dataset is expensive
        
        :arrow_right:search for a good architecture on a proxy dataset, for instance the smaller CIFAR-10 dataset, and then transfer the learned architecture to ImageNet
      - In NAS,
        a controller recurrent neural network (RNN) samples child
        networks with different architectures. The child networks
        are trained to convergence to obtain some accuracy on a
        held-out validation set. The resulting accuracies are used
        to update the controller so that the controller will generate
        better architectures over time. The controller weights are
        updated with policy gradient
      - uses a reinforcement learning search method to optimize architecture configurations.
      - :explode:Neural Architecture Search
    - - :fire:sample decisions from the uniform distribution
      - :sweat_drops:slightly worse than RL in CIFAR-10
  - - - :apple:blocks
        
        :bread:1 cell(Normal Cell / Reduction Cell) => B blocks; 1 block <=> 5 prediction steps made by 5 softmax classifiers
        
        :explode:From hi-1 & hi, we choose (hidden_state_1, hidden_state_2, operation_1, operation_2, combination) for B times in order to get B cases. Each case is called a block. Finally they are concated.
        
        :link:Best architectures for CIFAR-10
      - :moneybag:5 steps in each block
        
        :one:Step 1. Select a hidden state from hi, hi−1 or from the set of hidden states created in previous blocks.
        
        :two:Step 2. Select a second hidden state from the same options as in Step 1.
        
        :three:Step 3. Select an operation to apply to the hidden state selected in Step 1.
        
        :four:Step 4. Select an operation to apply to the hidden state selected in Step 2.
        
        :five:Step 5. Select a method to combine the outputs of Step 3 and 4 to create a new hidden state.
      - :apple:options
        
        :one:hidden states
        
        :two:hi
        
        :question:What do hi, hi-1 mean?
        
        :one:hi-1
        
        :warning:input image if it is the first cell
        
        :explode:the new hidden state created in Step 5 of previous cell
        
        :star:The final 2 hidden states selected by controller RNN is one of {(hi, hi), (hi, hi-1), (hi-1, hi), (hi-1, hi-1)}
        
        :two:operations
        
        • identity • 1x3 then 3x1 convolution • 1x7 then 7x1 convolution • 3x3 dilated convolution • 3x3 average pooling • 3x3 max pooling • 5x5 max pooling • 7x7 max pooling • 1x1 convolution • 3x3 convolution • 3x3 depthwise-separable conv • 5x5 depthwise-seperable conv • 7x7 depthwise-separable conv
        
        :fire:popular operations in CNN
        
        :three:methods
        
        :explode:Here, methods mean the options of method of combining 2 hidden states in Step 5
        
        :one:element-wise addition
        
        :two:concatenation between 2 hidden states along the filter dimension
        
        :coffee:2 * 5B predictions in total
        
        :baguette_bread:2=> Normal Cell & Reduction Cell; 5 => 5 values in prediction; B => number of blocks
        
        :bread: the first 5B predictions are for the Normal Cell and the second 5B predictions are for the Reduction Cell.
      - :!:selecting B=5 provides a good result
- - - - :question:What is the overall architecture
        
        :maple_leaf:the permutation of Image Input, Normal Cells, Reduction Cells
      - :question:Does it means that we construct the networks before training?
        
        :maple_leaf:We construct the overall architecture, but do not determine how to handle each cell
      - :explode:The overall architecture of convolutional nets is predetermined