atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Colorjam</title>
  
  
  <link href="/atom.xml" rel="self"/>
  
  <link href="http://yoursite.com/"/>
  <updated>2020-06-02T14:35:58.741Z</updated>
  <id>http://yoursite.com/</id>
  
  <author>
    <name>Colorjam</name>
    
  </author>
  
  <generator uri="http://hexo.io/">Hexo</generator>
  
  <entry>
    <title>深度学习优化器</title>
    <link href="http://yoursite.com/2019/12/12/NAG/"/>
    <id>http://yoursite.com/2019/12/12/NAG/</id>
    <published>2019-12-12T02:07:11.000Z</published>
    <updated>2020-06-02T14:35:58.741Z</updated>
    
    <content type="html"><![CDATA[<p>优化器从NAS到FISTA</p><h3 id="NAG"><a href="#NAG" class="headerlink" title="NAG"></a>NAG</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">nag</span><span class="params">(weight, lr, grad, mom)</span>:</span></div><div class="line">  mom = <span class="number">0.9</span> * mom + grad <span class="comment"># momentum</span></div><div class="line">  grad = grad + <span class="number">0.9</span> * mom  <span class="comment"># nesterov</span></div><div class="line">  weight = weight - lr * grad</div></pre></td></tr></table></figure><h3 id="APG"><a href="#APG" class="headerlink" title="APG"></a>APG</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">optim_apg</span><span class="params">(weight, lr, grad, mom, gamma)</span>:</span></div><div class="line">  z = weight - lr * grad</div><div class="line">  z = soft_thresholding(z, lr * gamma)</div><div class="line">  mom = z - weight + <span class="number">0.9</span> * mom</div><div class="line">  weight = z + <span class="number">0.9</span> * mom</div><div class="line">  </div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">soft_thresholding</span><span class="params">(x, gamma)</span>:</span></div><div class="line">  y = torch.max(torch.abs(x) - gamma, torch.zeros_like(x))</div><div class="line">  <span class="keyword">return</span> torch.sign(x) * y</div></pre></td></tr></table></figure><h3 id="FISTA"><a href="#FISTA" class="headerlink" title="FISTA"></a>FISTA</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">optim_fista</span><span class="params">(weight_prev, alpha_prev, lr, grad, mom, gamma)</span>:</span></div><div class="line">  alpha = (<span class="number">1</span> + (<span class="number">1</span> + <span class="number">4</span> * alpha_prev**<span class="number">2</span>).sqrt()) / <span class="number">2</span></div><div class="line">  z = weight + ((alpha - <span class="number">1</span>) / alpha_prev) * (weight - weight_prev)</div><div class="line">  z = z - lr * grad</div><div class="line">  weight = soft_thresholding(z, lr * gamma)</div></pre></td></tr></table></figure>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;优化器从NAS到FISTA&lt;/p&gt;
&lt;h3 id=&quot;NAG&quot;&gt;&lt;a href=&quot;#NAG&quot; class=&quot;headerlink&quot; title=&quot;NAG&quot;&gt;&lt;/a&gt;NAG&lt;/h3&gt;&lt;figure class=&quot;highlight python&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td 
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>模型压缩 | 剪枝乱炖</title>
    <link href="http://yoursite.com/2019/12/11/pruning/"/>
    <id>http://yoursite.com/2019/12/11/pruning/</id>
    <published>2019-12-11T06:48:12.000Z</published>
    <updated>2019-12-13T08:48:14.012Z</updated>
    
    <content type="html"><![CDATA[<p>剪枝是模型压缩的一个子领域，依据剪枝粒度可以分为非结构化/结构化剪枝，依据实现方法可以大致分为基于度量标准/基于重建误差/基于稀疏训练的剪枝，并且逐渐有向AutoML发展的趋势。由于实现方法在剪枝粒度上是有通用性的，本文主要从实现方法进行展开，康康近年来关于剪枝的有的没的，从个人角度对近几年经典的剪枝方法以及其拓展进行一下梳理。</p><h2 id="基于度量标准的剪枝"><a href="#基于度量标准的剪枝" class="headerlink" title="基于度量标准的剪枝"></a>基于度量标准的剪枝</h2><p>这类方法通常是提出一个判断神经元是否重要的度量标准，依据这个标准计算出衡量神经元重要性的值，将不重要的神经元剪掉。在神经网络中可以用于度量的值主要分为3大块：<strong>Weight / Activation / Gradient</strong>。各种神奇的组合就产出了各种metric玩法。</p><blockquote><p>这里的神经元可以为非结构化剪枝中的单个weight亦或结构化剪枝中的整个filter。</p></blockquote><p><strong>Weight：</strong>基于结构化剪枝中比较经典的方法是<a href="https://arxiv.org/pdf/1608.08710.pdf" target="_blank" rel="external">Pruning Filters for Efficient ConvNets</a>(ICLR2017)，基于L1-norm判断filter的重要性。<a href="https://arxiv.org/pdf/1811.00250.pdf" target="_blank" rel="external">Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration</a>(CVPR2019) 把绝对重要性拉到相对层面，认为与其他filters太相似的filter不重要。</p><p><strong>Activation：</strong><a href="https://arxiv.org/abs/1607.03250" target="_blank" rel="external">Network trimming: A data-driven neuron pruning approach towards efficient deep architectures</a> 用activations中0的比例 (Average Percentage of Zeros, APoZ)作为度量标准，<a href="https://arxiv.org/abs/1706.05791" target="_blank" rel="external">An Entropy-based Pruning Method for CNN Compression</a> 则利用信息熵进行剪枝。</p><p><strong>Gradient：</strong>这类方法通常从Loss出发寻找对损失影响最小的神经元。将目标函数用泰勒展开的方法可以追溯到上世纪90年代初，比如Lecun的<a href="http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf" target="_blank" rel="external">Optimal Brain Damage</a> 和 <a href="https://authors.library.caltech.edu/54983/3/647-second-order-derivatives-for-network-pruning-optimal-brain-surgeon%281%29.pdf" target="_blank" rel="external">Second order derivatives for network pruning: Optimal Brain Surgeon </a>。近年来比较有代表性的就是<a href="https://arxiv.org/abs/1611.06440" target="_blank" rel="external">Pruning Convolutional Neural Networks for Resource Efficient</a>(ICLR2017)，对activation在0点进行泰勒展开。原作者也很好的向我们展现了如何优雅地进行方法迁移 <a href="https://arxiv.org/abs/1906.10771" target="_blank" rel="external">Importance Estimation for Neural Network Pruning</a>(CVPR2019)，换成weight的展开再加个平方。类似的方法还有 <a href="https://arxiv.org/abs/1801.05787" target="_blank" rel="external">Faster gaze prediction with dense networks and Fisher pruning</a>，用Fisher信息来近似Hessian矩阵。<a href="https://arxiv.org/abs/1810.02340" target="_blank" rel="external">SNIP: Single-shot Network Pruning based on Connection Sensitivity</a>(ICLR2019)则直接利用导数对随机初始化的权重进行非结构化剪枝。相关工作同样可以追溯到上世纪80年代末<a href="https://papers.nips.cc/paper/119-skeletonization-a-technique-for-trimming-the-fat-from-a-network-via-relevance-assessment" target="_blank" rel="external">Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment</a>(NIPS1988)。历史总是惊人的相似：</p><p><img src="/Users/colorjam/Desktop/image-20191213134813210.png" alt="image-20191213134813210"></p><p>还有一些考虑实际硬件部署并结合度量标准进行剪枝的方法，对网络层的剪枝顺序进行了选择。<a href="https://arxiv.org/pdf/1611.05128.pdf" target="_blank" rel="external">Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning</a>(CVPR2017)利用每层的energy consumption来决定剪枝顺序，<a href="https://arxiv.org/abs/1804.03230" target="_blank" rel="external">NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications</a>(ECCV2018)建立了latency的表，利用贪心的方式决定该剪的层。</p><h2 id="基于重建误差的剪枝"><a href="#基于重建误差的剪枝" class="headerlink" title="基于重建误差的剪枝"></a>基于重建误差的剪枝</h2><p>这类方法通过最小化特征输出的重建误差来确定哪些filters要进行剪裁，即找到当前层对后面的网络层输出没啥影响的信息。<a href="https://arxiv.org/abs/1707.06342" target="_blank" rel="external">ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression</a> 采用贪心法，<a href="https://arxiv.org/abs/1707.06168" target="_blank" rel="external">Channel Pruning for Accelerating Very Deep Neural Networks</a>(ICLR2017) 则采用Lasso regression。<a href="https://arxiv.org/abs/1711.05908" target="_blank" rel="external">NISP: Pruning Networks using Neuron Importance Score Propagation</a>(CVPR2018) 通过最小化网络倒数第二层的重建误差，并将反向传播的误差累积考虑在内，来决定前面哪些filters需要裁剪。</p><h2 id="基于稀疏训练的剪枝"><a href="#基于稀疏训练的剪枝" class="headerlink" title="基于稀疏训练的剪枝"></a>基于稀疏训练的剪枝</h2><p>这类方法采用训练的方式，结合各种regularizer来让网络的权重变得稀疏，于是可以将接近于0的值剪掉。<a href="https://arxiv.org/abs/1608.03665" target="_blank" rel="external">Learning Structured Sparsity in Deep Neural Networks</a> 用group Lasso进行结构化稀疏，包括filters, channels, filter shapes, depth。 <a href="https://arxiv.org/abs/1707.01213" target="_blank" rel="external">Data-Driven Sparse Structure Selection for Deep Neural Networks</a>(ECCV2018)通过引入可学习的mask，用APG算法来稀疏mask达到结构化剪枝。由于每个通道的输出都会经过BN，可以巧妙地直接稀疏BN的scaling factor，比如 <a href="https://arxiv.org/abs/1708.06519" target="_blank" rel="external">Learning Efficient Convolutional Networks through Network Slimming</a>(ICCV2017) 采用L1 regularizer，<a href="https://arxiv.org/abs/1802.00124" target="_blank" rel="external">Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers</a>(ICLR2018) 则采用ISTA来进行稀疏。<a href="https://arxiv.org/abs/1711.06798" target="_blank" rel="external">MorphNet: Fast &amp; Simple Resource-Constrained Structure Learning of Deep Networks</a>(CVPR2018) 也是直接利用L1 regularizer，但是结合了MobileNet中的width-multiplier，加上了shink-expand操作，能够更好的满足资源限制。</p><h2 id="Random-and-Rethinking"><a href="#Random-and-Rethinking" class="headerlink" title="Random and Rethinking"></a>Random and Rethinking</h2><p>有采用各种剪枝方法的就有和这些剪枝方法对着干的。<a href="https://arxiv.org/pdf/1801.10447.pdf" target="_blank" rel="external">Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks</a> 就表明了度量标准都没啥用，随机赛高。<a href="">Rethinking the Value of Network Pruning</a>(ICLR2019) 则表示剪枝策略实际上是为了获得网络结构，挑战了传统的 train-prune-finetune的剪枝流程。<a href="https://arxiv.org/abs/1909.12579" target="_blank" rel="external">Pruning from Scratch</a> 则直接用Network Slimming的方法对训练过程中的剪枝结构进行了一波分析，发现直接采用random初始化的网络权重能够获得更丰富的剪枝结构。</p><h2 id="走向NAS的自动化剪枝"><a href="#走向NAS的自动化剪枝" class="headerlink" title="走向NAS的自动化剪枝"></a>走向NAS的自动化剪枝</h2><p>从<a href="https://arxiv.org/abs/1802.03494" target="_blank" rel="external">AMC: AutoML for Model Compression and Acceleration on Mobile Devices</a>[ECCV2018]开始将强化学习引入剪枝，剪枝的研究开始套上各种Auto的帽子，玩法更是层出不穷。<a href="https://arxiv.org/abs/1903.11728" target="_blank" rel="external">AutoSlim: Towards One-Shot Architecture Search for Channel Numbers</a>先训练出一个slimmable model（类似NAS中的SuperNet <a href="https://arxiv.org/abs/1908.09791" target="_blank" rel="external">Once for All: Train One Network and Specialize it for Efficient Deployment</a>），继而通过贪心的方式逐步对网络进行裁剪。 <a href="https://arxiv.org/abs/1905.09717" target="_blank" rel="external">Network Pruning via Transformable Architecture Search</a>(NIPS2019) 则把NAS可导的一套迁移过来做剪枝。<a href="http://proceedings.mlr.press/v97/ding19a.html" target="_blank" rel="external">Approximated Oracle Filter Pruning for Destructive CNN Width Optimization</a>(ICML2019)平行操作网络的所有层，用二分搜索的方式确定每层的剪枝数。<a href="https://arxiv.org/pdf/1911.07478.pdf" target="_blank" rel="external">Fine-Grained Neural Architecture Search</a>把NAS的粒度降到了通道，包含了空的操作即剪枝。还有各种拿进化来做的也就不提了。</p><p>此外，还有基于信息瓶颈的方法<a href="https://arxiv.org/abs/1802.10399" target="_blank" rel="external">Compressing Neural Networks using the Variational Information Bottleneck</a>(ICML2018)，聚类的方法<a href="http://openaccess.thecvf.com/content_CVPR_2019/html/Ding_Centripetal_SGD_for_Pruning_Very_Deep_Convolutional_Networks_With_Complicated_CVPR_2019_paper.html" target="_blank" rel="external">Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure</a>(CPVR2019)，等等等等等……</p><h2 id="剪枝之外"><a href="#剪枝之外" class="headerlink" title="剪枝之外"></a>剪枝之外</h2><p><strong>提升精度：</strong> 利用剪枝的方式来提升模型精度，比如<a href="https://arxiv.org/abs/1607.04381" target="_blank" rel="external">DSD: Dense-Sparse-Dense Training for Deep Neural Networks</a>(ICLR2017)利用非结构化剪枝，阶段性的砍掉某些权重再恢复。稀疏训练<a href="https://arxiv.org/abs/1907.04840" target="_blank" rel="external">Sparse Networks from Scratch: Faster Training without Losing Performance</a>在训练过程中保持网络的稀疏率不变，动态调整层间的稀疏率。</p><p><strong>动态结构：</strong>不同的输入图片可以走网络中的不同结构。<a href="https://arxiv.org/abs/1711.08393" target="_blank" rel="external">BlockDrop: Dynamic Inference Paths in Residual Networks</a>(CVPR2018)引入一个Policy Network，以Block为单位进行选择。<a href="https://openreview.net/pdf?id=BJxh2j0qYm" target="_blank" rel="external">Dynamic Channel Pruning: Feature Boosting and Suppression</a>(ICLR2019)引入SEBlock，以Channel为单位进行选择。<a href="https://arxiv.org/abs/1908.06294" target="_blank" rel="external">Improved Techniques for Training Adaptive Deep Networks</a>采用截断式的选择，简单的图片采用靠前的网路层解决，复杂的加入后面得网络层。</p><h3 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h3><p><a href="https://blog.csdn.net/jinzhuojun/article/details/100621397" target="_blank" rel="external">闲话模型压缩之网络剪枝（Network Pruning）篇</a></p><p><a href="https://draveness.me/sketch-and-sketch" target="_blank" rel="external">技术文章配图指南</a></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;剪枝是模型压缩的一个子领域，依据剪枝粒度可以分为非结构化/结构化剪枝，依据实现方法可以大致分为基于度量标准/基于重建误差/基于稀疏训练的剪枝，并且逐渐有向AutoML发展的趋势。由于实现方法在剪枝粒度上是有通用性的，本文主要从实现方法进行展开，康康近年来关于剪枝的有的没的，
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>Prune during training</title>
    <link href="http://yoursite.com/2019/11/05/Scalable%20training%20of%20artificial%20neural%20networks%20with%20adaptive%20sparse%20connectivity%20inspired%20by%20network%20science/"/>
    <id>http://yoursite.com/2019/11/05/Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science/</id>
    <published>2019-11-05T15:48:10.000Z</published>
    <updated>2020-06-02T14:39:48.522Z</updated>
    
    <content type="html"><![CDATA[<blockquote><p>Sparse evolutionary training (SET) simplifies prune–regrowth cycles by using heuristics for random growth at the end of each training epoch.</p></blockquote><p>主要针对全连接层，提出利用 sparse connected layer代替fully connected layer。</p><p>sparse connected layer首先是随机初始化的，利用的是Erdös–Rényi random graph来生成拓扑结构，其中$h^k_i$和$h_j^{k-1}$的连接概率为：<br>$$<br>p\left(W_{i j}^{k}\right)=\frac{\varepsilon\left(n^{k}+n^{k-1}\right)}{n^{k} n^{k-1}}<br>$$<br>超参$\epsilon$控制稀疏度。在每次训练时一部分具有smallest positive weights和largest negative weights被移除。下一阶段再生成与剪掉connections相同数量的weights。</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20191105100616370.png" alt="image-20191105100616370"></p><h3 id="NeST-A-Neural-Network-Synthesis-Tool-Based-on-a-Grow-and-Prune-Paradigm"><a href="#NeST-A-Neural-Network-Synthesis-Tool-Based-on-a-Grow-and-Prune-Paradigm" class="headerlink" title="NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm"></a>NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm</h3><ul><li><p>gorw connections and neurons based on gradient information </p></li><li><p>prune away insignificant connections based on magnitude information</p></li></ul><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20191105103114233.png" alt="image-20191105103114233"></p><p>剪枝：<br>$$<br>\mathbf{u}^{l}=\left[\left(\mathbf{W}^{l} \mathbf{x}^{l-1}+\mathbf{b}^{l}\right)-\mathbf{E}\right] \oslash \mathbf{V}=\mathbf{W}_{_}^{l} \mathbf{x}+\mathbf{b}_{_}^{l}<br>$$</p><p>$$<br>\mathbf{W}_{_}^{l}=\mathbf{W}^{l} \oslash \mathbf{V}, \mathbf{b}_{_}^{l}=\left(\mathbf{b}^{l}-\mathbf{E}\right) \oslash \mathbf{V}<br>$$</p><h3 id="Parameter-Efficient-Training-of-Deep-Convolutional-Neural-Networks-by-Dynamic-Sparse-Reparameterization"><a href="#Parameter-Efficient-Training-of-Deep-Convolutional-Neural-Networks-by-Dynamic-Sparse-Reparameterization" class="headerlink" title="Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization"></a>Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization</h3><p>ICML2019</p><blockquote><p>Dynamic Sparse Reparameterization (DSR) (Mostafa &amp; Wang, 2019) implements a prune– redistribute–regrowth cycle where target sparsity levels are redistributed among layers, based on loss gradients </p></blockquote><p>本文解决的其实下列问题：在训练中给定一个fixed budget of parameters, how to train it to yield the best generalization performance.</p><p>首先所有的参数随机初始化为相同的sparsity。训练过程也分为两步</p><ol><li>magnitude-based pruning。利用一个全局threshold $H$进行剪枝。</li><li>random growth。当移除掉K个parameters后，K个zero-initialized的参数会被重新分配，基于以下准则：有更多不为0值权重的层会有更多的free parameters。也就是说free parameters应该要被再分布到一些能够接受larger loss gradients的权重上。（就像NesT中grow的思想）</li></ol><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20191105103724605.png" alt="image-20191105103724605"></p><p>为了让prune和grow的权重数量是相同的，文章用了额外的保护措施。</p><p>本文的对比实验比较充分，与6个baseline进行了比较</p><ol><li>Full dense: 原始的dense网络</li><li>Thin dense：有更少层的模型</li><li>Static sparse: 训练过程中sparse的地方是不变的</li><li>Compress sparse: 通过prune-retrain一个大的模型</li><li>DeepR</li><li>SET</li></ol><p>thin dense和static sparse网络训了更多的epochs。</p><h3 id="Dynamic-pruning-with-feedback"><a href="#Dynamic-pruning-with-feedback" class="headerlink" title="Dynamic pruning with feedback"></a>Dynamic pruning with feedback</h3><p>ICCV2020</p><p>本文提出的DPF，通过反馈机制进行剪枝。在pruend的模型上计算的梯度$\tilde{\mathbf{w}}_{t}=\mathbf{m}_{t} \odot \mathbf{w}_{t}$，应用在dense 模型的权重$w_t$上：<br>$$<br>\mathbf{w}_{t+1}:=\mathbf{w}_{t}-\gamma_{t} \mathbf{g}\left(\mathbf{m}_{t} \odot \mathbf{w}_{t}\right)=\mathbf{w}_{t}-\gamma_{t} \mathbf{g}\left(\tilde{\mathbf{w}}_{t}\right)<br>$$<br>文章说这样可以让模型 recover form ”errors”。暂时mask掉一些特定weight，这些weight在后续训练中能够再次被激活。</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20191105102431690.png" alt="image-20191105102431690"></p><h3 id="Sparse-Networks-from-Scratch-Faster-Training-without-Losing-Performance"><a href="#Sparse-Networks-from-Scratch-Faster-Training-without-Losing-Performance" class="headerlink" title="Sparse Networks from Scratch: Faster Training without Losing Performance"></a>Sparse Networks from Scratch: Faster Training without Losing Performance</h3><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20191115191831435.png" alt="image-20191115191831435"></p><p>用momentum来计算magnitude。主要由三个阶段构成：</p><ul><li>redistribution of weights</li><li>pruning weights</li><li>regrowing weights</li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;blockquote&gt;
&lt;p&gt;Sparse evolutionary training (SET) simplifies prune–regrowth cycles by using heuristics for random growth at the end of each
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
  </entry>
  
  <entry>
    <title>每周论文 Vol.11</title>
    <link href="http://yoursite.com/2019/09/17/weekly-paper-11/"/>
    <id>http://yoursite.com/2019/09/17/weekly-paper-11/</id>
    <published>2019-09-17T02:07:11.000Z</published>
    <updated>2020-06-02T14:38:48.943Z</updated>
    
    <content type="html"><![CDATA[<h3 id="1️⃣-EFFICIENT-MULTI-OBJECTIVE-NEURAL-ARCHITEC-TURE-SEARCH-VIA-LAMARCKIAN-EVOLUTION"><a href="#1️⃣-EFFICIENT-MULTI-OBJECTIVE-NEURAL-ARCHITEC-TURE-SEARCH-VIA-LAMARCKIAN-EVOLUTION" class="headerlink" title="1️⃣ EFFICIENT MULTI-OBJECTIVE NEURAL ARCHITEC-TURE SEARCH VIA LAMARCKIAN EVOLUTION"></a>1️⃣ EFFICIENT MULTI-OBJECTIVE NEURAL ARCHITEC-TURE SEARCH VIA LAMARCKIAN EVOLUTION</h3><ul><li>多目标优化<ul><li>提出LEMONADE进化算法</li></ul></li><li>计算复杂度度高<ul><li>approximate network morphisms</li></ul></li></ul><p><strong>LEMONADE</strong></p><p>多目标：</p><ul><li>expensive-to-evaluate objectives (valid error)</li><li>cheap-to-evaluate objects (model size)</li></ul><p>变异：</p><ul><li><p>network morphism operators (insert convolution, insert skip connection, increase number of filters)</p></li><li><p>approximate network morphism operators (remove layer, prune filters, replace layer)</p></li></ul><p>算法迭代：</p><ol><li><p>基于cheap目标计算概率分布，从该概率分布中采样出父代网络</p></li><li><p>利用NM/ANM变异生成子代网络</p></li><li><p>基于cheap目标计算概率分布，从该概率分布中采样出子代网络</p></li><li><p>对采样出的子代进行评估</p></li><li>利用当前种群和生成的子代计算Pareto front，生成新一代种群</li></ol><p>步骤1.中的概率分布基于一个核密度估计：<br>$$<br>p_{\mathcal{P}}(N)=\frac{c}{p_{K D E}\left(f_{\text {cheap }}(N)\right)}<br>$$<br>​                           </p><p>进行两次采样的原因：变异出的子代网络cheap目标与父代很接近，很可能在$f_{cheap}$密度较低的区域生成子代。</p><h3 id="2️⃣-Deep-Learning-Architecture-Search-by-Neuro-Cell-based-Evolution-with-Function-Preserving-Mutations"><a href="#2️⃣-Deep-Learning-Architecture-Search-by-Neuro-Cell-based-Evolution-with-Function-Preserving-Mutations" class="headerlink" title="2️⃣ Deep Learning Architecture Search by Neuro-Cell-based Evolution with Function-Preserving Mutations"></a>2️⃣ Deep Learning Architecture Search by Neuro-Cell-based Evolution with Function-Preserving Mutations</h3><p>本文主要是Chen et al.的扩展工作，加入了separable conv.</p><p>假设：卷积网络由一系列神经元细胞构成，并且利用function-preserving操作不断进行杂交。</p><p><strong>Function-Preseving</strong></p><p>将teacher network $f$ 转化为student network $g$ 的当且仅当输出保持不变的操作：<br>$$<br>\forall \mathrm{x} : f\left(\mathrm{x} | \boldsymbol{\theta}^{(f)}\right)=g\left(\mathrm{x} | \boldsymbol{\theta}^{(g)}\right)<br>$$</p><ul><li><p>Layer Widening</p><p>将filters数从o扩充到$o’$，扩充部分从已有部分中随机采样。</p></li></ul><p>  $$<br>  V_{\cdot, \cdot, j}^{(l)}=\left\{\begin{array}{ll}{W_{\cdot, \cdot, j}^{(l)},} &amp; {j \leq o} \\ {W_{\cdot, \cdot, r}^{(l)}} &amp; {r \text { uniformly sampled from }\{1, \ldots, o\}}\end{array}\right.<br>  $$<br>  为了保持function-preserving特性，需要在下一层权重也做处理，除以重复数量$n_j$：<br>  $$<br>  V_{\cdot, j,}^{(l+1)}=\frac{1}{n_{j}} W_{\cdot, \cdot, j}^{(l+1)}<br>  $$<br>  本文将这个机制扩展到了separable conv</p><ul><li><p>Layer Deepening：也是扩展到了seprable conv</p></li><li><p>Kernel Widening： padding</p></li><li>Insert Skip Connections：添加权重为0的残差连接</li><li>Branch Layers</li></ul><h3 id="3️⃣-RETHINKING-THE-SMALLER-NORM-LESS-INFORMATIVE-ASSUMPTION-IN-CHANNEL-PRUNING-OF-CONVOLUTION-LAYERS-ICLR-2018"><a href="#3️⃣-RETHINKING-THE-SMALLER-NORM-LESS-INFORMATIVE-ASSUMPTION-IN-CHANNEL-PRUNING-OF-CONVOLUTION-LAYERS-ICLR-2018" class="headerlink" title="3️⃣ RETHINKING THE SMALLER-NORM-LESS-INFORMATIVE ASSUMPTION IN CHANNEL PRUNING OF CONVOLUTION LAYERS [ICLR 2018]"></a>3️⃣ RETHINKING THE SMALLER-NORM-LESS-INFORMATIVE ASSUMPTION IN CHANNEL PRUNING OF CONVOLUTION LAYERS [ICLR 2018]</h3><p>本文分析了regularization可能失败并且适用范围局限的两个点：</p><ol><li><p>Model Reparameterization</p><p>控制网络不同层的权重norm是很难的。需要与reparameterization patterns作斗争。以Lasso为例：<br>$$<br>\min _{\left\{W_{i}\right\}_{i=1}^{2 n}} \mathbb{E}_{(x, y) \sim \mathcal{D}}\left|W_{2 n} _ \ldots _ W_{2} _ W_{1} _ x-y\right|^{2}+\lambda \sum_{i=1}^{n}\left|W_{2 i}\right|_{j}<br>$$<br>由于解并不是唯一的，永远可以找到一个其它参数集合$\left\{W_{i}^{\prime}\right\}_{i=1}^{2 n}$ ，使得total loss更小，但也满足$l_0$norm不变：<br>$$<br>W_{i}^{\prime}=\alpha W_{i}, i=1,3, \ldots, 2 n-1 \text { and } W_{i}^{\prime}=W_{i} / \alpha, i=2,4, \ldots, 2 n<br>$$<br>基于梯度的学习在探索这样的reparameterization patterns时很不高效。</p></li><li><p>Transform Invariance</p><p>batch normalization与weight regularization是不兼容的。<br>$$<br>x^{l+1}=\max \left\{\gamma \cdot \mathrm{BN}_{\mu, \sigma, \epsilon}\left(W^{l} * x^{l}\right)+\beta, 0\right\}<br>$$<br>当对$W^l$进行uniform scaling，由于后续的BN操作，实际上对输出$x^{l+1}$是没有影响的。此外，如果对多层权重一起norm，是不知道如何选择每层合适的penalty。</p></li></ol><p>本文的剪枝策略是通过ISTA算法稀疏BN的$\gamma$。具体算法如下图所示：</p><p><img src="/var/folders/57/3j4d0hq111q_7_xy_7jvjbm40000gn/T/ro.nextwave.Snappy/ro.nextwave.Snappy/B6E153F0-7C45-490E-824D-FC3D5E9B6E50.png" alt="B6E153F0-7C45-490E-824D-FC3D5E9B6E50"></p><p>最主要的三个超参：</p><ol><li>$\mu$ (learning rate)：加速收敛和稀疏。但是过大会导致SGD不收敛。</li><li>$\rho$ (sparse penalty)：收敛时的稀疏率。</li><li>$\alpha$ (rescaling): 用于pre-trained模型的trick，可以让权重的优化慢于$\lambda$的优化。</li></ol><p>本文提了几点超参优化策略：</p><ol><li>检查 cross-enterpy loss 和 regularizatoin loss，选择一个$\rho$使得两个loss在一开始比较接近。</li><li>选择一个合适的学习率。</li><li>如果是pretrained模型，检查模型中 $\lambda$ 的平均数值，选择一个 $\alpha$ 使得 $\gamma^l$ 近似于$100\mu\lambda^l\rho$。</li><li>如果在一开始 regularizatoin loss 线性下降，$\lambda$s 在0周围，可能decrease $\alpha$ and restart；如果$\lambda$s 的稀疏率很快接近100%，可能需要decrease $\rho$ and restart；如果 cross-entropy loss保持不变/快速上升，可能需要decrease $\mu$ or $\rho$ and restart。</li></ol><blockquote><p>感觉和Network slimming相比，并没有什么很大的亮点。除了提出的$\gamma-W$ rescaling tirck，和用ISTA算法来稀疏$\gamma$。</p></blockquote><h3 id="4️⃣-Net2Net-ACCELERATING-LEARNING-VIA-KNOWLEDGE-TRANSFER"><a href="#4️⃣-Net2Net-ACCELERATING-LEARNING-VIA-KNOWLEDGE-TRANSFER" class="headerlink" title="4️⃣ Net2Net: ACCELERATING LEARNING VIA KNOWLEDGE TRANSFER"></a>4️⃣ Net2Net: ACCELERATING LEARNING VIA KNOWLEDGE TRANSFER</h3><p>本文提出了function-preseving transformations，让知识从小网络到大网络迁移。</p><h3 id="5️⃣-Data-Distillation-Towards-Omni-Supervised-Learning"><a href="#5️⃣-Data-Distillation-Towards-Omni-Supervised-Learning" class="headerlink" title="5️⃣ Data Distillation: Towards Omni-Supervised Learning"></a>5️⃣ Data Distillation: Towards Omni-Supervised Learning</h3><p>self-training：在无标签数据上进行预测，并用它们来更新模型。</p><p>本文通过ensemble多个data transformations来进行知识蒸馏。</p><p>self-pruning？</p><ol><li>在labeled data上训练模型</li><li>将讯号的模型应用在使用多个transformations的unlabeld data数据集上</li><li>将unlabeld data的预测结果通过ensemble转为labels</li></ol>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h3 id=&quot;1️⃣-EFFICIENT-MULTI-OBJECTIVE-NEURAL-ARCHITEC-TURE-SEARCH-VIA-LAMARCKIAN-EVOLUTION&quot;&gt;&lt;a href=&quot;#1️⃣-EFFICIENT-MULTI-OBJECTIVE-NEURAL-A
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
  </entry>
  
  <entry>
    <title>模型压缩 | 蒸馏乱炖</title>
    <link href="http://yoursite.com/2019/09/17/%E8%92%B8%E9%A6%8F%E4%B9%B1%E7%82%96/"/>
    <id>http://yoursite.com/2019/09/17/蒸馏乱炖/</id>
    <published>2019-09-17T02:07:11.000Z</published>
    <updated>2020-06-02T14:40:13.448Z</updated>
    
    <content type="html"><![CDATA[<p>参考<a href="https://github.com/HobbitLong/RepDistiller" target="_blank" rel="external">RepDistiller</a>，整理一下目前SOTA的蒸馏方法。</p><ol><li><p><strong>(KD) - Distilling the Knowledge in a Neural Network </strong></p><p>Hinton提的，在温度的作用下，最小化T和S输出概率的交叉熵：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">p_s = F.log_softmax(y_s/T, dim=<span class="number">1</span>)</div><div class="line">p_t = F.softmax(y_t/T, dim=<span class="number">1</span>)</div><div class="line">loss = F.kl_div(p_s, p_t, size_average=<span class="keyword">False</span>) * (T**<span class="number">2</span>) / y_s.shape[<span class="number">0</span>]</div></pre></td></tr></table></figure></li></ol><ol><li><p><strong>(FitNet) - Fitnets: hints for thin deep net,  _ICLR2015_</strong></p><p>可以解决T和S的feature map尺寸不一致的问题，对S的feature map进行变换后，和T的feature map做MSE loss：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">regress_s = ConvReg(feat_s[opt.hint_layer].shape, feat_t[opt.hint_layer].shape)</div><div class="line">f_s = regress_s(feat_s[opt.hint_layer])</div><div class="line">f_t = feat_t[opt.hint_layer]</div><div class="line">loss = F.mse_loss(f_s, f_t)</div></pre></td></tr></table></figure></li></ol><ol><li><p><strong>(AT) - Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer, _ICLR2017_</strong></p><p>把T和S的信息压缩到spatial维度 $\mathcal{F}: R^{C \times H \times W} \rightarrow R^{H \times W}$，最小化它们之间的距离：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">at</span><span class="params">(x)</span>:</span> F.normalize(x.pow(<span class="number">2</span>).mean(<span class="number">1</span>).view(x.size(<span class="number">0</span>), <span class="number">-1</span>))</div><div class="line">loss = (at(f_s) - at(f_t)).pow(<span class="number">2</span>).mean()</div></pre></td></tr></table></figure></li><li><p><strong>(SP) - Similarity-Preserving Knowledge Distillation, _ICCV2019_</strong></p><p>基于AT的改进，让T和S的信息在mini-batch的图像之间尽可能相似：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">bsz = f_s.shape[<span class="number">0</span>]</div><div class="line">f_s = f_s.view(bsz, <span class="number">-1</span>)</div><div class="line">f_t = f_t.view(bsz, <span class="number">-1</span>)</div><div class="line"></div><div class="line">G_s = F.normalize(torch.mm(f_s, torch.t(f_s)))</div><div class="line">G_t = F.normalize(torch.mm(f_t, torch.t(f_t)))</div><div class="line"></div><div class="line">loss = (G_t - G_s).pow(<span class="number">2</span>).view(<span class="number">-1</span>, <span class="number">1</span>).sum(<span class="number">0</span>) / (bsz * bsz)</div></pre></td></tr></table></figure></li></ol><ol><li><p><strong>(CC) - Correlation Congruence for Knowledge Distillation, _ICCV2019_</strong></p><p>在embedded space，不同层的feature map之间存在相关性，让这个相关性保持一致：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">embed_s = LinearEmbed(feat_s[<span class="number">-1</span>].shape[<span class="number">1</span>], opt.feat_dim)</div><div class="line">embed_t = LinearEmbed(feat_t[<span class="number">-1</span>].shape[<span class="number">1</span>], opt.feat_dim)</div><div class="line">delta = torch.abs(embed_s - embed_t)</div><div class="line">loss = torch.mean((delta[:<span class="number">-1</span>] * delta[<span class="number">1</span>:]).sum(<span class="number">1</span>))</div></pre></td></tr></table></figure></li></ol><ol><li><p><strong>(VID) - Variational Information Distillation for Knowledge Transfer, _CVPR2019_</strong></p><p>最大化T和S之间的互信息$I(t ; s)$。这个互信息可以理解为已知学生S的信息，对T中知识不确定性的减少量。最大化互信息即希望这个不确定性减少得尽可能多。由于互信息中的概率分布难以计算，用变分概率进行近似，即最大化activations间的条件概率分布$\mathbb{E}_{t, s}[\log q(\boldsymbol{t} | s)]$。本文采用的$q$是高斯分布（这个过程用了很多log, exp进行数值转换，需要恶补数学）</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">self.log_scale = torch.nn.Parameter(</div><div class="line">np.log(np.exp(init_pred_var-eps)<span class="number">-1.0</span>) * torch.ones(num_target_channels)</div><div class="line">)</div><div class="line"></div><div class="line">pred_mean = self.regressor(f_s)</div><div class="line">pred_var = torch.log(<span class="number">1.0</span>+torch.exp(self.log_scale))+self.eps</div><div class="line">pred_var = pred_var.view(<span class="number">1</span>, <span class="number">-1</span>, <span class="number">1</span>, <span class="number">1</span>)</div><div class="line">neg_log_prob = <span class="number">0.5</span>*(</div><div class="line">(pred_mean-target)**<span class="number">2</span>/pred_var+torch.log(pred_var)</div><div class="line">)</div><div class="line">loss = torch.mean(neg_log_prob)</div></pre></td></tr></table></figure></li></ol><ol><li><p><strong>(RKD) - Relational Knowledge Distillation, _CVPR2019_</strong></p><p>这个中心思想和CC是一致的，都是让T中信息的相对结构在S中也保持一致。但是CC首先对feature做了一个映射，本文提了pairwise的distance loss和ternary的angle loss。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line">student = f_s.view(f_s.shape[<span class="number">0</span>], <span class="number">-1</span>)</div><div class="line">teacher = f_t.view(f_t.shape[<span class="number">0</span>], <span class="number">-1</span>)</div><div class="line"></div><div class="line"><span class="comment"># RKD distance loss</span></div><div class="line">t_d = self.pdist(teacher, squared=<span class="keyword">False</span>)</div><div class="line">mean_td = t_d[t_d &gt; <span class="number">0</span>].mean()</div><div class="line">t_d = t_d / mean_td</div><div class="line"></div><div class="line">d = self.pdist(student, squared=<span class="keyword">False</span>)</div><div class="line">mean_d = d[d &gt; <span class="number">0</span>].mean()</div><div class="line">d = d / mean_d</div><div class="line"></div><div class="line">loss_d = F.smooth_l1_loss(d, t_d)</div><div class="line"></div><div class="line"><span class="comment"># RKD Angle loss</span></div><div class="line"><span class="keyword">with</span> torch.no_grad():</div><div class="line">td = (teacher.unsqueeze(<span class="number">0</span>) - teacher.unsqueeze(<span class="number">1</span>))</div><div class="line">norm_td = F.normalize(td, p=<span class="number">2</span>, dim=<span class="number">2</span>)</div><div class="line">t_angle = torch.bmm(norm_td, norm_td.transpose(<span class="number">1</span>, <span class="number">2</span>)).view(<span class="number">-1</span>)</div><div class="line"></div><div class="line">sd = (student.unsqueeze(<span class="number">0</span>) - student.unsqueeze(<span class="number">1</span>))</div><div class="line">norm_sd = F.normalize(sd, p=<span class="number">2</span>, dim=<span class="number">2</span>)</div><div class="line">s_angle = torch.bmm(norm_sd, norm_sd.transpose(<span class="number">1</span>, <span class="number">2</span>)).view(<span class="number">-1</span>)</div><div class="line"></div><div class="line">loss_a = F.smooth_l1_loss(s_angle, t_angle)</div><div class="line"></div><div class="line">loss = self.w_d * loss_d + self.w_a * loss_a</div><div class="line"></div><div class="line"><span class="keyword">return</span> loss</div></pre></td></tr></table></figure></li></ol><ol><li><p><strong>(PKT) - Learning Deep Representations with Probabilistic Knowledge Transfer, _ECCV2018_</strong></p><p>基于KDE计算features的条件概率分布，利用cosine similarity-based作为密度估计的核。用KL散度让老师和学生信息的条件概率分布相似。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># Normalize each vector by its norm</span></div><div class="line">output_net_norm = torch.sqrt(torch.sum(output_net ** <span class="number">2</span>, dim=<span class="number">1</span>, keepdim=<span class="keyword">True</span>))</div><div class="line">output_net = output_net / (output_net_norm + eps)</div><div class="line">output_net[output_net != output_net] = <span class="number">0</span></div><div class="line"></div><div class="line">target_net_norm = torch.sqrt(torch.sum(target_net ** <span class="number">2</span>, dim=<span class="number">1</span>, keepdim=<span class="keyword">True</span>))</div><div class="line">target_net = target_net / (target_net_norm + eps)</div><div class="line">target_net[target_net != target_net] = <span class="number">0</span></div><div class="line"></div><div class="line"><span class="comment"># Calculate the cosine similarity</span></div><div class="line">model_similarity = torch.mm(output_net, output_net.transpose(<span class="number">0</span>, <span class="number">1</span>))</div><div class="line">target_similarity = torch.mm(target_net, target_net.transpose(<span class="number">0</span>, <span class="number">1</span>))</div><div class="line"></div><div class="line"><span class="comment"># Scale cosine similarity to 0..1</span></div><div class="line">model_similarity = (model_similarity + <span class="number">1.0</span>) / <span class="number">2.0</span></div><div class="line">target_similarity = (target_similarity + <span class="number">1.0</span>) / <span class="number">2.0</span></div><div class="line"></div><div class="line"><span class="comment"># Transform them into probabilities</span></div><div class="line">model_similarity = model_similarity / torch.sum(model_similarity, dim=<span class="number">1</span>, keepdim=<span class="keyword">True</span>)</div><div class="line">target_similarity = target_similarity / torch.sum(target_similarity, dim=<span class="number">1</span>, keepdim=<span class="keyword">True</span>)</div><div class="line"></div><div class="line"><span class="comment"># Calculate the KL-divergence</span></div><div class="line">loss = torch.mean(target_similarity * torch.log((target_similarity + eps) / (model_similarity + eps)))</div><div class="line"></div><div class="line"><span class="keyword">return</span> loss</div></pre></td></tr></table></figure></li></ol><ol><li><p><strong>(AB) - Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons, _AAAI2019_</strong></p><p>本文探索的是权重的初始化方式。通过让T和S中activations大于0值的数量相似，提出了activation transfer loss。这个loss可以直接计算出来是一个常数不可导，于是将其转化为hinge loss，最小化activation transfer loss相当于以teacher的$\rho(x)$作为label，学习一个二值分类器。<br>$$<br>\begin{aligned} \mathcal{L}(\boldsymbol{I})=&amp; | \rho(\mathcal{T}(\boldsymbol{I})) \odot \sigma(\mu \mathbf{1}-\mathcal{S}(\boldsymbol{I})) \\ &amp;+(\mathbf{1}-\rho(\mathcal{T}(\boldsymbol{I}))) \odot \sigma(\mu \mathbf{1}+\mathcal{S}(\boldsymbol{I})) |_{2}^{2} \end{aligned}<br>$$</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">loss = ((source + self.margin) ** <span class="number">2</span> * ((source &gt; -self.margin) &amp; (target &lt;= <span class="number">0</span>)).float() +</div><div class="line">(source - self.margin) ** <span class="number">2</span> * ((source &lt;= self.margin) &amp; (target &gt; <span class="number">0</span>)).float())</div><div class="line">loss = torch.abs(loss).sum()</div></pre></td></tr></table></figure></li></ol><ol><li><p><strong>(FT) - Paraphrasing Complex Network: Network Compression via Factor Transfer, _NIPS2018_</strong></p><p>提出了两个卷积模块paraphraser和translator。前者通过无监督训练（最小化input feature maps和output feature masp）来提取teacher factors，后者用于提取student factors。(这个感觉没啥意思就不去看代码了）</p></li></ol><ol><li><p><strong>(FSP) - A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning, _CVPR2017_</strong></p><p>本文的方法也是用于初始化S权重的。知识由两个不同层的features maps之间的inner product构成，最小化T和S的FSP矩阵。</p></li></ol><ol><li><p><strong>(NST) - Like what you like: knowledge distill via neuron selectivity transfer</strong></p><p>本文提取的知识是neuron selectivity，通过最小化分布之间的Maximum Mean Discrepancy(MMD) metric 进行蒸馏。</p></li><li></li></ol>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;参考&lt;a href=&quot;https://github.com/HobbitLong/RepDistiller&quot; target=&quot;_blank&quot; rel=&quot;external&quot;&gt;RepDistiller&lt;/a&gt;，整理一下目前SOTA的蒸馏方法。&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>Multi-task Learning</title>
    <link href="http://yoursite.com/2019/07/26/Multi-task%20Learning/"/>
    <id>http://yoursite.com/2019/07/26/Multi-task Learning/</id>
    <published>2019-07-26T15:48:10.000Z</published>
    <updated>2020-06-02T14:38:39.712Z</updated>
    
    <content type="html"><![CDATA[<p>### </p><h1 id="Multi-task-Learning"><a href="#Multi-task-Learning" class="headerlink" title="Multi-task Learning"></a>Multi-task Learning</h1><p>多任务同时进行不分主次，多个相关的任务放在一起学习，任务之间的知识共享和共同学习。</p><ul><li><p>Cross-stitch Networks for Multi-task Learning </p><ul><li>提出一个”cross-stitch”单元，学习的<strong>activation</strong>之间的线性映射，期望找到最优的share / task-specific特征组合。</li><li>方法：基于一个AlexNet（one-task network），然后在两个任务上分别进行finetune获得网络A和B，引入corss-stitching。</li><li>数据集和任务：<ul><li>Semantic segmentation（SemSeg）and Surface Normal Prediction（SN）on NYU-v2 </li><li>object detection and attribute prediction on PASCAL VOC 2008</li></ul></li><li>实验：<ul><li>初始值$\alpha$的影响，one-task / ensemble两个网络 / split architecture / MTL-shared</li></ul></li></ul><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190803113347827.png" alt="image-20190803113347827"></p><blockquote><p>只考虑权重，没有考虑网络结构的影响。而且增加了模型大小。</p></blockquote></li><li><p><strong>DAN</strong>: Incremental Learning Through Deep Adaptation (ICLR2018)</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190803113821256.png" alt="image-20190803113821256"></p></li><li><ul><li><p>假设基于的是保持网络结构不变，在T1任务上训练好的网络N，通过改变网络权重能够迁移到T2任务。</p></li><li><p>方法：基于VGG-B，引入controller modules，封装了原先的卷积层，对原始权重做了一下线性变换，每个任务有一个二值变量$\alpha$，控制选择原始权重or新的权重。</p></li><li><p>数据集和任务：Caltech-256, CIFAR-10, Daimler, GTSR, Omniglot, Plankton imagery data, Human Sketch dataset, SVHN</p></li><li><p>实验：</p><ul><li>control-module中W的初始化方式</li><li>base network的选择</li><li>Visual Decathlon Challenge：</li></ul><blockquote><p>不同于cross-stitch的jointly learning，而是one-by-one，在一个网络的基础上训练出另外一个。</p></blockquote></li></ul></li></ul><ul><li><p>Learning multiple visual domains with residual adapters（NIPS2017）</p></li><li><p>Efficient parametrization of multi-domain deep neural networks（CVPR2018）</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190805213652297.png" alt="image-20190805213652297"></p></li></ul><ul><li><ul><li>和👆一篇是同一个作者，不同的是一个上一篇用串行的adapaters，这篇用并行的。</li><li>方法：基于ResNet结构引入residual adapters，引入较少的参数对feature进行了变换。</li><li>实验：<ul><li>adapters的位置（early / mid / late）</li><li>和在各自任务上获得finetune网络进行比较</li></ul></li></ul></li></ul><ul><li>LEARNING TO SHARE: SIMULTANEOUS PARAMETER TYING AND SPARSIFICATION IN DEEP LEARNING （ICLR2018）</li></ul><ul><li><p><strong>Piggyback</strong>: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights</p><p><a href="https://github.com/arunmallya/piggyback" target="_blank" rel="external">https://github.com/arunmallya/piggyback</a></p><ul><li>one-by-one learning</li><li>每个卷积核学习一个mask（剪枝），mask的值为0和1（量化）</li><li>数据集：CUBS / Stanford Cars / WikiArt / Sketch</li><li>方法：不改变pretrained模型的backbone，学习binary mask，让卷积核稀疏以达到适应新数据集的目的。</li></ul><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190807104608124.png" alt="image-20190807104608124"></p></li><li><p><strong>MTAN</strong>: End-to-End Multi-Task Learning with Attention  (cvpr2019)</p><p><a href="https://github.com/lorenmt/mtan" target="_blank" rel="external">https://github.com/lorenmt/mtan</a></p></li></ul><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190803164905002.png" alt="image-20190803164905002"></p><ul><li><ul><li>jointly learning，几个任务是同时训的。</li><li>有一个backbone网络作为task-shared网络（本文采用SegNet），每个任务有对应的attention module。</li><li>提出了DWA，动态平衡loss系数。</li></ul></li></ul><ul><li><p>Nerttailor (cvpr2019)</p><p><a href="https://github.com/pedro-morgado/nettailor" target="_blank" rel="external">https://github.com/pedro-morgado/nettailor</a></p><p><img src="https://github.com/pedro-morgado/nettailor/raw/master/docs/figs/teaser_row.png" alt="img"></p><ul><li><p>one-by-one learning，先在一个任务上训练完迁移到另一个任务上。</p></li><li><p>universarial network是在目标任务上fine-tune一个pre-trained网络，区别在于不仅改变权重，还改变了网络的结构。soft Attention+NAS。</p></li><li><p>动态改变网络结构，backbone是在一个域上训好的ResNet，通过搜索辅助单元。</p></li></ul></li></ul><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190804144750941.png" alt="image-20190804144750941"></p><ul><li><p>Efficient parametrization of multi-domain deep neural networks</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190805171712077.png" alt="image-20190805171712077"></p><ul><li>网络结构相同的网络，第一个卷积层不同（输入不同）</li><li>通过权值共享的方式进行两个域模型的压缩。首先怼了GrOWL(Group weighted order lasso)的方法。</li><li><strong>三个数据集：</strong>SUN-RGBD Dataset（RGB图像和深度图），UCF-101 Dataset（Youtube videos），HMDB-51 Dataset（video）</li><li><strong>两个任务：</strong>RGB-D Scene Classification：Alex-Net，Action Recognition Tasks：VGG-16</li></ul></li></ul><p>weight sharing?</p><p>基于特征</p><ul><li>不同任务特征转换，学习特征之间的线性组合（Cross-stitch、Deep Adaptation</li><li><p>特征选择，稀疏（Group sparsity）</p></li><li><p>分解，低秩分解</p></li></ul><p>基于任务聚类</p><ul><li>加权最近邻分类器。针对每个任务，通过调整权重实现最小化类内距离，最大化类间距离。每个任务之间构建转化矩阵A，其中$a_{ij}$表示使用任务$T_j$的分类器对任务$T_i$样本进行分类的繁花精度。基于矩阵A，将$m$个任务聚成$r$个簇。一个簇里各个任务的样本共享，每个簇训练出一个共同的加权最近邻分类器。</li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;### &lt;/p&gt;
&lt;h1 id=&quot;Multi-task-Learning&quot;&gt;&lt;a href=&quot;#Multi-task-Learning&quot; class=&quot;headerlink&quot; title=&quot;Multi-task Learning&quot;&gt;&lt;/a&gt;Multi-task Learni
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
  </entry>
  
  <entry>
    <title>The Reparameterization Trick</title>
    <link href="http://yoursite.com/2019/07/26/The%20Reparameterization%20Trick/"/>
    <id>http://yoursite.com/2019/07/26/The Reparameterization Trick/</id>
    <published>2019-07-26T15:48:10.000Z</published>
    <updated>2020-06-02T13:29:45.233Z</updated>
    
    <content type="html"><![CDATA[<h3 id="The-Reparameterization-Trick"><a href="#The-Reparameterization-Trick" class="headerlink" title="The Reparameterization Trick"></a>The Reparameterization Trick</h3><p>最近看了很多贝叶斯的方法，重参化技巧是其中的关键，之前在gumbel softmax一块有稍微了解了一下VAE，但感觉还是理解得不深，基于博客<a href="[http://gregorygundersen.com/blog/2018/04/29/reparameterization/">The Reparameterization Trick</a>记录一下。</p><p>先下定义：</p><blockquote><p><strong>Kingma:</strong> This reparameterization is useful for our case since it can be used to rewrite an expectation w.r.t $q_{\phi}(\textbf{z} \mid \textbf{x})$ such that the Monte Carlo estimate of the expectation is differentiable w.r.t.  $\phi$</p></blockquote><p>先验知识：期望的梯度等于梯度的期望<br>$$<br>\begin{aligned} \nabla_{\theta} \mathbb{E}_{p(z)}\left[f_{\theta}(z)\right] &amp;=\nabla_{\theta}\left[\int_{z} p(z) f_{\theta}(z) d z\right] \\ &amp;=\int_{z} p(z)\left[\nabla_{\theta} f_{\theta}(z)\right] d z \\ &amp;=\mathbb{E}_{p(z)}\left[\nabla_{\theta} f_{\theta}(z)\right] \end{aligned}<br>$$<br>问题引入：我们希望解决的问题是当$p$也有参数$\theta$：<br>$$<br>\begin{aligned} \nabla_{\theta} \mathbb{E}_{p_{\theta}(z)}\left[f_{\theta}(z)\right] &amp;=\nabla_{\theta}\left[\int_{z} p_{\theta}(z) f_{\theta}(z) d z\right] \\ &amp;=\int_{z} \nabla_{\theta}\left[p_{\theta}(z) f_{\theta}(z)\right] d z \\ &amp;=\int_{z} f_{\theta}(z) \nabla_{\theta} p_{\theta}(z) d z+\int_{z} p_{\theta}(z) \nabla_{\theta} f_{\theta}(z) d z \\ &amp;=\underbrace{\int_{z} f_{\theta}(z) \nabla_{\theta} p_{\theta}(z) d z}_{\text {What about this? }}  + \mathbb{E}_{p_{\theta}(z)}\left[\nabla_{\theta} f_{\theta}(z)\right]\end{aligned}<br>$$<br>Monte Carlo并不能直接估计第一项，于是引入重参化技巧：<br>$$<br>\begin{align}<br>\boldsymbol{\epsilon} &amp;\sim p(\boldsymbol{\epsilon})<br>\\ \\<br>\textbf{z} &amp;= g_{\boldsymbol{\theta}}(\boldsymbol{\epsilon}, \textbf{x})<br>\\ \\<br>\mathbb{E}_{p_{\boldsymbol{\theta}}(\textbf{z})}[f(\textbf{z}^{(i)})] &amp;= \mathbb{E}_{p(\boldsymbol{\epsilon})} [f(g_{\theta}(\boldsymbol{\epsilon}, \textbf{x}^{(i)}))]<br>\\ \\<br>\nabla_{\theta} \mathbb{E}_{p_{\boldsymbol{\theta}}(\textbf{z})}[f(\textbf{z}^{(i)})] &amp;= \nabla_{\theta} \mathbb{E}_{p(\boldsymbol{\epsilon})} [f(g_{\boldsymbol{\theta}}(\boldsymbol{\epsilon}, \textbf{x}^{(i)}))] &amp;&amp; \text{(1)}<br>\\<br>&amp;= \mathbb{E}_{p(\boldsymbol{\epsilon})} [f(\nabla_{\boldsymbol{\theta}} g_{\boldsymbol{\theta}}(\boldsymbol{\epsilon}, \textbf{x}^{(i)}))] &amp;&amp; \text{(2)}<br>\\<br>&amp;\approx \frac{1}{L} \sum_{l=1}^{L} f(\nabla_{\boldsymbol{\theta}} g_{\boldsymbol{\theta}}(\epsilon^{(l)}, \textbf{x}^{(i)})) &amp;&amp; \text{(3)}<br>\end{align}<br>$$<br>重参化技巧让，因此期望的梯度(1)可以写成梯度的期望(2)，然后又可以用Monte Carlo(3)进行估计。</p><p>在ELBO中，我们有：<br>$$<br>\begin{align}<br>\text{ELBO}(\boldsymbol{\theta}, \boldsymbol{\phi})<br>&amp;= \Big[\mathbb{E}_{q_{\boldsymbol{\phi}}(\textbf{z})}[\log p_{\boldsymbol{\theta}}(\textbf{x}, \textbf{z}) - \log q_{\boldsymbol{\phi}}(\textbf{z} \mid \textbf{x})] \Big] &amp;&amp; \text{(4)} \\<br>&amp;\downarrow \\<br>\nabla_{\theta, \phi} \text{ELBO}(\boldsymbol{\theta}, \boldsymbol{\phi}) &amp;= \underbrace{\nabla_{\theta, \phi} \Big[\mathbb{E}_{q_{\boldsymbol{\phi}}(\textbf{z})}[\log p_{\boldsymbol{\theta}}(\textbf{x}, \textbf{z}) - \log q_{\boldsymbol{\phi}}(\textbf{z} \mid \textbf{x})] \Big]}_{\text{Gradient w.r.t. $\phi$ over expectation w.r.t. $\phi$}}<br>\end{align}<br>$$<br>可以将ELBO分解成两部分的loss：<br>$$<br>\mathcal{L}^B = - \text{KL}[\overbrace{q_{\phi}(\textbf{z} \mid \textbf{x}^{(i)})}^{\text{Encoder}} \lVert \overbrace{p_{\theta}(\textbf{z})}^{\text{Fixed}}] + \frac{1}{L} \sum_{l=1}^{L} \log \overbrace{p_{\boldsymbol{\theta}}(\textbf{x}^{(i)} \mid \textbf{z}^{(l)})}^{\text{Decoder}}<br>$$</p><p>$$<br>\nabla_{\theta, \phi} \mathcal{L}^B = - \nabla_{\theta, \phi} \overbrace{\Bigg[\text{KL}[q_{\phi}(\textbf{z} \mid \textbf{x}^{(i)}) \lVert p_{\theta}(\textbf{z})]\Bigg]}^{\text{Analytically compute this}} + \nabla_{\theta, \phi} \overbrace{\Bigg[ \frac{1}{L} \sum_{l=1}^{L} \Big( \log p_{\boldsymbol{\theta}}(\textbf{x}^{(i)} \mid \textbf{z}^{(l)}) \Big)\Bigg]}^{\text{Monte Carlo estimate this}}<br>$$</p><p>假设先验和估计后验都服从高斯分布：<br>$$<br>\begin{align}<br>\boldsymbol{\mu}_x, \boldsymbol{\sigma}_x &amp;= M(\textbf{x}), \Sigma(\textbf{x}) &amp;&amp; \text{Push $\textbf{x}$ through encoder}<br>\\ \\<br>\boldsymbol{\epsilon} &amp;\sim \mathcal{N}(0, 1) &amp;&amp; \text{Sample noise}<br>\\ \\<br>\textbf{z} &amp;= \boldsymbol{\epsilon} \boldsymbol{\sigma}_x + \boldsymbol{\mu}_x  &amp;&amp; \text{Reparameterize}<br>\\ \\<br>\textbf{x}_r &amp;= p_{\boldsymbol{\theta}}(\textbf{x} \mid \textbf{z}) &amp;&amp; \text{Push $\textbf{z}$ through decoder}<br>\\ \\<br>\text{recon. loss} &amp;= \text{MSE}(\textbf{x}, \textbf{x}_r) &amp;&amp; \text{Compute reconstruction loss}<br>\\ \\<br>\text{var. loss} &amp;= -\text{KL}[\mathcal{N}(\boldsymbol{\mu}_x, \boldsymbol{\sigma}_x) \lVert \mathcal{N}(0, I)] &amp;&amp; \text{Compute variational loss}<br>\\ \\<br>\text{L} &amp;= \text{recon. loss} + \text{var. loss} &amp;&amp; \text{Combine losses}<br>\end{align}<br>$$</p><p><img src="http://gregorygundersen.com/image/reparam/vae.png" alt="img"></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h3 id=&quot;The-Reparameterization-Trick&quot;&gt;&lt;a href=&quot;#The-Reparameterization-Trick&quot; class=&quot;headerlink&quot; title=&quot;The Reparameterization Trick&quot;&gt;&lt;/a&gt;Th
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>每周论文 Vol.10</title>
    <link href="http://yoursite.com/2019/07/26/weekly-paper-10/"/>
    <id>http://yoursite.com/2019/07/26/weekly-paper-10/</id>
    <published>2019-07-26T15:48:10.000Z</published>
    <updated>2019-07-31T13:30:32.838Z</updated>
    
    <content type="html"><![CDATA[<p>### </p><h3 id="1️⃣-Interpretable-and-Fine-Grained-Visual-Explanations-for-Convolutional-Neural-Networks"><a href="#1️⃣-Interpretable-and-Fine-Grained-Visual-Explanations-for-Convolutional-Neural-Networks" class="headerlink" title="1️⃣ Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks"></a>1️⃣ Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks</h3><p><strong>本文是基于干扰项方法的可解释</strong>。可解释的区域$\mathbf{e}^__{C_T}$可以分为<strong>最小保留的区域</strong>和_*最小移除区域__，前者意味着这些区域是保证模型分类正确的部分，后者意味着这些区域必须移除以改变模型输出。</p><p>添加干扰项的图像可以表示为：$\mathbf{e}=\mathbf{m} \cdot \mathbf{x}+(1-\mathbf{m}) \cdot \mathbf{r}$，通过训练使mask稀疏。</p><ul><li><p>保留解释：<br>$$<br>\begin{aligned} \mathbf{e}_{c_{T}}^{_} &amp;=\mathbf{m}_{c_{T}}^{_} \cdot \mathbf{x} \\ \mathbf{m}_{c_{T}}^{*} &amp;=\underset{\mathbf{m}_{c_{T}}}{\arg \min }\left\{\varphi\left(y_{x}^{c_{T}}, y_{e}^{c_{T}}\right)+\lambda \cdot\left|\mathbf{m}_{c_{T}}\right|_{1}\right\} \end{aligned}<br>$$<br>图像中的e区域，保证模型的分类正确。</p></li><li><p>移除解释：<br>$$<br>\begin{aligned} \mathbf{e}_{c_{T}}^{_} &amp;=\mathbf{m}_{c_{T}}^{_} \cdot \mathbf{x} \\ \mathbf{m}_{c_{T}}^{*} &amp;=\underset{\mathbf{m}_{e_{T}}}{\arg \max }\left\{\varphi\left(y_{x}^{c_{T}}, y_{e}^{c_{T}}\right)+\lambda \cdot\left|\mathbf{m}_{c_{T}}\right|_{1}\right\} \end{aligned}<br>$$<br>图像中的e区域，使得模型分类错误。</p></li></ul><h3 id="2️⃣-THE-DEEP-WEIGHT-PRIOR"><a href="#2️⃣-THE-DEEP-WEIGHT-PRIOR" class="headerlink" title="2️⃣ THE DEEP WEIGHT PRIOR"></a>2️⃣ THE DEEP WEIGHT PRIOR</h3><p>【ICLR2019】</p><p>本文的目标是能够通过某个概率分布生成网络的权重。可以看作是增强网络初始化的一种方法。以前贝叶斯神经网络都是需要对参数的先验分布$p(W)$进行假设，通常是log-uniform。<br>$$<br>\mathcal{L}(\theta)=\sum_{i=1}^{N} \mathbb{E}_{q_{\theta}(W)} \log p\left(y_{i} | x_{i}, W\right)-D_{\mathrm{KL}}\left(q_{\theta}(W) | p(W)\right) \rightarrow \max _{\theta}<br>$$<br>VAE是通过隐变量$z_i$估计后验概率分布$q(z_i|x_i)$的方法。其中$x_i$是生成图像。<br>$$<br>\mathcal{L}(\theta, \phi)=\sum_{i=1}^{N} \mathbb{E}_{q_{\theta}\left(z_{i} | x_{i}\right)} \log p_{\phi}\left(x_{i} | z_{i}\right)-D_{\mathrm{KL}}\left(q_{\theta}\left(z_{i} | x_{i}\right) | p\left(z_{i}\right)\right) \rightarrow \max _{\theta, \phi}<br>$$<br>本文的假设是基于预训练的网络参数$\hat{p}_{l}(w)$，参数的先验概率分布为：<br>$$<br>\hat{p}_{l}(w)=\int p\left(w | z ; \phi_{l}\right) p_{l}(z) d z<br>$$<br>引入auxiliary lower bound KL:<br>$$<br>\begin{array}{l}{D_{\mathrm{KL}}(q(W) | \hat{p}(W))=\sum_{l, i, j} D_{\mathrm{KL}}\left(q\left(w_{i j}^{l} | \theta_{i j}^{l}\right) | \hat{p}_{l}\left(w_{i j}^{l}\right)\right) \leq \sum_{l, i, j}\left(-H\left(q\left(w_{i j}^{l} | \theta_{i j}^{l}\right)\right)+\right.} \\ {+\mathbb{E}_{q\left(w_{i j}^{l} | \theta_{i j}^{l}\right)}\left[D_{\mathrm{KL}}\left(r\left(z | w_{i j}^{l} ; \psi_{l}\right) | p_{l}(z)\right)-\mathbb{E}_{r\left(z | w_{i j}^{l} ; \psi_{l}\right)} \log p\left(w_{i j}^{l} | z ; \phi_{l}\right)\right] )=D_{\mathrm{KL}}^{b o u n d}}\end{array}<br>$$<br>这样就和VAE对上，用VAE的encoder估计参数的先验概率（文章假设隐变量$z_i$服从N(0,1)），然后用VAE估计网络参数的先验概率分布，然后从该分布中生成网络的参数。</p><p><img src="https://i.loli.net/2019/07/27/5d3c4f29da39269692.png" alt=""></p><h3 id="3️⃣-DSC-Dense-Sparse-Convolution-for-Vectorized-Inference-of-Convolutional-Neural-Networks"><a href="#3️⃣-DSC-Dense-Sparse-Convolution-for-Vectorized-Inference-of-Convolutional-Neural-Networks" class="headerlink" title="3️⃣ DSC: Dense-Sparse Convolution for Vectorized Inference of Convolutional Neural Networks"></a>3️⃣ DSC: Dense-Sparse Convolution for Vectorized Inference of Convolutional Neural Networks</h3><p>【CVPR2019】</p><p>本文是从很现实的角度做压缩，基于具体的Winograd convolution的压缩方式。</p><p><strong>计算单元向量化</strong>：从现实角度来看，从内存中读取8-bit整形和32-bit浮点型的能耗相同，从i7 CPU读取数据64-bits数据和Altera Arria 10度去32-bit数据的能耗相同。只读取同bit数据(align data)填充寄存器只需要一次操作，同时读取不同bit数据(unaligned data)则需要两次操作。通常CPU数据流缓存块的大小是64bytes (64*8bits)，意味着64x8-bit整形和 16x32-bit的数据可以平行填充寄存器。</p><p><strong>WInograd convolution</strong>：基于Winograd卷积是用更多的加法来减少惩罚操作，2D Winograd Convolution F(2x2, 3x3)的计算公式如下：</p><p><img src="https://i.loli.net/2019/07/28/5d3cfdb637a8236799.png" alt=""></p><p><strong>Dense-Sparse Convolution</strong></p><p><img src="https://i.loli.net/2019/07/28/5d3cfe28d812326769.png" alt=""></p><p>针对Sparse Convolution，把卷积核用CSR格式存放，进行direct sparse convolution。</p><p>针对Sparse-Dense Convolution，作者先通过一个threshold判断的卷积核的稀疏程度，然后用下面的公式进行计算：</p><p><img src="https://i.loli.net/2019/07/28/5d3cff84895ca39428.png" alt=""></p><h3 id="4️⃣-Efficient-Neural-Network-Compression"><a href="#4️⃣-Efficient-Neural-Network-Compression" class="headerlink" title="4️⃣ Efficient Neural Network Compression"></a>4️⃣ Efficient Neural Network Compression</h3><p>【CVPR2019】</p><p>本文的压缩方法是针对卷积核进行低秩分解，目标是找到针对整个网络的最优rank以进行压缩（相当于通道ratio）</p><p><img src="https://github.com/Hyeji-Kim/ENC/raw/master/fig/overall2.png" alt="i"></p><p>两种Layer-wise Accuracy Metrics</p><ul><li><p>PCA energy-based：$y_{p, l}\left(r_{l}\right)$<br>$$<br>y_{p, l}\left(r_{l}\right)=\frac{\sigma_{l}^{\prime}\left(r_{l}\right)-\sigma_{l}^{\prime}(1)}{\sigma_{l}^{\prime}\left(r_{l}^{\max }\right)-\sigma_{l}^{\prime}(1)}<br>$$<br>其中第$l$层的秩是$r_l$，$\sigma_l(d)$是经过分解的第$d$个对角值，$\sigma_{l}^{\prime}\left(r_{l}\right)=\sum^{r_l}_{d=1}\sigma_l(d)$表示卷积核分解后对应秩的元素之和，进行归一化。</p></li><li><p>Measurement-based Metric：$y_{m, l}\left(r_{l}\right)$</p><p>只改变网络层$l$的秩，所获得的整体精度。用VBMF进行秩的采样。</p></li></ul><p>假设每层的metric是独立的，联合概率分布表示网络整体的accuracy metric：<br>$$<br>\mathrm{P}(A ; R)=\prod_{l=1}^{L} \mathrm{P}\left(a_{l} ; r_{l}\right)<br>$$<br>三种Overall accuracy metric：</p><ul><li>Measurement-based：$A_{m}(R)=\prod_{l=1}^{L} y_{m, l}\left(r_{l}\right)$</li><li><p>PCA-based：$A_{p}(R)=\prod_{l=1}^{L} y_{p, l}\left(r_{l}\right)$</p></li><li><p>combied metric：$A_{c}(R)=\left\{A_{p}(R) \times \frac{C(R)}{C_{\text {orig}}}\right\}+A_{m}(R)$</p></li></ul><p><strong>ENC-Map</strong>：利用Accuracy-Complexity的映射来选择每层的rank配置。文章认为让网络每层的具有相同的精度损失与具有相同压缩率相比，是更合理的压缩策略。因此假设在VBMF生成的rank下，每层的metric都相同：<br>$$<br>R_{e}=R | y_{i, l}\left(r_{l}\right)=y_{i, k}\left(r_{k}\right)<br>$$<br>然后我们可以计算出$R_e$的复杂度$C(R)=\sum_{l=1}^{L} C_{l}\left(r_{l}\right)=\sum_{l=1}^{L} c_{l} r_{l}$</p><p>于是有了complexity和accuracy的映射：$f_{C-A}： \mathbb{R} \rightarrow \mathbb{R}$，进一步得到complexity和accuracy到rank的映射：$f_{C-R}：\mathbb{R} \rightarrow \mathbb{R}^L$。</p><p><strong>ENC-Model/Inf</strong>：将扩展ENC-Map至rank的组合问题，需要搜索合适的rank，通过1. 利用已知复杂度来限制 2. 把长得差不多的的卷积核的rank分到一组。</p><h3 id="5️⃣-ECC-Platform-Independent-Energy-Constrained-Deep-Neural-Network-Compression-via-a-Bilinear-Regression-Model"><a href="#5️⃣-ECC-Platform-Independent-Energy-Constrained-Deep-Neural-Network-Compression-via-a-Bilinear-Regression-Model" class="headerlink" title="5️⃣ ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model"></a>5️⃣ ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model</h3><p>本文用限制能耗来进行模型压缩，提出了用一个双线性回归模型来估计target硬件平台的能耗。</p><p>目标用下面的公式表示：<br>$$<br>\begin{array}{cl}{\min _{\mathcal{W}, \mathbf{s}}} &amp; {\ell(\mathcal{W})} \ \\{\text { s.t. }} &amp; {\phi\left(\mathbf{w}^{(u)}\right) \leq s^{(u)}, \quad u \in \mathcal{U}}\\ \ {} &amp; {\mathcal{E}(\mathbf{s}) \leq E_{\text { budget }}}\end{array}<br>$$<br>解决上面问题需要解决稀疏率到能量的映射模型$\mathcal{E}(\mathbf{s})$。用data-driven的方法来训练这个近似模型$\hat{\mathcal{E}}$：<br>$$<br>\hat{\mathcal{E}}=\underset{f \in \mathcal{F}}{\arg \min } \mathbb{E}_{\mathbf{s}}\left[(f(\mathbf{s})-\mathcal{E}(\mathbf{s}))^{2}\right]<br>$$<br>用双线性模型来估计网络整体能耗：<br>$$<br>\mathcal{F} :=\{f(\mathbf{s})=a_{0}+\sum_{j=1}^{|\mathcal{U}|} a_{j} s_{j} s_{j+1} : a_{0}, a_{1}, \ldots, a_{|\mathcal{U}|} \in \mathbb{R}_{+} \}<br>$$<br>ECC整体框架分为两个部分，Online和Offline部分。在Offline部分建立近似能量估计模型$\hat{\mathcal{E}}$</p><p><img src="https://i.loli.net/2019/07/28/5d3d42fd271e241820.png" alt=""></p><p>Online部分基于能量模型进行压缩和ADMM进行压缩。将目标转为minmax优化问题：<br>$$<br>\min _{\mathcal{W}, \mathbf{s}} \max _{z \geq 0, \mathbf{y} \geq \mathbf{0}} \mathcal{L}(\mathcal{W}, \mathbf{s}, \mathbf{y}, z)<br>$$<br>引入对偶变量$y$和$z$用于限制稀疏率，引入z用于限制能量：<br>$$<br>\mathcal{L}(\mathcal{W}, \mathbf{s}, \mathbf{y}, z) \quad :=\ell(\mathcal{W})+\mathcal{L}_{1}(\mathcal{W}, \mathbf{s}, \mathbf{y})+\mathcal{L}_{2}(\mathbf{s}, z)<br>$$<br>其中$\mathcal{L}_{1}(\mathcal{W}, \mathbf{s}, \mathbf{y}) \quad :=\quad \frac{\rho_{1}}{2} \sum_{u}\left[\phi\left(\mathbf{w}^{(u)}\right)-s^{(u)}\right]_{+}^{2}+\sum_{u} y^{(u)}\left(\phi\left(\mathbf{w}^{(u)}\right)-s^{(u)}\right), \mathcal{L}_{2}(\mathbf{s}, z)$</p><p>$\mathcal{L}_{2}(\mathbf{s}, z) :=\frac{\rho_{2}}{2}\left[\hat{\mathcal{E}}(\mathbf{s})-E_{\mathrm{budget}}\right]_{+}^{2}+z\left(\hat{\mathcal{E}}(\mathbf{s})-E_{\text { budget }}\right)$</p><p>算法通过迭代更新参数来达到最终目标</p><ul><li>Update $W$：用Proximal Adam</li><li><p>Update $s$：$\mathbf{s}^{t+1}=\mathbf{s}^{t}-\beta\left(\nabla_{\mathbf{s}} \mathcal{L}_{1}\left(\mathcal{W}, \mathbf{s}^{t}, \mathbf{y}\right)+\nabla_{\mathbf{s}} \mathcal{L}_{2}\left(\mathbf{s}^{t}, z\right)\right)$</p></li><li><p>Update 对偶变量：$\begin{aligned} y^{(u)^{t+1}} &amp;=\left[y^{(u)^{t}}+\rho_{1}\left(\phi\left(\mathbf{w}^{(u)}\right)-s^{(u)}\right)\right]_{+} \\ z^{t+1} &amp;=\left[z^{t}+\rho_{2}\left(\hat{\mathcal{E}}(\mathbf{s})-E_{\mathrm{budget}}\right)\right]_{+} \end{aligned}$</p></li></ul><h3 id="NETTAILOR-Tuning-the-architecture-not-just-the-weights"><a href="#NETTAILOR-Tuning-the-architecture-not-just-the-weights" class="headerlink" title="NETTAILOR: Tuning the architecture, not just the weights"></a>NETTAILOR: Tuning the architecture, not just the weights</h3><p>这篇文章很有意思，不止fintune网络权重，还FT网络结构。目前大部分网络使用的是相同的backbone，没有考虑到网络结构本身的影响。可能小一些的网络在目标数据集上就足够了。本文将pre-trained的backbone网络结构为universal blocks，加上一些task-specific网络来生成新的网络。通过soft-attention机制和网络的复杂度限制来学习新的网络结构和权重。</p><p>一些相关工作包括迁移学习、多任务学习（增强任务之间的泛化性），迁移学习假设图像来自不同的域，MTL假设所有任务是处于同域的。Domain adaptation解决两个不同域数据集的任务。Cascaded classifiers &amp; Adaptive inference graphs能够自动调整网络的拓扑结构。但是针对不同的任务要训练不同的网络，NETTAILOR通过重用universal blocks，只训练task相关的block来解决multi-domain transfer learning problems。</p><p>算法主要分成以下四步：1. 在目标任务上用pre-trained网络训练一个teacher network。2. 定义包括proxy layers的学生网络。3. 在目标任务上只训练task-specific参数，同时加上复杂度限制。4. 精简网络结构后进行finetune。</p><p><img src="https://github.com/pedro-morgado/nettailor/raw/master/docs/figs/teaser_row.png" alt="img"></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;### &lt;/p&gt;
&lt;h3 id=&quot;1️⃣-Interpretable-and-Fine-Grained-Visual-Explanations-for-Convolutional-Neural-Networks&quot;&gt;&lt;a href=&quot;#1️⃣-Interpretable-an
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>Approximated Oracle FIlter Pruning</title>
    <link href="http://yoursite.com/2019/07/24/Approximated%20Oracle%20FIlter%20Pruning/"/>
    <id>http://yoursite.com/2019/07/24/Approximated Oracle FIlter Pruning/</id>
    <published>2019-07-24T15:48:10.000Z</published>
    <updated>2020-06-02T14:38:30.860Z</updated>
    
    <content type="html"><![CDATA[<p>### </p><p>本文提出了AOFP框架，通过binary search搜索下一个剪枝filters，并且同时finetune模型。</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20191102155014824.png" alt="image-20191102155014824"></p><p>每个网络层$i$，利用前一层的输出和当前层的filters做如下映射：<br>$$<br>\boldsymbol{M}^{(i)}=\zeta^{(i)}\left(\boldsymbol{M}^{(i-1)}, \mathcal{F}_{i}\right)<br>$$<br>用$T$来代表filter的重要性，oracle的标准：<br>$$<br>T(\boldsymbol{F})=\sum_{(x, y) \in(X, Y)}(L(x, y, \mathcal{F}-\boldsymbol{F})-L(x, y, \mathcal{F}))<br>$$<br>每次只砍一个filters太耗时，引入剪枝粒度$g$，对少数filters一起剪枝。</p><h3 id="存在问题："><a href="#存在问题：" class="headerlink" title="存在问题："></a>存在问题：</h3><ol><li>Oracle Pruning的反馈机制太耗时 </li><li>每次feedback只能判断一层的重要性。 </li></ol><p>需要寻找方案缩短反馈循环，还要能够并行的独立判断每层的重要性。</p><h3 id="解决问题："><a href="#解决问题：" class="headerlink" title="解决问题："></a>解决问题：</h3><ol><li><p>缩短反馈循环，提出了<strong>Damage isolation</strong>：将CNN可以看作一个状态机，$i$层的改变不能被$i+2$层看到，由于$i+1$层的隔离。因此将fiilters的重要性$T’$设为对$i+1$输出的影响：<br>$$<br>T^{\prime}(\boldsymbol{F})=\frac{1}{|X|} \sum_{x \in X} t(\boldsymbol{F}, x)<br>$$</p><p>$$<br>t(\boldsymbol{F}, x)=\frac{\left|\boldsymbol{M}^{(i+1)}(x)-\zeta^{(i+1)}\left(\boldsymbol{M}_{\boldsymbol{F}}^{(i)}(x), \mathcal{F}^{(i+1)}\right)\right|_{2}^{2}}{\left|\boldsymbol{M}^{(i+1)}(x)\right|_{2}^{2}}<br>$$</p><p>$t(\boldsymbol{F}, x)$反映了对$i$层进行剪枝对$i+1$层输出的影响。</p><p>用上面那张图片理解，我们对conv1进行剪枝，用conv2的输出来计算conv1的重要性。</p></li><li><p>比较常见地是score完finetune（串行），本文的提出的框架并行地进行score和finetune。</p><p>对$i$层进行剪枝，之后将计算流分成两支base path和score path。</p><ul><li>bath path包含mask $u$，用这个部分来计算loss，更新网络参数</li><li>score path包含mask $v$，用这个部分计算$t$</li></ul><p>输入batch data，基于base path，在score path随机将一些filters置为0。对比base和score path的数值。</p><blockquote><p>由于在score path计算分数，其实不影响base path的输出，所以也不影响更高level的fiters。</p></blockquote><p>当所有filters都被打分以以后，将$g$个最低$T$值的filters永久置0。</p><p>本文将处理一个或多个filters的过程称为move操作。</p></li></ol><p>算法流程👇</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20191104142609953.png" alt="image-20191104142609953"></p><h3 id="具体操作："><a href="#具体操作：" class="headerlink" title="具体操作："></a>具体操作：</h3><p>作者针对自己提出的框架提出了几点需要解决的问题:</p><ol><li>针对特定层，由于finetune和prune两个操作是同时进行的，每次finetune的filters和prune的filters是不同的，导致每个filter的分数是不准确的。</li><li><p>不能很好地解决每次的剪枝粒度$g$。</p></li><li><p>不知道啥时候终止剪枝。</p></li></ol><p>一点点的来攻破：</p><ol><li>把对所有filters rank的操作利用二分搜索的方式进行近似，由<strong>粗到细</strong>的进行搜索。针对特定层：首先由保留的filters构成搜索空间$A$，在每次迭代中随机选择$|A|/2$ 的filters进行剪枝，finetune并收集$t$，总共迭代$\phi$个batches。对收集到的$\hat{T}$取均值近似作为每个filters的重要性。选择$\hat{T}$最小的$|A|/2$构成picked set $B$。然后缩小搜索空间$A \leftarrow B$。直到$B$为0或者max damage 小于某个阈值。如果max damage小于阈值，说明该层还可以继续剪。如果max damge大于某个阈值并且剩余通道为1时，说明该层就不能剪了。然后改变$u$，进行静态剪枝。</li><li>剪枝粒度$g$的问题其实包含在上述算法中，当$B$足够好时，结束某层的搜索即$g=B$</li></ol><p>Pros: </p><ul><li>并行操作每层的思想很好，因此可以在每层进行独立地二分搜索，用于决定which filters are important。</li></ul><p>Cons:</p><ul><li>本文自己也说了之前的filters会对之后的filters产生影响，那它提出的只计算下一层输出的damage还合理吗？(但是不这样就没办法并行了啊）</li><li>用二分搜索的方式可以确定每次的剪枝粒度，但是这个粒度其是受到一个全局超参$\theta$进行控制的，说明每层都一样？感觉这个超参很难进行选择。如何改进？</li></ul><blockquote><p>剪枝其实要解决的两个问题就是每次剪哪里，每次剪多少。剪哪里即代表了filters的重要性。剪多少则为剪枝粒度。本文把层和层分离开来，基于层内进行重要性的判断。先判断该剪哪些filters，慢慢剪掉最该剪掉的那些，在每次剪枝的时候通过一个全局超参来控制每次剪枝的粒度。</p></blockquote>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;### &lt;/p&gt;
&lt;p&gt;本文提出了AOFP框架，通过binary search搜索下一个剪枝filters，并且同时finetune模型。&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;/Users/colorjam/Library/Application Support/typora
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
  </entry>
  
  <entry>
    <title>进化策略</title>
    <link href="http://yoursite.com/2019/07/20/evolution-strategies/"/>
    <id>http://yoursite.com/2019/07/20/evolution-strategies/</id>
    <published>2019-07-19T23:44:06.000Z</published>
    <updated>2019-07-20T00:25:32.893Z</updated>
    
    <content type="html"><![CDATA[<p><a href="http://blog.otoro.net/2017/10/29/visual-evolution-strategies/" target="_blank" rel="external">这篇博客</a></p><p>进化策略是一种黑箱优化算法，防止参数陷入局部最优解。进化策略可以看作一种提供一系列候选解决方案来评估一个问题的算法。评估结果基于一个目标函数(objective function)，一个解决方案返回一个适应度(fitness value)，基于当前解决方案的适应度，再生成下一集合的候选者。最简单的伪代码如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div></pre></td><td class="code"><pre><div class="line">solver = EvolutionStrategy()</div><div class="line"></div><div class="line">while True:</div><div class="line"></div><div class="line">  # 请求ES生成候选者</div><div class="line">  solutions = solver.ask()</div><div class="line"></div><div class="line">  # 初始化适应度</div><div class="line">  fitness_list = np.zeros(solver.popsize)</div><div class="line"></div><div class="line">  # 评估候选者，生成与其对应的适应度</div><div class="line">  for i in range(solver.popsize):</div><div class="line">    fitness_list[i] = evaluate(solutions[i])</div><div class="line"></div><div class="line">  # 返回适应度结果给ES</div><div class="line">  solver.tell(fitness_list)</div><div class="line"></div><div class="line">  # 从ES中获取最优参数和最优适应度</div><div class="line">  best_solution, best_fitness = solver.result()</div><div class="line"></div><div class="line">  if best_fitness &gt; MY_REQUIRED_FITNESS:</div><div class="line">    break</div></pre></td></tr></table></figure><p>以简单的2D问题为例，参数由$\mu=\left(\mu_{x}, \mu_{y}\right)$和$\sigma=\left(\sigma_{x}, \sigma_{y}\right)$组成Simple ES和Simple GA都是固定$\sigma$不变，通过进化算法学习$\mu$，由此CMA-ES算法诞生。它是一种不基于梯度的算法，通过计算所有参数空间的协方差矩阵，在每次迭代时从多元正态分布中采样解决方案。</p><p>上面提到的一些算法只保留了最优解，忽略了其他解决方案，因此也可能忽略大部分对生成下一代有用的信息。结合RL算法，有人提出了REINFORCE-ES以及NES，遵循的原则是不论好坏综合所有候选者以估计梯度，往梯度方向更新参数。核心思想是<strong>最大化采样候选者的适应度期望值</strong>：<br>$$<br>J(\theta)=E_{\theta}[F(z)]=\int F(z) \pi(z, \theta) d z<br>$$<br><a href="http://www.jmlr.org/papers/volume15/wierstra14a/wierstra14a.pdf" target="_blank" rel="external">NES</a>提供了梯度的推导，利用log-likelihood trick和蒙特卡洛采样可以得到：<br>$$<br>\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} F\left(z^{i}\right) \nabla_{\theta} \log \pi\left(z^{i}, \theta\right)<br>$$<br><a href="http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf" target="_blank" rel="external">REINFORCE</a>为特殊的案例，当$\pi(z, \theta)$是一个factored multi-variate normal distribution，即每个参数服从一个一元正态分布$z_{j} \sim N\left(\mu_{j}, \sigma_{j}\right)$，给出了一个梯度的封闭解：<br>$$<br>\begin{array}{l}{\nabla_{\mu_{j}} \log N\left(z^{i}, \mu, \sigma\right)=\frac{z_{j}^{i}-\mu_{j}}{\sigma_{j}^{2}}} \\ {\nabla_{\sigma_{j}} \log N\left(z^{i}, \mu, \sigma\right)=\frac{\left(z_{j}^{i}-\mu_{j}\right)^{2}-\sigma_{j}^{2}}{\sigma_{j}^{3}}}\end{array}<br>$$<br>这些论文提出了一些tricks，比如PEPG中的antithetic sampling，NES中利用Fisher Information更新梯度等等。</p><p>在OenAI的<a href="https://blog.openai.com/evolution-strategies/" target="_blank" rel="external">论文</a>里，它们固定$\sigma$，只更新$\mu$，主要是解决执行层面的并行运算问题。</p><p>通常进化策略都会采样一个称为Fitness Shaping的trick，把种群适应度转化为种群内部的相对值，即rank一下fitness，保证评估指标的不变性。</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;&lt;a href=&quot;http://blog.otoro.net/2017/10/29/visual-evolution-strategies/&quot; target=&quot;_blank&quot; rel=&quot;external&quot;&gt;这篇博客&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;进化策略是一种黑箱优化算法，防止参数
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>每周论文 Vol.09</title>
    <link href="http://yoursite.com/2019/07/08/weekly-paper-09/"/>
    <id>http://yoursite.com/2019/07/08/weekly-paper-09/</id>
    <published>2019-07-08T01:07:11.000Z</published>
    <updated>2019-07-17T13:39:48.882Z</updated>
    
    <content type="html"><![CDATA[<h3 id="1️⃣-Natural-Evolution-Strategies"><a href="#1️⃣-Natural-Evolution-Strategies" class="headerlink" title="1️⃣ Natural Evolution Strategies"></a>1️⃣ Natural Evolution Strategies</h3><p>NES是一种利用搜索梯度(search gradients)更新搜索分布参数(parameters of the search distribution)的黑箱优化算法，与经典方法（EDAs）利用最大似然拟合采样分布的方法不同，本文提出的更新策略是沿着更高期望适应度的方向。</p><p>最大化所采样样本的适应度的期望值。核心思想就是在每次新的种群里更新mean和std，从新的结果中继续更新</p><h4 id="0-Search-Gradients"><a href="#0-Search-Gradients" class="headerlink" title="0/ Search Gradients"></a>0/ Search Gradients</h4><p>假设我们从分布$\pi(\mathbf z | \theta)$中采样$\mathbf z$，用$f(\mathbf z)$表示该采样的适应度。在该搜索分布下的期望适应度为：<br>$$<br>J(\theta)=\mathbb{E}_{\theta}[f(\mathbf{z})]=\int f(\mathbf{z}) \pi(\mathbf{z} | \theta) d \mathbf{z}<br>$$<br>利用 <a href="http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/" target="_blank" rel="external">log-likelihood trick</a>：<br>$$<br>\begin{aligned} \nabla_{\theta} J(\theta) &amp;=\nabla_{\theta} \int f(\mathbf{z}) \pi(\mathbf{z} | \theta) d \mathbf{z} \\ &amp;=\int f(\mathbf{z}) \nabla_{\theta} \pi(\mathbf{z} | \theta) d \mathbf{z} \\ &amp;=\int f(\mathbf{z}) \nabla_{\theta} \pi(\mathbf{z} | \theta) \frac{\pi(\mathbf{z} | \theta)}{\pi(\mathbf{z} | \theta)} d \mathbf{z} \\ &amp;=\int\left[f(\mathbf{z}) \nabla_{\theta} \log \pi(\mathbf{z} | \theta)\right] \pi(\mathbf{z} | \theta) d \mathbf{z} \\ &amp;=\mathbb{E}_{\theta}\left[f(\mathbf{z}) \nabla_{\theta} \log \pi(\mathbf{z} | \theta)\right] \end{aligned}<br>$$<br>利用种群大小$\lambda$对搜索梯度进行近似估计：<br>$$<br>\nabla_{\theta} J(\theta) \approx \frac{1}{\lambda} \sum_{k=1}^{\lambda} f\left(\mathbf{z}_{k}\right) \nabla_{\theta} \log \pi\left(\mathbf{z}_{k} | \theta\right)<br>$$<br>$\nabla_{\theta} J(\theta)$提供了梯度上升的方向，最直接的方法参数更新方法：$\theta \leftarrow \theta+\eta \nabla_{\theta} J(\theta)$</p><blockquote><p>我们希望最大化期望值，因此是梯度上升。</p></blockquote><p>这样，我们可以写出标准搜索梯度算法：</p><p><img src="https://i.loli.net/2019/07/08/5d229d3f6943c17863.png" alt=""></p><p>以多元正态分布为例，$\theta=\langle\boldsymbol{\mu}, \boldsymbol{\Sigma}\rangle$为需要学习的分布参数。我们还需要满足$\mathbf{A}^{\top} \mathbf{A}=\mathbf{\Sigma}$的协方差矩阵的平方根矩阵$\mathbf{A} \in \mathbb{R}^{d \times d}$，使得$\mathbf{z}=\boldsymbol{\mu}+\mathbf{A}^{\top} \mathbf{s}$将标准正态分布$\mathbf{s} \sim \mathcal{N}(0, \mathbb{I})$转化为种群个体$\mathbf{z} \sim \mathcal{N}(\boldsymbol{\mu}, \mathbf{\Sigma})$，需要对$\boldsymbol{\mu}, \mathbf{\Sigma}$求导，更新算法为：</p><p><img src="https://i.loli.net/2019/07/08/5d229f6ae0f3061548.png" alt=""></p><h4 id="1-Limitations"><a href="#1-Limitations" class="headerlink" title="1/ Limitations"></a>1/ Limitations</h4><p>针对普通搜索梯度算法的局限，本文提出的解决办法可以用下表总结：</p><p><img src="https://i.loli.net/2019/07/08/5d2299a8b796f92761.png" alt=""></p><p>这些问题和方法让我们来一个个攻破。</p><p><strong>a. Natural gradient</strong></p><p>令$\lambda=1, d=1$，$\mu \leftarrow \mu+\eta \frac{z-\mu}{\sigma^{2}},\quad \sigma \leftarrow \sigma+\eta \frac{(z-\mu)^{2}-\sigma^{2}}{\sigma^{3}}$</p><p>由于$\Delta \mu \propto \frac{1}{\sigma}, \Delta \sigma \propto \frac{1}{\sigma}$，$\sigma$同时控制了$\mu$和$\sigma$的更新，造成参数的更新不是尺度不变(scale-invariant)的：我们减小$\sigma$让$\mu$接近最优解的同时也增大了$\sigma$，使其在参数更新的时候再次远离最优解。</p><p>自然梯度的提出便是为了解决尺度不变的问题。原始梯度测量的是参数分布的欧拉距离。自然梯度利用的是参数分布的KL散度。<br>$$<br>\begin{aligned} \max _{\delta \theta} J(\theta+\delta \theta) &amp; \approx J(\theta)+\delta \theta^{\top} \nabla_{\theta} J \\ \text {s.t. } D(\theta+\delta \theta | \theta) &amp;=\varepsilon, \end{aligned}<br>$$<br>$J(\theta)$仍然是期望适应度，$\varepsilon$是一个很小的增量，通过二阶泰勒展开$\lim \delta \theta \rightarrow 0$，$D(\theta+\delta \theta | \theta)=\frac{1}{2} \delta \theta^{\top} \mathbf{F}(\theta) \delta \theta$。其中<br>$$<br>\begin{aligned} \mathbf{F} &amp;=\int \pi(\mathbf{z} | \theta) \nabla_{\theta} \log \pi(\mathbf{z} | \theta) \nabla_{\theta} \log \pi(\mathbf{z} | \theta)^{\top} d \mathbf{z} \\ &amp;=\mathbb{E}\left[\nabla_{\theta} \log \pi(\mathbf{z} | \theta) \nabla_{\theta} \log \pi(\mathbf{z} | \theta)^{\top}\right] \end{aligned}<br>$$<br>表示为费雪信息矩阵(Fisher information matrix)。若$ \mathbf{F}$可逆，则自然梯度表示为：<br>$$<br>\widetilde{\nabla}_{\theta} J=\mathbf{F}^{-1} \nabla_{\theta} J(\theta)<br>$$</p><blockquote><p>推导可以参考<a href="https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/" target="_blank" rel="external">blog</a></p></blockquote><p>然后我们就可以写出标准自然演化策略算法：</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190708145437180.png" alt="image-20190708145437180"></p><p>与算法1的区别就在于，更新的时候梯度乘以了一个费雪信息矩阵的逆。</p><p><strong>b. Fitness shaping</strong></p><p>用效用函数(utility)替an代适应度：<br>$$<br>\nabla_{\theta} J(\theta)=\sum_{k=1}^{\lambda} u_{k} \nabla_{\theta} \log \pi\left(\mathbf{z}_{k} | \theta\right)<br>$$<br>本文定义的效用函数为：<br>$$<br>u_{k}=\frac{\max \left(0, \log \left(\frac{\lambda}{2}+1\right)-\log (k)\right)}{\sum_{j=1}^{\lambda} \max \left(0, \log \left(\frac{\lambda}{2}+1\right)-\log (j)\right)}-\frac{1}{\lambda}<br>$$<br><strong>c. Adapation Sampling</strong></p><p>用质量函数判断，当前采样$\mathbf {z’}$很大程度上优于之前的采样$\mathbf {z}$，才进行参数更新。与单纯的最大化适应度函数本身不同，适应采样最大化的是进步的步伐(pace of progress)。以最重要的参数学习率$\eta_{\sigma}$为例：</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190708163252563.png" alt="image-20190708163252563"></p><p>首先利用$1.5\eta_{\sigma, t}$和$\theta_{t-1}$计算出假想参数$\theta ‘$，计算每个个体的权重$w_k ‘$，用Mann-Whitney test检验两个质量函数，当大于阈值$\rho=\frac{1}{2}-\frac{1}{3(d+1)}$时，增加学习率，否则让它靠近初始的学习率。</p><p><strong>d. Exponential parameterization &amp; Natural coordinate system</strong></p><p>NES不直接更新协方差$\mathbf \Sigma$，考虑以它的平方根$\mathbf A$为参数的正态分布，利用自然梯度进行更新。为了避免估计Fisher矩阵，xNES使用了局部坐标系和指数映射。局部坐标系下的自然梯度为：<br>$$<br>\begin{aligned} \nabla_{\boldsymbol{\delta}} J &amp;=\sum_{k=1}^{\lambda} f\left(\mathbf{z}_{k}\right) \cdot \mathbf{s}_{k} \\ \nabla_{\mathbf{M}} J &amp;=\sum_{k=1}^{\lambda} f\left(\mathbf{z}_{k}\right) \cdot\left(\mathbf{s}_{k} \mathbf{s}_{k}^{\top}-\mathbb{I}\right) \end{aligned}<br>$$<br>其中$\mathbf {s}_k$是局部坐标系下第$k$个最优样本，$\mathbf {z}_k$是目标2坐标系下的相同样本。把协方差因子$\mathbf{A}$分解为步长$\sigma &gt;0$和满足$\operatorname{det}(\mathbf{B})=1$的归一化协方差因子$\mathbf{B}$。这种分解使得两个正交成分可以有各自的学习率（$\eta_{\sigma}$, $\eta_{\mathbf {B}}$），对步长$\sigma$和$B$的更新做了指数映射。</p><p><img src="https://i.loli.net/2019/07/08/5d230b687c16942920.png" alt=""></p><p>本文还提了separable NES(SNES)，使用分离的搜索分布减少计算复杂度。本文的实现方式是限制$\mathbf {A}$为可逆的对角变换矩阵。</p><p><img src="https://i.loli.net/2019/07/08/5d230b390673572266.png" alt=""></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h3 id=&quot;1️⃣-Natural-Evolution-Strategies&quot;&gt;&lt;a href=&quot;#1️⃣-Natural-Evolution-Strategies&quot; class=&quot;headerlink&quot; title=&quot;1️⃣ Natural Evolution Strate
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
      <category term="每周论文" scheme="http://yoursite.com/tags/%E6%AF%8F%E5%91%A8%E8%AE%BA%E6%96%87/"/>
    
  </entry>
  
  <entry>
    <title>手撕PyTorch的Batch Normalization</title>
    <link href="http://yoursite.com/2019/07/02/batch-normalization/"/>
    <id>http://yoursite.com/2019/07/02/batch-normalization/</id>
    <published>2019-07-02T00:35:20.000Z</published>
    <updated>2019-07-05T09:03:38.738Z</updated>
    
    <content type="html"><![CDATA[<p>BN是防网络过拟合的一个很重要的模块，细微的差别可能对输出效果有很大影响。因此需要理解一下PyTorch中BN的具体实现。PyTorch的源码用C实现的torch.batchnorm。可以<a href="https://github.com/marvis/pytorch-yolo2/blob/master/layers/batchnorm/src/batchnorm.c" target="_blank" rel="external">yolo2</a>里找到具体实现。</p><h3 id="计算过程"><a href="#计算过程" class="headerlink" title="计算过程"></a>计算过程</h3><p>$Input : \mathcal{B}=\left\{x_{1}, \cdots, x_{m}\right\}$ 表示batch_size为$m$的输入数据。</p><p>$Output:  \gamma ,  \beta$   PyTorch中为weights和bias。</p><p><strong>更新过程：</strong></p><p>$\mu_{\mathcal{B}} \leftarrow \frac{1}{m} \sum_{i=1}^{m} x_{i}$</p><p>$\sigma_{\mathcal{B}}^{2} \leftarrow \frac{1}{m} \sum_{i=1}^{m}\left(x_{i}-\mu_{\mathcal{B}}\right)^{2}$</p><p>$\hat{x}_{i} \leftarrow \frac{x_{i}-\mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^{2}+\epsilon}}$</p><p>$y_{i} \leftarrow \gamma \hat{x}_{i}+\beta \equiv \mathrm{B} \mathrm{N}_{\gamma, \beta}\left(x_{i}\right)$</p><p>首先通过代码测试一下具体更新过程的实现。</p><p><strong>测试输出：</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line">x = torch.range(<span class="number">0</span>, <span class="number">35</span>).reshape(<span class="number">3</span>, <span class="number">3</span>, <span class="number">2</span>, <span class="number">2</span>)</div><div class="line"></div><div class="line"><span class="comment"># initilization</span></div><div class="line">bn = nn.BatchNorm2d(<span class="number">3</span>)</div><div class="line">bn.weight.data.fill_(<span class="number">1</span>)</div><div class="line">bn.bias.data.zero_()</div><div class="line">mean, new_mean = torch.zeros([<span class="number">3</span>]), torch.zeros([<span class="number">3</span>])</div><div class="line">var, new_var = torch.ones([<span class="number">3</span>]), torch.zeros([<span class="number">3</span>])</div><div class="line"></div><div class="line"><span class="comment"># compute mean</span></div><div class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">3</span>):</div><div class="line">  new_mean[i] = x[:, i, :, :].mean()</div><div class="line"></div><div class="line"><span class="comment"># compute variance</span></div><div class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">3</span>):</div><div class="line">  new_var[i] = torch.pow((x[:, i, :, :] - new_mean[i]), <span class="number">2</span>).mean()</div><div class="line">  <span class="comment"># 等同于 new_var[i] = x[:, i, :, :].var(False) </span></div><div class="line">  <span class="comment"># 计算方差时不使用贝塞尔校正</span></div><div class="line">  </div><div class="line">normalized_x = torch.zeros_like(x)</div><div class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">3</span>):</div><div class="line">  normalized_x[:, i, :, :] = (x[:, i, :, :] - new_mean[i]) / torch.sqrt(new_var[i] + bn.eps)</div><div class="line">  </div><div class="line">print(bn_x)</div><div class="line">print(normalized_x)</div></pre></td></tr></table></figure><p><strong>测试runing_mean和running_var：</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># moving average</span></div><div class="line">running_mean = bn.momentum * new_mean + (<span class="number">1</span> - bn.momentum) * mean</div><div class="line">running_var = bn.momentum * new_var + (<span class="number">1</span> - bn.momentum) * var</div><div class="line"></div><div class="line">print(bn.running_mean, bn.running_var)</div><div class="line">print(running_mean, running_var)</div></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="meta">&gt;&gt;&gt; </span>[<span class="number">1.3500</span>, <span class="number">1.7500</span>, <span class="number">2.1500</span>] [<span class="number">11.5091</span>, <span class="number">11.5091</span>, <span class="number">11.5091</span>]</div><div class="line"><span class="meta">&gt;&gt;&gt; </span>[<span class="number">1.3500</span>, <span class="number">1.7500</span>, <span class="number">2.1500</span>] [<span class="number">10.6250</span>, <span class="number">10.6250</span>, <span class="number">10.6250</span>]</div></pre></td></tr></table></figure><p>running_mean对上了，可是running_var却不对。仔细看了一下源代码在函数<code>variance_cpu</code>中注释掉了一句<code>float scale = 1./(batch * spatial - 1)</code>，这句就很关键了。修改一下上面var的计算方式：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">scale = x.size(<span class="number">0</span>) * x.size(<span class="number">2</span>) * x.size(<span class="number">3</span>)</div><div class="line"></div><div class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">3</span>):</div><div class="line">  new_var[i] = x[:, i, :, :].var(<span class="keyword">False</span>) </div><div class="line">  </div><div class="line">new_var = new_var * scale / (scale - <span class="number">1</span>)</div><div class="line">running_var = bn.momentum * new_var + (<span class="number">1</span> - bn.momentum)*var</div><div class="line">print(running_var)</div></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="meta">&gt;&gt;&gt; </span>[<span class="number">11.5091</span>, <span class="number">11.5091</span>, <span class="number">11.5091</span>]</div></pre></td></tr></table></figure><p>这回方差也🉑️了。说明归一化中的方差使用的是正常计算出的方差，而running_var的方差在scale上做了-1的处理。</p><blockquote><p>需要注意的是，running_mean和running_var的变化，与optimizer无关，不是执行step以后才更新值，而是每次做前向值都会改变。</p></blockquote><p><strong>完整代码</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_mean</span><span class="params">(x, running_mean, mom=<span class="number">0.1</span>)</span>:</span></div><div class="line">    nc = x.size(<span class="number">1</span>)</div><div class="line">    new_mean = torch.zeros([nc])</div><div class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(nc):</div><div class="line">        new_mean[i] = x[:, i, :, :].mean()</div><div class="line">    running_mean = mom * new_mean + (<span class="number">1</span> - mom) * running_mean</div><div class="line">    <span class="keyword">return</span> new_mean, running_mean</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_var</span><span class="params">(x, running_var, mom=<span class="number">0.1</span>)</span>:</span></div><div class="line">    nc = x.size(<span class="number">1</span>)</div><div class="line">    scale = x.size(<span class="number">0</span>) * x.size(<span class="number">2</span>) * x.size(<span class="number">3</span>)</div><div class="line">    new_var = torch.zeros([nc])</div><div class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(nc):</div><div class="line">        new_var[i] = x[:, i, :, :].var(<span class="keyword">False</span>)</div><div class="line">    temp_var = new_var * scale / (scale - <span class="number">1</span>)</div><div class="line">    running_var = mom * temp_var + (<span class="number">1</span> - mom) * running_var</div><div class="line">    <span class="keyword">return</span> new_var, running_var</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">norm_x</span><span class="params">(x, mean, var, eps=<span class="number">1e-5</span>)</span>:</span></div><div class="line">    normalized_x = torch.zeros_like(x)</div><div class="line">    nc = x.size(<span class="number">1</span>)</div><div class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(nc):</div><div class="line">        normalized_x[:, i, :, :] = (x[:, i, :, :] - mean[i]) / torch.sqrt(var[i] + eps)</div><div class="line">    <span class="keyword">return</span> normalized_x</div><div class="line">  </div><div class="line">mean, running_mean = get_mean(x, running_mean)</div><div class="line">var, running_var = get_var(x, running_var)</div><div class="line">x = norm_x(x, mean, var)</div></pre></td></tr></table></figure><p>小结一下，BN在训练和测试采取的是两种模式，训练阶段每个batch用的是当前batch算出的均值和方差进行归一化，而测试阶段每个用的是moving averages的统计值。在训练阶段学习一个线性映射，即$\gamma, \beta$使得每层的数据分布尽可能平稳。</p><h3 id="相关参数"><a href="#相关参数" class="headerlink" title="相关参数"></a>相关参数</h3><p>知道了具体更新过程的实现后，来看一下PyTorch中BatchNorm的API</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">torch.nn.BatchNorm1d(num_features, </div><div class="line">                     eps=<span class="number">1e-05</span>, </div><div class="line">                     momentum=<span class="number">0.1</span>, </div><div class="line">                     affine=<span class="keyword">True</span>, </div><div class="line">                     track_running_stats=<span class="keyword">True</span>)</div></pre></td></tr></table></figure><p>其中<code>affine</code>表明是否用$\gamma$和$\beta$进行仿射，当<code>affine=False</code>时，<code>weigthts=None, bias=None</code>，<code>track_running_stats</code>表明是否更新统计特性，当<code>track_running_stats=False</code>时，<code>running_mean=None, running_var=None</code>，即每次归一化的时候只用当前batch的均值和方差进行归一化，而不会对之前算出的均值和方差进行平滑。</p><p><strong>参考链接：</strong></p><ul><li><a href="https://blog.csdn.net/LoseInVain/article/details/86476010" target="_blank" rel="external">Pytorch的BatchNorm层使用中容易出现的问题</a></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;BN是防网络过拟合的一个很重要的模块，细微的差别可能对输出效果有很大影响。因此需要理解一下PyTorch中BN的具体实现。PyTorch的源码用C实现的torch.batchnorm。可以&lt;a href=&quot;https://github.com/marvis/pytorch-
      
    
    </summary>
    
    
      <category term="维修指南" scheme="http://yoursite.com/tags/%E7%BB%B4%E4%BF%AE%E6%8C%87%E5%8D%97/"/>
    
  </entry>
  
  <entry>
    <title>卷积核的低秩分解</title>
    <link href="http://yoursite.com/2019/07/01/tensor-decompositions/"/>
    <id>http://yoursite.com/2019/07/01/tensor-decompositions/</id>
    <published>2019-07-01T11:25:02.000Z</published>
    <updated>2020-02-03T13:08:50.357Z</updated>
    
    <content type="html"><![CDATA[<p>通过<a href="https://jacobgil.github.io/deeplearning/tensor-decompositions-deep-learning" target="_blank" rel="external">这篇blog</a>了解一下低秩分解。</p><p>低秩分解仅作用于线性网络层的权重，可能忽略不同网络层的联系。</p><blockquote><p>There are works that try to address these issues, and its still an active research area.</p></blockquote><h3 id="全连接层的张量分解"><a href="#全连接层的张量分解" class="headerlink" title="全连接层的张量分解"></a>全连接层的张量分解</h3><p>首先简单介绍一下SVD。奇异值分解(SVD)是对矩阵进行分解：<br>$$<br>A_{n \times m}=U_{n \times n} S_{n \times m} V_{m \times m}^{T}<br>$$<br>其中$S$是非负实数对角矩阵，对角线上元素即为$A$的<strong>奇异值</strong>。通常会将奇异值由大到小排序。$U$和$V$是酉矩阵，满足$U^{T} U=V^{T} V=I$。当我们取其最大的$t$个奇异值，将剩下的值置0，则能得到近似矩阵$\hat{A}=U_{n x t} S_{t x t} V_{m x t}^{T}$。</p><p>我们对全连接层的公式$A x+b$进行分解有：$\left(U_{n \times t} S_{t \times t} V_{m \times t}^{T}\right) x+b=U_{n \times t}\left(S_{t \times t} V_{m \times t}^{T} x\right)+b$</p><p>这样将一个大矩阵转化为两个小矩阵，参数量从 $n \times m$ 降为 $t  (n+m)$</p><h3 id="卷积层的张量分解"><a href="#卷积层的张量分解" class="headerlink" title="卷积层的张量分解"></a>卷积层的张量分解</h3><p>下面介绍对卷积层进行张量分解最经典的两种方法：CP分解和Tucker分解</p><p><strong>CP分解</strong></p><p>论文地址：<a href="https://arxiv.org/abs/1412.6553" target="_blank" rel="external">Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition</a></p><p>对$A$进行低秩分解有：<br>$$<br>A(i, j)=\sum_{n=1}^{R} A_{1}(i, r) A_{2}(j, r), \quad i=\overline{1, n}, \quad j=\overline{1, m}<br>$$<br>对d维的$A$进行CP分解有：<br>$$<br>A\left(i_{1}, \ldots, i_{d}\right)=\sum_{r=1}^{R} A_{1}\left(i_{1}, r\right) \ldots A_{d}\left(i_{d}, r\right)<br>$$<br>对$d \times d \times S \times T$的4D的卷积核进行CP分解<br>$$<br>K(i, j, s, t)=\sum_{r=1}^{R} K^{x}(i-x+\delta, r) K^{y}(j-y+\delta, r) K^{s}(s, r) K^{t}(t, r)<br>$$<br>则输出可以$V$ 表示为：<br>$$<br>V(x, y, t)=\sum_{r=1}^{R} K^{t}(t, r)\left(\sum_{i=x-\delta}^{x+\delta} K^{x}(i-x+\delta, r)\left(\sum_{j=y-\delta}^{y+\delta} K^{y}(j-y+\delta, r)\left(\sum_{s=1}^{S} K^{s}(s, r) U(i, j, s)\right)\right)\right)<br>$$<br>则1个卷积操作可以分解为4个卷积操作：<br>$$<br>\begin{aligned} U^{s}(i, j, r) &amp;=\sum_{s=1}^{S} K^{s}(s, r) U(i, j, s) \\ U^{s y}(i, y, r) &amp;=\sum_{j=1}^{y+\delta} K^{y}(j-y+\delta, r) U^{s}(i, j, r) \\ U^{s y z}(x, y, r) &amp;=\sum_{i=x-\delta}^{x+\delta} K^{x}(i-x+\delta, r) U^{s y}(i, y, r) \\ V(x, y, t) &amp;=\sum_{r=1}^{R} K^{t}(t, r) U^{s y x}(x, y, r) \end{aligned}<br>$$</p><ol><li>用$1\times1$的pointwise卷积将输入降至R纬度</li><li>在垂直维度做$d\times1$的depthwise卷积</li><li><p>在水平纬度做$d\times1$的depthwise卷积</p></li><li><p>用$1\times 1$的pointwise卷积获得$T$维的输出</p></li></ol><p>复杂度分析：</p><ul><li>原始卷积 $STd^2$</li><li>CP分解 $R(S+2d+T)$</li></ul><p><strong>Tucker分解</strong></p><p>论文地址：<a href="https://arxiv.org/abs/1511.06530" target="_blank" rel="external">Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications</a></p><p>Tucker分解也称为高阶SVD，4D卷积核表示为：<br>$$<br>K(i, j, s, t)=\sum_{r_{1}=1}^{R_{1}} \sum_{r_{2}=1}^{R_{2}} \sum_{r_{3}=1}^{R_{3}} \sum_{r_{4}=1}^{R_{4}} C’_{r_1, r_2, r_3, r_4} K_{r 1}^{x}(i) K_{r_2}^{y}(j) K_{r_3}^{s}(s) K_{r _4}^{t}(t)<br>$$<br>卷积核通常比较小($3 \times 3$)，就不再在空间维度进行分解<br>$$<br>K(i, j, s, t)=\sum_{r_{3}=1}^{R_{3}} \sum_{r_{4}=1}^{R_{4}} C’_{i,j, r_3, r_4} K_{r_3}^{s}(s) K_{r _4}^{t}(t)<br>$$<br>其中$C$代表$d \times d \times R_3 \times R_4$ 的核心张量。</p><p>于是将1个卷积操作分解为3个卷积操作：<br>$$<br>\begin{aligned} \mathcal{Z}_{h, w, r_{3}} &amp;=\sum_{s=1}^{S} U_{s, r_{3}}^{(3)} \mathcal{X}_{h, w, s} \\ \mathcal{Z}_{h^{\prime}, w^{\prime}, r_{4}}^{D} &amp;=\sum_{i=1}^{D} \sum_{j=1}^{D} \sum_{r_{3}=1}^{R_{3}} \mathcal{C}_{i, j, r_{3}, r_{4}} z_{h_{i} w_{j}, r_{3}} \\ y_{h^{\prime}, w^{\prime}, t} &amp;=\sum_{r_{4}=1}^{R_{4}} U_{t, r_{4}}^{(4)} \mathcal{Z}_{h^{\prime}, w^{\prime}, r_{4}}^{\prime} \end{aligned}<br>$$</p><ol><li>用$1\times1$的pointwise卷积将输入降至R纬度</li><li>进行$d \times d$的卷积</li><li>用$1\times 1$的pointwise卷积获得$T$维的输出</li></ol><h3 id="挑选分解的秩"><a href="#挑选分解的秩" class="headerlink" title="挑选分解的秩"></a>挑选分解的秩</h3><p>在分解的时候秩$R$的选择很重要，Tucker分解中用了VBMF的方法。</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;通过&lt;a href=&quot;https://jacobgil.github.io/deeplearning/tensor-decompositions-deep-learning&quot; target=&quot;_blank&quot; rel=&quot;external&quot;&gt;这篇blog&lt;/a&gt;了解一下低秩分解
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>每周论文 Vol.08</title>
    <link href="http://yoursite.com/2019/06/29/weekly-paper-08/"/>
    <id>http://yoursite.com/2019/06/29/weekly-paper-08/</id>
    <published>2019-06-29T02:42:16.000Z</published>
    <updated>2019-07-08T01:08:09.009Z</updated>
    
    <content type="html"><![CDATA[<h3 id="1️⃣-Centripetal-SGD-for-Pruning-Very-Deep-Convolutional-Networks-with-Complicated-Structure"><a href="#1️⃣-Centripetal-SGD-for-Pruning-Very-Deep-Convolutional-Networks-with-Complicated-Structure" class="headerlink" title="1️⃣  Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure"></a>1️⃣  Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure</h3><p>The main idea of this work is to make some filters close to each other in the same cluster during training, and propose Centripetal SGD (C-SGD). </p><p>the update rule of C-SGD is<br>$$<br>\boldsymbol{F}^{(j)} \leftarrow \boldsymbol{F}^{(j)}+\tau \Delta \boldsymbol{F}^{(j)}<br>$$</p><p>$$<br>\begin{aligned} \Delta \boldsymbol{F}^{(j)}=&amp;-\frac{\sum_{k \in H(j)} \frac{\partial L}{\partial \boldsymbol{F}^{(k)}}}{|H(j)|}-\eta \boldsymbol{F}^{(j)} \\ &amp;+\epsilon\left(\frac{\sum_{k \in H(j)} \boldsymbol{F}^{(k)}}{|H(j)|}-\boldsymbol{F}^{(j)}\right) \end{aligned}<br>$$</p><p>For the filters in the same cluster 1）objective function are averaged 2）weight decay 3）gradually eliminate the difference in the initial values.</p><h3 id="2️⃣-Variational-Convolutional-Neural-Network-Pruning"><a href="#2️⃣-Variational-Convolutional-Neural-Network-Pruning" class="headerlink" title="2️⃣ Variational Convolutional Neural Network Pruning"></a>2️⃣ Variational Convolutional Neural Network Pruning</h3><p>本文用变分推理来进行剪枝感觉还蛮有意思的。首先文章基于Network Slimming做了一些改进。对于BN的$\gamma$，如果只稀疏这个值，不考虑$\beta$的影响，那实际上归一化后的输出不会为0，而是加上$\beta$后的值，所以文章考虑的改进方式是令$\tilde{\beta} = \gamma \cdot \beta$，<br>$$<br>x_{o u t}=\gamma \cdot B N(x)+\tilde{\beta}<br>$$<br>文中将这个$\gamma$称为<em>channel saliency</em>，我们的目标是学习稀疏的$\gamma$同时最大化条件概率$p(y | x, \gamma)$。</p><p>首先利用贝叶斯公式我们可以得到：$ p(\gamma | \mathcal{D})= \frac{p(\gamma) p(\mathcal{D} | \gamma)} { p(\mathcal{D})}$</p><p>由于$p(\mathcal{D})=\int p(\mathcal{D}, \gamma) d \gamma$难以计算，这个后验概率分布我们很难直接求的。在变分推理中，我们可以用一个参数分布$q_{\phi}(\gamma)$来近似这个后验概率分布。利用KL散度拉近两个分布的距离：$\min _{\phi} D_{K L}\left(q_{\phi}(\gamma) | p(\gamma | \mathcal{D})\right)$。等价于最大化ELBO：<br>$$<br>\mathcal{L}(\phi)=L_{\mathcal{D}}(\phi)-D_{K L}\left(q_{\phi}(\gamma) | p(\gamma)\right)<br>$$<br>其中，$\mathcal{L}_{\mathcal{D}}(\phi)=\sum_{(x, y) \in \mathcal{D}} \mathbb{E}_{q_{\phi}(\gamma)}[\log p(y | x, \gamma)]$</p><p>可以看到目标函数由两部分构成，第一部分是重建项，是极大似然估计，第二部分为正则项，后面会引入一个先验分布对参数进行惩罚，即稀疏$\gamma$。</p><p>对于$\mathcal{L}(\phi)$需要解决两个问题：</p><ol><li>第一项中由于期望的存在，$\mathcal{L}_{\mathcal{D}}(\phi)$的梯度无法直接求得。</li><li>第二项参数分布$q_\phi(\gamma)$和先验分布$p(\gamma)$的选择。</li></ol><p>🔺 问题1的解决：</p><p>​    引入再参化技巧，则$q_{\phi}(\gamma)$可以表示为一个可导函数$\gamma=f(\phi, \epsilon)$，其中$\epsilon \sim \mathcal{N}(0,1)$<br>$$<br>\mathcal{L}_{\mathcal{D}}(\phi) \simeq \mathcal{L}_{\mathcal{D}}^{\mathcal{A}}(\phi)=\frac{N}{M} \sum_{m=1}^{M} \log p\left(y_{i m} | x_{i m}, \gamma_{i m}=f(\phi, \epsilon)\right)<br>$$<br>​    其中$M$为batch size，$N$为data数量。</p><p>​    将模型参数$\mathbf{w}$加入优化目标中：<br>$$<br>\mathcal{L}(\phi, \mathbf{w}) \simeq \mathcal{L}_{\mathcal{D}}^{\mathcal{A}}(\phi, \mathbf{w})-D_{K L}\left(q_{\phi}(\gamma) | p(\gamma)\right)<br>$$</p><p>🔺 问题2的解决：</p><p>​    本文选取高斯分布作为参数的分布：<br>$$<br>q_{\phi}(\gamma)=\prod_{i=1}^{C} q\left(\gamma_{i}\right), \quad \gamma_{i} \sim \mathcal{N}\left(\mu_{i}, \sigma_{i}\right)<br>$$<br>​    为了让学习出的参数$\phi=(\mu, \sigma)$使分布$q_\phi(\gamma)$尽可能稀疏，本文引入的先验分布为：<br>$$<br>p(\gamma)=\prod_{i=1}^{C} p\left(\gamma_{i}\right), \quad \gamma_{i} \sim \mathcal{N}\left(0, \sigma_{i}^{_}\right)<br>$$<br>​    这样就能让$\gamma$尽可能向0值靠近。<br>$$<br>\begin{aligned} D_{K L}\left(q_{\phi}(\gamma) | p(\gamma)\right) &amp;=\sum_{i} D_{K L}\left(q_{\phi}\left(\gamma_{i}\right) | p\left(\gamma_{i}\right)\right) \\ &amp;=\sum_{i} \log \frac{\sigma_{i}^{_}}{\sigma_{i}}+\frac{\sigma_{i}^{2}+\mu_{i}^{2}}{2\left(\sigma_{i}^{*}\right)^{2}}-\frac{1}{2} \end{aligned}<br>$$<br>​    让两个分布的方差相同，则上式可以表示为：<br>$$<br>D_{K L}\left(q_{\phi}(\gamma) | p(\gamma)\right)=\sum_{i} k \mu_{i}^{2}<br>$$</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h3 id=&quot;1️⃣-Centripetal-SGD-for-Pruning-Very-Deep-Convolutional-Networks-with-Complicated-Structure&quot;&gt;&lt;a href=&quot;#1️⃣-Centripetal-SGD-for-Pruni
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
      <category term="每周论文" scheme="http://yoursite.com/tags/%E6%AF%8F%E5%91%A8%E8%AE%BA%E6%96%87/"/>
    
  </entry>
  
  <entry>
    <title>每周论文 Vol.07</title>
    <link href="http://yoursite.com/2019/06/25/weekly-paper-07/"/>
    <id>http://yoursite.com/2019/06/25/weekly-paper-07/</id>
    <published>2019-06-25T00:55:36.000Z</published>
    <updated>2019-07-08T01:08:06.520Z</updated>
    
    <content type="html"><![CDATA[<h3 id="1️⃣-Universally-Slimmable-Networks-and-Improved-Training-Techniques"><a href="#1️⃣-Universally-Slimmable-Networks-and-Improved-Training-Techniques" class="headerlink" title="1️⃣  Universally Slimmable Networks and Improved Training Techniques"></a>1️⃣  Universally Slimmable Networks and Improved Training Techniques</h3><p>Slimmable network的拓展工作，将固定宽度的网络扩展到任意宽度。提出了3个挑战：</p><ol><li>如何解决包含batch-normalization的网络？</li><li>如何更有效地训练US-Nets</li><li>与单独训练某个宽度的网络相比，US-Nets是如何提升整体精度的？</li></ol><p>🔺 问题1的解决：</p><ol><li><p>训练阶段每次前向时，计算出该batch的均值和方差，然后对输出进行归一化：<br>$$<br>\hat{x}_{B}=\gamma \frac{x_{B}-E_{B}\left[x_{B}\right]}{\sqrt{\operatorname{Var}_{B}\left[x_{B}\right]+\epsilon}}+\beta<br>$$<br>训练过程会对均值和方差做<strong>moving averages</strong>：<br>$$<br>\begin{aligned} \mu_{t} &amp;=m \mu_{t-1}+(1-m) E_{B}\left[x_{B}\right] \\ \sigma_{t}^{2} &amp;=m \sigma_{t-1}^{2}+(1-m) \operatorname{Var}_{B}\left[x_{B}\right] \end{aligned}<br>$$</p><blockquote><p> 需要注意的是，在PyTorch的实现中，每次进行统计时$Var_B = \frac{n}{n-1}Var_B$，其中 $n=c \times h \times w$</p></blockquote><p>测试阶段，用统计值$\mu=\mu_{T}， \sigma^2=\sigma^2_T$进行归一化：<br>$$<br>\hat{x}_{t e s t}=\gamma^{_} \frac{x_{t e s t}-\mu}{\sqrt{\sigma^{2}+\epsilon}}+\beta^{_}<br>$$<br>其中$\gamma^_, \beta^_$是bn学出的weight和bias。进一步可以表示为：<br>$$<br>\hat{x}_{t e s t}=\gamma^{\prime} x_{t e s t}+\beta^{\prime}, \gamma^{\prime}=\frac{\gamma^{_}}{\sqrt{\sigma^{2}+\epsilon}}, \beta^{\prime}=\beta^{_}-\gamma^{\prime} \mu<br>$$</p></li></ol><p>   如果对不同宽度的网络采用Shared BN，由于特征是相加的，前一层是用不同的通道数，输出值就会有所不同，均值和方差也不同，导致了统计值不准确的问题。Slimmable Network的解决办法是对每个宽度都训练了一个单独的BN层，但如果对所有宽度都这样做代价太大了。本文的解决办法是做exact averages：<br>   $$<br>   \begin{aligned} m &amp;=(t-1) / t \\ \mu_{t} &amp;=m \mu_{t-1}+(1-m) E_{B}\left[x_{B}\right] \\ \sigma_{t}^{2} &amp;=m \sigma_{t-1}^{2}+(1-m) \operatorname{Var}_{B}\left[x_{B}\right] \end{aligned}<br>   $$<br>   统计值的计算不在训练过程中进行，而是训练结束后，用随机采样的训练数据进行估计。</p><p>🔺 问题2的解决：</p><ol><li>本文假设模型的表现限制于宽度$[0.25 \times, 1.0 \times]$，优化lower bound和upper bound就能优化整个网络。于是提出了<strong>Sandwich Rule</strong>，在每次训练时随机采样$n-2$个宽度，加上最小宽度和最大宽度一起训练。同时跟踪这两个模型的验证精度，能大概知道US-Net的lower bound和upper bound。并且，训练最大宽度的模型可以用于<strong>Inplace Distillation</strong>。最大宽度的模型用groud truth做loss，而其它宽度的模型可以用最大宽度模型预测出的soft-probabilities做loss。</li></ol><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190703111249984.png" alt="image-20190703111249984"></p><p>文章最后坐着讨论了几个话题：</p><ol><li>我们能不能训练一个非均匀的US-Net这样每层能够调整它自己的宽度比？</li><li></li></ol>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h3 id=&quot;1️⃣-Universally-Slimmable-Networks-and-Improved-Training-Techniques&quot;&gt;&lt;a href=&quot;#1️⃣-Universally-Slimmable-Networks-and-Improved-Train
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
      <category term="每周论文" scheme="http://yoursite.com/tags/%E6%AF%8F%E5%91%A8%E8%AE%BA%E6%96%87/"/>
    
  </entry>
  
  <entry>
    <title>Tensorflow中的PixelShuffle(depth_to_space)</title>
    <link href="http://yoursite.com/2019/06/23/tf-pixshuffle/"/>
    <id>http://yoursite.com/2019/06/23/tf-pixshuffle/</id>
    <published>2019-06-23T08:31:38.000Z</published>
    <updated>2019-06-28T02:10:01.249Z</updated>
    
    <content type="html"><![CDATA[<p>在尝试对PixelShuffle前的卷积层做剪枝时遇到了一些问题，对PixelShuffle的具体操作有了进一步的了解。</p><p>PixelShuffle通过将通道重排对图像进行上采样，tf中的函数是<code>tf.depth_to_sapce</code>，第一个参数是<code>Tensor</code>，第二个参数是需要放大倍数。当输入<code>X</code>的大小为<code>[1 2 2 16]</code>，放大倍数为2，H和W各乘2，C除以4，PixelShuffle后的结果就为<code>[1 4 4 4]</code>。</p><p>🔺坑点1：想当然的以为参与重排的通道是<code>[:, :, :, i:i+4]</code></p><p>测试代码：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">x = tf.range(<span class="number">64</span>)</div><div class="line">x = tf.reshape(x, [<span class="number">1</span>, <span class="number">2</span>, <span class="number">2</span>, <span class="number">16</span>])</div><div class="line">y = tf.depth_to_space(x, <span class="number">2</span>) <span class="comment"># [1, 4, 4, 4]</span></div></pre></td></tr></table></figure><p>下面看一下具体x和y每个通道的值：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="meta">&gt;&gt;&gt; </span>x[:, :, :, <span class="number">0</span>]</div><div class="line"><span class="meta">&gt;&gt;&gt; </span>[[[ <span class="number">0</span>, <span class="number">16</span>],</div><div class="line">[<span class="number">32</span>, <span class="number">48</span>]]]</div><div class="line"><span class="meta">&gt;&gt;&gt; </span>x[:, :, :, <span class="number">1</span>]</div><div class="line"><span class="meta">&gt;&gt;&gt; </span>[[[ <span class="number">1</span>, <span class="number">17</span>],</div><div class="line">[<span class="number">33</span>, <span class="number">49</span>]]]</div><div class="line"> </div><div class="line"><span class="meta">&gt;&gt;&gt; </span>y[:, :, :, <span class="number">0</span>]</div><div class="line"><span class="meta">&gt;&gt;&gt; </span>[[[ <span class="number">0</span>,  <span class="number">4</span>, <span class="number">16</span>, <span class="number">20</span>],</div><div class="line">[ <span class="number">8</span>, <span class="number">12</span>, <span class="number">24</span>, <span class="number">28</span>],</div><div class="line">[<span class="number">32</span>, <span class="number">36</span>, <span class="number">48</span>, <span class="number">52</span>],</div><div class="line">[<span class="number">40</span>, <span class="number">44</span>, <span class="number">56</span>, <span class="number">60</span>]]]</div></pre></td></tr></table></figure><p>可以看出<code>y</code>的第0维通道包含的是<code>x</code>通道数为0、4、8、12的特征图。可视化一下就是这样的效果：</p><p><img src="https://i.loli.net/2019/06/27/5d14b6a1d9bbf35069.png" alt=""></p><p>将Y的一个通道单独取出，看一下每个点属于原来X的哪个坐标：</p><p><img src="/Users/colorjam/Library/Application Support/typora-user-images/image-20190628095308642.png" alt="image-20190628095308642"></p><p>可以看到Y的一个通道实际上分成4个象限，在空间上由<code>(0,0)(0,1)(1,0)(1,1)</code>构成。在通道上每4个间隔提取对应通道。这里的间隔对应的是Pixshuffle后的通道数。</p><p>假设原始通道数为<code>c_out</code>，PS后的通道数为<code>ps_out</code>，实际上<code>y</code>的第<code>i</code>通道对应的是<code>x</code>的<code>[i, i+ps_out, i+2*ps_out, i+3*ps_out]</code></p><p>而我原先理解的通道排列方式是❌</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="meta">&gt;&gt;&gt; </span>y[:, :, :, <span class="number">0</span>]</div><div class="line"><span class="meta">&gt;&gt;&gt; </span>[[[ <span class="number">0</span>,  <span class="number">1</span>,  <span class="number">4</span>,  <span class="number">5</span>],</div><div class="line">[ <span class="number">2</span>,  <span class="number">3</span>,  <span class="number">6</span>,  <span class="number">7</span>],</div><div class="line">[ <span class="number">8</span>,  <span class="number">9</span>, <span class="number">12</span>, <span class="number">13</span>],</div><div class="line">[<span class="number">10</span>, <span class="number">11</span>, <span class="number">14</span>, <span class="number">15</span>]]]</div></pre></td></tr></table></figure><p>🔺坑点2：提取k个保留的通道时，只需取索引的前k个值</p><p>假设放大倍率是4，用L1的剪枝方式，需要保留的通道数为<code>c_keep</code>。当对应到具体的剪枝通道的时候，需要找到PixShuffle后剪掉通道所对应的原始卷积输出的4个通道。从上面的坐标我们就可以看出，剪掉Y的0通道时，需要对应剪掉X的0、4、8、12通道。来看看具体的实现。主要分为几步：</p><ol><li>计算Y对应X的通道</li><li>计算Y需要保留的通道</li><li>将Y的通道映射回X</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># 1. 计算Y对应X的通道</span></div><div class="line">norm_list, shuffled_idx_list = [], []</div><div class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(ps_out):</div><div class="line">shuffled_idx = [i+k*ps_out <span class="keyword">for</span> k <span class="keyword">in</span> range(<span class="number">4</span>)]  <span class="comment"># Y通道对应的4个X通道</span></div><div class="line">shuffled_idx_list.append(shuffled_idx)</div><div class="line">  norm_sum = tf.reduce_sum(tf.gather(norm_value, shuffled_idx)) <span class="comment"># 提取对应索引的通道</span></div><div class="line">  norm_list.append(sess.run(norm_sum))</div><div class="line">  </div><div class="line"><span class="comment"># 2. 计算需要保留的通道</span></div><div class="line">remain_idx = np.sort(np.argsort(norm_list)[::<span class="number">-1</span>][:int(c_keep/<span class="number">4</span>)])</div><div class="line"></div><div class="line">remain_list = []</div><div class="line"><span class="comment"># 3. 将Y的通道映射回X</span></div><div class="line"><span class="keyword">for</span> remain <span class="keyword">in</span> remain_idx:</div><div class="line">  remain_list.extend(shuffled_idx_list[remain])</div><div class="line">remain_list = np.sort(remain_list)</div></pre></td></tr></table></figure><p>这样<code>remain_list</code>即原始卷积输出需要剪掉的通道索引。</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;在尝试对PixelShuffle前的卷积层做剪枝时遇到了一些问题，对PixelShuffle的具体操作有了进一步的了解。&lt;/p&gt;
&lt;p&gt;PixelShuffle通过将通道重排对图像进行上采样，tf中的函数是&lt;code&gt;tf.depth_to_sapce&lt;/code&gt;，第一个
      
    
    </summary>
    
    
      <category term="维修指南" scheme="http://yoursite.com/tags/%E7%BB%B4%E4%BF%AE%E6%8C%87%E5%8D%97/"/>
    
  </entry>
  
  <entry>
    <title>每周论文 Vol.06</title>
    <link href="http://yoursite.com/2019/06/18/weekly-paper-06/"/>
    <id>http://yoursite.com/2019/06/18/weekly-paper-06/</id>
    <published>2019-06-18T01:49:16.000Z</published>
    <updated>2019-06-21T09:12:49.701Z</updated>
    
    <content type="html"><![CDATA[<h3 id="1️⃣-AutoSlim-Towards-One-Shot-Architecture-Search-for-Channel-Numbers"><a href="#1️⃣-AutoSlim-Towards-One-Shot-Architecture-Search-for-Channel-Numbers" class="headerlink" title="1️⃣ AutoSlim: Towards One-Shot Architecture Search for Channel Numbers"></a>1️⃣ AutoSlim: Towards One-Shot Architecture Search for Channel Numbers</h3><p>这篇和<a href="https://arxiv.org/abs/1812.08928" target="_blank" rel="external">ICLR2019</a>、<a href="https://arxiv.org/abs/1903.05134" target="_blank" rel="external">Universally Slimmable Networks</a>是同一个作者，解决的问题都是通道剪枝。下面先了解一下本文。</p><p><strong>Why?</strong></p><p>Most channel pruning methods are grouneded on <strong>the importance of trained weights</strong>, so the slimmed layer usually consists channels of discrete index. Most NAS methods have high computational cost and time cost.</p><p><strong>How?</strong></p><p>Extending the work of slimmable networks and propose AutoSlim. The training process is as following:</p><ol><li><p>Train a slimmable model for a few epochs to get a benchmark performance estimator.</p><ul><li><p>Searching space is defined between the upper bound and lower bound of channel numbers. In each training iteration, randomly sample the number of channels in each layer. In each layer remove a group of channels. </p><blockquote><p>in resents, first sample the channel number of residual dentity pathway and then randomly and independenly sample channel number inside each residual block.</p></blockquote></li></ul></li><li><p>Evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop on validation set.</p></li><li><p>Obtain the optimized channel configurations under different resource constraints.</p></li><li><p>Train optimized architectures individually or slimmable network for full training epochs.</p></li></ol><p>The paper is based on the assumption that <strong>the importance of weight is implicitly ranked by its index</strong>, which means that the smaller index of one filter the more important of this filter.</p><h3 id="2️⃣-AutoGrow-Automatic-Layer-Growing-in-Deep-Convolutional-Networks"><a href="#2️⃣-AutoGrow-Automatic-Layer-Growing-in-Deep-Convolutional-Networks" class="headerlink" title="2️⃣ AutoGrow: Automatic Layer Growing in Deep Convolutional Networks"></a>2️⃣ AutoGrow: Automatic Layer Growing in Deep Convolutional Networks</h3><p>The method can be easily found in the title, to gradually grow the depth of DNN.</p><p>The  <em>network</em> is  composed of <em>sub-netwok</em>, and <em>sub-network</em> is composed of <em>sub-modules</em>.<br>$$<br>g\left(\mathcal{X}_{0}\right)=l\left(\boldsymbol{f}_{M-1}\left(\boldsymbol{f}_{M-2}\left(\cdots \boldsymbol{f}_{1}\left(\boldsymbol{f}_{0}\left(\mathcal{X}_{0}\right)\right) \cdots\right)\right)\right)<br>$$<br>AutoGrow is based on Network Morphism, but propose to initilize the last Batch Normalization layer in a residual block of <em>AdamInit</em> insted of <em>ZeroInit</em></p><p><strong>AdamInit</strong></p><p>given the new layers $\mathcal{W}$, we have:<br>$$<br>g\left(\mathcal{X}_{0} ; \mathbb{W}\right)=g\left(\mathcal{X}_{0} ; \mathbb{W} \cup \mathcal{W}\right) \forall \mathcal{X}_{0}<br>$$<br>freeze all parameters except the last Batch Normalization layer in $\mathcal{W}$, use Adam optimizer to optimize the last Batch Normalization layer.</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h3 id=&quot;1️⃣-AutoSlim-Towards-One-Shot-Architecture-Search-for-Channel-Numbers&quot;&gt;&lt;a href=&quot;#1️⃣-AutoSlim-Towards-One-Shot-Architecture-Search-f
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
      <category term="每周论文" scheme="http://yoursite.com/tags/%E6%AF%8F%E5%91%A8%E8%AE%BA%E6%96%87/"/>
    
  </entry>
  
  <entry>
    <title>每周论文 Vol.05</title>
    <link href="http://yoursite.com/2019/06/10/weekly-paper-05/"/>
    <id>http://yoursite.com/2019/06/10/weekly-paper-05/</id>
    <published>2019-06-10T02:29:31.000Z</published>
    <updated>2019-06-18T01:49:25.901Z</updated>
    
    <content type="html"><![CDATA[<h3 id="1️⃣-Dynamic-Capacity-Networks"><a href="#1️⃣-Dynamic-Capacity-Networks" class="headerlink" title="1️⃣ Dynamic Capacity Networks"></a>1️⃣ Dynamic Capacity Networks</h3><p>use two alternative sub-networks: </p><ol><li>coarse layers $f_c$ on the whole input $\mathbf {x}$  </li><li>fine layers $f_f$ at salient regions </li></ol><p><strong>coarse representation vectors</strong><br>$$<br>f_{c}(\mathbf{x})=\left\{\mathbf{c}_{i, j} |(i, j) \in\left[1, s_{1}\right] \times\left[1, s_{2}\right]\right\}<br>$$</p><p>$$<br>h_{c}(\mathbf{x})= \mathbf{o}_c = g\left(f_{c}(\mathbf{x})\right)<br>$$</p><p>$\mathbf{c}_{i, j}=f_{c}\left(\mathbf{x}_{i, j}\right) \in \mathbb{R}^{D}$ </p><p><strong>salient input regions</strong><br>$$<br>H=-\sum_{l=1}^{C} \mathbf{o}_{c}^{(l)} \log \mathbf{o}_{c}^{(l)}<br>$$</p><p>$$<br>M_{i, j}=\left|\nabla_{\mathbf{c}_{i, j}} H\right|_{2}<br>$$</p><p>$C$ is the number of class labels, $\mathbf{M} \in \mathbb{R}^{s_{1} \times s_{2}}$</p><p>select top $k$ input regions $\mathbf{X}^{s}=\left\{\mathbf{x}_{i, j} |(i, j) \in \mathbf{I}^{s}\right\}$ based on $\mathbf{M}$</p><p><strong>fine representation vectors</strong><br>$$<br>f_{f}\left(\mathbf{X}^{s}\right)=\left\{\mathbf{f}_{i, j} |(i, j) \in \mathbf{I}^{s}\right\}<br>$$<br>refined representation $f_r(\mathbf {x})$ by combining $f_c(\mathbf{x})$ and $f_f(\mathbf{X}^s)$</p><p><strong>loss</strong></p><ol><li><p>Cross Entropy<br>$$<br>J=-\sum_{i=1}^{m} \log p\left(y^{(i)} | \mathbf{x}^{(i)} ; \theta\right)<br>$$</p></li><li><p>encourage similarity between the coarse and fine representations</p></li></ol><p>$$<br>\sum_{\mathbf{x}_{i, j} \in \mathbf{X}^{s}}\left|f_{c}\left(\mathbf{x}_{i, j}\right)-f_{f}\left(\mathbf{x}_{i, j}\right)\right|_{2}^{2}<br>$$</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h3 id=&quot;1️⃣-Dynamic-Capacity-Networks&quot;&gt;&lt;a href=&quot;#1️⃣-Dynamic-Capacity-Networks&quot; class=&quot;headerlink&quot; title=&quot;1️⃣ Dynamic Capacity Networks&quot;&gt;&lt;/a
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
      <category term="每周论文" scheme="http://yoursite.com/tags/%E6%AF%8F%E5%91%A8%E8%AE%BA%E6%96%87/"/>
    
  </entry>
  
  <entry>
    <title>Gumbel Softmax</title>
    <link href="http://yoursite.com/2019/05/28/gumbel-softmax/"/>
    <id>http://yoursite.com/2019/05/28/gumbel-softmax/</id>
    <published>2019-05-28T07:29:09.000Z</published>
    <updated>2019-07-16T00:34:11.969Z</updated>
    
    <content type="html"><![CDATA[<p>在PAG里发现了Gumbel Sampling Trick，把离散的采样过程用公式表达出来，于是可以放进神经网络中进行求导和反向，觉得是很有意思的工作，想要多加深一些了解。</p><h3 id="问题引入"><a href="#问题引入" class="headerlink" title="问题引入"></a>问题引入</h3><p>通过<a href="https://www.cnblogs.com/initial-h/p/9468974.html" target="_blank" rel="external">博客</a>入了一下小门，结合<a href="https://www.zhihu.com/question/62631725/answer/201338234" target="_blank" rel="external">知乎</a>，首先来理解一下Gumbel Sampling Trick用来做什么。</p><blockquote><p>已知一个离散随机变量X的分布，我们想得到一些服从这个分布的离散的x的值。</p></blockquote><p>比较简单的方法是用<code>np.random.choice</code>。比如我们现在有5个值，概率分布是<code>[0.1, 0, 0.3, 0.6, 0]</code>，即第4个元素最有可能被采样到：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="meta">&gt;&gt;&gt; </span>np.random.choice(<span class="number">5</span>, <span class="number">3</span>, p=[<span class="number">0.1</span>, <span class="number">0</span>, <span class="number">0.3</span>, <span class="number">0.6</span>, <span class="number">0</span>])</div><div class="line">array([<span class="number">3</span>, <span class="number">3</span>, <span class="number">0</span>])</div></pre></td></tr></table></figure><p>这样我们是获取到了值，但是这个过程在神经网络中无法求导和方向。于是gumbel-max出现了：</p><blockquote><p>将采样的过程公式化，公式中的参数为离散随机变量的概率分布。</p></blockquote><p>$$<br>z_{i}=\left\{\begin{array}{l}{1, i=\operatorname{argmax}_{j}\left(\log \left(p_{j}\right)+g_{j}\right)} \\ {0, \text { otherwise }}\end{array}\right.<br>$$</p><p>其中$g_{i}$代表gumbel噪声，$g_{i}=-\log \left(-\log \left(u_{i}\right)\right), u_{i} \sim U n i f o r m(0,1)$。输出$z_i$是一个$j$维的one-hot向量。</p><p>由于argmax不可导，用可导的softmax替代argmax<br>$$<br>\boldsymbol{z}=\operatorname{softmax}((\log (\boldsymbol{p})+\boldsymbol{g}) / \tau)<br>$$<br>参数$ \tau$越小，$z$越接近one-hot向量。</p><blockquote><p>我们把不可导的采样过程，从x本身转嫁到了求取x的公式中的一项g上面，而g不依赖于概率分布p。这样一来，x对p仍然是可导的，而我们得到的x仍然是离散值的采样。这样的采样过程转嫁的技巧叫再参化技巧(reparameterization trick)</p></blockquote><p>那么网络有哪些地方需要采样呢？接下来了解一下VAE的相关应用。</p><h3 id="相关应用"><a href="#相关应用" class="headerlink" title="相关应用"></a>相关应用</h3><p><strong>变分自动编码器VAE</strong></p><p><a href="http://kvfrans.com/variational-autoencoders-explained/" target="_blank" rel="external">这篇博客</a> 解释得很好，自动编码器由编码器(encoder, E)和解码器(decoder, D)构成，E对输入图像进行编码，生成隐向量， D对隐向量进行解码，输出图像。</p><p><img src="https://images2018.cnblogs.com/blog/1428973/201808/1428973-20180813165000500-1207992534.jpg" alt="img"></p><p>但是这样我们必须通过图像来生成隐向量，局限性较大，可不可以随便来一个隐向量，输入进D就能生成图片呢？于是VAE就出现了。</p><blockquote><p>限制编码器生成服从单元高斯分布的隐向量。</p></blockquote><p>因此学习目标就可以分为两部分：1）生成图像和真实图像尽可能接近； 2）隐变量服从单元高斯分布</p><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">generation_loss = mean(square(generated_image - real_image))  </div><div class="line">latent_loss = KL-Divergence(latent_variable, unit_gaussian)  </div><div class="line">loss = generation_loss + latent_loss</div></pre></td></tr></table></figure><p>为了优化KL散度，需要引入👆🏻提到过的再参化技巧。</p><blockquote><p>E不直接生成隐向量，而是生成一个均值向量和一个方差向量。再通过均值和方差采样出隐向量。</p></blockquote><p><img src="https://images2018.cnblogs.com/blog/1428973/201808/1428973-20180813165407236-1369432498.png" alt="img"> </p><p>至此我们明白了<strong>采样</strong>是为了让数据尽可能服从某一分布，通过<strong>再参化技巧</strong>来学习这个分布的参数。高斯分布是连续的，直接可求导，那一些不连续的离散分布怎么办呀？这就回到了一开始的问题「Gumbel Sampling Trick」。</p><p><strong>分类再参化(Categorical reparameterization)</strong></p><p>ICLR 2017的<a href="https://arxiv.org/pdf/1611.01144.pdf" target="_blank" rel="external">这篇文章</a> 就利用Gumbel-Softmax分布，将离散的分类概率分布采样过程转化为了可求导的过程。</p><p><img src="https://i.loli.net/2019/05/30/5cef4476d308a17728.png" alt=""></p><p>上图反映了参数$ \tau$对连续概率分布(a)和离散的one-hot类别分布的影响。当$ \tau$太小时会导致梯度的方差过大，所以文章在实验中用了退火的策略来逐渐减小参数$ \tau$。还可以利用熵正则来学习$\tau$，自动调整Gumbel-Softmax分布采样的置信度。</p><p>本文的训练过程采用Straight-Through (ST) Gumbel Estimator，即前向用argmax，梯度回传时用softmax的梯度。</p><p><strong>参考链接：</strong></p><ul><li><p><a href="https://www.cnblogs.com/initial-h/p/9468974.html" target="_blank" rel="external">Gumbel-Softmax Trick和Gumbel分布</a></p></li><li><p><a href="https://lips.cs.princeton.edu/the-gumbel-max-trick-for-discrete-distributions/" target="_blank" rel="external">The Gumbel-Max Trick for Discrete Distributions</a></p></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;在PAG里发现了Gumbel Sampling Trick，把离散的采样过程用公式表达出来，于是可以放进神经网络中进行求导和反向，觉得是很有意思的工作，想要多加深一些了解。&lt;/p&gt;
&lt;h3 id=&quot;问题引入&quot;&gt;&lt;a href=&quot;#问题引入&quot; class=&quot;headerlin
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
  </entry>
  
  <entry>
    <title>每周论文 Vol.04</title>
    <link href="http://yoursite.com/2019/05/25/weekly-paper-04/"/>
    <id>http://yoursite.com/2019/05/25/weekly-paper-04/</id>
    <published>2019-05-25T08:30:05.000Z</published>
    <updated>2020-06-02T14:40:49.792Z</updated>
    
    <content type="html"><![CDATA[<p>为了锻炼自己的英语写作能力，以后尽量用英文做进行归纳（⌘+C &amp; ⌘+V）～</p><h3 id="1️⃣-To-prune-or-not-to-prune-exploring-the-efficacy-of-pruning-for-model-compression"><a href="#1️⃣-To-prune-or-not-to-prune-exploring-the-efficacy-of-pruning-for-model-compression" class="headerlink" title="1️⃣ To prune, or not to prune: exploring the efficacy of pruning for model compression"></a>1️⃣ To prune, or not to prune: exploring the efficacy of pruning for model compression</h3><p>这篇是TensorFlow自己出的，直接在训练过程中融合L1剪枝。通过将操作融入TensoFlow的training graph，在训练过程中对权重进行排序，用一个mask将最小的weights置0。从inital sparsity values $s_i$开始，以$\Delta t$ 的剪枝频率，最终达到final sparsity value $s_f$<br>$$<br>s_{t}=s_{f}+\left(s_{i}-s_{f}\right)\left(1-\frac{t-t_{0}}{n \Delta t}\right)^{3} \text { for } t \in\left\{t_{0}, \quad t_{0}+\Delta t, \ldots, t_{0}+n \Delta t\right\}<br>$$<br>masks每隔$\Delta t$更新一次，直到达到$s_f$后不再更新。同时文章表明，$n$的选择与学习率的下降策略密切相关。</p><h3 id="2️⃣-OBJECT-DETECTORS-EMERGE-IN-DEEP-SCENE-CNNS"><a href="#2️⃣-OBJECT-DETECTORS-EMERGE-IN-DEEP-SCENE-CNNS" class="headerlink" title="2️⃣ OBJECT DETECTORS EMERGE IN DEEP SCENE CNNS"></a>2️⃣ OBJECT DETECTORS EMERGE IN DEEP SCENE CNNS</h3><p>📍<a href="https://github.com/metalbubble/cnnvisualizer" target="_blank" rel="external">Github Repo</a></p><p><strong>Contributions</strong></p><ul><li>object detection emerges inside a CNN trained to recognize scenes, even more than when trained with ImageNet</li><li>the same network can do both object localization and scene recognition in a single forward-pass.</li></ul><p><strong>Experiments</strong></p><ul><li><p>identify the differences in the type of images preferred at the different layers of each network</p></li><li><p>Places-CNN and ImageNet-CNN  prefer similar images in the earlier layers, while the later layers tend to be more specialized to the specific task of scene or object categorization.</p></li><li><p>understand the nature of the representation that the network is learning</p><ul><li><em>simplifying the input images:</em> 1) removing segments from the image to produce the smallest decrease of the correct classification score until the image is incorrectly classified 2) generate the minimal image representations using image set of SUN database. =&gt; use minimal image representations as inputs to show the contribute important information for the network to recognize the scene.</li><li><em>visualize the receptive fields (RFs) of units and their activatoin patterns:</em> use sliding-window to identify which regions of the image led to the high unit activations. =&gt; as the layers go deeper the RF size gradually increases and the activation regions become more semantically meaningful.</li><li><em>understan and quantify the precise semantic learnd by each unit: </em>ask AMT to indentify the common concepts that exists between the top scoring segmentations for each unit.</li></ul></li><li><p>emergence of objects as the internal representation</p><ul><li><p>what object classes emerge? =&gt; use pool5 to show the distribution of objects</p></li><li><p>why do those obejcts emerge? </p><ul><li><p>possibility 1:  the objects correspond to the most frequent ones in the database. (correlation is 0.54)</p></li><li><p>possibility 2:  the objects that allow discriminatin among scene categories. (correlation is 0.84)</p><p>=&gt; the network is automatically identifying the most discriminative object categories to a large extent</p></li></ul></li></ul></li></ul><h3 id="3️⃣-Network-Dissection-Quantifying-Interpretability-of-Deep-Visual-Representations"><a href="#3️⃣-Network-Dissection-Quantifying-Interpretability-of-Deep-Visual-Representations" class="headerlink" title="3️⃣ Network Dissection: Quantifying Interpretability of Deep Visual Representations"></a>3️⃣ Network Dissection: Quantifying Interpretability of Deep Visual Representations</h3><p>📍<a href="https://github.com/CSAILVision/NetDissect-Lite" target="_blank" rel="external">Github Repo</a></p><p><strong>Questions</strong></p><ul><li>What is a disentangled representation, and how can its factors be quantified and detected?</li><li><p>Do interpretable hidden units reflect a special alignment of feature space, or are interpretations a chimera?</p></li><li><p>What conditions in state-of-the-art training lead to representations with greater or lesser entanglement?</p></li></ul><p><strong>Measurement of interpretability: three-step process of Network Dissection</strong></p><ol><li><p>Identify a broad set of human-labeld visual concepts.</p></li><li><p>Gather hidden variables’ response to known concepts.</p><ul><li>draw concepts $c$ from the Broden dataset.</li></ul></li><li><p>Quantify alignment of hidden variable — concept pairs.</p><ul><li><p>Scoring Unit Interpretability</p><p>input image $x$, activation map $A_{k}(\mathbf{x}) \stackrel{scale up}{\longrightarrow}S_k(x) $，individual unit activations $a_k$</p><p>top quantile level $T_k$： $P\left(a_{k}&gt;T_{k}\right)=0.005$</p><p>binary segmentation：$M_{k}(\mathbf{x}) \equiv S_{k}(\mathbf{x}) \geq T_{k}$</p><p>input annotaion mask $L_c$ </p><p>score：the accuracy of unit $k$ in detecting concept $c$<br>$$<br>I o U_{k, c}=\frac{\sum\left|M_{k}(\mathbf{x}) \cap L_{c}(\mathbf{x})\right|}{\sum\left|M_{k}(\mathbf{x}) \cup L_{c}(\mathbf{x})\right|}<br>$$</p></li></ul></li></ol><h3 id="4️⃣-Pixel-wise-Attentional-Gating-for-Scene-Parsing"><a href="#4️⃣-Pixel-wise-Attentional-Gating-for-Scene-Parsing" class="headerlink" title="4️⃣ Pixel-wise Attentional Gating for Scene Parsing"></a>4️⃣ Pixel-wise Attentional Gating for Scene Parsing</h3><p><strong>Contributions:</strong></p><ul><li>Dynamic computation depth: insert PAG at multiple lyaers of ResNet to control computational parsimony.</li><li>Dynamic spatial pooling: adaptively chooses the proper pooling size for each pixel to aggregate information for inference.</li><li>Experimetns on various pixel labeling tasks, including semantic segmentation, boundary detection, monocular depth and surface normal estimation.</li></ul><p><img src="https://i.loli.net/2019/05/27/5ceb52642beaa24177.png" alt=""></p><p>binary spatial mask $\mathbf{G}$ on ResBottleneck:<br>$$<br>\begin{array}{ll}{\mathbf{X}=\mathcal{F}^{1}(\mathbf{I})} &amp; {\mathbf{X}=\mathcal{F}^{1}(\mathbf{I}), \mathbf{G}=\mathcal{G}(\mathbf{I})} \\ {\mathbf{Y}=\mathcal{F}^{2}(\mathbf{X})} &amp; {\mathbf{Y}=\mathcal{F}_{\mathbf{G}}^{2}(\mathbf{X})} \\ {\mathbf{Z}=\mathcal{F}^{3}(\mathbf{Y})} &amp; {\mathbf{Z}=\mathcal{F}_{\mathbf{G}}^{3}(\overline{\mathbf{G}} \odot \mathbf{X}+\mathbf{G} \odot \mathbf{Y})} \\ {\mathbf{O}=\mathbf{I}+\mathbf{Z}} &amp; {\mathbf{O}=\mathbf{I}+\mathbf{Z}}\end{array}<br>$$<br><strong>Methods:</strong></p><ul><li><p>Learning attention maps</p><blockquote><p>The key to the proposed PAG is the gating function G that produces a discrete (binary) mask which allows for reduced computation. However, producing the binary mask using hard thresholding is non-differentiable, and thus cannot be simply incorporated in CNN where gradient descent is used for training. To bridge the gap, we exploit the Gumbel-Max trick [19] and its recent continuous relaxation [39, 28].</p></blockquote><p>Gumbel distribution  $m \equiv-\log (-\log (u))$, where $u \sim \mathcal{U}[0,1]$</p><p>$g$ is a discrete random variable with probabilities  $P(g=k) \propto a_{k}$</p><p>$\left\{m_{k}\right\}_{k=1, \dots, K}$ is a sequence of i.i.d Gumbel random variables </p><p>sample from the discrete variable:<br>$$<br>g=\underset{k=1, \ldots, K}{\operatorname{argmax}}\left(\log \alpha_{k}+m_{k}\right)<br>$$<br>Gumbel Sampling Trick (replaces the argmax operation with a softmax): </p></li></ul><p>$$<br>\mathbf{g}=\operatorname{softmax}((\log (\boldsymbol{\alpha})+\mathbf{m}) / \tau)<br>$$</p><p>​        <strong>forwardd pass</strong>: discrete smaples of the argmax </p><p>​        <strong>backward pass</strong>: compute gradient of the softmax relaxation</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;为了锻炼自己的英语写作能力，以后尽量用英文做进行归纳（⌘+C &amp;amp; ⌘+V）～&lt;/p&gt;
&lt;h3 id=&quot;1️⃣-To-prune-or-not-to-prune-exploring-the-efficacy-of-pruning-for-model-compressi
      
    
    </summary>
    
    
      <category term="paper" scheme="http://yoursite.com/tags/paper/"/>
    
      <category term="每周论文" scheme="http://yoursite.com/tags/%E6%AF%8F%E5%91%A8%E8%AE%BA%E6%96%87/"/>
    
  </entry>
  
</feed>