Node Pruning Based on Entropy of Weights and Node Activity for Small-Footprint Acoustic Model Based on Deep Neural Networks

Ryu Takeda, Kazuhiro Nakadai, Kazunori Komatani


This paper describes a node-pruning method for an acoustic model based on deep neural networks (DNNs). Node pruning is a promising method to reduce the memory usage and computational cost of DNNs. A score function is defined to measure the importance of each node, and less important nodes are pruned. The entropy of the activity of each node has been used as a score function to find nodes with outputs that do not change at all. We introduce entropy of weights of each node to consider the number of weights and their patterns of each node. Because the number of weights and the patterns differ at each layer, the importance of the node should also be measured using the related weights of the target node. We then propose a score function that integrates the entropy of weights and node activity, which will prune less important nodes more efficiently. Experimental results showed that the proposed pruning method successfully reduced the number of parameters by about 6% without any accuracy loss compared with a score function based only on the entropy of node activity.


 DOI: 10.21437/Interspeech.2017-779

Cite as: Takeda, R., Nakadai, K., Komatani, K. (2017) Node Pruning Based on Entropy of Weights and Node Activity for Small-Footprint Acoustic Model Based on Deep Neural Networks. Proc. Interspeech 2017, 1636-1640, DOI: 10.21437/Interspeech.2017-779.


@inproceedings{Takeda2017,
  author={Ryu Takeda and Kazuhiro Nakadai and Kazunori Komatani},
  title={Node Pruning Based on Entropy of Weights and Node Activity for Small-Footprint Acoustic Model Based on Deep Neural Networks},
  year=2017,
  booktitle={Proc. Interspeech 2017},
  pages={1636--1640},
  doi={10.21437/Interspeech.2017-779},
  url={http://dx.doi.org/10.21437/Interspeech.2017-779}
}