Shallow RNNs: A Method for Accurate Time-series Classiﬁcation on Tiny Devices - Dennis, D.; Acar, D. A. E.; Mandikal, Vikram; Sadasivan, Vinu Sankar; Simhadri, Harsha; Saligrama, Venkatesh; Jain, Prateek (NeurIPS 2019, 2019)
Recurrent Neural Networks (RNNs) capture long dependencies and context, and hence are the key component of typical sequential data based tasks. However, the sequential nature of RNNs dictates a large inference cost for long sequences even if the hardware supports parallelization. To induce long-term dependencies, and yet admit parallelization, we introduce novel shallow RNNs. In this architecture, the ﬁrst layer splits the input sequence and runs several independent RNNs. The second layer consumes the output of the ﬁrst layer using a second RNN thus capturing long dependencies. We provide theoretical justiﬁcation for our architecture under weak assumptions that we verify on real-world benchmarks. Furthermore, we show that for time-series classiﬁcation, our technique leads to substantially improved inference time over standard RNNs without compromising accuracy. For example, we can deploy audio-keyword classiﬁcation on tiny Cortex M4 devices (100MHz processor, 256KB RAM, no DSP available) which was not possible using standard RNN models. Similarly, using ShaRNN in the popular Listen-Attend-Spell (LAS) architecture for phoneme classiﬁcation [4], we can reduce the lag in phoneme classiﬁcation by 10-12x while maintaining state-of-the-art accuracy.