Early Bird uses 10 times less energy to train deep neural networks

Novel training method could shrink carbon footprint for greener deep learning

Rice University’s Early Bird could care less about the worm; it’s looking for megatons of greenhouse gas emissions.

Early Bird is an energy-efficient method for training deep neural networks (DNNs), the form of artificial intelligence (AI) behind self-driving cars, intelligent assistants, facial recognition and dozens more high-tech applications.

Yingyan Lin, assistant professor of electrical and computer engineering

Yingyan Lin

Researchers from Rice and Texas A&M University unveiled Early Bird April 29 in a spotlight paper at ICLR 2020, the International Conference on Learning Representations. A study by lead authors Haoran You and Chaojian Li of Rice’s Efficient and Intelligent Computing (EIC) Lab showed Early Bird could use 10.7 times less energy to train a DNN to the same level of accuracy or better than typical training. EIC Lab director Yingyan Linled the research along with Rice’s Richard Baraniuk and Texas A&M’s Zhangyang Wang.

“A major driving force in recent AI breakthroughs is the introduction of bigger, more expensive DNNs,” Lin said. “But training these DNNs demands considerable energy. For more innovations to be unveiled, it is imperative to find ‘greener’ training methods that both address environmental concerns and reduce financial barriers of AI research.”

Chaojian Li is a graduate student in electrical and computer engineering at Rice University

Chaojian Li

Haoran You is a graduate student in electrical and computer engineering at Rice University

Haoran You

Training cutting-edge DNNs is costly and getting costlier. A 2019 study by the Allen Institute for AI in Seattle found the number of computations needed to train a top-flight deep neural network increased 300,000 times between 2012-2018, and a different 2019 study by researchers at the University of Massachusetts Amherst found the carbon footprint for training a single, elite DNN was roughly equivalent to the lifetime carbon dioxide emissions of five U.S. automobiles.

DNNs contain millions or even billions of artificial neurons that learn to perform specialized tasks. Without any explicit programming, deep networks of artificial neurons can learn to make humanlike decisions — and even outperform human experts — by “studying” a large number of previous examples. For instance, if a DNN studies photographs of cats and dogs, it learns to recognize cats and dogs. AlphaGo, a deep network trained to play the board game Go, beat a professional human player in 2015 after studying tens of thousands of previously played games.

A graphic illustrating how Rice's Early Bird training method reduces the carbon footprint for training deep neural networks.

Rice University’s Early Bird method for training deep neural networks finds key connectivity patterns early in training, reducing the computations and carbon footprint for the increasingly popular form of artificial intelligence known as deep learning. (Graphic courtesy of Y. Lin/Rice University)

“The state-of-art way to perform DNN training is called progressive prune and train,” said Lin, an assistant professor of electrical and computer engineering in Rice’s Brown School of Engineering. “First, you train a dense, giant network, then remove parts that don’t look important — like pruning a tree. Then you retrain the pruned network to restore performance because performance degrades after pruning. And in practice you need to prune and retrain many times to get good performance.”

Pruning is possible because only a fraction of the artificial neurons in the network can potentially do the job for a specialized task. Training strengthens connections between necessary neurons and reveals which ones can be pruned away. Pruning reduces model size and computational cost, making it more affordable to deploy fully trained DNNs, especially on small devices with limited memory and processing capability.

“The first step, training the dense, giant network, is the most expensive,” Lin said. “Our idea in this work is to identify the final, fully functional pruned network, which we call the ‘early-bird ticket,’ in the beginning stage of this costly first step.”

By looking for key network connectivity patterns early in training, Lin and colleagues were able to both discover the existence of early-bird tickets and use them to streamline DNN training. In experiments on various benchmarking data sets and DNN models, Lin and colleagues found Early Bird could emerge as little as one-tenth or less of the way through the initial phase of training.

Zhangyang Wang is an assistant professor of computer science at Texas A&M University

Zhangyang Wang

Richard Baraniuk

Richard Baraniuk

“Our method can automatically identify early-bird tickets within the first 10% or less of the training of the dense, giant networks,” Lin said. “This means you can train a DNN to achieve the same or even better accuracy for a given task in about 10% or less of the time needed for traditional training, which can lead to more than one order savings in both computation and energy.”

Developing techniques to make AI greener is the main focus of Lin’s group. Environmental concerns are the primary motivation, but Lin said there are multiple benefits.

“Our goal is to make AI both more environmentally friendly and more inclusive,” she said. “The sheer size of complex AI problems has kept out smaller players. Green AI can open the door enabling researchers with a laptop or limited computational resources to explore AI innovations.”

Additional co-authors include Pengfei Xu, Yonggan Fu and Yue Wang, all of Rice, and Xiaohan Chen of Texas A&M.

The research was supported by the National Science Foundation.

 

Author: JADE BOYD

New research addresses ‘link discovery’ problem for terahertz data networks

When you open a laptop, an access point can quickly locate it and connect it to the local Wi-Fi network. That ability, known as link discovery, is a basic element of any wireless network, yet very challenging to accomplish in emerging terahertz bands. The challenge arises from the fact the terahertz links need to be directional to compensate for the significantly higher propagation attenuation. Hence, the access point should know where the clients are in order to aim its beam at them. Conventional solutions for this problem involve a brute-force trial-and-error scan of all beam directions; however, the time overhead associated with such exhaustive training severely degrades throughout and disrupts low-latency applications. More importantly, the direction of transmission needs to be continually adjusted as the client moves requiring repeated training and incurring overhead each time.

The research introduces the first single-shot single-antenna link discovery technique that tracks user moves in nanosecond timescales in order to proactively adapt highly directional terahertz beams. We have proposed a novel node architecture in which the access point and clients are equipped with a leaky-wave antenna. This structure is composed of two metal plates with a space between them where radiation can propagate. One of the plates has a narrow slit cut into it, which allows radiation to leak out with the unique property that the emission angle from the waveguide is coupled to the frequency of the input signal.

We excite this antenna with a time-domain terahertz pulse (equivalently, a broadband signal in frequency-domain). Hence, different frequency components are decomposed and emerged at different angles, such that each direction has a unique spectral peak or “color” due to the leaky-wave antenna’s frequency-angle coupling. This forms a novel transmission pattern that we call a “Terahertz Rainbow” (see the picture above). Terahertz Rainbow enables scanning the entire space around the transmitter in nano-second-scale time and only via a single pulse transmission. Now imagine an access point equipped with a leaky-wave antenna and transmitting a terahertz rainbow. Depending upon where a client device is located relative to the access point, it’s going to see a different color coming out of the waveguide. The client just sends a signal back to the access point that says, “I saw yellow,” and now the access point knows exactly where the client is, and can continue tracking it. This technique allows for ultrafast adaptation, which is the key to achieving seamless connectivity.

This work is published in ACM MobiCom and Nature Communications 2020.