Energy-efficient system for detection of elephants with Machine Learning
Author: Marko Sagadin, Embedded Machine Learning Engineer at IRNAS
Introduction
I recently wrote a Master’s thesis on detecting elephants from thermal cameras, with the help of Machine Learning. Thesis is publicly available on my GitHub repository.
Classifying images has come a long way in the last decade; at this point, it is quite a norm in the field of web, mobile and desktop applications, however, it is not yet widely used in embedded systems.
Recent developments in the field of embedded Machine Learning, also known as TinyML, have pushed boundaries of what is possible. Thanks to TensorFlow Lite for Microcontrollers and Edge Impulse, we can now efficiently run neural networks on small embedded devices.
The main focus of this thesis was the design, implementation and evaluation of Machine Learning models running on a general-purpose microcontroller under low-power conditions. The majority of thermal images was provided by Arribada Initiative, who have already done some grunt work in elephant detection as seen here and here. Thermal images are publicly available on their GitHub repository.
This blog post is a short summary of my work, for an in-depth description of the problem, implementation and results, I suggest looking at my GitHub repository.
1. Early warning system
But why exactly would we want to classify elephants from thermal cameras?
The reason are Human-Elephant Conflicts (HEC), as they are a major problem in terms of elephant conservation.
According to WILDLABS, an average of 400 people and 100 elephants are killed every year in India alone because of them. One of the proposed solutions to the problem are the early warning systems.
Early warning systems replace the role of human watchers and warn local communities of nearby, potentially life-threatening, elephants, thus minimising Human-Elephant Conflicts.
A system capable of detecting the presence of nearby elephants would warn nearby communities and give them enough time to prepare and respond non-violently. The same kind of system would also provide information about common elephant paths, thus giving farmers knowledge on how to construct and place their fences better to minimise HEC.
The system should consist of several small, deployed devices with mounted cameras that will detect elephants and communicate wirelessly with a server. The server would display messages and alerts, and if needed, directly notify the local community, if elephants were detected in the local area.
Deployed devices will be placed far from the main server, in an area with an unreliable cellular connectivity, which makes sending a large amount of data a problem.
This limits the choice of wireless networks to long-range, low bandwidth technologies.
It is, therefore, preferable that elephant detection is done on deployed devices, and only results (which can be few bytes big) are sent over some radio network to the main server. Deployed devices will be placed in forests, fields, with no access to electricity, therefore, they need to be battery powered.
Low maintenance of deployed devices is a desirable quality, which means that they should be functional for longer periods without any human interactions. To achieve that with a limited power source, they should be energy-efficient, equipped with solar panels, and a low power radio. Devices should spend most of their time in sleep mode, conserving energy, only waking up to take a photo, processing it, and sending results to the main server.
2. Developing Neural Network
I developed a Neural Network with the help of TensorFlow, Keras and Jupyter Notebooks.
As I wanted to develop a small enough model, which would be able to fit on a microcontroller we decided to only classify four distinct classes: Elephants, humans, cows and nature/random.
These objects were highly likely to be captured by a thermal camera, thus they needed to be reliably classified. I cleaned up the image dataset and extracted images suitable for our classes.
For the model I chose a simple Convolutional Neural Network with a small number of layers, implementation in Keras can be seen below:
model = models.Sequential()model.add(Conv2D(FilterNum1, FilterSize,
activation='relu',
padding="same",
input_shape=(60,80, 1)))
model.add(MaxPooling2D((2, 2)))model.add(Conv2D(FilterNum2, FilterSize,
activation='relu',
padding="same")
model.add(MaxPooling2D((2, 2)))model.add(Conv2D(FilterNum3, FilterSize,
activation='relu',
padding="same")
model.add(Flatten())model.add(Dense(DenseSize, activation='relu'))
model.add(Dropout(DropoutRate))
model.add(Dense(4), activation='softmax')
Notice that I did not hard-code the number of filters, their size or size of fully connected layers.
Because these values are hard to determine heuristically, I decided on a possible hyperparameter range and used Keras Tuner module to perform a random search and train a large number of models. As a comparison we also used Edge Impulse Studio to train Neural Network models, some of them had similar general convolutional structure to ours, others were trained with Transfer Learning technique, which used a pre-trained MobileNetV2 network.
3. Hardware and firmware
To capture thermal images, we used FLIR Lepton 2.5 thermal camera, which was used by Arribada Initiative to create thermal image dataset. For running inference I chose STM32F767ZI microcontroller with 216 MHz system clock, 2 MB of flash and 512 KB of RAM. Using such powerful microcontroller guaranteed us that we would be able to fit a model in the memory and run it as fast as possible.
However, such high performance comes with a large current consumption, STM32 was reaching over 125 mA during inference. As we did not want both the thermal camera and STM32 microcontroller running all the time we needed another microcontroller which could turn them on and off. For that, we used NRF52840 microcontroller, which has a low current consumption during sleep state and is ideal for monitoring the whole system. The block diagram of the system can be seen below, for wireless communication module we used LoRa transceiver LR1110 chip.
General sequence of elephant detection is seen on the figure below.
A large part of this thesis was concerned with porting TensorFlow for Microcontrollers to the libopencm3, our platform of choice. During this process, we developed a small project that we named MicroML. MicroML enables users to develop ML applications on microcontrollers supported by libopencm3.
NRF52840 microcontroller firmware was developed with the help of Zepyhr RTOS.
4. Results
We saw that we could train CNN models with an accuracy as high as 98.04 % and we saw that models trained with the Transfer Learning technique reached 98.7 % without much additional work. We ran each trained model on the STM32 and measured its inference time. From the results, we saw that reaching an inference time below 200 ms is not hard if special care is devoted to the low number of parameters and correct microcontroller optimisations.
We also evaluated the performance of the embedded system from the battery lifetime perspective and we saw promising results. Assuming that our system would have to process 600 events per day, while connected from 6 battery cells, we estimated the lifetime of the system would be around 10 months.
Conclusion
This thesis shows that the field of Machine Learning on embedded devices has reached a point where it is viable enough to use it in real-life applications. Running inference directly on the embedded device can provide instant feedback and can extend the device’s lifetime, as data do not need to be sent to the server for processing. Many use cases from the Animal Conservation field would benefit from such technology, as there is a need for the embedded devices that require minimal manual care and can provide big value for their cost.
Machine Learning on the embedded devices is opening doors to various, wonderful applications that were not possible a few years ago. We can see that there is already a demand for intelligence for devices on the “edge” and we expect it to increase as the field matures.
It is quite possible that future embedded engineers would require some amount of Machine Learning knowledge to stay competitive and interesting to the market.
