Neural Networks and Photogrammetry for Analysis of Marine Remote Sensing Data

Project Summary

KC and Patrick led two hands-on data workshops for ENVIRON 335: Drones in Marine Biology, Ecology, and Conservation. These labs were intended to introduce students to examples of how drones are currently being used as a remote sensing tool to monitor marine megafauna and their environments, and how machine learning can be used to efficiently analyze remote sensing datasets. The first lab specifically focused on how drones are being used to collect aerial images of whales to measure changes in body condition to help monitor populations. Students were introduced to the methods for making accurate measurements and then received an opportunity to measure whales themselves. The second lab then introduced analysis methods using computer vision and deep neural networks to detect, count, and measure objects of interest in remote sensing data. This work provided students in the environmental sciences an introduction to new techniques in machine learning and remote sensing that can be powerful multipliers of effort when analyzing large environmental datasets.

Themes and Categories
Year
2019

Graduate Students: KC Bierlich and Patrick Gray

Faculty: Dr. David Johnston

Undergraduate Course: "Drones in Marine Biology, Ecology, and Conservation" (ENVIRON 335)

Course Summary

Learning Objectives

Lab 1:

  • Understand how to accurately measure objects from aerial images depending on different camera/sensors and the altitude of the photo.
  • Understand why it is important to measure body condition and health of top predators, such as whales, in an ecosystem.
  • Hands-on activity to measure the total length of blue, humpback, and minke whales from California and Antarctica using drone imagery.
  • Compare results to classmates/class and discuss reasons for variation in measurements.

Lab 2:

  • Understand basic remote sensing technology, temporal, spectral, spatial, and radiometric characteristics of different sensors.
  • Learn some intuitive ideas about machine learning and applications in marine science
  • Build up a library of examples of how remote sensing and machine learning can be used for marine science.
  • Learn more specifically how deep learning functions and gain an understanding of the theory behind convolutional neural networks. Use this knowledge and the general remote sensing knowledge to develop a neural network model to classify animals and everyday objects.

The first lab consisted a short lecture on the theory of photogrammetry and its conservation importance and then students worked directly with the drone imagery using ImageJ for hands-on photogrammetry. The second lab session began with a short crash course in machine learning and a summary of applications in marine remote sensing. Students then had time to familiarize themselves with the datasets and the python tools we used in a guided format using Jupyter notebooks stored on Github and run on the Google Colaboratory environment. We then trained our neural network models as a group but with each student running it independently on Colab.

Artificial intelligence-based techniques are bringing incredible advances to difficult research problems, but these methods are often inaccessible to non-technical ecologists. Developing an intuitive understanding of neural networks, without the complicated setup and coding challenges, was an essential goal of this, with the hope that students will be empowered enough to bring these new techniques into their own research.

Datasets

We used three datasets during this project. Lab 1 used images of blue, humpback, and Antarctic minke whales collected by our research group in Antarctica and California. Lab 2’s initial Neural Network Development was done on the publicly available MINST and CIFAR datasets. Students used these to experiment with model design and then as a final component of the project we demonstrated how a neural network could be used to identify and measure cetaceans using the same dataset as Lab 1.

Course Materials

Drone images of whales that students measured can be found here.

Students also downloaded ImageJ, a photogrammetry software, that they used for measuring. All code from Lab 2 can be found here. An overview of both deep learning and remote sensing intended to be an hour long lecture can be found here.

Graphics

Equation for measuring whales
Figure 1) Equation and example of measuring whales using aerial photogrammetry.
Drone imagery examples
Figure 2) Example of drone imagery in raw form and after being analyzed by a convolutional neural network to identify the species and measure each animal.
CIFAR dataset images
Figure 3) CIFAR Dataset for neural network model development

 

Related Projects

In this two-day, virtual data expedition project, students were introduced to the APIM in the context of stress proliferation, linked lives, the spousal relationship, and mental and physical health outcomes.

Stress proliferation is a concept within the stress process paradigm that explains how one person’s stressors can influence others (Thoits 2010). Combining this with the life course principle of linked lives explains that because people are embedded in social networks, stress not only can impact the individual but can also proliferate to people close to them (Elder Jr, Shanahan and Jennings 2015). For example, one spouse’s chronic health condition may lead to stress-provoking strain in the marital relationship, eventually spilling over to affect the other spouse’s mental health. Additionally, because partners share an environment, experiences, and resources (e.g., money and information), as well as exert social control over each other, they can monitor and influence each other’s health and health behaviors. This often leads to health concordance within couples; in other words, because individuals within the couple influence each other’s health and well-being, their health tends to become more similar or more alike (Kiecolt-Glaser and Wilson 2017, Polenick, Renn and Birditt 2018). Thus, a spouse’s current health condition may influence their partner’s future health and spouses may contemporaneously exhibit similar health conditions or behaviors.

However, how spouses influence each other may be patterned by the gender of the spouse with the health condition or exhibiting the health behaviors. Recent evidence suggests that a wife’s health condition may have little influence on her husband’s future health conditions, but that a husband’s health condition will most likely influence his wife’s future health (Kiecolt-Glaser and Wilson 2017).

Fluid mechanics is the study of how fluids (e.g., air, water) move and the forces on them. Scientists and engineers have developed mathematical equations to model the motions of fluid and inertial particles. However, these equations are often computationally expensive, meaning they take a long time for the computer to solve.

 

To reduce the computation time, we can use machine learning techniques to develop statistical models of fluid behavior. Statistical models do not actually represent the physics of fluids; rather, they learn trends and relationships from the results of previous simulation experiments. Statistical models allow us to leverage the findings of long, expensive simulations to obtain results in a fraction of the time. 

 

In this project, we provide students with the results of direct numerical simulations (DNS), which took many weeks for the computer to solve. We ask students to use machine learning techniques to develop statistical models of the results of the DNS.

Female baboons occasionally exhibit large swellings on their behinds. Although these ‘sexual swellings’ may evoke disgust from human on-lookers, they provide important information to group members about a female’s reproductive state. To figure out what these sexual swellings mean and whether male baboons notice, we need to look at the data.