Data Cleaning
The Challenge
The Piscataqua Region Estuaries Partnership (PREP) operates a single buoy in New Hampshire. They collaborate with the University of New Hampshire to experiment with brackish condition water quality sensors.
Water quality instrumentation gets fouled easily in brackish conditions, requiring regular cleaning and extensive field notes to create usable data. This is time consuming work and is a barrier to increasing water quality monitoring in these conditions.
PREP tested multiple water quality sensors on their buoy, including the YSI EXO2 Sonde, which monitors ammonium, chlorophyll, algae, and dissolved oxygen, and the Seabird SUNA V2 nitrate sensor.
Solutions using Dendra
PREP used Dendra as an offline data processing system. They manually uploaded their data from the buoy as CSV files. Then they wrote Python scripts to process the field notes of the field techs who cleaned and fixed the instruments, automatically creating Annotations. Using the Dendra API, they bulk uploaded all the data cleaning changes to the data as Annotation events. Currently there are 1,890 Annotations cleaning 40 datastreams.
The result is a cleaned dataset with a full record of all modifications.
This effort has shown the potential for integrating an electronic field note app with the Dendra annotation system. Doing so can make for an extremely efficient data cleaning workflow, even with sensors that require as much manual intervention as PREP’s water quality sensors do.
Learn More
