Pen Bay Marine Data documentation¶

Tools for working with Pen Bay Marine Data. This includes command line tools for downloading and processing data, specific to buoys and weather stations in Penobscot Bay, Maine.

penbay¶

Command Line Interface for weather and buoy data.

Usage

penbay [OPTIONS] COMMAND [ARGS]...

buoys¶

Interface for working with buoy data and firmware.

Usage

penbay buoys [OPTIONS] COMMAND [ARGS]...

db¶

Commands that interact with the buoy database.

Usage

penbay buoys db [OPTIONS] COMMAND [ARGS]...

create¶

Create a new Influx DB database.

Usage

penbay buoys db create [OPTIONS] DATABASE

Options

--org <org>¶: Required InfluxDB Organization ID

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

--timeout <timeout>¶: Timeout for the create request in seconds

--retention <retention>¶: Retention period for the bucket in seconds

Arguments

DATABASE¶: Required argument

Environment variables

INFLUX_ORGID: Provide a default for --org

INFLUX_SERVER_URL: Provide a default for --host

INFLUX_API_TOKEN: Provide a default for --token

delete¶

Delete the InfluxDB bucket.

Usage

penbay buoys db delete [OPTIONS] DATABASE

Options

--org <org>¶: Required InfluxDB Organization ID

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

Arguments

DATABASE¶: Required argument

Environment variables

INFLUX_ORGID: Provide a default for --org

INFLUX_SERVER_URL: Provide a default for --host

INFLUX_API_TOKEN: Provide a default for --token

describe¶

Read a measurement (table) back from the Influx database, dropping duplicate timestamps.

Usage

penbay buoys db describe [OPTIONS] {wynken|blynken} {diagnostic|sonde}
                         DATABASE

Options

--org <org>¶: Required InfluxDB Organization ID

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

--limit <limit>¶: Number of records to return

Arguments

NAME¶: Required argument

TABLE¶: Required argument

DATABASE¶: Required argument

Environment variables

INFLUX_ORGID: Provide a default for --org

INFLUX_SERVER_URL: Provide a default for --host

INFLUX_API_TOKEN: Provide a default for --token

diff¶

Determine whether any of the records in the CSV file are missing from the cloud database.

Usage

penbay buoys db diff [OPTIONS] FILENAME DATABASE

Options

--org <org>¶: Required InfluxDB Organization ID

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

Arguments

FILENAME¶: Required argument

DATABASE¶: Required argument

Environment variables

INFLUX_ORGID: Provide a default for --org

INFLUX_SERVER_URL: Provide a default for --host

INFLUX_API_TOKEN: Provide a default for --token

upload¶

Upload a single CSV file to database using the Influx client.

Usage

penbay buoys db upload [OPTIONS] FILENAME DATABASE

Options

--org <org>¶: Required InfluxDB Organization ID

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

Arguments

FILENAME¶: Required argument

DATABASE¶: Required argument

Environment variables

INFLUX_ORGID: Provide a default for --org

INFLUX_SERVER_URL: Provide a default for --host

INFLUX_API_TOKEN: Provide a default for --token

file¶

Interact with the buoy data file system.

Usage

penbay buoys file [OPTIONS] COMMAND [ARGS]...

derivatives¶

Calculate the first and second derivative of a time series to identify rapid changes and spikes in the series. Performs hourly resampling to fill gaps, so that time differences are consistent. The output values are in units per second and per hour, respectively, to use as inputs for a QARTOD configuration file.

Usage

penbay buoys file derivatives [OPTIONS] {wynken|blynken} {diagnostic|sonde} {s
                              ea_water_temperature|mass_concentration_of_oxyge
                              n_in_sea_water|sea_water_salinity|sea_water_chlo
                              rophyll_rfu|sea_water_phycoerythrin_rfu|barometr
                              ic_pressure|battery_voltage|water_pressure|disso
                              lved_oxygen|dissolved_oxygen_saturation}

Options

-p, --percentile <percentile>¶: Percentiles to calculate for the summary statistics. Accepts multiple values, e.g., -p 0.90 -p 0.95 -p 0.99

Arguments

NAME¶: Required argument

TABLE¶: Required argument

SERIES¶: Required argument

describe¶

Summarize available observations for a station.

Usage

penbay buoys file describe [OPTIONS] {wynken|blynken} {diagnostic|sonde}

Arguments

NAME¶: Required argument

TABLE¶: Required argument

export¶

Export buoy data to a CSV with single header row.

Usage

penbay buoys file export [OPTIONS] {wynken|blynken}

Options

-q, --qartod <qartod>¶: Required QARTOD configuration YAML file(s), which will be merged together in the order they are provided.

--test <test>¶

QARTOD test to plot.

Options:: gross_range | rate_of_change | spike | climatology | flat_line | rollup | gap | location

--flag <flag>¶

Filter out this QARTOD flag. 3 = suspect/failed, 4 = failed. Defaults to 3.

Options:: 3 | 4

Arguments

NAME¶: Required argument

firmware¶

Create firmware programs from a template.

Usage

penbay buoys firmware [OPTIONS] COMMAND [ARGS]...

template¶

Fill in a firmware template with options for a specific buoy station. Latitude and longitude are automatically filled based on expected station coordinates if not provided.

Usage

penbay buoys firmware template [OPTIONS] {wynken|blynken}

Options

--address <address>¶: Required Pakbus address, specific to hardware items.

--client <client>¶: Required Client ID, specific to geographic location

--file <file>¶: Template file

--latitude <latitude>¶: Latitude of the station

--longitude <longitude>¶: Longitude of the station

Arguments

NAME¶: Required argument

locations¶

Work with buoy location to assess mooring performance.

Usage

penbay buoys locations [OPTIONS] COMMAND [ARGS]...

names¶

List available stations from static data.

Usage

penbay buoys names [OPTIONS]

plot¶

Generate plots using buoy data.

Usage

penbay buoys plot [OPTIONS] COMMAND [ARGS]...

datastream¶

Plot a generic DataStream aggregated by either daily, weekly, or monthly. This will read in and concatenate all available data files for the station and table, then extract a deduplicated series for the data stream. The output is an image file formatted for a report or presentation.

Usage

penbay buoys plot datastream [OPTIONS] {wynken|blynken} {diagnostic|sonde} {se
                             a_water_temperature|mass_concentration_of_oxygen_
                             in_sea_water|sea_water_salinity|sea_water_chlorop
                             hyll_rfu|sea_water_phycoerythrin_rfu|barometric_p
                             ressure|battery_voltage|water_pressure|dissolved_
                             oxygen|dissolved_oxygen_saturation}

Options

--aggregate <aggregate>¶

Required Frequency for aggregation.

Options:: D | W | ME

--start <start>¶: Start date for filtering data.

--end <end>¶: End date for filtering data.

--size <size>¶: Figure size in inches (width, height).

--image-format <image_format>¶

The output format.

Options:: png | pdf

Arguments

NAME¶: Required argument

TABLE¶: Required argument

SERIES¶: Required argument

tail¶

Plot the most recent data from a buoy for a single data stream.

Usage

penbay buoys plot tail [OPTIONS] {wynken|blynken} {diagnostic|sonde} {sea_wate
                       r_temperature|mass_concentration_of_oxygen_in_sea_water
                       |sea_water_salinity|sea_water_chlorophyll_rfu|sea_water
                       _phycoerythrin_rfu|barometric_pressure|battery_voltage|
                       water_pressure|dissolved_oxygen|dissolved_oxygen_satura
                       tion}

Options

--image-format <image_format>¶

The output format.

Options:: png | pdf

-q, --qartod <qartod>¶: Required QARTOD configuration YAML file(s), which will be merged together in the order they are provided.

--test <test>¶

QARTOD test to plot.

Options:: gross_range | rate_of_change | spike | climatology | flat_line | rollup | gap | location

--days <days>¶: Number of days to plot backwards from the end date. Defaults to 30 days.

--end <end>¶: End date for the analysis and plotting. Defaults to the current date and time.

--scale, --no-scale¶: Scale the output plot to remaining data, otherwise use the full range. Defaults to True.

--figsize <figsize>¶: Size of the output figure in inches (width, height).

Arguments

NAME¶: Required argument

TABLE¶: Required argument

SERIES¶: Required argument

emolt¶

Command Line Interface group for Lobster Trap sensors.

Usage

penbay emolt [OPTIONS] COMMAND [ARGS]...

plot¶

Plot data

Usage

penbay emolt plot [OPTIONS] COMMAND [ARGS]...

acoustic-release¶

Plot acoustic release data

Usage

penbay emolt plot acoustic-release [OPTIONS]

loggers¶

Plot logger data

Usage

penbay emolt plot loggers [OPTIONS] {time|do_concentration|do_saturation|tempe
                          rature|water_detect}

Arguments

DIM¶: Required argument

island¶

Island CLI group.

Usage

penbay island [OPTIONS] COMMAND [ARGS]...

plot¶

Plot data from Govee or Sol Ark devices.

Usage

penbay island plot [OPTIONS] COMMAND [ARGS]...

grid¶

Plot grid data from Sol Ark devices.

Usage

penbay island plot grid [OPTIONS]

lorawan¶

LoRaWAN CLI commands.

Usage

penbay lorawan [OPTIONS] COMMAND [ARGS]...

db¶

Cloud database CLI commands.

Usage

penbay lorawan db [OPTIONS] COMMAND [ARGS]...

mock¶

Send a test message to the Cloudflare Worker that writes to InfluxDB. This is useful for testing the integration without sending real data from a device.

Usage

penbay lorawan db mock [OPTIONS] [DEVICE_ID]

Options

--secret <secret>¶: Webhook secret to use. If not provided, will use WEBHOOK_SECRET from environment.

Arguments

DEVICE_ID¶: Optional argument

Environment variables

WEBHOOK_SECRET: Provide a default for --secret

secret¶

Create a secret to add to TTN and Cloudflare Workers.

Usage

penbay lorawan db secret [OPTIONS]

sync¶

Sync the latest uplink messages from TTN to InfluxDB via the Cloudflare Worker. This is useful for backfilling data that may have been missed by the webhook. Unless values/keys have changed, this should be idempotent and safe to run multiple times.

Usage

penbay lorawan db sync [OPTIONS]

Options

--timeout <timeout>¶: Required Timeout for HTTP requests in seconds.

--api-key <api_key>¶: Required TTN API key with write permissions on devices

--device-id <device_id>¶: Required TTN device ID.

--application-id <application_id>¶: Required TTN application ID.

--subdomain <subdomain>¶: Required TTN subdomain.

--region <region>¶: Required TTN region.

--webhook-secret <webhook_secret>¶: Required Webhook secret to use.

--limit <limit>¶: Maximum number of uplink messages to fetch from TTN.

Environment variables

TTN_API_KEY: Provide a default for --api-key

WEBHOOK_SECRET: Provide a default for --webhook-secret

tail¶

Display a table of signal metrics (RSSI, SNR) for the given device.

Usage

penbay lorawan db tail [OPTIONS]

Options

--timeout <timeout>¶: Required Timeout for HTTP requests in seconds.

--api-key <api_key>¶: Required TTN API key with write permissions on devices

--device-id <device_id>¶: Required TTN device ID.

--application-id <application_id>¶: Required TTN application ID.

--subdomain <subdomain>¶: Required TTN subdomain.

--region <region>¶: Required TTN region.

--limit <limit>¶: Number of latest uplink messages to retrieve.

Environment variables

TTN_API_KEY: Provide a default for --api-key

describe¶

Describe LoRaWAN devices and messages.

Usage

penbay lorawan describe [OPTIONS] COMMAND [ARGS]...

device¶

Device management

Usage

penbay lorawan device [OPTIONS] COMMAND [ARGS]...

uplink¶

Use HTTP API to send a new decoder, so that we can keep the code in git instead of manually uploading.

Reference: https://www.thethingsindustries.com/docs/api/reference/http/routes/#asapplications{end_device.ids.application_ids.application_id}devices{end_device.ids.device_id}-put

Usage

penbay lorawan device uplink [OPTIONS]

Options

--timeout <timeout>¶: Required Timeout for HTTP requests in seconds.

--api-key <api_key>¶: Required TTN API key with write permissions on devices

--device-id <device_id>¶: Required TTN device ID.

--application-id <application_id>¶: Required TTN application ID.

--subdomain <subdomain>¶: Required TTN subdomain.

--region <region>¶: Required TTN region.

Environment variables

TTN_API_KEY: Provide a default for --api-key

plot¶

Plot CLI commands.

Usage

penbay lorawan plot [OPTIONS] COMMAND [ARGS]...

profiles¶

Profile CLI commands.

Usage

penbay profiles [OPTIONS] COMMAND [ARGS]...

plot¶

Plot CLI commands.

Usage

penbay profiles plot [OPTIONS] COMMAND [ARGS]...

map¶

Create a map

Usage

penbay profiles plot map [OPTIONS] PREFIX

Arguments

PREFIX¶: Required argument

single¶

Plot a single depth profile, e.g., temperature, salinity, or density.

Usage

penbay profiles plot single [OPTIONS] FILENAME

Options

--dim <dim>¶

Required Dimension for analysis.

Options:: DEP m | °C | Lat | Lon | DO % | DO mg/L | SAL-ppt | Chl RFU | BGA-PE RFU | density | DEP psia

--step <step>¶: Step size for depth binning.

--figsize <figsize>¶: Figure size as a tuple (width, height).

Arguments

FILENAME¶: Required argument

transect¶

Create a plot of a transect, with value linearly interpolated between sites.

Usage

penbay profiles plot transect [OPTIONS] PREFIX

Options

--dim <dim>¶

Required Dimension for analysis.

Options:: DEP m | °C | Lat | Lon | DO % | DO mg/L | SAL-ppt | Chl RFU | BGA-PE RFU | density | DEP psia

--levels <levels>¶: Bins for colormap breaks.

Arguments

PREFIX¶: Required argument

weather¶

Weather station commands.

Usage

penbay weather [OPTIONS] COMMAND [ARGS]...

db¶

Commands that interact with the weather database.

Usage

penbay weather db [OPTIONS] COMMAND [ARGS]...

backfill¶

Backfill missing data from local to database.

Usage

penbay weather db backfill [OPTIONS] {apprenticeshop|dev}

Options

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--measurement <measurement>¶: The InfluxDB measurement (table) name.

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

Arguments

STATION¶: Required argument

Environment variables

INFLUX_SERVER_URL: Provide a default for --host

INFLUXDB_MEASUREMENT: Provide a default for --measurement

INFLUX_API_TOKEN: Provide a default for --token

describe¶

Show information about the data already stored in Influx database.

Usage

penbay weather db describe [OPTIONS]

Options

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--measurement <measurement>¶: The InfluxDB measurement (table) name.

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

Environment variables

INFLUX_SERVER_URL: Provide a default for --host

INFLUXDB_MEASUREMENT: Provide a default for --measurement

INFLUX_API_TOKEN: Provide a default for --token

describe¶

Compare local and database data before merging or backfilling.

Usage

penbay weather describe [OPTIONS] {apprenticeshop|dev} {air_temperature|relati
                        ve_humidity|wind_speed|wind_from_direction|air_pressur
                        e|wind_speed_of_gust|wind_gust_from_direction|wind_chi
                        ll_of_air_temperature|solar_irradiance|ultraviolet_ind
                        ex|rainfall_amount|rainfall_rate|heat_index_of_air_tem
                        perature|dew_point_temperature|water_evapotranspiratio
                        n_flux}

Options

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--measurement <measurement>¶: The InfluxDB measurement (table) name.

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

Arguments

STATION¶: Required argument

SERIES¶: Required argument

Environment variables

INFLUX_SERVER_URL: Provide a default for --host

INFLUXDB_MEASUREMENT: Provide a default for --measurement

INFLUX_API_TOKEN: Provide a default for --token

file¶

Commands that interact with the weather file system.

Usage

penbay weather file [OPTIONS] COMMAND [ARGS]...

describe¶

Parse and normalize weather station data for display

Usage

penbay weather file describe [OPTIONS] {apprenticeshop|dev}

Arguments

STATION¶: Required argument

export¶

Export normalized weather station data to CSV.

Usage

penbay weather file export [OPTIONS] {apprenticeshop|dev}

Arguments

STATION¶: Required argument

plot¶

Commands that generate plots using weather data.

Usage

penbay weather plot [OPTIONS] COMMAND [ARGS]...

daily¶

Display a single DataStream aggregated by day.

Usage

penbay weather plot daily [OPTIONS] {apprenticeshop|dev} {air_temperature|rela
                          tive_humidity|wind_speed|wind_from_direction|air_pre
                          ssure|wind_speed_of_gust|wind_gust_from_direction|wi
                          nd_chill_of_air_temperature|solar_irradiance|ultravi
                          olet_index|rainfall_amount|rainfall_rate|heat_index_
                          of_air_temperature|dew_point_temperature|water_evapo
                          transpiration_flux}

Options

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--measurement <measurement>¶: The InfluxDB measurement (table) name.

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

--image-format <image_format>¶

The output format.

Options:: png | pdf

Arguments

STATION¶: Required argument

SERIES¶: Required argument

Environment variables

INFLUX_SERVER_URL: Provide a default for --host

INFLUXDB_MEASUREMENT: Provide a default for --measurement

INFLUX_API_TOKEN: Provide a default for --token

tail¶

Plot a comparison of local and InfluxDB data for a specific series.

Keyword arguments are passed through to the rendering function

Usage

penbay weather plot tail [OPTIONS] {apprenticeshop|dev} {air_temperature|relat
                         ive_humidity|wind_speed|wind_from_direction|air_press
                         ure|wind_speed_of_gust|wind_gust_from_direction|wind_
                         chill_of_air_temperature|solar_irradiance|ultraviolet
                         _index|rainfall_amount|rainfall_rate|heat_index_of_ai
                         r_temperature|dew_point_temperature|water_evapotransp
                         iration_flux}

Options

--host <host>¶: The InfluxDB server URL. Defaults to environment variable INFLUX_SERVER_URL

--measurement <measurement>¶: The InfluxDB measurement (table) name.

--token <token>¶: The InfluxDB API token. Defaults to environment variable INFLUX_API_TOKEN

--image-format <image_format>¶

The output format.

Options:: png | pdf

--days <days>¶: The time range to include in the plot (e.g., 7, 14).

--resample <resample>¶: How to resample the data (e.g., 1d, 1h).

--qartod <qartod>¶: Path to the QARTOD configuration file.

Arguments

STATION¶: Required argument

SERIES¶: Required argument

Environment variables

INFLUX_SERVER_URL: Provide a default for --host

INFLUXDB_MEASUREMENT: Provide a default for --measurement

INFLUX_API_TOKEN: Provide a default for --token