Command Line Interface¶
odpy¶
Ocean Data Parser command line main interface.
Usage:
Options:
--version Show the version and exit.
--verbose Logger format present more information
--log-level [TRACE|DEBUG|INFO|WARNING|ERROR]
Logger level used [default: INFO]
--log-file PATH Log to a file.
--log-file-level [TRACE|DEBUG|INFO|WARNING|ERROR]
Log file level used [default: INFO]
--log-file-rotation TEXT Rotate log file at a given interval. Given
value must be compatible with
pandas.TimeDelta
--log-file-retention TEXT Delete log file after a given time period.
Given value must be compatible with
pandas.TimeDelta (e.g. '1D', '1W'). If None,
file will be kept indefinitely.
--diagnose / --no-diagnose Run loguru diagnose on errors, to see all
variable inputs and stacktrace [default:
diagnose]
--backtrace / --no-backtrace Show stacktrace on error [default:
backtrace]
--backtrace-limit INTEGER Limit stacktrace to N lines [default: 5]
--help Show this message and exit.
convert¶
Ocean Data Parser Batch Conversion CLI Interface.
Usage:
Options:
-i, --input-path TEXT Input path to file list. It can be a glob
expression (ex: *.cnv) or a list of paths
separated by a colons [:] (linux,mac) and
semi-colons [;] (windows).
--exclude TEXT Glob expression of files to exclude.
-p, --parser TEXT Parser used to parse the data. Default to auto
detectection. Use --parser_list to retrieve
list of parsers available
--parser-kwargs TEXT Parser key word arguments to pass to the
parser. Expect a JSON string. (ex:
'{"globa_attributes": {"project": "test"}')
--overwrite Overwrite already converted files when source
file is changed.
--multiprocessing INTEGER Run conversion in parallel on N processors.
None == all processors available
-e, --errors [ignore|raise] Error hanlding method
--registry-path PATH File conversion registry path (*.csv or
*.parquet). If --registry_path=None, no
registry is used.
-o, --output-path PATH Output directory where to save converted
files.
--output-file-name PATH Output file path where to save converted
files.
--output-file-suffix PATH Output file name suffix to add
-c, --config PATH Path to configuration file
--new-config PATH Generate a new configuration file at the given
path
--parser-list Get the list of parsers available
--show-arguments [stop|True] Print present argument values. If stop
argument is given, do not run the conversion.
--version Show the version and exit.
--help Show this message and exit.
inspect-variables¶
Inspect NetCDF files variables and variables attributes.
Usage:
Options:
-i, --input TEXT Glob expression of input files [default:
**/*.nc]
--exclude TEXT Glob expression of files to exclude
--groupby TEXT Comma separated list of attributes to
regroup variables by. [default: variable,un
its,standard_name,sdn_parameter_urn,sdn_para
meter_name]
-m, --multiprocessing INTEGER RANGE
Load files in parallele with n processors.
If the option is set as a flag, all the
processors available (=4) will be used.
[default: 1; 1<=x<=4]
-t, --output-table TEXT Result table output: - None =
console(default) - path to csv or
markdown(*.md) file.
--output-erddap-xml, --xml TEXT
Output an ERDDAP XML blurb or all the
variables.
--show-arguments [stop|True] Print present argument values. If stop
argument is given, do not run the
conversion.
--help Show this message and exit.
Environment Variables
All the inputs available within the odpy command can be defined through environment variables: ODPY_*, ODPY_CONVERT_* and ODPY_COMPILE_* respectively.
Example:
- ODPY_LOG_LEVEL=WARNING will force odpy to log only the warning events.
- ODPY_CONVERT_OUTPUT_PATH=output will force odpy convert to output to the the local directory ./output/
Configuration¶
odpy convert can handle a configuration file which also integrates a number of other features. To generate a new configuration, you can use the --new-config flag:
Or copy the following yaml:
---
name: Batch Conversion Name
input_path: "" # glob expression of files to convert.
# It can be a str, a list of os path separator (: for unix, ; for windows)
# separated list of paths or a list of paths
input_table: # retrieve files to convert from tables (ignore input_path if set)
# input_table is used to retrieve files to convert from tables
# that are listing different glob expressions to retrieve files and associated metadata
# to append to generated converted files as global attributes (optionnal)
path: null # glob expression to retrieve input tables (e.g. 'inputs_tables/*.csv')
file_column: null # column use to generate glob expressions used to retrieve source files to parse
file_column_prefix: "" # prefix to add to the file_column value
file_column_suffix: "" # suffix to add to the file_column value
add_table_name: false # retrieve file name and add as table name column
table_name_column: "table_name" # column name associated with the file name new column to add table name
columns_as_attributes: true # add columns as attributes to the parsed file
exclude_columns: [] # columns to exclude from the table
exclude: null # glob expression of files to exclude
parser: null
parser_kwargs: {}
overwrite: false
multiprocessing: 1 # n processes to run [int] or null for all
errors: "ignore" # raise|ignore
registry:
path: null # file_registry(.csv | .parquet)
sentry:
dsn: Null
level: INFO
event_level: WARNING
traces_sample_rate: 1.0,
# Attributes and Metadata
variable_attributes: {}
global_attributes: {}
file_specific_attributes_path: null # Path to csv file with one column called "file"
global_attribute_mapping:
path: null # Path to csv file (accept glob parameter for multiple files)
mapping: null # mapping dataframe
by: [] # global attributes list
log_level: WARNING # [null, WARNING,ERROR] level log when no mapping exist
# Geospatial References
reference_stations:
path: null
maximum_distance_from_reference_station_km: null
reference_geograhical_areas:
path: null
# Transformations
xarray_pipe: []
# QC
ioos_qc: {}
manual_qc: {}
# Outputs
output:
path: null
file_name: null
file_preffix: ""
file_suffix: ""
output_format: .nc
summary: null # Path to save summary file (csv)