2.16. python_utils package
2.16.1. Submodules
2.16.2. python_utils.check_for_preexist_dir_file module
- python_utils.check_for_preexist_dir_file.check_for_preexist_dir_file(path, method)[source]
Check for a preexisting directory or file and, if present, deal with it according to the specified method
- Parameters
path – path to directory
method – could be any of [ ‘delete’, ‘reuse’, ‘rename’, ‘quit’ ]
- Returns
None
2.16.3. python_utils.check_var_valid_value module
2.16.4. python_utils.config_parser module
This file provides utilities for processing different configuration file formats. Supported formats include:
YAML
JSON
SHELL
INI
XML
Typical usage involves first loading the config file, then using the dictionary returnded by load_config to make queries.
- python_utils.config_parser.cfg_main()[source]
Main function for converting and formatting between different config file formats
- python_utils.config_parser.cfg_to_ini_str(cfg, kname=None)[source]
Get contents of config file as ini string
- python_utils.config_parser.cfg_to_json_str(cfg)[source]
Get contents of config file as a json string
- python_utils.config_parser.cfg_to_shell_str(cfg, kname=None)[source]
Get contents of config file as shell script string
- python_utils.config_parser.cfg_to_yaml_str(cfg)[source]
Get contents of config file as a yaml string
- python_utils.config_parser.check_structure_dict(dict_o, dict_t)[source]
Check if a dictionary’s structure follows a template. The invalid entries are returned as a dictionary. If all entries are valid, returns an empty dictionary
- Parameters
dict_o – target dictionary
dict_t – template dictionary to compare structure to
- Returns
Invalid key-value pairs.
- Return type
dict
- class python_utils.config_parser.custom_dumper(stream, default_style=None, default_flow_style=False, canonical=None, indent=None, width=None, allow_unicode=None, line_break=None, encoding=None, explicit_start=None, explicit_end=None, version=None, tags=None, sort_keys=True)[source]
Bases:
DumperCustom yaml dumper to correct list indentation
- python_utils.config_parser.cycstr(loader, node)[source]
Returns a cyclestring Element whose content corresponds to the input node argument
- python_utils.config_parser.days_ago(arg)[source]
A filter for jinja2 that gives us a date string for x number of days ago
- python_utils.config_parser.extend_yaml(yaml_dict, full_dict=None, parent=None)[source]
Updates yaml_dict inplace by rendering any existing Jinja2 templates that exist in a value.
- python_utils.config_parser.filter_dict(dict_o, keys_regex)[source]
Filter dictionary keys based on a list of keys :param dict_o: the source dictionary :param keys_regex: list of keys to retain (could be regex exp.)
- python_utils.config_parser.flatten_dict(dictionary, keys=None)[source]
Flatten a recursive dictionary (e.g.yaml/json) to be one level deep
- Parameters
dictionary – the source dictionary
keys – list of keys on top level whose contents to flatten, if None all of them
- Returns
A one-level deep dictionary for the selected set of keys
- python_utils.config_parser.get_ini_value(config, section, key)[source]
Finds the value of a property in a given section
- python_utils.config_parser.include(filepaths)[source]
Returns a dictionary that includes the contents of the referenced YAML file(s).
- python_utils.config_parser.load_config_file(file_name, return_string=0)[source]
Load config file based on file name extension
- python_utils.config_parser.load_ini_config(config_file, return_string=0)[source]
Load a config file with a format similar to Microsoft’s INI files
- python_utils.config_parser.load_shell_as_ini_config(file_name, return_string=1)[source]
Load shell config file with embedded structure in comments
- python_utils.config_parser.load_shell_config(config_file, return_string=0)[source]
Loads old style shell config files. We source the config script in a subshell and gets the variables it sets
- Parameters
config_file – path to config file script
- Returns
dictionary that should be equivalent to one obtained from parsing a yaml file.
- python_utils.config_parser.load_xml_config(config_file, return_string=0)[source]
Load xml config file
- python_utils.config_parser.startstopfreq(loader, node)[source]
Returns a Rocoto-formatted string for the contents of a cycledef tag. Assume that the items in the node are env variables, and return a Rocoto-formatted string
- python_utils.config_parser.structure_dict(dict_o, dict_t)[source]
Structure a dictionary based on a template dictionary
- Parameters
dict_o – dictionary to structure (flat one level structure)
dict_t – template dictionary used for structuring
- Returns
A dictionary with contents of dict_o following structure of dict_t
2.16.5. python_utils.create_symlink_to_file module
- python_utils.create_symlink_to_file.create_symlink_to_file(target, symlink, relative=True)[source]
Create a symbolic link to the specified target file.
- Parameters
target – target file
symlink – symbolic link to target file
relative – optional argument to specify relative symoblic link creation
- Returns
None
2.16.6. python_utils.define_macos_utilities module
2.16.7. python_utils.environment module
- python_utils.environment.date_to_str(d, format='%Y%m%d%H%M')[source]
Get string from python datetime object. By default it converts to YYYYMMDDHHMM format unless told otherwise by passing a different format
- Parameters
d – datetime object
- Returns
string in YYYYMMDDHHMM or shorter version of it
- python_utils.environment.export_vars(dictionary=None, source_dict=None, env_vars=None)[source]
Export all (or select few) global variables of the caller module’s to either the environement/dictionary. Call this function at the end of a function that updates environment variables.
- Parameters
dictionary – target dictionary to set (default=os.environ)
source_dict – source dictionary (default=caller modules globals())
env_vars – list of selected environement/dictionary variables to export, or None,
exported (in which case all environment/dictionary variables are) –
- Returns
None
- python_utils.environment.get_env_var(param)[source]
Get the value of an environement variable
- Parameters
param – the environement variable
- Returns
Returns either a string, list of strings or None
- python_utils.environment.import_vars(dictionary=None, target_dict=None, env_vars=None)[source]
Import all (or select few) environment/dictionary variables as python global variables of the caller module. Call this function at the beginning of a function that uses environment variables.
Note that for read-only environmental variables, calling this function once at the beginning should be enough. However, if the variable is modified in the module it is called from, the variable should be explicitly tagged as global, and then its value should be exported back to the environment with a call to export_vars()
import_vars() # import all environment variables global MY_VAR, MY_LIST_VAR MY_PATH = “/path/to/somewhere” MY_LIST_VAR.append(“Hello”) export_vars() # these exports all global variables
There doesn’t seem to an easier way of imitating the shell script doing way of things, which assumes that everything is global unless specifically tagged local, while the opposite is true for python.
- Parameters
dictionary – source dictionary (default=os.environ)
target_dict – target dictionary (default=caller module’s globals())
env_vars – list of selected environement/dictionary variables to import, or None,
imported (in which case all environment/dictionary variables are) –
- Returns
None
- python_utils.environment.list_to_str(v, oneline=False)[source]
Given a string or list of string, construct a string to be used on right hand side of shell environement variables
- Parameters
v – a string/number, list of strings/numbers, or null string(‘’)
- Returns
A string
- python_utils.environment.set_env_var(param, value)[source]
Set an environment variable
- Parameters
param – the variable to set
value – either a string, list of strings or None
- Returns
None
- python_utils.environment.str_to_date(s)[source]
Get python datetime object from string.
- Parameters
s – a string
- Returns
datetime object or None
- python_utils.environment.str_to_list(v, return_string=0)[source]
Given a string, construct a string or list of strings. Basically does the reverse operation of list_to_string.
- Parameters
v – a string
- Returns
a string, list of strings or null string(‘’)
- python_utils.environment.str_to_type(s, return_string=0)[source]
Check if the string contains a float, int, boolean, datetime, or just regular string. This will be used to automatically convert environment variables to data types that are more convenient to work with. If you don’t want this functionality, pass return_string = 1
- Parameters
s – a string
return_string – Set to 1 to return the string itself Set to 2 to return the string itself only for a datetime object
- Returns
a float, int, boolean, datetime, or the string itself when all else fails
2.16.8. python_utils.filesys_cmds_vrfy module
2.16.9. python_utils.fv3write_parms_lambert module
2.16.10. python_utils.misc module
- python_utils.misc.find_pattern_in_file(pattern, file_name)[source]
Find regex pattern in a file
- Parameters
pattern – regex expression
file_name – name of text file
- Returns
A tuple of matched groups or None
- python_utils.misc.find_pattern_in_str(pattern, source)[source]
Find regex pattern in a string
- Parameters
pattern – regex expression
source – string
- Returns
A tuple of matched groups or None
2.16.11. python_utils.print_input_args module
2.16.12. python_utils.print_msg module
- python_utils.print_msg.log_info(info_msg, verbose=True, dedent_=True)[source]
Function to print information message using the logging module. This function should not be used if python logging has not been initialized.
- Parameters
info_msg – info message to print
verbose – set to False to silence printing
dedent – set to False to disable “dedenting” (print string as-is)
- Returns
None
- python_utils.print_msg.print_err_msg_exit(error_msg='', stack_trace=True)[source]
Function to print out an error message to stderr and exit. It can optionally print the stack trace as well.
- Parameters
error_msg – error message to print
stack_trace – set to True to print stack trace
- Returns
None
- python_utils.print_msg.print_info_msg(info_msg, verbose=True)[source]
Function to print information message to stdout, when verbose is set to True. It does proper “dedentation” that is needed for readability of python code.
- Parameters
info_msg – info message to print
verbose – set to False to silence printing
- Returns
if message is successfully printed
- Return type
True