pykern package

Subpackages

• pykern.pkcli package
  • Submodules
  • pykern.pkcli.github module
  • pykern.pkcli.pkexample module
  • pykern.pkcli.projex module
  • pykern.pkcli.pytest module
  • pykern.pkcli.rsmanifest module
  • pykern.pkcli.sim module
  • Module contents

Submodules

pykern.base_pkconfig module

Default config

copyright:Copyright (c) 2015 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.base_pkconfig.alpha()

pykern.base_pkconfig.beta()

pykern.base_pkconfig.dev()

pykern.base_pkconfig.prod()

pykern.mpi module

MPI support routines

copyright:Copyright (c) 2017 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.mpi.checked_call(op)

Abort MPI if op raises an exception.

If op doesn’t handle an exception, then MPI needs to abort. This will terminate the MPI process, not just the one task.

If MPI is not running or mpi4py is not installed, then passes through the exception.

Parameters:op (func) – function to call

pykern.pkarray module

Wrapper for array to simplify and make future compatible.

Not a complete wrapper. New routines added as required.

copyright:Copyright (c) 2015 Bivio Software, Inc. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkarray.DOUBLE_TYPECODE = 'd'

Future-proof typecode for double

pykern.pkarray.FLOAT_TYPECODE = 'f'

Future-proof typecode for float

pykern.pkarray.new_double(*args, **kwargs)

Creates a new double (“d”) array

Args are the same as array.array() except for typecode, which is passed by this module.

Returns:New, initialized array Return type:array.array

pykern.pkarray.new_float(*args, **kwargs)

Creates a new float (“f”) array

Args are the same as array.array() except for typecode, which is passed by this module.

Returns:New, initialized array Return type:array.array

pykern.pkcollections module

Ordered attribute mapping type

Similar to argparse.Namespace, but is ordered, and behaves like dictionary except there are no public methods on OrderedMapping. All operations are operators or Python builtins so that the attribute names from clients of a OrderedMapping don’t collide.

copyright:Copyright (c) 2015 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

class pykern.pkcollections.Dict

Bases: dict

A subclass of dict that allows items to be read/written as attributes.

The purpose of this is as a convenience in coding. You can refer to dictionary keys as attributes as long as they don’t collide with the object’s attributes. You should always use the dict interface to refer to the items of the dictionary programmatically, just like you would do with Javascript.

You can reference any dict key as an attribute as long as it does not conflict with an attribute. For example, this works:

x = Dict()
x.a = 1
assert 1 == x.a
x['a'] = 3
assert 3 == x.a
assert 'a' == x.keys()[0]

You can’t set an attribute that already exists. These calls throw exceptions:

x.values = 1
delattr(x, 'values')

dict doesn’t allow this anyway. However, you can’t set or delete any existing attribute, even writable attributes. Indeed, you can’t delete attributes at all. Subclasses should be “containers” only, not general objects.

nested_get(dotted_key)

Split key on dots and return nested get calls

Throws KeyError if the dictionary key doesn’t exist.

Parameters:dotted_key (str) – what Returns:value of element Return type:object

exception pykern.pkcollections.DictNameError

Bases: exceptions.NameError

Raised when a key matches a builtin attribute in dict.

class pykern.pkcollections.OrderedMapping(*args, **kwargs)

Bases: object

Ordered mapping can be initialized by kwargs or single argument.

Parameters:

• first (object) – copy map in order of iterator, OR
• kwargs (dict) – initial values (does not preserver order)

All operations are munged names to avoid collisions with the clients of OrderedMapping so there are no “methods” on self except operator overloads.

pykern.pkcollections.json_load_any(obj, *args, **kwargs)

Read json file or str with object_pairs_hook=Dict

Parameters:

• obj (object) – str or object with “read”
• args (tuple) – passed verbatim
• kwargs (dict) – object_pairs_hook overriden

Returns:

parsed JSON

Return type:

object

pykern.pkcollections.map_items(value, op=None)

Iterate over mapping, calling op with key, value

Parameters:

• value (object) – Any object that implements iteration on keys
• op (function) – called with each key, value, in order (default: return (key, value))

Returns:

list of results of op

Return type:

list

pykern.pkcollections.map_keys(value, op=None)

Iterate over mapping, calling op with key

Parameters:

• value (object) – Any object that implements iteration on keys
• op (function) – called with each key, in order (default: return key)

Returns:

list of results of op

Return type:

list

pykern.pkcollections.map_to_dict(value)

Convert mapping to dictionary

Parameters:value (object) – mapping to convert Returns:Converted mapping Return type:dict

pykern.pkcollections.map_values(value, op=None)

Iterate over mapping, calling op with value

Parameters:

• value (object) – Any object that implements iteration on values
• op (function) – called with each key, in order (default: return value)

Returns:

list of results of op

Return type:

list

pykern.pkcollections.mapping_merge(base, to_merge)

Add or replace values from to_merge into base

Parameters:

• base (object) – Implements setitem
• to_merge (object) – implements iter and getitem

pykern.pkcollections.object_pairs_hook(*args, **kwargs)

Tries to use Dict if else uses dict

Returns:Dict or dict Return type:object

pykern.pkcollections.unchecked_del(obj, key)

Deletes the key from obj

Parameters:

• obj (object) – What to delete from (usually dict)
• key (object) – What to delete

pykern.pkcompat module

Python 2 and 3 compatbility routines

six and future.utils do most things, but there are some missing things here

copyright:Copyright (c) 2015 Bivio Software, Inc. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkcompat.isinstance_str(value)

Portable test for str

Parameters:value (object) – to test Returns:True if value is a str or unicode Return type:bool

pykern.pkcompat.locale_str(value)

Converts a value to a unicode str unless already unicode.

Parameters:value – The string or object to be decoded, may be None. Returns:decoded string (PY2: type unicode) Return type:str

pykern.pkcompat.unicode_getcwd()

os.getcwd() unicode wrapper

Returns:current directory (PY2: type unicode) Return type:str

pykern.pkconfig module

Declarative module configuration with dynamic value injection

Module Declaration

Modules declare their configuration via init. Here is how pkdebug declares its config params:

cfg = pkconfig.init(
    control=(None, re.compile, 'Pattern to match against pkdc messages'),
    want_pid_time=(False, bool, 'Display pid and time in messages'),
    output=(None, _cfg_output, 'Where to write messages either as a "writable" or file name'),
)

A param tuple contains three values:

0. Default value, in the expected type
1. Callable that can convert a string or other type into a valid value
2. A docstring briefly explaining the configuration element

The returned cfg object is ready to use after the call. It will contain the config params as defined or an exception will be raised.

Channel Files

Configuration files are python modules, which define functions for each channel to be configured. A channel is a stage of deployment. There are four channels:

dev

This is the default channel. It’s what developers use to configure their systems.

alpha

The first stage of a deployment. The configuration supports automated testing. Customer data is stored on alpha systems.

beta

First stage of customer use. The configuration supports both test and real users. Since there is customer data, encryption keys should be something randomly generated.

prod

Production systems contain customer data so they should be configured for backups, privacy, and scaling.

The name of the channel is specified by the environment variable $PYKERN_PKCONFIG_CHANNEL. If not set, the channel will be dev.

Config Files

Config modules are found along a load path defined by the environment variable $PYKERN_PKCONFIG_LOAD_PATH or set by an entry point module, e.g. pykern.pkcli. The load path consists of root-level package names (e.g. pykern) to identify modules.

Each package in the load path may contain a <pkg>.base_pkconfig.py, which will be imported first. All the base_pkconfig modules are loaded before any other files, and they are imported in the order of the load path.

Next anay “home files” of the form ~/.<pkg>_pkconfig.py are imported in the order of the load path.

Once loaded the channel method for each module in the load path are called. These Each loaded module can override any values.

One last level of configuration is environment values for individual parameters. If an environment variable exists that matches the upper case, underscored parameter name, it will override all other values. For example, you can set $PYKERN_PKDEBUG_OUTPUT to /dev/tty if you want to set the output parameter for pykern.pkdebug so that pkdebug writes debug output to the terminal.

Config Values

The values of parameters in config files are specified in nested dictionaries. The channel function must return a type level dictionary with the package roots as the first keys, then the submodules, and which then point to parameters.

Suppose we have my_app that uses Flask and wants pkdebug to stdout in development. Here’s what my_app/base_pkcoonfig.py might contain:

import os
import sys

def dev():
    return {
        'my_app': {
            'flask_init': {
                'db': 'sqlite://' + os.getcwd() + '/my_app.db',
            },
        },
        'pykern': {
            'pkdebug': {
                'output': sys.stdout,
            },
        },
    }

Configuration is returned as nested dicts. The values themselves could be any Python object. In this case, we have a string and a file object for the two parameters. We called os.getcwd and referred to sys.stdout in param values.

Summary

Here are the steps to configuring an application:

1. When the first module calls init, pkconfig reads all module config and environment variables to create a single dict of param values, unparsed, by calling merge repeatedly.
2. init looks for the module’s params by indexing with (root_pkg, submodule, param) in the merged config.
3. If the parameter is found, that value is used. Else, the default is merged into the dict and used.
4. The parameter value is then resolved with str.format. If the value is a list it will be joined with any previous value (e.g. default).
5. The resolved value is parsed using the param’s declared parser.
6. The result is stored in the merged config and also stored in the module’s Params object .
7. Once all params have been parsed for the module, init returns the Params object to the module, which can then use those params to initialize itself.

copyright:Copyright (c) 2015 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkconfig.BASE_MODULE = '{}.base_pkconfig'

Name of the module (required) for a package

pykern.pkconfig.CHANNEL_DEFAULT = 'dev'

Initialized channel (same as cfg.channel)

pykern.pkconfig.CHANNEL_ENV_NAME = 'PYKERN_PKCONFIG_CHANNEL'

Environment variable holding channel (defaults to ‘dev’)

pykern.pkconfig.HOME_FILE = '~/.{}_pkconfig.py'

Name of the file to load in user’s home directory if exists

pykern.pkconfig.INTERNAL_TEST_CHANNELS = ('dev', 'alpha')

Channels which can have more verbose output from the server

pykern.pkconfig.KEY_RE = <_sre.SRE_Pattern object>

Validate key – Cannot begin with non-letter or end with an underscore

pykern.pkconfig.LOAD_PATH_DEFAULT = ['pykern']

Load path default value

pykern.pkconfig.LOAD_PATH_ENV_NAME = 'PYKERN_PKCONFIG_LOAD_PATH'

Environment variable holding the load path

pykern.pkconfig.LOAD_PATH_SEP = ':'

Separator for load_path string

class pykern.pkconfig.Required

Bases: tuple, object

Container for a required parameter declaration.

Example:

cfg = pkconfig.init(
    any_param=(1, int, 'A parameter with a default'),
    needed=pkconfig.Required(int, 'A parameter with a default'),
)

Parameters:

• converter (callable) – how to string to internal value
• docstring (str) – description of parameter

pykern.pkconfig.STRING_TYPES

alias of __builtin__.basestring

pykern.pkconfig.THIS_PACKAGE = 'pykern'

Root package implicit

pykern.pkconfig.TUPLE_SEP = ':'

parse_tuple splits strings on this

pykern.pkconfig.VALID_CHANNELS = ('dev', 'alpha', 'beta', 'prod')

Order of channels from least to most stable

pykern.pkconfig.all_modules_in_load_path(path_module=None)

Loads all modules in path_module

Finds all modules in cfg.load_path matching the main_module sans root. If path_module is sirepo.pkcli, then the loaded modules will look like <root>.pkcli.<base>. Only goes one depth.

Parameters:path_module (module) – full path module [caller module] Returns:map of base names to module objects Return type:pkcollection.Dict

pykern.pkconfig.append_load_path(load_path)

Called by entry point modules to add packages into the load path

Parameters:load_path (str or list) – separate by : or list of packages to append

pykern.pkconfig.cfg = None

Configuration for this module – channel and load_path. Available after first init() call

pykern.pkconfig.channel_in(*args, **kwargs)

Test against configured channel

Parameters:

• args (str) – list of channels to valid
• channel (str) – channel to test (default: [cfg.channel])

Returns:

True if current channel in args

Return type:

bool

pykern.pkconfig.channel_in_internal_test(channel=None)

Is this a internal test channel?

Parameters:channel (str) – channel to test (default: [cfg.channel]) Returns:True if current channel in (alpha, dev) Return type:bool

pykern.pkconfig.flatten_values(base, new)

Merge flattened values into base

Keys are made all lowercase.

Lists instances are prepended and not recursively merged.

Parameters:

• base (object) – dict-like that is already flattened
• new (object) – dict-like that will be flattened and overriden

pykern.pkconfig.init(**kwargs)

Declares and initializes config params for calling module.

Parameters:kwargs (dict) – param name to (default, parser, docstring) Returns:pkcollections.OrderedMapping populated with param values Return type:Params

pykern.pkconfig.parse_bool(value)

Default parser for bool types

When the parser is bool, it will be replaced with this routine, which parses strings and None specially. bool values cannot be defaulted to None. They must be True or False.

String values which return true: t, true, y, yes, 1.

False values: f, false, n, no, 0, ‘’, None

Other values are parsed by bool.

Parameters:value (object) – to be parsed Returns:True or False Return type:bool

pykern.pkconfig.parse_none(func)

Decorator for a parser which can parse None

Parameters:callable – function to be decorated Returns:func with attr indicating it can parse None Return type:callable

pykern.pkconfig.parse_tuple(value)

Default parser for tuple types

When the parser is tuple, it will be replaced with this routine, which parses strings, lists, and tuples. It splits strings on ‘:’.

Parameters:value (object) – to be parsed Returns:may be an empty tuple Return type:tuple

pykern.pkconfig.reset_state_for_testing(add_to_environ=None)

Clear the raw values so we can change load paths dynamically

Only used for unit tests. add_to_environ overrides previous value.

Parameters:add_to_environ (dict) – values to augment to os.environ

pykern.pkdebug module

Logging or regex-controlled print statements

We take the view that there are two types of logging: permanent (pkdp) and dynamically controlled (pkdc). In both cases, you want to know where the print statement originated from. Python’s logging module supports logging the origin, but it doesn’t support user- defined dynamic filters in the form of a regular expression.

Permanent print statements (pkdp) is what people traditionally think of as “logging”. What we don’t care about is “logging levels”, because one person’s “critical” is another person’s “whatever”. It’s up to operations people to figure out what’s an error worth paying attention to, and what’s not. That’s managed by external rule bases, not inside some library module.

Python’s logging module gives programmers too many options and adds complexity to filters. If you don’t attach a handler to the loggers, you’ll never see some messages. Or, if the handler has logging.WARN set, and you really want to see logging.INFO messages, you won’t see them.

There are cases when you really don’t want to see output. We call these dynamically controlled print statements. They are typically used by developers when debugging a new module, and sometimes these lines get left in the code for a while so that you can get detailed (noisy) debugging information on production systems.

Dynamically controlled print statements (pkdc) would overwhelm logs so they are off by default and can be filtered by regular expressions supplied via configuration (including environment variables) or programmatically via init.

Example

In a module, you would write:

from pykern.debug import pkdc, pkdexc, pkdp

pkdp('user entered: {}', val)
pkdc('user context: name={name}, id={id}', **user_rec)

If you do nothing, the first print statement would come out always. The second wouldn’t come out unless you specified a “”control” via the environment variable $PYKERN_PKDEBUG_CONTROL:

PYKERN_PKDEBUG_CONTROL=my_mod python my_prog.py

Or, if you want a specific conditional print:

PYKERN_PKDEBUG_CONTROL=my_mod.py:52:

You can match any text in the line output with a regular expression, which is case insensitive.

If output is a string, will open the file to write to. The initial value of output is $PYKERN_PKDEBUG_OUTPUT.

copyright:Copyright (c) 2014-2016 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkdebug.MAX_EXCEPTION_COUNT = 5

Maximum number of exceptions thrown before printing stops

pykern.pkdebug.init(**kwargs)

May be called to (re)initialize this module.

control is a regular expression, which is used to control the output of pkdc(). Messages from pkdp() and pkdc() are written to output.

output is either an object which implements write or a str, in which case it is opened with io.open().

Parameters:

• control (str or re.RegexObject) – lines matching will be output
• output (str or file) – where to write messages [error output]
• redirect_logging (bool) – Redirect Python’s logging to output [True]
• want_pid_time (bool) – display PID and time in messages [False]

pykern.pkdebug.pkdc(fmt, *args, **kwargs)

Conditional print a message to output selectively based on control.

Controlled output that you can leave in your code permanently.

Parameters:

• fmt (str) – how to str.format()
• args – what to format
• kwargs – what to format

pykern.pkdebug.pkdexc()

Return last exception and stack as a string

Must be called from an except. This function removes the last two calls from the stack. It joins the traceback so you get a complete stack trace.

Will catch exceptions during the formatting and returns a string in all cases.

Example:

try:
    something
except:
    pkdp(pkdexc())

Returns:formatted exception and stack trace Return type:str

pykern.pkdebug.pkdlog(fmt_or_arg, *args, **kwargs)

Print a message to output unconditionally, possibly returning its arg

Use for print statements in your code that you don’t intend to keep in the system.

Use pkdlog for permanent log messages.

Parameters:

• fmt_or_arg (object) – how to str.format(), or object to print
• args – what to format
• kwargs – what to format

Returns:

Will return fmt_or_arg, if args and kwargs are empty

Return type:

object

pykern.pkdebug.pkdp(fmt_or_arg, *args, **kwargs)

Print a message to output unconditionally, possibly returning its arg

Use for print statements in your code that you don’t intend to keep in the system.

Use pkdlog for permanent log messages.

Parameters:

• fmt_or_arg (object) – how to str.format(), or object to print
• args – what to format
• kwargs – what to format

Returns:

Will return fmt_or_arg, if args and kwargs are empty

Return type:

object

pykern.pkdebug.pkdpretty(obj)

Return pretty print the object.

If obj is JSON, parse and print it. If it is a regular python data structure, pretty print that. Any exceptions are caught, and the return value will be obj.

Parameters:obj (object) – JSON string or python object Returns:pretty printed string Return type:str

pykern.pkexample module

Demonstrates a RadiaSoft style module.

This module demonstrates how we code at RadiaSoft. In general we follow the Google Python Style Guide and the Sphinx Napoleon Google Docstrings Example.

Some Rules:

We adhere to PEP8 unless overruled by Google’s Style Guide or explicit rules such as:

1. Indent by four (4) spaces and no tabs.
2. Avoid lines over 80 characters. Not everybody’s laptop can show two 80+ char pages side by side.
3. All modules are Python 2 and 3 compatible.
4. Modules are divided into groups of declarations: public constants, private constants, public global variables, private global variables, public classes, private classes, public functions, and private functions. Within each group, the declarations are sorted alphabetically.
5. Logs, exceptions and assertions contain values, not just messages. If a value is in error, include it in the message along with the expected value, range, etc.
6. :func:pykern.pkdebug.pkdp to print logging messages, and :func:pykern.pkdebug.pkdc to print trace messages.
7. Configuration is specified by :mod:pykern.pkconfig.
8. Use single quotes for strings. Use double quotes for docstrings.
9. TODO(robnagler) is how we represent things to do in a comment

Docstrings begin and end with three double quotes (“). On the line with the beginning double quotes, you write a one line summary of the function, class, or module.

copyright:Copyright (c) 2016 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

class pykern.pkexample.EMA(length)

Bases: object

Exponential moving average

Used as a demonstration of numerical computations. We document the __init__ method at the class level, since it is an implicit function call.

Parameters:length (int) – iterations

average

float – current value of the average

length

int – time period

compute(value)

Compute the next value in the average

Parameters:value (float) – next number in the average Returns:current value of the average Return type:float

value()

Get the average

Returns:current value of the average Return type:float

pykern.pkexample.ONE = 1

First constant is first alphabetically

pykern.pkexample.ZIPPITY = 'doodah'

Another constant

pykern.pkexample.cfg = OrderedMapping(any_length=1)

The module’s config is a public variable. It is initialized at the end.

pykern.pkinspect module

Helper functions for to inspect.

copyright:Copyright (c) 2015 RadiaSoft, Inc. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

class pykern.pkinspect.Call(frame_or_log)

Bases: pykern.pkcollections.Dict

Saves file:line:name of stack frame and renders as string.

Parameters:frame_or_log (frame or LogRecord) – values to extract

filename

str – full path (co_filename)

lineno

int – line number (f_lineno)

name

str – function name (co_name)

pykern.pkinspect.caller(ignore_modules=None, exclude_first=True)

Which file:line:func is calling the caller of this function.

Will not return the same module as the calling module, that is, will iterate until a new module is found. If ignore_modules is defined, will ignore those modules as well.

Note: may return __main__ module.

Will raise exception if calling from __main__

Parameters:

• ignore_modules (list) – other modules (objects) to exclude [None]
• exclude_first (bool) – skip first module found [True]

Returns:

keys: filename, lineno, name, module

Return type:

pkcollections.Dict

pykern.pkinspect.caller_module()

Which module is calling the caller of this function.

Will not return the same module as the calling module, that is, will iterate until a new module is found.

Note: may return __main__ module.

Will raise exception if calling from __main__

Returns:module which is calling module Return type:module

pykern.pkinspect.is_caller_main()

Is the caller’s calling module __main__?

Returns:True if calling module was called by __main__. Return type:bool

pykern.pkinspect.is_valid_identifier(string)

Is this a valid Python identifier?

Parameters:string (str) – what to validate Returns:True if is valid python ident. Return type:bool

pykern.pkinspect.module_basename(obj)

Parse the last part of a module name

For example, module_basename(pkinspect) is ‘pkinspect’.

Parameters:obj (object) – any python object Returns:base part of the module name Return type:str

pykern.pkinspect.module_name_join(names)

Joins names with ‘.’

Parameters:names (iterable) – list of strings to join Returns:module name Return type:str

pykern.pkinspect.module_name_split(obj)

Splits obj’s module name on ‘.’

Parameters:obj (object) – any python object Returns:base part of the module name Return type:str

pykern.pkinspect.root_package(obj)

Parse the root package in which obj is defined.

For example, root_package(module_basename) is ‘pykern’.

Parameters:obj (object) – any python object Returns:root package for the object Return type:str

pykern.pkinspect.submodule_name(obj)

Remove the root package in which obj is defined.

For example, root_package(module_basename) is ‘pkinspect’.

Parameters:obj (object) – any python object Returns:submodule for the object Return type:str

pykern.pkinspect.this_module()

Module object for caller

Returns:module object Return type:module

pykern.pkio module

Useful I/O operations

copyright:Copyright (c) 2015 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkio.exception_is_not_found(exc)

True if exception is IOError and ENOENT

Parameters:exc (BaseException) – to check Returns:True if is a file not found exception. Return type:bool

pykern.pkio.expand_user_path(path)

Calls expanduser on path

If pkunit_prefix is set, will prefix, too.

Parameters:path (str) – path to expand Returns:expanded path Return type:py.path.Local

pykern.pkio.has_file_extension(filename, to_check)

if matches any of the file extensions

Parameters:

• filename (str|py.path.local) – what to check
• to_check (str|tuple|list) – is without ‘.’ and lower

Returns:

if any of the extensions matches

Return type:

bool

pykern.pkio.mkdir_parent(path)

Create the directories and their parents (if necessary)

Parameters:path (str) – dir to create Returns:path Return type:py.path.local

pykern.pkio.mkdir_parent_only(path)

Create the paths’ parent directories.

Parameters:path (str) – children of dir to create Returns:parent directory of path Return type:py.path.local

pykern.pkio.pkunit_prefix = None

used during unit testing see pykern.pkunit.save_chdir

pykern.pkio.py_path(path=None)

Creates a py.path.Local object

Will expanduser, if needed.

If pkunit_prefix is set, will prefix, too.

Parameters:path (str) – path to convert (or None for current dir) Returns:path Return type:py.path.Local

pykern.pkio.read_text(filename)

Open file, read with preferred encoding text, and close.

Parameters:filename (str or py.path.Local) – File to open Returns:contest of filename Return type:str

pykern.pkio.save_chdir(*args, **kwds)

Save current directory, change to directory, and restore.

Parameters:

• dirname (str) – directory to change to
• mkdir (bool) – Make the directory?
• is_pkunit_prefix (bool) – If True, sets pkunit_prefix.

Returns:

current directory before chdir

Return type:

str

pykern.pkio.sorted_glob(path)

sorted list of py.path.Local objects, non-recursive

Parameters:path (py.path.Local or str) – pattern Returns:py.path.Local objects Return type:list

pykern.pkio.unchecked_remove(*paths)

Remove files or directories, ignoring OSError.

Will not remove ‘/’ or ‘.’

Parameters:paths (str) – paths to remove

pykern.pkio.walk_tree(dirname, file_re=None)

Return list files (only) as py.path’s, top down, sorted

If you want to go bottom up, just reverse the list.

Parameters:

• dirname (str) – directory to walk
• match_re (re or str) – Optionally, only return files which match file_re

Yields:

py.path.local – paths in sorted order

pykern.pkio.write_text(filename, contents)

Open file, write text with preferred encoding, and close.

Parameters:

• filename (str or py.path.Local) – File to open
• contents (str) – New contents

Returns:

filename as py.path.Local

Return type:

py.path.local

pykern.pkjinja module

Simplify rendering jinja2

copyright:Copyright (c) 2015 Bivio Software, Inc. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkjinja.render_file(filename, j2_ctx, output=None, strict_undefined=False)

Render filename as template with j2_ctx.

Parameters:

• basename (str) – name without jinja extension
• j2_ctx (dict) – how to replace values in Jinja2 template
• output (str) – file name of output; if None, return str
• strict_undefined (bool) – set jinja2.StrictUndefined if True

Returns:

rendered template

Return type:

str

pykern.pkjinja.render_resource(basename, *args, **kwargs)

Render a pkresource as a jinja template.

Parameters:

• basename (str) – name without jinja extension
• args (list) – see func:render_file for rest of args and return

pykern.pkjson module

JSON help

copyright:Copyright (c) 2017 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkjson.dump_pretty(obj, filename=None, pretty=True)

Formats as json as string

Parameters:

• obj (object) – any Pyton object
• filename (str or py.path) – where to write [None]
• pretty (bool) – pretty print [True]

Returns:

sorted and formatted JSON

Return type:

str

pykern.pkjson.load_any(obj)

Calls pkcollections.json_load_any

Parameters:obj (object) – str or object with “read” Returns:parsed JSON Return type:object

pykern.pkplatform module

Wrapper for Python’s platform to provide cleaner programmatic control of system features.

copyright:Copyright (c) 2015 Bivio Software, Inc. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkplatform.is_darwin()

All flavors of Mac (OS X)

Returns:True if sys.platform is Mac. Return type:bool

pykern.pkplatform.is_linux()

All flavors of linux

Returns:True if sys.platform is Linux. Return type:bool

pykern.pkplatform.is_unix()

All flavors of Unix. Currently, it’s darwin, linux, cygwin, freebsd, netbsd, sunos, solaris, unixware, irix, aix, and next.

Returns:True if sys.platform is a pure Unix system (e.g. not beos) Return type:bool

pykern.pkplatform.is_windows()

All flavors of Windows (32, 64, cygwin, etc.). If your program expects a unix flavor, you will want is_unix().

Returns:True if sys.platform is win32 or cygwin Return type:bool

pykern.pkresource module

Where external resources are stored

copyright:Copyright (c) 2015 Bivio Software, Inc. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkresource.filename(relative_filename, caller_context=None)

Return the filename to the resource

Parameters:

• relative_filename (str) – file name relative to package_data directory.
• caller_context (object) – Any object from which to get the root_package

Returns:

absolute path of the resource file

Return type:

str

pykern.pkrunpy module

Run python code

copyright:Copyright (c) 2015 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkrunpy.run_path_as_module(fname)

Runs fname in a module

Parameters:fname (str or py.path.local) – file to be exec’d Returns:imported file as a module Return type:module

pykern.pksetup module

Wrapper for setuptools.setup to simplify creating of setup.py files.

Python setup.py files should be short for well-structured projects. b_setup.setup assumes there are directories such as tests, docs, bin, etc. PyKern Projects use py.test so the appropriate PyTest class is provided by this module.

Example

A sample setup.py script:

setup(
    name='pyexample',
    description='Some Example app',
    author='Example, Inc.',
    author_email='somebody@example.com',
    url='http://example.com',
)

Assumptions:

• the use of pytest for tests. GUI and console scripts are found automatically by special suffixes _gui.py and _console.py. See setup documentation for an example.
• Under git control. Even if you are building an app for the first time, you should create the repo first. Does not assume anything about the remote (i.e. need not be a GitHub repo).

copyright:Copyright (c) 2015 Radiasoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

class pykern.pksetup.NullCommand(dist)

Bases: distutils.cmd.Command, object

Use to eliminate a cmdclass.

Does nothing but complies with distutils.cmd.Command protocol.

finalize_options(**kwargs)

Set final values for all the options that this command supports. This is always called as late as possible, ie. after any option assignments from the command-line or from other commands have been done. Thus, this is the place to code option dependencies: if ‘foo’ depends on ‘bar’, then it is safe to set ‘foo’ from ‘bar’ as long as ‘foo’ still has the same value it was assigned in ‘initialize_options()’.

This method must be implemented by all command classes.

initialize_options(**kwargs)

Set default values for all the options that this command supports. Note that these defaults may be overridden by other commands, by the setup script, by config files, or by the command-line. Thus, this is not the place to code dependencies between options; generally, ‘initialize_options()’ implementations are just a bunch of “self.foo = None” assignments.

This method must be implemented by all command classes.

run(**kwargs)

A command’s raison d’etre: carry out the action it exists to perform, controlled by the options initialized in ‘initialize_options()’, customized by other commands, the setup script, the command-line, and config files, and finalized in ‘finalize_options()’. All terminal output and filesystem interaction should be done by ‘run()’.

This method must be implemented by all command classes.

user_options = []

pykern.pksetup.PACKAGE_DATA = 'package_data'

The subdirectory in the top-level Python where to put resources

class pykern.pksetup.PKDeploy(dist)

Bases: pykern.pksetup.NullCommand

Run tests, build sdist or wheel, upload. Only use this on a clean git repo.

The command will build the distro, then run tests on it with tox, which sets up a virtual environment.

You must have the following environment variables:

$PKSETUP_PYPI_USER

Name of the user to login as on pypi

$PKSETUP_PYPI_PASSWORD

Name of the password

This optional variable is useful for testing out your distro:

$PKSETUP_PYPI_IS_TEST

If set, will use testpypi, otherwise uses pypi.python.org

All values provided by environment variables.

description = 'Runs git clean and tox; if successful, uploads to (test)pypi'

run()

A command’s raison d’etre: carry out the action it exists to perform, controlled by the options initialized in ‘initialize_options()’, customized by other commands, the setup script, the command-line, and config files, and finalized in ‘finalize_options()’. All terminal output and filesystem interaction should be done by ‘run()’.

This method must be implemented by all command classes.

pykern.pksetup.PYTEST_INI_FILE = 'pytest.ini'

Created only during PyTest run

class pykern.pksetup.PyTest(dist, **kw)

Bases: setuptools.command.test.test, object

Proper initialization of pytest for python setup.py test

See also :mod:pykern.pytest_plugin.

finalize_options()

Initialize test_args and set test_suite to True

run_tests()

Import pytest and calls main. Calls sys.exit with result

pykern.pksetup.SCRIPTS_DIR = 'scripts'

Where scripts live, you probably don’t want this

class pykern.pksetup.SDist(dist, **kw)

Bases: setuptools.command.sdist.sdist, object

Fix up a few things before running sdist

check_readme(*args, **kwargs)

Avoid README error message. We assert differntly.

Currently only supports README.txt and README, but we may have README.md.

pykern.pksetup.TESTS_DIR = 'tests'

Where the tests live

pykern.pksetup.TOX_INI_FILE = 'tox.ini'

Created only during Tox run

class pykern.pksetup.Tox(dist, **kw)

Bases: setuptools.Command, object

Create tox.ini file

description = 'create tox.ini and run tox'

finalize_options(*args, **kwargs)

Set final values for all the options that this command supports. This is always called as late as possible, ie. after any option assignments from the command-line or from other commands have been done. Thus, this is the place to code option dependencies: if ‘foo’ depends on ‘bar’, then it is safe to set ‘foo’ from ‘bar’ as long as ‘foo’ still has the same value it was assigned in ‘initialize_options()’.

This method must be implemented by all command classes.

initialize_options(*args, **kwargs)

Set default values for all the options that this command supports. Note that these defaults may be overridden by other commands, by the setup script, by config files, or by the command-line. Thus, this is not the place to code dependencies between options; generally, ‘initialize_options()’ implementations are just a bunch of “self.foo = None” assignments.

This method must be implemented by all command classes.

run(*args, **kwargs)

A command’s raison d’etre: carry out the action it exists to perform, controlled by the options initialized in ‘initialize_options()’, customized by other commands, the setup script, the command-line, and config files, and finalized in ‘finalize_options()’. All terminal output and filesystem interaction should be done by ‘run()’.

This method must be implemented by all command classes.

user_options = []

pykern.pksetup.install_requires()

Parse requirements.txt.

Returns:parsed requirements.txt Return type:dict

pykern.pksetup.setup(**kwargs)

Parses README.* and requirements.txt, sets some defaults, then calls setuptools.setup.

Scripts are found by looking for files in the top level package directory which end with _console.py or _gui.py. These files must have a function called main.

Example

The file pykern_console.py might contain:

def main():
    print('hello world')

This would create a program called command line program pykern which would call main() when invoked.

Parameters:kwargs – see setuptools.setup

pykern.pksubprocess module

Wrapper for subprocess.

copyright:Copyright (c) 2016 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pksubprocess.check_call_with_signals(cmd, output=None, env=None, msg=None)

Run cmd, writing to output.

stdin is os.devnull.

Passes SIGTERM and SIGINT on to the child process. If output is a string, it will be opened in write (‘w’) mode.

Parameters:

• cmd (list) – passed to subprocess verbatim
• output (file or str) – where to write stdout and stderr
• env (dict) – environment to use

pykern.pkunit module

Useful operations for unit tests

copyright:Copyright (c) 2015 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkunit.assert_object_with_json(basename, actual)

Converts actual to JSON and compares with data_dir/basename.json

Reads data_dir/basename.json and compares with actual converted to json. Trailing newline is managed properly. The keys are sorted and indentation is 4. actual written to work_dir.

Parameters:

• expected_basename (str) – file to be found in data_dir with json suffix
• actual (object) – to be serialized as json

pykern.pkunit.data_dir()

Compute the data directory based on the test name

The test data directory is always <test>_data, where <test> is the name of the test’s python module with the _test or test_ removed. For example, if the test file is setup_test.py then the directory will be setup_data.

Returns:data directory Return type:py.path.local

pykern.pkunit.data_yaml(base_name)

Load base_name.yml from data_dir

Parameters:base_name (str) – name of YAML file with .yml extension Returns:YAML data structure, usually dict or array Return type:object

pykern.pkunit.empty_work_dir()

Remove work_dir if it exists and create.

All contents of the test directory will be removed.

Returns:empty work directory Return type:py.path.local

pykern.pkunit.import_module_from_data_dir(module_name)

Add data_dir to sys.path and import module_name.

Note that module_name with be removed from the sys.modules cache before loading in case the module was loaded by another test.

Parameters:module_name (str) – module relative to data_dir to import. Returns:imported module Return type:module

pykern.pkunit.module_under_test = None

Set to the most recent test module by pykern.pytest_plugin

pykern.pkunit.pkeq(expect, actual, *args, **kwargs)

If actual is not expect, throw assertion with calling context.

Parameters:

• expect (object) – what to test for
• actual (object) – run-time value
• args (tuple) – passed to pkfail()
• kwargs (dict) – passed to pkfail()

pykern.pkunit.pkexcept(*args, **kwds)

Expect an exception to be thrown and match or output msg

If fmt_and_args is falsey, will generate a message saying what was expected and what was received.

Examples:

# Expect an exception (or its subclass)
with pkexcept(AssertionError, 'did not expect this'):
    assert 0

# Expect exception to contain a specific message
with pkexcept('match this', 'problem with matching'):
    assert 0, 'some string with "match this" in it'

# Use a default output message
with pkexcept(KeyError):
    something['key will not be found']

Parameters:

• exc_or_re (object) – BaseException, re, or str; if str, compiled with re.IGNORECASE
• fmt_and_args (tuple) – passed to format
• kwargs (dict) – passed to format

Yields:

None – just for context manager

pykern.pkunit.pkfail(fmt, *args, **kwargs)

Format message and raise AssertionError.

Parameters:

• fmt (str) – to be passed to string.format
• args (tuple) – passed to format
• kwargs (dict) – passed to format

pykern.pkunit.pkok(cond, fmt, *args, **kwargs)

If cond is not true, throw assertion with calling context

Parameters:

• cond (object) – expression which should evaluate to true
• fmt (str) – to be passed to string.format
• args (tuple) – passed to format
• kwargs (dict) – passed to format

Returns:

obj value

Return type:

object

pykern.pkunit.pkre(expect_re, actual, flags=18)

If actual does not match (re.search) expect_re, throw assertion with calling context.

Parameters:

• expect_re (object) – string or re object
• actual (object) – run-time value
• flags – passed on to re.search [IGNORECASE + DOTALL]

pykern.pkunit.random_alpha(length=6)

Random lowercase alpha string

Parameters:length (int) – how many chars Returns:lower case alpha string Return type:str

pykern.pkunit.save_chdir_work(is_pkunit_prefix=False)

Create empty work_dir and chdir

Parameters:is_pkunit_prefix (bool) – use as root of (most) file I/O (optional) Returns:empty work directory Return type:py.path.local

pykern.pkunit.work_dir()

Returns ephemeral work directory, created if necessary.

To enable easier debugging, the test directory is always <test>_work, where <test> is the name of the test’s python module with the _test or test_ removed. For example, if the test file is setup_test.py then the directory will be setup_work.

The name “work” distinguishes from “tmp”, which could imply anything. Also, with editor autocomplete, “setup_work” and “setup_test” are more easily distinguishable.

Returns:directory name Return type:py.path

pykern.pkyaml module

Wrapper for yaml

copyright:Copyright (c) 2015 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pkyaml.load_file(filename)

Read a file, making sure all keys and values are locale.

Parameters:filename (str) – file to read (Note: .yml will not be appended) Returns:pkcollections.Dict or list Return type:object

pykern.pkyaml.load_resource(basename)

Read a resource, making sure all keys and values are locale

Parameters:basename (str) – file to read without yml suffix Returns:pkcollections.Dict or list Return type:object

pykern.pkyaml.load_str(value)

Read a value, making sure all keys and values are locale.

Parameters:value (str) – string to parse Returns:pkcollections.Dict or list Return type:object

pykern.pykern_console module

Front-end command line for pykern.pkcli.

Example:

copyright:Copyright (c) 2015 Bivio Software, Inc. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pykern_console.main()

pykern.pytest_plugin module

PyTest plugin to setup pkconfig and add pkunit fixtures

This plugin will only be “active” if the setup.py in the package imports pykern.pksetup. This module turns on pytest-xdist’s --boxed option. It also calls pykern.pkconfig.append_load_path, which modifies global state.

copyright:Copyright (c) 2016 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html

pykern.pytest_plugin.pytest_configure(config)

See if package uses pykern, and set options accordingly

Parameters:config (_pytest.config.Config) – used for options

pykern.pytest_plugin.pytest_ignore_collect(path, config)

Ignore _work and _data directories

pykern.pytest_plugin.pytest_runtest_protocol(item, *args, **kwargs)

Make sure work directory is empty for a module.

If item is in a module not seen before, it removes the pkunit.work_dir.

Parameters:item (Item) – pytest test item (case) Returns:always so that the next hook runs the item. Return type:None

Module contents

pykern package

copyright:Copyright (c) 2018 RadiaSoft LLC. All Rights Reserved. license:http://www.apache.org/licenses/LICENSE-2.0.html