Results and files#

Module pianistpy.io

Writing results to files and reading results back from files.

A result is written by the result’s write method and read back by read. write_result and read_result implement the HDF5 file format that the write methods and read share. write_table exports columns of a result, each with a column name, as a text table.

class pianistpy.io.StoredResult(kind, identity, data_sha256, record, arrays, parents, provenance)#

A result as one file stores the result.

read_result returns a StoredResult; the reader registered for the result’s kind (register_reader) rebuilds the result object from the StoredResult.

Parameters:

kind (str) – The kind of result, for example "model".
identity (str) – The identity string of the result.
data_sha256 (str) – The SHA-256 digest of the file’s datasets (data_digest), as a hexadecimal string of 64 characters.
record (dict) – The inputs and settings that determine the result, in dictionary form.
arrays (dict of str to array) – The arrays of the result, by name.
parents (tuple of str) – The identity strings of the results from which the result was computed.
provenance (dict of str to str) – The creation date of the file and the versions and numerical backend of the session that wrote the file (backend_provenance).

See also

read_result: The function that reads a file into a StoredResult.
write_result: The function that writes a result to a file.

Examples

>>> import os
>>> import tempfile
>>> import numpy as np
>>> from pianistpy.io import read_result, write_result
>>> with tempfile.TemporaryDirectory() as folder:
...     path = os.path.join(folder, "example.h5")
...     arrays = {"x": np.arange(3.0)}
...     _ = write_result(path, "example", "example-1", {"n": 3}, arrays)
...     stored = read_result(path)
>>> stored.kind, stored.identity, stored.record, stored.parents
('example', 'example-1', {'n': 3}, ())
>>> stored.arrays["x"]
array([0., 1., 2.])

pianistpy.io.backend_provenance()#

Return the software versions and the numerical backend of the running session.

write_result stores the returned dictionary in every file the function writes.

Returns:: provenance – The keys python_version, numpy_version and scipy_version; jax_platforms_env and pianistpy_platform_env, the values of the environment variables JAX_PLATFORMS and PIANISTPY_PLATFORM, each "unset" where the variable is not set; and jax_version, jax_backend and jax_devices. A session in which JAX cannot be imported records "unavailable" for jax_version and jax_devices and "none" for jax_backend.
Return type:: dict of str to str

See also

write_result: The function that stores the provenance in each file.

Examples

>>> from pianistpy.io import backend_provenance
>>> sorted(backend_provenance())[:3]
['jax_backend', 'jax_devices', 'jax_platforms_env']

pianistpy.io.data_digest(source)#

Return the SHA-256 digest of every dataset of an HDF5 file or group.

The digest covers every dataset, in the sorted order of the datasets’ paths: each dataset’s path, data type, shape and raw bytes. The digest excludes the attributes, so a file can store the digest of its own datasets as an attribute.

Parameters:: source (h5py.Group or path) – An open HDF5 file or group, or the path of an HDF5 file.
Returns:: digest – The hexadecimal digest, 64 characters.
Return type:: str

See also

write_result: The function that records the digest in each file the function writes.
read_result: The function that compares the recorded digest with a recomputed digest.

Examples

>>> import os
>>> import tempfile
>>> import numpy as np
>>> from pianistpy.io import data_digest, write_result
>>> with tempfile.TemporaryDirectory() as folder:
...     path = os.path.join(folder, "example.h5")
...     written = write_result(path, "example", "example-1", {}, {"x": np.ones(2)})
...     recomputed = data_digest(path)
>>> recomputed == written, len(recomputed)
(True, 64)

pianistpy.io.load_result(source)#

Return a StoredResult, given the StoredResult or the path of a result file.

A StoredResult passed as source is returned unchanged. A path is read and checked by read_result.

Parameters:: source (StoredResult or str or path-like) – A StoredResult, or the path of a file that holds one result.
Returns:: stored – source itself when source is a StoredResult; otherwise the StoredResult that read_result reads from the file.
Return type:: StoredResult
Raises:: ResultError – If source is a path and read_result raises ResultError: the file lacks an attribute of a result file, the major number of the file’s format version differs from the major number the package reads, or the recomputed digest differs from the recorded digest.

See also

read_result: The function that reads and checks a file.

Examples

>>> import os
>>> import tempfile
>>> import numpy as np
>>> from pianistpy.io import load_result, read_result, write_result
>>> with tempfile.TemporaryDirectory() as folder:
...     path = os.path.join(folder, "example.h5")
...     _ = write_result(path, "example", "example-1", {}, {"x": np.ones(2)})
...     stored = load_result(path)
>>> load_result(stored) is stored
True

pianistpy.io.read(path)#

Read a result from a file, whatever kind of result the file holds.

The call reads and checks the file with read_result and passes the StoredResult to the reader registered for the file’s kind (register_reader).

Parameters:

path (str or path-like) – A file written by the write method of a result.

Returns:

result – The distribution function, model, particles or other result the file holds, equal to the result that was written.

Return type:

object

Raises:

ResultError – If the file lacks one of the attributes format_version, result_kind, identity, data_sha256, record and parents.
ResultError – If the major number of the file’s format version differs from the major number of the format version the installed package reads.
ResultError – If the recomputed digest of the datasets differs from the digest the file records.
ResultError – If no reader is registered for the kind the file records.

See also

register_reader: The registration of the function that rebuilds a kind.
read_result: The read and the check of the file.

Examples

>>> import os
>>> import tempfile
>>> import numpy as np
>>> from pianistpy.io import read, register_reader, write_result
>>> register_reader("doc_example", lambda stored: float(stored.arrays["x"].sum()))
>>> with tempfile.TemporaryDirectory() as folder:
...     path = os.path.join(folder, "example.h5")
...     arrays = {"x": np.arange(3.0)}
...     _ = write_result(path, "doc_example", "example-1", {}, arrays)
...     result = read(path)
>>> result
3.0

pianistpy.io.read_result(path)#

Read one result from an HDF5 file and check the file’s digest.

The call recomputes the SHA-256 digest of the file’s datasets (data_digest) and compares the recomputed digest with the digest the file records.

Parameters:

path (str or path-like) – The file to read.

Returns:

stored – The arrays, record, identity string, digest, parents and provenance the file holds.

Return type:

StoredResult

Raises:

ResultError – If the file lacks one of the attributes format_version, result_kind, identity, data_sha256, record and parents.
ResultError – If the major number of the file’s format version differs from the major number of the format version the installed package reads.
ResultError – If the recomputed digest differs from the digest the file records.

See also

read: The function that rebuilds the result object from the file.
write_result: The function that writes the file.

Examples

>>> import os
>>> import tempfile
>>> import numpy as np
>>> from pianistpy.io import read_result, write_result
>>> with tempfile.TemporaryDirectory() as folder:
...     path = os.path.join(folder, "example.h5")
...     arrays = {"x": np.arange(3.0)}
...     _ = write_result(path, "example", "example-1", {"n": 3}, arrays)
...     stored = read_result(path)
>>> stored.record, stored.arrays["x"]
({'n': 3}, array([0., 1., 2.]))

pianistpy.io.register_reader(kind, reader)#

Make reader the function that rebuilds every result of one kind.

read calls the function registered for the kind a file records. A second registration for the same kind replaces the first registration.

Parameters:

kind (str) – The kind of result, as write_result stores the kind in the file.
reader (callable) – A function that takes the StoredResult and returns the result object.

Return type:

None

See also

read: The function that calls the registered reader.

Examples

>>> import os
>>> import tempfile
>>> import numpy as np
>>> from pianistpy.io import read, register_reader, write_result
>>> register_reader("doc_example", lambda stored: float(stored.arrays["x"].sum()))
>>> with tempfile.TemporaryDirectory() as folder:
...     path = os.path.join(folder, "example.h5")
...     arrays = {"x": np.arange(3.0)}
...     _ = write_result(path, "doc_example", "example-1", {}, arrays)
...     total = read(path)
>>> total
3.0

pianistpy.io.write_result(path, kind, identity, record, arrays, parents=(), overwrite=False)#

Write one result to an HDF5 file and check the file against the request.

After writing, the call reads the file back and compares the file’s arrays, bit for bit, and the file’s record, identity string, kind and parents with the values passed. The call stores the SHA-256 digest of the datasets (data_digest) in the file only when every comparison holds.

Parameters:

path (str or path-like) – The file to write.
kind (str) – The kind of result, the key under which register_reader finds the function that rebuilds the result.
identity (str) – The identity string of the result.
record (mapping) – The inputs and settings that determine the result, in dictionary form. Every value is one that json.dumps writes.
arrays (mapping of str to array-like) – The arrays of the result, by name. A name may contain / to place the array in an HDF5 group.
parents (sequence of str, optional) – The identity strings of the results from which the result was computed. Default: (), which means the result has no parent results.
overwrite (bool, optional) – Whether to replace an existing file at path. Default: False, which means an existing file is kept and the call raises InputError.

Returns:

data_sha256 – The SHA-256 digest of the datasets as written (data_digest), the same digest the file stores.

Return type:

str

Raises:

InputError – If a file exists at path and overwrite is false.
ResultError – If the file read back differs from the request in an array, the record, the identity string, the kind or the parents.

See also

read_result: The function that reads the file back.
data_digest: The digest of the datasets.

Examples

>>> import os
>>> import tempfile
>>> import numpy as np
>>> from pianistpy.io import write_result
>>> with tempfile.TemporaryDirectory() as folder:
...     path = os.path.join(folder, "example.h5")
...     digest = write_result(path, "example", "example-1", {}, {"x": np.ones(2)})
>>> len(digest)
64

pianistpy.io.write_table(path, columns, header=(), overwrite=False)#

Write columns, each with a column name, as a fixed-width text table.

The file opens with the lines of header, each prefixed with # ``, followed by the line ``# columns: and the column names in order. Each following row holds one value of every column, each value in a field of 25 characters: a float in exponent form with 14 digits after the decimal point, an integer as the integer. After writing, the call reads the file back with numpy.loadtxt and compares the values read with the values written, rounded to the 14 digits the file holds.

Parameters:

path (str or path-like) – The file to write.
columns (mapping of str to array-like) – The columns, in order. Each column is one-dimensional, of integer or float type, and of the length of every other column. Each column name is non-empty and holds no whitespace.
header (sequence of str, optional) – The lines at the head of the file, each without the leading #. Default: (), which means the head holds only the line of column names.
overwrite (bool, optional) – Whether to replace an existing file at path. Default: False, which means an existing file is kept and the call raises InputError.

Raises:

InputError – If a file exists at path and overwrite is false.
InputError – If columns is empty.
InputError – If a column name is empty or holds whitespace.
InputError – If a column is not one-dimensional.
InputError – If a column is not of integer or float type.
InputError – If two columns differ in length.
ResultError – If a value read back differs from the value written, rounded to 14 digits after the decimal point.

Return type:

None

See also

write_result: The HDF5 file that records a result.

Examples

>>> import os
>>> import tempfile
>>> import numpy as np
>>> from pianistpy.io import write_table
>>> with tempfile.TemporaryDirectory() as folder:
...     path = os.path.join(folder, "table.txt")
...     columns = {"r": np.array([1.0, 2.0]), "n": np.array([3, 4])}
...     write_table(path, columns, header=["radii and counts"])
...     with open(path) as file:
...         print(file.read(), end="")
# radii and counts
# columns: r n
1.00000000000000e+00     3
2.00000000000000e+00     4