Validation#

Module pianistpy.validation

The accuracy bands of a result and the validation report that states them.

Every result of the package (a model, a distribution function, a sampler, a projected table) carries a ValidationReport as its validation attribute. A user reads the band of a result from result.validation.tier and the measured relative errors from result.validation.checks. tier_level maps one relative error \(\epsilon\) to its band:

\[\begin{split}\text{band}(\epsilon) = \begin{cases} \text{silent}, & \epsilon \le 10^{-6},\\ \text{recorded}, & 10^{-6} < \epsilon \le 10^{-4},\\ \text{warned}, & 10^{-4} < \epsilon \le 10^{-2},\\ \text{refused}, & \epsilon > 10^{-2}. \end{cases}\end{split}\]

The technical notes, Section T:0.5, state how each band is reported.

class pianistpy.validation.Check(error, where=None, reason=None, source=None, excluded=None)#

One check of a result: the relative error measured and where.

A check is either judged or not judged. A judged check has a measured relative error and the band of that error. A check that is not judged has the error NaN, a stated reason, and the name "not judged" in place of a band; a check that is not judged does not change the band of the result.

Parameters:

error (float) – The relative error the check measured. NaN for a check that is not judged.
where (float, optional) – The position of the largest relative error, in the variable the check runs over (a radius or an energy, in internal units). Default: None, which means the check has no single position of its largest error.
reason (str, optional) – The reason the check is not judged. Default: None, which means the check is judged.
source (str, optional) – For a judged check that read its input from a source other than the result’s own data: the source read, and the reason. The error and the band are those measured on that source. Default: None, which means the check read the result’s own data.
excluded (str, optional) – For a judged check that leaves part of its input unjudged: the part left out, and the reason. The error and the band are those of the remaining input. Default: None, which means the check judged its whole input.

Variables:

level (int) – The band of the check’s error as returned by tier_level; 0 for a check that is not judged.
tier (str) – The name of the check’s band, one of TIER_NAMES, or "not judged".

See also

ValidationReport: The checks of one result together.
tier_level: The band of a relative error.

Examples

>>> check = Check(error=3e-5, where=12.5)
>>> check.tier
'recorded'
>>> Check(error=float("nan"), reason="the table has no outer rows").tier
'not judged'

property level: int#: The band of the check’s error as a level from 0 to 3; 0 for a check that is not judged.

property tier: str#: The name of the check’s band, or "not judged".

class pianistpy.validation.ValidationReport(keys, checks, parents_level=0)#

The validation report of one result: its checks and its band.

Each kind of result reports a fixed set of keys, one Check per key. The band of the result is the worst band among the result’s own checks and the worst band among the results the result was built from (its parents). A report holds only the result’s own checks: a derived result carries the worst level of its parents as one number, not the parents’ reports.

Parameters:

keys (tuple of str) – The keys the kind of result reports, in the order the report lists them.
checks (mapping of str to Check) – One Check for each entry of keys.
parents_level (int, optional) – The worst level, from 0 to 3, among the parents of the result. Default: 0, which means the result has no parent above \(10^{-6}\).

Variables:

level (int) – The worst level among the result’s own checks and parents_level.
tier (str) – The name of the worst band, TIER_NAMES[level].

Raises:

InputError – If the keys of checks are not the entries of keys.

See also

Check: One check of the report.
tier_level: The band of a relative error.

Examples

>>> report = ValidationReport(
...     keys=("round_trip", "top_form"),
...     checks={"round_trip": Check(2e-7), "top_form": Check(3e-5)},
... )
>>> report.tier
'recorded'
>>> ValidationReport(report.keys, report.checks, parents_level=2).tier
'warned'

property level: int#: The worst level among the result’s own checks and its parents.

property tier: str#: The name of the worst band of the result.

classmethod from_dict(record, key_sets=None)#

Build a report from the dictionary that to_dict writes.

Parameters:

record (mapping) – The report in the form to_dict returns: the entries "parents_tier" and "checks".
key_sets (sequence of tuple of str, optional) – Every key set that a report of the record’s kind of result can have. With key_sets given, the report takes the smallest key set that holds every check of the record, and a key of that set that the record lacks becomes a check that is not judged, with the reason that the file does not hold the check. Default: None, which means the report takes the record’s own checks as its keys.

Returns:

report – The report the record describes.

Return type:

ValidationReport

Raises:

ResultError – If key_sets is given and a check of the record belongs to no key set of key_sets.
ResultError – If key_sets is given and no single key set of key_sets holds every check of the record.

See also

to_dict: The dictionary form of a report.

Examples

>>> report = ValidationReport(("a",), {"a": Check(3e-5)})
>>> record = report.to_dict()
>>> ValidationReport.from_dict(record) == report
True
>>> wider = ValidationReport.from_dict(record, [("a", "b")])
>>> wider.checks["b"].tier
'not judged'

to_dict()#

Return the report as a plain dictionary.

Returns:: record – The entries "tier" (the report’s band), "parents_tier" (the name of the parents’ worst band) and "checks" (one dictionary for each key, in the order of keys, with the entries "error", "where" and "tier", and "reason", "source" or "excluded" where the check states one).
Return type:: dict

See also

from_dict: The report built from the dictionary.

Examples

>>> report = ValidationReport(("a",), {"a": Check(3e-5, where=1.0)})
>>> record = report.to_dict()
>>> record["tier"], record["parents_tier"]
('recorded', 'silent')
>>> record["checks"]["a"]
{'error': 3e-05, 'where': 1.0, 'tier': 'recorded'}

pianistpy.validation.announce(what, report, detail)#

Report a result whose relative error is above \(10^{-6}\).

The announcement depends on the band of report. A result in the band "silent" (at most \(10^{-6}\)) is not announced. A result in the band "recorded" (above \(10^{-6}\), at most \(10^{-4}\)) is written as one line to the pianistpy logger at level INFO, and no warning is issued. A result in the band "warned" (above \(10^{-4}\), at most \(10^{-2}\)) is written to the pianistpy logger at level WARNING and issued as an AccuracyWarning, which the standard warnings filter can silence or turn into an error. A result in the band "refused" (above \(10^{-2}\)) reaches announce only if its settings set write_anyway=True; that result is announced as a result in the band "warned" is.

Parameters:

what (str) – The result’s name as messages write it, for example "The sampler of 'nfw'".
report (ValidationReport) – The validation report of the result.
detail (str) – The checks above \(10^{-6}\), each with its relative error and the position of its largest error.

Warns:

AccuracyWarning – If the band of report is "warned" or "refused".

Return type:

None

See also

refuse_unless_asked: The error for an input in the band "refused".
set_verbosity: The lowest level of the messages printed on the screen.

Examples

>>> silent = ValidationReport(("a",), {"a": Check(1e-8)})
>>> announce("The sampler of 'x'", silent, "a 1.0e-08")

pianistpy.validation.refuse_unless_asked(what, report, settings)#

Raise if an input result is in the band "refused" and the settings do not ask for the result anyway.

A result in the band "refused" (relative error above \(10^{-2}\)) exists only if the settings that built the result set write_anyway=True, or if the result was read from a file that holds such a result. A computation that takes such a result as input proceeds only if its own settings also set write_anyway=True; the new result then carries the band "refused".

Parameters:

what (str) – The input result’s name as messages write it, for example "The model 'nfw'".
report (ValidationReport) – The validation report of the input result.
settings (Settings) – The settings of the computation that takes the input result.

Raises:

AccuracyError – If the band of report is "refused" and settings does not set write_anyway=True.

Return type:

None

See also

announce: The message for a result above \(10^{-6}\).
Settings: The setting write_anyway.

Examples

>>> from pianistpy import Settings
>>> report = ValidationReport(("a",), {"a": Check(0.5)})
>>> report.tier
'refused'
>>> refuse_unless_asked("The model 'x'", report, Settings(write_anyway=True))
>>> refuse_unless_asked("The model 'x'", report, Settings())
Traceback (most recent call last):
...
pianistpy.exceptions.AccuracyError: The model 'x' is beyond 0.01 (tier ...

pianistpy.validation.tier_level(error)#

Return the band of a relative error as a level from 0 to 3.

The level indexes TIER_NAMES: level 0 is "silent", level 1 is "recorded", level 2 is "warned" and level 3 is "refused".

Parameters:: error (float) – Relative error \(\epsilon\), a non-negative number. A NaN error has level 3, the level of an error above \(10^{-2}\).
Returns:: level – 0 if error is at most \(10^{-6}\); 1 if error is above \(10^{-6}\) and at most \(10^{-4}\); 2 if error is above \(10^{-4}\) and at most \(10^{-2}\); 3 if error is above \(10^{-2}\) or is NaN.
Return type:: int

See also

TIER_NAMES: The name of each level.
TIER_BOUNDS: The three relative errors that separate the levels.
Check: One measured relative error and its level.

Examples

>>> [tier_level(e) for e in (1e-7, 1e-5, 1e-3, 1e-1, float("nan"))]
[0, 1, 2, 3, 3]
>>> TIER_NAMES[tier_level(3e-5)]
'recorded'