Skip to content

medcat.utils.regression.results

Classes:

Attributes:

STRICTNESS_MATRIX module-attribute 

STRICTNESS_MATRIX: dict[Strictness, set[Finding]] = {STRICTEST: {IDENTICAL}, STRICT: {IDENTICAL, FOUND_ANY_CHILD}, NORMAL: {IDENTICAL, FOUND_ANY_CHILD, FOUND_CHILD_PARTIAL, BIGGER_SPAN_RIGHT, BIGGER_SPAN_LEFT, BIGGER_SPAN_BOTH, SMALLER_SPAN, PARTIAL_OVERLAP}, LENIENT: {IDENTICAL, FOUND_ANY_CHILD, BIGGER_SPAN_RIGHT, BIGGER_SPAN_LEFT, BIGGER_SPAN_BOTH, SMALLER_SPAN, PARTIAL_OVERLAP, FOUND_DIR_PARENT, FOUND_DIR_GRANDPARENT}, ANYTHING: set(Finding)}

Finding

Bases: Enum

Describes whether or how the finding verified.

The idea is that we know where we expect the entity to be recognised and the enum constants describe how the recognition compared to the expectation.

In essence, we want to know the relative positions of the two pairs of numbers (character numbers): - Expected Start, Expected End - Recognised Start, Recognised End

We can model this as 4 numbers on the number line. And we want to know their position relative to each other. For example, if the expected positions are marked with * and recognised positions with #, we may have something like: __#_#_____ Which would indicate that there is a partial, but smaller span recognised.

Methods:

  • determine –

    Determine the finding type based on the input

  • has_correct_cui –

    Whether the finding found the correct concept.

Attributes:

  • BIGGER_SPAN_BOTH –

    The CUI is the same, but the recognised span is longer on both sides.

  • BIGGER_SPAN_LEFT –

    The CUI is the same, but the recognised span is longer on the left.

  • BIGGER_SPAN_RIGHT –

    The CUI is the same, but the recognised span is longer on the right.

  • FAIL –

    The concept was not recognised in any meaningful way.

  • FOUND_ANY_CHILD –

    The recognised CUI is a child of the expected CUI but the span is an exact match.

  • FOUND_CHILD_PARTIAL –

    The recognised CUI is a child yet the match is only partial (smaller/bigger/partial).

  • FOUND_DIR_GRANDPARENT –

    The recognised CUI is a grandparent of the expected CUI but the span is an exact match.

  • FOUND_DIR_PARENT –

    The recognised CUI is a parent of the expected CUI but the span is an exact match.

  • FOUND_OTHER –

    Found another CUI in the same span.

  • IDENTICAL –

    The CUI and the span recognised are identical to what was expected.

  • PARTIAL_OVERLAP –

    The CUI is the same, but the span overlaps partially.

  • SMALLER_SPAN –

    The CUI is the same, but the recognised span is smaller.

BIGGER_SPAN_BOTH class-attribute instance-attribute 

BIGGER_SPAN_BOTH = auto()

The CUI is the same, but the recognised span is longer on both sides.

If we use the notation from the class doc string, e.g: #______#

BIGGER_SPAN_LEFT class-attribute instance-attribute 

BIGGER_SPAN_LEFT = auto()

The CUI is the same, but the recognised span is longer on the left.

If we use the notation from the class doc string, e.g: #__#_

BIGGER_SPAN_RIGHT class-attribute instance-attribute 

BIGGER_SPAN_RIGHT = auto()

The CUI is the same, but the recognised span is longer on the right.

If we use the notation from the class doc string, e.g: _#____#

FAIL class-attribute instance-attribute 

FAIL = auto()

The concept was not recognised in any meaningful way.

FOUND_ANY_CHILD class-attribute instance-attribute 

FOUND_ANY_CHILD = auto()

The recognised CUI is a child of the expected CUI but the span is an exact match.

FOUND_CHILD_PARTIAL class-attribute instance-attribute 

FOUND_CHILD_PARTIAL = auto()

The recognised CUI is a child yet the match is only partial (smaller/bigger/partial).

FOUND_DIR_GRANDPARENT class-attribute instance-attribute 

FOUND_DIR_GRANDPARENT = auto()

The recognised CUI is a grandparent of the expected CUI but the span is an exact match.

FOUND_DIR_PARENT class-attribute instance-attribute 

FOUND_DIR_PARENT = auto()

The recognised CUI is a parent of the expected CUI but the span is an exact match.

FOUND_OTHER class-attribute instance-attribute 

FOUND_OTHER = auto()

Found another CUI in the same span.

IDENTICAL class-attribute instance-attribute 

IDENTICAL = auto()

The CUI and the span recognised are identical to what was expected.

PARTIAL_OVERLAP class-attribute instance-attribute 

PARTIAL_OVERLAP = auto()

The CUI is the same, but the span overlaps partially.

If we use the notation from the class doc string, e.g: _#__# (starts between expected start and end, but ends beyond) #_#__ (start before expected start, but ends between expected start and end)

SMALLER_SPAN class-attribute instance-attribute 

SMALLER_SPAN = auto()

The CUI is the same, but the recognised span is smaller.

If we use the notation from the class doc string, e.g: ##_ (neither start nor end match) ##__ (start matches, but end is before expected) __#_#_ (end matches, but start is after expected)

determine classmethod 

determine(exp_cui: str, exp_start: int, exp_end: int, tl: TranslationLayer, found_entities: dict[int, Entity], strict_only: bool = False, check_children: bool = True, check_parent: bool = True, check_grandparent: bool = True) -> tuple[Finding, Optional[str]]

Determine the finding type based on the input

Parameters:

  • exp_cui

    (str) –

    Expected CUI.

  • exp_start

    (int) –

    Expected span start.

  • exp_end

    (int) –

    Expected span end.

  • tl

    (TranslationLayer) –

    The translation layer.

  • found_entities

    (dict[int, Entity]) –

    The entities found by the model.

  • strict_only

    (bool, default: False ) –

    Whether to use a strict-only mode (either identical or fail). Defaults to False.

  • check_children

    (bool, default: True ) –

    Whether to check the children. Defaults to True.

  • check_parent

    (bool, default: True ) –

    Whether to check for parent(s). Defaults to True.

  • check_grandparent

    (bool, default: True ) –

    Whether to check for grandparent(s). Defaults to True.

Returns:

  • tuple[Finding, Optional[str]] –

    tuple['Finding', Optional[str]]: The type of finding determined, and the alternative.

Source code in medcat-v2/medcat/utils/regression/results.py 
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
@classmethod
def determine(cls, exp_cui: str, exp_start: int, exp_end: int,
              tl: TranslationLayer,
              found_entities: dict[int, Entity],
              strict_only: bool = False,
              check_children: bool = True, check_parent: bool = True,
              check_grandparent: bool = True
              ) -> tuple['Finding', Optional[str]]:
    """Determine the finding type based on the input

    Args:
        exp_cui (str): Expected CUI.
        exp_start (int): Expected span start.
        exp_end (int): Expected span end.
        tl (TranslationLayer): The translation layer.
        found_entities (dict[int, Entity]):
            The entities found by the model.
        strict_only (bool): Whether to use a strict-only mode
            (either identical or fail). Defaults to False.
        check_children (bool): Whether to check the children.
            Defaults to True.
        check_parent (bool): Whether to check for parent(s).
            Defaults to True.
        check_grandparent (bool): Whether to check for grandparent(s).
            Defaults to True.

    Returns:
        tuple['Finding', Optional[str]]:
            The type of finding determined, and the alternative.
    """
    return FindingDeterminer(exp_cui, exp_start, exp_end,
                             tl, found_entities, strict_only,
                             check_children, check_parent,
                             check_grandparent).determine()

has_correct_cui 

has_correct_cui() -> bool

Whether the finding found the correct concept.

Returns:

  • bool ( bool ) –

    Whether the correct concept was found.

Source code in medcat-v2/medcat/utils/regression/results.py 
76
77
78
79
80
81
82
83
84
85
86
87
def has_correct_cui(self) -> bool:
    """Whether the finding found the correct concept.

    Returns:
        bool: Whether the correct concept was found.
    """
    return self in (
        Finding.IDENTICAL, Finding.BIGGER_SPAN_RIGHT,
        Finding.BIGGER_SPAN_LEFT,
        Finding.BIGGER_SPAN_BOTH, Finding.SMALLER_SPAN,
        Finding.PARTIAL_OVERLAP
    )

FindingDeterminer 

FindingDeterminer(exp_cui: str, exp_start: int, exp_end: int, tl: TranslationLayer, found_entities: dict[int, Entity], strict_only: bool = False, check_children: bool = True, check_parent: bool = True, check_grandparent: bool = True)

A helper class to determine the type of finding.

This is mostly useful to split the responsibilities of looking at children/parents as well as to keep track of the already-checked children to avoid infinite recursion (which could happen in - e.g - a SNOMED model).

Parameters:

  • exp_cui

    (str) –

    The expected CUI.

  • exp_start

    (int) –

    The expected span start.

  • exp_end

    (int) –

    The expected span end.

  • tl

    (TranslationLayer) –

    The translation layer.

  • found_entities

    (dict[str, Entity]) –

    The entities found by the model.

  • strict_only

    (bool, default: False ) –

    Whether to use strict-only mode (either identical or fail). Defaults to False.

  • check_children

    (bool, default: True ) –

    Whether or not to check the children. Defaults to True.

  • check_parent

    (bool, default: True ) –

    Whether to check for parent(s). Defaults to True.

  • check_grandparent

    (bool, default: True ) –

    Whether to check for granparent(s). Defaults to True.

Methods:

  • determine –

    Determine the finding based on the given information.

Attributes:

Source code in medcat-v2/medcat/utils/regression/results.py 
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
def __init__(self, exp_cui: str,
             exp_start: int,
             exp_end: int,
             tl: TranslationLayer,
             found_entities: dict[int, Entity],
             strict_only: bool = False,
             check_children: bool = True,
             check_parent: bool = True,
             check_grandparent: bool = True,) -> None:
    self.exp_cui = exp_cui
    self.exp_start = exp_start
    self.exp_end = exp_end
    self.tl = tl
    self.found_entities = found_entities
    self.strict_only = strict_only
    self.check_children = check_children
    self.check_parent = check_parent
    self.check_grandparent = check_grandparent
    # helper for children to avoid infinite recursion
    self._checked_children: set[str] = set()

check_children instance-attribute 

check_children = check_children

check_grandparent instance-attribute 

check_grandparent = check_grandparent

check_parent instance-attribute 

check_parent = check_parent

exp_cui instance-attribute 

exp_cui = exp_cui

exp_end instance-attribute 

exp_end = exp_end

exp_start instance-attribute 

exp_start = exp_start

found_entities instance-attribute 

found_entities = found_entities

strict_only instance-attribute 

strict_only = strict_only

tl instance-attribute 

tl = tl

determine 

determine() -> tuple[Finding, Optional[str]]

Determine the finding based on the given information.

First, the strict check is done (either identical or not). Then, parents are checked (if required). After that, children are checked (if required).

Returns:

  • tuple[Finding, Optional[str]] –

    tuple[Finding, Optional[str]]: The appropriate finding, and the alternative (if applicable).

Source code in medcat-v2/medcat/utils/regression/results.py 
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
def determine(self) -> tuple[Finding, Optional[str]]:
    """Determine the finding based on the given information.

    First, the strict check is done (either identical or not).
    Then, parents are checked (if required).
    After that, children are checked (if required).

    Returns:
        tuple[Finding, Optional[str]]: The appropriate finding,
            and the alternative (if applicable).
    """
    finding, cui = self._determine()
    # NOTE: the point of this wrapper method is to add the preferred name
    #       to the CUI in one place and one place only
    return finding, self._descr_cui(cui)

MalformedFinding 

MalformedFinding(*args: object)

Bases: ValueError

Source code in medcat-v2/medcat/utils/regression/results.py 
775
776
def __init__(self, *args: object) -> None:
    super().__init__(*args)

MultiDescriptor

Bases: BaseModel

The descriptor of results over multiple different results (parts).

The idea is that this would likely be used with a regression suite and it would incorporate all the different regression cases it describes.

Methods:

Attributes:

findings property 

findings: dict[Finding, int]

The total findings.

Returns:

  • dict[Finding, int] –

    dict[Finding, int]: The total number of successes.

name instance-attribute 

name: str

The name of the collection being checked

parts class-attribute instance-attribute 

parts: list[ResultDescriptor] = []

The parts kept track of

calculate_report 

calculate_report(phrases_separately: bool = False, hide_empty: bool = False, examples_strictness: Optional[Strictness] = STRICTEST, strictness: Strictness = NORMAL, phrase_max_len: int = 80) -> tuple[int, int, int, str, int]

Calculate some of the major parts of the report.

Parameters:

  • phrases_separately

    (bool, default: False ) –

    Whether to include per-phrase information

  • hide_empty

    (bool, default: False ) –

    Whether to hide empty cases

  • examples_strictness

    (Optional[STRICTEST], default: STRICTEST ) –

    What level of strictness to show for examples. Set to None to disable examples. Defaults to Strictness.STRICTEST.

  • strictness

    (Strictness, default: NORMAL ) –

    The strictness of the success / fail overview. Defaults to Strictness.NORMAL.

  • phrase_max_len

    (int, default: 80 ) –

    The maximum length of the phrase in examples. Defaults to 80.

Returns:

  • tuple[int, int, int, str, int] –

    tuple[int, int, int, int, str]: The total number of examples, the total successes, the total failures, the delegated part, and the number of empty

Source code in medcat-v2/medcat/utils/regression/results.py 
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
def calculate_report(
        self, phrases_separately: bool = False,
        hide_empty: bool = False,
        examples_strictness: Optional[Strictness] = Strictness.STRICTEST,
        strictness: Strictness = Strictness.NORMAL,
        phrase_max_len: int = 80) -> tuple[int, int, int, str, int]:
    """Calculate some of the major parts of the report.

    Args:
        phrases_separately (bool): Whether to include
            per-phrase information
        hide_empty (bool): Whether to hide empty cases
        examples_strictness (Optional[Strictness.STRICTEST]):
            What level of strictness to show for examples.
            Set to None to disable examples.
            Defaults to Strictness.STRICTEST.
        strictness (Strictness): The strictness of the
            success / fail overview. Defaults to Strictness.NORMAL.
        phrase_max_len (int): The maximum length of the phrase in examples.
            Defaults to 80.

    Returns:
        tuple[int, int, int, int, str]: The total number of examples,
            the total successes, the total failures,
            the delegated part, and the number of empty
    """
    del_out = []  # delegation
    total_findings: dict[Finding, int] = {}
    total_s, total_f = 0, 0
    allowed_findings = STRICTNESS_MATRIX[strictness]
    total_total = 0
    nr_of_empty = 0
    for part in self.parts:
        (cur_add, total_total_add,
         total_s_add, total_f_add) = self._get_part_report(
             part, allowed_findings, total_findings, hide_empty,
             # NOTE: using STRICTEST strictness for examples means
             #       that all but IDENTICAL examples will be shown
             examples_strictness, phrases_separately, phrase_max_len)
        if hide_empty and total_total_add == 0:
            nr_of_empty += 1
        else:
            total_total += total_total_add
            total_s += total_s_add
            total_f += total_f_add
            del_out.append(cur_add)
    delegated = '\n'.join(del_out)
    return total_total, total_s, total_f, delegated, nr_of_empty

get_report 

get_report(phrases_separately: bool, hide_empty: bool = False, examples_strictness: Optional[Strictness] = STRICTEST, strictness: Strictness = NORMAL, phrase_max_len: int = 80) -> str

Get the report associated with this descriptor

Parameters:

  • phrases_separately

    (bool) –

    Whether to include per-phrase information

  • hide_empty

    (bool, default: False ) –

    Whether to hide empty cases

  • examples_strictness

    (Optional[STRICTEST], default: STRICTEST ) –

    What level of strictness to show for examples. Set to None to disable examples. Defaults to Strictness.STRICTEST.

  • strictness

    (Strictness, default: NORMAL ) –

    The strictness of the success / fail overview. Defaults to Strictness.NORMAL.

  • phrase_max_len

    (int, default: 80 ) –

    The maximum length of the phrase in examples. Defaults to 80.

Returns:

  • str ( str ) –

    The report string

Source code in medcat-v2/medcat/utils/regression/results.py 
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
def get_report(
        self, phrases_separately: bool,
        hide_empty: bool = False,
        examples_strictness: Optional[Strictness] = Strictness.STRICTEST,
        strictness: Strictness = Strictness.NORMAL,
        phrase_max_len: int = 80) -> str:
    """Get the report associated with this descriptor

    Args:
        phrases_separately (bool):
            Whether to include per-phrase information
        hide_empty (bool): Whether to hide empty cases
        examples_strictness (Optional[Strictness.STRICTEST]):
            What level of strictness to show for examples.
            Set to None to disable examples.
            Defaults to Strictness.STRICTEST.
        strictness (Strictness): The strictness of the
            success / fail overview.
            Defaults to Strictness.NORMAL.
        phrase_max_len (int): The maximum length of the phrase in examples.
            Defaults to 80.

    Returns:
        str: The report string
    """
    (total_total, total_s, total_f,
     delegated, nr_of_empty) = self.calculate_report(
         phrases_separately=phrases_separately,
         hide_empty=hide_empty,
         examples_strictness=examples_strictness,
         strictness=strictness,
         phrase_max_len=phrase_max_len)
    empty_text = ''
    allowed_findings = STRICTNESS_MATRIX[strictness]
    if hide_empty:
        empty_text = (
            f' A total of {nr_of_empty} cases '
            'did not match any CUIs and/or names.')
    ret_vals = [
        f'A total of {len(self.parts)} parts were kept track of within '
        f'the group "{self.name}".\n'
        f'                    And a total of {total_total} '
        f'(sub)cases were checked.{empty_text}']
    allowed_fingings_str = sorted([f.name for f in allowed_findings])
    ret_vals.extend([
        f"At the strictness level of {strictness} "
        f"(allowing {allowed_fingings_str}):",
        f"The number of total successful (sub) cases: {total_s} "
        f"({100 * total_s / total_total if total_total > 0 else 0:5.2f}%)",
        f"The number of total failing (sub) cases   : {total_f} "
        f"({100 * total_f / total_total if total_total > 0 else 0:5.2f}%)"
    ])
    ret_vals.extend([
        f"{f.name:24s}:{self.findings[f]:10d} "
        f"({ft100 / total_total if total_total > 0 else 0:5.2f}%)"
        # NOTE iterating over Finding so the order is the same
        #      as in the enum
        for f in Finding if f in self.findings
        # NOTE: this is just for the line length, really
        and (ft100 := 100 * self.findings[f]) >= 0
    ])
    return "\n".join(ret_vals) + f"\n{delegated}"

iter_examples 

iter_examples(strictness_threshold: Strictness) -> Iterable[tuple[FinalTarget, tuple[Finding, Optional[str]]]]

Iterate over all relevant examples.

Only examples that are not in the strictness matrix for the specified threshold will be used.

Parameters:

  • strictness_threshold

    (Strictness) –

    The threshold of avoidance.

Yields:

Source code in medcat-v2/medcat/utils/regression/results.py 
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
def iter_examples(self, strictness_threshold: Strictness
                  ) -> Iterable[
                      tuple[FinalTarget, tuple[Finding, Optional[str]]]]:
    """Iterate over all relevant examples.

    Only examples that are not in the strictness matrix for the specified
    threshold will be used.

    Args:
        strictness_threshold (Strictness): The threshold of avoidance.

    Yields:
        Iterable[tuple[FinalTarget, tuple[Finding, Optional[str]]]]:
            The examples
    """
    for descr in self.parts:
        yield from descr.iter_examples(
            strictness_threshold=strictness_threshold)

model_dump 

model_dump(**kwargs) -> dict
Source code in medcat-v2/medcat/utils/regression/results.py 
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
def model_dump(self, **kwargs) -> dict:
    if 'strictness' in kwargs:
        strict_raw = kwargs.pop('strictness')
        if isinstance(strict_raw, Strictness):
            strictness = strict_raw
        elif isinstance(strict_raw, str):
            strictness = Strictness[strict_raw]
        else:
            raise ValueError(f"Unknown stircntess specified: {strict_raw}")
    else:
        strictness = Strictness.NORMAL
    out_dict = cast(pydantic.BaseModel, super()).model_dump(
        exclude={'parts'}, **kwargs)
    out_dict['parts'] = [part.model_dump(strictness=strictness)
                         for part in self.parts]
    return out_dict

ResultDescriptor

Bases: SingleResultDescriptor

The overarching result descriptor that handles multiple phrases.

This class keeps track of the results on a per-phrase basis and can be used to get the overall report and/or iterate over examples.

Methods:

Attributes:

per_phrase_results class-attribute instance-attribute 

per_phrase_results: dict[str, SingleResultDescriptor] = {}

get_report 

get_report(phrases_separately: bool = False) -> str

Get the report associated with this descriptor

Parameters:

  • phrases_separately

    (bool, default: False ) –

    Whether to output descriptor for each phrase separately

Returns:

  • str ( str ) –

    The report string

Source code in medcat-v2/medcat/utils/regression/results.py 
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
def get_report(self, phrases_separately: bool = False) -> str:
    """Get the report associated with this descriptor

    Args:
        phrases_separately (bool): Whether to output descriptor
            for each phrase separately

    Returns:
        str: The report string
    """
    sr = super().get_report()
    if not phrases_separately:
        return sr
    children = '\n'.join([srd.get_report()
                         for srd in self.per_phrase_results.values()])
    return sr + '\n\t\t' + children.replace('\n', '\n\t\t')

iter_examples 

iter_examples(strictness_threshold: Strictness) -> Iterable[tuple[FinalTarget, tuple[Finding, Optional[str]]]]

Iterate suitable examples.

The strictness threshold at which to include examples.

Any finding that is assumed to be "correct enough" according to the strictness matrix for this threshold will be withheld from examples.

In simpler terms, if the finding is NOT in the strictness matrix for this strictness, the example is recorded.

To disable example keeping, set the threshold to

Strictness.ANYTHING.

Parameters:

  • strictness_threshold

    (Strictness) –

    The strictness threshold.

Yields:

Source code in medcat-v2/medcat/utils/regression/results.py 
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
def iter_examples(self, strictness_threshold: Strictness
                  ) -> Iterable[tuple[FinalTarget,
                                      tuple[Finding, Optional[str]]]]:
    """Iterate suitable examples.

    The strictness threshold at which to include examples.

    Any finding that is assumed to be "correct enough" according to
    the strictness matrix for this threshold will be withheld from
    examples.

    In simpler terms, if the finding is NOT in the strictness matrix
    for this strictness, the example is recorded.

    NOTE: To disable example keeping, set the threshold to
          Strictness.ANYTHING.

    Args:
        strictness_threshold (Strictness): The strictness threshold.

    Yields:
        Iterable[tuple[FinalTarget, tuple[Finding, Optional[str]]]]:
            The placeholder, phrase, finding, CUI, and name.
    """
    phrases = sorted(self.per_phrase_results.keys())
    for phrase in phrases:
        srd = self.per_phrase_results[phrase]
        # sort by finding 1st, found CUI 2nd, and used name 3rd
        sorted_examples = sorted(
            srd.examples, key=lambda tf: (
                tf[1][0].name, str(tf[1][1]), tf[0].name))
        for target, finding in sorted_examples:
            if finding[0] not in STRICTNESS_MATRIX[strictness_threshold]:
                yield target, finding

model_dump 

model_dump(**kwargs) -> dict
Source code in medcat-v2/medcat/utils/regression/results.py 
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
def model_dump(self, **kwargs) -> dict:
    if 'exclude' in kwargs and kwargs['exclude'] is not None:
        exclude: set = kwargs['exclude']
    else:
        exclude = set()
        kwargs['exclude'] = exclude
    # NOTE: ignoring here so that examples are only present
    #       in the per phrase part
    exclude.update(('examples', 'per_phrase_results'))
    d = cast(pydantic.BaseModel, super()).model_dump(**kwargs)
    if 'examples' in d:
        # NOTE: I don't really know why, but the examples still
        #       seem to be a part of the resulting dict, so I need
        #       to explicitly remove them
        del d['examples']
    # NOTE: need to propagate here manually so the strictness keyword
    #       makes sense and doesn't cause issues due being to
    #       unexpected keyword
    per_phrase_results = {
        phrase: res.model_dump(**kwargs) for phrase, res in
        sorted(self.per_phrase_results.items(), key=lambda it: it[0])
    }
    d['per_phrase_results'] = per_phrase_results
    return d

report 

report(target: FinalTarget, finding: tuple[Finding, Optional[str]]) -> None

Report a test case and its successfulness

Parameters:

Source code in medcat-v2/medcat/utils/regression/results.py 
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
def report(self, target: FinalTarget,
           finding: tuple[Finding, Optional[str]]) -> None:
    """Report a test case and its successfulness

    Args:
        target (FinalTarget): The final targe configuration
        finding (tuple[Finding, Optional[str]]):
            To what extent the concept was recognised
    """
    phrase = target.final_phrase
    super().report_success(target, finding)
    if phrase not in self.per_phrase_results:
        self.per_phrase_results[phrase] = SingleResultDescriptor(
            name=phrase)
    self.per_phrase_results[phrase].report_success(target, finding)

SingleResultDescriptor

Bases: BaseModel

The result descriptor.

This class is responsible for keeping track of all the findings (i.e how many were found to be identical) as well as the examples of the finding on a per-target basis for further analysis.

Methods:

Attributes:

examples class-attribute instance-attribute 

examples: list[tuple[FinalTarget, tuple[Finding, Optional[str]]]] = []

The examples of non-perfect alignment.

findings class-attribute instance-attribute 

findings: dict[Finding, int] = {}

The description of failures

name instance-attribute 

name: str

The name of the part that was checked

get_report 

get_report() -> str

Get the report associated with this descriptor

Returns:

  • str ( str ) –

    The report string

Source code in medcat-v2/medcat/utils/regression/results.py 
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
def get_report(self) -> str:
    """Get the report associated with this descriptor

    Returns:
        str: The report string
    """
    total = sum(self.findings.values())
    ret_vals = [f"Tested '{self.name}' for a total of {total} cases:"]
    ret_vals.extend([
        f"{f.name:24s}:{self.findings[f]:10d} "
        f"({100 * self.findings[f] / total if total > 0 else 0:5.2f}%)"
        # NOTE iterating over Finding so the order is the same
        #      as in the enum
        for f in Finding if f in self.findings
    ])
    return "\n".join(ret_vals)

json 

json(**kwargs) -> str
Source code in medcat-v2/medcat/utils/regression/results.py 
442
443
444
def json(self, **kwargs) -> str:
    d = self.model_dump(**kwargs)
    return json.dumps(d)

model_dump 

model_dump(**kwargs) -> dict
Source code in medcat-v2/medcat/utils/regression/results.py 
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
def model_dump(self, **kwargs) -> dict:
    if 'strictness' in kwargs:
        kwargs = kwargs.copy()  # so if used elsewhere, keeps the kwarg
        strict_raw = kwargs.pop('strictness')
        if isinstance(strict_raw, Strictness):
            strictness = strict_raw
        elif isinstance(strict_raw, str):
            strictness = Strictness[strict_raw]
        else:
            raise ValueError(f"Unknown stircntess specified: {strict_raw}")
    else:
        strictness = Strictness.NORMAL
    # avoid serialising multiple times
    if 'exclude' in kwargs and kwargs['exclude'] is not None:
        exclude: set = kwargs['exclude']
    else:
        exclude = set()
        kwargs['exclude'] = exclude
    exclude.update(('findings', 'examples'))
    serialized_dict = {
        key.name: value for key, value in self.findings.items()
    }
    serialized_examples = [
        (ft.model_dump(**kwargs), (f[0].name, f[1]))
        for ft, f in self.examples
        # only count if NOT in strictness matrix (i.e 'failures')
        if f[0] not in STRICTNESS_MATRIX[strictness]
    ]
    model_dict = cast(pydantic.BaseModel, super()).model_dump(**kwargs)
    model_dict['findings'] = serialized_dict
    model_dict['examples'] = serialized_examples
    return model_dict

report_success 

report_success(target: FinalTarget, found: tuple[Finding, Optional[str]]) -> None

Report a test case and its successfulness.

Parameters:

Source code in medcat-v2/medcat/utils/regression/results.py 
377
378
379
380
381
382
383
384
385
386
387
388
389
390
def report_success(self, target: FinalTarget,
                   found: tuple[Finding, Optional[str]]) -> None:
    """Report a test case and its successfulness.

    Args:
        target (FinalTarget): The target configuration
        found (tuple[Finding, Optional[str]]):
            Whether or not the check was successful
    """
    finding, _ = found
    if finding not in self.findings:
        self.findings[finding] = 0
    self.findings[finding] += 1
    self.examples.append((target, found))

Strictness

Bases: Enum

The total strictness on which to judge the results.

Attributes:

  • ANYTHING –

    Anything stricness allows ANY finding.

  • LENIENT –

    Lenient stictness also allows parents and grandparents.

  • NORMAL –

    Normal strictness also allows partial overlaps on target concept and children.

  • STRICT –

    A strict option which allows identical or children.

  • STRICTEST –

    The strictest option which only allows identical findings.

ANYTHING class-attribute instance-attribute 

ANYTHING = auto()

Anything stricness allows ANY finding.

This would generally only be relevant when disabling examples for results descriptors.

LENIENT class-attribute instance-attribute 

LENIENT = auto()

Lenient stictness also allows parents and grandparents.

NORMAL class-attribute instance-attribute 

NORMAL = auto()

Normal strictness also allows partial overlaps on target concept and children.

STRICT class-attribute instance-attribute 

STRICT = auto()

A strict option which allows identical or children.

STRICTEST class-attribute instance-attribute 

STRICTEST = auto()

The strictest option which only allows identical findings.