medcat.components.addons.relation_extraction.modernbert.tokenizer

Classes:

• TokenizerWrapperModernBERT –

  Wrapper around a huggingface ModernBERT tokenizer so that it works

Attributes:

• logger –

logger module-attribute

logger = getLogger(__name__)

TokenizerWrapperModernBERT

TokenizerWrapperModernBERT(hf_tokenizers=None, max_seq_length: Optional[int] = None, add_special_tokens: Optional[bool] = False)

Bases: BaseTokenizerWrapper

Wrapper around a huggingface ModernBERT tokenizer so that it works with the RelCAT models.

Parameters:

• hf_tokenizers

  (`transformers.PreTrainedTokenizerFast`, default: None ) –

  A huggingface Fast tokenizer.

Methods:

• load –

Attributes:

• name –
• pretrained_model_name_or_path –

Source code in medcat-v2/medcat/components/addons/relation_extraction/tokenizer.py

def __init__(self, hf_tokenizers=None,
             max_seq_length: Optional[int] = None,
             add_special_tokens: Optional[bool] = False):
    self.hf_tokenizers = hf_tokenizers
    self.max_seq_length = max_seq_length
    self._add_special_tokens = add_special_tokens

name class-attribute instance-attribute

name = 'tokenizer_wrapper_modern_bert_rel'

pretrained_model_name_or_path class-attribute instance-attribute

pretrained_model_name_or_path = 'answerdotai/ModernBERT-base'

load classmethod

load(tokenizer_path: str, relcat_config: ConfigRelCAT, **kwargs) -> TokenizerWrapperModernBERT

Source code in medcat-v2/medcat/components/addons/relation_extraction/modernbert/tokenizer.py

@classmethod
def load(cls, tokenizer_path: str, relcat_config: ConfigRelCAT, **kwargs
         ) -> "TokenizerWrapperModernBERT":
    tokenizer = cls()
    path = os.path.join(tokenizer_path, cls.name)

    if tokenizer_path:
        tokenizer.hf_tokenizers = PreTrainedTokenizerFast.from_pretrained(
            path, **kwargs)
    else:
        relcat_config.general.model_name = (
            cls.pretrained_model_name_or_path)
        tokenizer.hf_tokenizers = PreTrainedTokenizerFast.from_pretrained(
            relcat_config.general.model_name)
    return tokenizer