Healthcare NLP for Data Scientists

?Learning Objectives:

1. Understand how to use AverageEmbeddings.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to use BertSentenceChunkEmbeddings.

2. Become comfortable using the different parameters of the annotator.

This annotator that splits documents or sentences by chunks provided. Splitted parts can be named with the splitting chunks.

? Learning Objectives:

1. Understand how It is useful when you need to perform different models or analysis in different sections of your document (for example, for different headers, clauses, items, etc.).

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Comprehend the need for Entity Chunk Embeddings and their relationship with BERT Sentence embeddings.

2. Understand the concept of a weighted average vector representation of related entity chunks.

3. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

• Merging two or more same type annotation results in a spark nlp pipeline

Replacer allows to replace entities in the original text with the ones extracted by the annotators NameChunkObfuscatorApproach or DateNormalizer.

?Learning Objectives:

1. Understand how Replacer works.

2. Understand how Replacer can be used to with the DateNormalizer annotator and in the deintification process.

3. Become comfortable using the setUseReplacement parameter of the annotator.

?Learning Objectives:

1. Understand how to use Chunk2Token.

2. Become comfortable using the different parameters of the annotator.

?Learning Objectives:

1. Understand how to extract key phrases from texts.

2. Become comfortable using the different parameters of the ChunkKeyPhraseExtraction.

This annotator normalizes Date chunks into a chosen format.

?Learning Objectives:

1. Understand how it is useful when using data from different sources, some times from different countries that has different formats to represent dates.

2. Become comfortable using the different parameters of the annotator.

This annotator is designed to normalize unprocessed text extracted from clinical documents, like web pages or XML files, converting it into sentences.

?Learning Objectives:

1. Understand how it normalizes mentions of drugs in clinical text. You can utilize it to remove unwanted characters according to specific rules and perform lowercase formatting.

2. Become comfortable using the different parameters of the annotator.

IOBTagger merges token tags and NER labels from chunks in the specified format. For example output columns as inputs from NerConverter and Tokenizer can be used to merge.

This notebook will cover the different parameters and usages of IOBTagger.

? Learning Objectives:

Become comfortable using the different parameters of the IOBTagger.

This annotator links words of interest, such as names of persons, locations and companies, from an input text document to a corresponding unique entity in a target Knowledge Base (KB).

? Learning Objectives:

1. Background: Understand the NerDisambiguation

2. Colab setup

3. Become comfortable using the different parameters of the annotator.

This annotator extracts phrases that fits into a known pattern using the NER tags.

? Learning Objectives:

1. Understand how NerChunker works.

2. Become comfortable using the Regex parameter of the annotator.

The Flattener converts annotation results into a format that easier to use. This annotator produces a DataFrame with flattened and exploded columns containing annotation results, making it easier to interpret and analyze the information. It is particularly useful for extracting and organizing the results obtained from Spark NLP Pipelines.

? Learning Objectives:

1. Understand how to use the annotator.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to use NerQuestionGenerator.

2. Become comfortable using the different parameters of the annotator.

3. Programatically generate question to be used by Question-Answering models.

This lecture covers the uses of InternalDocumentSplitter. This annotator specifically target to split documents into relevant sections.

?Learning Objectives:

1. Understand how InternalDocumentSplitter works.

2. Become comfortable using the parameters of the annotator.

RegexMatcher

This notebook will cover the different parameters and usages of the RegexMatcher annotator. This annotator provides the ability to tag occurrences of regex patterns in raw text.

? Learning Objectives:

1. Find occurrences of regular expression (regex) patterns in text

2. Set one or more regex rules and assign an identifier for each regex rule

3. Create and use external regex rules file

4. Change the matching strategy of RegexMatcher

This annotator converts a IOB or IOB2 representation of a named entity to a user-friendly one, by associating the tokens of recognized entities and their label. Chunks with no associated entity (tagged “O”) are filtered out.

? Learning Objectives:

1. Understand how NerConverterInternal works.

2. Become comfortable using the different parameters of the annotator.

We will examine the MedicalNerApproach to to train an MedicalNerModel.

?Learning Objectives:

1. Understand the meaning and usage of Named Entity Recognition.

2. Learn how to preprocess your data before training a model.

3. Understand how to train a Named Entity Recognition model using MedicalNerApproach.

This Named Entity recognition annotator is a generic NER model based on Neural Networks. Neural Network architecture is Char CNNs - BiLSTM - CRF that achieves state-of-the-art in most datasets.

In the original framework, the CNN extracts a fixed length feature vector from character-level features. For each word, these vectors are concatenated and fed to the BLSTM network and then to the output layers. They employed a stacked bi-directional recurrent neural network with long short-term memory units to transform word features into named entity tag scores. The extracted features of each word are fed into a forward LSTM network and a backward LSTM network. The output of each network at each time step is decoded by a linear layer and a log-softmax layer into log-probabilities for each tag category. These two vectors are then simply added together to produce the final output. In the architecture of the proposed framework in the original paper, 50-dimensional pretrained word embeddings is used for word features, 25-dimension character embeddings is used for char features, and capitalization features (allCaps, upperInitial, lowercase, mixedCaps, noinfo) are used for case features.

?Learning Objectives:

1. Understand how to detect Named Entities by using pre-trained models.

2. Become comfortable using the different parameters of the annotator.

We will examine the MedicalBertForTokenClassifier annotator for Named-Entity-Recognition (NER) tasks.

? Learning Objectives:

1. Understand how to detect Named Entities by using Bert models with a token classification.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to use ChunkFilterer.

2. Become comfortable using the different parameters of the annotator.

?Learning Objectives:

1. Understand how to set filters, via a white list of terms or a regular expression.

2. Become comfortable using the different parameters of the ChunkFilterer.

We will examine the ChunkMapperApproach to create custom mapper model based on the given json file.

This annotator ensures creating of a mapper to map the chunks based on a pre-defined dictionary with no machine learning/deep learning model.

? Learning Objectives:

1. Understand how to create a mapper model by using pre-defined dictionary.

2. Become comfortable using the different parameters of the annotator.

We will cover the different parameters and usages of ChunkMergeModel. This annotator provides the ability to merge chunk columns coming from two or more annotators using a model generated by ChunkMergeApproach. Common parameters with ChunkMergeApproach can be used in ChunkMergeModel in the same way.

? Learning Objectives:

1. Merging two or more chunks results in a spark nlp pipeline

2. Using ChunkMergeModel annotator's parameters to get desired outputs

? Learning Objectives:

1. Understand how to use ChunkConverter.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to use ContextualParserModel.

2. Become comfortable using the different parameters of the annotator.

3. Train a ContextualParserApproach annotator and use that model with ContextualParserModel in the future.

?Learning Objectives:

1. Understand how to use ZeroShotNerModel.

2. Become comfortable using the different parameters of the annotator.

3. Identify clinical entities on text without training data.

? Learning Objectives:

1. Understand how to use ContextualParserModel.

2. Become comfortable using the different parameters of the annotator.

3. Train a ContextualParserApproach annotator and use that model with ContextualParserModel in the future.

EntityRuler

This notebook will cover the different parameter and usage of EntityRuler. There are 2 annotators to perform this task in Spark NLP; EntityRulerApproach and EntityRulerModel.

• EntityRulerApproach fits an Annotator to match exact strings or regex patterns provided in a file against a Document and assigns them an named entity. The definitions can contain any number of named entities.

• EntityRulerModel is instantiated model of the EntityRulerApproach

? Learning Objectives:

1. Understand how to extract entities with predefined regex patterns or match predefined exact strings.

2. Understand the difference between the EntityRulerApproach and EntityRulerModel.

3. Become comfortable using the different parameters of these annotators.

In this notebook, we will examine the TextMatcherInternal annotator and its model version TextMatcherInternalModel.

This annotator match exact phrases provided in a file against a Document.

? Learning Objectives:

1. Understand how to match exact phrases by using pre-defined dictionary.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand the meaning and use of assertion status.

2. Learn how to create a chunk column with metadata for training assertion status detection models.

3. Customize your assertion model by using the different parameters of the annotator.

? Learning Objectives:

1. Understand the meaning and use of assertion status.

2. Learn how to create a chunk column with metadata for training assertion status detection models.

3. Customize your assertion model by using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to use AssertionDLModel.

2. Become comfortable using the different parameters of the annotator.

3. Build a pretraine pipeline using AssertionDLModel annotator.

?Learning Objectives:

1. Understand how to use AssertionLogRegApproach.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to use AssertionLogRegModel.

2. Become comfortable using the different parameters of the annotator.

?Learning Objectives:

1. Understand how to use AssertionDLApproach.

2. Become comfortable using the different parameters of the annotator.

3. Train a AssertionDLApproach annotator and use that model with AssertionDLApproach in the future.

This Relation Extraction annotator extracts and classifies instances of relations between named entities. In contrast with RelationExtractionModel, RelationExtractionDLModel is based on BERT.

?Learning Objectives:

1. Understand how to extract and classify the relations between named entities by using pre-trained BERT models.

2. Become comfortable using the different parameters of the annotator.

We will examine the RelationExtractionApproach to to train an RelationExtractionModel.

?Learning Objectives:

1. Learn how to preprocess your data before training a Relation Extraction model.

2. Understand how to train a Relation Extraction model using RelationExtractionApproach.

3. Understand how to resume Relation Extraction model training.

We will examine the RelationExtractionApproach to to train an RelationExtractionModel.

? Learning Objectives:

1. Learn how to preprocess your data before training a Relation Extraction model.

2. Understand how to train a Relation Extraction model using RelationExtractionApproach.

3. Understand how to resume Relation Extraction model training.

The RENerChunksFilter annotator filters NER chunks that contains the desired NER entity pairs only.

?Learning Objectives:

1. Understand how to filters desired relation pairs for the RelationExtractionDLModel

2. Become comfortable using the different parameters of the annotator.

This Relation Extraction annotator extracts and classifies instances of relations between named entities.

? Learning Objectives:

1. Understand how to extract and classify the relations between named entities by using pre-trained models.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to use ZeroShotRelationExtractionModel.

2. Become comfortable using the different parameters of the annotator.

3. Identify clinical relations on text without training data.

? Learning Objectives:

1. Understand how to use FeaturesAssembler.

2. Using FeaturesAssembler with several columns.

3. Using FeaturesAssembler with embeddings.

? Learning Objectives: Become comfortable using the different parameters of the annotator.

? Learning Objectives: Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how trains a TensorFlow model for generic classification of feature vectors.

2. Become comfortable using the different parameters of the annotator.

?Learning Objectives:

1. Background: Understand the 'GenericSVMClassifierModel' annotator.

2. Colab setup.

3. Become comfortable with using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to use GenericLogRegClassifierApproach.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to map chunks by using pre-defined dictionary.

2. Become comfortable using the different parameters of the annotator.

?Learning Objectives:

1. Understand how trains a TensorFlow model for SVMClassifier of feature vectors.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Understand how to train a model to classify documents with a Logarithmic Regression algorithm.

2. Become comfortable using the different parameters of the annotator.

?Learning Objectives:

1. Background: Understand the 'DocumentMLClassifierModel' Annotator.

2. Colab setup.

3. Become comfortable with using the different parameters of the annotator.

This lecture covers the uses of FewShotClassifierModel. This annotator specifically targets few-shot classification tasks, which involves training models to make accurate predictions with limited labeled data.

?Learning Objectives:

1. Understand how FewShotClassifierModel works.

2. Become comfortable using the parameters of the annotator.

This lecture will cover the different parameters and usages of WindowedSentenceModel. This annotator helps you merge the previous and following sentences of a given piece of text, so that you add the context surrounding them.

?Learning Objectives:

1. Understand how it is super useful when using for especially context-rich analyses that require a deeper understanding of the language being used.

2. Become comfortable using the different parameters of the annotator.

This lecture covers the uses of DocumentLogRegClassifier. This annotator uses a supervised learning algorithm that learns to classify documents (or text) into predefined categories or classes based on the content of the text.

?Learning Objectives:

1. Understand how DocumentLogRegClassifier works.

2. Become comfortable using the parameters of the annotator.

This lecture covers the uses of DocumentFiltererByClassifier. This annotator is designed to filter documents based on outcomes generated by classifier annotators.

?Learning Objectives:

1. Understand how DocumentFiltererByClassifier works.

2. Become comfortable using the parameters of the annotator.

? Learning Objectives:

Become comfortable using the different parameters of the Resolution2Chunk.

This annotator maps chunks based on pre-defined dictionary with no machine learning/deep learning model.

?Learning Objectives:

1. Understand how to map chunks by using pre-defined dictionary.

2. Become comfortable using the different parameters of the annotator.

This annotator maps document typed strings based on pre-defined dictionary with no machine learning/deep learning model.

? Learning Objectives:

1. Understand how to map document based strings by using pre-defined dictionary.

2. Become comfortable using the different parameters of the annotator.

We will examine the DocMapperApproach to create custom mapper model based on the given json file.

This annotator ensures creating of a mapper to map the document typed strings based on a pre-defined dictionary with no machine learning/deep learning model.

? Learning Objectives:

1. Understand how to create a mapper model by using pre-defined dictionary.

2. Become comfortable using the different parameters of the annotator.

We will examine the ChunkMapperApproach to create custom mapper model based on the given json file.

This annotator ensures creating of a mapper to map the chunks based on a pre-defined dictionary with no machine learning/deep learning model.

?Learning Objectives:

1. Understand how to create a mapper model by using pre-defined dictionary.

2. Become comfortable using the different parameters of the annotator.

We will examine the ChunkMapperApproach to create custom mapper model based on the given json file.

This annotator ensures creating of a mapper to map the chunks based on a pre-defined dictionary with no machine learning/deep learning model.

?Learning Objectives:

1. Understand how to create a mapper model by using pre-defined dictionary.

2. Become comfortable using the different parameters of the annotator.

We will examine the ChunkMapperFilterer annotator.

This annotator filters the chunks based on whether the ChunkMapperModel annotator successfully mapped the chunk or not.

? Learning Objectives:

1. Understand how to filter the chunks that were passed through the ChunkMapperModel.

2. Become comfortable using the different parameters of the annotator.

?Learning Objectives:

1. Understand how to use Doc2ChunkInternal.

2. Become comfortable using the different parameters of the annotator.

This annotator provides the ability to split an output of an annotator for a selected metadata field and the value for that field.

?Learning Objectives:

1. Using Router annotators' parameters to get desired outputs

2. Using Router for optimizing getting sentence embeddings and multiple sentence entity resolver models.

This annotator maps clinical entities to a particilar ontology / curated dataset using sentence embeddings.

? Learning Objectives:

1. Understand the application and relevance of these models in healthcare data analysis

2. Map clinical entities to standard codes (ICD-10, RxNorm, SNOMED, etc.)

3. Become comfortable using the different parameters of the annotator.

This annotator trains a SentenceEntityResolverModel that maps sentence embeddings to entities in a knowledge base.

?Learning Objectives:

1. Understand the application and relevance of these models in healthcare data analysis, particularly in coding and classification tasks related to healthcare ontologies like ICD-10, RxNorm, SNOMED, etc.

2. Become comfortable using the different parameters of the annotator.

This annotator provides the ability to obfuscate or mask the entities that contains personal information.

?Learning Objectives:

1. Background: Understand the Deidentification module

2. Colab setup

3. Become comfortable with deidentiifcation using the different parameters of the annotator.

This annotator provides the ability to obfuscate or mask the entities that contains personal information.

?Learning Objectives:

1. Background: Understand the Deidentification module

2. Colab setup

3. Become comfortable with deidentiifcation using the different parameters of the annotator.

This annotator can reidentifies obfuscated entities by DeIdentification. It requires the outputs from the deidentification as input. Input columns need to be the deidentified document and the deidentification mappings set with DeIdentification.setMappingsColumn.

? Learning Objectives:

1. Background: Understand the Deidentification then ReIdentification

2. Colab setup

3. Become comfortable using the different parameters of the annotator.

This annotator provides the ability to merge sentence enitity resolver and chunk mapper model output columns.

?Learning Objectives:

• Merging sentence enitity resolver and chunk mapper results in a spark nlp pipeline

This module can replace name entities with consistent fakers.

?Learning Objectives:

1. Obfuscation background

2. Colab setup

3. Become comfortable using the different parameters of the annotator.

It allows to transform a dataset with an Input Annotation of type CHUNK, into its obfuscated version of by obfuscating the given CHUNKS. This module can replace name entities with consistent fakers, remain others same.

? Learning Objectives:

1. Obfuscation background

2. Colab setup

3. Become comfortable using the different parameters of the annotator.

DocumentHashCoder() annotator is used for determining shifts date information for deidentification purposes.

This annotator gets the hash of the specified column and creates a new document column containing day shift information.

? Learning Objectives:

1. Understand how to shift days in Deidentification tasks by using DocumentHashCoder.

2. Become comfortable using the different parameters of the annotator.

? Learning Objectives:

1. Background: Understand the MedicalSummarizer Annotator.

2. Colab setup.

3. Become comfortable with using the different parameters of the annotator.

? Learning Objectives:

1. Background: Understand the 'ExtractiveSummarization' Annotator.

2. Colab setup.

3. Become comfortable with using the different parameters of the annotator.

?Learning Objectives:

1. Understand how to use MedicalQuestionAnswering.

2. Become comfortable using the different parameters of the annotator.

3. Train an MedicalQuestionAnswering based on pattern matching.