Customizing the NLP models in Presidio Analyzer
Presidio uses NLP engines for two main tasks: NER based PII identification, and feature extraction for custom rule based logic (such as leveraging context words for improved detection).
While Presidio comes with an open-source model (the en_core_web_lg
model from spaCy), it can be customized by leveraging other NLP models, either public or proprietary.
These models can be trained or downloaded from existing NLP frameworks like spaCy and Stanza.
In addition, other types of NLP frameworks can be integrated into Presidio.
Replacing the default NLP model with a different model
Setting up a new NLP model
As mentioned before, Presidio supports both spaCy and Stanza as its internal NLP engine. This section describes how new models from either spaCy or Stanza could be obtained, and how to configure Presidio to use them.
Download / create the new model
Using a public spaCy/Stanza model
To replace the default model with a different public model, first download the desired spaCy/Stanza NER models.
-
To download a new model with spaCy:
python -m spacy download es_core_news_md
In this example we download the medium size model for Spanish.
-
To download a new model with Stanza:
import stanza stanza.download("en") # where en is the language code of the model.
For the available models, follow these links: spaCy, stanza.
Tip
For Person, Location and Organization detection, it could be useful to try out the en_core_web_trf
model which uses a more modern deep-learning architecture, but is slower than the default en_core_web_lg
model.
Training your own model
Note
A labeled dataset containing text and labeled PII entities is required for training a new model.
To train your own model, see these links on spaCy and Stanza:
Once models are trained, they should be installed locally in the same environment as Presidio Analyzer.
Configure Presidio to use the new model
Configuration can be done in two ways:
-
Via code: Create an
NlpEngine
using theNlpEnginerProvider
class, and pass it to theAnalyzerEngine
as input:from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer.nlp_engine import NlpEngineProvider # Create configuration containing engine name and models configuration = { "nlp_engine_name": "spacy", "models": [{"lang_code": "es", "model_name": "es_core_news_md"}, {"lang_code": "en", "model_name": "en_core_web_lg"}], } # Create NLP engine based on configuration provider = NlpEngineProvider(nlp_configuration=configuration) nlp_engine_with_spanish = provider.create_engine() # Pass the created NLP engine and supported_languages to the AnalyzerEngine analyzer = AnalyzerEngine( nlp_engine=nlp_engine_with_spanish, supported_languages=["en", "es"] ) # Analyze in different languages results_spanish = analyzer.analyze(text="Mi nombre es Morris", language="es") print(results_spanish) results_english = analyzer.analyze(text="My name is Morris", language="en") print(results_english)
-
Via configuration: Set up the models which should be used in the default
conf
file.An example Conf file:
nlp_engine_name: spacy models: - lang_code: en model_name: en_core_web_lg - lang_code: es model_name: es_core_news_md
The default conf file is read during the default initialization of the
AnalyzerEngine
. Alternatively, the path to a custom configuration file can be passed to theNlpEngineProvider
:from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer.nlp_engine import NlpEngineProvider LANGUAGES_CONFIG_FILE = "./docs/analyzer/languages-config.yml" # Create NLP engine based on configuration file provider = NlpEngineProvider(conf_file=LANGUAGES_CONFIG_FILE) nlp_engine_with_spanish = provider.create_engine() # Pass created NLP engine and supported_languages to the AnalyzerEngine analyzer = AnalyzerEngine( nlp_engine=nlp_engine_with_spanish, supported_languages=["en", "es"] ) # Analyze in different languages results_spanish = analyzer.analyze(text="Mi nombre es David", language="es") print(results_spanish) results_english = analyzer.analyze(text="My name is David", language="en") print(results_english)
In this examples we: a. create an
NlpEngine
holding two spaCy models (one in English:en_core_web_lg
and one in Spanish:es_core_news_md
). b. define thesupported_languages
parameter accordingly. c. pass requests in each of these languages.Note
Presidio can currently use one NLP model per language.
Using a previously loaded NLP model
If the app is already loading an NLP model, it can be re-used to prevent presidio from loading it again by extending the relevant engine.
```python
from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.nlp_engine import SpacyNlpEngine
import spacy
# Create a class inheriting from SpacyNlpEngine
class LoadedSpacyNlpEngine(SpacyNlpEngine):
def __init__(self, loaded_spacy_model):
self.nlp = {"en": loaded_spacy_model}
# Load a model a-priori
nlp = spacy.load("en_core_web_sm")
# Pass the loaded model to the new LoadedSpacyNlpEngine
loaded_nlp_engine = LoadedSpacyNlpEngine(loaded_spacy_model = nlp)
# Pass the engine to the analyzer
analyzer = AnalyzerEngine(nlp_engine = loaded_nlp_engine)
# Analyze text
analyzer.analyze(text="My name is Bob", language="en")
```
Leverage frameworks other than spaCy or Stanza for ML based PII detection
In addition to the built-in spaCy/Stanza capabitilies, it is possible to create new recognizers which serve as interfaces to other models. See the Remote recognizer documentation and samples for more information.
For considerations for creating such recognizers, see the best practices for adding ML recognizers documentation.