NLTK: Allowlisted pickle loaders still permit code execution in current source

Summary

The current source tree still allows arbitrary code execution during supposedly safer allowlisted pickle loading. The allowlist trusts whole module namespaces instead of exact safe globals, so crafted pickles can invoke dangerous in-namespace callables through pickle REDUCE.

Details

• Vulnerability type: Remote code execution via unsafe deserialization
• Affected component: nltk.picklesec.allowlisted_pickle_load, nltk.tokenize.punkt.punkt_pickle_load, nltk.parse.transitionparser.TransitionParser.parse
• Affected versions: Current source v3.10.0-rc2; published 3.9.4 was not the claim target for this bypass.
• Patched versions: Not yet patched
• Root cause: Module-prefix allowlists include dangerous callables such as nltk.tokenize.repp.ReppTokenizer._execute and numpy.f2py.crackfortran.myeval.

punkt_pickle_load() allowlists both nltk.tokenize.punkt and the whole nltk.tokenize namespace, which exposes ReppTokenizer._execute() and its subprocess.Popen(...) sink during unpickling. TransitionParser.parse() uses allowlisted_pickle_load(..., allowed_modules=("numpy", "scipy", "sklearn")), which permits numpy.f2py.crackfortran.myeval() and its attacker-controlled eval(...) path. I confirmed both gadgets create marker files before the caller returns or later aborts on type misuse.

PoC

Preconditions

• The application loads an attacker-controlled tokenizer or model artifact through these public loaders.

Steps

1. Create a pickle whose REDUCE callable is ReppTokenizer._execute and point its command to a harmless marker-file write.
2. Pass that payload to punkt_pickle_load(BytesIO(payload)) and observe the marker file is created during unpickling.
3. Create a second pickle whose REDUCE callable is numpy.f2py.crackfortran.myeval and load it through TransitionParser.parse().
4. Observe the second marker file is created before TransitionParser.parse() later fails on the returned object type.

Minimal reproducible excerpt

 1{'punkt_marker': 'PUNKT_RCE', 'transitionparser_marker': 'TP_RCE'}

Impact

Any caller that trusts these current allowlisted loaders can still execute attacker-controlled commands while loading model or tokenizer artifacts. This defeats the protection mechanism that replaced unrestricted pickle loading and creates a dangerous false sense of safety.

Remediation

Replace broad module-prefix allowlists with exact (module, qualname) pairs for the few safe classes or functions genuinely required. Do not allow entire namespaces such as nltk.tokenize or numpy, and keep post-load type validation only as a secondary defense.

Resources

• https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/tokenize/punkt.py#L120-L134
• https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/tokenize/repp.py#L111-L115
• https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/parse/transitionparser.py#L26-L30
• https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/parse/transitionparser.py#L565-L571