"""Drive defusedxml's three guards against the stdlib parser they replace, and print one key=value per line. ONE INTERPRETER START DOES THE WHOLE OF IT. Every python3 start on this arm is a fresh load of one large static image (99-python3 records the cost), so the suite that drives this asserts on the OUTPUT rather than calling python3 once per case. Every line is `key=value`; a key is emitted at most once. The suite reads the keys it names and ignores the rest, so adding a fact here cannot break it. Failure is reported as a key too -- `*_error` -- rather than only as an exit status, because the case that reds should say which stage stopped and why. argv[1] is a WRITABLE SCRATCH DIRECTORY. The external-entity fixtures name a file in it by absolute path, so the document cannot be a literal: it is built at run time around a path this process just wrote. That is also why the fixture lengths are NOT asserted the way a static document's would be -- the path length varies with $TMPDIR -- and why the suite pins the fixtures by SHAPE and by outcome instead. ============================================================================ THE BOMBS ARE BOUNDED ON PURPOSE, AND THAT IS NOT A WEAKENING OF THE TEST. ============================================================================ Each attack fixture is parsed TWICE: once through defusedxml, which must refuse it, and once through the stdlib `xml.etree.ElementTree`, which must behave differently. The stdlib leg is what stops a package that installed as a silent no-op from passing -- and it means the stdlib ACTUALLY EXPANDS these documents on target. So a true billion-laughs would be run, in full, by the control leg, on the guest, and would take the machine down -- wedging not just this suite but the shared batch image's whole verify. The fixtures are therefore the billion-laughs and quadratic-blowup SHAPES with their expansion factors cut to nothing: nested entity indirection three levels deep expanding to 27 bytes, and one entity referenced 16 times expanding to 1024. What each case asserts is the mechanism -- nested expansion, repeated reference -- and defusedxml stops both at the ENTITY DECLARATION, before expansion factor is a thing that exists. A bomb sized to hurt would test the guest's memory limits, not this library. ============================================================================ WHAT THE EXTERNAL-ENTITY CASE ACTUALLY RAISES -- MEASURED, NOT ASSUMED. ============================================================================ With defusedxml's DEFAULTS (`forbid_dtd=False, forbid_entities=True, forbid_external=True`) a document whose internal subset declares `` raises **EntitiesForbidden**, NOT ExternalReferenceForbidden. The entity DECLARATION handler fires first; the reference is never reached, so the external handler never runs. defusedxml is stopping the attack one step EARLIER than the obvious reading of its own exception names suggests. `ExternalReferenceForbidden` is reachable, and this probe reaches it, but only on its own path: `DefusedXMLParser(forbid_entities=False)` lets the declaration through so that the REFERENCE is what gets stopped. That is a real, documented API path and the only way to state that guard as behaviour. ============================================================================ AND WHAT THE STDLIB CONTROL DOES WITH IT -- ALSO MEASURED. ============================================================================ The stdlib does NOT read the file either. CPython's expat binding resolves no external entity by default, so the stdlib leg of the XXE fixture raises `ParseError: undefined entity`. The difference defusedxml makes here is therefore NOT "we refuse where the stdlib leaks" -- it is WHICH guard stops it and WHEN: a typed, catchable DefusedXmlException naming the entity, at declaration time, versus an untyped well-formedness error at reference time. The suite states it that way. Asserting that the stdlib resolves the entity would red against a CORRECT stdlib, which is a false red about the platform. The security property the ticket asks for -- the file is left unread -- is asserted directly and independently of either parser's exception: the file's marker string is searched for in every parse result and every exception text, and `secret_leaked` reports it. That case is about the bytes, so it holds however the parsers are wired. ============================================================================ THE PORTING PREREQUISITE THIS PACKAGE HAS AND THE OBVIOUS ONES DO NOT. ============================================================================ defusedxml.ElementTree does NOT subclass the XMLParser you get from `import xml.etree.ElementTree`. `_get_py3_cls()` masks `_elementtree` in sys.modules and re-imports the module to obtain the PURE-PYTHON XMLParser, because only that one exposes the expat handler slots the guards are installed into. Two consequences this image has to satisfy, and both are emitted here: - `xml/etree/ElementTree.py` must be present AS A SOURCE FILE the import system can reach. An image that shipped only frozen or byte-compiled stdlib, or a python3-xml trimmed to the C accelerator, breaks defusedxml at IMPORT with an error naming neither. - the parser really is the pure-Python one -- `parser_is_stdlib_xmlparser=no`. A build where that came back `yes` would mean the guards were installed on a class that ignores them, which is a SILENTLY DEFEATED defusedxml: it would import, expose every name, and refuse nothing. The second is the case worth having. It is the one shape of failure in which every other case here could pass while the package did nothing at all. """ import hashlib import os import sys import warnings _seen = set() def emit(key, value): if key in _seen: return _seen.add(key) # One line per fact: the suite greps by key, and an embedded newline would # land as a second, unkeyed line. print("%s=%s" % (key, str(value).replace("\n", " ").replace("\r", " "))) def fail(key, exc): emit(key, "%s: %s" % (type(exc).__name__, exc)) raise SystemExit(1) def sha16(s): if not isinstance(s, bytes): s = s.encode("utf-8") return hashlib.sha256(s).hexdigest()[:16] # ------------------------------------------------------------ the scratch dir if len(sys.argv) < 2: emit("argv_error", "no scratch directory given") raise SystemExit(1) SCRATCH = sys.argv[1] # The file the external-entity fixtures try to reach. Its content is a single # marker line: `secret_leaked' below greps every parse result and every exception # string for it, so the "left unread" property is asserted against the BYTES # rather than inferred from which exception came back. MARKER = "BRINTOS-1914-XXE-MARKER" SECRET = os.path.join(SCRATCH, "xxe-secret.txt") try: with open(SECRET, "w") as fh: fh.write(MARKER + "\n") except Exception as exc: # noqa: BLE001 - reported, not raised fail("scratch_error", exc) emit("secret_path", SECRET) # ------------------------------------------------------------------ the import try: import defusedxml except Exception as exc: # noqa: BLE001 fail("import_error", exc) # THE EXPAT BINDING, REPORTED SEPARATELY AND FIRST. defusedxml.common imports # `xml.parsers.expat' at module scope and raises ImportError("pyexpat") if it # lacks ParserCreate, so an interpreter without pyexpat presents AS a broken # defusedxml. On this image pyexpat is BUILTIN rather than a lib-dynload .so # (99-python3/builtin-modules asserts that directly); this key tells the two # apart. try: import pyexpat emit("pyexpat_builtin", "yes" if "pyexpat" in sys.builtin_module_names else "no") emit("expat_version", "%d.%d.%d" % pyexpat.version_info) except Exception as exc: # noqa: BLE001 fail("pyexpat_error", exc) try: import defusedxml.ElementTree as DET except Exception as exc: # noqa: BLE001 fail("et_import_error", exc) import xml.etree.ElementTree as SET emit("package_file", defusedxml.__file__) emit("version", getattr(defusedxml, "__version__", "")) try: from importlib.metadata import version as _dist_version emit("dist_version", _dist_version("defusedxml")) except Exception as exc: # noqa: BLE001 emit("dist_version_error", "%s: %s" % (type(exc).__name__, exc)) # THE PUBLIC SURFACE, from the package's own `__all__' rather than from dir(): # defusedxml DECLARES one, and it is exactly the five exception types -- the # facades are submodules, not names on the package. `et_all' is the ElementTree # facade's own declaration, which is the surface a caller actually imports. emit("api_all", ",".join(defusedxml.__all__)) emit("et_all", ",".join(DET.__all__)) emit("et_origin", DET.__origin__) # --------------------------------------------- the pure-Python parser, checked # See the header. The guards are installed into expat handler slots that only the # pure-Python XMLParser exposes, so defusedxml re-imports the module with # `_elementtree' masked to get it. If this came back `yes' the guards would be # sitting on a class that ignores them and defusedxml would refuse NOTHING while # passing every surface case here. emit("parser_is_stdlib_xmlparser", "yes" if DET.DefusedXMLParser.__bases__[0] is SET.XMLParser else "no") emit("parser_base", DET.DefusedXMLParser.__bases__[0].__name__) # ...and the source file that re-import needs, named from the interpreter's own # answer rather than guessed from a version string. emit("stdlib_et_file", getattr(SET, "__file__", "")) emit("stdlib_et_is_source", "yes" if str(getattr(SET, "__file__", "")).endswith(".py") else "no") # ------------------------------------------------------------- the submodules # Every facade the package ships, imported one at a time so a red names the one # that broke. `lxml' is expected to FAIL on this image -- lxml is not ported -- # and that is recorded as its own key rather than folded in with the others; see # the suite header for why the absence is asserted rather than skipped. with warnings.catch_warnings(): # cElementTree and lxml both emit DeprecationWarning at import. They are # upstream's own deprecations, not this port's business, and letting them # reach stderr would put unkeyed text in the middle of the probe's output. warnings.simplefilter("ignore") _ok, _bad = [], [] for _m in ("cElementTree", "common", "ElementTree", "expatbuilder", "expatreader", "minidom", "pulldom", "sax", "xmlrpc"): try: __import__("defusedxml." + _m) _ok.append(_m) except Exception as exc: # noqa: BLE001 _bad.append("%s(%s)" % (_m, type(exc).__name__)) emit("submodules_ok", ",".join(_ok)) emit("submodules_failed", ",".join(_bad) if _bad else "") try: __import__("defusedxml.lxml") emit("lxml_facade", "importable") except ImportError as exc: # noqa: BLE001 emit("lxml_facade", "ImportError") emit("lxml_facade_names_lxml", "yes" if "lxml" in str(exc) else "no") except Exception as exc: # noqa: BLE001 emit("lxml_facade", type(exc).__name__) # ------------------------------------------------------------- the fixtures # The billion-laughs SHAPE: nested entity indirection, three levels, expanding to # 27 bytes rather than to a gigabyte. See the header -- the stdlib control leg # really does expand this on the guest. BILLION = ( '' '' '' '' ']>&lol2;' ) # The quadratic-blowup SHAPE: ONE entity, referenced many times. 64 bytes x 16 # references = 1024, which is the whole expansion. QUAD = ( '' ']>' '' + ("&a;" * 16) + '' ) # The external-entity fixture, naming the file written above by absolute path. XXE = ( '' ']>' '&xxe;' % SECRET ) # A perfectly ordinary document, to show the hardening does not break input that # was never an attack. GOOD = ( '' 'textxy' ) emit("billion_sha", sha16(BILLION)) emit("quad_sha", sha16(QUAD)) emit("good_sha", sha16(GOOD)) emit("good_len", len(GOOD)) # Everything either parser ever returned or raised, for the leak check at the end. _witness = [] def run(label, doc, parse): """Parse `doc`, emit outcome/exception-type/text keys, and bank the result.""" try: el = parse(doc) except Exception as exc: # noqa: BLE001 _witness.append(str(exc)) emit(label + "_outcome", "raised") emit(label + "_exc", type(exc).__name__) # The exception TEXT is emitted for a reader of a red but is NOT the # thing asserted: every one of these strings interpolates the scratch # path, and the stdlib ParseError even carries a COLUMN NUMBER that moves # with the length of $TMPDIR. What is asserted is the type, plus the # derived `*_names_secret` below, which is a yes/no about the same string. emit(label + "_excstr", str(exc)) emit(label + "_names_secret", "yes" if SECRET in str(exc) else "no") return None text = el.text if el.text is not None else "" _witness.append(text) _witness.append(SET.tostring(el).decode("utf-8", "replace")) emit(label + "_outcome", "parsed") emit(label + "_textlen", len(text)) return el # ------------------------------------------- the guards, through the facade --- # defusedxml's DEFAULTS. All three attack fixtures must be refused, and the # header records the measured surprise: the external-entity one is stopped at the # DECLARATION, so it raises EntitiesForbidden and not ExternalReferenceForbidden. run("def_billion", BILLION, DET.fromstring) run("def_quad", QUAD, DET.fromstring) run("def_xxe", XXE, DET.fromstring) # ----------------------------------------------- the same three, undefused --- # The control leg. Without it a package that installed as a no-op -- or was never # installed at all, with `defusedxml' resolving to something else on the path -- # would pass every case above by refusing nothing that was ever going to happen. run("std_billion", BILLION, SET.fromstring) run("std_quad", QUAD, SET.fromstring) run("std_xxe", XXE, SET.fromstring) # -------------------------------- ExternalReferenceForbidden, on its own path -- # Reachable only with the entity guard stood down, so that the REFERENCE rather # than the declaration is what expat stops on. See the header. try: _p = DET.DefusedXMLParser(forbid_entities=False) SET.fromstring(XXE, parser=_p) emit("extref_outcome", "parsed") except Exception as exc: # noqa: BLE001 _witness.append(str(exc)) emit("extref_outcome", "raised") emit("extref_exc", type(exc).__name__) emit("extref_names_secret", "yes" if SECRET in str(exc) else "no") # ------------------------------------------------- DTDForbidden, on its path -- # Off by default -- a DTD is not itself an attack -- so this is the fourth guard # stated as behaviour rather than left as an unexercised constructor argument. try: _p = DET.DefusedXMLParser(forbid_dtd=True) SET.fromstring(BILLION, parser=_p) emit("dtd_outcome", "parsed") except Exception as exc: # noqa: BLE001 _witness.append(str(exc)) emit("dtd_outcome", "raised") emit("dtd_exc", type(exc).__name__) # ------------------------------------------ the exception hierarchy, asserted -- # Every guard raises a DefusedXmlException, which IS a ValueError. That is the # contract a caller writes `except ValueError' against, and it is what makes the # refusals catchable as a class rather than one name at a time. from defusedxml.common import ( # noqa: E402 DefusedXmlException, DTDForbidden, EntitiesForbidden, ExternalReferenceForbidden, NotSupportedError, ) emit("exc_base_is_valueerror", "yes" if issubclass(DefusedXmlException, ValueError) else "no") emit("exc_all_defused", "yes" if all( issubclass(c, DefusedXmlException) for c in (DTDForbidden, EntitiesForbidden, ExternalReferenceForbidden, NotSupportedError)) else "no") # ---------------------------------------------------- the well-formed document -- # The hardening must not change ordinary input. Compared as SERIALISED BYTES # through each parser's own tree, which is the strongest form available: same # elements, same attributes, same text. _gd = _gs = None try: _gd = DET.tostring(DET.fromstring(GOOD)) emit("good_defused", "parsed") except Exception as exc: # noqa: BLE001 emit("good_defused", "%s: %s" % (type(exc).__name__, exc)) try: _gs = SET.tostring(SET.fromstring(GOOD)) emit("good_stdlib", "parsed") except Exception as exc: # noqa: BLE001 emit("good_stdlib", "%s: %s" % (type(exc).__name__, exc)) emit("good_identical", "yes" if (_gd is not None and _gd == _gs) else "no") if _gd is not None: emit("good_out_sha", sha16(_gd)) emit("good_out_len", len(_gd)) # ------------------------------------------------------------ the leak check -- # THE SECURITY PROPERTY, ASSERTED AGAINST THE BYTES. Nothing either parser # returned, and nothing either parser said, may contain the marker the secret # file holds. This is independent of which exception came back, so it stays true # however the two parsers are wired -- and it is the case that would catch an # expat actually configured to resolve external entities. emit("secret_leaked", "yes" if any(MARKER in w for w in _witness) else "no") # ...and the file is still there, unchanged, so `secret_leaked=no' means "never # read" rather than "never written". try: with open(SECRET) as fh: emit("secret_intact", "yes" if fh.read() == MARKER + "\n" else "no") except Exception as exc: # noqa: BLE001 emit("secret_intact", "%s: %s" % (type(exc).__name__, exc)) emit("done", "yes")