Ë V¹� ®ãóÈ—dZddlZddlZddlZddlmZmZmZddlm Z ddl m Z ddl m Z mZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZGd„d«Zy)a Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco éN)ÚListÚOptionalÚUnioné)ÚCharSetGroupProber)Ú CharSetProber)Ú InputStateÚLanguageFilterÚ ProbingState)ÚEscCharSetProber)Ú Latin1Prober)ÚMacRomanProber)ÚMBCSGroupProber)Ú ResultDict)ÚSBCSGroupProber)Ú UTF1632Proberc óN—eZdZdZdZej d«Zej d«Zej d«Z dddd d d d d dœZ dddd ddddœZ e jdfde deddfd„Zedefd„«Zedefd„«Zedeefd„«Zd!d„Zdeeefddfd„Zdefd „Zy)"ÚUniversalDetectoraq The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result gš™™™™™É?s[€-ÿ]s(|~{)s[€-Ÿ]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)ú iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8ú iso-8859-9z iso-8859-13z ISO-8859-11ÚGB18030ÚCP949úUTF-16)Úasciirztis-620rÚgb2312zeuc-krzutf-16leFÚ lang_filterÚshould_rename_legacyÚreturnNcó—d|_d|_g|_ddddœ|_d|_d|_t j|_d|_ ||_ tjt«|_d|_||_|j#«y)Nç©ÚencodingÚ confidenceÚlanguageFó)Ú_esc_charset_proberÚ_utf1632_proberÚ_charset_probersÚresultÚdoneÚ _got_datar Ú PURE_ASCIIÚ _input_stateÚ _last_charrÚloggingÚ getLoggerÚ__name__ÚloggerÚ_has_win_bytesrÚreset)Úselfrrs ú>/usr/lib/python3.12/site-packages/chardet/universaldetector.pyÚ__init__zUniversalDetector.__init__dsˆ€ð @DˆÔ Ø8<ˆÔØ57ˆÔàØØñ# ˆŒ 𠈌 ؈ŒÜ&×1Ñ1ˆÔ؈ŒØ&ˆÔÜ×'Ñ'¬Ó1ˆŒ Ø#ˆÔØ$8ˆÔ!Ø � ‰ � r%có—|jS©N)r-©r5s r6Ú input_statezUniversalDetector.input_state{s€à× Ñ Ð r%có—|jSr9)r3r:s r6Ú has_win_byteszUniversalDetector.has_win_bytess€à×"Ñ"Ð"r%có—|jSr9)r(r:s r6Úcharset_probersz!UniversalDetector.charset_probersƒs€à×$Ñ$Ð$r%cóV—ddddœ|_d|_d|_d|_tj |_d|_|jr|jj«|jr|jj«|jD]}|j«Œy)zæ Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. Nr r!Fr%) r)r*r+r3r r,r-r.r&r4r'r()r5Úprobers r6r4zUniversalDetector.reset‡sŒ€ð $(°sÈÑMˆŒ ؈Œ ؈ŒØ#ˆÔÜ&×1Ñ1ˆÔ؈ŒØ × #Ò #Ø × $Ñ $× *Ñ *Ô ,Ø × Ò Ø × Ñ × &Ñ &Ô (Ø×+Ô+ˆFØ �L‰L�Nñ,r%Úbyte_strcóV —|jry|syt|t«s t|«}|jsø|j t j «r ddddœ|_n¯|j t jt jf«r ddddœ|_nt|j d«r ddddœ|_nW|j d «r d dddœ|_n:|j t jt jf«r d dddœ|_d |_|jd �d |_y|jtjk(r‹|jj!|«rtj"|_ nZ|jtjk(r=|j$j!|j&|z«rtj(|_ |dd|_|j*st-«|_|j*j.t0j2k(rk|j*j5|«t0j6k(r?|j*j8|j*j;«ddœ|_d |_y|jtj(k(r¦|j<st?|j@«|_|j<j5|«t0j6k(rS|j<j8|j<j;«|j<jBdœ|_d |_yy|jtj"k(�r:|jDs¡tG|j@«g|_"|j@tHjJzr#|jDjMtO««|jDjMtQ««|jDjMtS««|jDD]Z}|j5|«t0j6k(sŒ&|j8|j;«|jBdœ|_d |_n|jTj!|«rd |_+yyy)aý Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIGçð?Úr!zUTF-32sþÿzX-ISO-10646-UCS-4-3412sÿþzX-ISO-10646-UCS-4-2143rTr"éÿÿÿÿ),r*Ú isinstanceÚ bytearrayr+Ú startswithÚcodecsÚBOM_UTF8r)Ú BOM_UTF32_LEÚ BOM_UTF32_BEÚBOM_LEÚBOM_BEr-r r,ÚHIGH_BYTE_DETECTORÚsearchÚ HIGH_BYTEÚ ESC_DETECTORr.Ú ESC_ASCIIr'rÚstater Ú DETECTINGÚfeedÚFOUND_ITÚ charset_nameÚget_confidencer&r rr$r(rr ÚNON_CJKÚappendrr rÚWIN_BYTE_DETECTORr3)r5rBrAs r6rWzUniversalDetector.feedšsЀð �9Š9Ø áØ ä˜(¤IÔ.Ü  Ó*ˆHð�~Š~à×"Ñ"¤6§?¡?Ô3ð!,Ø"%Ø "ñ�• ð ×$Ñ$¤f×&9Ñ&9¼6×;NÑ;NÐ%OÔPð,4À3ÐTVÑW�• Ø×$Ñ$Ð%8Ô9ð!9Ø"%Ø "ñ �• ð ×$Ñ$Ð%8Ô9ð!9Ø"%Ø "ñ �• ð ×$Ñ$¤f§m¡m´V·]±]Ð%CÔDð,4À3ÐTVÑW�” à!ˆDŒNØ�{‰{˜:Ñ&Ð2Ø �” Øð × Ñ ¤ × 5Ñ 5Ò 5Ø×&Ñ&×-Ñ-¨hÔ7Ü$.×$8Ñ$8�Õ!à×!Ñ!¤Z×%:Ñ%:Ò:Ø×%Ñ%×,Ñ,¨T¯_©_¸xÑ-GÔHä$.×$8Ñ$8�Ô!à" 2 3˜-ˆŒð×#Ò#Ü#0£?ˆDÔ à × Ñ × %Ñ %¬×)?Ñ)?Ò ?Ø×#Ñ#×(Ñ(¨Ó2´l×6KÑ6KÒKà $× 4Ñ 4× AÑ AØ"&×"6Ñ"6×"EÑ"EÓ"GØ "ñ�” ð !�” Øð × Ñ ¤ × 4Ñ 4Ò 4Ø×+Ò+Ü+;¸Dר×/Ô/�Ø—;‘;˜xÓ(¬L×,AÑ,AÓAà$*×$7Ñ$7Ø&,×&;Ñ&;Ó&=Ø$*§O¡Oñ#�D”Kð !%�D”IÙð0ð×%Ñ%×,Ñ,¨XÔ6Ø&*�Õ#ð7ð#7r%c óH—|jr |jSd|_|js|jj d«�nD|j t jk(r ddddœ|_�n|j t jk(rýd}d}d}|jD]}|sŒ|j«}||kDsŒ|}|}Œ!|rÇ||jkDr¸|j}|€J‚|j«}|j«}|jd «r(|jr|j j#||«}|j$r.|j&j#|xsdj«|«}|||j(dœ|_|jj+«t,j.krà|jd €Ñ|jj d «|jD]§}|sŒt1|t2«rR|j4D]B}|jj d |j|j(|j««ŒDŒh|jj d |j|j(|j««Œ©|jS) zæ Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!rrDrEr!Nr ziso-8859r"z no probers hit minimum thresholdz%s %s confidence = %s)r*r)r+r2Údebugr-r r,rRr(rZÚMINIMUM_THRESHOLDrYÚlowerrIr3Ú ISO_WIN_MAPÚgetrÚ LEGACY_MAPr$ÚgetEffectiveLevelr/ÚDEBUGrGrÚprobers) r5Úprober_confidenceÚmax_prober_confidenceÚ max_proberrArYÚlower_charset_namer#Ú group_probers r6ÚclosezUniversalDetector.closesa€ð �9Š9Ø—;‘;Р؈Œ à�~Š~Ø �K‰K× Ñ Ð1Ö 2ð× Ñ ¤*×"7Ñ"7Ò 7Ø'.¸cÈrÑRˆDŽKð× Ñ ¤*×"6Ñ"6Ò 6Ø $Ð Ø$'Ð !؈JØ×/Ô/�ÙØØ$*×$9Ñ$9Ó$;Ð!Ø$Ð'<Ó<Ø,=Ð)Ø!'‘Jð 0ñÐ4°t×7MÑ7MÒMØ)×6Ñ6� Ø#Ð/Ð/Ð/Ø%1×%7Ñ%7Ó%9Ð"Ø'×6Ñ6Ó8� ð&×0Ñ0°Ô<Ø×*Ò*Ø'+×'7Ñ'7×';Ñ';Ø.° ó(˜ ð×,Ò,Ø#'§?¡?×#6Ñ#6Ø%Ò+¨×2Ñ2Ó4°ló$�Lð!-Ø",Ø *× 3Ñ 3ñ�” ð �;‰;× (Ñ (Ó *¬g¯m©mÒ ;Ø�{‰{˜:Ñ&Ð.Ø— ‘ ×!Ñ!Ð"DÔEØ$(×$9Ô$9�LÙ'Ø Ü! ,Ô0BÔCØ&2×&:Ô&:˜FØ ŸK™K×-Ñ-Ø 7Ø &× 3Ñ 3Ø &§¡Ø &× 5Ñ 5Ó 7õ ñ';🠙 ×)Ñ)Ø3Ø(×5Ñ5Ø(×1Ñ1Ø(×7Ñ7Ó9õ ð%:ð$�{‰{Ðr%)rN)r1Ú __module__Ú __qualname__Ú__doc__r`ÚreÚcompilerPrSr]rbrdr ÚALLÚboolr7ÚpropertyÚintr;r=rrr?r4rÚbytesrHrWrrm©r%r6rr8s7„ñð ÐØ#˜Ÿ™ NÓ3ÐØ�2—:‘:˜lÓ+€LØ"˜Ÿ ™  >Ó2Ðà$Ø$Ø$Ø$Ø$Ø$Ø$Ø%ñ €Kð Ø$Ø Ø$ØØØñ€Jð'5×&8Ñ&8Ø%*ñà#ðð#ðð ó ð.ð!˜Sò!óð!ðð#˜tò#óð#ðð%  mÑ!4ò%óð%óð&A+˜U 5¨)Ð#3Ñ4ðA+¸óA+ðFM�zôMr%r)rprJr/rqÚtypingrrrÚcharsetgroupproberrÚ charsetproberrÚenumsr r r Ú escproberr Ú latin1proberr ÚmacromanproberrÚmbcsgroupproberrÚ resultdictrÚsbcsgroupproberrÚ utf1632proberrrrxr%r6Úr„sHðñ8óÛÛ ß(Ñ(å2Ý(ß;Ñ;Ý'Ý&Ý*Ý,Ý"Ý,Ý(÷ròrr%