AHj& dZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZd d lmZmZd d lmZej.d ZeeddZej;edDcic]$}ej>e |ddk(s"|d&c}Z!edd1dedede"de#fdZ$de%edede%efdZ&d2dedede%efdZ'd2dedede%efd Z(d3dedede)fd!Z*ed"d#edede%efd$Z+Gd%d&Z,d'edefd(Z-d)\Z.Z/Z0Z1Gd*d+Z2e2eZ3d,eddfd-Z4d.ed/ede#fd0Z5ycc}w)4z>Code parts dedicated to duplicate removal and text similarity.N)SequenceMatcher) lru_cache)blake2b)add)RLock)Any)_Element)LRU_SIZE Extractor)trimz\.[^/?#]{2,63}$ bit_countc6t|jdS)N1)bincount)xs Q/root/tools/cai/cai_env/lib/python3.12/site-packages/trafilatura/deduplication.pyrsSV\\#5FiP i)maxsize reference new_string thresholdreturnctjd|}tjd|}td||j|k\S)zIReturn the similarity ratio between two short strings, here domain names.N)STRIP_EXTENSIONsubrratio)rrrs ris_similar_domainr#sE ##B 2I $$R4J 4J 7 = = ?9 LLrtokens target_lengthcg}tdddD]5}|Dcgc]}t||kDs|}}t||dz k\s3|cS|Scc}w)zCHelper function to get a sample of tokens based on length criteria.)rangelen)r$r%sampleits r_get_sample_by_lengthr/!s^ F 1b" #2s1vz!22 v;-!+ +M M3s A A  inputstringlengthc|jt}|jDcgc]}|js|}}t ||Scc}w)u This fallback implementation is used when the primary sample_tokens function generates an empty token list. This is mostly relevant for languages like mandarin where none latin-based punctuation is used e.g.: 。 ) translate PUNCT_TBLsplitisalnumr/)r0r1 clean_textr.r$s rsample_tokens_fallbackr8+sJ&&y1J#))+ ;Aqyy{a ;F ;  00ctjt|jdj d}t |Dcgc]}|d|zzrdndc}Scc}w)zFToken's contribution to a Simhash vector, cached across all instances.rAbigr r()int from_bytesrrDrEr*)r>r1 token_hashr-s r_vector_to_addrNOsS A F M M OQVWJ8=f F1qAv&AB . FF Fs A!c eZdZdZddgZ ddedededzddfdZdedefd Zdefd Z d ededzfd Z d eezdzdedzfd Z dddefdZ ddde fdZy)SimhashzAImplement a basic Charikar hashing approach of string similarity.hashr1Nr0 existing_hashrcd||_|j|xs|j||_y)z&Store length and existing or new hash.N)r1validate create_hashrQ)selfr0r1rRs r__init__zSimhash.__init__[s+ MM-0QD4D4D[4Q rc dg|jzt||jD]0}tttt ||j2t fdt|jDS)zCalculates a Charikar simhash. References used: https://github.com/vilda/shash/ https://github.com/sean-public/python-hashes/blob/master/hashes/simhash.py Optimized for Python by @adbar. rc3:K|]}|dk\s d|zyw)rr N).0r-vectors r z&Simhash.create_hash..psHaa16Hs  )r1r?listmaprrNsumr*)rVr0r>r\s @rrUzSimhash.create_hashesk t{{""; < PE#c6>%+MNOF PH5#5HHHrc2t|jddS)z3Convert the numerical hash to a hexadecimal string.r)N)hexrQrVs rto_hexzSimhash.to_hexrs499~ab!!r inputhashcF t|dS#ttf$rYywxYw)z2Convert the hexadecimal hash to a numerical value.N)rK TypeError ValueErrorrVres r _hash_to_intzSimhash._hash_to_intvs+ y"% %:&  s   ct|tr dtt|cxkrdkr|St|trA|j r dt|cxkrdkr t|S|j |Sy)z9Validate the input hash and return it, or None otherwise.N) isinstancerKr+strisdigitrkrjs rrTzSimhash.validate}sy i %"C N0C*Ir*I +J i %  "rS^'Ar'A9~%(B$$Y/ /r other_hashcFt|j|jz S)zJReturn distance between two hashes of equal length using the XOR operator.)BIN_COUNT_FUNCrQrVrrs rhamming_distancezSimhash.hamming_distancesdii*//9::rcX|j|j|z |jz S)zjCalculate how similar this hash is from another simhash. Returns a float from 0.0 to 1.0. )r1rvrus r similarityzSimhash.similaritys' d33J??4;;NNr)r@N)__name__ __module__ __qualname____doc__ __slots__rprKrWrUrdrkrTrvfloatrxrZrrrPrPVsG"I$( RRRTz R  R Is Is I""ccDj #)d"2 sTz ;9;;OYO5OrrPcontentc4t|jS)zACalculate a simhash hex value for meaningful bits of the content.)rPrd)rs rcontent_fingerprintrs 7  " " $$r)rr r)c^eZdZdZd deddfdZdedefdZdedefd Zde d eddfd Z dd Z y)LRUCachea Pure-Python Least Recently Used (LRU) cache using a circular doubly linked list Adapted from CPython functools.py lru_cache decorator implementation https://github.com/python/cpython/blob/3.9/Lib/functools.py#L524 First adapted by https://github.com/vbarbaresi rrNct|_||_i|_g|_|j|jddg|jddd|_y)NF)rlockrcacherootfull)rVrs rrWzLRUCache.__init__sEG  +- !  499dD9 !  rlinkc|\}}}}||c|t<|t<|jt}|x|t<|jt<||t<|j|t<|S)N)NEXTPREVr)rVr link_prev link_next_keyresultlasts r _move_linkzLRUCache._move_linksb-1* 9dF+4i( $4yy'++T TYYt_T YYT  rkeyc|j5|jj|}|r|j|cdddS dddy#1swYyxYw)zgTests if the key that is asked for is in the cache and retrieve its value from the linked list.Nr()rrgetr)rVrrs rrz LRUCache.getsSYY -::>>#&Dt, - - -  -s .AAvaluec|j5|jj|}|r)|j|||j|t<n|j r|j }||c|t<|t<|t|_|j t}dx|j t<|j t<|j|=||j|<nu|j t}||j ||g}|x|t<x|j t<|j|<t|j|jk\|_dddy#1swYyxYw)z Stores a given key in the cache.N) rrrrRESULTrrKEYrrr+r)rVrrroldrootoldkeyrs rputz LRUCache.puts*YY @::>>#&D%*/ 3'99"iiG471GCL'&/!( DI!YYs^F9==DIIcNTYYv%6 6*'.DJJsO 99T?D $))S%8DEIIDJI44::c?!$DJJ4<< ?DIA @ @ @s E E!!E*c|j5|jj|j|jddg|jddd|_dddy#1swYyxYw)zDelete all cache content.NF)rrclearrrrcs rrzLRUCache.clearsS YY  JJ    IItyy$=DIIaLDI   s A AA())rN) rzr{r|r}rKrWrrrrprrrZrrrrsd  d ssss#@s#@3#@4#@JrrrFcrtj|}|dk7r|dznd}tj||y)zImplement LRU cache.r(r N)LRU_TESTrr)rFcachevalrs r put_in_cachers/||J'H$NHqLE LLU#relementoptionsctdj|j}t||jkDr>t j |}||jkDrt j||dzyt|y)z(Check for duplicate text with LRU cache.rr TF) r rCitertextr+min_duplcheck_sizerrmax_repetitionsrr)rrrFrs rduplicate_testrslchhw//123J :333<< + g-- - LLX\ 2 r)g?)ry))6r}rer; unicodedatadifflibr functoolsrhashlibroperatorr threadingrtypingr lxml.etreer settingsr r utilsr compiler getattrrKrtrp maketransr*categorychrr4rboolr#r^r/r8r?bytesrGrNrPrrrrrrrrr)r-s0rrs@ #)"**/0k+FG MM5?ea>Rk>R>RSVWXSY>Z[\>]ad>d1c6e f  4MM#M%MRVMM$s)CDI 1 1S 1$s) 1sCc"E3EEUE 5G#GsGtCyGG :O:Oz%%% %dCPPf H %$S$T$ H y T Gfs 9#EE