AHj% d Z ddlZddlZddlmZ ddlmZ ddlmZ ddlm Z ddl
m Z m Z m
Z
mZmZmZ dd lmZ dd
lmZmZ dd lmZmZ ej2 e Z ej8 d Z ej8 d
ej< Z ej8 d Z ej8 d Z! ej8 d Z" ej8 d Z# ej8 d Z$ ej8 d Z% ej8 d Z&g dZ' G d d Z(dddefde)de)dz d e*d e+d e,d e-e) f d Z.de)d!e)dz d e*fd"Z/d#e)d e-e) fd$Z0d%e)dz d#e)d e-e) fd&Z1y)'z#
Deriving link info from sitemaps.
N)Callable)islice)Pattern)sleep) clean_urlextract_domain filter_urlsfix_relative_urls get_hostinfo lang_filter )is_similar_domain) fetch_url is_live_page) MAX_LINKSMAX_SITEMAPS_SEENz.(?:)?zzhref=["\'](.+?)["\']zg(?:blogger|blogpost|ghost|hubspot|livejournal|medium|typepad|squarespace|tumblr|weebly|wix|wordpress)\.z ^.{0,5}<\?xml| !1!128b
d&&'r' linkc L || j k( ryt | j | }t || j xs d}|rt || j syt
|d }|t j d| y| j sMt j | s8t | j | s"t j d| j | yt j | r | j j# | y| j$ j# | y)z^Examine a link and determine if it's valid and if it leads to
a sitemap or a web page.Nr" T)fastzcouldn't extract domain: %sz-link discarded, diverging domain names: %s %s)r r
r r r r r r) errorr WHITELISTED_PLATFORMSsearchr r warningDETECT_SITEMAP_LINKr appendr )r$ r. newdomains r% handle_linkzSitemapObject.handle_linkX s 4## # 5t / /06B;tT-=-=> "4d3 LL6 = }}%:%A%A)%LUfgkgrgrt}U~ NNJDKKYb c % %d + $ $T * II T "r' regexindexhandlerc fdt |j | j t D D ]
} || t j dt
| j t
| j | j y)zJExtract links from the content using pre-defined regex, index and handler.c 3 ( K | ] }| y w)N ).0mr: s r% z.SitemapObject.extract_links..v s X1ahXs z%%s sitemaps and %s links found for %sN)
r finditerr r r) r* lenr r r )r$ r9 r: r; matchs ` r%
extract_linkszSitemapObject.extract_linkst sc Xu~~dll/KY(WX E EN 3 !! " N
r' c d j vryt j d j dt j dt
ddf fd } j
t d| y) z7Extract links corresponding to a given target language.z hreflang=Nzhreflang=[\"'](z.*?|x-default)[\"']attrsr c j | r-t j | }|rj |d yyy)z!Examine language code attributes.r
N)r3 HREFLANG_REGEXr8 )rG
lang_match
lang_regexr$ s r% handle_lang_linkzASitemapObject.extract_sitemap_langlinks..handle_lang_link s@ ' + 2 25 9
$$Z]3 (r' r )r recompiler DOTALLstrrE XHTML_REGEX)r$ rL rK s ` @r% extract_sitemap_langlinksz'SitemapObject.extract_sitemap_langlinks sb dll * ZZ?43C3C2DDW XZ\ZcZcd
4C 4D 4
;+; r' r% r r 1 s I
I" #'
"" " 3i "
4Z "
"
"$( # # #8
73<
hPSuVZ{F[
`d
= <%r' r Fg @urlr r
sleep_time max_sitemapsr c t | \ }}|t j d| g S t | st j d| g S d}| j d r| g}n g }t | t | dz kD r| }t
||||| } | j s+t | xs t D
cg c] }
| d|
c}
| _ | j rt | j |k r| j j | _ | j | j | j D cg c] } | | j vs| c} | _ t | j |k r t | | j rt | j |k r|rt! | j" | | _ t j% dt | j" | | j" S c c}
w c c} w )ax Look for sitemaps for the given URL and gather links.
Args:
url: Webpage or sitemap URL as string.
Triggers URL-based filter if the webpage isn't a homepage.
target_lang: Define a language to filter URLs based on heuristics
(two-letter string, ISO 639-1 format).
external: Similar hosts only or external URLs
(boolean, defaults to False).
sleep_time: Wait between requests on the same website.
max_sitemaps: Maximum number of sitemaps to process.
Returns:
The extracted links as a list (sorted list of unique links).
Nzinvalid URL: %sz*base URL unreachable, dropping sitemap: %s)z.gzr z.xml /z %s sitemap links found for %s)r r) r4 r endswithrC r r find_robots_sitemapsGUESSESr popr r- r[ r r r r* ) rd r r re rf
domainnamebaseurl urlfilter sitemapurlsr gss r% sitemap_searchrt s . 's+J(#. CSI I
||./e s8c'lQ& &IGZk8TG 3G < c[b@cVWG9AaSAQ@c
3w||#4|#C % 2 2 6 6 8
+2+?+? Ya1GLLCX Y w|| | + *
3w||#4|#C "7<<;
LL0#gll2CZP <<# Ad Zs 1G38G8 G8contentsc |yt j d| } t j | r%t |t
r*t j | rd|dd j v rt j d| yy)zLCheck if the sitemap corresponds to an expected format,
i.e. TXT or XML.NFr" z r' r% r s $ - . 2 8 $
RZZI
J
bjj+RYY7 34"
n => bjj!?@ rzz./ bjj ' BJJz* q% q%l #) ? ?t? ? ?
?
#Y
?Dc S4Z D *7# 7$s) 7sTz C DI r'