U d͕@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlmZddlZddlmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+dd l,m-Z-e.d Z/e.d ej0Z1e.d Z2e.d ej0j3Z4d5Z6ddddgZ7dZ8dZ9e9j:dj:ej;edZddZ?dAddZ@dBddZAdCdd ZBdedfd!dZCd"d#ZDe.d$ej0ZEeDd%d&ZFGd'd(d(ZGGd)d*d*eGZHGd+ddeZIe.d,jJZKd-d.ZLd/d0ZMdDd1d2ZNd3d4ZOGd5d6d6ZPGd7d8d8e jQZRejSjTfd9d:ZUd;d<ZVeNe8eUZUd=d>ZWd?d@ZXdS)Ez$PyPI and direct package downloading.Nwraps) CHECKOUT_DIST Distribution BINARY_DISTnormalize_path SOURCE_DIST Environmentfind_distributions safe_name safe_version to_filename Requirement DEVELOP_DISTEGG_DIST parse_version)logDistutilsError) translate)Wheelunique_everseenz^egg=([-A-Za-z0-9_.+!]+)$zhref\s*=\s*['"]?([^'"> ]+)z([^<]+)\n\s+\(md5\)z([-+.a-z0-9]{2,}):z.tar.gz .tar.bz2 .tar .zip .tgz PackageIndexdistros_for_urlparse_bdist_wininstinterpret_distro_namez}ztd|f|W5d}~XYnXdS)Nz1Not a URL, existing file, or requirement spec: %r)rparse ValueErrorr)specer#@/tmp/pip-install-xwugzb3_/setuptools/setuptools/package_index.pyparse_requirement_argDs r%cCs|}d\}}}|dr|dr8|dd}d}nn|ddrb|d d }|dd}d}nD|d r~|dd }d }n(|ddr|d d }|dd}d }|||fS)z=Return (base,pyversion) or (None,None) for possible .exe name)NNN.exez .win32.exeNiwin32z .win32-pyiiz.win-amd64.exeiz win-amd64z .win-amd64-pyi)lowerendswith startswith)namer)basepy_verplatr#r#r$rMs$            c Csxtj|}|\}}}}}}tj|dd}|dkrX|dkrXtj|dd}d|krp|dd\}}||fS)N/zsourceforge.netdownload#)urllibrurlparseunquotesplit) urlpartsschemeserverpath parametersqueryfragmentr-r#r#r$egg_info_for_urles rBccs\t|\}}t|||D] }|Vq|rXt|}|rXt||d|tdD] }|VqLdS)zEYield egg or source distribution objects that might be found at a URLr5) precedenceN)rBdistros_for_location EGG_FRAGMENTmatchrgroupr)r:metadatar-rAdistrFr#r#r$rps   cCs|dr|dd}|dr8d|kr8t|||gS|drxd|krxt|}|s^gSt||j|jtddgS|d rt|\}}}|dk rt ||||t |St D]0}||r|dt | }t |||SqgS) z:Yield egg or source distribution objects based on basename.egg.zipNr(z.egg-z.whlr5)location project_nameversionrCr&) r*r from_locationr is_compatiblerMrNrrrr EXTENSIONSlen)rLbasenamerHwheelZwin_baser.platformextr#r#r$rD~s>    rDcCstt|tj||S)zEYield possible egg or source distribution objects based on a filename)rDrosr>rS)filenamerHr#r#r$distros_for_filenames  rYc cs|d}|s,tdd|ddDr,dStt|D]}||ddr8q^q8t|}t||d|d|d||d|||dVdS)zGenerate the interpretation of a source distro name Note: if `location` is a filesystem filename, you should call ``pkg_resources.normalize_path()`` on it before passing it to this routine! rKcss|]}td|VqdS)z py\d\.\d$N)rerF).0pr#r#r$ sz(interpret_distro_name..Nr5) py_versionrCrU)r9anyrangerRisdigitrjoin)rLrSrHr_rCrUr;r\r#r#r$rs cstfdd}|S)zs Wrap a function returning an iterable such that the resulting iterable only ever yields unique items. cst||SNr)argskwargsfuncr#r$wrapperszunique_values..wrapperr)rhrir#rgr$ unique_valuessrjz3<([^>]*\srel\s{0,10}=\s{0,10}['"]?([^'" >]+)[^>]*)>c cst|D]d}|\}}tttj|d}d|ksDd|kr t |D]}t j |t |dVqNq dD]@}||}|dkrtt ||}|rtt j |t |dVqtdS)zEFind rel="homepage" and rel="download" links in `page`, yielding URLs,Zhomepager2r5)z Home PagezDownload URLr1N)RELfinditergroupssetmapstrstripr)r9HREFr6rurljoin htmldecoderGfindsearch)r:pagerFtagrelZrelsposr#r#r$find_external_linkss   r|c@s(eZdZdZddZddZddZdS) ContentCheckerzP A null content checker that defines the interface for checking content cCsdS)z3 Feed a block of data to the hash. Nr#selfblockr#r#r$feedszContentChecker.feedcCsdS)zC Check the hash. Return False if validation fails. Tr#rr#r#r$is_validszContentChecker.is_validcCsdS)zu Call reporter with information about the checker (hash name) substituted into the template. Nr#)rreportertemplater#r#r$reportszContentChecker.reportN)__name__ __module__ __qualname____doc__rrrr#r#r#r$r}sr}c@sBeZdZedZddZeddZddZ dd Z d d Z d S) HashCheckerzK(?Psha1|sha224|sha384|sha256|sha512|md5)=(?P[a-f0-9]+)cCs||_t||_||_dSrd) hash_namehashlibnewhashexpected)rrrr#r#r$__init__s zHashChecker.__init__cCs>tj|d}|stS|j|}|s0tS|f|S)z5Construct a (possibly null) ContentChecker from a URLr1)r6rr7r}patternrw groupdict)clsr:rArFr#r#r$from_urls zHashChecker.from_urlcCs|j|dSrd)rupdater~r#r#r$rszHashChecker.feedcCs|j|jkSrd)r hexdigestrrr#r#r$r"szHashChecker.is_validcCs||j}||Srd)r)rrrmsgr#r#r$r%s zHashChecker.reportN) rrrrZcompilerr classmethodrrrrr#r#r#r$r s rcsTeZdZdZdNfdd Zfdd ZdOd d ZdPd dZdQddZddZ ddZ ddZ ddZ ddZ dRddZddZdSfdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*ZdTd+d,ZdUd-d.Zd/d0Zd1Zd2d3Zd4d5ZdVd6d7Zd8d9Zd:d;Zdd?Z d@dAZ!e"dWdBdCZ#dDdEZ$dFdGZ%dHdIZ&dJdKZ'dLdMZ(Z)S)Xrz;A distribution index that scans web pages for download URLshttps://pypi.org/simple/*NTcshtj|||dd|d |_i|_i|_i|_td t t |j |_ g|_tjj|_dS)Nr0|)superrr* index_url scanned_urls fetched_urls package_pagesrZrrcrprrFallowsto_scanr6requesturlopenopener)rrhostsZ ca_bundleZ verify_sslrekw __class__r#r$r-s zPackageIndex.__init__cs2zt|jWntk r$YdSXt|Srd)rrN Exceptionradd)rrIrr#r$r?s zPackageIndex.addFc Cs||jkr|sdSd|j|<t|s2||dStt|}|r\||sPdS|d||sn|rn||jkrtt|j |dS||sd|j|<dS| d|d|j|<d}| |||}|dkrdSt |t jjr|jdkr| d|jd|j|j<d|jd d kr(|dS|j}|}t |tsvt |t jjrXd }n|jd phd }||d }|t|D](} t j|t| d} |!| q|"|j#rt$|dddkr|%||}dS)zexistswarnisdirrealpathlistdirrrcrYrrrpr)rfnnestedr>itemrr#r#r$r}s    zPackageIndex.process_filenamecCsbt|}|o|ddk}|s8|tj|drentryr#r#r$r]s   z.PackageIndex.scan_egg_links..)filterrWr>rr itertoolsstarmap scan_egg_link)r search_pathdirsZ egg_linksr#r#r$scan_egg_linkss zPackageIndex.scan_egg_linksc Csttj||}ttdttj|}W5QRXt |dkrDdS|\}}t tj||D](}tjj|f||_ t |_ ||q^dS)Nr^)openrWr>rcrrrprqrrrRr rLrrCr)rr>rZ raw_lineslinesZegg_pathZ setup_pathrIr#r#r$rs  zPackageIndex.scan_egg_linkcCsd}||js|Stttjj|t|jdd}t|dksRd|dkrV|St |d}t |d}d|j | i|<t|t|fS)N)NNr0r^r4r5rT)r+rrrpr6rr8rRr9r r r setdefaultr)r )rrZNO_MATCH_SENTINELr;pkgverr#r#r$_scans &  zPackageIndex._scanc Cst|D]<}z"|tj|t|dWq tk rDYq Xq ||\}}|s^dSt ||D]H}t |\}}| dr|s|r|d||f7}n | || |qhtdd|S)z#Process the contents of a PyPI pager5r.pyz #egg=%s-%scSsd|dddS)Nz%sr5r^)rG)mr#r#r$z,PackageIndex.process_index..)rsrmrr6rrtrurGr r|rBr*need_version_infoscan_urlPYPI_MD5sub) rr:rxrFrrnew_urlr-fragr#r#r$rs&"   zPackageIndex.process_indexcCs|d|dS)NzPPage at %s links to .py file(s) without version info; an index scan is required.)scan_allrr:r#r#r$rszPackageIndex.need_version_infocGs:|j|jkr*|r |j|f||d||jdS)Nz6Scanning index of all packages (this may take a while))rrrrrrrrer#r#r$rs   zPackageIndex.scan_allcCsz||j|jd|j|js:||j|jd|j|jsR||t|j|jdD]}||qfdS)Nr0r#) rr unsafe_namerrkeyrMnot_found_in_indexr)r requirementr:r#r#r$ find_packagess zPackageIndex.find_packagescsR|||||jD]"}||kr0|S|d||qtt|||S)Nz%s does not match %s)prescanrrrrrobtain)rr installerrIrr#r$rs zPackageIndex.obtaincCsL||jd||sH|t|td|jjtj |fdS)z- checker is a ContentChecker zValidating %%s checksum for %sz7%s validation failed for %s; possible download problem?N) rrrrrWunlinkrrr,r>rS)rcheckerrXtfpr#r#r$ check_hash s zPackageIndex.check_hashcCsN|D]D}|jdks0t|r0|ds0tt|r<||q|j|qdS)z;Add `urls` to the list that will be prescanned for searchesNfile:)rrr+rrrappend)rurlsr:r#r#r$add_find_linkss  zPackageIndex.add_find_linkscCs"|jrtt|j|jd|_dS)z7Scan urls scheduled for prescanning (e.g. --find-links)N)rrrprrr#r#r$r&szPackageIndex.prescancCs<||jr|jd}}n |jd}}|||j|dS)Nz#Couldn't retrieve index page for %rz3Couldn't find index page for %r (maybe misspelled?))rrrrr)rrmethrr#r#r$r,s  zPackageIndex.not_found_in_indexcCs~t|tsjt|}|rR||d||}t|\}}|drN||||}|Stj |rb|St |}t | ||ddS)aLocate and/or download `spec` to `tmpdir`, returning a local path `spec` may be a ``Requirement`` object, or a string containing a URL, an existing local filename, or a project/version requirement spec (i.e. the string form of a ``Requirement`` object). If it is the URL of a .py file with an unambiguous ``#egg=name-version`` tag (i.e., one that escapes ``-`` as ``_`` throughout), a trivial ``setup.py`` is automatically created alongside the downloaded file. If `spec` is a ``Requirement`` object or a string containing a project/version requirement spec, this method returns the location of a matching distribution (possibly after downloading it to `tmpdir`). If `spec` is a locally existing file or directory name, it is simply returned unchanged. If `spec` is a URL, it is downloaded to a subpath of `tmpdir`, and the local filename is returned. Various errors may be raised if a problem occurs during downloading. r5rrLN)rrr _download_urlrGrBr* gen_setuprWr>rr%rfetch_distribution)rr!tmpdirr<foundr-rAr#r#r$r27s    zPackageIndex.downloadc sd|id}d fdd }|rH|||}|s^|dk r^|||}|dkrjdk rx||}|dkr|s|||}|dkrdrdpd|nd||j|jd SdS) a|Obtain a distribution suitable for fulfilling `requirement` `requirement` must be a ``pkg_resources.Requirement`` instance. If necessary, or if the `force_scan` flag is set, the requirement is searched for in the (online) package index as well as the locally installed packages. If a distribution matching `requirement` is found, the returned distribution's ``location`` is the value you would have gotten from calling the ``download()`` method with the matching distribution's URL or filename. If no matching distribution is found, ``None`` is returned. If the `source` flag is set, only source distributions and source checkout links will be considered. Unless the `develop_ok` flag is set, development and system eggs (i.e., those using the ``.egg-info`` format) will be ignored. zSearching for %sNcs|dkr }||jD]v}|jtkrFsF|krd|d|<q||ko\|jtkp\ }|r|j}||_tj |jr|SqdS)Nz&Skipping development or system egg: %sr5) rrCrrrr2rLdownload_locationrWr>r)reqenvrItestloc develop_okrskippedsourcer r#r$rvws"z-PackageIndex.fetch_distribution..findz:No local packages or working download links found for %s%sza source distribution of rzBest match: %s)rL)N)rrrrrcloner ) rrr  force_scanrrZ local_indexrIrvr#rr$r Ys2         zPackageIndex.fetch_distributioncCs"|||||}|dk r|jSdS)a3Obtain a file suitable for fulfilling `requirement` DEPRECATED; use the ``fetch_distribution()`` method now instead. For backward compatibility, this routine is identical but returns the ``location`` of the downloaded distribution instead of a distribution object. N)r rL)rrr rrrIr#r#r$fetchszPackageIndex.fetchc Cst|}|r*ddt||ddDp,g}t|dkrtj|}tj||krtj ||}tj |r~tj ||st |||}ttj |dd2}|d|dj|djtj|dfW5QRX|S|rtd||fntd dS) NcSsg|]}|jr|qSr#)rN)r[dr#r#r$ sz*PackageIndex.gen_setup..r5zsetup.pywzIfrom setuptools import setup setup(name=%r, version=%r, py_modules=[%r]) rzCan't unambiguously interpret project/version identifier %r; any dashes in the name or version should be escaped using underscores. %rzpCan't process plain .py files without an '#egg=name-version' suffix to enable automatic setup script generation.)rErFrrGrRrWr>rSdirnamercrsamefileshutilcopy2rwriterMrNsplitextr) rrXrAr rFrrSdstrr#r#r$rsD     zPackageIndex.gen_setupi c Cs|d|d}zt|}||}t|tjjrJt d||j |j f|}d}|j }d}d|kr| d} ttt| }||||||t|dV} ||} | r|| | | |d7}||||||qqq|||| W5QRX|WS|r|XdS) NzDownloading %szCan't download %s: %s %srr1zcontent-lengthzContent-Lengthwbr5)rrrrrrr6rrrrr dl_blocksizeget_allmaxrpint reporthookrrrrr) rr:rXfprrblocknumbssizeZsizesrrr#r#r$ _download_tos:        zPackageIndex._download_tocCsdSrdr#)rr:rXr)Zblksizer+r#r#r$r' szPackageIndex.reporthookc Cs|drt|Szt||jWSttjjfk r}z>ddd|j D}|rb| ||nt d||f|W5d}~XYnt j jk r}z |WYSd}~XYnt j jk r}z,|r| ||jnt d||jf|W5d}~XYntjjk rP}z.|r,| ||jnt d||jf|W5d}~XYnTtjjtj fk r}z*|r| ||nt d||f|W5d}~XYnXdS)Nr cSsg|] }t|qSr#)rq)r[argr#r#r$rsz)PackageIndex.open_url..z%s %szDownload error for %s: %sz;%s returned a bad status line. The server might be down, %s)r+ local_openopen_with_authrr httpclient InvalidURLrcrerrr6rrURLErrorreason BadStatusLineline HTTPExceptionsocket)rr:warningvrr#r#r$rsB $ zPackageIndex.open_urlcCst|\}}|r0d|kr4|dddd}qnd}|drJ|dd}tj||}|dksj|d rv|||S|d ks|d r|||S|d r| ||S|d krt j t j |dS||d|||SdS)Nz...\_Z__downloaded__rJr(svnzsvn+gitzgit+zhg+rr^T)rBreplacer*rWr>rcr+ _download_svn _download_git _download_hgr6r url2pathnamerr7r_attempt_download)rr<r:r r,rArXr#r#r$r0s$        zPackageIndex._download_urlcCs||ddS)NT)rrr#r#r$rMszPackageIndex.scan_urlcCs6|||}d|ddkr.||||S|SdS)Nrrr)r,rr)_invalid_download_html)rr:rXrr#r#r$rFPs zPackageIndex._attempt_downloadcCst|td|dS)NzUnexpected HTML page found at )rWrr)rr:rrXr#r#r$rGWs z#PackageIndex._invalid_download_htmlcCstd|dS)Nz/Invalid config, SVN download is not supported: r)rr: _filenamer#r#r$rB[szPackageIndex._download_svncCsptj|\}}}}}|ddd}|ddd}d}d|krR|dd\}}tj||||df}||fS)N+r5r1r4r@r)r6rurlsplitr9rsplit urlunsplit)r: pop_prefixr<netlocr>r@rrevr#r#r$_vcs_split_rev_from_url^sz$PackageIndex._vcs_split_rev_from_urlcCsl|ddd}|j|dd\}}|d||td||f|dk rh|d|td ||f|S) Nr4r5rTrNzDoing git clone from %s to %szgit clone --quiet %s %szChecking out %szgit -C %s checkout --quiet %sr9rQrrWsystemrr:rXrPr#r#r$rCps zPackageIndex._download_gitcCsl|ddd}|j|dd\}}|d||td||f|dk rh|d|td ||f|S) Nr4r5rTrRzDoing hg clone from %s to %szhg clone --quiet %s %szUpdating to %szhg --cwd %s up -C -r %s -qrSrUr#r#r$rDs zPackageIndex._download_hgcGstj|f|dSrd)rrrr#r#r$rszPackageIndex.debugcGstj|f|dSrd)rrrr#r#r$rszPackageIndex.infocGstj|f|dSrd)rrrr#r#r$rszPackageIndex.warn)rrNT)F)F)F)N)N)FFFN)FF)N)F)*rrrrrrrrrrrrrrrrrrrrrr2r rrr#r,r'rrrrFrGrB staticmethodrQrCrDrrr __classcell__r#r#rr$r*s\ 5      & P .$ " z!&(#(\d+|x[\da-fA-F]+)|[\w.:-]+);?cCs|d}t|S)Nr)rGrunescape)rFwhatr#r#r$ decode_entitys rZcCs tt|S)a Decode HTML entities in the given text. >>> htmldecode( ... 'https://../package_name-0.1.2.tar.gz' ... '?tokena=A&tokenb=B">package_name-0.1.2.tar.gz') 'https://../package_name-0.1.2.tar.gz?tokena=A&tokenb=B">package_name-0.1.2.tar.gz' ) entity_subrZ)textr#r#r$rus rucsfdd}|S)Ncsfdd}|S)Nc s2t}tz||WSt|XdSrd)r9getdefaulttimeoutsetdefaulttimeout)rerfZ old_timeout)rhtimeoutr#r$_socket_timeouts  z@socket_timeout.._socket_timeout.._socket_timeoutr#)rhr`r_rgr$r`sz'socket_timeout.._socket_timeoutr#)r_r`r#rar$socket_timeouts rbcCs2tj|}|}t|}|}|ddS)a9 Encode auth from a URL suitable for an HTTP header. >>> str(_encode_auth('username%3Apassword')) 'dXNlcm5hbWU6cGFzc3dvcmQ=' Long auth strings should not cause a newline to be inserted. >>> long_auth = 'username:' + 'password'*10 >>> chr(10) in str(_encode_auth(long_auth)) False  r)r6rr8encodebase64 b64encoderrA)authZauth_sZ auth_bytesZ encoded_bytesencodedr#r#r$ _encode_auths  ric@s(eZdZdZddZddZddZdS) Credentialz: A username/password pair. Use like a namedtuple. cCs||_||_dSrdusernamepassword)rrlrmr#r#r$rszCredential.__init__ccs|jV|jVdSrdrkrr#r#r$__iter__szCredential.__iter__cCs dt|S)Nz%(username)s:%(password)s)varsrr#r#r$__str__szCredential.__str__N)rrrrrrnrpr#r#r#r$rjsrjcs8eZdZfddZeddZddZddZZS) PyPIConfigcsNtdddgd}t|tjtjdd}tj|rJ| |dS)z% Load from ~/.pypirc rlrm repositoryr~z.pypircN) dictfromkeysrrrWr>rc expanduserrr)rdefaultsrcrr#r$rs   zPyPIConfig.__init__cs&fddD}ttj|S)Ncs g|]}|dr|qS)rr)rrr)r[sectionrr#r$rsz2PyPIConfig.creds_by_repository..)sectionsrtrp_get_repo_cred)rZsections_with_repositoriesr#rr$creds_by_repositorys zPyPIConfig.creds_by_repositorycCs6||d}|t||d||dfS)Nrrrlrm)rrrrj)rryrepor#r#r$r{s zPyPIConfig._get_repo_credcCs*|jD]\}}||r |Sq dS)z If the URL indicated appears to be a repository defined in this config, return the credential for that repository. N)r|itemsr+)rr:rrcredr#r#r$find_credentials zPyPIConfig.find_credential) rrrrpropertyr|r{rrWr#r#rr$rqs   rqcCs<tj|}|\}}}}}}|dr2tjd|dkrHt|\} } nd} | st |} | rt | } | j |f} t j d | | rdt| } || ||||f} tj| }tj|}|d| n tj|}|dt||}| r8tj|j\}}}}}}||kr8|| kr8||||||f} tj| |_|S) z4Open a urllib2 request, handling HTTP authentication:znonnumeric port: '')r1httpsN*Authenticating as %s for %s (from .pypirc)zBasic Authorizationz User-Agent)r)r6rr7r*r1r2r3 _splituserrqrrqrlrrri urlunparserRequest add_header user_agentr:)r:rparsedr<rOr>paramsr@rrgaddressrrr;rrr(s2h2path2Zparam2Zquery2Zfrag2r#r#r$r0s8          r0cCs |d\}}}|r|nd|fS)zNsplituser('user[:passwd]@host[:port]') --> 'user[:passwd]', 'host[:port]'.rJN) rpartition)hostuserdelimr#r#r$rAsrcCs|Srdr#)r:r#r#r$ fix_sf_urlLsrc Cstj|\}}}}}}tj|}tj|r{name})r,zB{url}{files}rc)r:files)OK)rzPath not foundz Not foundrz text/html)r6rr7rrErWr>isfilerr*rrrcrrrformatioStringIOrr)r:r<r=r>paramr@rrXrrfilepathr(bodyrstatusmessagerZ body_streamr#r#r$r/Ps.        r/)N)N)N)r)YrsysrWrZrrr9rerr configparserr http.clientr1 urllib.parser6urllib.request urllib.error functoolsrr pkg_resourcesrrrrrr r r r r rrrr distutilsrdistutils.errorsrfnmatchrZsetuptools.wheelrZ setuptools.extern.more_itertoolsrrrEIrsrrFrr9rQ__all__Z_SOCKET_TIMEOUTZ_tmplr version_inforr%rrBrrDrYrrjrlr|r}rrrr[rZrurbrirjRawConfigParserrqrrr0rrr/r#r#r#r$s @          #  "  !|  '/