/usr/share/cagefs-skeleton/usr/local/lib/python3.9/site-packages/pip/_internal/index/__pycache__
a ���h9? � @ s� d Z ddlmZ ddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlmZmZmZ ddlmZ ddlmZ ddlmZ ddlmZmZmZ dd l m Z dd l m Z dd l m!Z!m"Z" dd l#m$Z$ dd l%m&Z& ddl'm(Z( ddl)m*Z* ddl+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4m5Z5m6Z6 e �7e8�Z9ee:e:f Z;ddd�dd�Z<G dd � d e=�Z>d d d �d d!�Z?G d"d#� d#e=�Z@dd$d d%�d&d'�ZAdd$d d%�d(d)�ZBd*dd+�d,d-�ZCG d.d/� d/�ZDG d0d1� d1e�ZEd1d1d2�d3d4�ZFeFd5d6d7�d8d9��ZGed:d;�G d<d5� d5��ZHG d=d>� d>e�ZIdQd?d@dAd dB�dCdD�ZJdRd dEd5dF�dGdH�ZKd?d$dIdJ�dKdL�ZLG dMdN� dNe�ZMG dOdP� dP�ZNdS )SzO The main purpose of this module is to expose LinkCollector.collect_sources(). � )� annotationsN)�Iterable�MutableMapping�Sequence)� dataclass)� HTMLParser)�Values)�Callable� NamedTuple�Protocol)�requests)�Response)� RetryError�SSLError)�NetworkConnectionError)�Link)� SearchScope)� PipSession)�raise_for_status)�is_archive_file��redact_auth_from_url)�vcs� )�CandidatesFromPage� LinkSource� build_source�str� str | None��url�returnc C s6 t jD ]*}| �� �|�r| t|� dv r| S qdS )zgLook for VCS schemes in the URL. Returns the matched VCS scheme, or None if there's no match. z+:N)r �schemes�lower� startswith�len)r �scheme� r' �G/usr/local/lib/python3.9/site-packages/pip/_internal/index/collector.py�_match_vcs_scheme. s r) c s& e Zd Zdddd�� fdd� Z� ZS )�_NotAPIContentr �None)� content_type� request_descr! c s t � �||� || _|| _d S �N)�super�__init__r, r- )�selfr, r- �� __class__r' r( r0 : s z_NotAPIContent.__init__)�__name__� __module__� __qualname__r0 � __classcell__r' r' r2 r( r* 9 s r* r r+ )�responser! c C s6 | j �dd�}|�� }|�d�r$dS t|| jj��dS )z� Check the Content-Type header to ensure the response contains a Simple API Response. Raises `_NotAPIContent` if the content type is not a valid content-type. � Content-Type�Unknown)z text/htmlz#application/vnd.pypi.simple.v1+html�#application/vnd.pypi.simple.v1+jsonN)�headers�getr# r$ r* �request�method)r8 r, �content_type_lr' r' r( �_ensure_api_header@ s �rA c @ s e Zd ZdS )�_NotHTTPN)r4 r5 r6 r' r' r' r( rB V s rB r )r �sessionr! c C sF t j�| �\}}}}}|dvr$t� �|j| dd�}t|� t|� dS )z� Send a HEAD request to the URL, and ensure the response contains a simple API Response. Raises `_NotHTTP` if the URL is not available for a HEAD request, or `_NotAPIContent` if the content type is not a valid content type. > �http�httpsT)�allow_redirectsN)�urllib�parse�urlsplitrB �headr rA )r rC r& �netloc�path�query�fragment�respr' r' r( �_ensure_api_responseZ s rP c C sx t t| �j�rt| |d� t�dt| �� |j| d�g d��dd�d�}t |� t |� t�dt| �|j �d d �� |S ) aY Access an Simple API response with GET, and return the response. This consists of three parts: 1. If the URL looks suspiciously like an archive, send a HEAD first to check the Content-Type is HTML or Simple API, to avoid downloading a large file. Raise `_NotHTTP` if the content type cannot be determined, or `_NotAPIContent` if it is not HTML or a Simple API. 2. Actually perform the request. Raise HTTP exceptions on network failures. 3. Check the Content-Type header to make sure we got a Simple API response, and raise `_NotAPIContent` otherwise. �rC zGetting page %sz, )r; z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)�Acceptz Cache-Control)r< zFetched page %s as %sr9 r: ) r r �filenamerP �logger�debugr r= �joinr rA r< )r rC rO r' r' r( �_get_simple_responsel s& ��� �rW �ResponseHeaders)r< r! c C s< | r8d| v r8t j�� }| d |d< |�d�}|r8t|�S dS )z=Determine if we have any encoding information in our headers.r9 z content-type�charsetN)�email�message�Message� get_paramr )r< �mrY r' r' r( �_get_encoding_from_headers� s r_ c @ s: e Zd Zddd�dd�Zddd�d d �Zd d �d d�ZdS )�CacheablePageContent� IndexContentr+ ��pager! c C s |j s J �|| _d S r. )�cache_link_parsingrc �r1 rc r' r' r( r0 � s z CacheablePageContent.__init__�object�bool)�otherr! c C s t |t| ��o | jj|jjkS r. )� isinstance�typerc r )r1 rh r' r' r( �__eq__� s zCacheablePageContent.__eq__�int�r! c C s t | jj�S r. )�hashrc r �r1 r' r' r( �__hash__� s z CacheablePageContent.__hash__N)r4 r5 r6 r0 rk rp r' r' r' r( r` � s r` c @ s e Zd Zddd�dd�ZdS )� ParseLinksra �Iterable[Link]rb c C s d S r. r' re r' r' r( �__call__� � zParseLinks.__call__N)r4 r5 r6 rs r' r' r' r( rq � s rq )�fnr! c s> t jddd�� fdd� ��t �� �ddd�� �fdd � �}|S ) z� Given a function that parses an Iterable[Link] from an IndexContent, cache the function's result (keyed by CacheablePageContent), unless the IndexContent `page` has `page.cache_link_parsing == False`. r` z list[Link])�cacheable_pager! c s t � | j��S r. )�listrc )rv )ru r' r( �wrapper� s z*with_cached_index_content.<locals>.wrapperra rb c s | j r�t| ��S t� | ��S r. )rd r` rw )rc �ru rx r' r( �wrapper_wrapper� s z2with_cached_index_content.<locals>.wrapper_wrapper)� functools�cache�wraps)ru rz r' ry r( �with_cached_index_content� s r~ ra rr rb c c s� | j �� }|�d�rTt�| j�}|�dg �D ]"}t�|| j �}|du rHq,|V q,dS t | j �}| j pfd}|� | j� |�� | j }|jp�|}|jD ]$} tj| ||d�}|du r�q�|V q�dS )z\ Parse a Simple API's Index Content, and yield its anchor elements as Link objects. r; �filesNzutf-8)�page_url�base_url)r, r# r$ �json�loads�contentr= r � from_jsonr �HTMLLinkParser�encoding�feed�decoder� �anchorsZ from_element) rc r@ �data�file�link�parserr� r r� �anchorr' r' r( � parse_links� s& r� T)�frozenc @ sL e Zd ZU dZded< ded< ded< ded< d Zd ed < dd �d d�ZdS )ra a� Represents one response (or page), along with its URL. :param encoding: the encoding to decode the given content. :param url: the URL from which the HTML was downloaded. :param cache_link_parsing: whether links parsed from this page's url should be cached. PyPI index urls should have this set to False, for example. �bytesr� r r, r r� r Trg rd rm c C s t | j�S r. )r r ro r' r' r( �__str__ s zIndexContent.__str__N)r4 r5 r6 �__doc__�__annotations__rd r� r' r' r' r( ra � s c sJ e Zd ZdZddd�� fdd� Zdddd�d d �Zdd d �d d�Z� ZS )r� zf HTMLParser that keeps the first base HREF and a list of all anchor elements' attributes. r r+ r c s$ t � jdd� || _d | _g | _d S )NT)�convert_charrefs)r/ r0 r r� r� )r1 r r2 r' r( r0 s zHTMLLinkParser.__init__z list[tuple[str, str | None]])�tag�attrsr! c C sH |dkr,| j d u r,| �|�}|d urD|| _ n|dkrD| j�t|�� d S )N�base�a)r� �get_hrefr� �append�dict)r1 r� r� �hrefr' r' r( �handle_starttag s z HTMLLinkParser.handle_starttagr )r� r! c C s" |D ]\}}|dkr| S qd S )Nr� r' )r1 r� �name�valuer' r' r( r� ! s zHTMLLinkParser.get_href)r4 r5 r6 r� r0 r� r� r7 r' r' r2 r( r� s r� r zstr | ExceptionzCallable[..., None] | None)r� �reason�methr! c C s |d u rt j}|d| |� d S )Nz%Could not fetch URL %s: %s - skipping)rT rU )r� r� r� r' r' r( �_handle_get_simple_fail( s r� rg )r8 rd r! c C s& t | j�}t| j| jd || j|d�S )Nr9 )r� r rd )r_ r< ra r� r )r8 rd r� r' r' r( �_make_index_content2 s �r� �IndexContent | None)r� rC r! c C s | j �dd�d }t|�}|r0t�d|| � d S tj�|�\}}}}}}|dkr�tj � tj � |��r�|� d�sv|d7 }tj�|d�}t�d|� zt||d �}W �nL ty� t�d | � Y �n< ty� } z"t�d | |j|j� W Y d }~�n d }~0 t�y( } zt| |� W Y d }~n�d }~0 t�yV } zt| |� W Y d }~n�d }~0 t�y� } z,d } | t|�7 } t| | tjd � W Y d }~nld }~0 tj �y� } z t| d|� �� W Y d }~n6d }~0 tj �y� t| d� Y n0 t || j d�S d S )N�#r r zICannot look at %s URL %s because it does not support lookup as web pages.r� �/z index.htmlz# file: URL is directory, getting %srQ z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.z�Skipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r� zconnection error: z timed out)rd ) r �splitr) rT �warningrG rH �urlparse�osrL �isdirr> � url2pathname�endswith�urljoinrU rW rB r* r- r, r r� r r r �infor �ConnectionError�Timeoutr� rd ) r� rC r � vcs_schemer& �_rL rO �excr� r' r' r( �_get_index_content? sV � � � $$r� c @ s e Zd ZU ded< ded< dS )�CollectedSourceszSequence[LinkSource | None]� find_links� index_urlsN)r4 r5 r6 r� r' r' r' r( r� | s r� c @ sp e Zd ZdZdddd�dd�Zed dd d d d �d d ��Zedd�dd��Zddd�dd�Z dddd�dd �Z d S ) � LinkCollectorz� Responsible for collecting Link objects from all configured locations, making network requests as needed. The class's main method is its collect_sources() method. r r r+ )rC � search_scoper! c C s || _ || _d S r. )r� rC )r1 rC r� r' r' r( r0 � s zLinkCollector.__init__Fr rg )rC �options�suppress_no_indexr! c C sd |j g|j }|jr8|s8t�dd�dd� |D ��� g }|jp@g }tj|||jd�}t ||d�}|S )z� :param session: The Session to use to make requests. :param suppress_no_index: Whether to ignore the --no-index option when constructing the SearchScope object. zIgnoring indexes: %s�,c s s | ]}t |�V qd S r. r )�.0r r' r' r( � <genexpr>� rt z'LinkCollector.create.<locals>.<genexpr>)r� r� �no_index)rC r� ) � index_url�extra_index_urlsr� rT rU rV r� r �creater� )�clsrC r� r� r� r� r� �link_collectorr' r' r( r� � s$ � ��zLinkCollector.createz list[str]rm c C s | j jS r. )r� r� ro r' r' r( r� � s zLinkCollector.find_linksr r� )�locationr! c C s t || jd�S )z> Fetch an HTML page containing package links. rQ )r� rC )r1 r� r' r' r( �fetch_response� s z LinkCollector.fetch_responser r r� )� project_name�candidates_from_pager! c s� t �� ��fdd��j���D ���� }t �� ��fdd��jD ���� }t�tj �r�dd� t � ||�D �}t |�� d�� d�g| }t� d�|�� tt|�t|�d �S ) Nc 3 s& | ] }t |� �jjd d �d�V qdS )F�r� �page_validator� expand_dirrd r� N�r rC �is_secure_origin�r� �loc�r� r� r1 r' r( r� � s ��z0LinkCollector.collect_sources.<locals>.<genexpr>c 3 s& | ] }t |� �jjd d �d�V qdS )Tr� Nr� r� r� r' r( r� � s ��c S s* g | ]"}|d ur|j d urd|j � ��qS )Nz* )r� )r� �sr' r' r( � <listcomp>� s �z1LinkCollector.collect_sources.<locals>.<listcomp>z' location(s) to search for versions of �:� )r� r� )� collections� OrderedDictr� �get_index_urls_locations�valuesr� rT � isEnabledFor�logging�DEBUG� itertools�chainr% rU rV r� rw )r1 r� r� �index_url_sources�find_links_sources�linesr' r� r( �collect_sources� s* � � � ����z LinkCollector.collect_sourcesN)F) r4 r5 r6 r� r0 � classmethodr� �propertyr� r� r� r' r' r' r( r� � s �!r� )N)T)Or� � __future__r r� � email.messagerZ r{ r� r� r� r� � urllib.parserG �urllib.request�collections.abcr r r Z dataclassesr � html.parserr �optparser �typingr r r � pip._vendorr Zpip._vendor.requestsr Z pip._vendor.requests.exceptionsr r �pip._internal.exceptionsr �pip._internal.models.linkr �!pip._internal.models.search_scoper � pip._internal.network.sessionr �pip._internal.network.utilsr � pip._internal.utils.filetypesr �pip._internal.utils.miscr �pip._internal.vcsr �sourcesr r r � getLoggerr4 rT r rX r) � Exceptionr* rA rB rP rW r_ r` rq r~ r� ra r� r� r� r� r� r� r' r' r' r( �<module> sf ? � � =
.
Edit
..
Edit
__init__.cpython-39.pyc
Edit
collector.cpython-39.pyc
Edit
package_finder.cpython-39.pyc
Edit
sources.cpython-39.pyc
Edit