/usr/lib/python3.9/site-packages/future/backports/urllib/__pycache__
a �c� � @ s� d dl mZmZmZ d dlmZ d dlmZ d dlm Z m Z e e_ e e_ dgZ G dd� de�ZG dd� de�ZG d d � d e�Zd S ) � )�absolute_import�division�unicode_literals��str)�urllib)�parse�request�RobotFileParserc @ sZ e Zd ZdZddd�Zdd� Zdd� Zd d � Zd d � Zd d� Z dd� Z dd� Z dd� Z dS )r zs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. � c C s, g | _ d | _d| _d| _| �|� d| _d S )NFr )�entries� default_entry� disallow_all� allow_all�set_url� last_checked��self�url� r �G/usr/lib/python3.9/site-packages/future/backports/urllib/robotparser.py�__init__ s zRobotFileParser.__init__c C s | j S )z�Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r �r r r r �mtime& s zRobotFileParser.mtimec C s ddl }|� � | _dS )zYSets the time the robots.txt file was last fetched to the current time. r N)�timer )r r r r r �modified/ s zRobotFileParser.modifiedc C s& || _ tj�|�dd� \| _| _dS )z,Sets the URL referring to a robots.txt file.� � N)r r r �urlparse�host�pathr r r r r 7 s zRobotFileParser.set_urlc C s~ zt j�| j�}W nJ t jjy\ } z.|jdv r8d| _n|jdkrHd| _W Y d}~n&d}~0 0 |� � }| � |� d�� � � dS )z4Reads the robots.txt URL and feeds it to the parser.)i� i� Ti� Nzutf-8) r r Zurlopenr �errorZ HTTPError�coder r �readr �decode� splitlines)r �f�err�rawr r r r# < s zRobotFileParser.readc C s, d|j v r | jd u r(|| _n | j�|� d S �N�*)� useragentsr r �append)r �entryr r r � _add_entryI s zRobotFileParser._add_entryc C sn d}t � }|D �]D}|sH|dkr,t � }d}n |dkrH| �|� t � }d}|�d�}|dkrf|d|� }|�� }|stq|�dd�}t|�dkr|d �� �� |d< tj� |d �� �|d< |d dkr�|dkr�| �|� t � }|j � |d � d}q|d dk�r&|dk�rT|j � t |d d �� d}q|d d kr|dkr|j � t |d d �� d}q|dk�rj| �|� dS ) z�Parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. r r � �#N�:z user-agentZdisallowFZallowT)�Entryr. �find�strip�split�len�lowerr r �unquoter+ r, � rulelines�RuleLine)r �lines�stater- �line�ir r r r R sJ zRobotFileParser.parsec C s� | j r dS | jrdS tj�tj�|��}tj�dd|j|j|j |j f�}tj� |�}|s\d}| j D ] }|� |�rb|�|� S qb| jr�| j�|�S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr �/)r r r r r r8 � urlunparser �params�query�fragment�quoter � applies_to� allowancer )r � useragentr Z parsed_urlr- r r r � can_fetch� s" � zRobotFileParser.can_fetchc C s d� dd� | jD ��S )Nr c S s g | ]}t |�d �qS )� r )�.0r- r r r � <listcomp>� � z+RobotFileParser.__str__.<locals>.<listcomp>)�joinr r r r r �__str__� s zRobotFileParser.__str__N)r ) �__name__� __module__� __qualname__�__doc__r r r r r# r. r rH rN r r r r r s 3c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r: zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.c C s( |dkr|sd}t j�|�| _|| _d S )Nr T)r r rD r rF )r r rF r r r r � s zRuleLine.__init__c C s | j dkp|�| j �S r) )r � startswith)r �filenamer r r rE � s zRuleLine.applies_toc C s | j r dp dd | j S )NZAllowZDisallowz: )rF r r r r r rN � s zRuleLine.__str__N)rO rP rQ rR r rE rN r r r r r: � s r: c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd S ) r2 z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d S )N)r+ r9 r r r r r � s zEntry.__init__c C sH g }| j D ]}|�d|dg� q | jD ]}|�t|�dg� q&d�|�S )Nz User-agent: rI r )r+ �extendr9 r rM )r �ret�agentr= r r r rN � s z Entry.__str__c C sF |� d�d �� }| jD ](}|dkr* dS |�� }||v r dS qdS )z2check if this entry applies to the specified agentr? r r* TF)r5 r7 r+ )r rG rW r r r rE � s zEntry.applies_toc C s$ | j D ]}|�|�r|j S qdS )zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r9 rE rF )r rT r= r r r rF � s zEntry.allowanceN)rO rP rQ rR r rN rE rF r r r r r2 � s r2 N)Z __future__r r r Zfuture.builtinsr Zfuture.backportsr Zfuture.backports.urllibr �_parser Z_request�__all__�objectr r: r2 r r r r �<module> s
.
Edit
..
Edit
__init__.cpython-39.opt-1.pyc
Edit
__init__.cpython-39.pyc
Edit
error.cpython-39.opt-1.pyc
Edit
error.cpython-39.pyc
Edit
parse.cpython-39.opt-1.pyc
Edit
parse.cpython-39.pyc
Edit
request.cpython-39.opt-1.pyc
Edit
request.cpython-39.pyc
Edit
response.cpython-39.opt-1.pyc
Edit
response.cpython-39.pyc
Edit
robotparser.cpython-39.opt-1.pyc
Edit
robotparser.cpython-39.pyc
Edit