parse_response refactor

[EvilDrPurple]

Fixes

• Refactor parse_response() in collector.py #59

Description

• Changes made according to the original issue
• The tags dictionary is converted to a DataClass specified in DataClassTags.py
• A unique function is created for each attribute being pulled and for the verification code

Testing

1. Clone the repository and checkout the branch
2. Create a virtual environment in the root directory and install the requirements via requirements.txt
3. Run python html_tag_collector/collector.py [test_file.csv].
   test_file.csv can be any csv with a column labeled 'url' containing a set of urls.

[maxachis]

Looks pretty good! Before I approve, I'd like to see the module docstring added and the big if-conditional extracted to a separate function. The namedtuple is a nice-to-have, but I won't block on that.

[maxachis]

NIT: I won't block on this, but recommend converting multi-value responses to named tuples. At a glance, I don't know what each of these values represent, and a named tuple will greatly clarify this.

[EvilDrPurple]

@maxachis like this? I've never used a namedtuple before so correct me if this isn't what you had in mind

def verify_response(res):
    """Verifies the webpage response is readable and ok.

    Args:
        res (HTMLResponse|Response): Response object to verify.

    Returns:
        VerifiedResponse(bool, int): A named tuple containing False if verification fails, True otherwise and the http response code.
    """
    VerifiedResponse = namedtuple("VerifiedResponse", "verified http_response")
    # The response is None if there was an error during connection, meaning there is no content to read
    if res is None:
        return VerifiedResponse(False, -1)

    # If the connection did not return a 200 code, we can assume there is no relevant content to read
    http_response = res.status_code
    if not res.ok:
        return VerifiedResponse(False, http_response)
        
    return VerifiedResponse(True, http_response)

[maxachis]

Yep! Just make sure to have it outside of the function so that other parts of the code can reference it (and also don't forget that the fields are a comma-delimited list, so VerifiedResponse = namedtuple("VerifiedResponse", ["verified", "http_response"])

[maxachis]

Add a docstring describing the function of collector.py here at the top of the function. This will make it easier for people in the 🤖 future 🤖 to understand at a glance what this module is doing.

[EvilDrPurple]

Good description?

""" The tag collector is used to collect HTML tags and other relevant data from websites that is useful for training prediction models.
    Information being collected includes:
        - The URL's path
        - HTML title
        - Meta description
        - The root page's HTML title
        - HTTP response code
        - Contents of H1-H6 header tags
        - Contents of div tags
"""

[maxachis]

An excellent description 😀

[maxachis]

Looks good! Approved!