responses multipart_matcher does not consume the passed iterator; how to mock?

[kfot]

Describe the bug

Hi, I am developing some python code utilizing mulipart/form-data headers for uploading a file. When I try to mock the endpoint response with the following code, then it hangs.

@responses.activate
def test_multipart_upload(client, test_file_definition):
    responses.add(
        responses.POST,
        url=gen_url(<some_endpoint>),
        json=1,
    )
    client.multipart_upload(...)

Responses: 0.25.0
Python: 3.11

Additional context

Once I will add a multipart matcher, I can get an error stating that the iterator does not match the data (as the generator is just an object in the memory).

~#@❯ pytest -x --no-cov -k test_multipart_upload
Test session starts (platform: win32, Python 3.11.8, pytest 7.4.0, pytest-sugar 1.0.0)
rootdir: C:\Users\<REDACTED>
configfile: pytest.ini
testpaths: tests
plugins: anyio-4.3.0, nbmake-1.5.3, cov-4.1.0, icdiff-0.9, sugar-1.0.0, timeout-2.3.1


―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_create_new_version ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― 

client = Client(dummy@unknown(http://test:54422))>
tmp_path = WindowsPath('C:/Users/<REDACTED>/pytest-442/test_multipart_upload0')

    @responses.activate
    def test_multipart_upload(client, tmp_path):
        file_path = tmp_path / "test_data.txt"
        write_file(file_path)  # 1 kB

        responses.add(
            responses.POST,
            url=gen_url(<some_endpoint>),
            match=[
                responses.matchers.multipart_matcher({"file_name": b""})
                ],
            json=1,
        )
>       client.multipart_upload(...)

C:\Users\<REDACTED>\tests\client\test_multipart_upload.py
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
C:\Users\<REDACTED>\envs\mylib311\Lib\site-packages\responses\__init__.py:1173: in send
    return self._on_request(adapter, request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <responses.RequestsMock object at 0x000002AAF71E2B10>, adapter = <requests.adapters.HTTPAdapter object at 0x000002AAF743D210>, request = <PreparedRequest [POST]>, retries = None
kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': None, ...}, match = None
match_failed_reasons = ['multipart/form-data doesn\'t match. Request body differs. <generator object chunk_file_for_upload at 0x000002AAF73C0...: form-data; name="file_name"; filename="file_name"\\r\\n\\r\\n\\r\\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\\r\\n\'']
resp_callback = None
error_msg = 'Connection refused by Responses - the call doesn\'t match any registered mock.\n\nRequest: \n- POST http://test:54422... form-data; name="file_name"; filename="file_name"\\r\\n\\r\\n\\r\\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\\r\\n\'\n'
i = 0, m = <Response(url='http://test:54422/<some_endpoint>' status=200 content_type='application/json' headers='null')>

    def _on_request(
        self,
        adapter: "HTTPAdapter",
        request: "PreparedRequest",
        *,
        retries: Optional["_Retry"] = None,
        **kwargs: Any,
    ) -> "models.Response":
        # add attributes params and req_kwargs to 'request' object for further match comparison
        # original request object does not have these attributes
        request.params = self._parse_request_params(request.path_url)  # type: ignore[attr-defined]
        request.req_kwargs = kwargs  # type: ignore[attr-defined]
        request_url = str(request.url)

        match, match_failed_reasons = self._find_match(request)
        resp_callback = self.response_callback

        if match is None:
            if any(
                [
                    p.match(request_url)
                    if isinstance(p, Pattern)
                    else request_url.startswith(p)
                    for p in self.passthru_prefixes
                ]
            ):
                logger.info("request.allowed-passthru", extra={"url": request_url})
                return self._real_send(adapter, request, **kwargs)  # type: ignore

            error_msg = (
                "Connection refused by Responses - the call doesn't "
                "match any registered mock.\n\n"
                "Request: \n"
                f"- {request.method} {request_url}\n\n"
                "Available matches:\n"
            )
            for i, m in enumerate(self.registered()):
                error_msg += "- {} {} {}\n".format(
                    m.method, m.url, match_failed_reasons[i]
                )

            if self.passthru_prefixes:
                error_msg += "Passthru prefixes:\n"
                for p in self.passthru_prefixes:
                    error_msg += f"- {p}\n"

            response = ConnectionError(error_msg)
            response.request = request

            self._calls.add(request, response)
>           raise response
E           requests.exceptions.ConnectionError: Connection refused by Responses - the call doesn't match any registered mock.
E
E           Request: 
E           - POST http://test:54422/<some_endpoint>?complete=False&encoding=utf-16
E
E           Available matches:
E           - POST http://test:54422/<some_endpoint> multipart/form-data doesn't match. Request body differs. <generator object chunk_file_for_upload at 0x000002AAF73C0DC0> aren't equal b'--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1\r\nContent-Disposition: form-data; name="file_name"; filename="file_name"\r\n\r\n\r\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\r\n'

C:\Users\<REDACTED>\envs\mylib311\Lib\site-packages\responses\__init__.py:1100: ConnectionError

Version of responses

0.25.0

Steps to Reproduce

Add an generator to be consumed by the request.

Expected Result

Code using responses does not hang anymore.

Actual Result

Tests using responses hang.

[beliaev-maksim]

can you please provide a minimal reproducible ?

[kfot]

Feeding multipart requests that way is fine but hangs the responses (as the iterators are not consumed and file.tell() keeps pointing to 0).

def chunk_file(f, chunk_size, chunk_limit):
    chunks_read = 0
    while chunks_read < chunk_limit:
        chunk = f.read(chunk_size)
        if chunk == b"":
            break
        yield chunk
        chunks_read += chunk_size

import os
file_size = os.stat(filepath).st_size
with open(filepath, "rb") as f:
    while f.tell() < file_size:
        data_chunk = chunk_file(
                    f,
                    chunk_size=4*1024,
                    chunk_limit=8*1024,
                )
                ...
                requests.post(url, headers, data=data_chunk)

Avoiding the .tell() in the loop condition (like for _ in range(int(math.ceil(file_size / chunk_limit))):) worked pretty well but I still wonder to what extent the requests behavior should be mockable by the responses.