docs(proxy-wasm) document response body buffering behavior

docs/DIRECTIVES.md

@@ -26,6 +26,7 @@ By alphabetical order:
 - [tls_verify_cert](#tls_verify_cert)
 - [tls_verify_host](#tls_verify_host)
 - [wasm_call](#wasm_call)
+- [wasm_response_body_buffers](#wasm_response_body_buffers)
 - [wasm_socket_buffer_reuse](#wasm_socket_buffer_reuse)
 - [wasm_socket_buffer_size](#wasm_socket_buffer_size)
 - [wasm_socket_connect_timeout](#wasm_socket_connect_timeout)
@@ -66,6 +67,7 @@ By context:
     - [proxy_wasm_request_headers_in_access](#proxy_wasm_request_headers_in_access)
     - [resolver_add](#resolver_add)
     - [wasm_call](#wasm_call)
+    - [wasm_response_body_buffers](#wasm_response_body_buffers)
     - [wasm_socket_buffer_reuse](#wasm_socket_buffer_reuse)
     - [wasm_socket_buffer_size](#wasm_socket_buffer_size)
     - [wasm_socket_connect_timeout](#wasm_socket_connect_timeout)
@@ -758,6 +760,20 @@ return `HTTP 500`.
 
 [Back to TOC](#directives)
 
+wasm_response_body_buffers
+--------------------------
+
+**usage**    | `wasm_response_body_buffers <number> <size>;`
+------------:|:----------------------------------------------------------------
+**contexts** | `http{}`, `server{}`, `location{}`
+**default**  | `4 4096`
+**example**  | `wasm_response_body_buffers 2 16k;`
+
+Set the maximum `number` and `size` of buffers used for [response body
+buffering](PROXY_WASM.md#response-body-buffering).
+
+[Back to TOC](#directives)
+
 wasm_socket_buffer_reuse
 ------------------------
 

docs/PROXY_WASM.md

@@ -15,6 +15,8 @@
     - [Supported Entrypoints](#supported-entrypoints)
     - [Supported Host ABI](#supported-host-abi)
     - [Supported Properties](#supported-properties)
+- [Extended Specifications](#extended-specifications)
+    - [Response Body Buffering](#response-body-buffering)
 - [Examples]
 - [Current Limitations]
 
@@ -383,6 +385,7 @@ specifications and different SDK libraries:
 - [Tested SDKs](#tested-sdks)
 - [Supported Entrypoints](#supported-entrypoints)
 - [Supported Host ABI](#supported-host-abi)
+- [Supported Properties](#supported-properties)
 
 [Back to TOC](#table-of-contents)
 
@@ -578,7 +581,7 @@ implementation state in ngx_wasm_module:
 -------------------------------------------:|:------------------:|:-------------------:|:----------------
 *Request properties*                        |                    |
 `request.path`                              | :heavy_check_mark: | :x:                 | Maps to [ngx.request_uri](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_uri).
-`request.url_path`                          | :heavy_check_mark: | :x:                 | Maps to [ngx.uri](https://nginx.org/en/docs/http/ngx_http_core_module.html#uri). 
+`request.url_path`                          | :heavy_check_mark: | :x:                 | Maps to [ngx.uri](https://nginx.org/en/docs/http/ngx_http_core_module.html#uri).
 `request.host`                              | :heavy_check_mark: | :x:                 | Maps to [ngx.hostname](https://nginx.org/en/docs/http/ngx_http_core_module.html#hostname).
 `request.scheme`                            | :heavy_check_mark: | :x:                 | Maps to [ngx.scheme](https://nginx.org/en/docs/http/ngx_http_core_module.html#scheme).
 `request.method`                            | :heavy_check_mark: | :x:                 | Maps to [ngx.request_method](https://nginx.org/en/docs/http/ngx_http_core_module.html#request_method).
@@ -651,6 +654,59 @@ ngx_wasm_module, most likely due to a Host incompatibility.
 
 [Back to TOC](#table-of-contents)
 
+## Extended Specifications
+
+This section documents enhancements, specification extensions or known
+differences with the Proxy-Wasm specifications, or existing host implementations
+(e.g. Envoy).
+
+[Back to TOC](#table-of-contents)
+
+### Response Body Buffering
+
+Buffering of response body chunks is supported within ngx_wasm_module itself so
+filters don't have to implement buffering themselves.
+
+Buffered response chunks will be copied to buffers defined by the
+[wasm_response_body_buffers] directive while execution of Proxy-Wasm filter
+chain is suspended.
+
+To trigger this behavior from a filter based on Proxy-Wasm ABI v0.2.1, the
+filter must return the `AGAIN` action from `on_response_body`. Once enabled,
+ngx_wasm_module will accumulate subsequent body chunks until either `eof` is
+reached, or the buffers are full. When either of these conditions are met,
+`on_response_body` will be invoked again and the body buffer will contain the
+buffered chunks.
+
+In other words, once body buffering enabled, the next `on_response_body`
+invocation will contain the buffered body and **may or may not be invoked
+again**, depending on whether `eof` was reached.
+
+A typical response buffering flow could be:
+
+1. 1st `on_response_body` call: ignore 1st chunk, requesting buffering
+    1. Check for `eof=false`
+    2. Ensure buffering was not already requested
+    3. Return the `AGAIN` action, requesting buffering
+2. 2nd `on_response_body` call: buffering ended, but how?
+    1. If `eof=true`, the full response body is in the buffers
+    2. If `eof=false`, the buffers are full, but more chunks are expected
+       (users should treat the buffers as a single non-buffered chunk)
+3. nth `on_response_body` call - next chunks, if any
+
+Attempts to return `AGAIN` when buffering has already taken place will be
+ignored (i.e. treated as `CONTINUE`) and an error log will be printed.
+
+> Notes
+
+Keep in mind there are fundamental issues with buffering bodies at scale due to
+the nature of the workload, hard buffer limits defined by
+[wasm_response_body_buffers], and Wasm memory limits themselves (loading and
+manipulating the body in filters). This feature should be used with extreme
+caution in production environments.
+
+[Back to TOC](#table-of-contents)
+
 ## Examples
 
 - Functional filters written by the WasmX team:
@@ -671,12 +727,13 @@ factors are at play when porting the SDK to a new Host proxy runtime.
 
 Proxy-Wasm's design was primarily influenced by Envoy concepts and features, but
 because Envoy and Nginx differ in underlying implementations there remains a few
-limitations on some supported features:
+limitations on some supported features (non-exhaustive list):
 
 1. Pausing a filter (i.e. `Action::Pause`) can only be done in the following
    steps:
     - `on_http_request_headers`
     - `on_http_request_body`
+    - `on_http_response_body` (to enable body buffering)
     - `on_http_call_response`
 
 2. The "queue" shared memory implementation does not implement an automatic
@@ -697,6 +754,8 @@ limitations and increasing overall surface support for the Proxy-Wasm SDK.
 [Examples]: #examples
 [Current Limitations]: #current-limitations
 
+[wasm_response_body_buffers]: DIRECTIVES.md#wasm_response_body_buffers
+
 [WebAssembly]: https://webassembly.org/
 [Nginx Variables]: https://nginx.org/en/docs/varindex.html
 [Envoy Attributes]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes.html?highlight=properties#request-attributes