Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTTP/2 (D)DoS prevention #1346

Open
2 tasks
krizhanovsky opened this issue Aug 29, 2019 · 9 comments · May be fixed by #2108
Open
2 tasks

HTTTP/2 (D)DoS prevention #1346

krizhanovsky opened this issue Aug 29, 2019 · 9 comments · May be fixed by #2108

Comments

@krizhanovsky
Copy link
Contributor

krizhanovsky commented Aug 29, 2019

HTTTP/2 flood prevention

Low-Rate Denial-of-Service Attacks against HTTP/2 Services paper studies HTTP/2 floods using PING and WINDOW_UPDATE frames. It's not good to fight against the attacks using packet rate limit since a client can send large POST requests for example. From the other hand requests rate in such attacks could be zero. So a new limits with burst, just the same as for current request limiting, must be developed.

It makes sense to apply the limit to service frames only since HEADERS and DATA obey existing HTTP request limits. We can limit all the service frames at once, there is no need to limit them separately.

HTTTP/2 slow read

Slow Read Attack (CVE-2016-1546) uses many concurrent streams with little window updates, so

  1. The HTTP limits must be extended by stream_rate, stream_burst and concurrent_streams just similar to equal connection limits.

  2. response_timeout limit must be introduced for both the HTTP/1 and HTTP/2. The limit specifies how long we can be sending a given response to a client. To not to rate limit valid Comet responses one must specify the limit for vhosts other than servicing Comet resources.

HPACK bomb

Huffman decoding is an expensive operation and we must check http_field_len Frang limit before Huffman decoding execution (see also HPACK bomb). Considering implementation variants - it seems that the simplest and most effective way is to embed a hook into HTTP/2-parser (or into HPACK-decoder) directly to catch the long headers immediately.

Also see corresponding TODO comment for __frang_http_field_len() in http_limits.c - we should do the same for HTTP/1.

We have separate issue for this #1780

Header list size limit

HTTP limiting module provides important for HTTP/2 security limit http_header_cnt. According to RFC 7540 chapters 10.5.1 and 6.5.2 having the limit set, Tempesta should send SETTINGS_MAX_HEADER_LIST_SIZE settings parameter to a client.

Testing

  • HTTP/2 streams rate limit to prevent HTTP/2 slow read. See #88
  • HTTP/2 frames rate limit to prevent HTTP/2 floods. See #88

Test to reproduce for concurrent streams.

Documentation

Please update https://github.com/tempesta-tech/tempesta/wiki/HTTP-security#frang-security-limits-enforcing-module with the reference to the paper.

@RomanBelozerov
Copy link
Contributor

RomanBelozerov commented Jan 24, 2023

The HTTP limits must be extended by stream_rate, stream_burst and concurrent_streams just similar to equal connection limits.

Tempesta should send SETTINGS_MAX_CONCURRENT_STREAMS in SETTINGS frame when initializing connection with client.

Please enable http2_general.test_h2_streams.TestH2Stream.test_max_concurrent_stream test after fixes

@EvgeniiMekhanik
Copy link
Contributor

EvgeniiMekhanik commented Jun 28, 2023

During developing this task we should don't forget to change

if (closed_streams->num <= TFW_MAX_CLOSED_STREAMS
    || (max_streams == ctx->streams_num
	&& closed_streams->num))
``
in `tfw_h2_closed_streams_shrink`

@krizhanovsky
Copy link
Contributor Author

Related CVE and security advisory GHSA-93p3-5r25-4p75

@const-t
Copy link
Contributor

const-t commented Apr 16, 2024

  1. First of all we must not implement stream_rate, stream_burst, we have request_rate/burst that works the same.
  2. response_timeout it's very doubtful. It can help us only if limit whole connection lifetime, without any updates during connection lifetime. For example: Slow read client opens connection with 100 streams, stream 1 sends request, Tempesta scheduled it to upstream. Start response_timeout, if response timeout exceeded, reset connection.
    Cons of this method:
    1. We always must know how many time will take transferring of largest file with slowest client.
    2. It's per-connection not per-stream.

However, it seems in current task we must implement rate limits for SETTINGS, PING and maybe WINDOW_UPDATE frame. But let's start from SETTINGS and PING. WINDOW_UPDATE requires some discussion, because it's not so easy and maybe it not worth it. For instance, client can send unlimited number of WINDOW_UPDATE frames to closed streams, that also not good.

@const-t
Copy link
Contributor

const-t commented Apr 17, 2024

After some discussion, we came to the conclusion that we need one timer per connection and bitfield that indicates stream stream being read in the timer interval. On every shot timer clears bitfield. However this doesn't look like complete solution.

@kingluo
Copy link
Contributor

kingluo commented Apr 17, 2024

For "slow read":
The slow-read attack affects the CPU, memory, and incoming connection pool. So #498 does not cover the issue.

A simple solution:

  1. define a frang limit: client_read_timeout <timeout> <min_rate>
  2. HEADERS/CONTINUATION/DATA frames sending will count the sending bytes of the current stream
  3. reuse the keep-alive timer, and set it as 1 second resolution
  4. in the timer callback, we iterate all streams and update their rates, if the rate of any stream is lower than the <min_rate>, we increase the duration, and if the accumulated duration is over <timeout>, we abort the connection and stop the iteration.
  5. works for any response slow-read cases (e.g. tcp-level slow-read, small sockbuf) and HTTP versions.

It works similarly to apache httpd, but here we aim at the response direction and use and condition to combine the <timeout> and <min_rate>.

@kingluo
Copy link
Contributor

kingluo commented Apr 17, 2024

For "HTTTP/2 flood prevention":
I doubt the low-rate service frames like PING can form the DOS, so maybe we just focus on the high-rate case? That is the PING flood, as well as SETTING (as suggested by @const-t, we can postpone handling WINDOW_UPDATE).

A simple solution:
We add timing history for service frames (so far, only PING and SETTING) just like HEADERS and enforce it in frang.

@const-t
Copy link
Contributor

const-t commented Apr 18, 2024

In this issue must be implemented fixes for following CVEs:
CVE-2019-9511 “Data Dribble”
CVE-2019-9517 “Internal Data Buffering”
CVE-2019-9512 “Ping Flood”
CVE-2019-9515 “Settings Flood”

kingluo added a commit that referenced this issue Apr 24, 2024
@kingluo kingluo linked a pull request Apr 24, 2024 that will close this issue
@krizhanovsky
Copy link
Contributor Author

Crucial since DDoS affects our installation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants