Toxic with toxicity: 0 still affects proxy

[fmmagalhaes]

Contrary to my expectations, it seems that when the toxicity of a reset_peer toxic is set to 0, the toxic still has a residual impact on the proxy.

To pinpoint the issue, I have created a single toxic with toxicity: 0:

{
  "name": "connection_reset",
  "type": "reset_peer",
  "stream": "downstream",
  "toxicity": 0
}

Despite the toxicity set to 0, we can still see some exceptions thrown during the test (~1 every 10 seconds). This only happens when using toxiproxy.

The README states

toxicity: probability of the toxic being applied to a link

A toxicity value of 0 then means that the toxic has no probability of being applied, suggesting that it is the same as not having the toxic defined at all. However, this was not what I observed.

Evidence

To make sure the exceptions were not caused by my server, I conducted a test where I created a single toxic with toxicity: 0, followed by the deletion of the toxic. I repeated this process many times, with 5-minute waits between each create and delete operation.

1. Create the toxic

curl -X POST http://localhost:63264/proxies/my-proxy/toxics \
  -H "Content-Type: application/json" \
  -d '{
        "name": "connection_reset",
        "type": "reset_peer",
        "stream": "downstream",
        "toxicity": 0
      }'

2. Delete the toxic

curl -X DELETE http://localhost:63264/proxies/my-proxy/toxics/connection_reset

When the toxic is deleted, the proxy operates without issue. However, when it is introduced with toxicity set to 0, it causes some requests to fail with "Connection reset" errors. The chart below highlights this difference.

Conclusion

Based on the documentation, I expected that a toxicity of 0 would mean the toxic would have no effect, but the observed behavior suggests that toxicity: 0 does not completely disable the toxic.

[mattbardal]

Hey @fmmagalhaes! I'm trying to reproduce this behaviour with a newly created Rails app but I haven't noticed the toxic having any residual impact on the proxy when the toxicity is set to 0.

The steps I followed to try and reproduce are as follows. For reference, I used a brand new Rails 8.0.0.1 app running on port 3000 and Toxiproxy version 2.11.

I created a proxy along with the same reset_peer toxic specified above and used a simple Ruby script to make GET requests and kept track of the number of errors that occur when making the requests to the localhost:22220/up endpoint on the Rails app.

Proxy creation:

curl --request POST \
  --url http://localhost:8474/proxies \
  --header 'Content-Type: application/json' \
  --data '{
  "name": "my-proxy",
  "listen": "127.0.0.1:22220",
  "upstream": "127.0.0.1:3000"
}'

Toxic creation:

curl --request POST \
  --url http://localhost:8474/proxies/my-proxy/toxics \
  --header 'Content-Type: application/json' \
  --data '{
  "name": "my-toxic",
  "type": "reset_peer",
  "stream": "downstream",
  "toxicity": 0
}'

Ruby script:

require 'net/http'
require 'uri'

request_count = 0
error_count = 0
max_requests = 100

while request_count < max_requests
  begin
    response = Net::HTTP.get_response(URI.parse("http://localhost:22220/up"))
  rescue => e
    puts "Error: #{e.message}"
    error_count += 1
  end
  request_count += 1
  # sleep 0.5 # I've tried with a few different values (e.g., 0.25, 0.5, 1) and without sleeping
end

puts "Requests: #{request_count}"
puts "Errors: #{error_count}"

After running this script a few times with toxicity: 0, I did not encounter any Connection reset by peer errors. However, when I removed the toxic and recreated it with a toxicity: 0.25 and tried again, this time the Ruby script started encountering Connection reset by peer errors.

Is there anything I may be missing while trying to reproduce and are you still encountering this behaviour?

[fmmagalhaes]

Hello, @mattbardal

On my side, the client is a Spring Boot application. While I couldn’t replicate the issue locally with the same client, it consistently occurs in my CI/CD pipeline during tests that use Toxiproxy. I’ve managed to reproduce it with two independent applications in that environment.

Unfortunately, I can’t disclose much about the client itself, which I understand makes it harder to identify the problem.

I have also been unable to pinpoint the exact conditions under which the issue is reproducible. However, I believe the test I shared earlier still highlights a potential issue with the toxicity mechanism. Would you agree with that assessment?