Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network attack prevents API response #248

Open
senges opened this issue Sep 12, 2023 · 0 comments
Open

Network attack prevents API response #248

senges opened this issue Sep 12, 2023 · 0 comments

Comments

@senges
Copy link

senges commented Sep 12, 2023

Context:

I want to create a network attack that sets ens21 down for 3 minutes, using chaosd in server mode with chaos-mesh CRD integration.

Running naive version with chaosd CLI works:

$ chaosd attack network down -d "ens21" --duration "3m"

Then, here is a simple version of the attack manifest:

kind: PhysicalMachineChaos
apiVersion: chaos-mesh.org/v1alpha1
metadata: {}
spec:
  action: network-down
  duration: 4m
  selector: {}
  network-down:
    device: ens21
    duration: 3m

If I provision this CRD in my k8s cluster, here is what's gonna happen:

  • chaos-controller-manager sends API call to my PhysicalMachine host
  • My host starts disruption on ens21 which makes my host unreachable
  • chaos-controller-manager waits for API response but there is an ongoing network chaos (for 3 min) so it times-out
Failed to apply chaos: : Post "http://10.52.0.205:31767/api/attack/network": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I believe this is caused in server.go#L160 by ExecuteAttack() being called before function returns http.StatusOK. This behavior works for most of the chaos but this one is a bit different as it has its own "duration" and is autonomous on recovering.

Moreover, note that if spec.duration > spec.network-down.duration a retry will be sent to the host API during the spare time between the two "duration" (4th minute in the manifest above). And this second time HTTP will return HTTP 200 and a second network disruption will be triggered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant