Supports External Control of Agent Behavior Dynamically

[pepeshore]

Is your feature request related to a problem? Please describe.

When a sudden surge in business traffic causes an application to reach its performance limits, we often need to downgrade some functions of the mounted agent to ensure business availability. Currently, the configuration of the agent requires a restart to take effect, and some behaviors of the probes cannot be dynamically managed. This poses significant risks in real-world enterprise scenarios.

Describe the solution you'd like

We hope to dynamically control the following behaviors of the agent from the external:

• Control whether data is reported
• Control whether the instrumenter executes
• Adjust the sampling rate
• And so on

Describe alternatives you've considered

No response

Additional context

No response

[jackshirazi]

I've been looking at these. It will gradually become more dynamic but never fully. In terms of your top three there

• dynamic sampling rate has this implementation available
• dynamically reporting data currently needs a custom exporter, which just does nothing really other than decide whether to export or not. I think it's reasonable to add a dynamic config to the default exporter, but I haven't tried this yet
• dynamic instrumentation has two aspects
  • Firstly the actual bytecode transformations. I don't think that's going to be dynamic anytime within the next couple of years. (it's feasible but the agent isn't really designed to handle this nicely)
  • The second aspect is whether the instrumentation generates spans. This is in progress to enable/disable dynamically

[pepeshore]

I noticed that the community has been discussing OpAMPs recently. Is it possible to use them to achieve this functionality?

[pepeshore]

https://docs.google.com/document/d/1WK9h4p55p8ZjPkxO75-ApI9m0wfea6ENZmMoFRvXSCw/edit

[jackshirazi]

The short answer is OpAMP doesn't help.

The long answer is that OpAMP and dynamic capability are not tied. You can have the above dynamic capability without OpAMP. Conversely adding in OpAMP does not require dynamic capabilities in the agent or SDK. Current thinking is that with OpAMP you have a few possibilities:

• the agent waits until it receives the OpAMP configuration then initializes based on that
• a changed configuration obtained via OpAMP after agent initialization re-sets the entire SDK & agent (this is feasible for the SDK but unlikely to be achieved for the agent any time soon)
• a changed configuration obtained via OpAMP identifies a "diff" in the configuration which alters only those configuration options that are changed

Using OpAMP to get configuration changes is probably going to be done in a proprietary way for now because the OpAMP specification doesn't require any particular config structure or mechanism. It may evolve to recommend the use of declarative configuration, but that doesn't make it any more supportive of dynamic behaviour