Make --dry-run an actual dry-run

[STgversluis]

Describe the bug
The term dry-run suggests that apart from actually applying the config, everything is tested. However, this is not the case.
After a lot of time spent troubleshooting together with Dynatrace support (case #172865), we came to the conclusion that the dry-run only runs a syntax check. It does not connect to the actual tenant, it does not check for conflicting settings, it does not verify the token… which means it leaves plenty of opportunity for disaster when someone makes a wrong change.

How to reproduce
Perform a dry-run known to result in an error, such as:

• set an incorrect token in the environment variable
• use a config that will overwrite or modify an existing setting

Expected behavior

• If an incorrect token is used, the dry-run should indicate this in a correct error message (in the log snippet below, the token is NOT an old format - I used a valid token and changed only 1 letter)
• Do not state incorrect or confusing info in the log (in the log snippet below, the remark about a Dynatrace Classic environment is confusing - what is a Dynatrace Classic environment?)
• If a configuration is overwritten, this should be logged and warned about in the dry-run

Log output

Output from dry-run:
2023-08-18T12:20:20+02:00 info Validation finished without errors

Output from download, with same env var value for token:
2023-08-18T12:21:39+02:00 info Loading manifest "manifest.yaml". Restrictions: groups=[], environments=["test"]
2023-08-18T12:21:39+02:00 warn You used an old token format. Please consider switching to the new 1.205+ token format.
More information: https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication
2023-08-18T12:21:39+02:00 error Could not authorize against the environment with name "test" (https://dynatrace.europe.stater.corp/e/test) using token authorization: failed to query version of Dynatrace environment: (HTTP 401) Response was: {"error":{"code":401,"message":"Token Authentication failed"}}
2023-08-18T12:21:39+02:00 error Please verify that this environment is a Dynatrace Classic environment.

Environment (please complete the following information):

• Dynatrace Managed
• Monaco v2.6.0

Additional context
none.

[UnseenWizzard]

Hi @STgversluis.

You're correct that dry-run only validates configuration file syntax.
This is documented: https://www.dynatrace.com/support/help/manage/configuration-as-code/reference/commands#deploy-flags

Providing a full validation that connects to the environment is already on our internal roadmap, though at this point we can not provide any ETA for this feature.

To you other notes on expected behaviour: the CLI has no way of identifying how a token is incorrect, as it can only work on what error information returned by the Dynatrace API. (401 Unauthorized HTTP)
A few log lines for possible causes for such errors are thus always printed. A 'classic' environment is a non platform/"latest" Dynatrace environment.

I hope that helps a bit.

I will leave this github issue open to allow others to find this question/answer in the future.

[STgversluis]

Hi @UnseenWizzard ,

I don't agree on your statement that the documentation is clear about a dry-run only checking syntax. The documentation leaves out that no actual connection is made and the confusing log lines make things worse.

As you state the CLI behaviour is a response to 401 Unauthorized HTTP. Why not start those log lines with something like "The authentication failed. Please check the token in ." and then continue with "Other possible causes: You may be using an old token format or connecting to a Dynatrace environment that is not on a recent version." (please note I mentioned may be using an old token format, not "You used an old token format" which is an incorrect statement of fact.)

From my POV (not knowing the inner workings of the CLI or the workload of the development team) this shouldn't be too complex, and it would have saved both me and Dynatrace Support quite some time.

[UnseenWizzard]

Thanks for the feedback @STgversluis!
We will try to clarify both the documentation and the messages produced by the CLI with upcoming releases.

[UnseenWizzard]

Hi @STgversluis, after taking another look at the logs you're seeing in this case, I don't see your suggestion as different from what is already being logged?

One (technical) note on the logs just to clarify the cause:
The token format warning and the 401 Unauthorized error are independent and likely unrelated to each other. Unless the token is expired or missing permissions, Dynatrace still accepts tokens in the old format.

The token format warning happens as soon as your token is loaded but has no relation to whether this token is valid for the Dynatrace environment. It's just a hint that you could create a newer one with more fine grained options. (Note that Dynatrace 1.205 is nearly 3 years old.)

2023-08-18T12:21:39+02:00 warn You used an old token format. Please consider switching to the new 1.205+ token format.
More information: https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication

We'll extend the authentication error with a note to check if the token is correct.

2023-08-18T12:21:39+02:00 error Could not authorize against the environment with name "test" (https://dynatrace.europe.stater.corp/e/test) using token authorization: failed to query version of Dynatrace environment: (HTTP 401) Response was: {"error":{"code":401,"message":"Token Authentication failed"}}
2023-08-18T12:21:39+02:00 error Please verify that this environment is a Dynatrace Classic environment.

Would that cover what you had in mind, or am I missing something here?

edit: taking another look at the token format check, I see how it is misleading! Technically the CLI checks whether the token matches the new format, not that it's an old one.
As such any string that does not happen to be in the expected token format will produce the warning that it's an "old token", even if it's not a Dynatrace API token at all.
Was that the cause of your issue?

[STgversluis]

Hi @UnseenWizzard ,

I was using a new style token - actually, I took the valid token and only changed the last letter of that from an A to a B. IMO, that doesn't do anything with regards to the new format - it should still match.

And is there a way to check the version, even without a token? That way you'd be able to rule out old style environments. I can even imagine changing the 401 response to include something in the newer version... wouldn't change anything for the current stuff obviously, but would be helpful in the future.

Regards, Gerald

[UnseenWizzard]

Hi @STgversluis.

We've just released v2.7.1 which aims to improve that warning, now printing:

2023-08-25T10:37:33+02:00 warn The supplied token does not match the expected format and may be invalid. If authentication fails, please check your manifest and environment variable configuration.
If you are using a token created before Dynatrace 1.205, please consider generating a new token: https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication

There is sadly no way to reliably verify a Dynatrace environment's "generation" (which is where you're seeing the 401 Unauthorized) other than connecting to its API, which always requires valid credentials/tokens.
So if your access token is invalid you'll always see a combination of the token format warning and the first call to the Dynatrace API failing.