fix(utils): add ClearUnmatchingDeprecations function to align configs

[nowNick]

Summary

When a new plugin configuration is sent that contains information about some deprecated fields in order to produce
correct diff we need to align the API response with given config.

Description

This PR aims to align the configs by doing two things:

1. Adding: ClearUnmatchingDeprecations
2. Modifying: fillConfigRecord

More details can be found in the functions documentation.

Important changes:

• nil values are preserved and not changed when fillConfigRecord is run. Previously a config cluster_max_redirections: null would be "filled" with cluster_max_redirections: 5 if cluster_max_redirections had a default value of 5 defined. Right now sending explicit null preserves that. This reverts the change introduced in: https://github.com/Kong/go-kong/pull/309/files#diff-af8350ebad15aeeb0a2561433d1d0907140df8e5b8267a0d0f05bb53d0d53fcbR221-R249
• shorthand values are not backfilled. If a new config contains only new fields fillConfigRecord will not try to fill deprecated fields in that config
• if a deprecated field was defined fillConfigRecord will not populate the new fields that are linked to that deprecated field

Related PRs:

• (next one) fix(diff): clear unmatching deprecated fields go-database-reconciler#145

KAG-5577

[CLAassistant]

All committers have signed the CLA.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 93.98907% with 11 lines in your changes missing coverage. Please review.

Project coverage is 55.15%. Comparing base (8678be1) to head (8b77ac2).

Files with missing lines	Patch %	Lines
kong/utils.go	93.98%	8 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #473      +/-   ##
==========================================
+ Coverage   54.04%   55.15%   +1.11%     
==========================================
  Files          71       71              
  Lines        5599     5691      +92     
==========================================
+ Hits         3026     3139     +113     
+ Misses       1954     1935      -19     
+ Partials      619      617       -2     

Flag	Coverage Δ
2.1	33.38% <44.26%> (-0.16%)	⬇️
2.2	43.85% <44.26%> (-0.33%)	⬇️
2.3	44.42% <44.26%> (-0.34%)	⬇️
2.4	44.49% <44.26%> (-0.34%)	⬇️
2.5	44.49% <44.26%> (-0.34%)	⬇️
2.6	44.49% <44.26%> (-0.34%)	⬇️
2.7	45.96% <44.26%> (-0.37%)	⬇️
2.8	45.96% <44.26%> (-0.37%)	⬇️
3.0	49.51% <44.26%> (-0.43%)	⬇️
3.1	50.85% <44.26%> (-0.45%)	⬇️
3.2	50.85% <44.26%> (-0.45%)	⬇️
3.3	50.85% <44.26%> (-0.45%)	⬇️
3.4	52.87% <44.26%> (-0.48%)	⬇️
3.5	51.02% <44.26%> (-0.45%)	⬇️
3.6	51.02% <44.26%> (-0.45%)	⬇️
3.7	51.02% <44.26%> (-0.45%)	⬇️
3.8	52.62% <93.98%> (+1.15%)	⬆️
community	41.60% <91.80%> (+1.26%)	⬆️
enterprise	53.75% <93.98%> (+1.13%)	⬆️
integration	55.15% <93.98%> (+1.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[nowNick]

Thanks @Prashansa-K for the review. For now I've changed this PR back to draft because I've found a couple of issues.

[nowNick]

Ok the corresponding PR is green and I think I've tackled the edge cases I've discovered (in the tests in this PR and the other one: Kong/go-database-reconciler#145 ). I think it's ready to be reviewed again.

The requested changes were addressed and incorporated.

[nowNick]

I did some important changes to fillConfigRecord. @GGabriele @samugi could you guys take a look at it again? I've tried to document it as much as possible to ease reviewing process.

[pmalek]

Minor comments on Go style etc.

[Prashansa-K]

I tested these changes locally with an openid plugin (deck file added here)and noticed the following:

On gateway 3.8.1.0
No unnecessary diffs come up, except

+    "session_compressor": "none"
+    "session_cookie_maxsize": 4000
+    "session_cookie_renew": 600
+    "session_strategy": "default"

These fields don't have any translate_backwards or replaced_with property in the schema. So, not applicable with this change.

However, on Konnect, I noticed some diffs:

-    "session_redis_cluster_max_redirections": 5,
+    "session_redis_cluster_max_redirections": null,

-    "session_redis_connect_timeout": 2000,
+    "session_redis_connect_timeout": null,
    
-    "session_redis_read_timeout": 2000,
+    "session_redis_read_timeout": null,
-    "session_redis_send_timeout": 2000,
+    "session_redis_send_timeout": null,

decK and reconciler logic remains same for Konnect as far as I know. Is there anything different schema-wise for Konnect that could be causing this?

[nowNick]

@Prashansa-K it seems that it's a well known Koko issue. The reason why you see these diffs in Konnect is that Lua-CP and Koko treat null values differently. LuaCP preserves them but Koko falls back to default value if defined.

[pmalek]

Some minor comments. Other than that this looks solid.

One thought though: could we add an integration test that checks against the schema we receive from Gateway? Perhaps we can use

go-kong/kong/plugin_service.go

Line 64 in 5236d86

func (s *PluginService) GetFullSchema(ctx context.Context,

to get the schema and then use that in the test? We can use RunWhenKong() to constrain the versions against which we run these tests.

[samugi]

I verified the "important changes" described in the summary, LGTM

[nowNick]

@pmalek I've made the changes according to your comments and I've switched to calling Kong API in tests (didn't know we can do this from go-kong). Does it look better now? (feel free to check the latest commit - I didn't force-push)

[pmalek]

Minor nits about the testing.

[pmalek]

Looks good to me 👍