[Housekeeping] Add support for protobuf version 5 in flytekit

[eapolinario]

Describe the issue

Google released a new major version of the protobuf library in March/2024: https://pypi.org/project/protobuf/#history, but that version started to affect the flytekit ecosystem only after googleapis-common-protos released version 1.36.1, which dropped the constraint on the 5.X version of protobuf.

Using the new major version of protobuf broke the suggested way of using python backend described in https://github.com/protocolbuffers/protobuf/blob/main/python/README.md#implementation-backends, which involved setting an environment variable to instruct the protobuf runtime to use the python implementation. I opened googleapis/python-api-common-protos#232 to track on their side.

What if we do not do this?

We use some plugins in our CI checks that rely on the python implementation to run, since those haven't updated to the version 4 of the protobuf library.

Related component(s)

flytekit

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

The issue you described involves adding support for protobuf version 5 in Flytekit. This is necessary due to the recent release of protobuf 5.0.0 and its impact on the Flytekit ecosystem, particularly when using the Python backend. The problem is exacerbated by the googleapis-common-protos library dropping constraints on protobuf versions, leading to compatibility issues.

To address this, Flytekit needs to update its dependencies and ensure compatibility with protobuf 5.0.0. This involves modifying the environment variable settings and potentially updating CI plugins that rely on the older protobuf versions.

Would you like more detail on how to proceed with these updates?

Sources:

• Flyte Issue #4359
• Googleapis Python API Common Protos Issue #232

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[eapolinario]

The original issue was fixed in https://pypi.org/project/googleapis-common-protos/1.63.2/. We should give this another try.