While test integrating tonic into TiKV to replace grpc-rs, I found that tonic's performance is not as good as grpc-rs as it consumes more cpu. And from the profile, I saw that a lot of cpu time is cost by the tracing logic in h2. In some workloads, the qps with tonic can be 15% lower than grpc-rs. Here is a cpu profiling snapshot:

After patching tonic with a h2 package with all tracing code removed, tonic's performance is slightly better than gprc-rs in almost all workload without any further optimization.