Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add buf.registry.descriptor.v1 package #102

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
// Copyright 2023-2024 Buf Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

syntax = "proto3";

package buf.registry.descriptor.v1;

import "buf/registry/priv/extension/v1beta1/extension.proto";
import "buf/validate/validate.proto";

option go_package = "buf.build/gen/go/bufbuild/registry/protocolbuffers/go/buf/registry/descriptor/v1";

// Additional information about a FileDescriptorProto.
//
// FileDescriptoProtos are not extendable, so we choose to sideband this information as opposed
// to creating a new FileDescriptoProto type, so that the FileDescriptorProtoService can return
nicksnyder marked this conversation as resolved.
Show resolved Hide resolved
// standard FileDescriptorProtos that most expect.
message FileDescriptorProtoExtension {
option (buf.registry.priv.extension.v1beta1.message).response_only = true;

// The id of the Commit that the file was created from.
//
// If the file was created from a Well-Known Type, this will point to
// a Commit within a Module that contains only the Well-Known Types. For the public BSR,
// this is currently buf.build/protocolbuffers/wellknowntypes.
string commit_id = 1 [
(buf.validate.field).required = true,
(buf.validate.field).string.tuuid = true
];
// Whether or not the file is an import.
//
// A file is an import if:
//
// - It comes from a dependency of the Module specified via the ResourceRef used in the request.
// For example, if the ResourceRef specifies "buf.build/foo/bar", which depends on
// "buf.build/foo/baz", all files from "buf.build/foo/baz" will be imports.
// - It is a Well-Known Type, and the Well-Known Type was not part of the Module specified
// via the ResourceRef used in the request. Well-Known Types are special in the Protobuf ecosystem,
// and can be imported without being contained in a specified module.
// - It contains symbols that are referenced by symbols specified in the request. For example,
// if the request specified symbol "foo.v1.Foo" contained within "foo.proto", which has a message
// field of type "baz.v1.Baz", which is contained within "baz.proto", then the file "baz.proto"
// will be marked as an import, regardless of which Module it derived from. If the symbols field
// is not specified on the request, this situation will never occur.
//
// If exclude_imports was specified on the request, no FileDescriptorProtos will be imports.
bool is_import = 2;
// The explicitly-specified syntax within the file.
//
// If the syntax field is not set within a file, this is interpreted as the syntax "proto2".
// The protoc compiler does not set the syntax field if the syntax was "proto2", regardless of if
// the syntax was explicitly specified or not. This prevents programs (such as linters) from
// determining whether or not a syntax was explicitly specified within a file
//
// As opposed to syntax on a FileDescriptorProto, this field will be empty if no syntax was
// explicitly specified, and "proto2" if "proto2" was explicitly specified. Otherwise, this
// field matches the semantics of the syntax field on FileDescriptorProtos.
Comment on lines +59 to +68
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the middle paragraph here a bit of a tangent that was confusing on first read. Wondering if we can resequence this documentation so the behavior is documented first, and then the "why" is last (if necessary at all).

Suggested change
// The explicitly-specified syntax within the file.
//
// If the syntax field is not set within a file, this is interpreted as the syntax "proto2".
// The protoc compiler does not set the syntax field if the syntax was "proto2", regardless of if
// the syntax was explicitly specified or not. This prevents programs (such as linters) from
// determining whether or not a syntax was explicitly specified within a file
//
// As opposed to syntax on a FileDescriptorProto, this field will be empty if no syntax was
// explicitly specified, and "proto2" if "proto2" was explicitly specified. Otherwise, this
// field matches the semantics of the syntax field on FileDescriptorProtos.
// The explicitly-specified syntax within the file.
// If the syntax field is not set within a file, this is interpreted as the syntax "proto2".
//
// As opposed to syntax on a FileDescriptorProto, this field will be empty if no syntax was
// explicitly specified, and "proto2" if "proto2" was explicitly specified. This enables programs
// (such as linters) to determine whether or not a syntax was explicitly specified within a file.
// Otherwise, this field matches the semantics of the syntax field on FileDescriptorProtos.

string specified_syntax = 3;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional design note - this was a bool syntax_specified in image.proto, but it seems clearer to make this the actual specified syntax, if it exists.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to validate this with an in constraint?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for a proto file with syntax="proto2", or without a syntax specified, FileDescriptorProto.syntax is blank, and this field is "proto2".

What happens for a file with edition="2023"? FileDescriptorProto.syntax will be "editions" in this case, but syntax wasn't really specified. Is this field "", or "editions"?

// The indexes within the dependency field on the FileDescriptorProto that are not used.
//
// The matches the shape of the public_dependency and weak_dependency fields on FileDescriptorProto.
repeated int32 unused_dependency = 4;
}
130 changes: 130 additions & 0 deletions buf/registry/descriptor/v1/file_descriptor_proto_service.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
// Copyright 2023-2024 Buf Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

syntax = "proto3";

package buf.registry.descriptor.v1;

import "buf/registry/descriptor/v1/file_descriptor_proto_extension.proto";
import "buf/registry/module/v1/resource.proto";
import "buf/validate/validate.proto";
import "google/protobuf/compiler/plugin.proto";
import "google/protobuf/descriptor.proto";
import "google/protobuf/field_mask.proto";

option go_package = "buf.build/gen/go/bufbuild/registry/protocolbuffers/go/buf/registry/descriptor/v1";

// Provide compiled FileDescriptorProtos for Modules, Labels, or Commits.
service FileDescriptorProtoService {
// Get compiled FileDescriptorProtos for the given Module, Label, or Commit.
rpc GetFileDescriptorProtos(GetFileDescriptorProtosRequest) returns (GetFileDescriptorProtosResponse) {
option idempotency_level = NO_SIDE_EFFECTS;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late to the party, but this is the most important thing to me: the RPC should be side effect free, so we can expose it as an HTTP GET using Connect.

}

message GetFileDescriptorProtosRequest {
// The reference to get FileDescriptorProtos for.
//
// See the documentation on ResourceRef for resource resolution details.
//
// Once the resource is resolved, the following content is returned:
// - If a Module is referenced, the FileDescriptorProtos for the Commit of the default Label is returned.
// - If a Label is referenced, the FileDescriptorProtos for the Commit of this Label is returned.
// - If a Commit is referenced, the FileDescriptorProtos for this Commit is returned.
buf.registry.module.v1.ResourceRef resource_ref = 1 [(buf.validate.field).required = true];
// Exclude imports from the returned FileDescriptorProtos.
//
// A file is an import if:
//
// - It comes from a dependency of the Module specified via the ResourceRef used in the request.
// For example, if the ResourceRef specifies "buf.build/foo/bar", which depends on
// "buf.build/foo/baz", all files from "buf.build/foo/baz" will be imports.
// - It is a Well-Known Type, and the Well-Known Type was not part of the Module specified
// via the ResourceRef used in the request. Well-Known Types are special in the Protobuf ecosystem,
// and can be imported without being contained in a specified module.
// - It contains symbols that are referenced by symbols specified in the request. For example,
// if the request specified symbol "foo.v1.Foo" contained within "foo.proto", which has a message
// field of type "baz.v1.Baz", which is contained within "baz.proto", then the file "baz.proto"
// will be marked as an import, regardless of which Module it derived from. If the symbols field
// is not specified on the request, this situation will never occur.
//
// Note that if imports are excluded, the returned FileDescriptorProtos may not (and likely will not)
// be self-contained. If imports are included, all symbols referenced within the FileDescriptorProtos
// are contained within the returned FileDescriptorProtos.
bool exclude_imports = 2;
// Exclude SourceCodeInfo from the returned FileDescriptorProtos.
//
// SourceCodeInfo is optional additional information that is not required to describe a set
// of Protobuf APIs.
bool exclude_source_code_info = 3;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question whether this should be include or exclude - we decided for the HTTP GET endpoint to make this include.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my mind, the decision on whether to call this include or exclude depends on whatever you want to be returned by default. If this should be returned by default, then exclude is right (opt-out). If it shouldn't be returned by default, then include is right (opt-in). To decide what the default would be, I would think about the anticipated use cases and whether a majority of those use cases would need or not need this info. (similarly for the other fields)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Writing tools against descriptors, I find myself constantly forgetting that the info is not returned by default and I need to set a flag somewhere in the chain of tools to ensure the info is available. I'd rather need to ask for it to be stripped as an optimization than have to remember to include it, personally.

This goes for the other opt-in/out questions below, as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Including by default is more convenient for use cases like the CLI. Excluding by default is better for constrained environments, where things suddenly stop working because a giant module with many comments shows up.

Giant modules have other downsides. Including by default is the least surprising behavior in my book.

// Exclude source-retention options from the returned FileDescriptorProtos.
//
// With the new Editions work, options are either runtime-retention or source-retention.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// With the new Editions work, options are either runtime-retention or source-retention.
// With Protobuf Editions, options are either runtime-retention or source-retention.

// Setting this option will exclude the latter.
bool exclude_source_retention_options = 4;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question whether this should be include or exclude - we decided for the HTTP GET endpoint to make this include.

// Exclude FileDescriptorProtoExtensions from the response.
//
// FileDescriptorProtoExtensions contain additional information about FileDescriptorProtos.
// We include this information in a separate message as FileDescriptorProtos are not
// extendable, and we do not want to modify the FileDescriptorProto shape.
bool exclude_file_descriptor_proto_extensions = 5;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question whether this should be include or exclude.

// Only include specific symbols (and symbols that they reference, if exclude_imports is not set).
//
// A symbol is a package, message, enum, service, method, or extension.
//
// Symbols are referenced by their fully-qualified name. For example, if "Foo" is a message within
// the package "foo.v1", the fully-qualified name is "foo.v1.Foo". Fully-qualfied names should
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// the package "foo.v1", the fully-qualified name is "foo.v1.Foo". Fully-qualfied names should
// the package "foo.v1", the fully-qualified name is "foo.v1.Foo". Fully-qualified names should

// not start with a period, that is specify "foo.v1.Foo", not ".foo.v1.Foo".
//
// If no symbols are provided, all symbols are returned.
repeated string symbols = 6;
// An explicit, opt-in field mask for the fields populated on the returned FileDescriptorProtos.
//
// For example, if you would only like to return the names of messages, use the mask:
//
// {
// paths: [
// "message_type.name",
// "message_type.nested_type.name"
// ]
// }
//
// The filters exclude_imports, exclude_source_code_info, exclude_source_retention_options and
// symbols continue to be respected even if this field is set. Specifically, if exclude_source_code_info
// is set, but this mask includes source_code_info, SourceCodeInfo will continue to be excluded.
google.protobuf.FieldMask field_mask = 7;
}

message GetFileDescriptorProtosResponse {
option (buf.validate.message).cel = {
id: "len_file_descriptor_protos_equals_len_extensions",
message: "the length of file_descriptor_protos must equal the length of file_descriptor_proto_extensions if file_descriptor_proto_extensions is present",
expression: "this.file_descriptor_proto_extensions.size() == 0 || this.file_descriptor_protos.size() == this.file_descriptor_proto_extensions.size()"
};

// The FileDescriptorProtos representing the referenced Module, Label, or Commit.
repeated google.protobuf.FileDescriptorProto file_descriptor_protos = 1 [(buf.validate.field).required = true];
// Additional information about each FileDescriptorProto.
//
// We include this information in a separate message as FileDescriptorProtos are not
// extendable, and we do not want to modify the FileDescriptorProto shape.
//
// If present, the length of this field will be equal to the length of the file_descriptor_protos field, and each
// value corresponds to its corresponding value at the same index within file_descriptor_protos.
repeated FileDescriptorProtoExtension file_descriptor_proto_extensions = 2;
// The version of the compiler used to create the FileDescriptorProtos.
//
// This will match the semantics used within CodeGeneratorRequests.
google.protobuf.compiler.Version compiler_version = 3 [(buf.validate.field).required = true];
Comment on lines +128 to +129
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this imply we're doing something new here? I thought we used to always set the version to nil/unset since version of the Buf CLI weren't really comparable to versions of protoc.

}
Loading