-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Iterator Guidelines
#Introduction
Google Cloud Platform API clients for Go should be as consistent as possible. Since Go has no standard iterator pattern, this document establishes guidelines for iterators.
Most iterators will result from the standard Google APIs List method with the pagination pattern: each call to a List method for a resource returns a sequence (“page”) of resource items (e.g. Books) along with a “next-page token” that can be passed to the List method to retrieve the next page.
Each List method will result in an iterator (which we call a List iterator) with two methods, one for individual items and one for pages. Iterators may also arise from other sources, like streaming RPCs or Cloud PubSub message subscriptions. These may or may not support page-by-page iteration.
Here is what iterators written according to these guidelines will look like to users. Here is the List iterator for the Book resource in the library example:
it := client.Books(ctx, shelfName)
for {
book, err := it.Next()
if err == library.Done {
break
}
if err != nil {
return err
}
process(book)
}Here's the same code using a switch:
it := client.Books(ctx, shelfName)
loop: for {
book, err := it.Next()
switch err {
case nil: process(book)
case library.Done: break loop
default: return err
}Here is what iteration by pages looks like:
it := client.Books(ctx, shelfName)
for {
books, err := it.NextPage()
if err != nil && err != library.Done {
return err
}
for _, b := range books {
process(b)
}
if err == library.Done {
break
}
}Here we retrieve the first page of 25 (or fewer) books and display the page and the next-page token:
it := client.Books(ctx, shelfName)
it.SetPageSize(25)
books, err := it.NextPage()
if err == nil {
display(books, it.NextPageToken())
}When the next-page token is handed back to us later (possibly in another process), we can get the next page:
it := client.Books(ctx, shelfName)
it.SetPageSize(25)
it.SetPageToken(token)
books, err := it.NextPage()
if err == nil || err == library.Done {
display(books, it.NextPageToken())
}An iterator should be represented by a type whose name ends in Iterator. If the iterator is a List iterator, the type's name should be ResourceIterator, E.g. BookIterator. The type should have at least one method, called Next. List iterators will also have a NextPage method. Next and NextPage are described below.
Example:
type BookIterator struct { ... }Typically, the client will have a single method that returns an iterator of a particular type. We will call this the creating method.
The name of the creating method for a List iterator should be the plural of the resource, e.g. Books (not ListBooks, which is a bit verbose for Go). For other kinds of iterators, the name of the creating method should be a plural noun, but a different name can be used if it makes more sense.
The first argument to the creating method should be a context.Context. The iterator will use that context throughout the iteration. In the unlikely event that neither the creating method nor the iterator make any RPCs, the context can be omitted.
The creating method may accept other arguments after the context, as needed.
In most cases, the creating method will simply create an instance of the iterator type and return it, leaving the initial work, including any RPCs, to the first call to Next. In other cases, a creating method may return an error along with the iterator.
An iterator over values of type T will have a method called Next that returns (T, error). For example,
func (it *BookIterator) Next() (*Book, error) { ... }Next will typically have no arguments, but it may in some cases. (For example, the Datastore iterator's Next method takes an argument into which it copies the entity.) None of the arguments should be a context, because Next should use the context passed when the iterator was created.
Following standard Go convention, if Next’s second return value is non-nil, then the first must be the zero value for T.
A special error value returned by Next signals the successful end of the iteration. This sentinel value will be named Done (unless that results in a conflict) and will be declared as a variable in the same package as the iterator. After Next returns Done, all subsequent calls to it will return Done.
If feasible, the user should be able to continue calling Next even if it returns an error that is not Done. If that is not feasible, it should be so documented.
The documentation comment for Next should be as follows:
Next returns the next result. Its second return value is Done if there are no more results. Once Next returns Done, all subsequent calls will return Done.
Internally, Next retrieves results in bulk. You can call SetPageSize as a performance hint to affect how many results are retrieved in a single RPC.
SetPageToken should not be called when using Next.
Next and NextPage should not be used with the same iterator.
Iterators that support pagination, such as List iterators, should also have a NextPage method that returns ([]Resource, error). For example,
func (it *BookIterator) NextPage() ([]*Book, error) { ... }If NextPage's second return value is neither nil nor Done (in other words, an actual error), then its first return value must be nil. If the second return value is Done, the first may be nil or a zero-length slice, or it may contain a final page of items. Hence loops with NextPage should check first for a true error, then process the list of items, and finally check for Done. See the third example above.
After NextPage returns Done, all subsequent calls to NextPage should return (nil, Done).
See the next section for the recommended documentation comment for NextPage.
Iterators that support pagination should have a SetPageSize method which sets the page size for all subsequent RPCs from the current iterator. It should take an int32 argument and have no return value:
func (it *BookIterator) SetPageSize(int32)Callers who use only the iterator's Next method should treat SetPageSize as a hint, much as a buffer size argument in an I/O method.
The documentation comment for SetPageSize should be as follows:
SetPageSize sets the page size for all subsequent calls to NextPage.
Page size semantics may differ across clients.
In those intended to reflect the underlying API closely (e.g. machine-generated clients), the page size provided by SetPageSize is a maximum; fewer items may be returned in subsequent NextPage calls. Indeed, it is valid for NextPage to return (r, nil) where len(r) == 0. In other words, there can be empty pages in the middle of the iteration. These clients should not impose a default page size; they should let the underlying service choose.
Other clients aspire to be a higher-level wrapper on top of the API, built to favor user convenience over faithfulness to the underlying RPCs. The NextPage methods in such clients should return exactly the page size (if that many items are available), to facilitate the common use case of displaying pages of items in a web or command-line interface. They should specify a default page size in a constant named DefaultPageSize, to be used if SetPageSize is not called or is called with a value less than 1.
For iterators that reflect the underlying API, the documentation comment for NextPage should be as follows:
NextPage returns the next page of results. It will return at most the number of results specified by the last call to SetPageSize. If SetPageSize was never called or was called with a value less than 1, the page size is determined by the underlying service.
NextPage may return a second return value of Done along with the last page of results.
After NextPage returns Done, all subsequent calls to NextPage will return (nil, Done).
Next and NextPage should not be used with the same iterator.
For iterators that return the exact page size, the first paragraph should instead read:
NextPage returns the next page of results. It will return exactly the number of results specified by the last call to SetPageSize, unless there are not enough results available. If SetPageSize was never called or was called with a value less than 1, it uses DefaultPageSize.
Google API calls that support pagination typically return a next-page token, which can be passed to a subsequent RPC to obtain the next page of results. Iterators built on such RPCs should provide two methods to support this, NextPageToken and SetPageToken. Note that for a simple loop implementing page-by-page iteration as in the example above, explicit use of the page token is unnecessary; the iterator will perform the necessary manipulations (the equivalent of SetPageToken(NextPageToken())) itself. The page token methods are only necessary when an iteration needs to be resumed later, possibly in another process.
The NextPageToken method should take no arguments and return the next-page token, typically a string:
func (it *BookIterator) NextPageToken() stringThe SetPageToken method should accept a string and have no return value:
func (it *BookIterator) SetPageToken(string) { ... }It should set the page token for the next RPC.
The documentation comment for NextPageToken should be as follows:
NextPageToken returns a page token that can be used with SetPageToken to resume iteration from the next page. It returns the empty string if there are no more pages.
The documentation comment for SetPageToken should be as follows:
SetPageToken sets the page token for the next call to NextPage, to resume the iteration from a previous point.
An iterator may require clean-up work to happen after it completes, or if the user abandons it before reaching the end. The iterator type should have a method of no arguments that performs this clean-up.
If the clean-up work can result in an error that should be returned, the method should be named Close. If no error is possible (or if the error is to be exposed in another way), the method should be named Stop:
func (it *CleanupWithErrorIterator) Close() error { ... }
func (it *CleanupNoErrorIterator) Stop() { ... }If an iterator does not require clean-up, it should define neither Close nor Stop.
If Close or Stop does not support being called multiple times, that should be documented.
If an iterator supports both Next and NextPage, it is required to be “smart” about interleaved calls to those methods. That is, they need not support any particular semantics for a call to Next followed by a call to NextPage, or vice versa. For the life a single iterator, we expect users to call only Next, or only NextPage.
The Next and NextPage methods are not required to be thread-safe, and Close (or Stop), Next and NextPage are not required to be concurrently callable.
Iterators may have fields and methods other than those described here.