feat (provider/togetherai): Add TogetherAI provider. #3781
+4,164
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shaper
force-pushed
the
shaper/pr/together
branch
from
ec3e6b1
to
feb0fe3
Compare
shaper
force-pushed
the
shaper/pr/together
branch
from
fcffdd5
to
8c6420c
Compare
shaper
force-pushed
the
shaper/pr/together
branch
from
8c6420c
to
10915e2
Compare
shaper
changed the title
shaper
force-pushed
the
shaper/pr/together
branch
from
10915e2
to
6d44bf7
Compare
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+9
to
+24
| export function mapOpenAICompatibleCompletionLogProbs( | ||
| logprobs: OpenAICompatibleCompletionLogProps | null | undefined, | ||
| ): LanguageModelV1LogProbs | undefined { | ||
| return logprobs?.tokens.map((token, index) => ({ | ||
| token, | ||
| logprob: logprobs.token_logprobs[index], | ||
| topLogprobs: logprobs.top_logprobs | ||
| ? Object.entries(logprobs.top_logprobs[index]).map( | ||
| ([token, logprob]) => ({ | ||
| token, | ||
| logprob, | ||
| }), | ||
| ) | ||
| : [], | ||
| })); | ||
| } |
There was a problem hiding this comment.
Let's remove logprobs from the compatible provider. I'm leaning towards moving it into a provider extension since it's rarely supported.
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+18
to
+19
| // TODO(shaper): This is really model-specific, move to config or elsewhere? | ||
| defaultObjectGenerationMode?: 'json' | 'tool' | undefined; |
There was a problem hiding this comment.
can you make it a constructor parameter (in the options) for the model that is then defined in the providers? (which would have that knowledge)
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+148
to
+151
| // TODO(shaper): Review vs. OpenAI impl here. | ||
| throw new UnsupportedFunctionalityError({ | ||
| functionality: 'object-json mode', | ||
| }); |
There was a problem hiding this comment.
copy what openai has including schemas
lgrammel
reviewed
Nov 20, 2024
| constructor( | ||
| modelId: OpenAICompatibleChatModelId, | ||
| settings: OpenAICompatibleChatSettings, | ||
| config: OpenAICompatibleChatConfig, |
There was a problem hiding this comment.
include the default object generation mode in config
lgrammel
reviewed
Nov 20, 2024
| completionTokens: response.usage.completion_tokens, | ||
| }, | ||
| finishReason: mapOpenAICompatibleFinishReason(choice.finish_reason), | ||
| logprobs: mapOpenAICompatibleCompletionLogProbs(choice.logprobs), |
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+343
to
+349
| logprobs: z | ||
| .object({ | ||
| tokens: z.array(z.string()), | ||
| token_logprobs: z.array(z.number()), | ||
| top_logprobs: z.array(z.record(z.string(), z.number())).nullable(), | ||
| }) | ||
| .nullish(), |
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+370
to
+376
| logprobs: z | ||
| .object({ | ||
| tokens: z.array(z.string()), | ||
| token_logprobs: z.array(z.number()), | ||
| top_logprobs: z.array(z.record(z.string(), z.number())).nullable(), | ||
| }) | ||
| .nullish(), |
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+132
to
+133
| // 'openai-organization': 'test-organization', | ||
| // 'openai-project': 'test-project', |
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+111
to
+112
| // organization: 'test-organization', | ||
| // project: 'test-project', |
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+9
to
+12
| /** | ||
| Override the parallelism of embedding calls. | ||
| */ | ||
| supportsParallelCalls?: boolean; |
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+4
to
+19
| // TODO(shaper): Reconcile this with openai-error.ts. We derived from `xai`. | ||
|
|
||
| export const openAICompatibleErrorDataSchema = z.object({ | ||
| code: z.string(), | ||
| error: z.string(), | ||
| }); | ||
|
|
||
| export type OpenAICompatibleErrorData = z.infer< | ||
| typeof openAICompatibleErrorDataSchema | ||
| >; | ||
|
|
||
| export const openAICompatibleFailedResponseHandler = | ||
| createJsonErrorResponseHandler({ | ||
| errorSchema: openAICompatibleErrorDataSchema, | ||
| errorToMessage: data => data.error, | ||
| }); |
There was a problem hiding this comment.
make provider-specific. different providers have different structures. have default that matches openai
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+19
to
+21
| L extends string = string, | ||
| C extends string = string, | ||
| E extends string = string, |
There was a problem hiding this comment.
Prefer full words for generic in upper case, e.g. CONTEXT
lgrammel
reviewed
Nov 20, 2024
Comment on lines
+94
to
+96
| // TODO(shaper): | ||
| // - consider throwing if baseUrl, name, sufficient api key info not available | ||
| // - force only 'compatible' -- look into whether we can remove some 'strict' logic/configs entirely |
There was a problem hiding this comment.
yes - goal is that this works w/ compatible mode but is more flexible. then eventually openai provider should only be strict mode and we can simplify
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Built on top of a new
openai-compatiblepackage for better code sharing as we add more top level providers that follow this pattern.