About text embedding from text tokens.

[ZhaojunCP]

Thank you for your work! May I ask if you input the tokens of the text into text-embedding-ada-002? As far as I know, text-embedding-ada-002 requires a string rather than a list of integer like tokens. Could I get your explanation? Thank you.

[DVampire]

Our OpenAI LLM provider primarily refers to LangChain's implementation. The specific reference code and reasons are as follows:

1. The LangChain code is https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/embeddings/openai.py.
   At line 397, LangChain states that it primarily refers to OpenAI's codebook.
2. The openai codebook link is https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb. They wrote it this way primarily to address embedding texts that are longer than the model's maximum context length.

Please refer to the above code. If you have any questions, feel free to contact us. Thanks.