default embedding

[nashid]

If we do not provide embedding like word2vec, how does it know to represent the words?

Does it use one hot encoding by default or ngram, CBOW, skip grams?

[luozhouyang]

No. If you do not provide the pretrained embeddings, it will create an trainable variable, and initialize it by some algorithm. When you train the model on your data, this variable will be updated too.

[nashid]

@luozhouyang I understand if we do not provide the pre-trained embedding it uses the default implementation of embedding in this framework.

However, I would like to know what algorithm is used to build the embedding.

[luozhouyang]

Word embeddings here are actually an 2-d tensor, with shape (vocab_size, embedding_size).
This tensor will be updated along with other params by BP.

[nashid]

@luozhouyang I understand this. But what algorithm it is using (like word2vec, gloVe, ...)?

[luozhouyang]

No special algorithm is used. Not word2vec, not GloVe, just a learnable 2-d matrix.