Skip to content

Commit

Permalink
updated README
Browse files Browse the repository at this point in the history
  • Loading branch information
amansrivastava17 committed Aug 29, 2019
1 parent 40df89d commit cc90a1e
Showing 1 changed file with 11 additions and 1 deletion.
12 changes: 11 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,12 +85,22 @@ array([[ 0.48388457, -0.01327741, -0.76577514, ..., -0.54265064,
>>> vectors.shape
(2, 256, 768)
```
#### 5. Using Tokenizer
#### 5. Show embedding Tokens
```python
>>> en.tokenize(texts=['hello aman', 'how are you?'])
[['_hello', '_aman'], ['_how', '_are', '_you', '?']]
```

#### 6. Using your own tokenizer
```python
>>> texts = ['hello aman!', 'how are you']

# a naive whitespace tokenizer
>>> tokens = [s.split() for s in texts]
>>> vecs = en.encode(tokens, is_tokenized=True)
```



<h2 align="center" href="#supported-models">Supported Embeddings and Models</h2>

Expand Down

0 comments on commit cc90a1e

Please sign in to comment.