Implement the ModernBert model

[kozistr]

What does this PR do?

Close #457

• upgrade the tokenizer crate from 0.19.1 to 0.21.0 to address a ModernBert tokenizer issue.
• implement ModernBert model
  • it may work on CPU, CUDA (w/o FA), and MPS.
  • ModernBert uses local attention. however, I'm unfamiliar with candle_flash_attn and don't have any GPU to test FA2 w/ local attn, so the FlashModernBert implementation remains unsupported at this time.
• implement a classification head for ModernBert

Log

$ ./target/release/text-embeddings-router --model-id ./ModernBERT-base/ --port 8888 --pooling cls --dtype float32
2024-12-25T07:34:46.753673Z  INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "./Mod*******-*ase/", revision: None, tokenization_workers: None, dtype: Some(Float32), pooling: Some(Cls), max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "0.0.0.0", port: 8888, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: None, payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }
2024-12-25T07:34:46.817444Z  WARN text_embeddings_router: router/src/lib.rs:184: Could not find a Sentence Transformers config
2024-12-25T07:34:46.817472Z  INFO text_embeddings_router: router/src/lib.rs:188: Maximum number of tokens per request: 8192
2024-12-25T07:34:46.817622Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:28: Starting 8 tokenization workers
2024-12-25T07:34:46.883933Z  INFO text_embeddings_router: router/src/lib.rs:230: Starting model backend
2024-12-25T07:34:46.884247Z  INFO text_embeddings_backend_candle: backends/candle/src/lib.rs:239: Starting ModernBert model on Cpu
2024-12-25T07:34:47.138974Z  WARN text_embeddings_router: router/src/lib.rs:258: Backend does not support a batch size > 4
2024-12-25T07:34:47.139002Z  WARN text_embeddings_router: router/src/lib.rs:259: forcing `max_batch_requests=4`
2024-12-25T07:34:47.139930Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1812: Starting HTTP server: 0.0.0.0:8888
2024-12-25T07:34:47.139955Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1813: Ready
2024-12-25T07:34:52.701893Z  INFO embed{total_time="115.486302ms" tokenization_time="322.4µs" queue_time="363.6µs" inference_time="114.688702ms"}: text_embeddings_router::http::server: router/src/http/server.rs:714: Success

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@OlivierDehaene OR @Narsil