Add Support for Reindexing and Incremental Indexing

[dmayboroda]

Currently, the implementation indexes all contents of a folder (with recursive crawling). Ideally, we need to add the ability to:
1. Track the appearance of new files and add them to the index.
2. Detect changes in existing files and re-add them to the index.

This will ensure that the index stays up to date without the need for reindexing the entire folder.