Example-based #165

AntoninPoche · 2024-09-25T09:52:33Z

Introduce the Example-based module

This pull request introduces example-based methods from four families of methods. Namely:

Similar Examples
Semi-factuals
Counterfactuals
Prototypes

The API

The API is common for the four families, and it has three steps:

Construct a projection function with the model.
Initialize the method with a dataset, projection, and other parameters.
Explain given samples with a local explanation.

projection = ProjectionMethod(model)

explainer = ExampleMethod(
    cases_dataset=cases_dataset, 
    k=k,
    projection=projection,
    case_returns=case_returns,
    distance=distance,
)

examples = explainer.explain(inputs, targets)

…que into antonin/example-based-merge

…operator

…-ai/xplique into antonin/example-based-merge

AntoninPoche

First part of my review. I will wait to review all to make a final comment.

AntoninPoche · 2024-09-30T09:04:46Z

README.md

+| `KLEORSimMiss` | Semi Factuals | [KLEOR](../semifactuals/kleor/) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) |
+| `KLEORGlobalSim` | Semi Factuals | [KLEOR](../semifactuals/kleor/) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) |
+||||
+| `ProtoGreedy` | Prototypes | [ProtoGreedy](../prototypes/proto_greedy/) | **TODO** |


Add link when tutorial is ready.

AntoninPoche · 2024-09-30T09:05:24Z

TUTORIALS.md

@@ -20,6 +20,8 @@ Here is the lists of the available tutorial for now:
 | Metrics                         | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1WEpVpFSq-oL1Ejugr8Ojb3tcbqXIOPBg)            |
 | Concept Activation Vectors      | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1iuEz46ZjgG97vTBH8p-vod3y14UETvVE)            |
 | Feature Visualization           | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1st43K9AH-UL4eZM1S4QdyrOi7Epa5K8v)            |
+| Example-Based Methods           | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF)            |
+| Prototypes                      | **TODO**            |


Add a link when the tutorial is ready.

AntoninPoche · 2024-09-30T09:05:53Z

TUTORIALS.md

+| **Tutorial Name**                      | Notebook                                                                                                                                                |
+| :------------------------------------- | :-----------------------------------------------------------------------------------------------------------------------------------------------------: |
+| Example-Based Methods: Getting started | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) |
+| Prototypes: Getting started            | **TODO**                                                                                     |


Add a link when the tutorial is ready.

AntoninPoche · 2024-09-30T09:07:06Z

docs/api/example_based/api_example_based.md

+# API: Example-based
+
+- [**Example-based Methods**: Getting started](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) <sub> [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) </sub>
+- [**TODO: Add the Getting Started on Prototypes**]()


Add a link when the tutorial is ready.

docs/api/example_based/api_example_based.md

xplique/example_based/base_example_method.py

lucashervier

I think we have done a nice first draft. I have made some comments on the points I do not want to forget to discuss with you. I have made very few comments on prototypes as I am not familiar with the methods.

lucashervier · 2024-09-30T12:45:36Z

.pylintrc

We removed it to have all configurations in the setup.cfg file instead. Why do we add it back?

I will try to remove it and see it works again.

lucashervier · 2024-09-30T12:46:46Z

TUTORIALS.md

@@ -20,6 +20,8 @@ Here is the lists of the available tutorial for now:
 | Metrics                         | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1WEpVpFSq-oL1Ejugr8Ojb3tcbqXIOPBg)            |
 | Concept Activation Vectors      | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1iuEz46ZjgG97vTBH8p-vod3y14UETvVE)            |
 | Feature Visualization           | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1st43K9AH-UL4eZM1S4QdyrOi7Epa5K8v)            |
+| Example-Based Methods           | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF)            |
+| Prototypes                      | **TODO**            |


We need to actually add the link of the notebook here

lucashervier · 2024-09-30T12:46:54Z

TUTORIALS.md

+| **Tutorial Name**                      | Notebook                                                                                                                                                |
+| :------------------------------------- | :-----------------------------------------------------------------------------------------------------------------------------------------------------: |
+| Example-Based Methods: Getting started | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) |
+| Prototypes: Getting started            | **TODO**                                                                                     |


lucashervier · 2024-09-30T12:47:09Z

docs/api/example_based/api_example_based.md

+# API: Example-based
+
+- [**Example-based Methods**: Getting started](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) <sub> [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) </sub>
+- [**TODO: Add the Getting Started on Prototypes**]()


lucashervier · 2024-09-30T12:47:29Z

docs/api/example_based/api_example_based.md

+    | `KLEORSimMiss` | Semi Factuals | [KLEOR](../semifactuals/kleor/) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) |
+    | `KLEORGlobalSim` | Semi Factuals | [KLEOR](../semifactuals/kleor/) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1gA7mhWhWzdKholZWkTvAg4FzFnzS8NHF) |
+    ||||
+    | `ProtoGreedy` | Prototypes | [ProtoGreedy](../prototypes/proto_greedy/) | **TODO** |


lucashervier · 2024-09-30T14:00:36Z

xplique/commons/data_conversion.py



 def tensor_sanitize(inputs: Union[tf.data.Dataset, tf.Tensor, np.ndarray],
-                    targets: Optional[Union[tf.Tensor, np.ndarray]]) -> Tuple[tf.Tensor, tf.Tensor]:
+                    targets: Union[tf.Tensor, np.ndarray]) -> Tuple[tf.Tensor, tf.Tensor]:


We can't use None anymore?

It was never the case for classic attribution methods. The typing did not correspond to what the method supported.

lucashervier · 2024-09-30T14:05:53Z

xplique/example_based/datasets_operations/convert_torch_to_tf.py

If I am correct, there are no tests for this script.

All functions in this file are tested through the tests/example_based/test_torch.py file.

lucashervier · 2024-09-30T14:07:51Z

xplique/example_based/datasets_operations/tf_dataset_operations.py

+    return False
+
+
+def is_not_shuffled(dataset: Optional[tf.data.Dataset]) -> bool:


The name is not really well suited. In addition, I would rather use an iterator to check that is two instantiations of the same dataset..

I agree on the name, but what do you mean on the iterator?

xplique/example_based/projections/base.py

xplique/example_based/projections/commons.py

AntoninPoche

Second part of my review.

xplique/example_based/counterfactuals.py

xplique/example_based/projections/commons.py

xplique/example_based/projections/hadamard.py

xplique/example_based/projections/latent_space.py

AntoninPoche

Third part of the review, there are many small modifications in docstrings.

xplique/example_based/prototypes.py

xplique/example_based/similar_examples.py

xplique/plots/image.py

AntoninPoche

To summarize remaining comments:

No link or wrong link for prototypes tutorial.
Some questions on example based test, for pytorch ones and the ones regarding datasets operations.
Questions on the FilterKNN on how the mask is applied.
A pylint problem too-many-positional-arguments. If resolved, this could lead to loose retro-compatibility. See pylint doc on this warning for more informations.

AdelinCO and others added 30 commits February 14, 2024 16:26

example-based: add cole method

c8df34e

tests: unit testing cole

c331801

example based: introduce projections

e1e4d82

example based: introduce search methods

eda3317

example based: add base class

4b0a340

cole: update and improve

0daf281

example based tests: update and complete

62a621d

plots: add image visualization for example based

2aa133d

commons: add operations for tf dataset

bed4747

pylint: disable similarities for signatures

9f9acfb

example based: introduce base example method abstraction

d29f92f

example based: adapt similar examples

69fef12

example based: adapt cole

539c387

example based: adapt tests

7a58c9f

base example method: dataset projections in projections

2255896

base projection: dataset projections in projections

ead19ea

latent space projection: dataset projections in projections

48bc853

attribution projection: dataset projections in projections

fcacfd1

example based: introduce hadamard projection

4dbb6b6

base search method: remove projection from search

6e8ddb1

knn search method: remove projection from search

5abb298

cole: add hadamard product possibility

92056f1

projections: factorize model splitting

3f69e7b

example based projections: fuse custom projection with base class

5cdf1c1

projections tests: adapt to changes and complete

0c141bb

similar examples tests: adapt to changes

56ddd00

cole tests: adapt to changes and add hadamard

c44d256

tf operations: add get device for dataset mapping

445fce1

feat: add a new KNN object and improve distance computation efficiency

7cfa2f3

tests: add tests for KNNs

52d1c6a

ChafikBak and others added 15 commits September 19, 2024 13:33

docs: update prototypes

0f92127

Merge branch 'antonin/example-based-merge' of github.com:deel-ai/xpli…

35e1962

…que into antonin/example-based-merge

projections: correct from splitted model

0e0ab2e

projections: correct target free operator

9662bca

projections: small fix to base

4ddb859

commons: sanitize inputs targets disable pylint warning

8d9975d

tests projections: add tests for from splitted model and target free …

4f94e6b

…operator

plots: small fix

f515f14

base example methods: projection can now be none

f8bc136

example based: small pylint fixes

f8bb9f8

search methods: small pylint fixes

cb8eee9

prototypes: optimize and harmonize

f436713

tests prototypes: adapt to prototypes changes and complete

145f0d5

Merge branch 'antonin/example-based-merge' of https://github.com/deel…

222527a

…-ai/xplique into antonin/example-based-merge

prototypes: small fix

c8a0abe

AntoninPoche requested review from lucashervier and ChafikBak September 25, 2024 09:53

AntoninPoche added 2 commits September 25, 2024 16:05

ci: restrict numpy version and specify torch test

97fd2cf

test example based: small fix

a4414dd

AntoninPoche self-assigned this Sep 26, 2024

AntoninPoche commented Sep 30, 2024

View reviewed changes

rise: fix pylint

4debfd6

lucashervier requested changes Sep 30, 2024

View reviewed changes

AntoninPoche commented Oct 1, 2024

View reviewed changes

AntoninPoche commented Oct 2, 2024

View reviewed changes

AntoninPoche added 3 commits October 3, 2024 12:28

all: resolve part of example based pr comments

ef7db46

example based: small fix

b684ad3

example based: small fix

a876d7e

AntoninPoche commented Oct 3, 2024

View reviewed changes

example based: small fix

3bbf406

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example-based #165

Example-based #165

AntoninPoche commented Sep 25, 2024 •

edited

Loading

AntoninPoche left a comment

AntoninPoche Sep 30, 2024

AntoninPoche Sep 30, 2024

AntoninPoche Sep 30, 2024

AntoninPoche Sep 30, 2024

lucashervier left a comment

lucashervier Sep 30, 2024

AntoninPoche Oct 2, 2024

lucashervier Sep 30, 2024

lucashervier Sep 30, 2024

lucashervier Sep 30, 2024

lucashervier Sep 30, 2024

lucashervier Sep 30, 2024

AntoninPoche Oct 2, 2024

lucashervier Sep 30, 2024

AntoninPoche Oct 1, 2024

lucashervier Sep 30, 2024

AntoninPoche Oct 1, 2024

AntoninPoche left a comment

AntoninPoche left a comment

AntoninPoche left a comment

		return False


		def is_not_shuffled(dataset: Optional[tf.data.Dataset]) -> bool:

Example-based #165

Are you sure you want to change the base?

Example-based #165

Conversation

AntoninPoche commented Sep 25, 2024 • edited Loading

Introduce the Example-based module

The API

AntoninPoche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucashervier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AntoninPoche left a comment

Choose a reason for hiding this comment

AntoninPoche left a comment

Choose a reason for hiding this comment

AntoninPoche left a comment

Choose a reason for hiding this comment

AntoninPoche commented Sep 25, 2024 •

edited

Loading