[SSPN][Migration] Affiliation matching values

[zzacharo]

Context

ROR organization API docs: https://ror.readme.io/docs/api-affiliation

ROR matching algorithms

PHRASE: the entire phrase matched to a variant of the organization's name
COMMON TERMS: the matching was done by comparing the words separately
FUZZY: the matching was done by fuzzy-comparing the words separately
HEURISTICS: "University of X" was matched to "X University"
ACRONYM: matched by acronym
EXACT: exact match of the entered string in name, aliases, labels or acronyms fields

We used the ROR v1 API to analyze the following scenarios:

• API returned a high confident result using all matching algorithms
• API returned a high confident result using only the EXACT matching algorithm

Results

• The file affiliations_exact_matching_algorithm.csv gives you the highly confident results of only the records matched only with the EXACT algorithm.
• The file affiliations_exclude_exact_matching_algorithm.csv gives you the highly confident results of only the records matched with any algorithm other than the EXACT.

Action

• We need an evaluation of how many False positives we have in the 2 attached files separately so we can have an idea on which criteria to use to identify values (confident matches) that will replace the legacy arbitrary input and will not require curation.
• To analyze the false positives we would need to check manually the 2 files.

[PaulinaBaranowska]

Some thoughts:
Affiliations can be input through the form or manually or in the controlled INSPIRE format (ICN).
Perhaps we could split the three different cases:
When the affiliation comes from autocomplete though a submission form, it has the country code in brackets and an email address, and there is the CCID in $$0
When the affiliation has been submitted manually through a submission form, there will not be an email associated
When the affiliation is in the INSPIRE format, the record will have 035__a:oai:inspirehep.net:* or a thesis

Maybe we could look at this in these three groups? For things that are coming through autocomplete (which is coming from Foundation?), the strings are often poor, maybe we could use extra data available in Foundation for the matching?

For INSPIRE, we often have RORs already, so we can do some matching there.

[zzacharo]

Hey @PaulinaBaranowska thanks a lot for your input, I will have a look and come back to you. In either case, we would really appreciate also some feedback on the ROR API responses before we would need in any case when we try to disambiguate the free text values.

[PaulinaBaranowska]

@zzacharo Would it be possible to provide us the example .csv files with an extra column with the name of the institution from ROR that matches the ROR identifier? It would make comparing much easier and faster.

[zzacharo]

Hey @PaulinaBaranowska I updated the files with a new column called ror_match_info where you get the ROR information regarding each match.

[zzacharo]

Some thoughts: Affiliations can be input through the form or manually or in the controlled INSPIRE format (ICN). Perhaps we could split the three different cases: When the affiliation comes from autocomplete though a submission form, it has the country code in brackets and an email address, and there is the CCID in $$0 When the affiliation has been submitted manually through a submission form, there will not be an email associated When the affiliation is in the INSPIRE format, the record will have 035__a:oai:inspirehep.net:* or a thesis

Maybe we could look at this in these three groups? For things that are coming through autocomplete (which is coming from Foundation?), the strings are often poor, maybe we could use extra data available in Foundation for the matching?

For INSPIRE, we often have RORs already, so we can do some matching there.

I created #260 to address improvements on affiliation matching as in SSPN there is no presence of 035__a:oai:inspirehep.net:*