26186 returns 261872T00 but not 261860000, why?

[stravanato]

I have this strange problem with search results.
I'm not talking about autocomplete results, which seem to work, but search results you get when you press enter.

Searching partial product code as 26186 I get:
261872T00, 26187M250 and 26187M500
But I don't get 261860000

I get 261860000 only if I type 2618600
I thought that an exact substring should be found first and then possible little changes.
Is there any configuration I should look at?

[rbayet]

Hello @stravanato,

There are quite a lot of parameters that can dictate how and why an EAN-like string would get matches.

First things first, please note that there is no notion of "substring" matching.
What can occur is the fuzziness doing its magic when an fuzzy query is performed.
A fuzzy query is performed when Elasticsuite determines that what the user searched is not present "exactly" in the search engine.

Secondly, what is present in the engine is determined by the (text) analyzer that you can configured on a per-attribute level.
The "standard" analyzer does a bit of "alphanumerical string splitting" and split strings along special characters.
So there is a pretty good chance that the following strings are generating "tokens" (search engine "words") like these :

• 261872T00 : should generate "261872", "T", "00" but "261872T00" is probably preserved too
• 26187M250 : should generate "26187", "M", "250" but "26187M250" is probably preserved too
• 26187M500 : should generate "26187", "M", "500" but "26187M250" is probably preserved too
• 261860000 : should only generate "261860000"

You can check for yourself using the "Elasticsuite > Analysis" screen in your Magento admin.

The "reference" analyzer which we recommend using for "EAN-like" attributes (like sku) works a bit like that too.

But long story short, if your attribute is using the "standard" analyzer, when searching "26186", there will be no exact match amongst all the tokens I listed above.
So Elasticsuite enables the fuzziness which will allow up to 2 characters change per token to match the user request

• 261872T00 will match through the token "261872" (removal of terminal 2, changing the 7 by a 6)
• 26187M250 will match through the token "26187" (changing the 7 by a 6)
• same thing for 26187M500
• on the other hand there are too many changes to go from 261860000 to 26186 (4 removals of trailing "0")

Give a try to the "Elasticsuite > Analysis" to see if the "reference" analyzer would be able to generate from 261860000 a token "not too far" from 26186 (my guess is probably not).
If that's the case, I would encourage you to enable the following "experimental" settings in the "Elasticsuite > Search Relevance" configuration screen

• Relevance configuration > Exact Match Configuration > [Experimental] Use default analyzer in exact matching filter query
• Relevance configuration > Exact Match Configuration > [Experimental] Use reference analyzer in exact matching filter query
• Spellchecking configuration > Term Vectors Configuration > [Experimental] Use reference analyzer in term vectors
• Spellchecking configuration > Term Vectors Configuration > [Experimental] Use all tokens from term vectors

If that's not the case, you can try :

• either to tweak the "reference" analyzer through the XML configuration to allow ignoring trailing 0, while keep the complete form, so "261860000" would be indexed as both "261860000" and "26186" (and "261872T00" as both "261872T00" and 261872T", and so on)
• or to assign to your attribute the "standard_edge_ngram" analyzer which adds (to the cost of index size and a bit of performance) the "standard" analyzer the ability to perform real autocompletion
  • "261860000" will still be indexed as "261860000" but also "261", "2618", "26186", "261860", "2618600" and "26186000"
  • if you go this route, then enabling the following experimental settings in the "Elasticsuite > Search Relevance" menu will be mandatory too
    • Relevance configuration > Exact Match Configuration > [Experimental] Use default analyzer in exact matching filter query
    • Spellchecking configuration > Term Vectors Configuration > [Experimental] Use all tokens from term vectors
    • Spellchecking configuration > Term Vectors Configuration > [Experimental] Use edge ngram analyzer in term vectors

Regards,

[stravanato]

@rbayet thanks a lot.
I'm afraid that my elasticsuite version (ElasticSuite Open Source ver. 2.10.11) is missing most of the configurations you refer to.
So I was trying to do the tweak on reference analyzer through the XML configuration to allow ignoring trailing 0, while keep the complete form.
But there must something I don't understand:
I made a new elasticsuite_analysis.xml file and copied inside the configuration I found for "reference" analyzer
<?xml version="1.0"?>
<analysis xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Smile_ElasticsuiteCore:etc/elasticsuite_analysis.xsd">
<analyzers>
<analyzer name="reference" tokenizer="standard" language="default">
<filters>
<filter ref="ascii_folding" />
<filter ref="trim" />
<filter ref="reference_word_delimiter" />
<filter ref="lowercase" />
<filter ref="elision" />
<filter ref="reference_shingle" />
</filters>
<char_filters>
<char_filter ref="html_strip" />
</char_filters>
</analyzer>
</analyzers>
</analysis>

Should I add filters? Can you please point me to the direction?

[rbayet]

Hello @stravanato,

Concerning

So I was trying to do the tweak on reference analyzer through the XML configuration to allow ignoring trailing 0, while keep the complete form.

Come to think of it, you don't really need to "keep the complete" form since when you use a given field/attribute for searching, Elasticsearch will re-apply the same analyzer to your input string.
So if you push "261860000" through an analyzer that trims trailing 0s, it will index "26186".
But when you search for "261860000" it will also trim the trailing 0s, so it will actually search for "26186" and you'll get a match.

Concerning your exact question, yes, the idea was to add a filter after the "reference_word_delimiter" and to use a variation of the exiting "trim" filter by

1. adding a custom "trim_zeros" at the analysis/filters level
2. adding that filter in your "reference" analyzer at the analysis/analyzers level

BUT the trim token filter does not accept any parameters: there is no way to tell him to trim (leading and) trailing zeros.

So either you try your luck with a "pattern replace" token filter using inspiration from https://stackoverflow.com/questions/32569369/elasticsearch-how-to-remove-leading-zeros-from-string and using instead the pattern (.*)0+$ to remove trailing zeros.
(You might need a slight variation of this new filter if you also want to remove leading zeros).

Or, you could also do it in PHP using a dedicated data source to be declared in the DI (see src/module-elasticsuite-catalog/etc/di.xml)

    <type name="Smile\ElasticsuiteCore\Index\DataSourceResolver">
        <arguments>
            <argument name="datasources" xsi:type="array">
                <item name="catalog_product" xsi:type="array">
                    <item name="prices" xsi:type="object">Smile\ElasticsuiteCatalog\Model\Product\Indexer\Fulltext\Datasource\PriceData</item>
                    <item name="categories" xsi:type="object">Smile\ElasticsuiteCatalog\Model\Product\Indexer\Fulltext\Datasource\CategoryData</item>
                    <item name="attributes" xsi:type="object">Smile\ElasticsuiteCatalog\Model\Product\Indexer\Fulltext\Datasource\AttributeData</item>
                    <item name="stock" xsi:type="object">Smile\ElasticsuiteCatalog\Model\Product\Indexer\Fulltext\Datasource\InventoryData</item>
                    <item name="searchPositions" xsi:type="object">Smile\ElasticsuiteCatalog\Model\Product\Indexer\Fulltext\Datasource\SearchPositionData</item>
                    <item name="ignoreOutOfStockPositions" xsi:type="object">Smile\ElasticsuiteCatalog\Model\Product\Indexer\Fulltext\Datasource\IgnoreOutOfStockPositionsData</item>
                </item>
                <item name="catalog_category" xsi:type="array">
                    <item name="attributes" xsi:type="object">Smile\ElasticsuiteCatalog\Model\Category\Indexer\Fulltext\Datasource\AttributeData</item>
                </item>
            </argument>
        </arguments>
    </type>

You can take inspiration from Smile\ElasticsuiteCatalog\Model\Product\Indexer\Fulltext\Datasource\IgnoreOutOfStockPositionsData which runs when all the attribute and stock data has been collected.
Let's imagine your attribute is the sku, you could do something like this to remove both leading and trailing zeros

foreach ($indexData as &$productData) {
      if (array_key_exists('sku', $productData)) {
          $productData['sku'] = trim($productData['sku'], "0");
      }
  }

Regards,

[stravanato]

Thanks @rbayet again.
I did the PHP way, made a a module with a dedicated data source declared in DI. Just one method addData() with the foreach cicle you suggested.
Looks like it worked!!!!
It looks more like a "substring" matching: if I search for 26186 I get only 261860000, and searching 26187 I get 261872T00, 26187M250 and 26187M500, as expected.
That's fantastic, I'm really grateful to you.

PS: should I post the complete code?

[rbayet]

Hello @stravanato,

You're welcome.
You can post your code if you feel like it.

Please be aware though that the code I provided might have a few hiccups :

• it can happen that when you enter your datasource, the $productData['sku'] record might be an array instead of a scalar value: that will happen on composite products (configurables, bundles) and so at the very least it will through a PHP warning
  • so you should accomodate for that case
• contrary to the analyzer approach, the datasource one only applies to the indexed data, not what the user searches
  • so while if the user searches for 26186 he'll find the product with sku 261860000, it will not be the case if he searches for 261860000
  • it could be partially addressed if you make your datasource return both the original and trimmed values of your SKUs (Elasticsearch natively allow any "scalar" field to be actually multivalued)
    • buuuuuuuuuuut it still won't work in every case: searching for 261860 might also fail

So while in essence the analyzer is the best approach, you could be satisfied (it all depends on your users search behavior, really) with the PHP version in this form

            foreach ($indexData as &$productData) {
                if (array_key_exists('sku', $productData)) {
                    if (!is_array($productData['sku'])) {
                        $productData['sku'] = [$productData['sku']];
                    }
                    $skus = [];
                    foreach ($productData['sku'] as $sku) {
                        $skus[] = $sku;
                        $skus[] = trim($sku, "0");
                    }
                    $productData['sku'] = $skus;
                }
            }

Regards

[stravanato]

Great Richard! Thank you
I got the point and made the changes.
I think it can be enough now but I understand that if I have sku like 1234567890 and I search for 1234 I won't get it.
I was wondering if it's a good idea to split skus in 2 or 3 chunks and return them in the $productData array, given that fuzziness allow up to 2 characters change per token, a good percentage of partial searches could be covered.

[rbayet]

Hello @stravanato,

If in the end you would also like to match 1234567890 from 1234 and if 1234567890 does not exist match 1239[something] (through fuzziness), then you can try your approach or a more straight-forward approach is this one mentionned at the bottom of my initial reply

If that's not the case, you can try :

• either to tweak the "reference" analyzer [...]
• or to assign to your attribute the "standard_edge_ngram" analyzer which adds (to the cost of index size and a bit of performance) the "standard" analyzer the ability to perform real autocompletion
  • "261860000" will still be indexed as "261860000" but also "261", "2618", "26186", "261860", "2618600" and "26186000"
  • if you go this route, then enabling the following experimental settings in the "Elasticsuite > Search Relevance" menu will be mandatory too
    • Relevance configuration > Exact Match Configuration > [Experimental] Use default analyzer in exact matching filter query
    • Spellchecking configuration > Term Vectors Configuration > [Experimental] Use all tokens from term vectors
    • Spellchecking configuration > Term Vectors Configuration > [Experimental] Use edge ngram analyzer in term vectors

Regards