Add the `SearchParamType` parameter to the `Map` method of `IElementIndexer`.

[whyfate]

Fields of type string within a SearchParameter definition may be classified as either string or token. Unless these types are handled separately, queries must account for the differing storage mechanisms of both types, necessitating or queries and thereby impacting performance.

spark/Libraries/Spark.Store.MongoDB/Search/CriteriaMongoExtensions.cs

Lines 352 to 353 in beb0c47

	var plainStringQueries = new List<FilterDefinition<BsonDocument>>{
	Builders<BsonDocument>.Filter.Type(parameterName, BsonType.String)};

spark/Libraries/Spark.Store.MongoDB/Search/CriteriaMongoExtensions.cs

Lines 383 to 386 in beb0c47

	return modifier == Modifier.NOT ?
	Builders<BsonDocument>.Filter.And(Builders<BsonDocument>.Filter.Not(arrayEqQuery),
	Builders<BsonDocument>.Filter.Not(noArrayEqQuery), Builders<BsonDocument>.Filter.Not(plainStringQuery))
	: Builders<BsonDocument>.Filter.Or(arrayEqQuery, noArrayEqQuery, plainStringQuery);

If the Map method accepts a SearchParamType parameter, the type can be evaluated during the ToExpressions phase:

if (searchParamType == SearchParamType.Token)
{
    var values = new List<IndexValue>
    {
        new IndexValue("code", new StringValue(element.Value)),
    };

    return ListOf(new CompositeValue(values));
}

[kennethmyhra]

FYI, I am looking into this one

[whyfate]

oh,this is really good news.

[kennethmyhra]

@whyfate can you try out the branch in this PR #1401 . You need to run the reindex operation for the plainQueryString to be filtered out

[whyfate]

@kennethmyhra Thank you very much for working on this issue and taking the time to put together this PR!

I pulled the PR and carefully went through the changes. This is a really great improvement, and I think it’s a very well-designed solution to the plainStringQueries issue. I especially appreciate that it also takes compatibility with previous versions into consideration.

However, I ran into a couple of issues while testing it:

1. A fresh database still requires a reIndex/migration

   I’m testing this with a completely new database, so I would expect it to work directly with the latest design without any additional steps. However, it currently still requires running reIndex (or a migration) first.

2. Queries by _id may become more expensive

   Since >3.1.0 changes the type of id from String to Token, which is certainly more consistent with the design. However, this also means that querying by _id now goes through the noArrayQueries + arrayQueries OR query used for token queries.

   With a large amount of data, I’m concerned that this could potentially have a significant performance impact. It might be worth considering whether _id queries could use a more direct query path.

[kennethmyhra]

@whyfate thanks for the feedback. I will look into 1. On 2 I do not really want to have special cases for certain search parameters. I will look into #476 separately and look at how expensive the noArrayQueries/arrayQueries are

[whyfate]

@kennethmyhra Thanks again. Regarding the noArrayQueries/arrayQueries, I’ve found a few clues that might help us understand what’s happening #243 .

spark/Libraries/Spark.Engine/Search/BaseElementIndexer.cs

Lines 83 to 102 in 24ecb8b

	/// <summary>
	/// Maps an element to a list of Expression.
	/// </summary>
	/// <param name="element">The element to be indexed.</param>
	/// <param name="searchParamType">The search parameter type this element is associated with.</param>
	/// <returns>A list of expression, and an empty List if no mapping occurred or was possible.</returns>
	public List<Expression> Map(Element element, SearchParamType searchParamType)
	{
	List<Expression> expressions = MapInternal(element);
	if (searchParamType != SearchParamType.Token)
	return expressions;
	return
	[
	.. expressions.Select(expression => expression is StringValue stringValue
	? new CompositeValue([new IndexValue("code", stringValue)])
	: expression
	)
	];
	}

This change seems to resolve the issue with noArrayQueries / arrayQueries. It looks like we may no longer need arrayQueries at all.