[SOLR-17726] Fix CloudMLTQParser to support copyField in qf

[ilariapet]

https://issues.apache.org/jira/browse/SOLR-17726

Description

When using CloudMLTQParser, fields that are populated exclusively via copyField are not taken into account when constructing the MoreLikeThis query.

This happens because CloudMLTQParser relies on a RealTime Get (/get) request to retrieve the source document by ID, and the document returned by RealTime Get does not include fields generated via copyField (i.e. are not part of the original SolrIDocument).

As a result, even if the copyField target is stored and has proper termVectors configured, CloudMLTQParser skips the field silently, and the MLT query ends up empty.

Solution

The issue was addressed by introducing a helper method getFieldOrCopyFieldValues that retrieves field values from the source document returned by RealTime Get.

If a field passed in the qf parameter of the MLT query is not directly present in the document (as is the case for fields populated exclusively via copyField), the method looks up its copyField sources in the schema and aggregates their values. These values are then used to construct the input for MoreLikeThis#like, ensuring that all content (including that of copyField-derived fields) is properly considered when building the MLT query.

Tests

Four tests were added to cover different scenarios involving copyField usage in the qf parameter of the MoreLikeThis query:

• testUsesACopyFieldInQf_shouldUseTheSourceFieldAndReturnResults
  Verifies that when a copyField destination is used in the qf parameter, the MLT query correctly retrieves values from the source field(s) and returns relevant results.
• testCopyFieldSourceMissing_shouldReturnNoResults
  Ensures that no results are returned when the copyField source field is missing in the source document.
• testCopyFieldDestinationNotStored_shouldReturnResults
  Confirms that even if the copyField destination field is not stored, results can still be returned as long as the source field contains the relevant content.
• testCopyFieldDestinationMultiple_shouldReturnResults
  Validates that when multiple source fields map to a single copyField destination, their values are correctly combined to construct the MLT query, and appropriate results are returned.

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide

[epugh]

This all makes sense... Two things:

1. Instead of working around how RTG works, would it make sense to have RTG return copyField values instead and not need this? I don't know, there could be good technical reasons why RTG does it what it does....

2. I love the descriptive text for the tests that are in your PR. It would be nice to preserve that as a comment in the actual CloudMLTQParserTest.java file for the next person! I sometimes struggle to understand why a test exists..

[epugh]

Love the description. It could also be a javadoc right? On the method? But now I am just nit picking!

[epugh]

Looks like we need gradlew tidy

[ilariapet]

This all makes sense... Two things:

1. Instead of working around how RTG works, would it make sense to have RTG return copyField values instead and not need this? I don't know, there could be good technical reasons why RTG does it what it does....
2. I love the descriptive text for the tests that are in your PR. It would be nice to preserve that as a comment in the actual CloudMLTQParserTest.java file for the next person! I sometimes struggle to understand why a test exists..

Thanks a lot for the feedback!

Regarding the first point, according to the documentation, the goal of RealTime Get (RTG) is to return the latest version of a document. Maybe the reason why copyField values aren’t included is to avoid duplicating existing data... including them would just introduce redundancy.
That said, we could consider modifying RTG by introducing a parameter like includeCopyFields (defaulting to false) to preserve the current behavior while still offering flexibility when those fields are actually needed.
Let’s see what others think as well.