Feat: Adds OpenSearch2.19.1 as the vector_database support

[pyyuhao]

What problem does this PR solve?

This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow.

Main Benefit

1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch
2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema
3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema

Changes

• Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py
• Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py
• Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json
• Support OpenSearch python sdk : pyproject.toml
• Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template

How to use

• I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work.

Others

Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well.
All the conifg for OpenSearch is necessary.

Type of change

• New Feature (non-breaking change which adds functionality)

[yingfeng]

Thanks for the contribution. OpenSearch can be added as a doc engine alternative. However, maintaining OpenSearch is not an easy work, after this PR is merged, OpenSearch might not be able to work after several weeks.

[asiroliu]

the default DOC_ENGINE should be elasticsearch

[pyyuhao]

@asiroliu I'v changed it into elasticsearch by default in my commit. It's a config mistake I forgot to change back from my local environment

[pyyuhao]

Thanks for the contribution. OpenSearch can be added as a doc engine alternative. However, maintaining OpenSearch is not an easy work, after this PR is merged, OpenSearch might not be able to work after several weeks.

Thanks for you reply, and I'v changed some litte problems mentioned above. I am an search-engine engineer focusing on the stuff about OpenSearch/Elasticsearch for years, and also write some plugins for Opnsearch.I will still pay much attention on ES/OS continuously. During these two years, We gave more attention on RAG stuff

[pyyuhao]

@asiroliu @yingfeng @KevinHuSh ,hi: I've made some commits mainly about fomat and comment. Please review again, thanks a lot

[yingfeng]

It can not pass CI, the container of elasticsearch can not be started. See the CI logs https://github.com/infiniflow/ragflow/actions/runs/14533575135/job/40777952188

[pyyuhao]

It can not pass CI, the container of elasticsearch can not be started. See the CI logs https://github.com/infiniflow/ragflow/actions/runs/14533575135/job/40777952188

It worked well now at my local environment, I will check the code again and create a virtual machine to verify it.
Maybe because I use opensearch on 9200 port which is the same as elasticsearch(on most cases these are the same), is there a rule about this that will block the test job for es?
I will check these in a few days

Have a nice weekend

[yingfeng]

Still the same error for CI:
https://github.com/infiniflow/ragflow/actions/runs/14573690004/job/40875405448?pr=7140

Error response from daemon: container 41fe9ca43b5b1f497269237c45423bf7c4e3b53687d48b7f94b235cee24095e0 is not running

[pyyuhao]

Still the same error for CI: https://github.com/infiniflow/ragflow/actions/runs/14573690004/job/40875405448?pr=7140

Error response from daemon: container 41fe9ca43b5b1f497269237c45423bf7c4e3b53687d48b7f94b235cee24095e0 is not running

hi，there were some config adaption mistakes before . I reckeced on my new environment both on ES and OpenSearch . I've repushed today and I see the ci has passed
Thanks for your review

[wanpdsantos]

Shouldn't tests to OpenSearch be added here: https://github.com/infiniflow/ragflow/blob/main/.github/workflows/tests.yml ?

[asiroliu]

@pyyuhao
Apologies for not reviewing your PR immediately - I've been fully occupied with finalizing the 0.18.0 release. I'll prioritize reviewing your contribution as soon as the release is complete.

[pyyuhao]

@pyyuhao Apologies for not reviewing your PR immediately - I've been fully occupied with finalizing the 0.18.0 release. I'll prioritize reviewing your contribution as soon as the release is complete.

Thanks for your review.
no rush, take your time.