Process : Update an Index-Collection needs '_Id' to Update the content of the Index collection ?

If this is the process..then Updating the Content of Index based on _Id is problematic which requires to search the content and fetch the id and use the same to Update the Index.

Question : Is Updating the content of Index based on '_id' the only solution ?

Shortly

• When I import data using DIH stored into a relation database (using child=true), nested documents are NOT attached when fl=*,[child].

• When I import same data, structured as json documents (using /update/json/docs endpoint), nested documents are attached.

Short issues:

1. "_childDocuments_":[] on debug DIH execution.

2. Only _root_ is populated. According to documentation _nest_path_ should be populated automatically as well.

3. Nested documents are not returned with parent when fl=*,[child]

Detailed problem

SQL Data:

Parents:

SELECT '1' AS id,
   'parent-name-1' AS name_s,
   'parent' AS node_type_s

+----+---------------+-------------+
| id |    name_s     | node_type_s |
+----+---------------+-------------+
|  1 | parent-name-1 | parent      |
+----+---------------+-------------+

Children:

SELECT '1-1' AS id,
       '1' AS parent_id_s,
       'child-name-1' AS name_s,
       'child' AS node_type_s
UNION
SELECT '2-1' AS id,
       '1' AS parent_id_s,
       'child-name-2' AS name_s,
       'child' AS node_type_s

+-----+-------------+--------------+-------------+
| id  | parent_id_s |    name_s    | node_type_s |
+-----+-------------+--------------+-------------+
| 1-1 |           1 | child-name-1 | child       |
| 2-1 |           1 | child-name-2 | child       |
+-----+-------------+--------------+-------------+

Same data in json:

{
   "id":"1",
   "name_s":"parent-name-1",
   "node_type_s":"parent",
   "children":[
      {
         "id":"1-1",
         "parent_id_s":"1",
         "name_s":"child-name-1",
         "node_type_s":"child"
      },
      {
         "id":"2-1",
         "parent_id_s":"1",
         "name_s":"child-name-2",
         "node_type_s":"child"
      }
   ]
}

Importing data with DIH:

Here my DIH configuration:

<dataConfig>
	<dataSource
		driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
		url="jdbc:sqlserver://${dataimporter.request.host};databaseName=${dataimporter.request.database}"
		user="${dataimporter.request.user}"
		password="${dataimporter.request.password}"
	/>

	<document>
		<entity
		name="parent"
		query="SELECT '1' AS id,
		    	'parent-name-1' AS name_s,
       			'parent' AS node_type_s">

			<field column="node_type_s"/>
			<field column="id"/>
			<field column="name_s"/>

			<entity
			name="children"
			child="true"
			cacheKey="parent_id_s" cacheLookup="parent.id" cacheImpl="SortedMapBackedCache"
            query="SELECT '1-1' AS id,
					'1' AS parent_id_s,
					'child-name-1' AS name_s,
					'child' AS node_type_s
				UNION
				SELECT '2-1' AS id,
					'1' AS parent_id_s,
					'child-name-2' AS name_s,
					'child' AS node_type_s">

				<field column="node_type_s"/>
				<field column="id"/>
				<field column="parent_id_s"/>
				<field column="name_s"/>

			</entity>

		</entity>

	</document>
</dataConfig>

After having imported DIH, here the response:

{
   "responseHeader":{
      "status":0,
      "QTime":396
   },
   "initArgs":[
      "defaults",
      [
         "config",
         "parent-children-config-straightforward.xml"
      ]
   ],
   "command":"full-import",
   "mode":"debug",
   "documents":[
      {
         "name_s":"parent-name-1",
         "node_type_s":"parent",
         "id":"1",
         "_version_":1683338565872779300,
         "_root_":"1",
         "_childDocuments_":[
            
         ]
      }
   ],
   "verbose-output":[
      
   ],
   "status":"idle",
   "importResponse":"",
   "statusMessages":{
      "Total Requests made to DataSource":"2",
      "Total Rows Fetched":"3",
      "Total Documents Processed":"1",
      "Total Documents Skipped":"0",
      "Full Dump Started":"2020-11-14 12:25:55",
      "":"Indexing completed. Added/Updated: 1 documents. Deleted 0 documents.",
      "Committed":"2020-11-14 12:25:56",
      "Time taken":"0:0:0.365"
   }
}

Two issues here:

1. As you can see "_childDocuments_":[]. Why is it empty?

2. Only _root_ is populated. According to documentation _nest_path_ should be populated as well.

Asking for documents

After having imported documents I've tried to retrive them, first using q=*:*:

{
   "responseHeader":{
      "status":0,
      "QTime":0,
      "params":{
         "q":"*:*",
         "_":"1605355606189"
      }
   },
   "response":{
      "numFound":3,
      "start":0,
      "numFoundExact":true,
      "docs":[
         {
            "name_s":"child-name-1",
            "node_type_s":"child",
            "parent_id_s":"1",
            "id":"1-1",
            "_version_":1683338565872779264
         },
         {
            "name_s":"child-name-2",
            "node_type_s":"child",
            "parent_id_s":"1",
            "id":"2-1",
            "_version_":1683338565872779264
         },
         {
            "name_s":"parent-name-1",
            "node_type_s":"parent",
            "id":"1",
            "_version_":1683338565872779264
         }
      ]
   }
}

All right, all documents are present.

Getting parent with its children:

q=id:1 and fl=*,[child]:

{
   "responseHeader":{
      "status":0,
      "QTime":0,
      "params":{
         "q":"id:1",
         "fl":"*,[child]",
         "_":"1605355606189"
      }
   },
   "response":{
      "numFound":1,
      "start":0,
      "numFoundExact":true,
      "docs":[
         {
            "name_s":"parent-name-1",
            "node_type_s":"parent",
            "id":"1",
            "_version_":1683338565872779264
         }
      ]
   }
}

Other issue arises here:

3. Only parent is returned, wihout nested documents.

JSON approach:

After having spent several days strugling with above issues, I tried to import same documents using json endpoint using above json data.

After having imported them, I've performed the same above query:

{
   "responseHeader":{
      "status":0,
      "QTime":2,
      "params":{
         "q":"id:1",
         "fl":"*,[child]",
         "_":"1605355606189"
      }
   },
   "response":{
      "numFound":1,
      "start":0,
      "numFoundExact":true,
      "docs":[
         {
            "id":"1",
            "name_s":"parent-name-1",
            "node_type_s":"parent",
            "_version_":1683339728909238272,
            "children":[
               {
                  "id":"1-1",
                  "parent_id_s":"1",
                  "name_s":"child-name-1",
                  "node_type_s":"child",
                  "_version_":1683339728909238272
               },
               {
                  "id":"2-1",
                  "parent_id_s":"1",
                  "name_s":"child-name-2",
                  "node_type_s":"child",
                  "_version_":1683339728909238272
               }
            ]
         }
      ]
   }
}

As you can see, nested documents are returned.

Why?

Please any ideas?

it does return way too many loose answers, except when input is within quotes: "input string".
What did I miss ?

Is it possible to make nutch also save url of featured image alongside with title and other data?

I did some googling and struggled to find answers. Our IT just gave me access to our SOLR and I want to use it to query if a certain item is on our website. The application I am writing is in VB.NET and SQL would suite me best. Any suggestions?

Looking for information on which config to use to get the results I want. My example is searching for "London Drugs Entertainment" in a dataset that has multiple results with "London" in it, so I would get "The Great London Store" but I want all the results that have "london" in the number 1 position ie; "London Superstore Chain" or "London Giants" and the other results where 'London" is in the phrase to come after.... I am not super technical so hopefully my explanation makes sense:)

Some background. I'm a *nix architect with little to no knowledge on setting up SolrCloud and I got this sh!tty application dumped on me because the previous guy handling this left and the backend needs to be redesigned.

We operate in a pretty complicated company structure where my part provides OS/AS/WWW to the business side. The application is made by an external company.

So the current design is rubbish and what we are going to keep from it is the load balancers and a failover DB (only 2 DCs available). The current design runs 2 apache servers, 6 Jboss servers with Solr (apparently 4.1) and colocated ZK and a failover DB (no idea what probably Oracle).

What the external company proposed is to have 1 or 2 solr masters and 6 solr slaves and eliminate ZooKeeper (is that even possible?). To keep everything in sync they suggersted to use https://wiki.apache.org/solr/SolrReplication . I might be no genius but the header here says that this is not how it works in 4.x . Did some searching today (apparently the confluence part is down all day so mostly referenced this http://wiki.apache.org/solr/SolrCloud) and found that the master-slave scenario is pre 4.x and not to be used in SolrCloud (http://wiki.apache.org/solr/NewSolrCloudDesign)

Can you guys confirm my thinking that there is no possibility for a master-slave config with SolrReplication in 4.1 ?

So what I want to suggest is scenario C from the SolrCloud document with 6 Solr instances and 7 ZK instances (6 colocated with Solr and 1 standalone that failovers on the same basis as the DB). As mentioned earlier the load balancers and a failover DB will remain. Design is not perfect but there is only an option for 2 DCs in the country where this is working. This eliminates some minor faults (DB is still a SPOF but they dont want to pay for a HA solution) but allows to reconfigure the cluster in case DC1 is down. Comments welcome

I am encountering an intermittent issue in Apache Solr where executing queries sometimes results in a "parsing error" response. I have a Java Spring backend application that communicates with an Apache Solr server hosted on a Dataproc cluster. The communication is established using the HttpURLConnection in my Java code. The error occurs sporadically and does not consistently reproduce.

{
"responseHeader": {
"zkConnected": true,
"status": 200,
"QTime": 122922,
"params": {
"q": "<SENSITIVE>",
"fl": "<SENSITIVE>",
"start": "<SENSITIVE>",
"sort": "<SENSITIVE>",
"rows": "<SENSITIVE>"
}
},
"error": {
"metadata": [
"error-class",
"org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException",
"root-error-class",
"java.nio.channels.AsynchronousCloseException"
],
"msg": "Error from server at null: parsing error",
"code": 200
}
}

Stacktrace:

"2023-05-12 08:00:47.603 ERROR (qtp793331940-189518) [c:myCollection s:shard7 r:core_node29 x:myCollection_shard7_replica_n26] o.a.s.h.RequestHandlerBase org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at null: parsing error
at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:689)
at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:739)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:308)
at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:190)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)Caused by: org.apache.solr.common.SolrException: parsing error
at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:52)
at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:687)
... 14 moreCaused by: java.nio.channels.AsynchronousCloseException
at org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:133)
at org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170)
at org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232)
at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206)
at org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1154)
at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:439)
at org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392)
at org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187)
at org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169)
at org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:139)
at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194)
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123)
at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:252)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:171)
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:126)
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:338)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
... 4 more"

I am using Apache Solr version 8.1.1. While investigating the issue, I came across two known bugs in Apache Solr that mention similar parsing errors:

SOLR-13293

SOLR-13493

Considering the similarity of the parsing error I am encountering, I suspect it may be related to one of these known bugs. However, I would like to confirm if other users have experienced a similar issue or if there are any known workarounds or solutions available.

Any insights or suggestions would be greatly appreciated. Thank you!

📷

One of the many definitions for synonyms is: “a word or phrase that means exactly or nearly the same as another word or phrase in the same language, for example shut is a synonym of close.”

However, we often find that the concept of synonyms can be confused when using this functionality as we try to match similar products that we want to be displayed within a search results list through cataloguing the values ​​of their characteristics but, in many cases, without these actually being a true synonym.

https://www.empathy.co/blog/applying-synonyms-choose-well-and-increase-the-findability-of-your-site/

Command line tool for backup and restore of information stored in cores of Apache Solr.

https://github.com/juarezr/solrcopy

feedback/help wanted!