Блог

Я пытаюсь понять эффект нечеткости при нечетком поиске, используя несколько предложений span_near.

Здесь я делаю подсчет документов. Вот один запрос

 GET wikipedia-20200820/_search
{  
   "query":{  
      "bool":{  
         "must":[  
            {  
               "span_near":{  
                  "clauses":[
                     {  
                        "span_multi":{  
                           "match":{  
                              "fuzzy":{  
                                 "text":{  
                                    "value":"labor",
                                    "fuzziness":"2"
                                 }
                              }
                           }
                        }
                     },
                     {  
                        "span_multi":{  
                           "match":{  
                              "fuzzy":{  
                                 "text":{  
                                    "value":"unionism",
                                    "fuzziness":"2"
                                 }
                              }
                           }
                        }
                     },
                     {  
                        "span_multi":{  
                           "match":{  
                              "fuzzy":{  
                                 "text":{  
                                    "value":"among",
                                    "fuzziness":"2"
                                 }
                              }
                           }
                        }
                     },
                     {  
                        "span_multi":{  
                           "match":{  
                              "fuzzy":{  
                                 "text":{  
                                    "value":"negros",
                                    "fuzziness":"2"
                                 }
                              }
                           }
                        }
                     }
                  ],
                  "slop":1,
                  "in_order":true
               }
            }
         ],
         "filter": [ 
        { "term":  { "ns": 0 }},
        { "range": { "timestamp": { "gte": "2020-01-01" }}}
      ]
      }
   }
}
  

это дает мне ожидаемый результат:

 {
  "took" : 284,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 2238.6646,
    "hits" : [
      {
        "_index" : "wikipedia-20200820",
        "_type" : "_doc",
        "_id" : "GhxaT3UBz_UQYd3DEAJT",
        "_score" : 2238.6646,
        "_source" : {
          "page_title" : "National Brotherhood of Workers of America",
          "ns" : 0,
          "page_id" : 28365131,
          "redirect" : null,
          "revision_id" : 965097134,
          "revision_parent_id" : 891907364,
          "timestamp" : "2020-06-29T11:26:55Z",
          "contributor" : {
            "username" : "JJMC89 bot III",
            "id" : 35936988
          },
          "comment" : "Moving [[:Category:African-American history between emancipation and the Civil Rights Movement]] to [[:Category:African-American history between emancipation and the civil rights movement]] per [[Wikipedia:Categories for discussion/Speedy]]",
          "sha1" : "q1rz1h5n6qk9cpdt5qf0zqrydd9vx7e",
          "minor" : "TRUE",
          "model" : "wikitext",
          "format" : "text/x-wiki",
          "text" : """The '''National Brotherhood of Workers of America''' (NBWA) was the largest body of organised [[African American]] workers in the [[United States of America]] in 1919.amp;<refamp;>''Revolutionary radicalism: its history, purpose and tactics with an exposition and discussion of the steps being taken and required to curb it, being the report of the Joint Legislative Committee Investigating Seditious Activities'', filed 24 April 1920, in the Senate of the State of New Yorkamp;</refamp;>

==First congress==
The organisation was formed by T.J. Pree and R.T. Sims. [[Philip Randolph]] was also on the board.amp;<ref name=yhoamp;>[http://hierographics.org/yourhistoryonline/yourhistoryonlineVIII.htm Your History online], accessed 17 August 2010amp;</refamp;> The NBWA held its congress in [[Washington DC]] from  8–14 September 1919. There were 115 delegates primarily from the South. Three delegates were from the [[Industrial Workers of the World]] and fifteen from the Society for the Advancement of Trade Unionism among Negroes. No delegates were accepted from the [[American Federation of Labor]].amp;<refamp;>''Revolutionary radicalism: its history, purpose and tactics with an exposition and discussion of the steps being taken and required to curb it, being the report of the Joint Legislative Committee Investigating Seditious Activities'', filed 24 April 1920, in the Senate of the State of New Yorkamp;</refamp;>

==Resolutions==
They passed resolutions to the effect that:

amp;quot;. . . The combination of black and white workers will be a powerful lesson to the capitalists of the solidarity of labor. It will show that labor, black and white, is conscious of its interests and power. This will serve to convert a class of worker, which has been used by the capitalist class 
to defeat organised labor, into an ardent, class-conscious, intelligent, militant group.amp;quot;

amp;quot;and be it further resolved, that we recommend to all the working people of our race, that they immediately make themselves acquainted more in detail with the 
aims, objects and methods of said organization, the National Association for the Protection of 
Labor Unionism Among Negroes, in order that we may, as speedily as possible, align ourselves with and join the industrial unions that have already organized, and help to organize new industrial unions in such industries where they do not yet exist. 


amp;quot;and be it further resolved, that we shall henceforth devote all our energies to building up the new order of society along lines above indicated, to the exclusion of efforts hitherto expended in other directions.amp;quot; amp;<refamp;>''Revolutionary radicalism: its history, purpose and tactics with an exposition and discussion of the steps being taken and required to curb it, being the report of the Joint Legislative Committee Investigating Seditious Activities'', filed 24 April 1920, in the Senate of the State of New Yorkamp;</refamp;>

The NBWA functioned as an independent Union, drawing most of its support from shipyard and dock workers in the [[Tidewater region of Virginia]] area.amp;<refamp;>''Crisis'', November 1951, p626amp;</refamp;> The union dissolved in 1921, under pressure from the American Federation of Labor.amp;<ref name=yho/amp;>

==References==
{{reflist}}
[[Category:African-American leftism]]
[[Category:African-American history between emancipation and the civil rights movement]]
[[Category:Trade unions established in 1919]]
[[Category:Trade unions disestablished in 1921]]
[[Category:African-American trade unions]]"""
        }
      }
    ]
  }
}
  

Хотя документально подтверждено, что fuzziness параметр представляет расстояние редактирования / расстояние Левенштейна, мне это показалось непоследовательным. Затем, чтобы понять эффект нечеткости, я попытался изменить какой-то один термин в запросе и иногда менял fuzziness параметр, сохраняя все остальное неизменным.

Я обобщаю некоторые изменения, которые я попробовал в запросе в следующей таблице, и независимо от того, привел ли запрос к тому же результату или нет:

  ---------------- --------------- ----------- --------------- ----------------------------- 
| iniatial_query | updated_query | fuzziness | edit_distance | elastic_returns_same_result |
 ---------------- --------------- ----------- --------------- ----------------------------- 
| "labor"        | "labot"       |         2 | 1             | Yes                         |
| "labor"        | "labori"      |         2 | 1             | No                          |
| "labor"        | "lab"         |         2 | 2             | No                          |
| "labor"        | "lab"         |         3 | 2             | No                          |
| "labor"        | "lab"         |         4 | 2             | No                          |
| "labor"        | " labor"      |         2 | 1 (1 space)   | Yes                         |
| "labor"        | " labor"      |         1 | 1 (1 space)   | Yes                         |
| "labor"        | "  labor"     |         2 | 2 (2 space)   | No                          |
| "unionism"     | "unionismi"   |         2 | 1             | Yes                         |
| "unionism"     | "unionismi"   |         1 | 1             | Yes                         |
| "among"        | "amonng"      |         2 | 1             | Yes                         |
| "among"        | "amonng"      |         1 | 1             | Yes                         |
| "among"        | "amonnng"     |         2 | 2             | No                          |
| "among"        | "amonnng"     |         3 | 2             | No                          |
| "among"        | "amonnng"     |        10 | 2             | No                          |
 ---------------- --------------- ----------- --------------- ----------------------------- 
  

Может кто-нибудь объяснить, как влияет fuzziness параметр на запрос?

Параметры нечеткости :

АВТО

Генерирует расстояние редактирования на основе длины термина. Параметры низкого и высокого расстояния могут быть дополнительно предоставлены АВТОМАТИЧЕСКИ: [низкий], [высокий]. Если не указано, значения по умолчанию равны 3 и 6, что эквивалентно AUTO: 3,6, которые определяют длины:

0 ..2 должны точно совпадать

3 ..5 Разрешено одно редактирование

5 Два разрешенных редактирования AUTO обычно должно быть предпочтительным значением для нечеткости.

В вопросе вы упомянули, что когда значение updated_value равно lab , тогда elasticsearch не вернул тот же результат, что и раньше. Но, как вы можете видеть ниже, результаты приходят точно. Это связано с тем, что расстояние редактирования равно 2.

 lab --> labo --> labor
  

Аналогично, результаты поступают точно для labori , labor и amonnng

Индексировать данные:

 {
  "text":"The organisation was formed by T.J. Pree and R.T. Sims. [[Philip Randolph]] was also on the board.amp;<ref name=yhoamp;>[http://hierographics.org/yourhistoryonline/yourhistoryonlineVIII.htm Your History online], accessed 17 August 2010amp;</refamp;> The NBWA held its congress in [[Washington DC]] from  8–14 September 1919. There were 115 delegates primarily from the South. Three delegates were from the [[Industrial Workers of the World]] and fifteen from the Society for the Advancement of Trade Unionism among Negroes. No delegates were accepted from the [[American Federation of Labor]].amp;<refamp;>''Revolutionary radicalism: its history, purpose and tactics with an exposition and discussion of the steps being taken and required to curb it, being the report of the Joint Legislative Committee Investigating Seditious Activities'', filed 24 April 1920, in the Senate of the State of New Yorkamp;</refamp;>"
}
  

Поисковый запрос:

 {
  "query": {
    "bool": {
      "must": [
        {
          "span_near": {
            "clauses": [
              {
                "span_multi": {
                  "match": {
                    "fuzzy": {
                      "text": {
                        "value": "lab",         <-- note this
                        "fuzziness": "2"
                      }
                    }
                  }
                }
              }
            ],
            "slop": 1,
            "in_order": true
          }
        }
      ]
    }
  }
}
  

Результат поиска:

  "hits": [
      {
        "_index": "64492853",
        "_type": "_doc",
        "_id": "1",
        "_score": 2.2105305,
        "_source": {
          "text": "The organisation was formed by T.J. Pree and R.T. Sims. [[Philip Randolph]] was also on the board.amp;<ref name=yhoamp;>[http://hierographics.org/yourhistoryonline/yourhistoryonlineVIII.htm Your History online], accessed 17 August 2010amp;</refamp;> The NBWA held its congress in [[Washington DC]] from  8–14 September 1919. There were 115 delegates primarily from the South. Three delegates were from the [[Industrial Workers of the World]] and fifteen from the Society for the Advancement of Trade Unionism among Negroes. No delegates were accepted from the [[American Federation of Labor]].amp;<refamp;>''Revolutionary radicalism: its history, purpose and tactics with an exposition and discussion of the steps being taken and required to curb it, being the report of the Joint Legislative Committee Investigating Seditious Activities'', filed 24 April 1920, in the Senate of the State of New Yorkamp;</refamp;>"
        }
      }
    ]