package bm25

All checked builds pass
  1. Overview
  2. Docs
An implementation of Okapi BM25 (a search engine)

Install 

dune-project
 Dependency

github.com Readme Changelog MIT License Edit opam file Versions (2)

Authors

  1. R
    Romain Calascibetta <romain.calascibetta@gmail.com>

Maintainers

  1. R
    Romain Calascibetta <romain.calascibetta@gmail.com>

Sources

stem-0.0.2.tbz
sha256=830e8693a19996caa7745fc67c6d3655b29128d253ff78088e77faba9ad7b488
sha512=d89a8246864f204fbaa159d7c09517003838913944f4a6235b63efb8e4e96fb1872baaa8563392b8e3a4e2354baf363b5e15ed2899f60d8cbf66bbcfdb6d2d85

Description

Added to opam-repository:

README

stem, a stemming algorithm in OCaml

A stemming algorithm is an algorithm that attempts to find the root of words. This library allows you to "tokenize" a document and apply the stemming algorithm to these tokens (considered to be words). It then calculates the frequency of occurrence of these words and produces a CSV document mapping the "stems" to their frequencies.

The purpose of stemming is to be able to treat several words (such as "tout", "toutes", and "tous") as a single root. This way, the resulting stems and their frequencies better reflect the information the document is trying to convey. The idea is then to enable document indexing based on these stems.

How to install it and use it?

stem is a package available through OPAM. It provides two tools: stemmer and stem.ts. The latter allows you to specify multiple tokenizers, the language, and the way the result is displayed in CSV format:

$ opam install stem
$ stem.ts -l french -a bert:isolate -a whitespace:remove file.txt
"est",14                             
"son",13
"tout",11
"Julien",11
"plus",9
"trouv",8
"dan",8
"bien",7
"m\195\170m",7
...

A little search engine (BM25)

The distribution also offers the implementation of a small search engine (based on the Okapi BM25 algorithm). It uses stems to calculate occurrences (which avoids duplicates such as "should" or "shoulds").

A search tool is available that allows the relevance of documents to be rated according to a query, based on a given corpus.

$ opam install bm25
$ okapi -d file0.txt -d file1.txt -d file2.txt -l french "un chat noir"
file0.txt: 1.356894
file2.txt: 0.439572
file1.txt: 0.000000

Dependencies (5)

  1. stem = version
  2. flux
  3. miou
  4. dune >= "2.9.0"
  5. ocaml >= "5.2.0"

Dev Dependencies (1)

  1. alcotest with-test

Optional Dependencies

None

Used by

None

Conflicts

None

Build status

Results of building bm25 with each compiler on check.ci.ocaml.org, run of against opam-repository 8a77e58. All results for this package.

Each result links to its build log. To keep crawlers from overloading the site, build logs require a GitHub sign-in (see ocurrent/opam-health-check). On your first log, GitHub asks you to “Authorize opam-health-check”, the OCurrent app that runs these checks; it sees only your public profile, not your repositories. The sign-in lasts 30 days.

Version OCaml 4.11 OCaml 4.14 OCaml 5.2 OCaml 5.4 OCaml 5.5
0.0.2 (this version) n/a n/a good good good
  • good Package successfully built
  • partial One of the dependencies failed to build
  • bad Package failed to build
  • n/a Package is not available in this environment
  • internal Internal failure