Adding full text Search via FlexSearch to a Blog

When I migrated from wordpress to the static site generator hexo, I know not to have as many features out of the box available. One of them was search. Do be honest i did not care about a search feature.

To my surprise, I even had a search feature. The theme I chose has a search input in its header. When using it, google.com is opened with a result, limited to my site. I found that is a good solution and for about 4 years happy with it.

But as I recently write much more on dev.to and my website, it makes sense to have a more custom solution. As javascript and node.js developer, I have my personal website server powered by express.js.

Recently I had some time at hand and searched for other solutions. It was not easy to find a good solution. Because you can try yourself to google javascript reverse index or js text index. The google search will only show documentations for javascript’s array and string methods.

However I found two solutions but now I went with the first I tried. I tried flexSearch, because it claimed to have a smallest index size and can perform more searches per second. And I did not get disappointed.

How to use flexSearch.js

As most modules it was quickly installed using npm install --save flexsearch. In code It is as simple as new FlexSearch to get a new index. Then add all the documents with there ID and then you are ready to do a search.

For my hexo site, all the content get stored into a single json file. and preparing the posts and pages for the search, actually took me longer than setting up the index and the search API.

Here is the general usage:

// create the index
var FlexSearch = require("flexsearch");
var index = FlexSearch.create();
// load posts from a file
var pages = JSON.parse(require('fs').readFileSync('db.json').toString()).models.Posts;

// add posts to the index
posts.forEach(post=>{
  index.add(post.link, post.content)
});

// export a simple express middleware to search the index
module.exports.searchHandler = function searchHandler(req, res) {
  if (typeof req.query.q !== 'string') {
    res.json([]);
  }
  res.json(index.search(req.query.q);
}

you see? It is very simple to create the index, add all the documents to be searched and provide an API for searching. If you like you can try this search at tnickel.de/search.

The search that I use there is extended by pages(pages and posts are different in hexo ssg) and I return a little object, not only the page link to the frontend.

The search handler in this example, is for express.js, and I am sure you understand how to implement a handler for your framework of choice or cloud function.

More features of the flexSearch

For my little blog, the basic usage is enough. But flexsearch can be used in more complex scenarios fulling more requirements. It can index search.

You can define a depth argument, that let you specify how many words the user can type into your search.

With the threshold parameter you can say thay the searched word need to appear multiple times in the content, before it will get into the result set.

Then there are some more options for paging pagination (limit, next). And for suggestions. Maybe the user (or yourself had a typo) then suggestions can still bring the user to relevant content.

flexSearch has build in support for english and german. For supporting other languages, a stemmer and a tokenizer function need to be implemented. I think it is possle to use the implementation of the module natural.

Result of my new Search Feature

Looking at the speed, I find it very good on my little website with its few articles. When searching for tobias, you will get the about page. Searching for javascript and you find almost every post.

The search index is in memory and for a small blog, does not need to much. And it will be hard to write enough to outgrow this solution.