What We Learned at Activate 2018
Last week I was privileged to get some insight into the state of AI at Lucidworks’ annual conference, rebranded “Activate.” The focus was largely on Machine Learning (ML/AI) and their slogan, “Solr activates AI”, was reinforcing that. Speakers from the field were invited to talk about their experience with machine learning and how they see it evolving. The change in conference name & focus is not surprising; ML/AI is the new hotness in tech right now, and probably for the foreseeable future. From chatbots to smart homes to autonomous driving, AI is powering the next technical revolution and indexing engines are well poised to take advantage of that.
Access to Information Game Changers
Information is the commodity of the 21st century. Access to it has created the titans of the industry we see today.
However, Google did not come to dominate search by listing the site you’re seeking on page 5. It exemplifies that access to information is nothing without relevance & context, and that’s the secret sauce of their success. Because of them, we’ve grown accustomed to websites understanding what we mean not what we say. If a CIO wants the “expense report” he’s probably looking for a balance sheet, not a form to fill out. If a person in India asks for the “World Cup” he’s likely referring to cricket, not hockey or soccer. Moreover, looking beyond the “10 blue links” we begin to see the introduction of Natural Language Processing in voice-activated personal assistants, as well as sophisticated concept searches such as “What are my employees talking about today?”
Imbuing search engines with this level of intelligence requires heavy-duty machine learning and a lot of data. So it’s not surprising that companies with thousands of servers & millions of users are the best positioned to pursue it. But Lucidworks aims to democratize AI such that it’s available to more modest enterprises. Using their toolkits & expertise, while leveraging the latest features of Solr, Lucidworks lets organizations collect signals from their users. A signal is anything that can offer context, such as a user’s location, department, previous clicks, and so on. These can be used to extract relevance and boost rankings of search results, providing a better/customized search experience.
Questions to Answer
For the most part, the attendees were very receptive to Lucidworks’ efforts. You can tell from the Q&A that several people were already deep into grappling with these issues and excited to take advantage of the new technologies. Others were mulling whether it makes sense for them, and curious to find out how to start dabbling in it. Still, others were less concerned with technicalities and more concerned with the transparency of such technologies.
What signals are collected, is the user aware, and does the user have control? How can one find biases in the ranking algorithm?How easy is it to troubleshoot or adjust these mechanisms if it’s misbehaving? At what scale does it begin to make sense to shift from traditional search? These are all excellent & valid questions. ML/AI is not for everyone, so careful consideration of these aspects is crucial before embarking on this path.