{"id":118895,"date":"2022-08-07T21:53:05","date_gmt":"2022-08-07T21:53:05","guid":{"rendered":"https:\/\/livablesoftware.com\/?p=118895"},"modified":"2023-06-03T09:02:18","modified_gmt":"2023-06-03T09:02:18","slug":"chatbots-open-data-project","status":"publish","type":"post","link":"https:\/\/livablesoftware.com\/chatbots-open-data-project\/","title":{"rendered":"Bots for open data interactions – BODI project"},"content":{"rendered":"

Open data is a wonderful idea, but we have already argued that all the public time and money invested in creating and publishing open data resources has a very minimal impact<\/a> on our society. For instance, city halls complain that very few people use their open data sources. And the few that do are most data scientists or expert journalist. A regular citizen finds no use in all these open data initiatives.<\/p>\n

In the last years, our contribution to solve the open data problem has focused on the backend, by trying to combine individual open data sources<\/a> into richer, more useful ones.<\/p>\n

We have new started a new project, the BODI project<\/a>, funded by the Spanish Ministry of Science and Innovation and the EU’s Next Generation programme.<\/p>\n

Conversational interfaces for open data<\/h2>\n

In this project, we focus on fixing the front-end. We feel that open data sources should offer a conversational interface if we want citizens to be able to benefit from them. Right now, releasing an open data set typically means uploading a CSV, XML or JSON file. None of them really usable for non-technical people.<\/p>\n

I think we can all agree that being able to ask a chatbot a question like “what is the neighborhood with less average pollution in my city” is much easier than downloading the full CSV and starting “playing” and operating with the data to reach the same result.<\/p>\n

Architecture of an open data chatbot<\/h2>\n

The architecture of our open data bots is described in this figure. As you can see, we combine a set of concrete questions the bot is able to recognize and answer with an advanced fallback<\/a> strategy for those questions the bot doesn’t understand.<\/p>\n

This advanced fallback strategy is important as the number of questions users can ask on a dataset is too broad. We try to predefine as many questions as possible but we need to assume that citizens will always be able to surprise us and ask questions that we didn’t expect.<\/p>\n

Our fallback implementation is triggered every time the intent recognition component<\/a> fails to match the citizen question. The fallback relies on the salesforce<\/a><\/span>\/<\/span>TabularSemanticParsing<\/a><\/strong> model to automatically translate the user query to SQL that we then run on the CSV data thanks to ApacheDrill<\/a>. For questions in Catalan or Spanish we need to first translate the query to English. We use the softcatala\/opennmt-cat-eng<\/a> and Helsinki-NLP\/opus-mt-es-en<\/a> models, respectively.<\/p>\n

Obviously, the quality of the answer in this case is lower than for questions that the bot is actually trained with. But it’s better than nothing and has shown to be (surprisingly) accurate in numerous instances even when starting from Catalan and Spanish (which implies concatenating two language models, each one potentially introducing errors). Note that to maximize the chances of a good answer, we need to translate the names of the columns in all target languages so that the SQL translation model is able to match the attributes referenced in the question with column names in the CSV.<\/p>\n

\"BODI<\/a>

NLP architecture to answer citizen questions on open data<\/p><\/div>\n

Massive generation of open data bots<\/h2>\n

With over 1M of open data sources (and this only counting those in the European data portal<\/a>), creating the chatbots manually would not scale. For sure, city halls cannot afford to create a chatbot for every single data source, so a key component of the BODI project is to generate the chatbots automatically from the own open data set description.<\/p>\n

For this, we are creating a set of heuristic rules that generate potential questions users could ask depending on the structure of the data source. Most of the time, this structure is limited to the names of the columns and an analysis of the data each column stores. For instance, if we can determine that a certain column holds integer values, our rules will generate questions to ask for the top value of that column, the average, all those that have a value greater than X, etc. Similarly, if another column holds date values, we will add questions to ask about results before (or after or between,…) a certain date. We could go one step further and generate questions not only based on the type of the data but based on the semantics of the concept represented by the column. E.g. if I know a column is not only an integer but that it represents a price<\/em> I could generate predefined questions for price columns.<\/p>\n

The systematic application of these heuristics over every column generates the set of questions the bot will be trained with.\u00a0All this process can be completely automated so that given a CSV file (or any other tabular representation) we automatically generate three working bots<\/strong> (one for each language) ready to use. But with some simple configuration \/ annotation of the input CSV we can improve a lot the quality of conversation experience.<\/p>\n

Let’s see some screenshots of the whole process.<\/p>\n

From the web interface main menu we can import the data for a new bot, configure the generation process, the bot deployment properties (e.g. whether to deploy the bot on DialogFlow or on other NLP engines) and finally deploy the bots and get them running.<\/p>\n

\"Main<\/a>

Main menu of the bot generation interface<\/p><\/div>\n

When importing the data we show a preview of the import results for verification.<\/p>\n

\"Preview<\/a>

Preview of the imported CSV data<\/p><\/div>\n

We can then start with the configuration. All configuration is always optional. Via this form we can define the readable\u00a0<\/em>names for each column (instead of the sometimes obscure COL_X names and similars we find in the CSVs). We can also provide synonyms that help matching information requests with that column. All this in the three languages we support right now, though adding new languages would not really be a problem.<\/p>\n

\"Configuration<\/a>

Configuration of the CSV columns<\/p><\/div>\n

You can also merge fields (e.g. first name and last name) if the CSV keeps them separated but we know citizens we’ll use them together.<\/p>\n

\"Merging<\/a>

Merging CSV fields for a more natural bot<\/p><\/div>\n

Finally, we have different sets of properties that configure the bot deployment and execution.<\/p>\n

\"Configuration<\/a>

Configuration of the bot deployment properties<\/p><\/div>\n

At this point we’re ready to deploy and start using the bot.<\/p>\n

I want to know more!<\/h2>\n

Take a look at our technical description in the paper: Towards the Automatic Generation of Conversational Interfaces to Facilitate the Exploration of Tabular Data.<\/a><\/p>\n

Can I try it with my data?<\/h2>\n

Absolutely. We’re obviously interested in testing how the BODI infrastructure works in different open data scenarios. Get in touch<\/a> and we’ll take it from there.<\/p>\n

Later on, we will release first some examples\u00a0 to validate the generator and once that feedback is integrated, we’ll start releasing all the BODI components as open source for those institutions that prefer to try it on their own while we will be available to help those that prefer a more assisted process (or even have us host the bots for them).<\/p>\n

More long-term work include go beyond structured and tabular data and also be able to plug to the bot knowledge base full documents that the bot could use to answer additional questions based on Q&A language models<\/a>.<\/p>\n

Stay tuned for more!<\/p>\n<\/span>","protected":false},"excerpt":{"rendered":"

Make sure all your citizens can benefit from your open data sources by offering them a conversational interface to talk with your data<\/p>\n","protected":false},"author":13,"featured_media":118897,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[47,7,9],"tags":[161,115,107,162],"_links":{"self":[{"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/posts\/118895"}],"collection":[{"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/comments?post=118895"}],"version-history":[{"count":1,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/posts\/118895\/revisions"}],"predecessor-version":[{"id":118932,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/posts\/118895\/revisions\/118932"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/media\/118897"}],"wp:attachment":[{"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/media?parent=118895"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/categories?post=118895"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/tags?post=118895"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}