HomePage Forums Development Text to Intent Comparing Adapt and Padatious

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #585

    Adapt and Padatious are two text to intent parsers that have been published by the Mycroft project. I’d like to be able to have a stable interface for developers to easily be able to specify how their plugins should be activated.

    Currently, the method Naomi uses is to parse a block of speech into text, then pass that text to each speechhandler in turn (based on the value returned by the plugin’s get_priority() method) and checking to see if the plugin’s is_valid() method returns true. This means that a plugin could currently hijack all requests by simply setting the value returned by get_priority to sys.maxsize and having is_valid() always return true. This is not particularly scalable, and forces every plugin author to basically write their own intent parser.

    Adapt and Padatious have much different methods for building intents. With Adapt, you build word categories and then declare what groups of categories should be used to trigger the intent, while Padatious uses a neural network to build a model of ways the intent could be triggered from sample intents provided by the plugin author.

    I started with an example written for Padatious and re-wrote it for Adapt since that seemed easier than going the other way around, although I later added some of the Adapt examples just to show that it was possible to translate in both directions.

    Here’s the example in Adapt

    #!/usr/bin/env python3
    # -*- coding: utf-8 -*-
    import json
    import re
    import sys
    from adapt.entity_tagger import EntityTagger
    from adapt.tools.text.tokenizer import EnglishTokenizer
    from adapt.tools.text.trie import Trie
    from adapt.intent import IntentBuilder
    from adapt.parser import Parser
    from adapt.engine import IntentDeterminationEngine
    
    tokenizer = EnglishTokenizer()
    trie = Trie()
    tagger = EntityTagger(trie, tokenizer)
    parser = Parser(tokenizer, tagger)
    engine = IntentDeterminationEngine()
    
    # Hello
    for word in ['hi', 'hello']:
        engine.register_entity(word, "HelloKeyword")
    engine.register_intent_parser(IntentBuilder("HelloIntent").require("HelloKeyword").build())
    # Goodbye
    for word in ['see you', 'goodbye', 'bye']:
        engine.register_entity(word, "GoodbyeKeyword")
    engine.register_intent_parser(IntentBuilder("GoodbyeIntent").require("GoodbyeKeyword").build())
    # Search
    for word in ['search']:
        engine.register_entity(word, "SearchKeyword")
    for word in ['google', 'youtube', 'instagram']:
        engine.register_entity(word, "EngineKeyword")
    engine.register_regex_entity("for (?P<Query>) on")
    engine.register_regex_entity("for (?P<Query>.*)$")
    engine.register_intent_parser(IntentBuilder("SearchIntent").require("SearchKeyword").require("Query").optionally("EngineKeyword").build())
    # Weather
    for word in ['weather', 'snowing', 'raining', 'windy', 'sleeting', 'sunny']:
        engine.register_entity(word, "WeatherTypeKeyword")
    for word in ['seattle', 'san francisco', 'tokyo']:
        engine.register_entity(word, "LocationKeyword")
    engine.register_intent_parser(IntentBuilder("WeatherIntent").require("WeatherTypeKeyword").optionally("LocationKeyword").build())
    # Music
    for word in ["listen", "hear", "play"]:
        engine.register_entity(word, "MusicVerb")
    for word in ["music", "song"]:
        engine.register_entity(word, "MusicKeyword")
    for word in ["third eye blind", "the who", "the clash"]:
        engine.register_entity(word, "ArtistKeyword")
    engine.register_intent_parser(IntentBuilder("MusicIntent").require("MusicVerb").optionally("MusicKeyword").optionally("ArtistKeyword").build())
    
    # Test
    test_phrases = [
        "hi there",
        "bye bye",
        "search for cats on instagram",
        "search for calico cats on instagram",
        "search instagram for cats",
        "search instagram for calico cats",
        "what's the weather like in san francisco",
        "is it raining in seattle",
        "play some music by the clash",
        "add cereal to my shopping list"
    ]
    for phrase in test_phrases:
        print()
        print(phrase)
        for intent in engine.determine_intent(phrase):
            if intent and intent.get("confidence")>0:
                print(json.dumps(intent,indent=4))

    and here is basically the same example in Padatious:

    #!/usr/bin/env python3
    # -*- coding: utf-8 -*-
    import json
    from padatious import IntentContainer
    
    container = IntentContainer('intent_cache')
    # Hello
    container.add_intent('HelloIntent', [
        'hi', 
        'hello.'
    ])
    # Goodbye
    container.add_intent('GoodbyeIntent', [
        'see you', 
        'goodbye',
        'bye'
    ])
    # Search
    container.add_entity('EngineKeyword', ["google", "youtube", "instagram"])
    container.add_intent('SearchIntent', [
        'search for {Query} (using|on) {EngineKeyword}',
        'search {EngineKeyword} for {Query}'
    ])
    # Weather
    container.add_entity('WeatherTypeKeyword', ['snowing', 'raining', 'windy', 'sleeting', 'sunny'])
    container.add_entity('LocationKeyword', ['seattle', 'san francisco', 'tokyo'])
    container.add_intent('WeatherIntent', [
        "what's the weather in {LocationKeyword}",
        "is it {WeatherTypeKeyword} in {LocationKeyword}"
    ])
    # Music
    container.add_entity('ArtistKeyword', ['third eye blind', 'the who', 'the clash'])
    container.add_intent('MusicIntent', [
        "play {ArtistKeyword}",
        "play music"
    ])
    # container.add_intent('UnknownIntent', ['{Unknown}'])
    container.train()
    
    test_phrases = [
        "hi there",
        "bye bye",
        "search for cats on instagram",
        "search for calico cats on instagram",
        "search instagram for cats",
        "search instagram for calico cats",
        "what's the weather like in san francisco",
        "is it raining in seattle",
        "play some music by the clash",
        "add cereal to my shopping list"
    ]
    for phrase in test_phrases:
        print()
        print(phrase)
        intent = container.calc_intent(phrase)
        print(json.dumps(intent.__dict__,indent=4))

    As you should be able to see, the methods for building intents are quite different between the engines. Adapt is more labor intensive and less intuitive to write, but requires the plugin author to spell out all of the keywords which is good for limited vocabulary STT parsers like Pocketsphinx, whereas Padatious is happy to pull in text based mostly on where it appears in the sentence, and doesn’t require as much planning by the author

    Let’s see some output:
    adapt_test.py

    
    hi there
    {
        "intent_type": "HelloIntent",
        "HelloKeyword": "hi",
        "target": null,
        "confidence": 1.0
    }
    
    bye bye
    {
        "intent_type": "GoodbyeIntent",
        "GoodbyeKeyword": "bye",
        "target": null,
        "confidence": 0.5
    }
    
    search for cats on instagram
    {
        "intent_type": "SearchIntent",
        "SearchKeyword": "search",
        "Query": "cats on",
        "EngineKeyword": "instagram",
        "target": null,
        "confidence": 0.4166666666666667
    }
    
    search for calico cats on instagram
    {
        "intent_type": "SearchIntent",
        "SearchKeyword": "search",
        "Query": "calico cats on",
        "EngineKeyword": "instagram",
        "target": null,
        "confidence": 0.4166666666666667
    }
    
    search instagram for cats
    {
        "intent_type": "SearchIntent",
        "SearchKeyword": "search",
        "Query": "cats",
        "EngineKeyword": "instagram",
        "target": null,
        "confidence": 0.4166666666666667
    }
    
    search instagram for calico cats
    {
        "intent_type": "SearchIntent",
        "SearchKeyword": "search",
        "Query": "calico cats",
        "EngineKeyword": "instagram",
        "target": null,
        "confidence": 0.4166666666666667
    }
    
    what's the weather like in san francisco
    {
        "intent_type": "WeatherIntent",
        "WeatherTypeKeyword": "weather",
        "LocationKeyword": "san francisco",
        "target": null,
        "confidence": 1.0
    }
    
    is it raining in seattle
    {
        "intent_type": "WeatherIntent",
        "WeatherTypeKeyword": "raining",
        "LocationKeyword": "seattle",
        "target": null,
        "confidence": 1.0
    }
    
    play some music by the clash
    {
        "intent_type": "MusicIntent",
        "MusicVerb": "play",
        "MusicKeyword": "music",
        "ArtistKeyword": "the clash",
        "target": null,
        "confidence": 1.0
    }
    
    add cereal to my shopping list
    

    So here you can see that it figured out that the last phrase didn’t match any of the intents, but I was unable to get the search engine query not to attach “on” to the end of the query when written in the “search for {query} on {searchengine}” form. However, it was able to identify “cats” and “calico cats” as search terms despite not being defined as keywords.

    Here is the output from Padatious:

    Regenerated {EngineKeyword}.
    Regenerated HelloIntent.
    Regenerated {LocationKeyword}.
    Regenerated {WeatherTypeKeyword}.
    Regenerated {ArtistKeyword}.
    Regenerated GoodbyeIntent.
    Regenerated SearchIntent.
    Regenerated MusicIntent.
    Regenerated WeatherIntent.
    
    hi there
    {
        "name": "HelloIntent",
        "sent": "hi there",
        "matches": {},
        "conf": 0.6034999697076322
    }
    
    bye bye
    {
        "name": "GoodbyeIntent",
        "sent": "bye bye",
        "matches": {},
        "conf": 0.6225372755994739
    }
    
    search for cats on instagram
    {
        "name": "SearchIntent",
        "sent": "search for {query} on {enginekeyword}",
        "matches": {
            "query": "cats",
            "enginekeyword": "instagram"
        },
        "conf": 0.9091592209921133
    }
    
    search for calico cats on instagram
    {
        "name": "SearchIntent",
        "sent": "search for {query} on {enginekeyword}",
        "matches": {
            "query": "calico cats",
            "enginekeyword": "instagram"
        },
        "conf": 0.9091592209921133
    }
    
    search instagram for cats
    {
        "name": "SearchIntent",
        "sent": "search {enginekeyword} for {query}",
        "matches": {
            "query": "cats",
            "enginekeyword": "instagram"
        },
        "conf": 0.9131614920899851
    }
    
    search instagram for calico cats
    {
        "name": "SearchIntent",
        "sent": "search {enginekeyword} for {query}",
        "matches": {
            "query": "calico cats",
            "enginekeyword": "instagram"
        },
        "conf": 0.9131614920899851
    }
    
    what's the weather like in san francisco
    {
        "name": "WeatherIntent",
        "sent": "what ' s the weather like in {locationkeyword}",
        "matches": {
            "locationkeyword": "san francisco"
        },
        "conf": 0.8760099391646735
    }
    
    is it raining in seattle
    {
        "name": "WeatherIntent",
        "sent": [
            "is",
            "it",
            "raining",
            "in",
            "seattle"
        ],
        "matches": {
            "WeatherTypeKeyword": "raining",
            "LocationKeyword": "seattle"
        },
        "conf": 1.0
    }
    
    play some music by the clash
    {
        "name": "MusicIntent",
        "sent": "play {artistkeyword}",
        "matches": {
            "artistkeyword": "some music by the clash"
        },
        "conf": 0.9070281000702336
    }
    
    add cereal to my shopping list
    {
        "name": "HelloIntent",
        "sent": "add cereal to my shopping list",
        "matches": {},
        "conf": 0.12597418802567345
    }

    In padatious, not only is the code shorter and the intents are basically just sentences stating what sorts of phrases should trigger the plugin, but it also got the query term correct while I was not able to get adapt to. Unfortunately, it does match the unmatched intent to something but with a very low confidence. I tried putting in an unknown intent type to catch anything that didn’t match the other intents, but that ended up matching a lot of the phrases with slight variations from the patterns (“hi there”, “bye bye”, and “what’s the weather like in san francisco”).

    Also, when matching “play some music by the clash” the search is returning “some music by the clash” as the match, even though ArtistKeyword is given explicitly, making me think it isn’t actually using the keyword entities.

    So both engines have good and bad points. As far as coming up with a system that could be used with both of them, Adapt is much closer to what Naomi currently uses than Padatious. It should be possible to use this “required words” “optional words” style to generate a bunch of phrases to feed to padatious, but I don’t think that would really be fair to padatious as a test of its parsing abilities.

    The style for writing intents for the Amazon Echo, Snips and, from what I can glean from their website, PicoVoice/Rhino, appear to be closer to Padatious, and intent-parser is closer to Adapt.

    Interestingly enough, the Mycroft documentation for skill authors (skills being the Mycroft and Amazon Echo equivalent of Naomi Speechhandlers) basically has them using both Adapt and Padatious styles when writing skills. I don’t know if that is because they want the option of switching back and forth, or they are using both options simultaneously somehow.

    I think our development guidelines for speechhandler authors should require both approaches since intent parsers seem to fall into one or the other camps right now. This could be seen as creating extra, unnecessary work for plugin authors, but I think both activities, writing out lists of ways people might activate your handler and listing keywords you would like your handler to respond to, actually will help the author identify issues with the intent parser, and that way the user will be free to choose any intent parser.

    I’m currently working on a generic intent parser language for Naomi that basically uses both approaches and can be used by either Padatious or Adapt. Let me know if you have any thoughts.

    #591

    After rereading my previous post, I realize that the difference between Adapt and Padatious may not be as clear as I thought, partly because of the length of the examples and partly because the interesting parts got cut off to the right. The primary difference to me is in the definition of an intent.

    Adapt:
    IntentBuilder(“WeatherIntent”)
        .require(“WeatherTypeKeyword”)
        .optionally(“LocationKeyword”)
        .optionally(“TimeKeyword”)
        .build()

    Padatious:
    container.add_intent(‘WeatherIntent’, [
        “what’s the weather in {LocationKeyword}”,
        “is it {WeatherTypeKeyword} in {LocationKeyword}”
    ])

    Adapt builds up intents atomically. The author of the intent decides on a set of keywords that indicate that this intent is what what the user was attempting to trigger. Additionally you can have “optional” keywords. These keywords, if present, absolutely identify the intent. Given the above intent and the following definitions:

    for word in [‘weather’, ‘forecast’, ‘snow’, ‘snowing’, ‘rain’, ‘raining’, ‘wind’, ‘windy’, ‘sleet’, ‘sleeting’, ‘sunny’]:
        engine.register_entity(word, “WeatherTypeKeyword”)
    for word in [‘seattle’, ‘san francisco’, ‘tokyo’]:
        engine.register_entity(word, “LocationKeyword”)
    for word in [‘today’, ‘tomorrow’, ‘this weekend’]:
        engine.register_entity(word, “TimeKeyword”)

    The following test phrases all have a probability of 1.0 — Adapt is 100% sure that the user is asking a question about the weather:

    test_phrases = [
        “weather”,
        “what’s the forecast for seattle”,
        “is it raining in seattle”,
        “will it rain in seattle tomorrow”
    ]

    That despite the fact that the first phrase contains only a single trigger word and the last phrase contains a weather word, a location word and a time word. Since Adapt is 100% sure with just a trigger word, it can’t be any more sure.

    Now one problem that might strike you right away is that “weather” and “forecast” are not quite the same category as “rain” or “snow”. The second thing is that we have to list “rain” and “raining” separately because we anticipate questions like “Will it rain tomorrow?” or “Is it raining?” and Adapt has no built-in concept of morphology.

    To deal with the “rain/raining” issue, we would have to add a morphology parser that would compare each word in the utterance to the root word in the intent. That is not all that difficult since the nltk module already contains a stem variant for most languages, but it is something to keep in mind, since the author of the plugin might want to handle “rain” and “raining” differently. In the absence of a time keyword, for example, a phrase containing “raining” might check only whether it is currently raining, but a phrase containing rain might check the ten day forecast and respond with “There is an 87% chance that it will rain on Wednesday”

    To deal with the “weather/forecast” issue, we can re-write the intent as
    for word in [‘weather’, ‘forecast’]:
        engine.register_entity(word, “WeatherKeyword”)
    for word in [‘snow’, ‘snowing’, ‘rain’, ‘raining’, ‘wind’, ‘windy’, ‘sleet’, ‘sleeting’, ‘sunny’]:
        engine.register_entity(word, “WeatherConditionKeyword”)
    for word in [‘seattle’, ‘san francisco’, ‘tokyo’]:
        engine.register_entity(word, “LocationKeyword”)
    for word in [‘today’, ‘tomorrow’, ‘this weekend’]:
        engine.register_entity(word, “TimeKeyword”)
    engine.register_intent_parser(IntentBuilder(“WeatherIntent”)
        .require(“WeatherKeyword”)
        .optionally(“WeatherConditionKeyword”).
        optionally(“LocationKeyword”).
        optionally(“TimeKeyword”)
        .build()
    )
    engine.register_intent_parser(IntentBuilder(“WeatherIntent”)
        .require(“WeatherConditionKeyword”)
        .optionally(“LocationKeyword”)
        .optionally(“TimeKeyword”)
        .build()
    )

    This still results in a 100% match for all of our examples, but reads a bit better.

    It becomes a bit more interesting when we get rid of the require keywords and just use optional keywords:

    weather
    {
        “intent_type”: “WeatherIntent”,
        “WeatherKeyword”: “weather”,
        “target”: null,
        “confidence”: 1.0
    }

    what’s the forecast for seattle
    {
        “intent_type”: “WeatherIntent”,
        “WeatherKeyword”: “forecast”,
        “LocationKeyword”: “seattle”,
        “target”: null,
        “confidence”: 0.4838709677419355
    }

    is it raining in seattle
    {
        “intent_type”: “WeatherIntent”,
        “WeatherConditionKeyword”: “raining”,
        “LocationKeyword”: “seattle”,
        “target”: null,
        “confidence”: 0.5833333333333334
    }

    will it rain in seattle tomorrow
    {
        “intent_type”: “WeatherIntent”,
        “WeatherConditionKeyword”: “rain”,
        “LocationKeyword”: “seattle”,
        “TimeKeyword”: “tomorrow”,
        “target”: null,
        “confidence”: 0.59375
    }

    Now we start to see some level of doubt on the part of the parser, where it’s only really sure it knows what you want when you use the single word “weather”, a term that doesn’t match any other intent. It’s more confident when all three slots for the intent are filled (weather condition, location and time) but not much more than when only the weather condition and location slots are filled. It is interesting to me that it is much less sure when matching on the first case where only three out of four optional slots are filled than in the second case when all three option slots are filled in. This means that your algorithm is actually being penalized for having a bunch of things to check for and match, something that would make a human listener more confident.

    But it should seem obvious that words like ‘weather’ or ‘forecast’ should carry more weight than just the fact that a sentence contains two out of three optional keywords. Say if I set up the computer to tell me both about sports events in addition to weather events:

    engine.register_intent_parser(IntentBuilder(“WeatherIntent”).optionally(“WeatherKeyword”).optionally(“WeatherConditionKeyword”).optionally(“LocationKeyword”).optionally(“TimeKeyword”).build())
    for word in [‘game’, ‘play’]:
        engine.register_entity(word, “GameKeyword”)
    for word in [‘seahawks’, ‘bengals’]:
        engine.register_entity(word, “TeamKeyword”)
    engine.register_intent_parser(IntentBuilder(“GameIntent”).optionally(“GameKeyword”).optionally(“TeamKeyword”).optionally(“TeamKeyword”).optionally(“LocationKeyword”).optionally(“TimeKeyword”).build())

    Which responds with
    will it rain in seattle tomorrow
    {
        “intent_type”: “WeatherIntent”,
        “WeatherConditionKeyword”: “rain”,
        “LocationKeyword”: “seattle”,
        “TimeKeyword”: “tomorrow”,
        “target”: null,
        “confidence”: 0.59375
    }

    are the seahawks playing in seattle tomorrow
    {
        “intent_type”: “GameIntent”,
        “TeamKeyword”: “seahawks”,
        “LocationKeyword”: “seattle”,
        “TimeKeyword”: “tomorrow”,
        “target”: null,
        “confidence”: 0.5227272727272727
    }

    are the seahawks playing the bengals tomorrow
    {
        “intent_type”: “GameIntent”,
        “TeamKeyword”: “seahawks”,
        “TimeKeyword”: “tomorrow”,
        “target”: null,
        “confidence”: 0.3407407407407408
    }

    what’s happening in seattle tomorrow
    {
        “intent_type”: “WeatherIntent”,
        “LocationKeyword”: “seattle”,
        “TimeKeyword”: “tomorrow”,
        “target”: null,
        “confidence”: 0.41666666666666663
    }

    In the last case, since only location and time keywords are used, and no weather or game specific keywords are used, you might expect something like “the seahawks will be playing in seattle tomorrow evening. there is an 85% chance that it will rain.” since it should match both equally. Even when I removed the second “TeamKeyword” (which failed to match the second team anyway) I got the exact same result.

    There are a couple of odd structural problems here. Adapt is unable to return two results when the likelihood seems evenly split, and unable to return additional keywords when they appear in the query, which probably means that it’s not taking those additional keywords into account. We also see that there are some keywords that almost definitely, but not always (“add game to my shopping list”, or “play it’s raining men” for instance), mean that the intent should be activated and other words that strongly indicate that an intent is requested without being required. But coming up with weights for those words would be difficult, especially if the plugin authors are coming up with their own weights and trying to balance against every other plugins weights without knowing that there is a shopping list app, or songs with weather conditions in the titles.

    Wouldn’t it be great if there was some way of automatically calculating which words are most important for all the plugins at once? Well, good news. I’ll be talking about Padatious tomorrow.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.