Adding conversations to your mobile app

Tags: Xamarin.Forms, UWP, Android

A few weeks ago I gave a presentation for the Global Azure Bootcamp in Belgium. The session combines the awesome Microsoft Bot Framework with mobile apps created with Xamarin to integrate the customer service of a company with a chat experience on your mobile device.

Getting started

Before we’re able to start, there are a few points to prepare:

  • Install Visual Studio 2015 or 2017 with Xamarin.
  • Download the Visual Studio Template for the Microsoft Bot Framework and extract this to:
    • VS 2015: %USERPROFILE%\Documents\Visual Studio 2015\Templates\ProjectTemplates\Visual C#
    • VS 2017: %USERPROFILE%\Documents\Visual Studio 2017\Templates\ProjectTemplates\Visual C#
  • Download the Bot Framework Emulator
  • Create a new project and test the echo bot on http://localhost:0000/api/messages (replace 0000 by the port the project is running on).

Microsoft Bot Framework

Concepts

If you have time, certainly go read on the Microsoft Bot Framework Portal on what a bot actually is and how it works, but these are the key concepts to take home:

  • Conversation: aggregation of messages between bot and
    • Single user
    • Multiple users
    • Other bots
  • Activity: message container with
    • Routing information
    • User metadata
    • Optional attachments
  • Channel: the medium over which the communication is done, e.g. Skype, Facebook, REST, etc.
    Note that not all channels support group conversations, while each channel does support one-one communication.
  • Bot Connecter Service: REST component that makes the connection between the bot and the difference channels (with users on them).
  • Bot State Service: takes care of a bot’s state, enabling a stateless web service and thus improving scalability. This is what enables keeping track of a conversation, e.g. what was the previous question asked.
  • Dialog: the conversational process based on the exchange of messages encapsulated with its own state.
  • Formflow: a guided conversation that provides a lot less flexibility but supports an increased complexity of the conversation without extra work (see below).

Code

When you start a new Bot Framework project, you’ll have a MessagesController, which is a Web API controller.

namespace ShoppingBot.Controllers
{
    [BotAuthentication]
    public class MessagesController : ApiController
    {
        /// <summary>
        /// POST: api/Messages
        /// Receive a message from a user and reply to it
        /// </summary>
        public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
        {
            if (activity.Type == ActivityTypes.Message)
            {
                await Conversation.SendAsync(activity, () => new RootDialog());
            }
            else
            {
                HandleSystemMessage(activity);
            }
            var response = Request.CreateResponse(HttpStatusCode.OK);
            return response;
        }

        private Activity HandleSystemMessage(Activity message)
        {
            if (message.Type == ActivityTypes.DeleteUserData)
            {
                // Implement user deletion here
                // If we handle user deletion, return a real message
            }
            else if (message.Type == ActivityTypes.ConversationUpdate)
            {
                // Handle conversation state changes, like members being added and removed
                // Use Activity.MembersAdded and Activity.MembersRemoved and Activity.Action for info
                // Not available in all channels
            }
            else if (message.Type == ActivityTypes.ContactRelationUpdate)
            {
                // Handle add/remove from contact lists
                // Activity.From + Activity.Action represent what happened
            }
            else if (message.Type == ActivityTypes.Typing)
            {
                // Handle knowing tha the user is typing
            }
            else if (message.Type == ActivityTypes.Ping)
            {
            }

            return null;
        }
    }
}

This controller forwards an incoming HTTP call to the RootDialog, which is nothing more than a service that sends back the word you’ve entered and its length.

namespace ShoppingBot.Dialogs
{
    [Serializable]
    public class RootDialog : IDialog<object>
    {
        public Task StartAsync(IDialogContext context)
        {
            context.Wait(MessageReceivedAsync);

            return Task.CompletedTask;
        }

        private async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> result)
        {
            var activity = await result as Activity;

            // calculate something for us to return
            int length = (activity.Text ?? string.Empty).Length;

            // return our reply to the user
            await context.PostAsync($"You sent {activity.Text} which was {length} characters");

            context.Wait(MessageReceivedAsync);
        }
    }
}

Adding state

A bot service without state would be pretty useless as you can’t build a real conversation continuing on previous steps. Simply add private fields to track anything you’d like and since the Dialog object is Serializable, these fields are serialized with your Dialog. Note that because of the Bot Connector Service, the framework knows which HTTP calls to couple to which conversation and the Bot State Service keeps tracks of all messages sent in the conversation. As soon as you’re keeping track of the current state of an object, it’s wise to add some logic to be able to reset this state as well. This can easily be done by checking the input for a keyword and act on it. To prevent unwanted resets, we added a PromptDialog asking the user if he is sure to reset.

namespace ShoppingBot.Dialogs
{
    [Serializable]
    public class EchoStateDialog : IDialog<object>
    {
        private int _count = 1;

        public Task StartAsync(IDialogContext context)
        {
            context.Wait(MessageReceivedAsync);

            return Task.CompletedTask;
        }

        public virtual async Task MessageReceivedAsync(IDialogContext context, IAwaitable<IMessageActivity> argument)
        {
            var message = await argument;
            if (message.Text == "reset")
            {
                PromptDialog.Confirm(
                    context,
                    AfterResetAsync,
                    "Are you sure you want to reset the count?",
                    "Didn't get that!",
                    promptStyle: PromptStyle.Auto);
            }
            else
            {
                await context.PostAsync($"{_count++}: You said {message.Text}");
                context.Wait(MessageReceivedAsync);
            }
        }

        public async Task AfterResetAsync(IDialogContext context, IAwaitable<bool> argument)
        {
            var confirm = await argument;
            if (confirm)
            {
                _count = 1;
                await context.PostAsync("Reset count.");
            }
            else
            {
                await context.PostAsync("Did not reset count.");
            }
            context.Wait(MessageReceivedAsync);
        }
    }
}

Stepping up to a real ordering service

So far we haven’t done much special yet and you’re probably still asking "Why would I use the Microsoft Bot Framework for this?”. Well, the next step is create a meal ordering service, showing quite a bit of ‘human-like’ interaction steered by nothing more than a few enumerations using Formflow. All the rest is done for you by the Microsoft Bot Framework.

namespace ShoppingBot.Dialogs
{
    public enum SandwichOptions
    {
        BLT, BlackForestHam, BuffaloChicken, ChickenAndBaconRanchMelt, ColdCutCombo, MeatballMarinara,
        OvenRoastedChicken, RoastBeef, RotisserieStyleChicken, SpicyItalian, SteakAndCheese, SweetOnionTeriyaki, Tuna,
        TurkeyBreast, Veggie
    }
    public enum LengthOptions { SixInch, FootLong }
    public enum BreadOptions { NineGrainWheat, NineGrainHoneyOat, Italian, ItalianHerbsAndCheese, Flatbread }
    public enum CheeseOptions { American, MontereyCheddar, Pepperjack }
    public enum ToppingOptions
    {
        Avocado, BananaPeppers, Cucumbers, GreenBellPeppers, Jalapenos,
        Lettuce, Olives, Pickles, RedOnion, Spinach, Tomatoes
    }
    public enum SauceOptions
    {
        ChipotleSouthwest, HoneyMustard, LightMayonnaise, RegularMayonnaise,
        Mustard, Oil, Pepper, Ranch, SweetOnion, Vinegar
    }

    [Serializable]
    public class SandwichOrder
    {
        //[Prompt("What kind of {&} would you like? {||}")]
        public SandwichOptions? Sandwich;
        public LengthOptions? Length;
        public BreadOptions? Bread;
        public CheeseOptions? Cheese;
        public List<ToppingOptions> Toppings;
        public List<SauceOptions> Sauce;

        public static IForm<SandwichOrder> BuildForm()
        {
            return new FormBuilder<SandwichOrder>()
                .Message("Welcome to the simple sandwich order bot!")
                .Build();
        }
    }
}

The main difference between this Dialog and the previous ones, is that you use a FormBuilder object and use the Chain keyword to tell the framework that it has to chain all questions into a nice conversation. In a console this output is very basic, but if you integrate with a client like Skype, the user will have buttons to click.

Please select a sandwich
1. BLT
2. Black Forest Ham
3. Buffalo Chicken
4. Chicken And Bacon Ranch Melt
5. Cold Cut Combo
6. Meatball Marinara
7. Oven Roasted Chicken
8. Roast Beef
9. Rotisserie Style Chicken
10. Spicy Italian
11. Steak And Cheese
12. Sweet Onion Teriyaki
13. Tuna
14. Turkey Breast
15. Veggie
>

This chain will ask human-like questions, interpret the answers (numbered choice, partial answers, …), allows you to go back a step, reset the whole flow, etc. All with just a few enumerations. You can improve this conversation by providing phrases and expressions.

Adding LUIS

While this already quite impressive, you can even improve your bot by adding LUIS. LUIS, Language Understanding Intelligent Service, is part of the cognitive services toolkit and takes conversations to the human level by being able to map variances of a single sentence onto the correct request, called intent. These variances can be configured on the LUIS portal and machine learning will improve your model. Simply add the API-key, change to a LuisDialog and add the LuisIntentAttribute on your methods and you’re good to go. You can find the complete sample on GitHub.

Going live

All this code is useless if we can’t use our bot. We’re only a few steps away:

  • Publish your project to an Azure web site and register your bot with the Microsoft Bot Framework.
  • Update the web.config file with your own Microsoft App Id and password (this is empty for local testing) and republish.
  • Create a DirectLine channel for REST or use any of the other channels, e.g. Skype.

More information

If this teaser sparked your interest in the Microsoft Bot Framework, don’t hesitate to get over to the official pages and see what else you can do.

Xamarin.Forms client

You’ve probably already heard of Xamarin.Forms and its promise to write your app once and run it on all mobile platforms. The challenges that could with creating a decent cross-platform application are:

  • Keeping the code maintainable. I usually do this by using MVVM, separating the actual views from the real code.
  • Make the UI look decent on all platforms. The most simple screens can be written once, but as soon as you want to go a step further and have a more than input fields, you’ll find yourself writing native views or native renderers very soon.

Since we want a chat interface, we want the UI/UX to be quite close to the chat/SMS interfaces of each platform, which brings us to writing a renderer to tweak the default rendering. In this particular case a ListViewCellRenderer to change the layout of a single message which is still shown in a list of messages.

UWP

Since my roots are with Windows 10, I first wrote the UWP custom renderer in just a few minutes time. Simply tell which DataTemplate has to be used when getting the template for the ViewCell.

[assembly: ExportRenderer(typeof(MessageViewCell), typeof(MessageRenderer))]
namespace ShoppingForms.UWP.CustomRenderers
{
    public class MessageRenderer : ViewCellRenderer
    {
        public override DataTemplate GetTemplate(Cell cell)
        {
            return Application.Current.Resources["MessageDataTemplate"] as DataTemplate;
        }
    }
}

Android

As I don’t have a MacBook (or any other Apple device) to compile the app on, there was only Android left to fix. About two hours later I finally got it working because of a long-known bug (which is now fixed) where you have to implement a CellRenderer instead of a ViewCellRenderer.

[assembly: ExportRenderer(typeof(MessageViewCell), typeof(MessageRenderer))]
namespace ShoppingForms.Droid.CustomRenderers
{
    // bug https://bugzilla.xamarin.com/show_bug.cgi?id=38989 CellRenderer instead of ViewCellRenderer
    public class MessageRenderer : CellRenderer
    {
        protected override View GetCellCore(Cell item, View convertView, ViewGroup parent, Context context)
        {
            var inflatorservice = (LayoutInflater)Forms.Context.GetSystemService(Context.LayoutInflaterService);
            var dataContext = item.BindingContext as BotMessage;

            if (dataContext != null)
            {
                var template = (LinearLayout)inflatorservice.Inflate(dataContext.IsMine()
                            ? Resource.Layout.message_item_owner
                            : Resource.Layout.message_item_opponent, null, false);
                template.FindViewById<TextView>(Resource.Id.nick).Text =
                    (dataContext.IsMine() ? Constants.MyName : Constants.FriendName) + ":";
                template.FindViewById<TextView>(Resource.Id.message).Text =
                    dataContext.Text;
                return template;
            }

            return base.GetCellCore(item, convertView, parent, context);
        }
    }
}

This is a lot of text to dig through, so I suggest to just download the code and run it yourself.

Comments

comments powered by Disqus