Integrate Android widgets with Google Assistant

1. Overview

In the first App Actions codelab, you learned how to extend Google Assistant to a sample fitness app by implementing built-in intents (BII) from the Health and Fitness BII category.

App Actions let users launch directly into specific app features from Assistant by asking things like, "Hey Google, start a run on ExampleApp." In addition to launching apps, Assistant can display an interactive Android widget to the user to fulfill requests for eligible BIIs.

$A screen showing Assistant returning a widget in response to a\nuser query that triggered an apps GET_EXERCISE_OBSERVATION BII capability.$

What you'll build

In this codelab, you learn how to return Android widgets to fulfill Assistant user requests. You also learn to:

User BII parameters to personalize widgets.
Provide text-to-speech (TTS) introductions in Assistant for your widgets.
Use the Built-in intent reference to determine which BIIs support widget fulfillment.

Prerequisites

Before continuing, ensure your development environment is ready for App Actions development. It should have:

A terminal to run shell commands, with git installed.
The latest stable release of Android Studio.
A physical or virtual Android device with Internet access.
A Google Account that is signed into Android Studio, the Google app, and the Google Assistant app.

If you are using a physical device, connect it to your local development machine.

2. Understand how it works

Google Assistant uses natural language understanding (NLU) to read a user's request and match it to an Assistant built-in intent (BII). Assistant then maps the intent to the capability (that implements the BII), which you register for that intent in your app. Finally, Assistant fulfills the user's request by displaying the Android widget your app generates using the details found in the capability.

In this codelab, you define a capability that registers support for the GET_EXERCISE_OBSERVATION BII. In this capability, you instruct Assistant to generate an Android intent to the FitActions widget class to fulfill requests for this BII. You update this class to generate a personalized widget for Assistant to display to the user, and a TTS introduction for Assistant to announce.

The following diagram demonstrates this flow:

The FitActions sample app contains a workout information widget that users can add to their home screen. This widget is a great candidate for fulfilling user queries that trigger the GET_EXERCISE_OBSERVATION BII.

When a user adds a widget to the home screen, the widget pings the device Broadcast Receiver. This service retrieves information about the widget from the widget's receiver definition in the app's AndroidManifest.xml resource. It uses this information to generate a RemoteViews object representing the widget.

The sample app defines the receiver widgets.StatsWidgetProvider, which corresponds to the StatsWidgetProvider class:

<!-- app/src/main/AndroidManifest.xml -->

<receiver
  android:name=".widgets.StatsWidgetProvider"
  android:exported="false">
  <intent-filter>
    <action android:name="android.appwidget.action.APPWIDGET_UPDATE" />
  </intent-filter>
  <meta-data
    android:name="android.appwidget.provider"
    android:resource="@xml/stats_widget" />
</receiver>

The StatsWidgetProvider class, StatsWidgetProvider.kt, manages the StatsWidget object creation flows. It handles these responsibilities:

Creating widget instances and populating them with exercise data from the app database.
Formatting workout data for readability, with formatDataAndSetWidget().
Providing default values if workout data is unavailable, using setNoActivityDataWidget().

Add Assistant support

In this codelab, you update the sample app to handle App Actions functionality. These changes include:

Configuring the GET_EXERCISE_OBSERVATION BII capability to return an instance of the StatsWidget object.
Updating the StatsWidget class to use App Actions features like:
- Using BII parameters, allowing users to view specific workout statistics by asking things like, "Hey Google, show my run stats on ExampleApp."
- Providing TTS introduction strings.
- Managing special cases, like when the user query does not include a workout type parameter.

3. Prepare your development environment

Download your base files

Run this command to clone the sample app's GitHub repository:

git clone --branch start-widget-codelab https://github.com/actions-on-google/appactions-fitness-kotlin.git

Once you've cloned the repository, follow these steps to open it in Android Studio:

In the Welcome to Android Studio dialog, click Import project.
Find and select the folder where you cloned the repository.

To see a version of the app representing the completed codelab, clone the sample app repo using the --branch master flag.

Update the Android application ID

Updating the app's application ID uniquely identifies the app on your test device and avoids a "Duplicate package name" error if the app is uploaded to the Play Console. To update the application ID, open app/build.gradle:

android {
...
  defaultConfig {
    applicationId "com.MYUNIQUENAME.android.fitactions"
    ...
  }
}

Replace "MYUNIQUENAME" in the applicationId field to something unique to you.

Install the test plugin

The Google Assistant plugin lets you test your App Actions on a test device. It works by sending information to Assistant through the Google app on your Android device. If you do not already have the plugin, install it with these steps:

Go to File > Settings (Android Studio > Preferences on MacOS).
In the Plugins section, go to Marketplace and search for "Google Assistant". You can also manually download and install the test tool.
Install the tool and restart Android Studio.

Test the app on your device

Before making more changes to the app, it helps to get an idea of what the sample app can do.

Run the app on your test device:

In Android Studio, select your physical or virtual device and select Run > Run app or click Run in the toolbar.
Long-press the Home button to set up Assistant and verify that it works. You will need to sign in to Assistant on your device, if you haven't already.

For more information on Android virtual devices, see Create and manage virtual devices.

Briefly explore the app to see what it can do. The app prepopulates 10 exercise activities and displays this information on the first view.

Tap the Home button to go to your test device's home screen.
Long-press an empty space on the home screen and select Widgets.
Scroll down the widget list to FitActions.
Long-press the FitActions icon and place its widget on the home screen.

4. Add the App Action

In this step, you add the GET_EXERCISE_OBSERVATION BII capability. You do this by adding a new capability element in shortcuts.xml. This capability specifies how the capability is triggered, how BII parameters are used, and which Android intents to invoke to fulfill the request.

Add a new capability element to the sample project shortcuts.xml resource with this configuration:

<!-- fitnessactions/app/src/main/res/xml/shortcuts.xml -->

<capability android:name="actions.intent.GET_EXERCISE_OBSERVATION">
  <app-widget
    android:identifier="GET_EXERCISE_OBSERVATION"
    android:targetClass="com.devrel.android.fitactions.widgets.StatsWidgetProvider"
    android:targetPackage="PUT_YOUR_APPLICATION_ID_HERE">
    <parameter
      android:name="exerciseObservation.aboutExercise.name"
      android:key="aboutExerciseName"
      android:required="true">
    </parameter>
    <extra android:name="hasTts" android:value="true"/>
  </app-widget>
  <!-- Add Fallback Intent-->
</capability>

Replace the android:targetPackage value, PUT_YOUR_APPLICATION_ID_HERE, with your unique applicationId.

This capability maps the GET_EXERCISE_OBSERVATION BII to the app-widget intent so that when the BII is triggered, the widget instantiates and displays to the user.

Before triggering the widget, Assistant extracts supported BII parameters from the user query. This codelab requires the BII parameter exerciseObservation.aboutExercise.name, which represents the user's requested exercise type. The app supports three exercise types: "running", "walking", and "cycling." You provide an inline inventory to inform Assistant of these supported values.

Define these inventory elements by adding this configuration, above the GET_EXERCISE_OBSERVATION capability, to shortcuts.xml:

<!-- shortcuts.xml -->

<!-- shortcuts are bound to the GET_EXERCISE_OBSERVATION capability and
     represent the types of exercises supported by the app. -->

<shortcut
  android:shortcutId="running"
  android:shortcutShortLabel="@string/activity_running">
  <capability-binding android:key="actions.intent.GET_EXERCISE_OBSERVATION">
    <parameter-binding
      android:key="exerciseObservation.aboutExercise.name"
      android:value="@array/runningSynonyms"/>
  </capability-binding>
</shortcut>

<shortcut
  android:shortcutId="walking"
  android:shortcutShortLabel="@string/activity_walking">
  <capability-binding android:key="actions.intent.GET_EXERCISE_OBSERVATION">
    <parameter-binding
      android:key="exerciseObservation.aboutExercise.name"
      android:value="@array/walkingSynonyms"/>
  </capability-binding>
</shortcut>

<shortcut
  android:shortcutId="cycling"
  android:shortcutShortLabel="@string/activity_cycling">
  <capability-binding android:key="actions.intent.GET_EXERCISE_OBSERVATION">
    <parameter-binding
      android:key="exerciseObservation.aboutExercise.name"
      android:value="@array/cyclingSynonyms"/>
  </capability-binding>
</shortcut>

<capability android:name="actions.intent.GET_EXERCISE_OBSERVATION">
  <!-- ... -->
</capability>

Add a fallback intent

Fallback intents handle situations where a user query cannot be fulfilled because the query is missing parameters required by the capability. The GET_EXERCISE_OBSERVATION capability requires the exerciseObservation.aboutExercise.name parameter, specified by the attribute android:required="true". For these situations, Assistant requires you to define a fallback intent to allow the request to succeed, even if no parameters are provided in the query.

In shortcuts.xml, add a fallback intent to the GET_EXERCISE_OBSERVATION capability using this configuration:

<!-- shortcuts.xml -->

<capability android:name="actions.intent.GET_EXERCISE_OBSERVATION">

  <app-widget>
    <!-- ... -->
  </app-widget>

  <!-- Fallback intent with no parameters needed to successfully execute.-->
  <intent
    android:identifier="GET_EXERCISE_OBSERVATION_FALLBACK"
    android:action="android.intent.action.VIEW"
    android:targetClass="com.devrel.android.fitactions.widgets.StatsWidgetProvider">
  </intent>
</capability>

In this sample configuration, the fallback fulfillment is an Android intent with no parameters in its Extra data.

5. Enable the widget for Assistant

With the GET_EXERCISE_OBSERVATION capability established, update the widget class to support App Actions voice invocation.

Add the Widgets Extension library

The App Actions Widgets Extension library enhances your widgets for voice-forward Assistant experiences. Specifically, it enables you to provide a custom TTS introduction for your widgets.

Add the Widgets Extension library dependency to the sample app /app/build.gradle resource:
```
// app/build.gradle

dependencies {
  //...
  implementation "com.google.assistant.appactions:widgets:0.0.1"
}
```
Click Sync Now in the warning box that appears in Android Studio. Syncing after every build.gradle change helps you avoid errors when building the app.

A Service is an application component that can perform long-running operations in the background. Your app needs to provide a service to process widget requests.

Add a service to the sample app's AndroidManifest.xml resource with this configuration:

<!-- AndroidManifest.xml -->
<service
   android:name=".widgets.StatsWidgetProvider"
   android:enabled="true"
   android:exported="true">
   <intent-filter>
       <action
           android:name="com.google.assistant.appactions.widgets.PIN_APP_WIDGET" />
   </intent-filter>
</service>

During a voice query that triggers widget fulfillment, Assistant uses this service to send requests to the app. The service receives the request along with the BII data. The service uses this data to generate a RemoteView widget object to render within Assistant.

Your app is now configured to route GET_EXERCISE_OBSERVATION capability requests to your widget class. Next, update the StatsWidget.kt class to generate a widget instance that is personalized to the user request, using BII parameter values.

Open the StatsWidget.kt class and import the App Actions Widget Extension library:

// StatsWidget.kt

// ... Other import statements
import com.google.assistant.appactions.widgets.AppActionsWidgetExtension

Add these private variables, which you use when determining the information that should populate the widget:

// StatsWidget.kt

private val hasBii: Boolean
private val isFallbackIntent: Boolean
private val aboutExerciseName: String
private val exerciseType: FitActivity.Type

Add the init function to let the class use the widget options data passed from Assistant:

// StatsWidget.kt

init {
  val optionsBundle = appWidgetManager.getAppWidgetOptions(appWidgetId)
  val bii = optionsBundle.getString(AppActionsWidgetExtension.EXTRA_APP_ACTIONS_BII)
  hasBii = !bii.isNullOrBlank()
  val params = optionsBundle.getBundle(AppActionsWidgetExtension.EXTRA_APP_ACTIONS_PARAMS)

  if (params != null) {
    isFallbackIntent = params.isEmpty
    if (isFallbackIntent) {
      aboutExerciseName = context.resources.getString(R.string.activity_unknown)
    } else {
        aboutExerciseName = params.get("aboutExerciseName") as String
      }
  } else {
      isFallbackIntent = false
      aboutExerciseName = context.resources.getString(R.string.activity_unknown)
  }
  exerciseType = FitActivity.Type.find(aboutExerciseName)
}

Let's walk through how these updates enable the StatsWidget.kt class to respond to Android intents generated by the GET_EXERCISE_OBSERVATION capability:

optionsBundle = Bundle
- Bundles are objects that are intended to be used across process boundaries, between activities with intents, and to store transient state across configuration changes. Assistant uses Bundle objects to pass configuration data to the widget.
bii = actions.intent.GET_EXERCISE_OBSERVATION
- The name of the BII is available from the Bundle using the AppActionsWidgetExtension.
hasBii = true
- Checks to see if there is a BII.
params = Bundle[{aboutExerciseName=running}]
- A special Bundle, generated by App Actions, is nested inside the widget options Bundle. It contains the key/value pairs of the BII. In this case, the value running was extracted from the example query, "Hey Google, show my running stats on ExampleApp."
isFallbackIntent = false
- Checks for the presence of required BII parameters in the intent Extras.
aboutExerciseName = running
- Gets the intent Extras value for aboutExerciseName.
exerciseType = RUNNING
- Uses aboutExerciseName to look up the corresponding database type object.

Now that the StatsWidget class can process incoming App Actions Android intent data, update the widget creation flow logic to check if the widget was triggered by an App Action.

In StatsWidget.kt, replace the updateAppWidget() function with this code:

// StatsWidget.kt

fun updateAppWidget() {
   /**
    * Checks for App Actions BII invocation and if BII parameter data is present.
    * If parameter data is missing, use data from last exercise recorded to the
    *  fitness tracking database.
    */
   if (hasBii && !isFallbackIntent) {
       observeAndUpdateRequestedExercise()
   } else observeAndUpdateLastExercise()
}

The preceding code references a new function, observeAndUpdateRequestedExercise. This function generates widget data using the exerciseType parameter data passed by the App Actions Android intent.

Add the observeAndUpdateRequestedExercise function with this code:

// StatsWidget.kt

/**
* Create and observe the last exerciseType activity LiveData.
*/
private fun observeAndUpdateRequestedExercise() {
  val activityData = repository.getLastActivities(1, exerciseType)

   activityData.observeOnce { activitiesStat ->
       if (activitiesStat.isNotEmpty()) {
           formatDataAndSetWidget(activitiesStat[0])
           updateWidget()
       } else {
           setNoActivityDataWidget()
           updateWidget()
       }
   }
}

In the preceding code, use an existing repository class found in the app to retrieve fitness data from the app's local database. This class provides an API simplifying access to the database. The repository works by exposing a LiveData object when performing queries against the database. In your code you observe this LiveData to retrieve the latest fitness activity.

Enable TTS

You can provide a TTS string for Assistant to announce when displaying your widget. We recommend including this to provide audible context with your widgets. This functionality is provided by the App Actions Widgets Extension library, which lets you set the text and TTS introductions that accompany your widgets in Assistant.

A good place to provide your TTS introduction is in the formatDataAndSetWidget function, which formats the activity data returned from the app database.

In StatsWidget.kt, add this code to the formatDataAndSetWidget function:

// StatsWidget.kt

private fun formatDataAndSetWidget(
  activityStat: FitActivity,
) {
      // ...

      // Add conditional for hasBii for widget with data
      if (hasBii) {
         // Formats TTS speech and display text for Assistant
         val speechText = context.getString(
             R.string.widget_activity_speech,
             activityExerciseTypeFormatted,
             formattedDate,
             durationInMin,
             distanceInKm
         )
         val displayText = context.getString(
             R.string.widget_activity_text,
             activityExerciseTypeFormatted,
             formattedDate
         )
         setTts(speechText, displayText)
      }
}

The preceding code references two string resources: one for speech, and the other for text. Check out the Text-to-Speech style recommendation portion of our widgets video for TTS recommendations. The sample also refers to setTts, a new function that provides the TTS information to the widget instance.

Add this new setTts function to StatsWidget.kt using this code:

// StatsWidget.kt

/**
 * Sets TTS to widget
 */
private fun setTts(
  speechText: String,
  displayText: String,
) {
  val appActionsWidgetExtension: AppActionsWidgetExtension =
      AppActionsWidgetExtension.newBuilder(appWidgetManager)
        .setResponseSpeech(speechText)  // TTS to be played back to the user
        .setResponseText(displayText)  // Response text to be displayed in Assistant
        .build()

  // Update widget with TTS
  appActionsWidgetExtension.updateWidget(appWidgetId)
}

Finally, complete the TTS logic by setting TTS information when the exercise database returns empty data for a requested workout type.

Update the setNoActivityDataWidget() function in StatsWidget.kt with this code:

// StatsWidget.kt

private fun setNoActivityDataWidget() {
  // ...
  // Add conditional for hasBii for widget without data
  if (hasBii) {
    // formats speech and display text for Assistant
    // https://developers.google.com/assistant/app/widgets#library
    val speechText =
      context.getString(R.string.widget_no_activity_speech, aboutExerciseName)
    val displayText =
      context.getString(R.string.widget_no_activity_text)

    setTts(speechText, displayText)
  }
}

6. Test the App Action

During development, use the Google Assistant plugin to preview Assistant App Actions on a test device. You can adjust intent parameters for an App Action with the tool to test how your action handles the various ways a user might ask Assistant to run it.

Create a preview

To test your App Action with the plugin:

Go to Tools > Google Assistant > App Actions Test Tool. You may be asked to sign in to Android Studio using your Google account.
Click Create Preview. If asked, review and accept the App Actions policies and terms of service.

Test an expected exercise type

Return a widget showing information about the last run completed in the app by following these steps in the test tool:

In the first step where the tool asks you to select and configure a BII, select actions.intent.GET_EXERCISE_OBSERVATION.
In the exerciseObservation box, update the default Exercise name from climbing to run.
Click Run App Action.

Test an unexpected exercise type

To test an unexpected exercise type in the test tool:

In the exerciseObservation box, update the name value from Run to Climbing.
Click Run App Action.

Assistant should return a widget displaying "No activity found" information.

Test the fallback intent

Queries triggering the fallback intent should return a widget displaying information about the last logged activity of any exercise type.

To test the fallback intent:

In the exerciseObservation box, delete the aboutExercise object.
Click Run App Action.

Assistant should return a widget displaying information for the last completed exercise.

7. Next steps

Congratulations!

You now have the power to fulfill users' queries using an Android Widget with Assistant.

What we've covered

In this codelab, you learned how to:

Add an app widget to a BII.
Modify a widget to access parameters from Android Extras.

What's next

From here, you can try making further refinements to your fitness app. To reference the finished project, see the main repo on GitHub.

Here are some suggestions for further learning about extending this app with App Actions:

Visit the App Actions built-in intents reference to discover more ways to extend your apps to Assistant.

To continue your Actions on Google journey, explore these resources:

developers.google.com/assistant/app: Official documentation site for Google Assistant App Actions.
App Actions sample index: Sample apps and code for exploring App Actions capabilities.
Actions on Google GitHub repo: Sample code and libraries.
r/GoogleAssistantDev: Official Reddit community for developers working with Google Assistant.

Follow us on Twitter @ActionsOnGoogle to stay tuned to our latest announcements, and tweet to #appactions to share what you have built!

Feedback survey

Finally, please fill out this survey to give feedback about your experience with this codelab.