Google’s new AI feature tells phones when to scroll

(Subscribe to our Today's Cache newsletter for a quick snapshot of top 5 tech stories. Click here to subscribe for free.)

Google has introduced a new machine learning-based feature in Recorder, its recording app, that automatically marks important sections in the transcript, selects the important keywords from each section and surfaces them on scrollbar like chapter headings.

Through Smart Scrolling, the user can scroll through the keywords or tap on them to quickly navigate to the sections of interest.

The feature functions by extracting representative keywords from each section, and then picking sections in the text that are most informative and unique.

Google said the models used are lightweight and can be executed on-device without the need to upload the script, which also helps preserve user privacy.

How does the model work?

The California-based company utilised bidirectional transformer (BERT) model, pre-trained on data sourced from a Wikipedia dataset, alongside a modified extractive term frequency–inverse document frequency (TF-IDF) model to build Smart Scrolling feature.

Google’s ‘Verse by Verse’ can help you write poetry

BERT mechanism helped achieve context-aware processing of the input text- to identify contextual clues both before and after a given position in the transcript.

While the TF-IDF approach rated terms based on their frequency in the text compared to their inverse frequency in the trained dataset. This enabled the finding of unique representative terms in the text. The model detected informative keywords by giving each word a score, depending on how representative the keyword is within the text.

In parallel, the BERT model was fine-tuned on the task of extracting keywords. It provided a semantic understanding of the text, enabling it to extract precise context-aware keywords.

To rate a section’s importance, the model took the TF-IDF scores of all the keywords in the section and weighted them by their respective number of appearances in the section, followed by a summation of individual keyword scores.

The Google team computed the second score by running the section text through the bidirectional transformer model, which was also trained on the sections rating task. The scores from both models were normalised and combined to yield the section score.

It selected the top scored sections comprising of highly rated keywords with the number of sections highlighted proportional to the length of the recording to identify whether a section or keyword is important to an individual or not.

In the context of the Smart Scrolling features, a keyword was more highly rated if it better represented the unique information of the section.

Google said both models were trained on publicly available conversational datasets that were labelled and evaluated by independent raters