AI Builder Object Detection and Mixed Reality Play Together

Ever since the preview days of AI Builder, I’ve been an enthusiast of creating AI-powered apps – much like my passion for diving into Mixed Reality. The evolution of these technologies continues to fuel my personal and professional excitement, as I eagerly anticipate their integration into our everyday experiences. It’s fascinating and fulfilling that you could build an application and flow using AI and Mixed Reality in Power Platform without code.  

After finishing up the AI Skills Challenge, which introduces you to AI Builder, build and train the models, and use the model in your Power Apps and Power Automate. Three years back, I built an app with two preview features. Today, those features have flourished into essential elements of modern software. This journey underscores the importance of embracing change and fuels my enthusiasm for continued innovation. Here is how it works.

BUSINESS SCENARIO:

As we are into buying online, you wanted spare parts for your toy collection. By uploading the image, you can see if they have the item, and you can interact with the item you want to buy. 

https://videopress.com/embed/Kk8Hj7kf?cover=1&preloadContent=metadata&useAverageColor=1&hd=1Explore the collection with AI Builder and Mixed Reality.

HOW IT WORKS:

I used AI Builder: Object Detection and Mixed Reality in this application. AI Builder Object detection is a custom AI model. It provides object recognition capabilities. You need to train this model and have a premium license. Mixed Reality component in Power Apps to view the object using 3D. This component does not need a premium license.

HOW I BUILD IT:

I’m going to show you how I build it. Let’s start with the AI Builder Object detection.

Object Detection

AI Builder Object Detection and Mixed Reality Play Together
  • After selecting the object detection, this will show tips and documentation on how you will do it. Select create custom model.
  • Select the model’s domain from Common objects, Objects on retail shelves, or brand logos.
AI Builder Object Detection and Mixed Reality Play Together
Objects on retail shelves AI model detection
Brand logos AI model detection

In the app, I selected the Common objects as my object is not a brand logo or on the retail shelves.

  • Select NextChoose objects for your model. You need to select your database or add the object manually. You can select from a database or add them manually. If you have existing data in your Dataverse, you can use it. On my end, I manually input the object names.
AI Builder Object Detection and Mixed Reality Play Together
  • Next is Add images. These images are objects that AI needs to detect. A minimum of 15 images is needed. You can select your data source from My Device, SharePoint, or Azure Blob Storage. I uploaded it from my device.
AI Builder Object Detection and Mixed Reality Play Together
  • After uploading 15 images, tag the image to its corresponding object names. Select Done tagging when you are done.
AI Builder Object Detection and Mixed Reality Play Together
  • Train your model.
AI Builder Object Detection and Mixed Reality Play Together

You can do a quick test with your model or train more if you think the performance is below what you expected. It means you need to upload more images and train your model again.

AI Builder Object Detection and Mixed Reality Play Together

If you’re satisfied, we will use the model in our app.

USE THE MODEL IN POWER APPS:

I used Microsoft Lists to store the TagId, Images, Name of the collection, and Object Name. Each item has a .GLB file for my 3D object, which I will use in our app. The name of my list is Product Details.

AI Builder Object Detection and Mixed Reality Play Together

ESPC call for speakers 2024
The TagId is a unique id from the Object Detection model that the object uses. We will use this to display the 3D object to the app and other properties we need from an item.

  • To get the TagId, Insert the Object detector component from the Power Apps.
AI Builder Object Detection and Mixed Reality Play Together
  • Insert a Gallery and set the Items to ObjectDetector.Results. In my app, I have Objectector2.Results. I used the Title and Subtitle layout. The name of my gallery is Gallery4. (You can rename the gallery)
AI Builder Object Detection and Mixed Reality Play Together
  • Set the title to ThisItem.TagName
  • Set the subtitle to ThisItem.TagId
AI Builder Object Detection and Mixed Reality Play Together
  • Run the app in testing mode, and insert the images to copy the TagId. After copying the TagId, paste it to its corresponding item in the Lists.
AI Builder Object Detection and Mixed Reality Play Together
  • We need to add another gallery. The items are from the Product Details Lists. We filter the item from the Gallery4 TagId to display the corresponding item base on the image detected. The name of my gallery inserted is Gallery5. (You can rename it)
Filter('Product Details',Gallery4.Selected.TagId = TagId)
AI Builder Object Detection and Mixed Reality Play Together

Now we are done setting up our Object detector and querying the data from the list. We will now set up the Mixed Reality component to display the 3D object in the app.

  • Select insert >> Media >> 3D object.
AI Builder Object Detection and Mixed Reality Play Together

In the 3D object source settings, set this function below:

First(Gallery5.Selected.Attachments).Value

This function will query the attached .GLB file in the Product Details list item.

AI Builder Object Detection and Mixed Reality Play Together

Save it and publish your app! 

SUMMARY

MY ENTHUSIASM FOR CRAFTING AI-POWERED APPS HAS ONLY GROWN from AI Builder’s early preview to my ongoing journey with Mixed Reality. Building AI and Mixed Reality within Power Platform, all without writing a single line of code. Welcome to the future of innovation! 

REFERENCE

About the Author

I am working as Productivity Analyst focus on SharePoint, Power Platform (Power Apps, and Power Automate) and Microsoft 365. Dual MVP (Business Applications & Microsoft 365 Apps & Services).

Reference

Cerbolles, A., 2023, Add New Line Items to Repeating Tables in Power Apps and SharePoint List, Anjcerbolles.com, Available at: https://anjcerbolles.com/2023/07/31/add-new-line-items-to-repeating-tables-in-power-apps-and-sharepoint-list/ [Accessed on 7 September 2023]

STAY UP TO DATE

Catch up on the latest blogseBookswebinars, and how-to videos.
Not a member? Sign up today to unlock all content.
Subscribe to our YouTube channel for the latest community updates.

Share this on...

Rate this Post:

Share: