Virtual Entities – How to Create a Custom Data Provider

This is the second blog post in a series of four blog posts about Virtual Entites. In this one I will let you in on how to create a Custom Data Provider. As mentioned in my previous blog post, Virtual Entities provide a possibility to display and work with data (display in views, make advanced finds, show the records etc.) without having the data stored in the Common Data Service.

I also mentioned that there are two built-in Data Providers you can use and that is an OData v4 Data Provider and a Cosmos DB Data Provider. If you want to create a Virtual Entity and your data is not located on a Cosmos DB or does not support OData v4, then you have the possibility to create a Custom Data Provider.  

SpaceX Rocket Launches in a Model-Driven App

For this example we will use the SpaceX Rocket Launch API, described here. What we will do is to create a Custom Data Provider which connects to the API and retrieves information about Rocket Launches. In the upcoming blog posts we will register this Data Provider, create a Virtual Entity and we till use that entity in a Model-Driven App in order to display information about SpaceX Rocket Launches.

Picture from the SpaceX Rocket Launch API

Creating a Custom Data Provider

First of all, you need to know C#. What you need to do is to create a Retrieve plugin and a RetrieveMultiple plugin. That is a good start. You also need to think about what is unique for your data. In the Common Data Service guids are used. If you do not have guids in your external data source, then you will need to handle that somehow. In my example I have int values as unique values and I will transform the int values to guids and then the other way around.

Please note that this is just a start. You will need to implement some more to get a full functioning solution. E.g. you need to write more code in order to be able to filter and search the data properly. If I explore that further I will let you know later. One step at a time – for now let us take a look at the code and how it looks so far.

Launch.cs contains attributes for the information that comes from the SpaceX API about Rocket Launches. In fact, it is a class representation for the JSON that is returned from the API call. It also contains a method getLaunchAsEntity which creates a new instance of an Entity, takes the flight number returned from the API, transform that int value to a guid and then add the values from the API call to the instance of the Entity and return that instance (record). This class is used in the Retrieve and RetrieveMultiple plugins.

public class Rocket
{
    public string rocket_name { get; set; }
}
 
public class Links
{
    public string mission_patch { get; set; }
    public string mission_patch_small { get; set; }
    public string reddit_campaign { get; set; }
    public string reddit_launch { get; set; }
    public string reddit_recovery { get; set; }
    public string reddit_media { get; set; }
    public string presskit { get; set; }
    public string article_link { get; set; }
    public string wikipedia { get; set; }
    public string video_link { get; set; }
    public string youtube_id { get; set; }
    public List<object> flickr_images { get; set; }
}
 
public class Launch
{
    public Rocket rocket { get; set; }
    public int flight_number { get; set; }
    public string mission_name { get; set; }
    public string launch_year { get; set; }
    public DateTime launch_date_utc { get; set; }
    public Links links { get; set; }
 
    public Entity getLaunchAsEntity(ITracingService tracingService)
    {
        Entity entity = new Entity("cc_spacex_rocket_launch");
 
        // Transform int unique value to Guid
        var id = flight_number;
        var uniqueIdentifier = CDPHelper.IntToGuid(id);
        tracingService.Trace("Flight Number: {0} transformed into Guid: {1}", flight_number, uniqueIdentifier);
 
        // Map data to entity
        entity["cc_spacex_rocket_launchid"] = uniqueIdentifier;
        entity["cc_name"] = mission_name;
        entity["cc_flight_number"] = flight_number;
        entity["cc_rocket"] = rocket.rocket_name;
        entity["cc_launch_year"] = launch_year;
        entity["cc_launch_date"] = launch_date_utc;
        entity["cc_mission_patch"] = links.mission_patch;
        entity["cc_presskit"] = links.presskit;
        entity["cc_video_link"] = links.video_link;
        entity["cc_wikipedia"] = links.wikipedia;
 
        return entity;
    }
}

CDPHelper.cs contains methods for transforming an int to a guid and a guid to an int. It is used by the Retrieve and RetrieveMultiple plugins. I have already mentioned this, but I will do it again – the CDS requires guids and the API I am using uses ints.

static class CDPHelper
{
    public static Guid IntToGuid(int value)
    {
        byte[] bytes = new byte[16];
        BitConverter.GetBytes(value).CopyTo(bytes, 0);
        return new Guid(bytes);
    }
 
    public static int GuidToInt(Guid value)
    {
        byte[] b = value.ToByteArray();
        int bint = BitConverter.ToInt32(b, 0);
        return bint;
    }
}

RetrieveMultiplePlugin.cs inherits from IPlugin just as any plugin. In the Exceute method I create a new instance of an EntityCollection, to which I will add data from the API and pass that collection to the BusinessEntityCollection output parameter of the context.

When I make an HTTP request to the SpaceX Rocket Launch endpoint I use a filter in the URL in order to get only the data that I am are interested in. Then I deserialize the JSON into a list of Launches, remember I had a class Launch.cs representing the JSON object.

Next I use the getLaunchAsEntity method, which I have already described. The EntityCollection gets filled with entities containing data from the API.

The last important part is to put the EntityCollection object back to the context. Then we are done. This plugin will make sure we can get the data into a view in a Model-Driven App e.g. when we click on the menu item for SpaceX Rocket Launches in the Sitemap.

public class RetrieveMultiplePlugin : IPlugin
{
    public void Execute(IServiceProvider serviceProvider)
    {
        var context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
        var service = ((IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory))).CreateOrganizationService(new Guid?(context.UserId));
        var tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));
 
        EntityCollection ec = new EntityCollection();
 
        tracingService.Trace("Starting to retrieve SpaceX Launch data");
 
        try
        {
            // Get data about SpaceX Launches
            var webRequest = WebRequest.Create("https://api.spacexdata.com/v3/launches?filter=rocket/rocket_name,flight_number,mission_name,launch_year,launch_date_utc,links") as HttpWebRequest;
 
            if (webRequest == null)
            {
                return;
            }
 
            webRequest.ContentType = "application/json";
 
            using (var s = webRequest.GetResponse().GetResponseStream())
            {
                using (var sr = new StreamReader(s))
                {
                    var launchesAsJson = sr.ReadToEnd();
                    var launches = JsonConvert.DeserializeObject<List<Launch>>(launchesAsJson);
                    tracingService.Trace("Total number of Launches: {0}", launches.Count);
                    ec.Entities.AddRange(launches.Select(l => l.getLaunchAsEntity(tracingService)));
                }
            }
        }
        catch (Exception e)
        {
            tracingService.Trace("Exception with message: {0}", e.Message);
        }
 
        // Set output parameter
        context.OutputParameters["BusinessEntityCollection"] = ec;
    }
}

RetrievePlugin.cs inherits from IPlugin just as any plugin. It basically does the same thing as the RetrieveMultiple plugin but here we need to get the int value – the flight number – from the guid of the record. I take the PrimaryEntityId from the context and transform it to an int value and that int value is used in the HTTP request. One important part here is to put the entity back to the context as output parameter. This plugin is used when the user opens a record from a view.

public class RetrievePlugin : IPlugin
{
    public void Execute(IServiceProvider serviceProvider)
    {
        var context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
        var service = ((IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory))).CreateOrganizationService(new Guid?(context.UserId));
        var tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));
 
        Entity entity = null;
 
        tracingService.Trace("Starting to retrieve SpaceX Launch data");
 
        try
        {
            var guid = context.PrimaryEntityId;
            tracingService.Trace("Guid: {0}", guid);
 
            // Translate guid into the Flight Number ID
            var flightNumber = CDPHelper.GuidToInt(guid);
            tracingService.Trace("Flight Number: {0}", flightNumber);
 
            // Now we know which Launch to search for, let us go and do the search
            var webRequest = WebRequest.Create($"\nhttps://api.spacexdata.com/v3/launches/{flightNumber}?filter=rocket/rocket_name,flight_number,mission_name,launch_year,launch_date_utc,links") as HttpWebRequest;
 
            if (webRequest != null)
            {
                webRequest.ContentType = "application/json";
                using (var s = webRequest.GetResponse().GetResponseStream())
                {
                    using (var sr = new StreamReader(s))
                    {
                        var launchAsJson = sr.ReadToEnd();
                        tracingService.Trace("Response: {0}", launchAsJson);
 
                        var launch = JsonConvert.DeserializeObject<Launch>(launchAsJson);
                        if (launch != null)
                        {
                            entity = launch.getLaunchAsEntity(tracingService);
                        }
                    }
                }
            }
        }
        catch (Exception e)
        {
            tracingService.Trace("Exception with message: {0}", e.Message);
        }
 
        // Set output parameter
        context.OutputParameters["BusinessEntity"] = entity;
    }
}

This code is available on GitHub. Please feel free to get inspiration from it if you want to develop a Custom Data Provider of your own or to try out this example with the SpaceX Launches.

Conclusions

In order to create a class representation for your JSON, there is a super simple way to do that. In Visual Studio, go to Edit -> Paste Special -> Paste JSON as classes. Another way is to go to the website http://json2csharp.com/, paste your JSON and classes will be generated.

In order to parse the JSON you can use this:  

JsonConvert.DeserializeObject<Launch>(launchAsJson);

It is important to choose the right version of Newtonsoft.Json. It is not supported to use Ilmerge anymore, Scott Durows has written about it here. He has another way for the deserialization part.   

What I did was to use Newtonsoft.Json version 10.0.3.

custom data provider
Newtonsoft.JSON

It is included in CoreAssembly nowadays. Take a look here.

As I have mentioned, when working with Virtual Entities, guids are expected by the platform. There were no guids in my data source. I read here about an example for int to guid and guid to int and I chose to get inspiration from there.

To be continued

This was the second blog post in a series of four. In my next blog post I will let you know what to do with this code that we have created. How to register the plugins and how to register a Custom Data Provider. Until then I will leave you with this beautiful picture, taken from Nasas API APOD (Astronomy Picture of the Day).

custom data provider
The Hyades Star Cluster – the closest cluster of stars to the Sun.

About the Author:

Dynamics 365 and Power Platform Solution Architect from Sweden. 

Introduced to CRM 2006, Dynamics CRM/365 CE enthusiast since 2009. 

Interested in learning new technologies, problem solving and sharing my technical journey with Community fellows. 

Feel free to connect with me on Twitter @CarinaMClaesson or LinkedIn https://www.linkedin.com/in/carinamclaesson/.     

Reference:

Claesson, C. (2020). Virtual Entities – How to create a Custom Data Provider. Available at: https://carinaclaesson.com/2020/01/30/virtual-entities-how-to-create-a-custom-data-provider/ [Accessed: 24th March 2020].

Find more great Power Platform content here.

Share this on...

Rate this Post:

Share: