Visualizing Azure Digital Twins in 3D
By Jonathan Girroir • This article originally appeared on the Microsoft Tech Community
3D visualization of a real-world environment with all the things, people, business processes in it is no longer found only in science fiction movies! Digital Twins is a term describing digital representations of environments, things, people and their relationship, and Azure Digital Twins (ADT) is a platform that allows creating and interacting with such digital representations. Not only does it allow to create models but also offers a graph API to query and interact with its twins. ADT Explorer is a tool allowing users to visualize ADT models and Twins and explore the relationship between different things, people, and processes. ADT Explorer is useful for visualizing this graph and exploring the relationship between different objects.
But these 2D graphs don’t resemble the actual objects’ size, shape, color, or location. A flexible 3D representation is extremely helpful to navigate, understand, monitor, and react to changes in this information. Operators who have to interact with Digital Twins need to do so in a way that is simple and intuitive for them.
The HOOPS Web Platform is a set of software development kits that can complement ADT by providing a web-based 3D viewer that can import CAD models and connect to data and events stored in ADT. Using HOOPS and ADT we can create a 3D Digital Twin of a factory with an intuitive interface that allows operators to easily see the state of each machine.
Let's walk through how you can bring a 3D visualization to life by connecting it to Azure Digital Twins. We’ll start with what you need to follow along and how to import 3D models into HOOPS. Then we’ll connect the 3D models to ADT nodes, display markup, react to events, and learn how to edit the 3D world we’ve created.
Prerequisites
The repository at https://github.com/techsoft3d/communicator-adt-tutorial shows a simple example of combining these two frameworks. We’ll use this project to highlight essential ideas and integration points.
DTDL is the language used in Azure Digital Twins to describe models and twins. Learn more here. We use a DTDL graph based on this ADT Learning Module. You need to connect to a hosted ADT graph similar to this one.
To use custom 3D data from CAD models, you will need the HOOPS Web Platform.
Importing Data
We must first create or source 3D data. Many times, 3D models of the devices, machinery, infrastructure, and buildings already exist. Other times, they need to be created using a CAD package like SOLIDWORKS or Inventor for mechanical design or AutoCAD, Revit, or SketchUp for building design. A 3D model close to the original may suffice. Numerous 3D model libraries like GrabCAD,TurboSquid, and SketchFab provide ready-made 3D models that can be used or adapted.
Each 3D model can be converted into a stream cache file using the HOOPS Converter command-line application.
> converter --input factory-floor.skp --license_file license.txt --output_scs factory-floor.scsThese optimized files can be combined and loaded into the HOOPS Web Viewer for visualization.
Note: There are two main variants of stream cache data:
Stream Cache Standalone (SCS) - These singular files can be easily loaded with REST API functions, just as you would some other file. The entire file must be downloaded before visualizing. In this article, we use this approach to manage 3D data.
Stream Cache Compressed (SCZ) – These compressed files are streamed using an instance of the HOOPS SC Server app, which establishes a WebSocket connection between server and client. 3D model data is progressively streamed through this connection and can be instantly interacted with. This format helps visualize very large models.
A complete file list of supported file formats can be foundhere. In most cases, formats not supported by HOOPS Converter can be saved to a neutral format, like STEP, STL, or FBX, and then converted to a stream cache file.
Some nodes in the ADT graph may have associated 3D data, including the physical building or site, machines, sensors, and additional objects, both IoT enabled or not. For ADT nodes that have 3D data, you can save the associated model name as a property. Although the model file names are saved within ADT, the model files themselves should be saved elsewhere, possibly in a /scs_models directory on the server that hosts your backend.
The DTDL we use:
{
"@id": "dtmi:com:microsoft:iot:e2e:digital_factory:model_metadata;1",
"@type": "Interface",
"displayName": "Model Metadata - Interface",
"@context": "dtmi:dtdl:context;2",
"contents": [
{
"@type": "Property",
"name": "SCSFile",
"schema": "string"
},
{
"@type": "Property",
"name": "Transformation",
"writable": true,
"schema": {
"@type": "Map",
"mapKey": {
"name": "index",
"schema": "string"
},
"mapValue": {
"name": "value",
"schema": "double"
}
}
}
]
}In addition to the SCSFile path, we also save a transformation matrix that helps us place the object in 3D space. The transformation matrix is a 4x4 matrix that encodes its location, rotation, and scale in the 3D scene. Because DTDL does not support array as a property, this matrix is stored as a map. Index “1” through “16” represents each of the 16 elements in the matrix. It's recommended that the SCSFile name and transformation be saved as part of the ADT graph but they can also be hardcoded in your specific application, kept in a JSON file or saved in an external database. This sample assumes that it is present and will use these properties to load the associated 3D data automatically.
Visualizing Data
The HOOPS Web Viewer (HWV) is a browser-based application for viewing and interacting with 3D data. It is configurable through an extensive JavaScript API. hoops_webviewer_sample.html shows how to configure the WebViewer and connects a toolbar and other commonly used tools to the viewer. This sample serves as the basis for our sample project.
Instantiate the WebViewer using the client code served from hoops_web_viewer.js. We begin by loading an empty viewer attached to a named HTML <div>, which the WebViewer will control and render into.
window.onload = function () {
hwv = viewer = new Communicator.WebViewer({
empty: true,
containerId: "viewerContainer",
streamingMode: 1
});
viewer.start();
}Next we query ADT for all nodes that have "SCSFile" and "Transformation" defined and load these into the WebViewer using loadSubtreeFromScsFile().
query_twins("SELECT * FROM digitaltwins T WHERE IS_DEFINED(T.SCSFile) AND IS_DEFINED(Transformation)").then(result => {
let resultData = JSON.parse(result);
resultData.forEach(twinData => {
twins[twinData["$dtId"]] = {};
let scsFile = './scs_models/' + twinData["SCSFile"] + ".scs";
let array = [];
for (let i = 1; i <= 16; i += 1) {
array.push(twinData["Transformation"][i]);
}
let transformationMatrix = Communicator.Matrix.createFromArray(array);
hwv.model.loadSubtreeFromScsFile(-2, scsFile, transformationMatrix);
});
});Connecting ADT Twins to 3D Objects
Each model loaded into the WebViewer has a unique node Id. We can capture this when loading and store in a map indexed by the ADT Id.
hwv.model.loadSubtreeFromScsFile(-2, scsFile, transformationMatrix).then((nodeIds) => {
let twin = twins[twinData["$dtId"]];
twin.nodeId = nodeIds[0];
});This creates a binding between objects in the WebViewer and ADT. When something changes in ADT, we use the ADT Id to lookup the WebViewer node Id, and then use the node Id to change the 3D representation in the WebViewer. We use this binding to display metadata and react to events.
Alternatively, interaction with objects in the 3D viewer can impact the ADT graph by using this array to look up the associated ADT Id from a WebViewer Node Id. For example, this is done when repositioning objects or changing the 3D model associated with an ADT node.
Markup
Displaying static or real-time information next to 3D objects is commonly needed. Each entity in ADT can have metadata associated with it. The WebViewer can show this as text in several ways. One way is through a piece of markup.
In our sample, when we load a 3D model into the WebViewer, we also create a piece of markup to display its properties. To place it properly, we take into account the bounding box of the entire 3d scene.
function createMarkup(twinData) {
let twin = twins[twinData["$dtId"]];
hwv.model.getNodesBounding([twin.nodeId]).then((box) => {
var markup = new CustomMarkupItem(hwv, -5, "",
new Communicator.Point3((box.max.x - box.min.x) / 2 + box.min.x, 2500, 0),
new Communicator.Point3(0, 1, 0),
0);
var uid = hwv.markupManager.registerMarkup(markup);
hwv.markupManager.refreshMarkup();
twin.markup = markup;
updateMarkup(twinData);
})
}A reference to the JavaScript markup object is saved alongside the WebViewer node Id in our twinMap. To update the metadata for each object we query the ADT graph:
query_twins("SELECT * FROM digitaltwins T WHERE IS_DEFINED(T.SCSFile) AND IS_DEFINED(Transformation)").then(result => {
let adtData = JSON.parse(result);
adtData.forEach(twin => {
updateMarkup(twin);
});
});Then within updateMarkup(), we use the ADT twin Id to find the associated JavaScript markup object and update its text with the correct properties parsed from ADT. Finally, calling markupManager.refreshMarkup() automatically updates the text in the 3d view.
Reacting to events
Another typical workflow is to react to events, especially error states for individual nodes or propagated errors within the ADT graph. We treat event states as just another piece of metadata. We don’t display it as text but instead highlight the entire object in an error state.
Our ADT instance sets an alert when the grinding vibration goes above 300. The “Trigger” button in the UI automatically sets vibration above 300 in the ADT graph for testing.
During our polling function we check to see if an alert has been set on a node and then highlight:
hwv.selectionManager.selectNode(object.nodeId);
Repositioning Objects
Each object’s location, rotation, and size are stored as a 16-element float array representing a 4x4 transformation matrix (https://en.wikipedia.org/wiki/Transformation_matrix). This helps place each object in 3D space. Leaving a transformation matrix empty will place the objects at the center of the 3D scene (x=0, y=0, z=0).
We’ve set appropriate transformations for each node in our graph. Usegeometry handlesif a new object is added or you want to reposition an existing object. These can be enabled within the demo by right-clicking on an object and selecting “show handles” from the context menu. Once done repositioning, the new matrix will be output to the console and can be used to update the ADT graph.
Future work
Several initiatives that would make an integration between ADT and HOOPS more developer-friendly:
Authoring Tool– We assume that 3D model data (name, location, and orientation) already exist within the ADT graph. Sample code and accompanying UI that allows authoring and editing of this data both within the 3D scene and then saved to the ADT graph would be beneficial.
IFC to ADTL– IFC is a standard open-source building and construction file format that naturally lends itself to hierarchical object organization. IFC files are commonly organized by building domain (architecture, mechanical, electrical, plumbing), floor, space (room), and object. HOOPS Exchange provides API access to the IFC data and can prepopulate an ADTL representation of a building.
SignalR– The reference implementation uses a restful API to query the ADT graph on a five-second timer. The preferred way to get real-time data is throughSingalR, a wrapper for WebSocket data transfer.
Object Instancing– An instance of each 3D object referenced in ADT is loaded into the 3D scene. This is inefficient if the same object is loaded multiple times in different locations. Instead, a single instance of the 3D object can be streamed, loaded to memory, and drawn multiple times. Instead of using loadSubtreeFromScsFile(), create ashattered SCZ file.
Make Your Own 3D Digital Twin
There are endless possibilities when combining HOOPS with ADT. What will you create? Start with a copy of this project at https://github.com/techsoft3d/communicator-adt-tutorialor the numerous tutorials within theHOOPS Web Platform.
Please keep the discussion going. Leave your questions and comments below.