Hello, good morning, good afternoon and good evening everyone. My name is Gunjan Agarwal and I'm thrilled to be here to share insights on crafting compelling AR VR products and leveraging AI to accelerate development and improve user experience. A bit about me, having worked at Meta, Nutanix, and Amazon, I have been fortunate to lead some groundbreaking projects combining technical innovation with user centric design. Today, I'll share some of my learnings to help you as technical leaders navigate this exciting space. A quick disclaimer, this presentation is based entirely on my personal experience and independent research with no connection to my professional roles or responsibilities at Meta. With that, let's dive into the agenda to frame today's discussion. So we'll start by defining XR and why it's a game changer for industries. Next we'll examine how XR development compares to traditional platforms like mobile and web. Then we'll explore the challenges unique to XR and how to overcome them before concluding with actionable tips for building scalable products and integrating AI into the processes and practices. By the end of this talk, I hope you leave with both strategic insights and practical tools to apply in your road. And for those of you completely new to this space, I hope you will go back excited, knowing a bit more about the possibilities in this upcoming Skibase. Now, a couple of more points on what to expect and what not to expect from this session. Firstly, given the limited time we have today, I won't be able to do a deep dive into all the integrate details of XR development. However, my goal is to equip you with the foundational knowledge and point you in the right resources direction. Secondly, while much of my focus today will center on VR and X, AR development, particularly with headsets, it's important to note that many of the concepts and techniques I'll discuss, especially from a software development perspective, are really relevant to 2D game and application development. Now this makes this knowledge transferable and valuable across a wide range of platforms. With that, let's get going. So what is XR? Now XR or Extended Reality is basically an umbrella term for immersive technologies that blend the physical and the digital worlds to create interactive experiences. It encompasses three key categories, Augmented Reality, Virtual Reality and Mixed Reality. What is augmented reality now AR overlays digital elements like images text or example Or animations onto the real world enhancing what and interact with For example apps like ikea place lets you visualize the furniture in your home Or snapchat filters overlay fun effects on your face in real time VR, on the other hand, fully immerses you in computer generated environment, cutting you off from the physical world. Using devices like MetaQuest, PlayStation VR, we can explore virtual spaces, play games, or even train for surgery in completely simulated environments. Now, mixed reality is basically a mix of both of it. It merges the two. The physical and the digital world allowing them to coexist and interact seamlessly Unlike ar which only overlays digital content Mr enables users to manipulate and interact with virtual objects as though they are part of the real world Devices like microsoft's hololens are good. Good example and metacoast 3 are good example of mr technology Now why is xr development more relevant than ever You Now XR is revolutionizing gaming, healthcare, education, and retail. Think of PlayStation VR to defining, redefining gaming, or also VR, enhancing surgical training in healthcare. So across the industry, XR is basically making a huge impact and bringing about a lot of change. In terms of market opportunity, with a projected 250 billion market size by 2027, driven by device innovations and heavy investment from companies like Meta, Microsoft, Google, Sony, XR is quickly transitioning from futuristic to mainstream. Now, but developing for XR brings its own complexities, which we'll explore shortly. Many of us here are coming from software development background. would have experience with developing for mobile desktop. So I thought it would be useful to contrast the two. Unlike mobile and web, XR requires real time 3d environments, which demand for 3d modeling, complex rendering, and physics. Advanced spatial awareness and low latency are a must for seamless user experience and to avoid discomfort. Development involves leveraging game engines like Unity or Unreal, optimizing for 3D assets, and ensuring secure and responsive interactions across devices, and understanding unique privacy concerns in shared spaces. With that, let's dive into the capabilities and tools that power XR development. We'll first start by spending some time understanding the anatomy of an XMR headset. A mixed reality headset is an advanced device that blends virtual elements with immersive experiences. Key components include lenses and screens for displaying visuals, sensors and cameras for tracking, spatial audio systems and microphones for audio and voice, controllers with haptics and powerful CPUs and GPUs for processing. High refresh rates, typically 90 hertz or more, are essential to deliver smooth visuals and minimize latency, ensuring a convincing and comfortable user experience. Now, what gives a sense of immersion and realism in an MR headset? A key element of realism in 3D headsets lies in how they simulate depth, sound, and spatial awareness to mimic real world experiences. As far as realism is concerned, there are three aspects that we should talk about. First is Lenses and Depth Perception. The lenses in a 3D headset magnify the visuals displayed on the screen and present slightly different perspectives of the same image to the eye. Now this process known as stereoscopy creates a sense of depth replicating how our eyes perceive the real world. Combined with motion tracking, it enables users to see their environment shift naturally as they move their heads. Other aspect is spatial audio, which gives a sense of realism. Realism is further enhanced through spatial audio, which adapts sound based on distance, direction, and environmental factors like echos. Excuse me. For example, sounds grow louder as you approach a sound, as you approach a sound source, or shift position as you turn your head, providing an immersive and dynamic auditory experience. Field of view and tracking is another aspect which basically give a sense of immersion and realism in a MR headset. Now headsets further expand on the realism by offering a wide field of view and precise head tracking, allowing users to explore environments naturally. As user turn their head, they can see more of the virtual or the augmented space just as they would in the physical world. So all of these things come together to give a sense of immersion and realism when someone puts on a MR headset. Now with that, let's move on to understand some of the code software concepts for AR VR development. With Now one of the core pieces of software that we need to familiarize ourselves with is a game engine like unity or unreal engine A game engine is a very powerful software framework used in ar vr development to render 3d graphics Manage physics process interactions and more Now we'll talk about some of the key capabilities provided by the game engine. Let's start by talking about 3d graphics and rendering Game engines are optimized to render realistic, high quality 3D graphics essential for immersive AR VR experiences. They manage lighting, shading, textures, and other visual elements that help create a convincing virtual world. Physics simulation and 3D lighting is another capability provided by the game engines. The game engines have physics engine that simulate real world behaviors such as gravity, collision, and object interactions. This is crucial in VR to maintain a sense of realism as user expect virtual objects to respond to actions similarly to physical. objects. Game engines are also optimized for real time processing. Now ARVR requires real time performance to reduce latency, which is necessary to avoid motion sickness. Game engines are optimized to process actions and render changes in milliseconds. So this is another crucial aspect of game engine. Spatial audio integration, like we briefly discussed about earlier, is also a capability provided with the game engine. Sound is crucial, as we know, in AR VR for immersion, game engines typically support spatial audio, which allows sound to be positioned in 3D space relative to the user. This enhances the feeling of presence, as user can perceive sound coming from specific direction and distance, like we briefly spoke about earlier as well. Animation scripting is another capability provided by a game engine. Game engines support custom animation scripting, often using C sharp in Unity or C in Unreal, allowing developers to create interactive scenarios and dynamic object behaviors. Support for interactive. interaction input is another capability that the game engines provide These game engines are designed to handle user input from vr controllers hand tracking systems and even voice commands This enable more natural and intuitive way to interact with virtual objects making arvr applications more engaging Other aspect or capability provided by game engine is scene management. Now these game engines allow developers to organize complex scenes with multiple objects, layers, and environments. In AR VR, this is especially helpful for creating layered experiences where virtual objects can interact with the physical spaces and each other in highly complex ways. Developers tools and asset store is another aspect which is very important and provided by most of the game engines. These game engines come with a wide range of developer tools, debugging features, and asset stores with pre made objects, textures, and animations. These tools help streamline AR VR development, enabling faster iterations and prototyping. Now lastly cross platform support is another big capability generally provided by game engines Now these game engines often support multiple platforms like ios android oculus and hololens Etc allowing arvr application that can run on various devices Now we covered unity which is basically the core piece of software that one needs to be very familiar with if they are developing for arvr Or games But there is another aspect which we need to be very familiar with when developing for AR VR Now for anyone working on creating immersive experiences like virtual worlds a deep understanding of concepts such as shaders and materials Lighting and cameras vfx and animation is very important These are particularly useful for 3d artists involved in creating assets and the words which basically are used by engineers to put together the whole You know experience of the game together I'll briefly explain these Shaders and materials, basically, let's start by talking about shaders and materials. 3D shapes are generally represented using meshes. To draw something in Unity, you must provide information that describes the shape and the information that describes the appearance of its surface. You use meshes to describe shape and materials to describe appearance and surfaces. Shaders, pixel colors for realistic or stylized visuals. There's another thing we should talk about, which is a render pipeline, which basically process scene objects for display, right? They basically look at what is visible in the scene and cull what is out of the scene, include what is there in the scene to continuously update, render, update and render the display. There are different kinds of pipelines. render pipelines like URP, HDRP, or built in pipelines, which one needs to familiarize themselves with. Let's talk about now lighting and cameras. Now lighting models, real world light behavior with adjustable global illumination and lighting modes. There are things like baked lighting, real time lighting which one needs to understand. Now cameras basically flatten 3D scenes for 2D screens capturing and presenting the game world to players. Okay, so what is vfx and animation now vfx include particle systems decals large scale graphical effects to enhance immersion So for example a fire coming on the screen or a vfx Effect of emotes and all of those things are basically the vfx capabilities which basically are designed by artists Now animation provides features like retargeting, runtime control, state machines, and blend, blend shapes for lifelike motion and facial expression. So animation is something which is also very important for us to understand when developing for ARVR. Another crucial aspect that one needs to familiarize themselves with is post processing. Now, post processing is basically about visual enhancements via camera applied effects like filters and image corrections. Realistic 3D audio support, like directional sound, distance effects, and environmental acoustics like echoes are also part of post processing. Lastly, UI design and prototyping. This basically involves tools for creating user friendly interfaces and quick prototyping of assets and ideas. There are various tools which basically are used by the creative community to do all of this work, and then create, immersive 3D experiences. There are tools like Blender, Unity, ProBuilder, Maya, which basically are very popular among the creative community for creating these assets. And lastly, let's take a quick view of different, AR SDKs that can be used for AR development. we'll talk a few, talk about a few popular ones like ARKit, ARCore and Vuforia. For example, ARKit is provided by Apple, quite popular among, iOS developers. ARCore likely supported by Google. Very famous among Android developers. Euphoria is another very popular, AR SDK, and with a very good integration with Unity, is very, is another popular, AR SDK. Likewise, there are many others which, which can be leveraged to do AR. Now we covered about, tools like Game Engine, AR SDKs, and talk about different creative concepts. Now, despite these tools, building XR applications is quite challenging, which we'll discuss now. So what are the challenges unique to XR development? Now, creating a successful AR VR product is no small feat. Let's explore some of these unique challenges developers face and techniques to overcome them. First, let's start by talking about hardware limitations. Now, there are several hardware limitations, such as limited processing power, battery life, form factor and comfort, display quality, etc., that one needs to pay attention to when developing for XR. Let's start talking about each of these, basically hardware related challenges. So let's start by talking about processing power. Now AR VR experiences are resource intensive, demanding high computation power to render realistic graphic at high frame rates. Developers must optimize their code, manage memory usage efficiently to ensure a great, great, result from the processing power perspective. Other hardware limitation is battery or challenges battery life. Achieving a balance between performance and battery life consumption is critical, especially for standalone devices. Techniques like 4 bit rendering where only area that the user is directly looking at is rendered in high detail helps save processing power and battery life. Likewise, there are other techniques which are very important to, which are very important for taking consideration of battery life. Form factor and comfort is another aspect from the hardware perspective, which XR developers need to pay attention to. The physical design of headset impacts user adoption. Reducing weight, improving ergonomics, and balancing the headset are priorities to ensure user can wear devices for extended periods of time comfortably. Lastly, from the hardware perspective, let's talk about display quality. Now, fast refresh rates and high resolutions are necessary to deliver a smooth visual like we have spoken about earlier, but they also introduce challenges like heat generation and increased power consumption. Display technologies like OLED, micro LED are being explored to provide better visuals without these trade offs. With that, let's move on to talk about latency and motion sickness as another challenge unique to XR development. Now latency is a significant challenge in VR. If there is even a small delay between the user's physical environment and visuals being displayed, it can lead to cyber sickness ensuring. Now ensuring low latency requires efficient tracking algorithms. Now real time tracking and rendering algorithms are very important to keep the experience. This includes head tracking, eye tracking, and position tracking. Basically, all the tracking that we are doing is basically has to be in real time to avoid any sort of latency. Otherwise, users generally get motion sickness. Another technique which is very useful to avoid latency and reduce motion sickness is using predictive algorithms. Now, predictive algorithms can help compensate for latency by predicting the user's next move and adjusting the visuals accordingly, ensuring that the user remains immersed without any lag. Then let's talk about tracking through multiple sensors, which is challenging as far as XR development is concerned. Now, reliable tracking of head, hand, and body movements is essential. AR, VR devices rely on sensors like cameras, gyroscopes, and accelerometers. Developing robust sensor fusion algorithms that integrates data from multiple sensors is one of the techniques that allow for accurate and responsive tracking, which is crucial for immersion. Now content creation in 3D modeling is another challenge very unique to XR development. High quality 3D content is essential for creating immersive experiences. However, creating a detailed model that looks realistic while running smoothly across different hardware configurations is a big challenge. Developers use several optimization techniques like level of details and texture compression. Level of detail reduce the complexity of 3D model when they are far away and ensuring performance without sacrificing visual quality. Text compression reduce texture size without, texture compression reduce texture size without significant loss in detail to optimize load time and memory usage. Now let's talk about debugging and testing challenges. Testing AR VR requires evaluating performance across various environments, device types, and user scenarios. Developers use a combination of simulated environments and real world testing to address potential issues. Automated testing tools for stress testing, the rendering pipeline, and sensor inputs can also help identify certain bottlenecks. The user safety and health is another challenge which is very unique to XR development Ensuring the safety and comfort of user is critical extended use of VR can lead to eye strain and physical discomfort as well as Hazards, so developers need to design experiences that minimize this effect and encourage regular breaks Integrating safety features like a boundary system like guardian can prevent users from bumping into real world objects Now, we spoke about these challenges and some of these techniques to overcome these challenges, but nonetheless, overcoming these challenges require a careful optimization around strategies, which we will talk about next. So what does it take to build a lovable and scalable XR product? Now buildings a lovable and scalable excel product that scales across devices and user exp, and users and is loved by customers require paying attention to several key aspects. And we'll talk about three key areas, which is focus on user design and experience optimizing performance and focus of inter focus on interoperability and ecosystem integration. Let's start by talking about focus on user experience design. Now AR and VR is all about spatial interactions. The way users move and interact with virtual objects must feel natural. Some of the areas to pay attention to are spatial design. Developers need to leverage principles of spatial design to create intuitive and engaging experiences. Environment variability is another aspect which is related to user experience. Now AR experiences must work well in diverse settings from bright outdoor environments to dim indoor ones. Developers can use techniques like adaptive lighting and robust tracking to ensure consistency. Adaptive UI UX for varied user scenarios. is another thing which one needs to pay attention to. Now, users may switch between indoor and outdoor setting, or from sitting to standing, designing adaptive UI UX ensures that experience remain intuitive and responsive across varied scenarios. Now, optimizing for performance, is another critical aspect of building a lovable and great, XR experience. As you've already seen, performance is a critical aspect for XR application. We can optimize performance in several ways. First is we can adopt a low latency architecture like edge computing which can really help minimize delays. We can also optimize 3d content as you have seen 3d content is very computation heavy and have involved involves a performance penalty. So we can use best practices in optimizing the 3d entry content 3d content which can Ensure high performance without sacrificing visual quality. This involves efficient use of textures shader and lighting We should also pay attention to minimizing bandwidth usage by finding ways To reduce the bandwidth usage without finding qualities, techniques like adaptive streaming and compression can be very beneficial in this regard. Lastly, let's talk about interoperability and ecosystem integration. We briefly touched upon the concept of interoperability earlier, but creating a robust ecosystem that supports multiple devices and platforms ensures that the users have seamless experience, whether they are on mobile, VR, or desktop. Interoperability can also encourage third party developers to build on your platform. Encouraging application, you know spread from this aspect we can focus on following like You know following like we can start with we can think about x device and platform compatibility So for this we should build using standard methods that allow your apps to work across devices and headsets and platforms We can also adopt standard tools and skill sets Now allowing use of standards and popular tools such as unity for development encourages third party developers to develop apps And integrations enriching user experience and enhancing further reach of your application So we've spoken about, so far about the challenges unique to XR development, what the, what are the key software concepts of XR development and how can we overcome some of the challenges we've spoken about and different tools used, right? I want to end this, session by spending some time talking about how can we leverage AI to basically create compelling XR experiences, right? As we've spoken about, creating compelling XR products demands significant effort. But AI can accelerate development, enhance user experience, and empower creator. So let's start, to look at each of these aspects in a little bit more detail. Let's start by talking about improving AR VR development through XI. Now there are multiple ways in which AI can, accelerate or improve AR VR development experience for the developer themselves. First is by automated asset creation. Now 3D assets, textures and environments saving time for developers. Tools like DALI enable creators to produce high quality visuals from simple inputs. realistic physics and animations is another area where AI can help significantly. AI powered physics engines can simulate real world behaviors more accurately, making AR VR applications feel natural. Another aspect is procedural generation. Now AI enables the automated generation of complex environment, which is useful for open world VR games and simulation. Another final aspect is enhanced testing and debugging. Now AI can assist in QA processes by automatically testing interactions and identifying glitches or bugs in AR VR environment, which are difficult to catch. Algorithms, these algorithms can simulate user interaction, finding edge cages that humans might otherwise miss. another area where AI can, significantly, help improve, the user experience is, another area is basically where AI can significantly help is elevating the user experience. so there are multiple, aspects here as well. for example, personal personalized interaction. Now a LL algorithms can personalize content based on user's preferences and behaviors, creating more engaging and tailored experiences. For instance, VR experiences can adapt in real time to a user's emotional response. Tracked through AI based, analysis of biometrics. AI can also help with. Natural language processing. Now NLP powered AI can enable natural, conversational interactions with AR VR experiences. This is useful for applications like VR education or training simulations where users can interact with virtual characters like, like in a human like way. Now enhanced spatial audio is another aspect where AI can help improve the user experiences. AI can help enhance spatial audio, making sound appear to come from specific direction, distance, or even specific objects. Deep learning models can also personalize audio experience based on user preferences, immersive experience. Immersive experience. Finally, emotion detection and adaptation is another area in which AI can significantly help improve the user experience. For example, AI can monitor facial expressions and body language to direct the user's emotional state and adapt the experience accordingly. This is especially useful in VR training, therapy and gaming. where real world adaptation can improve engagement and outcomes. Lastly, let's talk about how can AI empower creators. So there are multiple ways, for example, content creation assistance. Now, AI driven tools help creators draft narratives, creative scripts, or storylines that adapt to user choices and behaviors in real time. For example, a generative model can assist creators Game writers by suggesting branching story paths. AI can also help by, word authoring, which is basically, a significant effort otherwise. lastly, AI can significantly democratize access to tools, encouraging more people to create, for AR. now AI powered low code or no code platforms like, which basically empower creators with minimal programming skills to develop compelling AR VR experiences, allowing a wide range of creators to enter the field, which otherwise would be limited because of the steep learning curve of tools and technology in this particular area. alright, with that we have come to the end of the presentation. Hope this was useful as we explore the transformative potential of xr, the unique challenges and for of developing for this platform and the role of creating, role of AI in creating scalable LAN applications. With that, I want to thank you, all for your time today. Please feel free to connect with me on LinkedIn if you're working on or looking to explore this space. Thank you so much for your time. Bye.