Virtual Reality for Surgery: Key Aspects

VR for surgery - ScienceSoft

ScienceSoft has 16 years of experience in software development for the healthcare industry, and 25 years in 3D modeling combined with experience in VR development.

Market Info: Virtual Reality in Healthcare

The worldwide market of virtual reality in healthcare has reached $336.9 million in 2020. It is forecasted to grow at a CAGR of 30.7% from 2020 till 2027. The surgery field is one of the fastest VR adopters due to the need for more efficient training of students and residents.

How Virtual Reality for Surgery Works

VR system architecture

The client VR application extracts the data from the database web server and outputs it on a user’s visualizer (a head-mounted display (HMD) or a desktop). The web admin panel is used for database management and settings access.

VR system architecture

Use cases

Surgical training

Surgery simulation for immersive training and surgeons’ skills assessment. For example, if a trainee needs to remove a tumor, the script of the simulation describes where it is located, how it looks, medical device readings and how they change after each step of the operation protocol is completed.

Pre-surgery planning

Patient data (e.g., MRI, CT) is converted into 3D models. A surgeon can navigate the models (scale, rotate, mark, etc.) in VR to plan a surgical intervention and prepare a patient.

Medical marketing

Medical device companies can use VR software to let the surgeons try new products in the virtual environment.


Features inherent to all use cases:

Multi-user mode

Users can collaborate as if they were in one operating room, for example, hand over surgical instruments.

Simulation recording

The surgery simulation 3D recording can be replayed to show the patients the plan of an upcoming surgery or to analyze a trainee’s performance.

Remote telepresence

The feature can be used by teachers for guided training, by surgeons for remote case discussion, or by medical sales reps to guide surgeons on the usage of new medical devices in VR demo.

Assets libraries

3D models of virtual patients (each with the unique patient data, response of the patient’s organs), surgical instruments, an operating room layout, etc.

Features specific to surgical training:

Scripts editor

The difficulty of certain tasks in the surgery simulation can be adjusted based on the skill level of a trainee.

Performance tracking and assessment

VR system analyses the actions of a trainee in comparison to performance metrics, gives feedback on their skills, and tracks their progress over time.

Want to Create a Virtual Reality Application for Surgery?

ScienceSoft’s team is ready to take up your idea all the way from shaping a user-centered product vision to the development of a reliable VR application for surgery.


VR engine

Unreal Engine
Amazon Sumerian

Unreal Engine (C++)

Unity (C#)

CRYENGINE (C++ and C#)

Amazon Sumerian (HTML, CSS, JS)

3D modeling

Autodesk Maya
Autodesk 3ds Max
Substance Painter

Simulation modeling

Nvidia FleX
Bullet Physics


Valve Index

Challenge #1

Virtual reality for surgery simulation should render the interaction of surgical instruments with tissue structures with high precision. A user should receive a real-life response to their manipulations.

Check out the solution


With a multi-point collision detection method, a group of sample points is placed evenly across the surface of the surgical instruments. The 3D model of an operated organ is built with voxels. When sample points of the instruments come in contact with voxels, the system computes the applied force in each sample point and deletes the affected voxels, thus accurately deforming the tissue.


Challenge #2

Virtual reality creates the environment to practice skills but can’t assess them in detail.

Check out the solution


Integration of AI in a VR system to measure trainees’ performance. The data for performance benchmarks (e.g., a surgical instrument’s position, the force applied to an instrument, timeliness, amount of the tissue removed) is derived from the users with different experience (surgeons, residents, med students) going through the same surgical procedure in a VR simulator. Then a machine learning algorithm is fed with this data to give feedback on performance to future trainees.

Note: for tasks, where human verification is still needed (e.g., medical exams), multi-player mode can be used.


General cost drivers

  • Method of creating 3D models (3D CAD rendering, photo- or video shoot)
  • Number of 3D models (in case of 3D CAD rendering).
  • Number of user roles.
  • Number and complexity of surgery simulation scripts.

Additional costs of the entire solution

  • Development of an AI-based performance assessment module.
  • Development of software for MRI and CT scans conversion to 3D models.
  • VR hardware: PC (with high-end GPU and CPU), HMDs and controllers.

Operational costs

  • Infrastructure services (e.g., cloud services).

VR Consulting and Development by ScienceSoft

ScienceSoft combines experience in VR and healthcare to help create innovative apps tailored to the surgery specifics.

VR consulting

Our consultants help:

  • Decide on the system’s functionality.
  • Choose the right tech stack.
  • Define product development and management roadmap.
  • Estimate TCO of the VR system.
  • Estimate ROI.

VR development

The service covers all the stages of VR system creation:

  • Business analysis.
  • Architecture design.
  • 3D modeling and software development.
  • Testing and QA.

About ScienceSoft

ScienceSoft is a technology consulting and implementation provider with 32 years of practice in IT. Being ISO 13485-certified, we design and develop medical software according to the requirements of the FDA and the Council of the European Union and help product companies across the globe implement the cutting-edge technology of virtual reality.