Surgical Console

The problems

• Project 1 · Arm identity

  Which arm is it?

  • The arm was only a number.
  • A nurse had to match it to a real arm in the room.

  Success: the right person knows which arm, and what to do.

• Project 2 · Gallery and video editor

  Where is that moment?

  • The old editor worked on still images only.
  • A resident had to scrub through an hour of footage.

  Success: find one moment without scrubbing.

• Project 3 · Live annotation

  What is the surgeon pointing at?

  • Scrub nurses at the wall monitor could not see it.
  • The surgeon’s hands stayed on the controls.

  Success: a mark reaches the main screen within a second.

The old screens showed real patient video, so I redrew them as wireframes. Project 3 had no old screen: the old design had no way to mark the live video.

Three projects

• Arm identity

  A color bar shows which arm has an error or warning.

• Gallery and video editor

  A gallery named by procedure step, and a video editor.

• Live annotation

  Marks made on a tablet appear on the main screen.

Constraints

• Tight timelines

  In development for 8 years, so requirements had to move quickly.

• Many stakeholders to align

  Surgeons, engineers, PMs and executive leadership.

• The operating room

  Screen distance, lighting, gloves and noise.

The product

Users and environment

Environment constraints

• Bright endoscope feed

  Warm interface colors looked like anatomy.

• Brief, fragmented attention

  Surgeons looked for only a second at a time.

• Safety alerts first

  Machine safety warnings beat every other message.

Annotation became a mode

Annotation took 6 versions (pills, switches, toggles). I ended with a mode and a check button to exit.

1. V1

   Watch and Annotate segments

   Watching is not a mode.

2. V2

   A switch in the toolbar

   It looked like one more tool.

3. V3

   A floating glass pill

   Too prominent over the video.

4. V4

   Tap the pill again to leave

   Leaving needs its own button.

5. V5

   Done with a cross icon

   A cross means cancel; Done means keep.

6. V6

   A named mode, ended by a round tick

   The current version.

Strategy

Engineering had limited time, so I did not plan one huge redesign. I chose three steps:

1. Crawl

   Fit into the old layouts

   I added basic capture, pin, and measure tools to the old editor.

2. Walk

   Rebuild with components

   I made annotation its own mode and used the new console grid.

3. Run

   Use the same rules everywhere

   I used one alert scale and one component library in the editor, gallery, and service screens.

Vision & principles

The team’s main goal was “designing for confidence.” Six principles came from it.

• Informed control

  You can tell what the system did and will do.

• Trust and reliability

  The same tap gives the same result.

• Situational awareness

  The anatomy matters most.

• Extensibility

  Components first, so one color scale works everywhere.

• Clarity

  Every mode is named and has a way out.

• Familiarity

  Controls people already know.

Lo-fi sketches

Component system

I wrote the component rules first, so every screen looks and acts the same.

• Rule 1 · Severity

  One alert scale

  Green running, amber warning, red error, grey inactive. Cyan only for selection.

• Rule 2 · Identity

  Arms keep their own color

  Shown only as a thin bar on top of the panel.

• Tools versus actions

  An active tool is a solid block; actions stay outlines. Only Delete is red.

• Saying it three ways

  Color, word and icon together, for color-blind staff.

Project 1 · Arm identity

I kept each arm’s own color, shown only as a bar along the top of the panel.

The state shows as a dot and a word: Running, Warning, Error or Inactive.

Instead of

Filling the panel with the arm color. It hid which arm it was and what it was doing.

Arm sandbox

Project 2 · Gallery and video editor

• The editor handled video

  A timeline marks recordings and pins.

  Instead of: still images only.

• One labeled tool bar

  Every tool in one bottom bar, each with a word.

  Instead of: a side panel of icon tabs.

• One gallery filter

  All, Images and Videos share one track.

  Instead of: separate filter chips.

• Recording changed the screen

  A red Stop, and the markup tools dim.

  Instead of: a state that looked like any other.

Inside the video editor

I placed each region by how often people look at it and reach for it: the live video, marks and action ribbon all the time; status and timeline sometimes; menus rarely.

Five key decisions and what each one replaced

• Annotation became a mode

  A tool rail, a cyan border and a banner.

  Instead of: five tool buttons always on.

• Captures in a corner player

  Saved clips open in the bottom right.

  Instead of: a drawer that pushed the video off center.

• The wall monitor shows part of the screen

  Only the video and the surgeon’s marks.

  Instead of: a cluttered copy of the whole screen.

• Pins got numbers

  So a pin can be called out by voice.

  Instead of: unnumbered pins and a side list.

• Voice commands got a visible log

  Time, what was said and the reply.

  Instead of: only a chime.

Project 3 · Live annotation

Still to do: an Edit button. I ran out of time, so a dashed placeholder marks where it goes.

From wireframe to prototype

I built a prototype in Figma to check annotation mode, tap target size and how views switch.

I can’t show the prototype because of privacy rules. I am happy to walk through it on a call.

Outcome

• >50%less task-completion time in exploratory usability testing
• 6 of 10usability rules fixed in my own review of the prototype

Exploratory usability testing, moderated and unmoderated, with A/B tests and timed sessions, including unmoderated tests with senior software engineers, cut task-completion time by more than 50%, with fewer interaction steps and fewer observed errors.

These were exploratory tests of a prototype, so they are early signals, not product results.

Heuristic evaluation

6 of 10 fixed, 4 still open.

• H1 · FixedVisibility of System Status

  I added 3 cues (a cyan border on the video, a top banner, and a tool rail) so surgeons knew marks were live.

• H2 · Still openMatch Between the System and the Real World

  I replaced engineering codes with clinical words. Some system errors still needed a review.

• H3 · FixedUser Control and Freedom

  I made a separate mode with a check button to exit, instead of a confusing toggle.

• H4 · FixedConsistency and Standards

  I used the same 4-color alert scale (green, amber, red, grey) on every console view.

• H5 · FixedError Prevention

  I stopped taps on a tool from leaving a mark on the live surgery video by accident.

• H6 · FixedRecognition Rather than Recall

  I added a word and an icon to every button, so staff did not have to remember.

• H7 · Still openFlexibility and Efficiency of Use

  Keyboard shortcuts worked for expert users. Custom tool presets stayed on the to-do list.

• H8 · FixedAesthetic and Minimalist Design

  I folded the screen into 3 regions that collapsed. This kept the video clean.

• H9 · Still openHelp Users Recognize, Diagnose, and Recover from Errors

  I grouped errors by robotic arm. Repeated error messages still needed to merge into one.

• H10 · Still openHelp and Documentation

  Help sat in the settings. I left out a full guided tour, because it could distract people in the operating room.

All three together

1. Before

   Arm identity

   A nurse reads the arm bars and knows which arm needs help.

2. During

   Video editor

   The surgeon marks, measures and records on the live feed.

3. During

   Live annotation

   Marks from another screen show on the main screen.

4. After

   Gallery

   Captures wait there, named by procedure step.

I built all three from the same components, the same severity scale and the same arm colors.

What I handed over

• A working prototype

  Working flows for annotation, captures, the corner player, case setup and the gallery.

• A component library

  One alert scale, the rules for using the existing arm colors, and the rules for how buttons act.

• A test plan

  A full test plan with 10 tasks and 9 ideas to test.

Still unfinished

• You could not undo one mark, only clear all.
• Repeated error alerts did not merge into one row.
• A medical expert still had to review the clinical words.
• Live annotation still needed an Edit button.

Takeaways

1. 01

   Write the reason before drawing

   Name the old pattern each choice replaces.

2. 02

   Components before full screens

   Early alert rules kept four parts consistent.

3. 03

   Prototype early

   A running prototype caught a tap-through bug.

4. 04

   Design system states, not controls

   Decide how modes switch before polishing buttons.

5. 05

   Familiar beats new when mistakes cost a lot

   The corner player worked because people knew it.

6. 06

   Test earlier, with rougher designs

   Ten minutes with a surgeon beats six versions alone.

To follow privacy and NDA rules, I changed the screens and hid all patient and staff names.