Shadow DOM Automation with Selenium 4

What is DOM ?

The Document Object Model (DOM) is a programming interface for web documents. It represents the page so that programs can change the document structure, style, and content. The DOM represents the document as nodes and objects; that way, programming languages can interact with the page.

A web page is a document that can be either displayed in the browser window or as the HTML source. In both cases, it is the same document but the Document Object Model (DOM) representation allows it to be manipulated.

HTML Structure

What is Shadow DOM ?

• An important aspect of custom elements is encapsulation, because a custom element, by definition, is a piece of reusable functionality: it might be dropped into any web page and be expected to work. So it’s important that code running in the page should not be able to accidentally break a custom element by modifying its internal implementation.
• Shadow DOM works by allowing you to attach a hidden, separate Document Object Model (DOM) to an element. This hidden DOM is known as the ‘Shadow DOM’, and the element it’s attached to is referred to as the ‘Shadow Host’.
• It’s a straightforward way to create components with isolated CSS and JavaScript.
• Shadow DOM allows hidden DOM trees to be attached to elements in the regular DOM tree — this shadow DOM tree starts with a shadow root, underneath which you can attach any element, in the same way as the normal DOM.

Shadow Root Tree

Web Components & Shadow DOM

• Web Components are a collection of technologies that allow developers to create reusable custom HTML elements with encapsulated functionality.
• Shadow DOM is a specific technology within Web Components that provides a way to isolate the internal structure (HTML, CSS, and JavaScript) of a custom element. This isolation prevents conflicts with the rest of the web page.

Tech Stacks Embracing Web Components (and Likely Shadow DOM):

• Single-Page Application (SPA) Frameworks: Powerhouses like React, Angular, and Vue.js heavily rely on Web Components for building reusable UI components. While they don’t directly expose Shadow DOM for manipulation, they likely use it internally to keep components self-contained.
• Web Component Libraries: Frameworks like Polymer and Lit offer pre-built, reusable Web Components that developers can drop into their applications. These components almost certainly leverage Shadow DOM to ensure their styles and functionality don’t clash with the surrounding page.
• UI Component Frameworks: Material Design and Bootstrap provide pre-built UI components that streamline development. While not every component might use Shadow DOM, some could benefit from it for style encapsulation.

Real-World Examples with Web Components (and Likely Shadow DOM):

• Salesforce Lightning Web Components (LWC): LWC leverages Shadow DOM extensively for its component architecture, promoting modularity and maintainability.
• YouTube’s Player Interface: The familiar video player interface on YouTube likely utilizes Shadow DOM to isolate its controls (play, pause, volume) and their styles from the rest of the web page.
• Interactive Maps on Google Maps: Similar to YouTube, the interactive map elements and their controls on Google Maps might benefit from Shadow DOM for isolation, ensuring a smooth user experience.
• Twitter’s Tweet Components: The individual tweet components on Twitter could potentially leverage Shadow DOM to encapsulate their content (text, images, etc.) and styling, creating a consistent look and feel.
• E-commerce Applications: Building reusable product cards or shopping cart components with Shadow DOM can streamline development and prevent style conflicts in complex e-commerce applications.

Handle Shadow Dom In Selenium

1. If there is a shadow element in the webpage we cannot interact directly with selenium code, normally if we locate the element by finding the Element method (driver.findElement()) ,  then it will throw a NosuchElement Exception.
2. In HTML there can be multiple shadow root sections; each section’s shadow properties are completely hidden from the actual Main DOM.
3. So, we cannot access the shadow elements directly from Main DOM
4. In order to overcome this situation, we have to use Javascript two methods:

• a) querySelector()
• b) shadowRoot property

Example of Basic HTML DOM

In the below screenshot, the text field can be easily accessible through XPath as it is not available under the shadow DOM in HTML DOM. We can see the result is 1 of 1 matching.

HTML DOM

Example of Shadow DOM

In the below screenshot, we are typing to inspect the ‘Request OTP’ button through XPath, but the result is 0 matching as the element is available under the Shadow Root.

Shadow DOM Example

How to inspect Shadow DOM elements in Selenium

Handling shadow DOM elements in Selenium 4 involves using the “execute_script” method of Javascript Executor to interact with shadow DOM elements.

Step 1: Open the Console in Dev tool and  create the path String of the element using Javascript ‘document.querySelector() and shadowRoot in the console.

Element path using document.querySelector()

Step 2: Convert the String into WebElement using the JavaScriptExecutor ‘executeScript()’ method.

((JavascriptExecutor) driver).executeScript(value);

Step 3: Perform the action on the WebElement using Javascript Click method.

((JavascriptExecutor) driver).executeScript("arguments[0].click();", saveButton);

Below is the three lines of code to be written to automate each Web Element which is Shadow DOM in Selenium Java framework:

Final code

Conclusion

# The approach we discussed above is helpful if you have any existing Selenium framework [with any technology say Java/Python etc.] and your web application has used Shadow DOM on few web pages.

# If your web application has thoroughly used Shadow DOM and yet you need to kick-off your automation framework then I would suggest go for WebdriverIO tool , which implicitly support Shadow DOM elements {for more details, please have a look on references}.

# Beyond above examples, any web application that prioritizes component-based development and style isolation can potentially benefit from Shadow DOM. It’s a powerful tool for building modular, reusable, and maintainable web interfaces.

Hope in future, Selenium will eventually provide complete support for Shadow DOM custom elements.

References

a) Understand Shadow DOM:

https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_shadow_DOM

b) Actions on Shadow DOM Elements using WebdriverIO:

https://webdriver.io/blog/2019/02/22/shadow-dom-support.html