Progressively Enhanced WebGL Lens Refraction

There are a lot of different ways to add WebGL effects to websites. However, building it in a way that is responsive, accessible, and easy to disable for mobile is hard.

There are cases where you might want to go all-in and define your layouts in WebGL, but in our experience, most projects need a more flexible approach. For instance, clients might prefer a more scaled back traditional mobile version, or the requirement to use WebGL might change along the way.

At 14islands, we decided to base our approach on Progressive Enhancement, and bundled our learnings into a library called r3f-scroll-rig. It allows us to use semantic markup and CSS to create responsive layouts, and easily enhance them with WebGL.

What you will learn

This tutorial will show you an easy way to extend your React website with WebGL (react-three-fiber) items that are synched to your layout during scroll. We are going to use our open-source library r3f-scroll-rig.

We will:

• Add the r3f-scroll-rig library
• Enhance DOM images to be rendered using WebGL
• Enhance DOM text to be rendered using WebGL
• Add a 3D model that is tied to the layout and reacts to scroll events
• Spice it up using a lens refraction component from the React-Three-Fiber ecosystem

Adding the r3f-scroll-rig library

The scroll-rig library is compatible with most React frontend frameworks. We’ll be using Create-React-App for the simplicity of this demo, although we mostly use Next.js on our client projects. It’s also compatible with Gatsby.js or Vite for instance.

// import the scroll-rig canvas and scrollbar
import { GlobalCanvas, SmoothScrollbar } from '@14islands/r3f-scroll-rig'

// these global styles are only needed if you want to use the 
// built in classes for hiding elements. (see next section)
import '@14islands/r3f-scroll-rig/css'

export default function App() {
  return (
    <>
      <!-- Page markup goes here -->
      <article>
        <h1>Hello World</h1>
      </article>

      <!-- This enables smooth scrolling with Lenis -->
      <SmoothScrollbar />

      <!-- A position fixed canvas that covers the viewport -->
      <GlobalCanvas />
    </>
  )
}

The only way to perfectly sync WebGL objects moving on the fixed canvas with DOM elements is by animating scrolling on the main thread. This is what the SmoothScrollbar is doing for us (in an accessible way) using the excellent Lenis library.

Enhancing images to render with WebGL

The basic use case is to track a DOM element on the page and render a Threejs mesh with the same scale and position that updates in sync with the scrollbar.

The <UseCanvas> component acts as a tunnel to the GlobalCanvas. Anything we add inside it will be rendered on the global canvas while this component stays mounted. It is also automatically removed from the canvas when it unmounts. This allows us to bundle WebGL specific code inside the UI components they belong to.

For this use case we will also use the <ScrollScene> component which takes care of tracking and measuring the size of a single DOM reference. The children of this component will be positioned over the DOM element and move when we scroll the page.

import { ScrollScene, UseCanvas, styles } from '@14islands/r3f-scroll-rig'

function Image({ src }) {
  const el = useRef()
  return (
    <>
      <img ref={el} className={styles.hiddenWhenSmooth} src={src} />
      <UseCanvas>
        <ScrollScene track={el}>
          {({ scale }) => (
            <mesh scale={scale}>
              <planeGeometry />
              <meshBasicMaterial color="red" />
            </mesh>
          )}
        </ScrollScene>
      </UseCanvas>
    </>
  )
}

We should now be rendering a red WebGL plane covering the image. The ScrollScene takes care of moving the plane on scroll, and the scale property will match the exact dimensions of the DOM element.

The class styles.hiddenWhenSmooth will hide the HTML image when the SmoothScrollbar is enabled since we only want to see one of them. In our demo we will toggle the enabled flag of the scrollbar to switch between DOM/WebGL content.

// HMR will work on me since I'm defined here!
const MyScrollScene = ({ el }) => <ScrollScene track={el}>/* ... */</ScrollScene>

function MyHtmlComponent() {
  return (
    <UseCanvas>
      <MyScrollScene />
    </UseCanvas>
  )
}

Replacing the plane with an actual image

In order to do this, we need to load the image as a Three.js Texture. Instead of making a separate request, the scroll-rig has a hook called useImageAsTexture() that let’s you re-use the image from the DOM that was already loaded by the browser. You can even use reponsive images with srcset and sizes and the hook will make sure to fetch the currentSrc.

Technically it’s still making a second request, but since the URL is identical the browser will serve it directly from its cache.

Let’s wrap this image logic in a new component called <WebGLImage> and pass it the ref to the DOM image. In this case, we can re-use the same ref as the ScrollScene is tracking as it already points to the <img> tag.

function Image({ src }) {
  const el = useRef()
  return (
    <>
      <img ref={el} className={styles.hiddenWhenSmooth} src={src} />
      <UseCanvas>
        <ScrollScene track={el}>
          {(props) => (
            <WebGLImage imgRef={el} {...props} />
          )}
        </ScrollScene>
      </UseCanvas>
    </>
  )
}

The WebGLImage component loads the texture and passes it to the very helpful Image component from Drei. The Image receives the correct scale as part of the props passed down from the ScrollScene.

import { useImageAsTexture } from '@14islands/r3f-scroll-rig'
import { Image } from '@react-three/drei'

function WebGLImage({ imgRef, scrollState, dir, ...props }) {
  const ref = useRef()

  // Load texture from the <img/> and suspend until it's ready
  const texture = useImageAsTexture(imgRef)

  useFrame(({ clock }) => {
    // visibility is 0 when image enters viewport and 1 when fully visible
    ref.current.material.grayscale = clamp(1 - scrollState.visibility ** 3, 0, 1)
    // progress is 0 when image enters viewport and 1 when image has exited
    ref.current.material.zoom = 1 + scrollState.progress * 0.66
    // viewport is 0 when image enters and 1 when image reach top of screen
    ref.current.material.opacity = clamp(scrollState.viewport * 3, 0, 1)
  })

  // Use the <Image/> component from Drei
  return <Image ref={ref} texture={texture} transparent {...props} />
}

The scrollState property passed in from the ScrollScene contains some usefull info on how far the tracked element has travelled through the viewport. In this case we use it in an animation frame to update the shader uniforms.

The useImageAsTexture() hook uses the ImageBitmapLoader from Threejs if supported which uploads the image to the GPU off the main thread to avoid jank.

Enhancing text with WebGL

Replacing text with WebGL text works in a similar way, again using the ScrollScene to match the DOM element’s position and scale. We can use the Text component from Drei to render WebGL text.

We created a helper component WebGLText which calculates the WebGL text size, letter spacing, line height and color from the calculated style of the HTML text. It’s available from a separate powerups import target as it’s not a core part of the scroll-rig (and the process of getting an exact match is admittedly a bit fiddly).

In this demo we pass in the MeshDistortMaterial from Drei to make the text wobble, but this can be any custom material. Here’s how it works:

import { ScrollScene, UseCanvas, useScrollRig, styles } from '@14islands/r3f-scroll-rig'
import { WebGLText } from '@14islands/r3f-scroll-rig/powerups'
import { MeshDistortMaterial } from '@react-three/drei'

export function Text({ children, font, as: Tag = 'span' }) {
  const el = useRef()
  return (
    <>
      {/* 
        This is the real DOM text that we want to replace with WebGL   
      */}
      <Tag ref={el} className={styles.transparentColorWhenSmooth}>
        {children}
      </Tag>

      <UseCanvas>
        <ScrollScene track={el}>
          {(props) => (
            // WebGLText uses getComputedStyle() to calculate font size,
            // letter spacing, line height, color and text align
            <WebGLText
              el={el} // getComputedStyle is called on this element
              font={font} // path to the typeface (*.woff)
              glyphGeometryDetail={16} // needed for distortion to look good
              {...props} // contains scale from the ScrollScene
              >
                <MeshDistortMaterial speed={1.4} distort={0.14} />
                {children}
              </WebGLText>
            )}
          </ScrollScene>
        </UseCanvas>
    </>
  )
}

Note: It’s important to match the exact font as used in the CSS if you want the measurements to be correct.

???? The Text component uses Troika text under the hood and it only supports the woff format for now. Make sure you also use woff instead of woff2 in the CSS if you want to avoid loading two font files.

The class styles.transparentColorWhenSmooth sets the text to transparent when SmoothScrollbar is enabled. The benefit of using transparent color, instead of visibility hidden, is that the real DOM text is still selectable in the background.

Adding 3D geometries or models

You can add anything inside the UseCanvas or ScrollScene. In the demo we create a BoxGeometry for the last image and use image as a texture on each side of the box. But you can also use loaders like useGLTF to load models and adjust their scale based on the ScrollScene props.

Check it out to see how easy it is to pair it up with `MeshWobbleMaterial` from Drei, the scroll velocity from the scroll-rig and React-spring for a wobbly enter animation.

????The ScrollScene passes a reactive prop called inViewport to its children which is useful for kicking of viewport based animations.

How to handle touch devices

When it comes to touch devices we basically have two options: either disabling all the scroll-bound effects, effectively falling back to the original DOM content, or, if the site is a more immersive WebGL experience, we can tell the SmoothScrollbar to also hijack to the scroll on touch devices.

Hijack scroll on touch devices

This requires some extra settings on the underlying Lenis scrollbar as it’s not enabled by default. The reason is that most users expect the native scroll experience on these devices and it’s hard to make it feel nice.

In our demo we are using this approach as way to showcase both approaches. In our experience, the best feeling is obtained by enabling the syncTouch option on Lenis:

<SmoothScrollbar config={{ syncTouch: true }} />

The config property is a way to pass custom configuration directly to the underlying Lenis instance.

Disable scroll effects on mobile

We usually opt for disabling WebGL effects on touch devices like tablets and smartphones because it’s hard to make the scroll experience nice.

// hook in your logic here, disable if touch device or below a certain breakpoint?
const enabled = false

<SmoothScrollbar enabled={enabled} />

Remember the classes styles.hiddenWhenSmooth and styles.transparentColorWhenSmooth that we used to hide the DOM content in the earlier sections? These are automatically disabled when the SmoothScrollbar is disabled – allowing the DOM element to be visible.

Additionally we’ll want to disable the WebGL mesh from rendering as well. You can do this by accessing the global state hasSmoothScrollbar from the useScrollRig() hook:

export function Image({ src }) {
  const { hasSmoothScrollbar } = useScrollRig()
  return (
    <>
      <img src={src} className={styles.hiddenWhenSmooth} />

      {hasSmoothScrollbar && (
        <UseCanvas>
          ....
        </UseCanvas>
      )}
    </>
  )
}

And there you have it. Flipping the enabled property on the <SmoohtScrollbar> will toggle visibility of all your DOM and WebGL meshes – allowing you to easily switch between the two.

We can still keep WebGL content that is not scroll-bound, such as interactive fullscreen backgrounds and more; they will render just fine on the fixed canvas behind the scrollable content.

Lens Refraction

The scroll rig is 100% compatible with the React Three Fiber ecosystem. Let’s try adding this Lens refraction component created by Paul Henschel.

You can control where to render the <UseCanvas> children if you pass a render function as the single child to the <GlobalCanvas>. This allows us to wrap all the children in the <Lens> component:

<GlobalCanvas>
  {(globalChildren) => (
    <Lens>
      <WebGLBackground />
      {globalChildren}
    </Lens>
  )}
</GlobalCanvas>

The lens effect requires a background in WebGL to blend the content with, so we pass in a persistent <WebGLBackground /> component that renders behind all the canvas children.

Big thanks to the Poimandres collective for their contributions to the R3F ecosystem!

Wrapping up

We have found this approach very useful when accessibility and SEO is a top priority. By defining the layout using CSS, some developers can focus on building a solid responsive layout, and other can focus on the WebGL enhancements in parallel.

More documentation and common pitfalls of the scroll rig can be found at https://github.com/14islands/r3f-scroll-rig

We’re excited to see what you build with it!