This example demonstrates how to integrate Steel with OpenAI's Computer Use Assistant (CUA) API in a Node.js application. It creates a browser automation agent where the assistant sees the browser through Steel's cloud sessions, analyzes the screen, and performs actions like clicking, typing, and navigating.

Prerequisites

A Steel API key — Get one here
An OpenAI API key with access to the Computer Use Assistant preview
Node.js 18+ and npm installed

Installation

Clone this repository and navigate to the project directory:

git clone https://github.com/steel-dev/steel-cookbook
cd steel-cookbook/examples/steel-oai-computer-use-node-starter

# Install dependencies
npm install

Setup

Create a .env file in the project directory by copying the example:

cp .env.example .env

Edit the .env file and add your API keys:

STEEL_API_KEY=your_steel_api_key_here
OPENAI_API_KEY=your_openai_api_key_here

How it Works

This example demonstrates:

Creating a Steel browser session - Launches a remote browser in the cloud
Connecting with Playwright - Establishes a direct connection to control the browser
Integrating with OpenAI's Computer Use Assistant - Sends screenshots to OpenAI and receives actions to execute
Action execution - Translates OpenAI's commands into browser actions (click, type, scroll, etc.)
Continuous interaction loop - Maintains a cycle of screenshots and actions until the task is complete

Running the Example

There are two ways to run the example:

Using ts-node (for development):

npm run dev

Building and running the compiled JavaScript:

npm start

You'll be prompted to enter a task for the assistant to perform. Examples:

"Search for Steel browser on Bing and tell me about it"
"Find today's weather for New York City"
"Go to Wikipedia and find information about machine learning"

The script will:

Create a Steel session (you'll see a URL where you can watch the session live)
Send the initial screenshot to OpenAI
Execute the commands received from OpenAI
Send updated screenshots after each action
Continue this loop until the task is complete

Key Components

SteelBrowser Class

A TypeScript class that wraps around the Steel session and Playwright browser, providing methods for:

Creating and managing a browser session
Taking screenshots
Executing various browser actions (click, type, scroll, etc.)

The class implements additional browser actions like:

Navigation (goto, back, refresh)
Mouse interactions (click, doubleClick, move, drag)
Keyboard input (type, keypress)
Waiting and scrolling

OpenAI Integration

The script connects to OpenAI's Computer Use Assistant API to:

Send browser screenshots
Receive actions to execute
Process text responses from the assistant

It handles the complete interaction loop, including:

Initial task submission
Action execution
Screenshot capture and response processing
Final task completion and cleanup

Customization

You can modify the example to:

Change the initial URL (currently Bing.com)
Adjust the browser dimensions
Add more action types
Implement additional error handling
Customize the UI/UX of the interaction

TypeScript Support

This example is written in TypeScript for better type safety and developer experience. The TypeScript configuration is set up to:

Target modern JavaScript (ES2020)
Enable strict type checking
Generate declaration files
Support ES modules