diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index 1a2c39a..9859ccd 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -28,6 +28,8 @@ title: Master you knowledge base with agentic RAG - local: examples/multiagents title: Orchestrate a multi-agent system + - local: examples/web_browser + title: Build a web browser agent using vision models - title: Reference sections: - local: reference/agents diff --git a/docs/source/en/examples/web_browser.md b/docs/source/en/examples/web_browser.md new file mode 100644 index 0000000..f8db929 --- /dev/null +++ b/docs/source/en/examples/web_browser.md @@ -0,0 +1,218 @@ +# Web Browser Automation with Agents šŸ¤–šŸŒ + +[[open-in-colab]] + +In this notebook, we'll create an **agent-powered web browser automation system**! This system can navigate websites, interact with elements, and extract information automatically. + +The agent will be able to: +āœ… Navigate to web pages +āœ… Click on elements +āœ… Search within pages +āœ… Handle popups and modals +āœ… Take screenshots +āœ… Extract information + +Let's set up this system step by step. + +First, run these lines to install the required dependencies: + +```bash +pip install smolagents selenium helium pillow python-dotenv -q +``` + +Let's import our required libraries and set up environment variables: + +```python +from io import BytesIO +from time import sleep + +import helium +from dotenv import load_dotenv +from PIL import Image +from selenium import webdriver +from selenium.webdriver.common.by import By +from selenium.webdriver.common.keys import Keys + +from smolagents import CodeAgent, tool +from smolagents.agents import ActionStep + +# Load environment variables +load_dotenv() +``` + +Now let's create our core browser interaction tools that will allow our agent to navigate and interact with web pages: + +```python +@tool +def search_item_ctrl_f(text: str, nth_result: int = 1) -> str: + """ + Searches for text on the current page via Ctrl + F and jumps to the nth occurrence. + Args: + text: The text to search for + nth_result: Which occurrence to jump to (default: 1) + """ + elements = driver.find_elements(By.XPATH, f"//*[contains(text(), '{text}')]") + if nth_result > len(elements): + raise Exception(f"Match n°{nth_result} not found (only {len(elements)} matches found)") + result = f"Found {len(elements)} matches for '{text}'." + elem = elements[nth_result - 1] + driver.execute_script("arguments[0].scrollIntoView(true);", elem) + result += f"Focused on element {nth_result} of {len(elements)}" + return result + +@tool +def go_back() -> None: + """Goes back to previous page.""" + driver.back() + +@tool +def close_popups() -> str: + """ + Closes any visible modal or pop-up on the page. Use this to dismiss pop-up windows! + This does not work on cookie consent banners. + """ + webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform() +``` + +Let's set up our browser with Chrome and configure screenshot capabilities: + +```python +# Configure Chrome options +chrome_options = webdriver.ChromeOptions() +chrome_options.add_argument("--force-device-scale-factor=1") +chrome_options.add_argument("--window-size=1000,1350") +chrome_options.add_argument("--disable-pdf-viewer") +chrome_options.add_argument("--window-position=0,0") + +# Initialize the browser +driver = helium.start_chrome(headless=False, options=chrome_options) + +# Set up screenshot callback +def save_screenshot(memory_step: ActionStep, agent: CodeAgent) -> None: + sleep(1.0) # Let JavaScript animations happen before taking the screenshot + driver = helium.get_driver() + current_step = memory_step.step_number + if driver is not None: + for previous_memory_step in agent.memory.steps: # Remove previous screenshots for lean processing + if isinstance(previous_memory_step, ActionStep) and previous_memory_step.step_number <= current_step - 2: + previous_memory_step.observations_images = None + png_bytes = driver.get_screenshot_as_png() + image = Image.open(BytesIO(png_bytes)) + print(f"Captured a browser screenshot: {image.size} pixels") + memory_step.observations_images = [image.copy()] # Create a copy to ensure it persists + + # Update observations with current URL + url_info = f"Current url: {driver.current_url}" + memory_step.observations = ( + url_info if memory_step.observations is None else memory_step.observations + "\n" + url_info + ) +``` + +Now let's create our web automation agent: + +```python +# Initialize the model +model_id = "meta-llama/Llama-3.3-70B-Instruct" # You can change this to your preferred model +model = HfApiModel(model_id) + +# Create the agent +agent = CodeAgent( + tools=[go_back, close_popups, search_item_ctrl_f], + model=model, + additional_authorized_imports=["helium"], + step_callbacks=[save_screenshot], + max_steps=20, + verbosity_level=2, +) + +# Import helium for the agent +agent.python_executor("from helium import *", agent.state) +``` + +The agent needs instructions on how to use Helium for web automation. Here are the instructions we'll provide: + +```python +helium_instructions = """ +You can use helium to access websites. Don't bother about the helium driver, it's already managed. +We've already ran "from helium import *" +Then you can go to pages! +Code: +```py +go_to('github.com/trending') +``` + +You can directly click clickable elements by inputting the text that appears on them. +Code: +```py +click("Top products") +``` + +If it's a link: +Code: +```py +click(Link("Top products")) +``` + +If you try to interact with an element and it's not found, you'll get a LookupError. +In general stop your action after each button click to see what happens on your screenshot. +Never try to login in a page. + +To scroll up or down, use scroll_down or scroll_up with as an argument the number of pixels to scroll from. +Code: +```py +scroll_down(num_pixels=1200) # This will scroll one viewport down +``` + +When you have pop-ups with a cross icon to close, don't try to click the close icon by finding its element or targeting an 'X' element (this most often fails). +Just use your built-in tool `close_popups` to close them: +Code: +```py +close_popups() +``` + +You can use .exists() to check for the existence of an element. For example: +Code: +```py +if Text('Accept cookies?').exists(): + click('I accept') +``` +""" +``` + +Now we can run our agent with a task! Let's try finding information on Wikipedia: + +```python +search_request = """ +Please navigate to https://en.wikipedia.org/wiki/Chicago and give me a sentence containing the word "1992" that mentions a construction accident. +""" + +agent_output = agent.run(search_request + helium_instructions) +print("Final output:") +print(agent_output) +``` + +You can run different tasks by modifying the request. For example, here's for me to know if I should work harder: + +```python +github_request = """ +I'm trying to find how hard I have to work to get a repo in github.com/trending. +Can you navigate to the profile for the top author of the top trending repo, and give me their total number of commits over the last year? +""" + +agent_output = agent.run(github_request + helium_instructions) +print("Final output:") +print(agent_output) +``` + +The system is particularly effective for tasks like: +- Data extraction from websites +- Web research automation +- UI testing and verification +- Content monitoring + +Best Practices: +1. Always provide clear, specific instructions +2. Use the screenshot callback for debugging +3. Handle errors gracefully +4. Clean up old screenshots to manage memory +5. Set reasonable step limits for your tasks \ No newline at end of file