This rule raises an issue when attributes, text content, or methods of BeautifulSoup elements are accessed without first checking if they exist.

BeautifulSoup’s search methods like find(), find_next(), and select_one() return None when they cannot locate a matching element. If you try to access properties or methods on None, Python raises an AttributeError or TypeError, causing your program to crash.

This commonly happens when:

You assume an HTML element will always be present in the parsed content
You chain multiple find() calls without intermediate checks
You access element attributes using subscript notation (e.g., element['href'])
You access the .text, .string, or other properties directly after a search

The following methods and access patterns can return None and are covered by this rule:

Method / Access	Description	Returns `None` when
`find()`	Searches descendants for the first tag or string matching the given criteria.	No matching element exists.
`find_next()`	Finds the first tag or string that follows the current element in document order matching the given criteria.	No matching element exists after the current position.
`find_previous()`	Finds the first tag or string that precedes the current element in document order matching the given criteria.	No matching element exists before the current position.
`find_next_sibling()`	Finds the first sibling tag after the current element matching the given criteria.	No matching sibling exists after the current element.
`find_previous_sibling()`	Finds the first sibling tag before the current element matching the given criteria.	No matching sibling exists before the current element.
`find_parent()`	Finds the first parent tag matching the given criteria by traversing upward in the tree.	No matching parent exists.
`select_one()`	Returns the first tag matching the given CSS selector.	No element matches the CSS selector.

Why is this an issue?

HTML and XML parsing libraries provide search methods to locate elements in parsed document trees. These search methods typically return a null or none value when they cannot find a matching element, rather than throwing an exception. This is expected behavior because:

HTML structures vary between pages and websites
Content may be dynamically loaded or conditionally present
Selectors might be incorrect or outdated
The page structure may change over time

The problem occurs when code assumes an element will always be found and immediately accesses its properties or methods. Attempting to operate on a null result causes runtime errors when:

Accessing properties like text content, tag names, or other element attributes on null values
Using subscript or indexing operations to access element attributes on null values
Calling methods to navigate the document tree or retrieve information on null values
Chaining multiple search operations without intermediate null checks amplifies the risk

These runtime errors cause the program to crash, often in production when encountering unexpected HTML structures. The errors can be difficult to diagnose because they depend on the specific content being parsed, which may vary between:

Different pages on the same website
The same page at different times
Different user sessions or geographic locations
Development versus production environments

What is the potential impact?

When HTML/XML parser element references are accessed without null checks, the application crashes with null reference exceptions or type errors. This has several consequences:

Service disruption: Web scraping scripts fail mid-execution, leaving incomplete or corrupt data
Poor user experience: Applications that parse HTML for features crash unexpectedly
Difficult debugging: Errors only occur with specific HTML content, making them hard to reproduce
Data loss: Partially processed data may be lost when the program terminates
Maintenance burden: Developers must handle support requests and emergency fixes

The impact is particularly severe in production web scraping pipelines and applications that process user-provided HTML content.

How to fix it

Check if the element exists before accessing its attributes. Store the search result in a variable and verify it’s not None using a conditional statement. This approach makes the code self-documenting and provides a clear place to handle missing elements.

Code examples

Noncompliant code example

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_content, "html.parser")
figure = soup.find("dd", class_="investment_sought").text  # Noncompliant
print(figure)

Compliant solution

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_content, "html.parser")
element = soup.find("dd", class_="investment_sought")
if element:
    figure = element.text
    print(figure)
else:
    print("Element not found")

Resources

Documentation

BeautifulSoup Documentation - find()
BeautifulSoup Documentation - Navigating the tree - Navigating the tree
Python AttributeError - Python documentation on AttributeError exceptions