This is an issue when test code verifies that an exception was raised without checking its type or its message.

Why is this an issue?

Assertions are statements that check whether certain conditions are true. They validate that the actual results of a code snippet match the expected outcomes. By using assertions, developers ensure their code behaves as intended and identify potential bugs or issues early in the development process.

When testing exception handling, it is not sufficient to verify only that some exception was raised. Tests should also validate the exception type or message to differentiate expected exceptions from unexpected ones.

Consider a function that should raise a ValueError when given invalid input. If your test only checks that an exception occurred, it cannot distinguish between:

• The expected ValueError indicating proper input validation
• An unexpected TypeError from a bug in the code
• An unexpected AttributeError from a null reference error

All three scenarios would make the test pass, even though two represent actual bugs.

In Python testing frameworks like pytest and unittest, several patterns can lead to insufficiently specific exception testing:

• Using pytest.raises(Exception) without specifying the exception type or checking the message
• Using unittest.assertRaises(Exception) which catches any exception
• Not validating the exception message when the same exception type could be raised for different reasons
• Catching exceptions in try-except blocks without asserting on the caught exception

These patterns create fragile tests that may pass for the wrong reasons, allowing bugs to slip through your test suite undetected.

What is the potential impact?

Tests that do not verify the exception type or message can:

• Miss bugs: Your tests may pass even when your code throws unexpected exceptions due to bugs
• Reduce confidence: You cannot trust that your error handling works correctly
• Hide breaking changes: Refactoring that changes exception types will not be caught by tests
• Complicate debugging: When tests fail, you will not have clear information about what went wrong

This weakens the safety net that your test suite provides and can lead to production issues that could have been caught during development.

How to fix it in pytest

Code examples

Noncompliant code example

import pytest

class InvalidConfigError(Exception):
    def __init__(self, field, message):
        self.field = field
        super().__init__(message)

def load_config(data):
    if 'api_key' not in data:
        raise InvalidConfigError('api_key', 'Missing required field')

def test_missing_config():
    with pytest.raises(Exception):  # Noncompliant: does not verify the type or the message
        load_config({})

Compliant solution

With pytest, use pytest.raises() with a specific exception type instead of the generic Exception class, or use the match parameter to validate the exception message using a regular expression pattern. You can also use both when appropriate.

import pytest

class InvalidConfigError(Exception):
    def __init__(self, field, message):
        self.field = field
        super().__init__(message)

def load_config(data):
    if 'api_key' not in data:
        raise InvalidConfigError('api_key', 'Missing required field')

def test_missing_config():
    with pytest.raises(InvalidConfigError) as exc_info:
        load_config({})

    assert exc_info.value.field == 'api_key'
    assert 'Missing required field' in str(exc_info.value)

How to fix it in unittest

With unittest, use assertRaises() with a specific exception type, or assertRaisesRegex() when you want to verify the message as well. You can use either approach depending on what the test needs to verify.

Code examples

Noncompliant code example

import unittest

def parse_age(value):
    age = int(value)
    if age < 0:
        raise ValueError("Age cannot be negative")
    return age

class TestAgeParser(unittest.TestCase):
    def test_negative_age(self):
        with self.assertRaises(Exception):  # Noncompliant: does not verify which exception
            parse_age("-5")

Compliant solution

import unittest

def parse_age(value):
    age = int(value)
    if age < 0:
        raise ValueError("Age cannot be negative")
    return age

class TestAgeParser(unittest.TestCase):
    def test_negative_age(self):
        with self.assertRaises(ValueError):
            parse_age("-5")

Resources

Documentation

• pytest documentation - raises
• unittest documentation - assertRaises