Using Python 'in' operator to check if Dataframe column values are in list of strings results in ValueError

With df[col] in some_tuple you are searching df[col] inside some_tuple, which is obviously not what you want. What you want is df[col].isin(some_tuple), which returns a new series of booleans of the same length of df[col].

So, why you get that error anyway? The function for searching a value in a tuple is more or less like the following:

for v in some_tuple:
    if df[col] == v:
        return True
return False

• df[col] == v evaluates to a series result; no problem here
• then Python try to evaluate if result: and you get that error because you have a series in a condition clause, meaning that you are (implicitly) trying to evaluate a series as a boolean; this is not allowed by pandas.

For your problem, anyway, I would use DataFrame.apply. It takes a function that map a value to another; in your case, a function that map each ethnicity to a category. There are many ways to define it (see options below).

import numpy as np
import pandas as pd

d = pd.DataFrame({
    'field': range(6),
    'ethnicity': list('ABCDE') + [np.nan]
})

# Option 1: define a dict {ethnicity: category}
category_of = {
    'A': 'X',
    'B': 'X',
    'C': 'Y',
    'D': 'Y',
    'E': 'Y',
    np.nan: np.nan,
}
result = d.assign(category=d['ethnicity'].apply(category_of.__getitem__))
print(result)

# Option 2: define categories, then "invert" the dict.
categories = {
    'X': ['A', 'B'],
    'Y': ['C', 'D', 'E'],
    np.nan: [np.nan],
}
# If you do this frequently you could define a function invert_mapping(d):
category_of = {eth: cat
               for cat, values in categories.items()
               for eth in values}
result = d.assign(category=d['ethnicity'].apply(category_of.__getitem__))
print(result)

# Option 3: define a function (a little less efficient)
def ethnicity_to_category(ethnicity):
    if ethnicity in {'A', 'B'}:
        return 'X'
    if ethnicity in {'C', 'D', 'E'}:
        return 'Y'
    if pd.isna(ethnicity):
        return np.nan
    raise ValueError('unknown ethnicity: %s' % ethnicity)

result = d.assign(category=d['ethnicity'].apply(ethnicity_to_category))
print(result)