Automated feature discovery and engineering

- When training predictive or even some generative AI system models, it's common for there to be a need to perform some level of feature engineering. This means that either a human AI engineer or the system's internal feature engineering module, or both, will attempt to choose which of the available features are most relevant to the model, or they might try to derive new features from existing ones. They could fill in missing values or encode text-based features like CD into a numerical format as well. While this is important to preparing and optimizing the training data for a given model, it's a tedious process that often relies on human judgment. Even the internal logic within the system's feature engineering module will often be limited to static pre-programed rules and functions. With the automated feature discovery and engineering pattern, a separate autonomous feature engineering agent is introduced with capabilities that allow it to assess training data and carry out various feature engineering tasks as required to tailor the features for the model. This saves a large amount of time and can further improve the quality of the feature engineering process itself. The agent is equipped with logic such as search logic, whereby the agent uses various search algorithms to explore the training data's features. It can measure how a given new feature might impact the model, and it can then use this result to guide its next steps, focusing on creating more features similar to the ones that worked well. Heuristic logic where the agent relies on pre-programmed rules of thumb and domain knowledge to guide its search for features, like it might follow a rule to prioritize certain types of features over others. Performance optimization logic, which is when the agent is focused on ensuring that whatever feature engineering tasks it performs either improve or don't negatively affect a model's performance. One potential risk when applying this pattern is that the agent might end up creating overly complex features that might be difficult for humans to understand. This can then affect a model's transparency or explainability because the decisions the model might make based on these features could be much harder for us to interpret.