Download as PDF, PPTX
![A travel between a quark
and the observable universe
[10-17, 1024]
Powers of Ten (1977)
Image credit: Powers of Ten, Charles and Ray Eames](https://image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-4-2048.jpg)
![How to cope with this complexity?
E = mc2
Free[F[_], A]
M[A]
Functor[F[_]]
Cofree[S[_], A]
Months -> Hours](https://image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-10-2048.jpg)
![Type Safety Everywhere
• Value operations
• Feature operations
• Transformation Pipelines (aka Workflows)
// Typed value operations
def tokenize(t: Text): TextList = t.map(_.split(" ")).toTextList
// Typed feature operations
val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor
val tokens: Feature[TextList] = title.map(tokenize)
// Transformation pipelines
new OpWorkflow().setInput(books).setResultFeatures(tokens.vectorize())](https://image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-16-2048.jpg)
![Book Price Prediction
// Raw feature definitions: author, title, description, price
val authr = FeatureBuilder.PickList[Book].extract(_.author).asPredictor
val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor
val descr = FeatureBuilder.Text[Book].extract(_.description).asPredictor
val price = FeatureBuilder.RealNN[Book].extract(_.price).asResponse
// Feature engineering: tokenize, tf-idf, vectorize, sanity check
val tokns = (title + description).tokenize(removePunctuation = true)
val tfidf = tokns.tf(numTerms = 1024).idf(minFreq = 0)
val feats = Seq(tfidf, author).transmogrify()
val chckd = feats.sanityCheck(price, removeBadFeatures = true)
// Model training: spin up spark, load books.csv, define model -> train
implicit val spark = SparkSession.builder.config(new SparkConf).getOrCreate
val books = spark.read.csv("books.csv").as[Book]
val preds = RegressionModelSelector().setInput(price, chckd).getOutput
new OpWorkflow().setInput(books).setResultFeatures(chckd, preds)
.withRawFeatureFilter().train()](https://image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-17-2048.jpg)
![Book Price Prediction
// Raw feature definitions: author, title, description, price
val authr = FeatureBuilder.PickList[Book].extract(_.author).asPredictor
val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor
val descr = FeatureBuilder.Text[Book].extract(_.description).asPredictor
val price = FeatureBuilder.RealNN[Book].extract(_.price).asResponse
// Feature engineering: tokenize, tf-idf, vectorize, sanity check
val tokns = (title + description).tokenize(removePunctuation = true)
val tfidf = tokns.tf(numTerms = 1024).idf(minFreq = 0)
val feats = Seq(tfidf, author).transmogrify() // <- magic here
val chckd = feats.sanityCheck(price, removeBadFeatures = true)
// Model training: spin up spark, load books.csv, define model -> train
implicit val spark = SparkSession.builder.config(new SparkConf).getOrCreate
val books = spark.read.csv("books.csv").as[Book]
val preds = RegressionModelSelector().setInput(price, chckd).getOutput
new OpWorkflow().setInput(books).setResultFeatures(chckd, preds)
.withRawFeatureFilter().train()](https://image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-18-2048.jpg)
![Automatic Feature Engineering
ZipcodeSubjectPhoneEmail Age
Age
[0-15]
Age
[15-35]
Age
[>35]
Email Is
Spammy
Top Email
Domains
Country
Code
Phone
Is Valid
Top TF-
IDF Terms
Average
Income
Feature Vector](https://image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-21-2048.jpg)
![A travel between a quark
and the observable universe
[10-17, 1024]
Powers of Ten (1977)
Image credit: Powers of Ten, Charles and Ray Eames](https://crownmelresort.com/image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-4-2048.jpg)
![How to cope with this complexity?
E = mc2
Free[F[_], A]
M[A]
Functor[F[_]]
Cofree[S[_], A]
Months -> Hours](https://crownmelresort.com/image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-10-2048.jpg)
![Type Safety Everywhere
• Value operations
• Feature operations
• Transformation Pipelines (aka Workflows)
// Typed value operations
def tokenize(t: Text): TextList = t.map(_.split(" ")).toTextList
// Typed feature operations
val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor
val tokens: Feature[TextList] = title.map(tokenize)
// Transformation pipelines
new OpWorkflow().setInput(books).setResultFeatures(tokens.vectorize())](https://crownmelresort.com/image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-16-2048.jpg)
![Book Price Prediction
// Raw feature definitions: author, title, description, price
val authr = FeatureBuilder.PickList[Book].extract(_.author).asPredictor
val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor
val descr = FeatureBuilder.Text[Book].extract(_.description).asPredictor
val price = FeatureBuilder.RealNN[Book].extract(_.price).asResponse
// Feature engineering: tokenize, tf-idf, vectorize, sanity check
val tokns = (title + description).tokenize(removePunctuation = true)
val tfidf = tokns.tf(numTerms = 1024).idf(minFreq = 0)
val feats = Seq(tfidf, author).transmogrify()
val chckd = feats.sanityCheck(price, removeBadFeatures = true)
// Model training: spin up spark, load books.csv, define model -> train
implicit val spark = SparkSession.builder.config(new SparkConf).getOrCreate
val books = spark.read.csv("books.csv").as[Book]
val preds = RegressionModelSelector().setInput(price, chckd).getOutput
new OpWorkflow().setInput(books).setResultFeatures(chckd, preds)
.withRawFeatureFilter().train()](https://crownmelresort.com/image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-17-2048.jpg)
![Book Price Prediction
// Raw feature definitions: author, title, description, price
val authr = FeatureBuilder.PickList[Book].extract(_.author).asPredictor
val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor
val descr = FeatureBuilder.Text[Book].extract(_.description).asPredictor
val price = FeatureBuilder.RealNN[Book].extract(_.price).asResponse
// Feature engineering: tokenize, tf-idf, vectorize, sanity check
val tokns = (title + description).tokenize(removePunctuation = true)
val tfidf = tokns.tf(numTerms = 1024).idf(minFreq = 0)
val feats = Seq(tfidf, author).transmogrify() // <- magic here
val chckd = feats.sanityCheck(price, removeBadFeatures = true)
// Model training: spin up spark, load books.csv, define model -> train
implicit val spark = SparkSession.builder.config(new SparkConf).getOrCreate
val books = spark.read.csv("books.csv").as[Book]
val preds = RegressionModelSelector().setInput(price, chckd).getOutput
new OpWorkflow().setInput(books).setResultFeatures(chckd, preds)
.withRawFeatureFilter().train()](https://crownmelresort.com/image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-18-2048.jpg)
![Automatic Feature Engineering
ZipcodeSubjectPhoneEmail Age
Age
[0-15]
Age
[15-35]
Age
[>35]
Email Is
Spammy
Top Email
Domains
Country
Code
Phone
Is Valid
Top TF-
IDF Terms
Average
Income
Feature Vector](https://crownmelresort.com/image.slidesharecdn.com/implementingautomltechniquesatsalesforcescale-180606040621/75/Implementing-AutoML-Techniques-at-Salesforce-Scale-21-2048.jpg)
Uploaded byMatthew Tovbin
Implementing AutoML Techniques at Salesforce Scale
This document discusses the implementation of AutoML techniques at Salesforce, emphasizing the risks and uncertainties associated with the company's forward-looking statements regarding new products and services. It outlines the steps for building machine learning applications and introduces 'Optimus Prime,' an AutoML library designed for simplicity and efficient machine learning workflows. The document highlights the automation of feature engineering, selection, and model tuning, demonstrating significant time savings and improved accuracy for deployed models.
Implementing AutoML Techniques at Salesforce Scale
- 1. Implementing AutoML Techniques atSalesforce Scale mtovbin@salesforce.com, @tovbinm Matthew Tovbin, Principal Engineer, Einstein
- 2. Forward Looking Statement Statementunder the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward- looking statements.
- 3. “This lonely scene– the galaxies like dust, is what most of space looks like. This emptiness is normal. The richness of our own neighborhood is the exception.” – Powers of Ten (1977), by Charles and Ray Eames Image credit: NASA
- 4. A travel betweena quark and the observable universe [10-17, 1024] Powers of Ten (1977) Image credit: Powers of Ten, Charles and Ray Eames
- 5. • Data collection •Data preparation • Feature engineering • Feature selection • Sampling • Algorithm implementation • Hyperparameter tuning • Model selection • Model serving (scoring) • Prediction insights • Metrics Powers of Ten for Machine Learning Image credit: Powers of Ten, Charles and Ray Eames
- 6. 1. Define requirements& outcomes 2. Collect & explore data 3. Define hypothesis 4. Prepare data 5. Build prototype 6. Verify hypothesis 7. Train & deploy model Steps for Building Machine Learning Application Image credit: Bethesda Software
- 7.
- 8. Make an intelligentguess How long does it take to build a machine learning application? a. Hours b. Days c. Weeks d. Months e. More Image credit: NASA
- 9. Einstein Prediction Builder •Product: Point. Click. Predict. • Engineering: any customer can create any number of ML applications on any data, eh?!
- 10. How to copewith this complexity? E = mc2 Free[F[_], A] M[A] Functor[F[_]] Cofree[S[_], A] Months -> Hours
- 11. “The task ofthe software development team is to engineer the illusion of simplicity.” – Grady Booch Image credit: Object-Oriented Analysis and Design with Applications, by Grady Booch
- 12. Complexity vs Abstraction Imagecredit: Wikipedia
- 13. Appropriate Level ofAbstraction Language Syntax & Semantics Degrees of Freedom Lower Abstraction Higher Abstraction define higher low er • Less flexible • Simpler syntax • Reuse • Suitable for complex problems • Difficult to use • More complex • Error prone ???
- 14. Codename: Optimus Prime AnAutoML library for building modular, reusable, strongly typed ML workflows on Spark • Declarative & intuitive syntax • Proper level of abstraction • Aimed for simplicity & reuse • >90% accuracy with 100X reduction in time
- 15. Types Hide theComplexity FeatureType OPNumeric OPCollection OPSetOPList NonNullable Text Email Base64 Phone ID URL ComboBox PickList TextArea OPVector OPMap BinaryMap IntegralMap RealMap DateList DateTimeList Integral Real Binary Percent Currency Date DateTime MultiPickList TextMap TextListCity Street Country PostalCode Location State Geolocation StateMap SingleResponse RealNN Categorical MultiResponse Legend: bold - abstract type, regular - concrete type, italic - trait, solid line - inheritance, dashed line - trait mixin ... CityMap https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/field_types.htm
- 16. Type Safety Everywhere •Value operations • Feature operations • Transformation Pipelines (aka Workflows) // Typed value operations def tokenize(t: Text): TextList = t.map(_.split(" ")).toTextList // Typed feature operations val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor val tokens: Feature[TextList] = title.map(tokenize) // Transformation pipelines new OpWorkflow().setInput(books).setResultFeatures(tokens.vectorize())
- 17. Book Price Prediction //Raw feature definitions: author, title, description, price val authr = FeatureBuilder.PickList[Book].extract(_.author).asPredictor val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor val descr = FeatureBuilder.Text[Book].extract(_.description).asPredictor val price = FeatureBuilder.RealNN[Book].extract(_.price).asResponse // Feature engineering: tokenize, tf-idf, vectorize, sanity check val tokns = (title + description).tokenize(removePunctuation = true) val tfidf = tokns.tf(numTerms = 1024).idf(minFreq = 0) val feats = Seq(tfidf, author).transmogrify() val chckd = feats.sanityCheck(price, removeBadFeatures = true) // Model training: spin up spark, load books.csv, define model -> train implicit val spark = SparkSession.builder.config(new SparkConf).getOrCreate val books = spark.read.csv("books.csv").as[Book] val preds = RegressionModelSelector().setInput(price, chckd).getOutput new OpWorkflow().setInput(books).setResultFeatures(chckd, preds) .withRawFeatureFilter().train()
- 18. Book Price Prediction //Raw feature definitions: author, title, description, price val authr = FeatureBuilder.PickList[Book].extract(_.author).asPredictor val title = FeatureBuilder.Text[Book].extract(_.title).asPredictor val descr = FeatureBuilder.Text[Book].extract(_.description).asPredictor val price = FeatureBuilder.RealNN[Book].extract(_.price).asResponse // Feature engineering: tokenize, tf-idf, vectorize, sanity check val tokns = (title + description).tokenize(removePunctuation = true) val tfidf = tokns.tf(numTerms = 1024).idf(minFreq = 0) val feats = Seq(tfidf, author).transmogrify() // <- magic here val chckd = feats.sanityCheck(price, removeBadFeatures = true) // Model training: spin up spark, load books.csv, define model -> train implicit val spark = SparkSession.builder.config(new SparkConf).getOrCreate val books = spark.read.csv("books.csv").as[Book] val preds = RegressionModelSelector().setInput(price, chckd).getOutput new OpWorkflow().setInput(books).setResultFeatures(chckd, preds) .withRawFeatureFilter().train()
- 19. Workflow Book Price Prediction BookBookBooks Author Title Description Price FeatureVector Transmogrify SanityChecker RawFeatureFilter RegressionModel Selector Best Model Evaluator Cross Validation + Tokenize Pivot TF IDF Vectors Combiner
- 20. AutoML with OptimusPrime • Automated Feature Engineering • Automated Feature Selection • Automated Model Selection
- 21. Automatic Feature Engineering ZipcodeSubjectPhoneEmailAge Age [0-15] Age [15-35] Age [>35] Email Is Spammy Top Email Domains Country Code Phone Is Valid Top TF- IDF Terms Average Income Feature Vector
- 22. Automatic Feature Engineering NumericCategorical SpatialTemporal Augment with external data e.g avg income Spatial fraudulent behavior e.g: impossible travel speed Geo-encoding Text Time difference Circular Statistics Time extraction (day, week, month, year) Closeness to major events Tokenization Hash Encoding TF-IDF Word2Vec Sentiment Analysis Language Detection Imputation Track null value One Hot Encoding Dynamic Top K pivot Smart Binning LabelCount Encoding Category Embedding Imputation Track null value Log transformation for large range Scaling - zNormalize Smart Binning
- 23. Metadata! ● The nameof the feature the column was made from ● The name of the RAW feature(s) the column was made from ● Everything you did to get the column ● Any grouping information across columns ● Description of the value in the column Image credit: https://ontotext.com/knowledgehub/fundamentals/metadata-fundamental
- 24. Automatic Feature Selection Analyzefeatures & calculate statistics • Min, max, mean, variance • Correlations with label • Contingency matrix • Confidence & support • Mutual information against label • Point-wise mutual information against all label values • Cramér’s V • Etc. Image credit: http://csbiology.com/blog/statistics-for-biologists-nature-com
- 25. Automatic Feature Selection •Ensure features have acceptable ranges • Is this feature a leaker? • Does this feature help our model? • Is it predictive? Image credit: ://srcity.org/2252/Find-Fix-Leaks
- 26. Automatic Feature Selection //Sanity check your features against the label val checked = price.sanityCheck( featureVector = feats, checkSample = 0.3, sampleSeed = 42L, sampleLimit = 100000L, maxCorrelation = 0.95, minCorrelation = 0.0, correlationType = Pearson, minVariance = 0.00001, removeBadFeatures = true ) new OpWorkflow().setInput(books).setResultFeatures(checked, preds).train()
- 27. Automatic Model Selection •Multiple algorithms to pick from • Many hyperparameters for each algorithm • Automated hyperparameter tuning • Faster model creation with improved metrics • Search algorithms to find the optimal hyperparameters. e.g. grid search, random search, bandit methods
- 28. Automatic Model Selection //Model selection and hyperparameter tuning val preds = RegressionModelSelector .withCrossValidation( dataSplitter = DataSplitter(reserveTestFraction = 0.1), numFolds = 3, validationMetric = Evaluators.Regression.rmse(), trainTestEvaluators = Seq.empty, seed = 42L) .setModelsToTry(LinearRegression, RandomForestRegression) .setLinearRegressionElasticNetParam(0, 0.5, 1) .setLinearRegressionMaxIter(10, 100) .setLinearRegressionSolver(Solver.LBFGS) .setRandomForestMaxDepth(2, 10) .setRandomForestNumTrees(10) .setInput(price, checked).getOutput new OpWorkflow().setInput(books).setResultFeatures(checked, preds).train()
- 29.
- 30.
- 31. How well doesit work? • Most of our models deployed in production are completely hands free • 2B+ predictions per day
- 32. • Define appropriatelevel of abstraction • Use types to express it • Automate everything • feature engineering & selection • model selection & evaluation • Etc. Takeaway Months -> Hours
- 33. Join Optimus Primepilot today! • Open for pilot • In production at Salesforce • Piloted by large tech companies • 100% Scala https://sfdc.co/join-op-pilot
- 34. Talks for FurtherExploration • “When all the world’s data scientists are just not enough”, Shubha Nabar, @Scale 17 • “Low touch Machine Learning”, Leah McGuire, Spark Summit 17 • “Fantastic ML apps and how to build them”, Matthew Tovbin, Scale By The Bay 17 • “The Black Swan of perfectly interpretable models”, Leah McGuire and Mayukh Bhaowal, QCon AI 18