"Outlier Detection in Time Series Data Published by Springer"

What does the A.I.-enabled golden age of qualitative mean for quantitative research - when rigorous, statistically significant data collection and analysis still matters? On Oct 15, our Leigh Kessler joins Smith Geiger Group's Dan Reines at Greenbook #iiex.ai to examine how quant-focused researchers are adding the power and insights of Qual-at-Scale to survey work. https://lnkd.in/eDCCR-rY

"We identify a critical mistake: the tendency to erroneously infer causation from correlation when sample sizes are large." Large samples improve precision, not bias. The paper starts by highlighting that most big data are observational, not experimental. While we can infer causality with observational data (using quasi-experimental methods), most analyses/reports/articles use correlations (and often make abusive causal claims). The thing is, increasing sample size (everything else equal) will improve precision (reduce the variance of the estimator and hence the confidence intervals). A large n will not mechanically reduce bias. Bad news, as the whole idea of causal inference is to have an unbiased estimator. As the author states: “Most educated decision-makers know the mantra that ‘correlation does not imply causation.’ Yet our research reveals that this knowledge provides little protection against the big data fallacy.” This is exactly why I always go further than this mantra in my talks/courses. I explain why correlation does not imply causation. Link to the paper by Gaia Giambastiani and @joachim vosgerau in comment

Enhancing Synthetic Data Generation via Adaptive Kernel Density Estimation with Bayesian Optimization This paper details a novel methodology for enhancing synthetic data generation in the realm of data augmentation, specifically targeting tabular datasets with complex feature interactions. Our approach, Adaptive Kernel Density Estimation with Bayesian Optimization (AKDE-BO), combines the established strengths of Kernel Density Estimation (KDE) with the optimization capabilities of Bayesian Optimization to produce synthetic samples that more accurately capture the underlying data distribution, exhibiting a 15-20% improvement in downstream model performance compared to traditional KDE techniques. The resulting methodology offers practical, immediate benefits for data scientists and engineers seeking improved model accuracy with limited real data. 1. Introduction & Problem Definition Data augmentation via synthetic data generation is critical when dealing with imbalanced datasets, privacy restrictions, or limited access to real-world data. Traditional methods, such as simple data duplicat https://lnkd.in/gpfJRrZ4

⚠️ Preprint out! ⚠️ Estimating causal effects efficiently doesn’t have to mean discovering the entire causal graph! In fact, local information around variables is also enough. We introduce Local Optimal Adjustments Discovery (LOAD), a local causal discovery method that identifies ✨statistically efficient✨ adjustment sets while staying ⚡computationally efficient⚡ 📜 Preprint: https://lnkd.in/eDdM6txV 👾 Code: https://lnkd.in/g9UvDbVR 🌐 Global discovery methods can find optimal adjustment sets, but at a huge computational cost. 📍 Local discovery methods are fast, but only find sub-optimal adjustment sets. 🎯 LOAD combines the best of both worlds: it finds the same optimal adjustments as global methods, but uses only local discovery around variables! In experiments, LOAD runs nearly as fast as local discovery methods while retaining the statistical efficiency of global methods, enabling scalable, reliable causal discovery and effect estimation even in high-dimensional settings.

SAT Benchmarks with Built-In Behavioural Intelligence Each Byt-Wyze SAT benchmark instance now ships with a complete behavioural analysis file (.md) derived directly from solver logs — transforming every CNF into a traceable research artifact. Example instance: sat-Moderate-C-44008conflicts-transition.cnf Naming Convention: sat → satisfiable instance (unsat for unsatisfiable) Moderate-C → Byt-Wyze hardness tier 44008conflicts → recorded solver conflict count transition → denotes phase-region proximity Included in the .md file: A structured summary of solver dynamics and structural metrics: 🔢 CNF Hardness Taxonomy • Hardness Tier: Moderate-C • Conflict Range: 35K–55K • Tier Traits: ~90% long-learnt, mild symbolic constraints 📐 Structural Density • Clause-to-Variable Ratio: 4.10:1 • Region Type: Phase-Transition 🧬 Behavioural Fingerprint • Branching: vsid ↔ vmtf cycles • Restarts: geom-stb pattern • Tags: phase-transition, conflict-saturated, fast-switching 📊 Core Metrics • Conflicts: 44 008 • Decisions: 62 199 • Time: 3.76 s • Decisions/Conflict: 1.41 • Conflicts/Restart: 258.9 Each instance becomes a data-rich reference point for solver research — complete with structural ratios, branching behaviour, restart signatures, and performance metrics. Every CNF is delivered with its log and Markdown companion file, forming a documented solver event — traceable, structured, and competition-ready. This approach defines the Byt-Wyze benchmark ecosystem: data-driven, taxonomy-consistent, and built for transparent reproducibility across solver research, teaching, and performance evaluation. This level of traceability moves the Byt-Wyze benchmark ecosystem beyond simple pass/fail testing — into a framework built for behavioural analysis, solver profiling, and reproducible performance studies. #BytWyze #SAT #Research #SATCompetition #Benchmarks #SolverAnalytics #ComputationalComplexity #ResearchData

Excited to share the latest milestone in my research journey! Our new paper, "Theoretically Based Dynamic Regression (TDR)—A New and Novel Regression Framework for Modeling Dynamic Behavior," has just been published in MDPI Stats. Modeling complex dynamic systems—especially for real-time monitoring and edge-based control—often faces tough constraints, like limited compute resources. Traditional lagged-based methods can be parameter-heavy and lack physical interpretability, making them inefficient for practical applications. TDR flips the script by integrating theoretical dynamic structures (like first- or second-order-plus-dead-time models) with static regression techniques. This results in fewer parameters, physically meaningful constraints, and better performance on real-world data, from human weight dynamics to distillation processes and glucose forecasting for diabetes management. Proud to have collaborated with an amazing team on this: Derrick K. Rollins, Marit Nilsen-Hamilton, Kendra Kreienbrink, Spencer Wolfe, and Jacob Oyler. This builds on our ongoing work to tackle edge-compute challenges in system modeling. Check it out here: https://lnkd.in/gf6KXgBZ #DynamicModeling #RegressionAnalysis #AIinEngineering #EdgeComputing #DataScience #ResearchPublication

A Dual Approach to Identifying Market Regimes Identifying structural breaks in financial time series is a foundational task in quantitative analysis. While deterministic methods provide point estimates of changepoints, a more robust approach involves a dual framework using both Bayesian inference and state-space models. 1. Bayesian Change Point Detection (BCPD): Quantifying the "When" BCPD moves beyond simple segmentation by treating changepoint locations as random variables. It computes the full posterior probability of a structural break at every point in the time series. By combining a prior belief about changepoint frequency with the data's likelihood, this method quantifies the uncertainty of each detected inflection point. The output isn't a single answer, but a probability distribution across time, allowing us to assess the confidence of each potential regime shift. 2. Hidden Markov Models (HMMs): Characterizing the "What" In parallel, HMMs provide a powerful framework for characterizing the regimes themselves. The model assumes the market operates in a set of unobservable (hidden) states. It learns two key sets of parameters from the data: a) Emission Probabilities: The statistical profile of the data within each state (e.g., a Gaussian distribution with a specific mean/variance). b) Transition Probabilities: The likelihood of moving from one state to another. The Viterbi algorithm then infers the most likely sequence of hidden states, effectively clustering the time series into its constituent regimes. Synergy and Application: Using these methods in tandem provides powerful cross-validation. A high-probability changepoint from BCPD that aligns with a state transition in the HMM provides high-confidence validation of a historical regime boundary. This rigorous historical labeling is the critical first step. It yields the high-quality ground-truth data necessary to train supervised machine learning models for a forward-looking view, enabling us to forecast the probability of future regime transitions using macroeconomic and market-based features. #QuantitativeFinance #BayesianStatistics #MarkovModels #TimeSeries #MachineLearning #RiskManagement #DataScience #InvestmentManagement

✍️ 𝐃𝐚𝐲 𝟕/𝟑𝟎 In the last two days, we discussed Pooling and External/Synthetic Control Arms (ECA). ❓ But here’s the key question: how do we know if historical trial data is “fit” for either approach? That’s where the Test-Then-Pool method comes in. 🔹 𝐓𝐡𝐞 𝐋𝐨𝐠𝐢𝐜 When we have historical control data and a concurrent control arm in the new trial, we first test comparability: Null hypothesis (𝐇₀): No difference between historical and new controls. Tests used (depending on endpoint): • Survival → log-rank test, Cox proportional hazards • Binary → chi-square / Fisher’s exact • Continuous → t-test / ANOVA /Wilcoxon rank-sum / Kruskal–Wallis 🔹 𝐓𝐡𝐞 𝐃𝐞𝐜𝐢𝐬𝐢𝐨𝐧 ➡️ If not significantly different → ✅ Pool the historical and current controls (gain precision & power). ➡️ If significantly different → ❌ Do not pool. Instead: • Use historical data only as an external control arm, or • Exclude it entirely if too inconsistent. 🔗 https://lnkd.in/gb5bbGH6 (free full text not available) 🔹 𝐄𝐯𝐢𝐝𝐞𝐧𝐜𝐞 𝐟𝐫𝐨𝐦 𝗪𝐢𝐞𝐝 𝐞𝐭 𝐚𝐥. (𝟐𝟎𝟐𝟒, 𝐁𝐌𝐂 𝐌𝐞𝐝 𝐑𝐞𝐬 𝐌𝐞𝐭𝐡𝐨𝐝𝐨𝐥𝐨𝐠𝐲) This study simulated thousands of trial scenarios and showed: - Naive pooling (blindly combining) inflates Type I error → false positives. - Test-Then-Pool controlled Type I error properly, giving regulatory confidence. - When historical ≈ current, it improved efficiency and reduced required sample size. 🔗 https://lnkd.in/gb5bbGH6 💡 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲 𝐟𝐨𝐫 𝐞𝐩𝐢 & 𝐛𝐢𝐨𝐬𝐭𝐚𝐭𝐢𝐬𝐭𝐢𝐜𝐬 Test-Then-Pool is the frequentist safeguard: it protects against bias and false positives when deciding if history can be pooled. It’s simple, transparent, and regulator-friendly. 👉 Tomorrow, we’ll move into the Bayesian borrowing approaches that try to solve this conservatism by adaptively weighting history.

Experimentation & Causal Inference: Beyond the p‐Value Randomized experiments are the gold standard but not always feasible. When you rely on observational data, lean on causal methods (matching, difference‐in‐differences, synthetic controls) and pre‐registration to reduce researcher degrees of freedom. Design wins happen upfront. Power your tests, define guardrails (e.g., fraud, supply constraints), and choose evaluation windows that reflect real behavior. Resist “peeking” unless you’ve planned sequential analysis. What’s your most surprising experiment outcome? When you couldn’t randomize, which causal approach held up best under scrutiny?

Advancing image-based meta-analysis through systematic use of crowdsourced NeuroVault data: (B) Multi-level image selection framework: Three complementary selection methods were applied to address data quality issues. Initial selection ...