Software Testing Basics

🎢 How To Roll Out New Features Without Breaking UX. Practical guidelines to keep in mind before releasing a new feature ↓ 🚫 We often assume that people don’t like change. 🤔 But people go through changes their entire lives. ✅ People accept novelty if they understand/value it. ✅ But: breaking changes disrupt habits and hurt efficiency. ✅ Roll out features slowly, with multiple layers of testing. ✅ First, study where a new feature fits in key user journeys. ✅ Research where different user types would find and apply it. ✅ Consider levels of proficiency: from new users to experts. ✅ Actively support existing flows, and keep them a default. 🚫 Assume low adoption rate: don’t make a feature mandatory. ✅ First, test with internal employees and company-wide users. ✅ Then, run a usability testing with real users and beta testers. ✅ Then, test with users who manually opt in and run a split-test. ✅ Allow users to try a new feature, roll back, dismiss, remind later. ✅ Release slowly and gradually and track retention as you go. As designers, we often focus on how a new feature fits in the existing UI. Yet problems typically occur not because components don’t work visually, but rather when features are understood and applied in unexpected ways. Rather than zooming in too closely, zoom out repeatedly to see a broader scope. Be strategic when rolling out new versions. Especially in complex environments, we need to be rather cautious and slow, especially when operating on a core feature. That’s a strategy you could follow in such scenarios: 1. Seek and challenge assumptions. 2. Define how you’ll measure success. 3. Have a rollback strategy in place. 4. Test with designers and developers. 5. Test with internal company-wide users. 6. Test with real users in a usability testing. 7. Start releasing slowly and gradually. 8. Test with beta testers (if applicable). 9. Test with users who manually opt in. 10. Test with a small segment of customers first. 11. Split-test the change and track impact. 12. Wait and track adoption and retention rates. 13. Roll out a feature to more user segments. 14. Run UX research to track usage patterns. 15. Slowly replace deprecated flows with the new one. With a new feature, the most dangerous thing that can happen is that loyal, experienced users suddenly lose their hard-won efficiency. It might be caused by oversimplification, or mismatch of expectations, or — more often than not — because a feature has been designed with a small subset of users in mind. As we work on a shiny new thing, we often get blinded by our assumptions and expectations. What really helps me is to always wear a critical hat in each design crit. Relentlessly question everything. Everything! One wrong assumption is a goldmine of disastrous decisions waiting to be excavated. [continues in comments ↓]

As a client project manager, I consistently found that offshore software development teams from major providers like Infosys, Accenture, IBM, and others delivered software that failed 1/3rd of our UAT tests after the provider's independent dedicated QA teams passed it. And when we got a fix back, it failed at the same rate, meaning some features cycled through Dev/QA/UAT ten times before they worked. I got to know some of the onshore technical leaders from these companies well enough for them to tell me confidentially that we were getting such poor quality because the offshore teams were full of junior developers who didn't know what they were doing and didn't use any modern software engineering practices like Test Driven Development. And their dedicated QA teams couldn't prevent these quality issues because they were full of junior testers who didn't know what they were doing, didn't automate tests and were ordered to test and pass everything quickly to avoid falling behind schedule. So, poor quality development and QA practices were built into the system development process, and independent QA teams didn't fix it. Independent dedicated QA teams are an outdated and costly approach to quality. It's like a car factory that consistently produces defect-ridden vehicles only to disassemble and fix them later. Instead of testing and fixing features at the end, we should build quality into the process from the start. Modern engineering teams do this by working in cross-functional teams. Teams that use test-driven development approaches to define testable requirements and continuously review, test, and integrate their work. This allows them to catch and address issues early, resulting in faster, more efficient, and higher-quality development. In modern engineering teams, QA specialists are quality champions. Their expertise strengthens the team’s ability to build robust systems, ensuring quality is integral to how the product is built from the outset. The old model, where testing is done after development, belongs in the past. Today, quality is everyone’s responsibility—not through role dilution but through shared accountability, collaboration, and modern engineering practices.

A conversation between a QA lead and a client related to test automation. QA Lead: Good morning! I'm excited to talk to you about an important enhancement to our testing strategy: test automation. Client: Hello! I've heard a bit about test automation, but I'm not sure how it fits into our current process. We've been doing fine with exploratory testing, haven't we? QA Lead: You're right, our exploratory testing has been effective, but there's a key area where automation can greatly help. Consider how our development team typically takes two weeks to develop a new feature, and then our testers spend a week testing it. As our software grows with more features, exploratory testing becomes a bottleneck. Client: How so? QA Lead: Well, with each new feature, our testers aren't just testing the new functionality. They also need to ensure all the previous features are still working — this is called regression testing. With exploratory testing, the time required for this grows exponentially with each new feature. Client: I see. So, testing becomes slower as our software grows? QA Lead: Exactly. For instance, by the time we reach feature number 15, testing could take much longer than it did for the first feature, because testers have to cover everything we've built so far. Client: That would slow down our entire development cycle. QA Lead: Right, and this is where test automation comes in. By automating repetitive and regression tests, we can execute them quickly and frequently. This dramatically reduces the time required for each testing cycle. Client: But does this mean we're replacing exploratory testing with automation? QA Lead: Not at all. Test automation doesn't replace exploratory testing; it complements it. There will always be a need for the human judgment and creativity that exploratory testers provide. Automation takes care of the repetitive, time-consuming tasks, allowing our exploratory testers to focus on more complex testing scenarios and exploratory testing. Client: That sounds like a balanced approach. So, we speed up testing without losing the quality that exploratory testing brings? QA Lead: Precisely. This combination ensures faster release cycles, maintains high quality, and keeps testing costs under control over the long term. It's a sustainable approach for growing software projects like ours. Client: Understood. Implementing test automation seems like a necessary step to keep up with our software development. Let's proceed with this strategy. QA Lead: Excellent! I'm confident that this will significantly improve our testing efficiency and overall product quality. #testautomation #exploratorytesting #regression #QA #testing

A few months ago, a junior dev on the team sent me 10 merge requests in a single day. It all started at 10 AM with: "feat/JIRA-111-QA-FIX" I reviewed it, merged it. Pipeline passed. All good. Ten minutes later—ding. Another MR: "feat/JIRA-111-QA-FIX" By 3 PM, I’d merged 10 MRs, all for the same feature, all with the same name. Finally, I asked: “Why so many MRs for the same thing?” He replied: “I dunno, but QA keeps finding bugs.” So I asked again: “Did you merge and test it in dev before sending it over?” The answer: “Uhh... no.” And that’s a perfect example of why dev testing is crucial. When you skip dev testing, you’re just pushing your unfinished work downstream. It causes unnecessary work for your QA team, delays the process, and introduces avoidable bugs into the pipeline. Dev testing isn’t just a step in the workflow—it’s your responsibility. It’s what ensures the code you push is functional and ready for the next stage. Takeaway: Test in dev. Ship clean. Save everyone’s time, including your own.

Andrew Ng's team once made a mistake in a paper. (It happened due to random splitting) It is common to generate train and validation sets using random splitting. However, in many situations, it can be fatal for model building. Consider building a model that generates captions for images. Due to the inherent nature of language, every image can have many different captions. This dataset will be like this: ↳ (Image-1, Caption-1) ↳ (Image-1, Caption-2) ↳ (Image-1, Caption-3) ↳ (Image-2, Caption-1) ↳ (Image-2, Caption-2) ↳ and so on... If we use random splitting: - the same data point (image) will be available in the train and validation sets. As a result, we end up evaluating the model on instances it was trained on. This is an example of data leakage (also called group leakage), resulting in overfitting! Group shuffle split solves this. There are two steps: 1) Group all training instances corresponding to one image. 2) After grouping, the ENTIRE GROUP (all examples of one image) must be randomly dedicated to the train or validation set. This will prevent the group leakage. The same thing happened in Andrew Ng's paper where they prepared a medical dataset to detect pneumonia. - Total images = 112k - Total patients = 30k Due to random splitting, the same patient's images were available both in train and validation sets. This unknowingly led to data leakage and validation scores looked much better than they should have. A few days later, the team updated the paper after using the group shuffle split strategy to ensure the same patients did not end up in both train and validation sets. P.S. It is natural to make mistakes. That's how we grow. It is important we learn from them. 👉 P.P.S. Will you be double cautious now during data splitting? ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

Too much to test, never enough time. Traditional testing approaches often lead to testing everything equally, resulting in bloated test suites and inefficient resource use. This not only slows down the process but also risks missing critical issues. It’s frustrating to see effort wasted on low-impact areas, while high-risk components don’t get the attention they deserve. This can lead to last-minute firefighting, costly fixes, and a lack of clear communication with stakeholders about what truly matters. Risk-based testing is the first step towards a solution. By prioritising high-risk areas, we focus our efforts where they have the most impact. This approach reduces redundancies, optimises resource allocation, and ensures our testing is aligned with business priorities. The result? Leaner processes, better quality software, and more efficient use of time. How are you tackling redundancies in your testing process?

How do you ship features to specific users without breaking prod? (drumroll…..) Feature Flagging Before joining an enterprise tech company, I had no idea this was even a thing. In freelance projects? Never needed it. In side projects? Never thought about it. But what even is feature flagging? Like it sounds, it’s basically a switch for your product features. It lets you roll out features to a specific set of users without affecting everyone. Imagine you: - have 1,000 users but only want 200 premium users to access a new feature? - need to gradually roll out a change instead of deploying to all users at once? - want a kill switch in case something goes wrong? How do you do that without a global rollout? Feature flags. But how do you implement it? - Database-driven flags Store feature flags in a table mapping to user ids. - Environment/config-based flags Toggle features using config files or env variables. - Third-party services (easier & scalable) Use tools like Statsig or LaunchDarkly to manage feature rollouts easily. But why does it matter? Not every company needs feature flags. I didn’t think I needed them…until I shipped a breaking query optimization change that caused 500 errors in staging. (don’t ask me how it got slipped in the PR review 💀 ) Had we used a feature flag, I could’ve just toggled it off instead of reverting & redeploying. But why does no-one talk about it? No full-stack SSR Next.js app with Neon tutorial will ever mention this. You only see it in large-scale products where: - Downtime is not an option. - Features must be rolled out in stages. - A bad release can cost serious $$$. Have you ever broken prod because of a bad deployment? Have feature flags ever saved your butt after a bad release?

𝗦𝘁𝗼𝗽 𝗨𝘀𝗶𝗻𝗴 𝗖𝗼𝗻𝗳𝗶𝗴 𝗦𝘄𝗶𝘁𝗰𝗵𝗲𝘀! 𝗘𝗺𝗯𝗿𝗮𝗰𝗲 𝗙𝗲𝗮𝘁𝘂𝗿𝗲 𝗙𝗹𝗮𝗴𝘀 𝗳𝗼𝗿 𝗦𝗺𝗮𝗿𝘁𝗲𝗿 .𝗡𝗘𝗧 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 Although Feature Flags have been available in .NET for some time, they still aren’t widely adopted. Feature Flags, or Feature Toggles, allow you to enable or disable features dynamically without redeploying code. They’re ideal for running new and legacy features in parallel, ensuring smoother transitions and reliable testing environments. With .NET’s Microsoft.FeatureManagement library, managing Feature Flags becomes straightforward and effective. Imagine you’re gradually migrating a legacy app and want to test new functionality incrementally. Using Feature Flags combined with the Strangle Pattern, you can replace old components step-by-step, toggling between new and legacy services as needed. 🤷 Why Choose Feature Flags Over Config Switches? Feature Flags provide a seamless way to control feature rollouts, essential for minimizing risk during updates. Using Microsoft.FeatureManagement in .NET, you can set up an API with multiple service versions and toggle between methods based on a flag. A simple check with IsEnabledAsync controls which code path runs, all managed through a setting in appsettings.json. 💡 Benefits of Microsoft.FeatureManagement • Condition-Based Rules: Set feature access rules based on conditions like user profile, location, or time, offering granular control over feature availability. • Azure Integration: Seamless integration with Azure for centralized feature management across applications. • Monitoring and Metrics: Built-in monitoring helps track feature usage and identify potential issues, making management easier. For example, Azure Feature Flags can be set to refresh periodically, enabling real-time changes in production. Integrating with Azure App Configuration is simple, allowing you to adjust cache expiration for updates without redeployment. Feature Flags are a powerful tool for managing gradual feature rollouts and legacy migrations. Microsoft.FeatureManagement enhances this practice by offering robust control, monitoring, and seamless integration with other Microsoft services. 💡 Example in .NET In a typical setup, a MyService class toggles between NewMethod and LegacyMethod based on the “EnableNewFeature” flag in appsettings.json. The Microsoft.FeatureManagement library makes these checks quick and flexible, allowing you to choose which functionality to enable without code redeployments. To highlight just how simple this setup can be, check out the images I’ve included below. These visuals walk you through the straightforward setup and usage, showing that you can get started with Feature Flags in minutes! #CSharp #DotNet #FeatureFlags #FeatureToggle #Azure

Over coffee the other day, a friend asked me, "How can we really trust a machine learning model's accuracy?" It got me thinking hiw to explain this, and I thought I'd share some insights from my own experience. You know that feeling when your model hits 95% accuracy? Feels like you've struck gold, right? But hold on a second—how do we know that number truly reflects the model's performance? Let me break it down in simple terms. Train-Test Split: This is the basic approach. You take your dataset and split it into two parts: - **Training Set**: Where the model learns patterns. - **Test Set**: Where you evaluate the model's performance. Sounds straightforward, but there's a catch. If the split isn't representative of the overall data, your accuracy score might be misleading. It's like studying only half the book before an exam covering the whole thing. K-Fold Cross-Validation: This is where things get interesting. Instead of a single split, you divide your data into *k* equally sized folds. Here's how it works: 1. **Split** your data into *k* folds. 5 is standard 2. **Train** the model *k* times, each time using a different fold as the test set and the remaining folds as the training set. 3. **Aggregate** the results to get an average performance metric. Why go through all this? Because it gives you a more reliable estimate of how your model will perform on unseen data. It's like taking multiple practice exams to gauge how you'd do on the real thing. So, how can we be sure about a model's accuracy? From my experience, relying solely on a single train-test split can be risky. The model might perform well on one specific split but poorly on another. K-fold cross-validation mitigates this by testing the model across multiple data subsets. Here's a funny thing I encountered: I once trusted a model with high accuracy from a train-test split, only to have it flop in production. Turns out, I'd accidentally trained it on data that wasn't representative of real-world scenarios. Lesson learned! Key Takeaways: - **No model is perfect**: Accuracy is just one metric. Consider others like precision, recall, and F1 score. - **Understand your data**: Make sure your data is representative of the problem you're trying to solve. - **Use cross-validation**: It provides a more robust evaluation of your model. Hope this sheds some light on model accuracy! Always happy to chat more about this over coffee. Cheers!

Chip Validation is Broken—And It's Costing the Industry Billions NVIDIA ($2T market cap), Intel ($150B+), AMD ($300B) — all pushing the limits of chip design. But nobody’s talking about the hidden bottleneck slowing their progress: Validation inefficiencies that delay launches and burn resources. These aren’t isolated issues. They’re systemic failures. Before a chip hits the market, it undergoes extensive testing to ensure it performs as expected under real-world conditions. This process—chip validation—is crucial for verifying functionality, performance, and reliability. But validation today is painfully slow. Engineers spend weeks scripting and running tests manually, dealing with fragmented tools, and sifting through scattered data to debug issues. The result? Delays, inefficiencies, and skyrocketing costs. The Broken Playbook -Weeks spent scripting tests instead of analyzing results -Scattered data across tools, slowing debugging cycles -Manual processes introducing errors and inefficiencies Behind the cutting-edge innovation lies a brutal truth: Validation is stuck in the past the last software that was built for validation released in 2001 (LabView). The Future of Validation: A new era is emerging: Automated, AI-driven validation workflows. And the results are game-changing: 20x faster test execution Fewer errors, more actionable insights Seamless collaboration across teams At Atoms, we’re not just talking about the future—we’re building it: One platform to automate and accelerate chip validation AI-assisted workflows to streamline debugging Real-time insights for faster decision-making Because while traditional validation burns months, the companies of the future are already moving at light speed. The old methods had their time. The age of AI-powered validation has begun. #Semiconductors #ChipValidation #FasterTimeToMarket #TestFlow