Change Management Audits For Compliance Assurance

Just because it looks good on paper doesn’t mean it works in practice. Ever seen a control that looks great on paper but fails in real life? That’s exactly why we document both design and operating effectiveness. This concept changed the way I approach every audit. Let me break it down. Every control exists to address a risk. That’s its core job. So as auditors, our responsibility is to evaluate: 1. Is the control designed well enough to address the risk? 2. Is it actually working in practice? Let’s take both parts one by one: 1. Design Effectiveness This answers the question: Does the control make sense on paper? - You review how the control is structured. - You assess if the steps align with the risk it’s meant to address. - You typically use one instance to understand how it’s expected to operate. If it looks solid in theory, you move to the next phase. 2. Operating Effectiveness Now we ask: Does it actually work over time? This is where theory meets reality. - You select a sample from a defined audit period (e.g. 5–7 months). - You check if the control followed the process consistently across those instances. - If even one key step fails repeatedly - you’ve got a problem. Think of it like this: Your college syllabus (design) might look great. But if the course doesn’t actually help you apply it in real life (operation), was it effective? Same with controls. Key Insight: - Design effectiveness = one point in time - Operating effectiveness = across a period of time If you’re testing a change management control: - For design, test one change per change type to see if the process makes sense. - For operating effectiveness, test a sample of multiple changes over months to check consistency. A control isn’t strong just because it’s designed well. It’s strong when it works repeatedly. What’s been the toughest control you’ve evaluated for operating effectiveness? #itaudit #cisa #crisc #audit

Auditing is proposed in laws, regulations, and industry guidelines to mitigate AI risks, but there's a lack of established norms and standardized practices for compliance and assurance audits. Despite varied approaches like adversarial pressure testing and quantitative assessments, consensus on norms and practices is still evolving. The term 'audit' is used broadly to encompass diverse evaluations of algorithmic tools, including pressure-testing by external entities, internal pre-deployment assessments, collaborative audits, and external audits ensuring compliance with legislative or standardized framework requirements. External audits differ from risk or impact assessments in two main aspects. Firstly, algorithmic impact or risk assessments primarily focus on internal evaluations. Secondly, external audits require a conclusive outcome for stakeholders to act upon, while risk or impact assessments usually provide open-ended outputs, such as prioritized lists of risks or impacts. This paper below specifically focuses on 'external audits,' also known as 'compliance audits,' which aim to ensure adherence to specified requirements. This paper introduces the 'criterion audit' as a practical way to do external audits, inspired by how financial audits work. It is defined as: "A criteria-based independent external evaluation E of an algorithmic system S conducted by an auditor A to determine whether the given system S meets the requirements set by a normative framework." The criterion audit is characterized by 4 key features: 1. Standardized Criteria: Transparent evaluation against publicly accessible criteria. 2. Normative Framework: Measuring compliance against a specific normative framework. 3. Auditor Training: Standardized training and accreditation for auditors. 4. Public Disclosure: Results disclosed, ensuring transparency while addressing security concerns. The standard process for a criterion audit includes target scoping, documentation submission, evidence verification, publication of the audit report, and certification of the audited algorithmic system based on the evaluation against normative framework requirements. The paper demonstrates the application of the proposed approach to comply with NYC Local Law 144. The paper stresses that auditors for the criterion audit, like financial auditors, need professional values, subject matter expertise, and rigorous audit processes. It advocates for standardized audit training and suggests combining this with responsible AI education for a comprehensive understanding of complex considerations in algorithm audits. Title: "A Framework for Assurance Audits of Algorithmic Systems": Authors: BABL AI research team, led by Khoa Lam, Dr. Benjamin Lange, and Borhane Blili-Hamelin, PhD. Contributions from Shea Brown, Jovana Davidovic, and Ali Hasan.

Dear Auditors, Auditing CI/CD Change Controls Continuous Integration and Continuous Deployment (CI/CD) pipelines have become the backbone of modern IT operations. Teams push code daily, sometimes multiple times a day, with the help of automation. While this accelerates delivery, it creates a new challenge. How do you audit change controls in an environment where traditional ticket-based approvals no longer apply? This can be done by adapting the audit approach without slowing down the business. 📌 Code Review as Approval: In pipelines like GitHub Actions, GitLab, or Azure DevOps, peer review is the new approval process. An auditor should test whether all production changes require pull requests, with at least one independent reviewer before merging. 📌 Segregation of Duties: The person who develops code should not be the one approving their own pull request or deploying directly to production. Look at repository permissions, branch protection rules, and pipeline access rights. 📌 Automated Testing: Unit, integration, and security tests are often embedded in the pipeline. An audit should confirm these steps exist and that the pipeline blocks deployments when tests fail. Evidence comes from pipeline logs, not just screenshots. 📌 Rollback and Recovery: Speed without safety is dangerous. Review whether the team can roll back a failed deployment. Blue-green or canary deployments should leave an evidence trail showing when and how a rollback was triggered. 📌 Audit Trail: Every pipeline run generates metadata: who triggered it, what code was deployed, and whether it passed controls. Auditors should confirm that this metadata is retained, tamper-proof, and available for review during compliance checks. 📌 Culture of Shared Accountability: The shift to DevOps means developers, security, and operations share responsibility for controls. Auditors must approach with the mindset of validating what’s working, not just enforcing outdated processes. If your audits still ask for manual change tickets, you’re missing the point. CI/CD pipelines are not the enemy of control; they’re the new evidence source. The future of assurance lies in understanding automation, not resisting it. #ITAudit #ChangeManagement #CI/CD #DevOps #CloudSecurity #InternalAudit #RiskManagement #ITGC #Automation #CyberAudit #GRC #CyberVerge #CyberYard

Internal Audit does have a role ensuring audit issues are remediated. It is not acceptable to communicate an issue in an audit report and then "check back in" to see if management has corrected the issue. By using change management best practices, Internal Audit can help drive remediation efforts when audit issues: - Are tied to company goals - Are mapped to transformation initiatives - Have buy-in from front-line workers to senior management - Deadlines are based on work needed to be done, vs 30/60/90 day dates - Action plans broken down to incremental steps to achieve short-term wins - Implemented actions are celebrated with audit report recipients - Identified actually have a negative impact corporate objectives, opposed to internal audit just “writing them up” Managing an issue remediation strategy with intent can result in faster corrective actions and help maintain the strong reputation of internal audit. AuditBoard #internalaudit #enablingpositivechange