Learning From a Software Incident Without Blaming the Wrong Person

A software incident often includes a visible action: an employee changed a setting, a developer released an update, or a manager approved a new process. That action may be important, but stopping the investigation there can leave the conditions that enabled the failure unchanged. A useful review asks how the incident developed and what would prevent the same outcome through a comparable sequence.

Reconstruct the sequence from evidence

Establish a timeline covering the first symptom, relevant decisions, response actions, and restoration. Distinguish confirmed events from assumptions. Use records and accounts from the people involved rather than selecting an explanation before the facts are assembled.

Consider a hypothetical wholesaler whose prices are accidentally changed during an import. The employee selected the wrong file, but the product may also have accepted the file without a meaningful preview, provided little warning, and offered no simple way to reverse the change. All of those conditions deserve examination.

Google's guidance on learning from incidents describes a blameless review culture intended to support learning. This is not the removal of responsibility. The business interpretation is to examine contributing conditions fairly so that corrective work addresses more than the last person in the sequence.

Ask what made the action reasonable at the time

Interview people about the information and options they had. An employee may have followed the documented procedure or used a workaround that the company had informally accepted. Understanding that context helps distinguish individual conduct from a process the business itself made difficult.

For the wholesaler, similar filenames and an unclear preview may have increased the chance of a mistaken selection. A rushed deadline could also have prevented a second review. These are hypotheses to verify, not excuses to assume. The review should identify evidence for each contributing factor.

Avoid hindsight judgments that rely on information unavailable during the event. A manager's later knowledge that the file was incorrect does not establish that the employee could readily recognize the error at the time. Focus on what the workflow could have made clearer before the damaging action.

Review detection and response as well as prevention

Ask how the company noticed the problem and whether it could have noticed sooner. Investigate which records or checks helped establish the extent of the change. A failure can become more consequential when detection and recovery are slow or unclear.

OWASP's logging guidance discusses using records of relevant events to support investigation. In business terms, the company should have enough evidence to reconstruct important changes without unnecessarily storing sensitive material. The appropriate record depends on the workflow.

For the wholesaler, a history of the import and previous prices may help correction. The company also needs a route for staff to report unusual results and authority to pause further imports. Examine whether the response relied on one person's memory or on a process others could follow.

Turn findings into specific improvements

Describe actions that address verified conditions. A better preview, a permission boundary, a reversible import, or an appropriate approval step may be more effective than telling everyone to be more careful. Match the improvement to the cause and the operating consequence.

Assign a responsible person and a way to check completion. If the product changes, test the original scenario and legitimate work that must still succeed. If staff instructions change, confirm that employees can follow them in a realistic exercise. A completed action should produce evidence of improvement.

Keep uncertainty visible. Some incidents may not have enough records to establish one definitive cause. State what is known, what remains unresolved, and which safeguards are justified despite that uncertainty. Avoid presenting a symptom-level workaround as proof that the underlying condition has been removed.

Use Holism's model for accountable learning

Holism combines AI, a human developer, and a Software Product Line of reusable capabilities and consistent rules. The developer can examine the relationship between a local symptom and rules shared across the product. AI supports investigation and implementation, while professional judgment evaluates the proposed correction.

The owner contributes operating context, deadlines, and accepted working practices. Together, the parties can prioritize changes that make responsible work easier. A reusable foundation may help apply a corrected rule consistently, but the result still needs checking against the actual incident and relevant workflows.

A fair incident review preserves accountability while widening the investigation beyond the most visible action. Evidence, contributing conditions, and verified improvements make the business less dependent on perfect individual behavior and better equipped to learn from a failure.