How CXOs Can Forestall a Southwest Airways-Type Tech Debacle With IT Oversight

Greater than 100 million People traveled through the December 2022 vacation season. For these taking one in all Southwest Airways’ (SWA) 15,000 canceled flights, it was a depressing time. It seems that severe weather overwhelmed old technology and created a cascading catastrophe that took days to repair. Now comes the aftermath: repairing the substantial injury SWA did to its model.

If you happen to’re a CXO or board director, you’re most likely asking, “Might this occur to us?” I’m afraid the reply is “Sure.” Think about SWA’s public shaming as a wake-up name to evaluate your expertise oversight. On this evaluation, we’ll discover 4 areas: capability risk, mission threat, catastrophe threat, and technical debt, all of which must be reviewed to forestall your small business from present process its personal, very public meltdown.

Capability Threat

Does your trade have busy instances? Possibly it’s vacation journey; midsummer electrical energy surges; periodic peaks (like “late cost day,” the ten^th of the month within the mortgage enterprise); or particular occasions (World Cup is coming to Dallas). Are your techniques and processes sized to deal with these busy instances?

What about when you’re at a peak time, and the flu has 10% of your workforce out sick? Or if a freak climate occasion provides extra complexity to your operations? What when you’re in the course of an acquisition or main system improve when a peak occasion hits?

IT Oversight Tip #1: Perceive trade peaks, and stress check techniques/processes to deal with expectations — plus “headroom.”

Venture Threat

Can your group deal with main system/course of upgrades? It’s comparatively straightforward to plan and finances for a expertise system set up/improve: If it’s a bought expertise product, the seller will hand data expertise (IT) a “pattern mission plan.” However guess what? The “techie stuff” most IT departments concentrate on represents lower than half the work/time/value/threat of a real “get everybody to make use of the brand new processes/expertise instruments successfully (and cease utilizing the previous manner)” mission.

Venture implementation failures run the gamut from “merely” costly (e.g., the ERP that prices double and takes twice so long as budgeted) to existential (e.g., FoxMeyer Drugs bankruptcy).

Why do these failures occur? I may write 10,000 phrases cataloging all of the methods my shoppers have failed. At a excessive stage, what I’ve largely seen throughout 30 years and quite a few industries is that IT departments are underneath stress to ship options cheaply and that enterprise models, additionally underneath stress, abdicate duty for enterprise course of change to IT.

IT Oversight Tip #2: Be sure that main expertise tasks are scoped and resourced as organizational change administration tasks reasonably than as expertise tasks, and that satisfactory contingency prices are a part of the finances. Assign a senior government or skilled advisor as mission proprietor and incentivize the C-suite to stay engaged with the method and the result.

Catastrophe Threat

Each mid-size and bigger group has a “catastrophe plan” that purports to assist the group recuperate from a variety of calamities. Actuality test: Most such plans exist in dusty binders designed to fulfill auditors and received’t work when wanted.

The 4 predominant issues I’ve seen are”

• Lack of creativeness when defining “catastrophe” varieties or scope
  Everybody is aware of about hearth, flood, energy, or community outage. What a few railcar filled with chlorine derailing subsequent to a essential enterprise location? How about 10,000 gallons of sugar syrup cascading down from the roof? (True story: My dad ran a sweet manufacturing facility that saved liquid sugar in a roof tank. When the provider’s pump truck overfilled the tank, extremely corrosive syrup ran down stairwells and inside partitions rendering the manufacturing facility unusable for 2 years whereas it was gutted and rebuilt.) What about scope? One facility is simple to plan for. What a few regional catastrophe (hurricane/earthquake/snowstorm) that impacts your group together with staff’ households plus your regional provide chain? Or a world pandemic, for that matter?
  • Viewing catastrophe planning as an IT drawback
    Disruptions to operations are an organizational drawback, not a departmental or expertise drawback. When a hurricane hits, and staff are attempting to get households to security, they’re not coming to work!
  • Expertise stack adjustments
    Suppose you keep a “catastrophe website” (an outdated notion). You’ll uncover that beginning it up when a catastrophe hits (in the course of the night time on a vacation weekend whereas a storm is raging) won’t go easily as a result of elements will likely be lacking or damaged, and adjustments to your “manufacturing” atmosphere aren’t mirrored at your catastrophe threat (DR) website.
  • Restoration
    Even organizations which are critical about DR typically overlook that transferring again to the first website(s) when the catastrophe ends may be fairly advanced, with the necessity to get knowledge synchronized and techniques operating nominally.

IT Oversight Tip #3: Transfer from a “catastrophe planning” mindset to a “steady distributed availability” mindset, by which any native or regional catastrophe shifts work from affected websites to different websites in unaffected areas. If you happen to can’t undertake that strategy, stress check your catastrophe plans utilizing enterprise outcomes (e.g., course of buyer orders) reasonably than IT objectives, comparable to “Get server ‘x’ operating.”

Technical Debt

The SWA story is filled with references to “technical debt,” which I recently discussed in-depth with Bob Evans. Let’s outline it: Technical debt is the sum whole of upkeep to IT elements ({hardware}, software program, community) that ought to have been carried out however wasn’t. Like several advanced machine — a automobile or a automobile manufacturing facility — your IT “stack” of {hardware} and software program wants upkeep. Bodily gadgets break or put on out; wires fray; buying and selling companions change their techniques in ways in which have to be accommodated; hackers uncover weaknesses that require patching; distributors replace their software program to repair bugs and add options.

It may be tempting to defer IT upkeep to ship new capabilities, which generate goodwill and bonuses for the IT workforce. In spite of everything, no one will get praised for updating stuff that appears to be working — particularly when a few of these updates trigger issues for workers and clients.
However, whereas it’s OK to handle the IT upkeep mission backlog to attenuate disruptions and favor cool new options, deferring upkeep for months or years has dire penalties, because the world not too long ago witnessed with Southwest Airways.

Mounting technical debt weakens the IT stack slowly, however the manifestation of that weak spot occurs all of the sudden and sometimes when the group is pressured by added quantity (See “Capability Threat” above), by new techniques being applied (See “Venture Threat”), or by exterior pressures (See “Catastrophe Threat”)—i.e., when the group is most weak, and sources are stretched to the restrict.

The most important subject with technical debt is that it’s often undocumented and unmanaged: IT hardly ever retains an inventory of upkeep that isn’t being carried out; neither GAAP (usually accepted accounting rules) nor IFRS (Worldwide Finance Reporting Requirements) require any documentation; enterprise leaders haven’t been educated to ask about it. No person is incented to repair it. I consider technical debt as an “off-balance-sheet legal responsibility” that may must be repaid sometime.

IT Oversight Tip #4: Enterprise executives should contemplate technical debt when making enterprise selections, together with mergers-and-acquisitions selections. Tasks have to be budgeted primarily based on “whole lifecycle prices” reasonably than up-front implementation prices alone (CFOs, I’m taking a look at you). CIOs have to be courageous sufficient to withstand stress to “do extra with much less” when budgeting tasks, particularly when digging out of a technical debt gap (What’s step one when filling in a gap? Cease digging!).

And somebody with the proper connections, please persuade FASB (Monetary Accounting Requirements Board) and PCAOB (Public Firm Accounting Oversight Board) to place technical debt on companies’ steadiness sheets so traders can acquire a clearer image of the ever-increasing threat companies face as automation turns into extra pervasive!

Last Ideas

What does all this imply to a corporation’s prime administration and governance executives?

• Perceive all your expertise dangers, not simply cybersecurity
• Perceive your technical debt, and the way it aligns along with your threat tolerance
• Recruit a professional expertise professional (QTE) director to the board to strengthen oversight of expertise dangers and alternatives
  

---

In search of extra insights into all issues knowledge? Subscribe to the Knowledge Modernization channel: