The Gold Rush of AI in 2024 as depicted by AI (image:MidJourney
It is easily observable in the Gold rush of the new AI industry: companies are focused on grabbing a very skilled star quarterback (VP of engineering or Director of AI/ML). To win the game, you need to know the style of game you are going to play (are you a running or passing team?) and what resources you have to carry out your strategy (for example, do you have wide receivers that match your QB’s passing skills?). Apologies for the sports analogy, but we can all blame Jason Gay at the Wall Street Journal for his excellence in relatable sports humor. In short, you need someone (the coach or AI Strategist) who can tie together the internal assets and external challenges and opportunities (SWOT) and create a winning strategy. This is your opportunity to impart some reflection in this white hot industry.
4 Benefits of an AI Strategist (image:MidJourney)
First, it is not your fault. Life has some basic truths, and two are very important and replicated at home and in business. First, the “arrow of time” moves towards less useful energy. In the big picture, black holes vacuum organized things and radiate out Hawking radiation and that may go on until the “heat death” of the universe (energy is conserved but it is cold and there is nothing unique) [1]. In that journey, clumps of organized and useful energy emerge and regions of decay and useless energy emerge. The human pursuit of progress is to harness this truth and bring organized and useful energy (order) into our life and work.
The March of the Arrow of Time (image:MidJourney)
Second, there is no perpetual motion machine, and it always takes more energy than you are left with to make something more organized (think heat from a light bulb wasted while light is generated). You can organize something in a specific place and time (bounded box) and are always left with increased waste outside the box (in your life, work etc..). At work, focus on quality may lead to lower volume output from your factory, or focus on lowering cost with a global outsourcing solution may result in lower quality and a worse experience for your customer.
Your hard fought projects create “useful energy”, but be prepared to handle the “useless energy” that will be created.
While these laws of nature and life swirl around us, we can clean our bathrooms, create new products and constantly optimize our business for growth and added value (all daily examples of creating order). Left unattended, the first two rules ensure that a robust, efficient, predictive and valuable machine learning project (and overall business process reengineering plan) is unlikely give you the result you intend without focus and energy greater than what you expect. Which is also why you can not abdicate accountability in an organizational structure. Leaders like Jack Welch (share the values and over deliver), General Petraeus (get the big ideas right, communicate, and harvest best practice) and Admiral Rickover (who invented the modern nuclear navy with 0 incidents in over 50 years of operations) knew this well.
Constant focus and good definition of goals overcomes internal and external challenges. Identify your stars, set vision and communicate and finally ensure you are preventing low occurrence and high risk issues.
Hopefully I have not left you with a sense of existential crisis and feeling of despair (our sun will last materially longer than our interest), but to reinforce how important structure and order are in attending to our activities every day. Consistent and above average effort makes for an excellent outcome. When choosing how to start your AI Enterprise journey, choose a project small enough that you are committed to attending to it every day, can observe its predictive shortcomings, and continuously improving it for three years. This is especially true if your Enterprise AI asset is on the critical path and touching your customers (or their outcome) directly.
AI and specifically Machine Learning (ML) is considered “A Whole New Industry” as described recently in the Wall Street Journal [2]. The leaders that are being recruited often are being asked for very specific computer science and ML skills, and often hands-on coding and neural net expertise. Those quarterbacks are needed on the team and often need to lead others on the field, but what is missing is the coach or strategy professional who can optimize and create order for the team on the field. Mature industries create this division of labor for a reason. The blueprint for division of labor for a winning team in this new industry has not been tested and refined. We are not good at being in a flow state and playing the game at the same time we are analyzing and creating a strategy. So, strongly consider who will be your AI Strategist to support your newly hired star quarterback Generative AI Computer Science PhD. Here are the three disciplines that are needed for successful Enterprise AI:
Enterprise AI requires Business Excellence, Causal Structured Solutions and Dynamic Knowledge of ML Tools and Operations.
1. Business strategy and excellence. The key goal here is not only to give your customers freedom in their affairs [3], but to organize resources (time, people and capital) towards a vision that grows and creates value that is differentiated and useful in the marketplace beyond what others can do. Create a meaningful process that is sustainable, understandable and repeatable that you can oversee and steer and not have to constantly grab the steering wheel during every out-of-control incident requiring corrective and preventative action.
2. Leadership and management. Leadership is setting the right vision that is meaningful in the marketplace (making rain), building other leaders and getting the vision/story to resonate inside and outside the organization.
Leadership (vision) and Management (division of labor) are both required for success.
Developing yourself and how you interact set a foundation for leadership in your project and company.
3. Development of technology and innovating to useful products. Your technology funnel needs to have a phase review process that focuses on the most important needs, allows ideas to progress and risks to be identified and managed in parallel with other necessary development. Innovation should not be confused with invention or improvement. Innovation is delivering technology (application of science) towards an unmet need. Invention is creating a higher order outcome from a set of materials and ideas. Improvement is a linear process of continuous improvement like Six Sigma where the ideal situation can be observed and you improve other situations in your business to mirror the ideal situation (variance reduction).
Innovation is the sweet spot for bringing your product and company to a limitless future.
4. Continuous Improvement for the way business is done. Six Sigma, Lean and Total Quality Management all come from the field of operations research which fundamentally is based on observation and some form of the scientific method, and consistent with Deming’s Plan>Do>Check>Act cycle.
Tools like Kepner Tregoe Potential Problem Analysis, 8D, FMEA etc.. help to speed the rate of learning.
5. Physics and Process based mindset when making your Enterprise AI model.
A well-structured and causal model has more positive impact than monitoring in production. Build quality into the product.
Today, much ML model building takes data and optimization parameters and iterates until there is an outcome that appears reasonable. In almost all cases, this testing is done within itself (even if the data is split), and is interpolative and based on correlation. But if it works, who cares? You care in Enterprise AI because your business depends on that prediction over and over and in a world where the blocking variables are constantly creating new input conditions not previously seen. At worst, these models are alchemy and sometimes giving Gold as an output, and at some point giving useless rocks, and according to Murphy’s Law, at the wrong time (reference the 2 rules of life which are the backbone to Murphy). The examples have been well established, models that choose a Husky over a Wolf because of snow in the background of the picture, or worse AI in Human Resources that chooses resumes by name, gender or sport played in college without our knowledge (i.e. Blake who played squash).
How Machine Learning decides to associate and solve our problem is often not what a human expects or sometimes wants.
We often loose thousands of years of collective knowledge with these training methods. In this example by Steve Brunton [4], a Machine Learning model can be made to explain a ball drop experiment, but never follows the physics we have discovered over centuries. The problem is that the next time a ball drops in your business, you are likely to get the prediction wrong especially as the balls get fuzzier, heavier, smoother etc.. Adding physics based principals to the model (constraints like global sparsity and loss functions related to governing equations) build knowledge back into the model and make them more robust and more usable in your enterprise business (think Fault Tolerant). Steve has a great new video going into much more detail on this topic [5].
Machine Learning can be steered into more robust solutions that include centuries of human learning.
Another good example is the journey Bloomberg went on to build BloombergGPT and requiring a new form of encoding data going into the model so numbers were not chopped up and eroding the ability to provide better financial results from the model[6].
Systems engineering principals directly transfer to Enterprise AI. Systems need to be sensitive to how they are implemented (this is the analogy to strategy needs to be based on assets, or you can’t be a passing team without a wide receiver). An unfortunate and clear example is the MCAS system for airplanes which worked flawlessly in a military application (3 sensors and when 1 fails, you have 2 sensors giving you the correct answer). However, when MCAS was applied to the commercial sector, there were only two sensors and when one failed, you can not tell which sensor is faulty [7].
To drive this example home we can think about driving a car which on the surface seems simple enough that at least 1 billion+ people do it. Building ML to be a general purpose reasoning and decision making machine is complicated (however, computers are way better than humans at staying focused).
1. Just working at the quarterback level, you are training recognition of street signs, cars etc.. to be as accurate as possible. Some quarterbacks will disagree on how best to do this step and either train the specifics or create a reinforcement agent to learn by general association (Human Reinforced Unsupervised Transformer models). Some companies believe in multiple sensors to do that, some rely only on vision to keep it simple and rely on rate of learning and massive data to create appropriate responses.
2. Known issues can often be overlooked and could have been predicted. A recent issue for vision recognition in self-driving cars is a tendency for a car to drive into emergency vehicles at night. After the fact, Occam’s Razor comes into focus: flashing lights from an emergency vehicle confuse (or over-saturate) the image recognition. This hypothesis is still under investigation [8], but code has already been released to help the situation [9]. Having a team that considers a known and often occurring situation like emergency vehicles on the road can lead to an architecture that tests and solves this corner case before Enterprise AI is on the critical path and impacting your Brand’s Goodwill.
3. Regardless, we can already see that the structure of the machine learning is critical in this task. Without a “fail safe” mentality, detecting anything in front of the car will not ensure the car takes every effort to prevent a collision [10] especially if it is confused in categorizing the object in-front of it. With a fail-safe mentality, your algorithm engages the higher-level safety protocol before categorizing an object. Of course, it is not as simple as that example, because if the car has too many false-positives, it will cause other problems like blocking traffic.
In conclusion, an AI strategist looks at your assets, the specifics of the game that will be played, the conditions reasonably expected during the game and builds a strategy for your star quarterback to win the game and opponent you are facing next.
4. Expert knowledge in ML tools and operations is the core of your project and why you have that super-star team of Computer Scientists. The strategy has been created, and the work-breakdown-structure (WBS) has been visualized enough that the team can be given clear and well-defined challenges to make their contribution to the entire Enterprise AI system that will be on the critical path. Here are the 3 pillars that support this development cycle.
Asset management and trackability of your ML Model is necessary as it loops thru the lifecycle in your Enterprise.
1. Asset management is as critical to Machine Learning as it is to the rest of your operations (ERP, capital asset management etc..). A model has a lifecycle from experiments, training, deployment, retraining, new features and optimization. Tracking the provenance of the model back to the experiment and data is a part of this. Additionally, you need to consider that your computer scientists needs to recreate the experiment at a future time. That could be one year or more after the original work was done to address new features, bugs or retraining based on new trends in the market. The results between the old and new experiment needs to be comparable to know if you are making improvements before you replace what you have in production (these are often called the artifacts, which are not the model). ML Operations is a quickly evolving business and there are many companies that are tacking various parts of the pipeline or management from CometML in experiment management, to DVC to track model assets, and Run.ai to more efficiently schedule and run experiments on idle GPUs. You can not rely on or make your computer scientists responsible for managing this enterprise requirement from their desktop. This requires a systems solution that frees the computer scientists to be the best Machine Learning developer every day.
2. As you scale and depending on the scope of your Enterprise AI project, you need to consider efficient operation of your model. Tools like Nvidia’s Triton server offer a solution to batch data, run multiple copies of your model and manage new versions of your model. Your team should also consider connivence vs. operational speed and this part of ML development is also advancing quickly. There are alternatives to Python like Rust and also new entrants like Mojo [11] that may offer quicker execution with a language that is compiled. Even configuration and YAML engineering are getting new tools like Apple’s PKL (Pickle) language [12]. Then there are challenges with having multiple and regional Enterprise AI solutions if your global company has factories across many continents and needs to support both synchronization and regional customization.
3. An ever-complicated discussion for global companies is how to make decisions at the edge (IOT, IT 4.0, AMD, Silicon Labs etc..). There are macro decisions like having your assets in a public or hybrid cloud. For example, AWS has created new solutions like CleanRoom. More interestingly is creating your solution to run in a location that is limited by compute power or data available. These are very exciting use cases because the promise of Machine Learning at the edge can offer decision at the point of differentiation. For example, examining a finished product like a car at the end of the line is expensive, slow and not preventing the issue. So, the desirable continuous improvement activity is to move the inspection and corrective action further upstream where the defect is being created. This is the same opportunity for putting ML (decision making) in locations and products where the activity is happening (think of augmented reality ski goggles that can tell you if you are exceeding your personal limits on the ski slope — probably not far off). Today, companies like Teal Systems are making real time decisions in buildings for improved water and energy management. LatentAI has an even stronger DeepTech solution which is funded by Booz Allen’s VC arm.
In conclusion, with all the tools and solutions moving so quickly for the business of building Machine Learning, you need an AI Strategist who is invariant to programming language, ML Operations library or hosting solution to reflect, update and continuously improve your Enterprise AI solutions.
AI Strategist: progress over chaos, strategy based on internal assets and external opportunities, build quality into the product and prioritize investments.
Go be Awesome!
[1] Veritasium, The Most Misunderstood Concept in Physics (2023), YouTube
[2] Asa Fitch, Nvidia Declares AI a ‘Whole New Industry’ — and Investors Agree (February 22, 2024), Wall Street Journal.
[3] Peter Koestenbaum, Leadership, The Inner Side of Greatness (2002)
[4] Steve Brunton, Discrepancy Modeling with Physics Informed Machine Learning (2023), YouTube
[5] Steve Brunton, Physics Informed Machine Learning: High Level Overview of AI and ML in Science and Engineering (2024), YouTube
[6] Toronto Machine Learning Summit 2023, BloombergGPT: How We Built a 50 Billion Parameter Financial Language Model (2023), YouTube
[7] Steve Mollman, The 737 Max’s most problematic system has a military predecessor that works better (2019), Quartz
[8] Steven Posads, US Department of Transportation Investigation: EA 22–002, (2022)
[9] Tesla, Model X Owners Manual, Emergency Vehicles (2024)
[10] Wikipedia, Death of Elaine Herzberg (2018)
[11] ArjanCodes, Choosing Your Language: Python or Mojo? (2024), YouTube
[12] Theo, Pkl: Apple’s New JSON/YAML Killer (2024), YouTube
Comentarios