GARP's Risk and AI (RAI) Certificate

Registration opened today. Some quick notes and my initial impression.

Today GARP opened registration for their new Risk and AI (RAI) Certificate. I’ve downloaded the Candidate Guide¹. As you may know, I founded Bionic Turtle² and was its lead instructor; I taught the entire Financial Risk Manager (FRM) curriculum for over a decade and a half. I will admit straightaway that I was highly skeptical when I first noticed GARP was going to develop this certificate. There were two reasons for my skepticism, one general and one specific. First, I’m unconvinced that any multiple-choice (aka, item response theory³) exam structure can effectively support learning or mastering artificial intelligence⁴. Second, this particular association doesn’t appear to have the expertise in-house to lead on the emergent, cross-disciplinary domain of artificial intelligence (AI)—which is necessarily technical—although it certainly has a reputation in the traditional domain of risk management. I just want to be candid about my bias.

The exam contains five modules (the weights are ranges; the number in parenthesis is merely the expected number of questions out of 80):

• AI and Risk: Overview (8 questions, on average)

• Tools and techniques (24)

• Risks and risk factors (16)

• Responsible and ethical AI (16)

• Data and AI model governance (16)

The 12-page Candidate Guide gives a brief description of each module. At the moment, apparently, you have to register (at a cost of $625 to $750⁵; minus $100 for early registration) in order to receive the “complementary” Study Guide/Learning Objectives (LOs) and a single RAI Practice Exam. It’s very difficult for me to evaluate until I see the Study Guide and LOs. Frankly, you should find a copy before you decide (that’s just my advice). The fact that you only get a single practice exam on the exam’s first cycle is totally expected if you know anything about this association: over the last decade, GARP provided very few practice exams for the FRM⁶. These exam-based certifications are a big spend of money and time, so what’s the time commitment? Says GARP, “Exam preparation is estimated at 100-130 hours. This will vary based on prior professional experience, academic background, and familiarity with the concepts covered in the curriculum.”⁷

Here’s my super-quick take on the modules (aside from the overview):

Tools and techniques. it’s a typical list of supervised learning, unsupervised learning, and reinforcement learning. To me, this technique list resembles the machine learning material in the P1.T2 FRM; e.g., regressions, decision trees, k-nearest neighbors, support vector machines. I do not see deep learning listed. I notice this list includes the broader natural language processing (NLP) rather than large language models (LLMs).

Risks and risk factors. This should be GARP’s strength and, indeed, this looks like a good starter list, including: algorithmic bias and fairness; explainability and interpretability; autonomy and manipulation; safety; reputational risk; existential risk; and global trends. At first glance, missing are privacy and security (but maybe they are tucked under the governance?).

Responsible and ethical AI. This module “looks at various ethical frameworks that can be applied to AI, the governance challenges associated with AI, and the current AI governance landscape.”⁸ Based on the FRM’s terribly thin ethical frameworks (after 20 years, the Code of Conduct is a scant 2 1/2 pages, last I checked; in sharp contrast to the CFAI’s developed ethics framework, the FRM’s ethics is vague generalities), and the high degree of difficulty, my bet is that this will be the weakest module.

Data and AI model governance. Until we see the readings⁹ + LOs, it’s not enough to render a judgment. We’re told to expect “a starting point to establish a firm-specific model validation framework for AI/ML. While the principles presented apply to a wide range of industries, the primary focus is on the financial sector, and specifically quantitative risk models (QRMs), due to the established formal regulatory framework around QRMs.”¹⁰ This could be great or this could be lame. One challenge (inherent in any certification exam approach) is that AI governance is so emergent and dynamic that today’s tentative framework might be quasi-obsolete next year. Another is the selection problem: there are currently many competing frameworks in various stages of development. What strike me is the reference (emphasis mine) to “established formal regulatory framework around QRM.” To who exactly does that refer? On the one hand, this is risk, so “established” might refer to Basel (BCBS), Solvency II (in the EU), Fed/OCC supervisory guidance, etc. However, to my knowledge these are model-based and haven’t gone very far with respect to artificial intelligence. So I don’t think it means those that are studied in the FRM. If the emphasis is regulatory approaches to AI, then my first thought is the NIST AI risk management framework. There are others! But really it’s more the case that “due to its novelty and ongoing transformation, defined guidelines around AI are still developing”¹¹. For example, the EU is widely considered to be more aggressive but, coincidentally, Europe’s AI Act (“the first-ever legal framework on AI”) enters into force literally today (August 1st). So, it’s all moving fast.

Another potential concern (especially in regard to ethical AI and AI model governance) is illustrated by GARP’s current approach to much of Operational Risk (P2.T7) in the FRM. In that topic, too many conceptual frameworks are presented as a way to categorize or organize abstract verbs or nouns. Although sometimes they are useful, too often these consultant-like frameworks are insufficiently realistic (or technical or concrete…) to be actionable.

GARP’s first chapter on Operational Risk features (from top to bottom) a risk management cycle, an operational risk management framework (ORMF), and a set of operational resilience building blocks. Source—2023 FRM P2.T7.Operational Risk, Chapter 1, Figures 1.1, 1.2, and 1.3.

My advice: on the one hand, it's only 100+ hours of study. But it’s a significant cost, and there is always the possibility that you pay money, spend time and then don’t pass the exam. I definitely wouldn’t signup until I saw the LOs (that’s just me). Also, it’s the exam’s first cycle. In affectionate terms, you could be called an early adopter¹². Yay! You could always wait and read the inevitable feedback on reddit (that’s what I myself would do. Hint, hint). Nobody knows what the RAI is today; in fact, it’s possible some might confuse it with RIA, yes?

Per my prior beliefs, my key problem is that this certificate is that it cannot possibly teach you any real AI because:

1. “Coding or programming is not required to work through the curriculum and pass the Exam”¹³. You can’t really gain any mastery until you engage with at least some code. Everybody should learn git/github and get minimum exposure to code.

2. Related, given the time input, there probably cannot be much math included: The FAQ includes a a query about math prerequisites. The reply includes, “while the fields of AI and ML can be quantitative, the RAI program is designed to be accessible for a wide range of candidates with diverse backgrounds and training … The level of mathematical difficulty is on par with an advanced undergraduate or introductory graduate level finance, statistics, or economics course at most universities.”¹⁴ I certainly accept that for some of the topics (ethics and, to a degree, governance) but those tools and techniques are largely quantitative and code-based. I mean, if you really want to begin to understand them.

On the other hand, I will admit that the long list of Advisors and Content Developers is impressive. You have got to give GARP credit here: few organizations could recruit such a list of established thought leaders and influencers. Not only does it include practitioners with real-world responsibilities (e.g., Chief Model Risk Officer, Head of Quantitative Modeling) but the professors on the list are world-class (at least four are household names in quantitative finance). We can’t infer their degree of involvement (sometimes marquee names are attached to a project rather than deeply engaged in a project. Do you know what I mean?), but if any group could generate the syllabus, it’s this group.

I hope that’s a helpful quick take.

1. I would share but I don’t have (and didn’t ask for) permission, but it’s easy to download yourself if you give them your email.

2. I sold to private equity (CeriFi, a Leeds company) and stayed on for three years after selling

3. I realize item set theory, IRT, is broader than multiple choice question exams but I’m just being associative. My experience with IRT was always multiple choice questions. Last year I wrong a long post on The art of writing a great practice question.

4. To support my argument, I’d enter into evidence Exhibit A: the Financial Data Professional which was so disrupted by AI that they had to literally pause the exam to regroup. Would you sit for this exam, in the last cohort, before it gets totally redesigned? I wouldn’t. What’s the signaling value of a FDP certificate that represents the disrupted syllabus? The new FDP will be developed by Alpha Development. I’m not criticizing Alpha Development, of course: I hope they solve the problem. I’m just saying that anything AI is not naturally amenable to clunky development and testing cycle of traditional exams like these.

5. Subtract $100 for early registration. Therefore, the lowest price to register (available to Certified FRM/ERPs) is $525 but others will pay at least $550 for early registration

6. In point of fact, as detailed in my memo How to Fix the FRM, there appeared to be several FRM Practice Exams, but many of the questions were simply bug fixes of prior versions. Over 15 years, GARP published at most 15 distinct Practice Exams (as an extremely generous estimate). There was a recent 4-year stretch where basically two exams were recycled. Those candidates just did not get a ton of exam-level practice questions. That’s was a big factor in our success: candidates needed to purchase questions for some prep provider, and that was our competitive differentiator.

7. 2023-2025 RAI Candidate Guide, GARP. Page 8

8. 2023-2025 RAI Candidate Guide, GARP. Page 6

9. A key question is: are the readings developed for the RIA, or are they chapter licensed from textbooks. As far as I can tell, the candidate guide does not say which. The FRM, for example, is currently a mix: P1 was brought in-house a few years ago, but before that, the curriculum was (almost) entirely assigned chapters from licensed textbooks. This is an inexpensive way to develop a curriculum but it has drawback. Each approach has pros and cons. The “problem” with an in-house curriculum on the first cycle is that paying customers are willing (or unwilling) beta testers, any way you look at it.

10. 2023-2025 RAI Candidate Guide, GARP. Page 6

11. Global Relay https://www.globalrelay.com/resources/blog/how-are-financial-regulators-approaching-ai-integration/

12. Notice I did not write guinea pig :)

13. 023-2025 RAI Candidate Guide, GARP. FAQ. Page 10

14. Ibid