Dec 12, 2012

What's your Risk Intelligence Quotient?

One of the main problems in risk management is that we (oblivious) overestimate our risk knowledge.

Example
If for example financial institutional boards have to define a risk-return strategy, they may overestimate the probability that the historic return level of a certain asset class will also be the expected future return level.
Or they might simply overestimate the quality of their investment advisor.... ;-)

To define an optimal asset mix on basis of a risk-return strategy, it takes more than just estimating future returns and/or risks of certain asset classes.

To make these kind of high-impact decisions it's important to train board members on knowledge of economic schools and theories and also on the relationship between economic developments and financial parameters like unemployment, inflation, GDP-growth, specific asset class return and risk parameters, linear and non-linear effects, and so on......

But more than that, it's important that board members - as they have learned all this - become aware of the fact that the more they know about risk and uncertainty, the more they'll realize that the outcome or certainty of a future development is intrinsically highly unsure. This last recognition will have significant consequences for the final choice with regard to the optimal asset mix given the risk appetite.

Risk Intelligence Test 
Eventually it all comes down to
Kowing how much you know
as Dylan Evans, author of the book "Risk Intelligence"states in the Dylan Ratigan Show

According to Dylan : 'Risk intelligence is not about solving probability puzzles; it is about how to make decisions when your knowledge is uncertain.'

Dylan Evans developed a short (5 min) Risk Intelligence Test.
See, if you can pas the test as an actuary or risk manager...
You can take the test here.
The test is also available in Dutch.



Links:
- Homepage Dylan Evans
- Dylan Evans on Twitter
- Dutch Risk Intelligence Test
- Dylan Evans: Emotional Equations (Pdf)

Dec 3, 2012

Solvency II or Basel III ? Model Fallacy

Managing investment models - ALM models in particular - is a professional art. One of the most tricky risk management fallacies when dealing with these models, is that they are being used for identifying so called 'bad scenarios', which are then being 'hedged away'.

To illustrate what is happening, join me in a short every day ALM thought experiment...

Before that, I must warn you... this is going to be a long, technical, but hopefully interesting Blog. I'll try to keep the discussion on 'high school level'. Stay with me, Ill promise: it actuarially pays out in the end!

ALM Thought Experiment
  • Testing the asset Mix
    Suppose the board of our Insurance Company or Pension Fund is testing the current strategic asset mix with help of an ALM model in order to find out more about the future risk characteristics of the chosen portfolio.
     
  • Simulation
    The ALM model runs a 'thousands of scenarios simulation', to find out under which conditions and in which scenarios the 'required return' is met and to test if results are in line with the defined risk appetite.
     
  • Quantum Asset Return Space
    In order to stay as close to reality as possible, let's assume that the 'Quantum Asset Return Space' in which the asset mix has to deliver its required returns for a fixed chosen duration horizon N, consists of: 
    1. 999,900 scenarios with Positive Outcomes ( POs ),
      where the asset returns weigh up to the required return) and 
    2. 100 scenarios with Negative Outcomes ( NOs ),
      where the asset returns fail to weigh up to the required return.
       
    Choose 'N' virtual anywhere between 1 (fraction) of a year up to 50 years, in line with your liability duration.
     

  • Confidence (Base) Rate
    From the above example, we may conclude that the N-year confidence base rate of a positive scenario outcome (in short: assets meet liabilities) in reality is 99.99% and the N-year probability of a company default due to a lack of asset returns in reality is 0.01%.
     
  • From Quantum Space to Reality
    As the strategic asset mix 'performs' in a quantum reality, nobody - no board member or expert - can tell which of the quantum ('potential') scenarios will come true in the next N years or (even) what the exact individual quantum scenarios are.

    Nevertheless, these quantum scenarios all exist in "Quantum Asset Return Space" (QARS) and only one of those quantum scenarios will finally turn out as the one and only 'return reality'.

    Which one...(?), we can only tell after the scenario has manifested itself after N years.
     
  • Defining the ALM Model
    Now we start defining our ALM Model. As any model, our ALM model is an approach of reality (or more specific: the above defined 'quantum reality') in which we are forced to make simplifications, like: defining an 'average return', defining 'risk' as standard deviation, defining a 'normal' or other type of model as basis for drawing 'scenarios' for our ALM's simulation process.
    Therefore our ALM Model is and cannot be perfect.

    Now, because of the fact that our model isn't perfect, let's assume that our 'high quality' ALM Model has an overall Error Rate of 1% (ER=1%), more specific simplified defined as:
    1. The model generates Negative Scenario Outcomes (NSOs) (= required return not met) with an error rate of 1%. In other words: in 1% of the cases, the model generates a positive outcome scenario when it should have generated a negative outcome scenario
       
    2. The model generates Positive Scenario Outcomes (PSOs) (= required return met) with an error rate of 1%. In other words: in 1% of the cases, the model generates a negative outcome scenario when it should have generated a positive outcome scenario
       

The Key Question!
Now that we've set the our ALM model, we run it in a simulation with no matter how much runs. Here is the visual outcome:


As you may notice, the resulting ALM graph tells us more than a billion numbers....At once it's clear that one of the scenarios (the blue one) has a very negative unwanted outcome.
The investment advisor suggests to 'hedge this scenario away'. You as an actuary raise the key question:

What is the probability that a Negative Outcome (NO) scenario in the ALM model is indeed truly a negative outcome and not a false outcome due to the fact that the model is not perfect?

With this question, you hit the nail (right) on the head...
Do you know the answer? Is it 99% exactly, more or less?

Before reading further, try to answer the question and do not cheat by scrolling down.....

To help you prevent reading further by accident, I have inserted a pointful youtube movie:



Answer 
Now here is the answer: The probability that any of the NOs (Negative Outcomes) in the ALM study - and not only the very negative blue one - is a truly a NO and not a PO (Positive Outcome) and therefore false NO, is - fasten your seat belts  - 0.98%! (no misspelling here!)

Warning
So there's a 99.02% (=100%-0.98%) probability that any Negative Outcome from our model is totally wrong, Therefore one must be very cautious and careful with drawing conclusions and formulating risk management actions upon negative scenarios from ALM models in general.

Explanation
Here's the short Excel-like explanation, which is based on Bayes' Theorem.
You can download the Excel spreadsheet here.


There is MORE!
Now you might argue that the low probability (0.98%) of finding true Negative Outcomes is due to the high (99,99%) Positive Outcome rate and that 99,99% is unrealistic much higher than - for instance - the Basel III confidence level of 99,9%. Well..., you're absolutely right. As high positive outcome rates correspond one to one with high confidence levels, here are the results for other positive outcome rates that equal certain well known (future) standard confidence levels (N := 1 year):


What can we conclude from this graph?
If the relative part of positive outcomes and therefore the confidence levels rise, the probability that an identified Negative Output Scenario is true, decreases dramatically fast to zero. To put it in other words:

At high confidence levels (ALM) models can not identify negative scenarios anymore!!!


Higher Error Rates
Now keep in mind we calculated all this still with a high quality error rate of 1%. What about higher model error rates. Here's the outcome:


As expected, at higher error rates, the situation of non detectable negative scenarios gets worse as the model error rate increases......

U.S. Pension Funds
The 50% Confidence Level is added, because a lot of U.S. pension funds are in this confidence area. In this case we find - more or less regardless of the model error rate level - a substantial probability ( 80% - 90%) of finding true negative outcome scenarios. Problem here is, it's useless to define actions on individual negative scenarios. First priority should be to restructure and cut ambition in the current pension agreement, in order to realize a higher confidence level. It's useless to mop the kitchen when your house is flooded with water.....

Model Error Rate Determination
One might argue that the approach in this blog is too theoretical as it's impossible to determine the exact (future) error rate of a model. Yes, it's true that the exact model error rate is hard to determine. However, with help of backtesting the magnitude of the model error rate can be roughly estimated and that's good enough for drawing relevant conclusions.

A General Detectability Equation
The general equation for calculating the Detectability (Rate) of Negative Outcome Scenarios (DNOS) given the model error rate (ER)  and a trusted Confidence Level (CL) is:

DNOS = (1-ER) (1- CL) / ( 1- CL + 2 ER CL -ER )

Example
So a model error rate of 1%, combined with Basel III confidence level of 99.9% results in a low 9.02% [ =(1-0.01)*(1-0.999)/(1-0.999+2*0.01*0.999-0.01) ] detectability of Negative Outcome scenarios.

Detectability Rates
Here's a more complete oversight of detectability rates:


It would take (impossible?) super high quality model error rates of 0.1% or lower to regain detectability power in our (ALM) models, as is shown in the next table:



Required  Model Confidence Level
If we define the Model Confidence Level as MCL = 1 - MER, the rate of Detectability of Negative Outcome Scenarios as DR= Detectability Rate = DNOS and the CL as CL=Positive Outcome Scenarios' Confidence Level, we can calculate an visualize the required  Model Confidence Levels (MCL) as follows:

From this graph it's at a glance clear that already modest Confidence Levels (>90%) in combination with a modest Detectability Rate of 90%, leads to unrealistic required Model Confidence Rates of around 99% or more. Let's not discuss the required Model Confidence Rates for Solvency II and/or Basel II/III.

Conclusions
  1. Current models lose power
    Due to the effect that (ALM) models are limited (model error rates 1%-5%) and confidence levels are increasing (above > 99%) because of more severe regulation, models significantly lose power an therefore become useless in detecting true negative outcome scenarios in a simulation. This implies that models lose their significance with respect to adequate risk management, because it's impossible to detect whether any negative outcome scenario is realistic.
     
  2. Current models not Solvency II and Basel II/III proof
    From (1) we can conclude in general that - despite our sound methods -our models probably are not Solvency II and Basel II/III proof. First action to take, is to get sight on the error rate of our models in high confidence environments...
     
  3. New models?
    The alternative and challenge for actuaries and investment modelers is to develop new models with substantial lower model error rates (< 0.1%).

    Key Question: Is that possible?

    If you are inclined to think it is, please keep in mind that human beings have an error rate of 1% and computer programs have an error rate of about 3%.......
     

Links & Sources:

Nov 24, 2012

Dying Age Quiz

Ever heard of Club 27? It turns out that famous pop artists have a preferred age of dying: 27.

Among this 'club', with around 50 unlucky 'members' that all died at the age of 27, are well known names like Brian Jones, Jimi Hendrix, Janis Joplin, Jim Morrison and (lately, 2011) Amy Winehouse.

There's been a lot of (actuarial) discussion whether this club 27 phenomenon is a mortality anomaly or not.

In a statistical study from BMJ (British Journal of Medicine) called "Is 27 really a dangerous age for famous musicians? Retrospective cohort study", it's shown that  there's no peak in the risk of death for famous musicians at age 27.


Club 27, or its movie,  is therefore a nice opportunity to study some interesting artists who died young, but not based on any statistical relevance.

Quiz
Not only some top musicians died young, but also some 'historical' celebrities.

Now take the next quiz to test your knowledge on the dying age of the next famous people who changed the world, each on in his/her own way:





Links and sources:
- BMJ Statistical Study
- Dying Age Quiz of Famous People
Death, Actuarial Science and Rock n’ Roll-the 27 Club