Programming with Data

class: center, middle, inverse, title-slide

# Programming with Data
## Session 9: LASSO and Tree Ensembles for Fraud Detection
### Dr. Wang Jiwei
### Master of Professional Accounting

---

class: inverse, center, middle

# Corporate/Securities Fraud

---

## Typical accounting fraud

.pull-left[
- A company is underperforming, motivated to inflate earnings or reduce expenses
    - [Worldcom (1999-2002)](https://www.sec.gov/Archives/edgar/data/723527/000093176303001862/dex991.htm)
        - Capitalizing costs (should be expensed)
    - [Olympus (late 1980s-2011)](https://en.wikipedia.org/wiki/Olympus_scandal): Hide losses in a separate entity
        - "Tobashi scheme"
    - [Wells Fargo (2011-2018?)](https://en.wikipedia.org/wiki/Wells_Fargo_account_fraud_scandal)
        - Fake/duplicate customers and transactions
        - "improperly encouraging" to actively trade
]

.pull-right[
- A company is overperforming, motivated to save for future rainy days (smoothing of earning)
    - [Dell (2002-2007)](https://www.economist.com/newsbook/2010/07/23/taking-away-dells-cookie-jar)
        - Cookie jar reserves
    - [Brystol-Myers Squibb (2000-2001)](https://www.sec.gov/news/press/2004-105.htm)
        - Cookie jar reserves
]

---

## Other accounting fraud types

- [Apple (2001)](https://www.sec.gov/news/press/2007/2007-70.htm)
    - *Options backdating*
- [Commerce Group Corp (2003)](https://www.sec.gov/litigation/admin/2007/34-56403.pdf)
    - Using an auditor that [*isn't registered*](https://www.sec.gov/news/press/2007/2007-183.htm)
- [Cardiff International (2017)](https://www.sec.gov/litigation/admin/2018/34-84258.pdf)
    - Releasing 10Q fin. statements that were *not reviewed by an auditor*
- [China North East Petroleum](https://www.sec.gov/litigation/litreleases/2012/lr22552.htm)
    - *Related party transactions* (transferring funds to family members)
- *Insufficient internal controls*
    - [Citigroup (2008-2014)](https://www.sec.gov/litigation/admin/2018/34-83858.pdf) via Banamex
    - [Asia Pacific Breweries](http://eresources.nlb.gov.sg/infopedia/articles/SIP_422_2005-01-25.html)
- *Bribery*
    - [Keppel O&M (2001-2014)](https://mothership.sg/2018/01/what-is-the-keppel-corruption-scandal-all-about/): US$55M to Brazilian officials
    - Baker Hughes ([2001](https://www.sec.gov/litigation/admin/34-44784.htm), [2007](https://www.nytimes.com/2007/04/27/business/worldbusiness/27settle.html): Payments to officials in Indonesia, and possibly to Brazil/India (2001), Angola/Indonesia/Nigeria/Russia/Uzbekistan (2007)
- [ZZZZ Best (1982-1987)](https://www.nytimes.com/1990/02/25/books/nothing-but-zzzz-best.html): *Fake the whole company*
    - Also faked a real project to get a clean audit to take the company public

---

## More interesting fraud types

- [Bernard Madoff](https://www.nytimes.com/2009/01/25/business/25bernie.html): Ponzi scheme
    1. Get money from individuals for "investments"
    2. Pretend as though the money was invested
    3. Use new investors' money to pay back anyone withdrawing
- [Applied Wellness Corporation (2008)](https://www.sec.gov/litigation/admin/2010/34-61344a.pdf)
    - Failed to file annual and quarterly reports
- [Tesla (2018)](https://www.sec.gov/news/press-release/2018-219): Misleading statements on Twitter
    - [Settled the charge in one week](https://www.sec.gov/news/press-release/2018-226)
- [Am-Pac International (1997)](https://www.sec.gov/litigation/litreleases/lr17024.htm)
    - Auditors lacked independence: maintain books, generate FS, then audit their own work
- [Keppel Club (2014)](https://www.straitstimes.com/singapore/courts-crime/keppel-club-duo-convicted-for-37m-membership-scam)
    - Employees created 1,280 fake memberships, sold them, and retained all profits (SG$37.5M)

---

## What will we look at today?

> Misstatements: non-conmpliance with accounting standards, maybe an unintentional error but in many cases seemingly intentionally done by management or other employees at the firm (ie, fraud).

How do misstatements come to light?

1. The company/management admits to it publicly
2. A government entity forces the company to disclose
    - In more egregious cases, government agencies may disclose the fraud publicly as well
3. Investors sue the firm, forcing disclosure

---

## Where are these disclosed? (US)

By the company:

1. [10-K/A filings](https://www.sec.gov/forms) (10-K: annual report, /A: amendment)
    - [Big R](https://assets.ey.com/content/dam/ey-sites/ey-com/en_us/topics/assurance/accountinglink/ey-frdbb2752-05-11-2020-v2.pdf): material corrections to previously issued financial statements
    - Note: not all 10-K/A filings are caused by fraud!
        - [Audit Analytic](https://www.auditanalytics.com/blog/reasons-for-an-amended-10-k-2017/)'s write-up on this
2. In a note inside a 10-K filing
    - [Little r](https://assets.ey.com/content/dam/ey-sites/ey-com/en_us/topics/assurance/accountinglink/ey-frdbb2752-05-11-2020-v2.pdf): immaterial corrections
3. In a press release, which is later filed with the US SEC as an 8-K
    - 8-Ks are filed for many other reasons too though

By the regulator:

4. By the SEC through the [Section 13(b)](https://www.sec.gov/spotlight/fcpa/fcpa-recordkeeping.pdf)
    -  Section 13(b) of the Securities Exchange Act, commonly called the “books and records” provision, requires issuers to “make and keep books, records, and accounts, which, in reasonable detail, accurately and fairly reflect the transactions and dispositions of the assets of the issuer."
5. [SEC AAERs](https://www.sec.gov/divisions/enforce/friactions.shtml): Accounting and Auditing Enforcement Releases
    - Generally highlight larger or more important cases

---

## AAERs

- Today we will examine these AAERs
    - Using a proprietary data set of > 1,000 such releases
    - The data is [available here](https://sites.google.com/usc.edu/aaerdataset/home)

> To get a sense of the data we're working with, read the *Summary* section (starting on page 2) of this AAER against Sanofi [SEC against Sanofi](https://www.sec.gov/litigation/admin/2018/34-84017.pdf)

---
class: middle, inverse, center

# Predicting Fraud

---

## Main question

> How can we *detect* if a firm *is* involved in a major instance of missreporting?

- This is a pure forensic analytics question
- "Major instance of misreporting" will be implemented using AAERs

> Unfortunately we don't have an "Altman Z" type of measure from the academia yet, partly due to the complexity of fraudulent activities.

---

## Approaches

- In these slides, we'll walk through the primary detection methods since the 1990s, up to currently used methods
- 1990s: Financials and financial ratios
    - Follow up in 2011
- Late 2000s/early 2010s: Characteristics of firm's disclosures
- mid 2010s: More holistic text-based measures of disclosures
    - This will tie to a future topic where we will explore how to work with text

> All of these are discussed in Section 2.1 of <a target="_blank" href="https://doi.org/10.1111/1475-679X.12294">Brown, Crowley and Elliott (2020)</a> -- we'll refer to the research paper as **BCE** for short

---

## The data

- Preprocessed data from **BCE(2020)**
- It contains 401 variables from Compustat, CRSP, and SEC
    - Many precalculated measures including:
        - Firm characteristics, such as auditor type (**bigNaudit**)
        - Financial measures, such as total accruals (**rsst_acc**)
        - Financial ratios, such as ROA (**ni_at**)
        - AR characteristics, such as mean sentence length (**sentlen_u**)
        - ML based content analysis (everything with **Topic_** prepended)
- Testing data (2004) and training data (1999-2003) by variable **Test**
    - Testing == 1: testing data
    - Testing == 0: training data

> Reminder: data provided in the course are mainly through the WRDS subscription. You are prohibited by law from sharing the data with people outside of SMU community.

---

## Event frequency

- Very low event frequencies (<10%) can make things tricky

```r
df %>%
  group_by(year) %>%
  mutate(total_AAERS = sum(AAER), total_observations = n(),
         percent = scales::percent(total_AAERS/total_observations,
                                   accuracy = 0.01)) %>%
  slice(1) %>%  ungroup() %>%
  select(year, total_AAERS, total_observations, percent) %>% html_df
```

<table class="table table-striped table-hover" style="width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> year </th>
   <th style="text-align:center;"> total_AAERS </th>
   <th style="text-align:center;"> total_observations </th>
   <th style="text-align:center;"> percent </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 1999 </td>
   <td style="text-align:center;"> 46 </td>
   <td style="text-align:center;"> 2195 </td>
   <td style="text-align:center;"> 2.10% </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2000 </td>
   <td style="text-align:center;"> 50 </td>
   <td style="text-align:center;"> 2041 </td>
   <td style="text-align:center;"> 2.45% </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2001 </td>
   <td style="text-align:center;"> 43 </td>
   <td style="text-align:center;"> 2021 </td>
   <td style="text-align:center;"> 2.13% </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2002 </td>
   <td style="text-align:center;"> 50 </td>
   <td style="text-align:center;"> 2391 </td>
   <td style="text-align:center;"> 2.09% </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2003 </td>
   <td style="text-align:center;"> 57 </td>
   <td style="text-align:center;"> 2936 </td>
   <td style="text-align:center;"> 1.94% </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2004 </td>
   <td style="text-align:center;"> 49 </td>
   <td style="text-align:center;"> 2843 </td>
   <td style="text-align:center;"> 1.72% </td>
  </tr>
</tbody>
</table>

---

## Dealing with imbalanced data

- imbalanced dataset: the number of obs belonging to one class is significantly lower than those belonging to the other classes.
- A few ways to handling imblanced dataset:
    * (1). Resampling to a balanced dataset (packages incl. `package:imbalance`, `package:ROSE`, `package:DMwR`, etc)

.center[<img src="../../../Figures/resampling.png" width="800px">]

---

## Oversampling by SMOTE

- Not good to use undersampling alone as it reduces data points.
- <a target="_blank" href="https://arxiv.org/abs/1106.1813">SMOTE: Synthetic Minority Over-sampling Technique</a>
    - Combine oversampling minority and undersampling majority
    - Introducing synthetic examples along the line joining all of the **k nearest neighbors** of minority class.
    - `package:DMwR` and `package:smotefamily` have the *SMOTE()* function
    
.center[<img src="../../../Figures/smote.png" width="800px" height="300px">]

---

## Dealing with imbalanced data

- imbalanced dataset: the number of obs belonging to one class is significantly lower than those belonging to the other classes.
- A few ways to handling imblanced dataset:
    * (2). Algorithmic ensemble techniques such as bagging (eg, Random Forest) and boosting (eg, XGBoost)
        - we will try the ensembling techniques
        
.center[<img src="../../../Figures/bagging.png" width="800px">]

---
class: inverse, middle, center

# 1990s approach

---

## The 1990s model

- Many financial measures and ratios can help to predict fraud

.pull-left[
- EBIT
- Earnings / revenue
- ROA
- Log of liabilities
- liabilities / equity
- liabilities / assets
- quick ratio
- Working capital / assets
- Inventory / revenue
- inventory / assets
- earnings / PP&E
- A/R / revenue
]
.pull-right[
- Change in revenue
- Change in A/R + 1
- `$>10\%$` change in A/R
- Change in gross profit + 1
- `$>10\%$` change in gross profit
- Gross profit / assets
- Revenue minus gross profit
- Cash / assets
- Log of assets
- PP&E / assets
- Working capital
]

- Misreporting firms' financials should be different from expected
    - odd financials

---

## Approach

```r
fit_1990s <- glm(AAER ~ ebit + ni_revt + ni_at + log_lt + ltl_at + lt_seq +
                   lt_at + act_lct + aq_lct + wcap_at + invt_revt + invt_at +
                   ni_ppent + rect_revt + revt_at + d_revt + b_rect + b_rect +
                   r_gp + b_gp + gp_at + revt_m_gp + ch_at + log_at +
                   ppent_at + wcap,
                 data = df[df$Test == 0, ], #Test=0 is the training data
                 family = binomial)
tidy(fit_1990s)
```

```
## # A tibble: 26 x 5
##    term          estimate std.error statistic      p.value
##    <chr>            <dbl>     <dbl>     <dbl>        <dbl>
##  1 (Intercept) -4.66       0.834      -5.59   0.0000000226
##  2 ebit        -0.000356   0.000109   -3.26   0.00112     
##  3 ni_revt      0.0366     0.0306      1.20   0.231       
##  4 ni_at       -0.320      0.233      -1.37   0.169       
##  5 log_lt       0.149      0.341       0.438  0.661       
##  6 ltl_at      -0.231      0.707      -0.326  0.744       
##  7 lt_seq      -0.0000283  0.000457   -0.0619 0.951       
##  8 lt_at       -0.856      0.927      -0.923  0.356       
##  9 act_lct      0.140      0.0701      2.00   0.0455      
## 10 aq_lct      -0.175      0.0916     -1.91   0.0559      
## # ... with 16 more rows
```

---

## ROC

```
##     In sample AUC Out of sample AUC 
##         0.7483132         0.7292981
```

---

## Code for last slide's curve

```r
library(ROCR)
# Have to remove all NA in prediction() with ROCR 1.0-11 version
# You may install the older version which does not require so
# https://cran.r-project.org/src/contrib/Archive/ROCR/ROCR_1.0-7.tar.gz
pred <- predict(fit_1990s, df, type="response")
*ROCpred <- prediction(as.numeric(pred[df$Test == 0 & !is.na(pred)]),
*                     as.numeric(df[df$Test == 0 & !is.na(pred), ]$AAER))
*ROCpred_out <- prediction(as.numeric(pred[df$Test == 1 & !is.na(pred)]),
*                         as.numeric(df[df$Test==1 & !is.na(pred), ]$AAER))
ROCperf <- performance(ROCpred, 'tpr', 'fpr')
ROCperf_out <- performance(ROCpred_out, 'tpr', 'fpr')
df_ROC_1990s <- data.frame(FalsePositive = c(ROCperf@x.values[[1]]),
                           TruePositive = c(ROCperf@y.values[[1]]))
df_ROC_out_1990s <- data.frame(FalsePositive = c(ROCperf_out@x.values[[1]]),
                               TruePositive = c(ROCperf_out@y.values[[1]]))

ggplot() +
  geom_line(data = df_ROC_1990s, aes(x = FalsePositive, y = TruePositive,
                                     color = "In Sample")) +
  geom_line(data = df_ROC_out_1990s, aes(x = FalsePositive, y = TruePositive,
                                         color = "Out of Sample")) + 
  geom_abline(slope = 1)

auc <- performance(ROCpred, measure = "auc")
auc_out <- performance(ROCpred_out, measure = "auc")
aucs_1990s <- c(auc@y.values[[1]], auc_out@y.values[[1]])
names(aucs_1990s) <- c("In sample AUC", "Out of sample AUC")
aucs_1990s
```

---
class: inverse, middle, center

# The 2011 follow up

---

## The 2011 model

.pull-left[
- Log of assets
- Total accruals
- % change in A/R
- % change in inventory
- % soft assets
- % change in sales from cash
- % change in ROA
- Indicator for stock/bond issuance
- Indicator for operating leases
- BV equity / MV equity
]
.pull-right[
- Lag of stock return minus value weighted market return
- **Below are BCE's additions**
- Indicator for mergers
- Indicator for Big N auditor
- Indicator for medium size auditor
- Total financing raised
- Net amount of new capital raised 
- Indicator for restructuring
]

> Based on <a target="_blank" href="https://doi.org/10.1111/j.1911-3846.2010.01041.x">Dechow, Ge, Larson and Sloan (2011)</a>

---

## The model

```r
fit_2011 <- glm(AAER ~ logtotasset + rsst_acc + chg_recv + chg_inv +
                  soft_assets + pct_chg_cashsales + chg_roa + issuance +
                  oplease_dum + book_mkt + lag_sdvol + merger + bigNaudit +
                  midNaudit + cffin + exfin + restruct,
                data = df[df$Test == 0, ],  family = binomial)
tidy(fit_2011)
```

```
## # A tibble: 18 x 5
##    term               estimate std.error statistic  p.value
##    <chr>                 <dbl>     <dbl>     <dbl>    <dbl>
##  1 (Intercept)       -7.15        0.534   -13.4    6.82e-41
##  2 logtotasset        0.321       0.0355    9.04   1.53e-19
##  3 rsst_acc          -0.219       0.301    -0.728  4.67e- 1
##  4 chg_recv           1.10        1.06      1.04   2.98e- 1
##  5 chg_inv            0.0390      1.25      0.0311 9.75e- 1
##  6 soft_assets        2.31        0.333     6.94   3.81e-12
##  7 pct_chg_cashsales -0.000691    0.0109   -0.0635 9.49e- 1
##  8 chg_roa           -0.270       0.255    -1.06   2.91e- 1
##  9 issuance           0.144       0.319     0.453  6.51e- 1
## 10 oplease_dum       -0.203       0.197    -1.03   3.03e- 1
## 11 book_mkt           0.0150      0.0111    1.36   1.75e- 1
## 12 lag_sdvol          0.0517      0.0555    0.932  3.52e- 1
## 13 merger             0.353       0.151     2.33   1.96e- 2
## 14 bigNaudit         -0.200       0.360    -0.555  5.79e- 1
## 15 midNaudit         -0.489       0.512    -0.956  3.39e- 1
## 16 cffin              0.456       0.344     1.33   1.85e- 1
## 17 exfin             -0.00536     0.0393   -0.136  8.92e- 1
## 18 restruct           0.383       0.147     2.60   9.30e- 3
```

---

## ROC

```
##     In sample AUC Out of sample AUC 
##         0.7445378         0.6849225
```

---

## Code for last slide's curve

```r
library(ROCR)
pred <- predict(fit_2011, df, type = "response")
ROCpred <- prediction(as.numeric(pred[df$Test == 0 & !is.na(pred)]),
                      as.numeric(df[df$Test==0 & !is.na(pred), ]$AAER))
ROCpred_out <- prediction(as.numeric(pred[df$Test == 1 & !is.na(pred)]),
                          as.numeric(df[df$Test == 1 & !is.na(pred), ]$AAER))
ROCperf <- performance(ROCpred, 'tpr', 'fpr')
ROCperf_out <- performance(ROCpred_out, 'tpr', 'fpr')
df_ROC_2011 <- data.frame(FalsePositive = c(ROCperf@x.values[[1]]),
                          TruePositive=c(ROCperf@y.values[[1]]))
df_ROC_out_2011 <- data.frame(FalsePositive = c(ROCperf_out@x.values[[1]]),
                              TruePositive = c(ROCperf_out@y.values[[1]]))

ggplot() +
  geom_line(data = df_ROC_2011, aes(x = FalsePositive, y = TruePositive,
                                    color = "In Sample")) +
  geom_line(data = df_ROC_out_2011, aes(x = FalsePositive, y = TruePositive,
                                        color = "Out of Sample")) + 
  geom_abline(slope = 1)

auc <- performance(ROCpred, measure = "auc")
auc_out <- performance(ROCpred_out, measure = "auc")
aucs_2011 <- c(auc@y.values[[1]], auc_out@y.values[[1]])
names(aucs_2011) <- c("In sample AUC", "Out of sample AUC")
aucs_2011
```

---
class: middle, center, inverse

# Late 2000s/early 2010s approach

---

## The late 2000s/early 2010s model

.pull-left[
- Log of # of bullet points + 1
- \# of characters in file header
- \# of excess newlines
- Amount of html tags
- Length of cleaned file, characters
- Mean sentence length, words
- S.D. of word length
- S.D. of paragraph length (sentences)
]
.pull-right[
- Word choice variation
- Readability
    - Coleman Liau Index
    - Fog Index
- % active voice sentences
- % passive voice sentences
- \# of all cap words
- \# of exclamation mark !
- \# of question mark ?
]

> From a variety of papers

- Generally pulled from the communications literature
    - Sometimes ad hoc
- The main idea:
    - Companies that are misreporting probably write their annual report differently

---

## The late 2000s/early 2010s model

```r
fit_2000s <- glm(AAER ~ bullets + headerlen + newlines + alltags +
                   processedsize + sentlen_u + wordlen_s + paralen_s +
                   repetitious_p + sentlen_s + typetoken + clindex + fog +
                   active_p + passive_p + lm_negative_p + lm_positive_p +
                   allcaps + exclamationpoints + questionmarks,
                 data=df[df$Test == 0, ], family = binomial)
tidy(fit_2000s)
```

```
## # A tibble: 21 x 5
##    term               estimate   std.error statistic    p.value
##    <chr>                 <dbl>       <dbl>     <dbl>      <dbl>
##  1 (Intercept)   -5.66         3.14           -1.80  0.0716    
##  2 bullets       -0.0000263    0.0000263      -1.00  0.316     
##  3 headerlen     -0.000294     0.000348       -0.846 0.397     
##  4 newlines      -0.0000482    0.000122       -0.395 0.693     
##  5 alltags        0.0000000506 0.000000257     0.197 0.844     
##  6 processedsize  0.00000571   0.00000129      4.44  0.00000919
##  7 sentlen_u     -0.0379       0.0690         -0.550 0.583     
##  8 wordlen_s      0.128        1.20            0.107 0.915     
##  9 paralen_s     -0.0481       0.0305         -1.58  0.115     
## 10 repetitious_p -1.67         1.67           -1.00  0.315     
## # ... with 11 more rows
```

---

## ROC

```
##     In sample AUC Out of sample AUC 
##         0.6377783         0.6295414
```

---

## Code for last slide's curve

```r
library(ROCR)
pred <- predict(fit_2000s, df, type = "response")
ROCpred <- prediction(as.numeric(pred[df$Test == 0 & !is.na(pred)]),
                      as.numeric(df[df$Test == 0 & !is.na(pred), ]$AAER))
ROCpred_out <- prediction(as.numeric(pred[df$Test == 1 & !is.na(pred)]),
                          as.numeric(df[df$Test == 1 & !is.na(pred), ]$AAER))
ROCperf <- performance(ROCpred, 'tpr', 'fpr')
ROCperf_out <- performance(ROCpred_out, 'tpr', 'fpr')
df_ROC_2000s <- data.frame(FalsePositive = c(ROCperf@x.values[[1]]),
                           TruePositive = c(ROCperf@y.values[[1]]))
df_ROC_out_2000s <- data.frame(FalsePositive = c(ROCperf_out@x.values[[1]]),
                               TruePositive = c(ROCperf_out@y.values[[1]]))

ggplot() +
  geom_line(data = df_ROC_2000s, aes(x = FalsePositive, y = TruePositive,
                                     color = "In Sample")) +
  geom_line(data = df_ROC_out_2000s, aes(x = FalsePositive, y = TruePositive,
                                         color = "Out of Sample")) + 
  geom_abline(slope = 1)

auc <- performance(ROCpred, measure = "auc")
auc_out <- performance(ROCpred_out, measure = "auc")
aucs_2000s <- c(auc@y.values[[1]], auc_out@y.values[[1]])
names(aucs_2000s) <- c("In sample AUC", "Out of sample AUC")
aucs_2000s
```

---

## Combining the 2000s and 2011 models

> Why is it appropriate to combine the 2011 model with the 2000s model?

- 2011 model: Parsimonious financial model
- 2000s model: Textual characteristics

> Little theoretical overlap

> Limited multicollinearity across measures

---

## The model

```r
fit_2000f <- glm(AAER ~ logtotasset + rsst_acc + chg_recv + chg_inv +
                   soft_assets + pct_chg_cashsales + chg_roa + issuance +
                   oplease_dum + book_mkt + lag_sdvol + merger + bigNaudit +
                   midNaudit + cffin + exfin + restruct + bullets + headerlen +
                   newlines + alltags + processedsize + sentlen_u + wordlen_s +
                   paralen_s + repetitious_p + sentlen_s + typetoken +
                   clindex + fog + active_p + passive_p + lm_negative_p +
                   lm_positive_p + allcaps + exclamationpoints + questionmarks,
                 data = df[df$Test == 0, ], family = binomial)
tidy(fit_2000f)
```

```
## # A tibble: 38 x 5
##    term              estimate std.error statistic  p.value
##    <chr>                <dbl>     <dbl>     <dbl>    <dbl>
##  1 (Intercept)       -1.63       3.41     -0.479  6.32e- 1
##  2 logtotasset        0.344      0.0392    8.77   1.86e-18
##  3 rsst_acc          -0.212      0.299    -0.709  4.78e- 1
##  4 chg_recv           1.02       1.05      0.969  3.33e- 1
##  5 chg_inv           -0.0946     1.23     -0.0772 9.38e- 1
##  6 soft_assets        2.57       0.339     7.60   3.01e-14
##  7 pct_chg_cashsales -0.00113    0.0110   -0.103  9.18e- 1
##  8 chg_roa           -0.270      0.247    -1.09   2.75e- 1
##  9 issuance           0.147      0.322     0.457  6.48e- 1
## 10 oplease_dum       -0.286      0.202    -1.42   1.57e- 1
## # ... with 28 more rows
```

---

## ROC

```
##     In sample AUC Out of sample AUC 
##         0.7664115         0.7147021
```

---
class: middle, center, inverse

# The BCE model

---

## The BCE approach

- Retain the variables from the other regressions
- Add in a machine-learning based measure of *topics*
    - Every document is a mixture of topics. It may contain words from several topics in particular proportions. Eg, we could say "Document 1 is 90% topic A and 10% topic B, while Document 2 is 30% topic A and 70% topic B."
    - Every topic is a mixture of words. Eg, we imagine a two-topic model of annual reports, "decrease in income" and "bad debt." The most common words in the income decreasing topic might be "income decreased" and "company expects", while the bad debt topic may be made up of words such as "accounts receivable" and "allowance for doubtful". Importantly, words can be shared between topics.

> Don't worry, we will cover this in a future topic

---

## What the topics look like

.center[<img src="../../../Figures/LDA2004.png" width="800px">]

---

## Theory behind the BCE model

> Why use document content?

- From communications and psychology:
    - When people are trying to deceive others, what they say is carefully picked
        - Topics chosen are intentional
- Putting this in a business context:
    - If you are manipulating inventory, you don't talk about it

---

## The model

```r
BCE_eq = as.formula(paste("AAER ~ logtotasset + rsst_acc + chg_recv + chg_inv +
                          soft_assets + pct_chg_cashsales + chg_roa + issuance +
                          oplease_dum + book_mkt + lag_sdvol + merger +
                          bigNaudit + midNaudit + cffin + exfin + restruct +
                          bullets + headerlen + newlines + alltags +
                          processedsize + sentlen_u + wordlen_s + paralen_s +
                          repetitious_p + sentlen_s + typetoken + clindex + fog +
                          active_p + passive_p + lm_negative_p + lm_positive_p +
                          allcaps + exclamationpoints + questionmarks +",
                          paste(paste0("Topic_", 1:30, "_n_oI"), collapse=" + "),
                          collapse = "")) # Topic_1_n_oI is the topic variable

fit_BCE <- glm(BCE_eq, data = df[df$Test == 0, ], family = binomial)
```

---

## BCE model output

```r
tidy(fit_BCE) # Topic_1_n_oI is the topic variable
```

```
## # A tibble: 68 x 5
##    term              estimate std.error statistic  p.value
##    <chr>                <dbl>     <dbl>     <dbl>    <dbl>
##  1 (Intercept)       -8.03      3.87       -2.07  3.81e- 2
##  2 logtotasset        0.388     0.0455      8.52  1.62e-17
##  3 rsst_acc          -0.194     0.306      -0.634 5.26e- 1
##  4 chg_recv           0.858     1.07        0.801 4.23e- 1
##  5 chg_inv           -0.261     1.22       -0.213 8.31e- 1
##  6 soft_assets        2.55      0.380       6.73  1.70e-11
##  7 pct_chg_cashsales -0.00198   0.00700    -0.282 7.78e- 1
##  8 chg_roa           -0.253     0.279      -0.909 3.64e- 1
##  9 issuance           0.0969    0.327       0.296 7.67e- 1
## 10 oplease_dum       -0.345     0.210      -1.65  9.99e- 2
## # ... with 58 more rows
```

---

## ROC

```
##     In sample AUC Out of sample AUC 
##         0.7941841         0.7599594
```

---

## Comparison across all models

```
##        1990s         2011        2000s 2000s + 2011          BCE 
##    0.7292981    0.6849225    0.6295414    0.7147021    0.7599594
```

---
class: middle, center, inverse

# Simplifying models with LASSO

---

## What is lasso?

.center[<img src="../../../Figures/lassorope.jpg" width="800px">]

---

## What is LASSO?

- Least Absolute Shrinkage and Selection Operator
    - Least absolute: uses an error term like `$|\varepsilon|$`
    - Shrinkage: it will make coefficients smaller
        - Less sensitive and less overfitting issues
    - Selection: it will completely remove some variables
        - Less variables and less overfitting issues
- Sometimes called `$L^1$` regularization
    - `$L^1$` means 1 dimensional distance, i.e., `$|\varepsilon|$`

> Great if you have way too many inputs in your model

- This is how we can, in theory, put more variables in our model than data points

---

## How does it work?

`$$\min_{ \beta \in \mathbb{R}^p } \left\{ \frac{1}{N} \left\| y - X \beta \right\|_2^2 + \lambda \| \beta \|_1 \right\}$$`

.pull-left[
- where `$\| \beta \|_p = \left( \sum_{i=1}^N | \beta_i |^p \right)^{1/p}$`
- to minimize training loss and regularization term
- Add an additional penalty term that is increasing in the absolute value of each `$\beta$`
    - Incentivizes lower `$\beta$`s, *shrinking* them
]
.pull-right[
.center[<img src="../../../Figures/LASSO.png" width="400px">]
]

---

## Why we use it?

- A better balance of **bias** (low) and **variance** (low)
    - **bias**: the inability for a ML method to capture the true relationship (intuitively, the prediction error in the training dataset)
    - **variance**: the difference in fits between datasets (intuitively, the prediction error in the testing dataset)
    - mitigate the overfitting (high variance) and underfiting (high bias) problems
    - regularization, bagging and boosting are used to achieve the balance
- Model selection by the machine
    - `$\beta$` could be 0, ie, selection of features to best fit the model
    - discourages learning a more complex model
- Automate the <a target="_blank" href="https://towardsdatascience.com/the-5-feature-selection-algorithms-every-data-scientist-need-to-know-3a6b566efd2">feature selection</a> process

---

## Package for LASSO

- [`package:glmnet`](https://glmnet.stanford.edu), pronounced glimnet
1. For all regression commands, they expect a **y** vector and an **x** matrix instead of our usual `y ~ x` formula
    - R has a helper function to convert a formula to a matrix: `model.matrix()`
        - Supply it the right hand side of the equation, starting with `~`, and your data
        - It outputs the matrix `x`
    - Alternatively, use `as.matrix()` on a data frame of your input variables
2. Its family argument should be specified in quotes, i.e., `"binomial"` instead of `binomial`

---

## What else can the package do?

.pull-left[
**Ridge regression**

- Similar to LASSO, but with an `$L^2$` penalty (Euclidean norm)

<img src="../../../Figures/ridge.png" width="400px">
]
.pull-right[
**Elastic net regression**

- Hybrid of LASSO and Ridge
- Below image by <a target="_blank" href="https://jaredlander.com/content/2015/11/LassoForEveryone.html">Jared Lander</a>

]

---

## How to run a LASSO

- To run a simple LASSO model, use <a target="_blank" href="https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html">`glmnet()`</a>

`$\min_{\beta_0,\beta} \frac{1}{N} \sum_{i=1}^{N} w_i l(y_i,\beta_0+\beta^T x_i) + \lambda\left[(1-\alpha)||\beta||_2^2/2 + \alpha ||\beta||_1\right]$`

```r
library(glmnet)
x <- model.matrix(BCE_eq, data = df[df$Test == 0, ])[ , -1] # remove intercept
y <- model.frame(BCE_eq, data = df[df$Test == 0, ])[ , "AAER"]
fit_LASSO <- glmnet(x = x, y = y,
                    family = "binomial", # "gaussian" for least squares (default)
                    alpha = 1  # LASSO, the default.  alpha = 0 is ridge
                               # alpha between 0 and 1: elastic net
                    )
```

- Note: the **x** and **y** can be more elegantly done using the [`package:useful`](https://github.com/jaredlander/useful), <a target="_blank"  href="https://www.jaredlander.com/2018/02/using-coefplot-with-glmnet/">see here for an example</a>
    - `useful::build.x(formula, data)` and `useful::build.y(formual, data)`

---

## Visualizing Lasso

```r
plot(fit_LASSO, xvar = 'lambda', label = TRUE)
```

- the top x-axis is the number of degrees of freedom of the model, i.e., the number of variables **minus 1**.

---

## Interactive visualizing Lasso

```r
coefpath(fit_LASSO)
```

<div id="coefpath_M09zXQuzvTPp" style="width:100%;height:360px;" class="dygraphs html-widget"></div>
<script type="application/json" data-for="coefpath_M09zXQuzvTPp">{"x":{"attrs":{"axes":{"x":{"pixelsPerLabel":60},"y":[]},"labels":["lambda","logtotasset","rsst_acc","chg_recv","chg_inv","soft_assets","pct_chg_cashsales","chg_roa","issuance","oplease_dum","book_mkt","lag_sdvol","merger","bigNaudit","midNaudit","cffin","exfin","restruct","bullets","headerlen","newlines","alltags","processedsize","sentlen_u","wordlen_s","paralen_s","repetitious_p","sentlen_s","typetoken","clindex","fog","active_p","passive_p","lm_negative_p","lm_positive_p","allcaps","exclamationpoints","questionmarks","Topic_1_n_oI","Topic_2_n_oI","Topic_3_n_oI","Topic_4_n_oI","Topic_5_n_oI","Topic_6_n_oI","Topic_7_n_oI","Topic_8_n_oI","Topic_9_n_oI","Topic_10_n_oI","Topic_11_n_oI","Topic_12_n_oI","Topic_13_n_oI","Topic_14_n_oI","Topic_15_n_oI","Topic_16_n_oI","Topic_17_n_oI","Topic_18_n_oI","Topic_19_n_oI","Topic_20_n_oI","Topic_21_n_oI","Topic_22_n_oI","Topic_23_n_oI","Topic_24_n_oI","Topic_25_n_oI","Topic_26_n_oI","Topic_27_n_oI","Topic_28_n_oI","Topic_29_n_oI","Topic_30_n_oI"],"retainDateWindow":false,"xlabel":"Log Lambda","ylabel":"Coefficients","legend":"onmouseover","labelsDivWidth":250,"labelsShowZeroValues":true,"labelsSeparateLines":false,"showRangeSelector":true,"rangeSelectorHeight":40,"rangeSelectorPlotFillColor":" #A7B1C4","rangeSelectorPlotStrokeColor":"#808FAB","interactionModel":"Dygraph.Interaction.defaultModel","highlightCircleSize":3,"highlightSeriesBackgroundAlpha":0.5,"highlightSeriesOpts":{"strokeWidth":3},"hideOverlayOnMouseOut":true},"annotations":[{"x":-11.1302367532268,"shortText":"logtotasset","text":"logtotasset","width":50,"attachAtBottom":false,"series":"logtotasset"},{"x":-11.1302367532268,"shortText":"rsst_acc","text":"rsst_acc","width":50,"attachAtBottom":false,"series":"rsst_acc"},{"x":-11.1302367532268,"shortText":"chg_recv","text":"chg_recv","width":50,"attachAtBottom":false,"series":"chg_recv"},{"x":-11.1302367532268,"shortText":"chg_inv","text":"chg_inv","width":50,"attachAtBottom":false,"series":"chg_inv"},{"x":-11.1302367532268,"shortText":"soft_assets","text":"soft_assets","width":50,"attachAtBottom":false,"series":"soft_assets"},{"x":-11.1302367532268,"shortText":"pct_chg_cashsales","text":"pct_chg_cashsales","width":50,"attachAtBottom":false,"series":"pct_chg_cashsales"},{"x":-11.1302367532268,"shortText":"chg_roa","text":"chg_roa","width":50,"attachAtBottom":false,"series":"chg_roa"},{"x":-11.1302367532268,"shortText":"issuance","text":"issuance","width":50,"attachAtBottom":false,"series":"issuance"},{"x":-11.1302367532268,"shortText":"oplease_dum","text":"oplease_dum","width":50,"attachAtBottom":false,"series":"oplease_dum"},{"x":-11.1302367532268,"shortText":"book_mkt","text":"book_mkt","width":50,"attachAtBottom":false,"series":"book_mkt"},{"x":-11.1302367532268,"shortText":"lag_sdvol","text":"lag_sdvol","width":50,"attachAtBottom":false,"series":"lag_sdvol"},{"x":-11.1302367532268,"shortText":"merger","text":"merger","width":50,"attachAtBottom":false,"series":"merger"},{"x":-11.1302367532268,"shortText":"bigNaudit","text":"bigNaudit","width":50,"attachAtBottom":false,"series":"bigNaudit"},{"x":-11.1302367532268,"shortText":"midNaudit","text":"midNaudit","width":50,"attachAtBottom":false,"series":"midNaudit"},{"x":-11.1302367532268,"shortText":"cffin","text":"cffin","width":50,"attachAtBottom":false,"series":"cffin"},{"x":-11.1302367532268,"shortText":"exfin","text":"exfin","width":50,"attachAtBottom":false,"series":"exfin"},{"x":-11.1302367532268,"shortText":"restruct","text":"restruct","width":50,"attachAtBottom":false,"series":"restruct"},{"x":-11.1302367532268,"shortText":"bullets","text":"bullets","width":50,"attachAtBottom":false,"series":"bullets"},{"x":-11.1302367532268,"shortText":"headerlen","text":"headerlen","width":50,"attachAtBottom":false,"series":"headerlen"},{"x":-11.1302367532268,"shortText":"newlines","text":"newlines","width":50,"attachAtBottom":false,"series":"newlines"},{"x":-11.1302367532268,"shortText":"alltags","text":"alltags","width":50,"attachAtBottom":false,"series":"alltags"},{"x":-11.1302367532268,"shortText":"processedsize","text":"processedsize","width":50,"attachAtBottom":false,"series":"processedsize"},{"x":-11.1302367532268,"shortText":"sentlen_u","text":"sentlen_u","width":50,"attachAtBottom":false,"series":"sentlen_u"},{"x":-11.1302367532268,"shortText":"wordlen_s","text":"wordlen_s","width":50,"attachAtBottom":false,"series":"wordlen_s"},{"x":-11.1302367532268,"shortText":"paralen_s","text":"paralen_s","width":50,"attachAtBottom":false,"series":"paralen_s"},{"x":-11.1302367532268,"shortText":"repetitious_p","text":"repetitious_p","width":50,"attachAtBottom":false,"series":"repetitious_p"},{"x":-11.1302367532268,"shortText":"sentlen_s","text":"sentlen_s","width":50,"attachAtBottom":false,"series":"sentlen_s"},{"x":-11.1302367532268,"shortText":"typetoken","text":"typetoken","width":50,"attachAtBottom":false,"series":"typetoken"},{"x":-11.1302367532268,"shortText":"clindex","text":"clindex","width":50,"attachAtBottom":false,"series":"clindex"},{"x":-11.1302367532268,"shortText":"fog","text":"fog","width":50,"attachAtBottom":false,"series":"fog"},{"x":-11.1302367532268,"shortText":"active_p","text":"active_p","width":50,"attachAtBottom":false,"series":"active_p"},{"x":-11.1302367532268,"shortText":"passive_p","text":"passive_p","width":50,"attachAtBottom":false,"series":"passive_p"},{"x":-11.1302367532268,"shortText":"lm_negative_p","text":"lm_negative_p","width":50,"attachAtBottom":false,"series":"lm_negative_p"},{"x":-11.1302367532268,"shortText":"lm_positive_p","text":"lm_positive_p","width":50,"attachAtBottom":false,"series":"lm_positive_p"},{"x":-11.1302367532268,"shortText":"allcaps","text":"allcaps","width":50,"attachAtBottom":false,"series":"allcaps"},{"x":-11.1302367532268,"shortText":"exclamationpoints","text":"exclamationpoints","width":50,"attachAtBottom":false,"series":"exclamationpoints"},{"x":-11.1302367532268,"shortText":"questionmarks","text":"questionmarks","width":50,"attachAtBottom":false,"series":"questionmarks"},{"x":-11.1302367532268,"shortText":"Topic_1_n_oI","text":"Topic_1_n_oI","width":50,"attachAtBottom":false,"series":"Topic_1_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_2_n_oI","text":"Topic_2_n_oI","width":50,"attachAtBottom":false,"series":"Topic_2_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_3_n_oI","text":"Topic_3_n_oI","width":50,"attachAtBottom":false,"series":"Topic_3_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_4_n_oI","text":"Topic_4_n_oI","width":50,"attachAtBottom":false,"series":"Topic_4_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_5_n_oI","text":"Topic_5_n_oI","width":50,"attachAtBottom":false,"series":"Topic_5_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_6_n_oI","text":"Topic_6_n_oI","width":50,"attachAtBottom":false,"series":"Topic_6_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_7_n_oI","text":"Topic_7_n_oI","width":50,"attachAtBottom":false,"series":"Topic_7_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_8_n_oI","text":"Topic_8_n_oI","width":50,"attachAtBottom":false,"series":"Topic_8_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_9_n_oI","text":"Topic_9_n_oI","width":50,"attachAtBottom":false,"series":"Topic_9_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_10_n_oI","text":"Topic_10_n_oI","width":50,"attachAtBottom":false,"series":"Topic_10_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_11_n_oI","text":"Topic_11_n_oI","width":50,"attachAtBottom":false,"series":"Topic_11_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_12_n_oI","text":"Topic_12_n_oI","width":50,"attachAtBottom":false,"series":"Topic_12_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_13_n_oI","text":"Topic_13_n_oI","width":50,"attachAtBottom":false,"series":"Topic_13_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_14_n_oI","text":"Topic_14_n_oI","width":50,"attachAtBottom":false,"series":"Topic_14_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_15_n_oI","text":"Topic_15_n_oI","width":50,"attachAtBottom":false,"series":"Topic_15_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_16_n_oI","text":"Topic_16_n_oI","width":50,"attachAtBottom":false,"series":"Topic_16_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_17_n_oI","text":"Topic_17_n_oI","width":50,"attachAtBottom":false,"series":"Topic_17_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_18_n_oI","text":"Topic_18_n_oI","width":50,"attachAtBottom":false,"series":"Topic_18_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_19_n_oI","text":"Topic_19_n_oI","width":50,"attachAtBottom":false,"series":"Topic_19_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_20_n_oI","text":"Topic_20_n_oI","width":50,"attachAtBottom":false,"series":"Topic_20_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_21_n_oI","text":"Topic_21_n_oI","width":50,"attachAtBottom":false,"series":"Topic_21_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_22_n_oI","text":"Topic_22_n_oI","width":50,"attachAtBottom":false,"series":"Topic_22_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_23_n_oI","text":"Topic_23_n_oI","width":50,"attachAtBottom":false,"series":"Topic_23_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_24_n_oI","text":"Topic_24_n_oI","width":50,"attachAtBottom":false,"series":"Topic_24_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_25_n_oI","text":"Topic_25_n_oI","width":50,"attachAtBottom":false,"series":"Topic_25_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_26_n_oI","text":"Topic_26_n_oI","width":50,"attachAtBottom":false,"series":"Topic_26_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_27_n_oI","text":"Topic_27_n_oI","width":50,"attachAtBottom":false,"series":"Topic_27_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_28_n_oI","text":"Topic_28_n_oI","width":50,"attachAtBottom":false,"series":"Topic_28_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_29_n_oI","text":"Topic_29_n_oI","width":50,"attachAtBottom":false,"series":"Topic_29_n_oI"},{"x":-11.1302367532268,"shortText":"Topic_30_n_oI","text":"Topic_30_n_oI","width":50,"attachAtBottom":false,"series":"Topic_30_n_oI"}],"shadings":[],"events":[],"format":"numeric","data":[[-11.1302367532268,-11.0372030121824,-10.9441692711379,-10.8511355300935,-10.7581017890491,-10.6650680480047,-10.5720343069602,-10.4790005659158,-10.3859668248714,-10.292933083827,-10.1998993427825,-10.1068656017381,-10.0138318606937,-9.92079811964924,-9.82776437860481,-9.73473063756038,-9.64169689651595,-9.54866315547152,-9.4556294144271,-9.36259567338267,-9.26956193233824,-9.17652819129381,-9.08349445024938,-8.99046070920496,-8.89742696816053,-8.8043932271161,-8.71135948607167,-8.61832574502724,-8.52529200398282,-8.43225826293839,-8.33922452189396,-8.24619078084953,-8.1531570398051,-8.06012329876068,-7.96708955771625,-7.87405581667182,-7.78102207562739,-7.68798833458296,-7.59495459353854,-7.50192085249411,-7.40888711144968,-7.31585337040525,-7.22281962936082,-7.12978588831639,-7.03675214727197,-6.94371840622754,-6.85068466518311,-6.75765092413868,-6.66461718309425,-6.57158344204983,-6.4785497010054,-6.38551595996097,-6.29248221891654,-6.19944847787211,-6.10641473682769,-6.01338099578326,-5.92034725473883,-5.8273135136944,-5.73427977264997,-5.64124603160554,-5.54821229056112,-5.45517854951669,-5.36214480847226,-5.26911106742783,-5.1760773263834,-5.08304358533898,-4.99000984429455,-4.89697610325012,-4.80394236220569,-4.71090862116126,-4.61787488011684,-4.52484113907241,-4.43180739802798,-4.33877365698355,-4.24573991593912],[0.387258614362204,0.387149123033305,0.387015107982467,0.386867967430202,0.386706220408972,0.386530734219673,0.386339760212247,0.38612409420419,0.385808585941777,0.385432242065738,0.385026052897258,0.384585713809497,0.384105835285457,0.383572439450778,0.382992687377003,0.382349588931308,0.381662031305963,0.380909383935607,0.380097751363317,0.379217434485594,0.378259901596004,0.377210050499165,0.376144337643192,0.375009938156609,0.373784552622442,0.372607826197874,0.371433795667746,0.37006387516075,0.36841382402244,0.36704862772017,0.365710155498879,0.36420554272698,0.362620374865566,0.360858455929355,0.358840310778038,0.35693208047231,0.354766472518965,0.352488090980531,0.350165209401659,0.347900677244839,0.345833272543105,0.343382673434121,0.340731899616889,0.337778458402478,0.335114911714556,0.33260953174923,0.329183390116478,0.324496895123121,0.319328014389491,0.314669303238656,0.30881821542528,0.302490302832393,0.29610108065244,0.289431675274407,0.283021252143457,0.275912442527813,0.26806515702572,0.260631110959998,0.253634093214053,0.247113250150829,0.239302958721946,0.230081049920529,0.221346574484978,0.213014141550343,0.202687074520784,0.19151532525549,0.178462392038583,0.163279953938002,0.146611364095921,0.128285857721697,0.108104177907154,0.0853979749832359,0.0600563056472091,0.0317400050632684,0],[-0.184698417525348,-0.184145575475959,-0.183277459987904,-0.18232608885469,-0.181284231547344,-0.180141137439963,-0.178888479483637,-0.177527215897265,-0.176132792126023,-0.174744252496009,-0.173198041387322,-0.171435162365026,-0.169486157749354,-0.167363654154822,-0.165035272279669,-0.162462130414464,-0.159688903940343,-0.156594486809859,-0.153288352572591,-0.149594799471782,-0.14557912243834,-0.141317418940242,-0.136533539742937,-0.13128812643646,-0.125405361612631,-0.118831682266553,-0.111964911555681,-0.103427535642352,-0.0932350702709083,-0.0809682192050808,-0.068067261241493,-0.0533506579750267,-0.0372083865585492,-0.0191689170435445,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.847154633743494,0.845659554535189,0.844069201137858,0.842324987729052,0.840408421447367,0.838319451244682,0.836038753688426,0.83354762994044,0.830751524562709,0.827499166025558,0.823803981210133,0.819578583792096,0.814723714581376,0.809246106248573,0.803328856113918,0.796689166767371,0.789651713778589,0.781841705992605,0.773360633026385,0.76395282027666,0.753647573049668,0.74234466741661,0.730086915410463,0.71696912018557,0.703648131538821,0.688184181036246,0.670968166614735,0.650697278106031,0.629999467255572,0.610492713075071,0.590905768798911,0.567607256261135,0.542069462866325,0.513181106698581,0.483427828720606,0.463274238933428,0.441408063588867,0.41874793617108,0.39409402198571,0.372632848612504,0.355711618958462,0.335219755013546,0.313273852874142,0.292645395866784,0.270983850232505,0.245614180770655,0.216129586881937,0.183907565028758,0.146101638896586,0.0708085787257971,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.240989695794015,-0.23879257460682,-0.236677431768481,-0.23436386152229,-0.231832163156658,-0.229064919653072,-0.226040190366819,-0.222755932685967,-0.219383328966443,-0.215695009543333,-0.211539995464837,-0.206851002951574,-0.201623174035055,-0.195869543229587,-0.189591783516983,-0.182670332922681,-0.175098735560831,-0.166743907106086,-0.157664891549018,-0.147635075884086,-0.136706406571461,-0.124855649091711,-0.111885807801211,-0.0972029223279174,-0.0798718409489976,-0.0602511660839431,-0.0383460510422994,-0.0136936719297007,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[2.55280813946404,2.55235848472154,2.55187050672681,2.55133247582468,2.55073907773788,2.55009439020133,2.54939193110159,2.54859652059602,2.5474345507657,2.54610064045417,2.54465669298451,2.5430384626582,2.54122055619628,2.53917739191082,2.53694847123869,2.53440110837514,2.53169735010311,2.52865314438768,2.52545872740253,2.52188172630063,2.51803200814456,2.51396555882622,2.5094938769674,2.50419347508778,2.49627818996479,2.48777689575044,2.47882263851791,2.46832290701119,2.45596118898778,2.44214428097858,2.42793505657037,2.41277823441866,2.39624458991564,2.37822192994925,2.36008071546126,2.34151127113722,2.32128401087372,2.29954960546538,2.27789846526401,2.2540458763208,2.22907157937774,2.20201054701197,2.1722332947158,2.13763403827713,2.10147729970293,2.06146425355188,2.02411308326318,1.98221129631589,1.93298123572888,1.88710714007595,1.8413034713745,1.78974620994698,1.73399769293816,1.67809047178687,1.62530041688698,1.56742067785003,1.5039024442098,1.44508715914968,1.38316054835189,1.31810473734687,1.2530827434054,1.18312890247691,1.10166385462467,1.00192894574895,0.897287238914458,0.786052323218488,0.659440255732948,0.515515997131769,0.360987821198776,0.195132713380253,0.0171321272775771,0,0,0,0],[-0.00182864345014665,-0.00181441821587495,-0.0017977876798512,-0.00177934820213445,-0.0017588764333735,-0.00173613119963113,-0.00171082272184005,-0.00168261550027291,-0.00165081182112104,-0.00161464345504151,-0.0015735206237078,-0.00152703732726556,-0.00147492260260443,-0.00141619659645765,-0.00134840650420381,-0.0012709203358312,-0.00118071682565639,-0.00107606429537501,-0.000952320863229185,-0.000805927658346821,-0.000629743271129,-0.000414844297049844,-0.000150780392469031,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.253291980734029,-0.253167737773656,-0.253069030396844,-0.252954645080875,-0.252821153468702,-0.252667897198343,-0.252490403815395,-0.252277573074315,-0.251997845707336,-0.251567656642543,-0.251017928659856,-0.250385625947829,-0.249704676906694,-0.248989940902555,-0.248090134479929,-0.247136710467674,-0.245940817019829,-0.244654486924884,-0.243077387097394,-0.241347098964403,-0.239341195646898,-0.236996832527072,-0.234054987042025,-0.230513063146109,-0.226685990709003,-0.222471607933888,-0.216768238163357,-0.210672079207408,-0.204399517510333,-0.197516948792891,-0.191302076156085,-0.182474023380807,-0.17265667936858,-0.161247069243941,-0.149328880727297,-0.128141378361663,-0.105163691795344,-0.0807533775780023,-0.0557038896919421,-0.0384971812821184,-0.0344953805563418,-0.0297573494636433,-0.0241807598969733,-0.0172848588684052,-0.00903237542126326,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.0911265986067846,0.0906129707914842,0.0899680144058004,0.0892616227674966,0.0884880771274732,0.0876431221819533,0.0867204198454662,0.0857128773391401,0.0845794096413674,0.0833186565893609,0.0819284714456393,0.0803972798797498,0.0787321138047589,0.076925432116941,0.0749340933339626,0.072773662189463,0.0704305424358268,0.0678723339407629,0.0651051561109719,0.0620865022274214,0.0588204924391054,0.0553010135191665,0.0514992228693779,0.0473862455123198,0.0431618808556446,0.0387075088632995,0.0339448137622505,0.0289463626709033,0.0241053144243358,0.0193314794016803,0.0142067847328112,0.00881257679024187,0.0030094088875355,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.33344326288239,-0.332958013945001,-0.332099723207655,-0.331163561434203,-0.330145169412822,-0.329025748901756,-0.327799479846751,-0.32647882981216,-0.325259245960229,-0.324153520078916,-0.323016724676198,-0.321809451121523,-0.32052792002634,-0.319160856845169,-0.31768382300146,-0.316134084891025,-0.314449160320913,-0.312615650830974,-0.310624944807212,-0.308456897482274,-0.306098902789004,-0.303539794999696,-0.300779129042869,-0.297804243654795,-0.294536638146006,-0.291110501180395,-0.287408813657399,-0.28322407448363,-0.27847809715978,-0.27320828018057,-0.267593065547097,-0.261237885376972,-0.254362769156654,-0.246883720247265,-0.239008505443616,-0.230603307336781,-0.221332454729205,-0.211216710400938,-0.200112817080453,-0.187762186109094,-0.174584011593936,-0.160233211598445,-0.144512891209275,-0.126953068273576,-0.10789079936157,-0.0882762651987815,-0.0655428376628372,-0.0407887710794732,-0.0129838529798069,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.0132924244445936,0.0132662277078579,0.0132318666480257,0.0131940428172761,0.0131523890798374,0.0131065121279175,0.0130559794423845,0.013000540479497,0.0129419693190543,0.0128783225632429,0.0128054942806719,0.0127226746638036,0.0126315509020116,0.0125329023088259,0.0124215766062883,0.0122998911255224,0.0121637024362233,0.0120135685172373,0.0118468630742342,0.0116621284471024,0.0114576836089192,0.0112338183963186,0.0109744583902329,0.0106811188291972,0.0103573283629488,0.00999709993293657,0.00958031153572038,0.00909755859440708,0.00855521540561056,0.00792802239625803,0.00729725329908084,0.00652962151721413,0.00568431525631661,0.00472953223899386,0.00371927729965565,0.00299969531005185,0.00222491964262259,0.00140227837575128,0.000553255459486951,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.0442737914251237,0.0441877472806927,0.0440725263951086,0.0439468994447188,0.0438099620823755,0.0436607321650427,0.0434983417451039,0.0433246892644232,0.0431547513706144,0.0429764671343559,0.0427761648240474,0.0425496142534419,0.0423002282476129,0.0420306862070039,0.0417370931269029,0.0414174241444147,0.0410753604766315,0.0406974362062333,0.0402944652065217,0.0398463577922518,0.0393613441410601,0.0388402842428274,0.0382966676901191,0.0376966077049256,0.0370040254622287,0.036182169271653,0.0353109750765982,0.0343527970712069,0.0334002176387646,0.0322288270392734,0.0308824232327402,0.0294240173087616,0.0278463529647436,0.0261294030473737,0.0242545544262149,0.0230819427593688,0.0217961550544135,0.0204237973149454,0.0189409186338983,0.0172269115845882,0.0151501059844563,0.0127542665302397,0.0101603084677559,0.00734977204873047,0.00420220081402502,0.00060076964253564,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.324168205958174,0.324154688296241,0.324222597869978,0.324297339200503,0.324378533490543,0.324471462253983,0.324576668934454,0.324695750369442,0.324783659704749,0.324819491790999,0.324817819358809,0.324776462043309,0.324714442247774,0.324645154851985,0.324562079651021,0.324469598271597,0.324365824150945,0.324235178456733,0.324118595393319,0.323964491876824,0.323803273251888,0.323650063646088,0.323547513530084,0.323445966116351,0.323285235522602,0.323105570522926,0.323019260117944,0.322690613767671,0.322586181114294,0.322642785224192,0.322574010916451,0.32248421504003,0.322374687062204,0.322172800251783,0.322195098544752,0.323978059883988,0.325548610319924,0.327186510765296,0.328640355182848,0.329622867169784,0.330442825063098,0.331734557111246,0.332722679998688,0.332293098805111,0.331777155058301,0.331347229260435,0.329832233904744,0.328896398030711,0.327282533431615,0.322197768192585,0.317388386396159,0.310003572036789,0.301606876866066,0.292377279489018,0.282212809372194,0.270973479953213,0.258653678316413,0.244220854619486,0.228277917260572,0.208621477473013,0.185817289677494,0.1602968969493,0.131618066596398,0.0994932645738456,0.0644116685825434,0.024477818554121,0,0,0,0,0,0,0,0,0],[-0.184232367233308,-0.183194625729511,-0.181499634748592,-0.179640066664485,-0.177600300845676,-0.175360854802702,-0.172902526080773,-0.170201963799533,-0.167247112151019,-0.164009061736311,-0.160390880293576,-0.156426883713014,-0.152250693886745,-0.147811784460458,-0.142803751792626,-0.137383582430322,-0.13135552836194,-0.124790123121521,-0.117538364680049,-0.109622306323485,-0.100920735216037,-0.0913571119860839,-0.0808579472009146,-0.0696729906385661,-0.0579103079187869,-0.0452746809226953,-0.0312578519792111,-0.0156719484586794,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.444621959832311,-0.443222604286473,-0.441222165671097,-0.439029458773576,-0.436625744477157,-0.43399053695005,-0.431102063609685,-0.427947452566613,-0.424594567628715,-0.420937230720395,-0.416833112588294,-0.412277868459437,-0.407373462113411,-0.402093282028776,-0.396214348612528,-0.389788218774779,-0.382729644432386,-0.374996244229839,-0.366534748600456,-0.357248321011564,-0.347089100274423,-0.336014027016373,-0.323887197230844,-0.310983859629381,-0.297551478764852,-0.282944819519052,-0.26699087225723,-0.248508479576508,-0.228971863810548,-0.223312057312959,-0.216902422167138,-0.209732475821282,-0.201970817296678,-0.193508969406809,-0.184490410457352,-0.174213284740164,-0.162939712131477,-0.150663151169767,-0.1369757455217,-0.122100730417012,-0.105799163530013,-0.0870588116327465,-0.0669685778762931,-0.0457220829492771,-0.0230711752864176,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.596412474678525,0.596064196569806,0.5954402448511,0.59475515050745,0.594002759901585,0.593176775124659,0.592270011257079,0.591275407610691,0.590159808488962,0.58885918923729,0.58741065694341,0.585868643248255,0.584282823725852,0.582629499547749,0.580731182131321,0.578736545162656,0.576497380144866,0.574096222830131,0.571431250289834,0.568566228980049,0.565442522217516,0.562124435476653,0.557740330784584,0.553011313061587,0.548259504350413,0.543268376986259,0.537769211014221,0.531672450093713,0.525695601043528,0.518632241412001,0.510259470227675,0.501187779162252,0.491024846725211,0.479422540664767,0.465221060365167,0.449961070269483,0.433677695053699,0.415475853473698,0.394648045596257,0.370461978053667,0.343143341190242,0.314416968006688,0.282200151388037,0.244920746866068,0.203530704390441,0.162654543848062,0.113324954481104,0.0604906271458651,0.00510523184497398,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.00616473078291425,-0.00610066333425712,-0.00603015364821501,-0.00595182093539118,-0.00586467689248673,-0.00576772025089579,-0.00565960453575297,-0.00553825594403451,-0.00539998274717133,-0.00524437154703967,-0.00506949083890941,-0.00487141730915921,-0.00464583089690028,-0.00438820817359881,-0.00409614246749424,-0.00376310497336022,-0.00338435077841875,-0.00294460204171663,-0.00242902427204607,-0.0018176185791688,-0.00108087124949804,-0.000164598909727697,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.193882613536484,0.193801682784499,0.193920083151664,0.194050651543863,0.194193638367673,0.194355293354051,0.194536979054084,0.194742295564645,0.194947871219207,0.195118916735766,0.195278496053024,0.195408721861219,0.195489715342907,0.19553823938334,0.195633960111507,0.195700384363089,0.195820905861999,0.195927434438082,0.196100619920087,0.196263262692043,0.196474173139113,0.196767623286317,0.197024990261825,0.197225134380034,0.197118803146308,0.196821363979796,0.196587503453404,0.196196450554113,0.195730103672346,0.195174880789693,0.19440039253602,0.193729061119763,0.192950122384783,0.192114598149357,0.190980968762232,0.188218469100979,0.184603587618347,0.180752482322642,0.176752625978582,0.172206269899441,0.167080884368012,0.161197480876744,0.155365858688226,0.150377915606901,0.145079274958059,0.138707718710092,0.132970078782759,0.128238820674244,0.122949504038394,0.117923206826859,0.111645085172829,0.105707721760756,0.0997715629598503,0.0934884866751658,0.0869078977006126,0.0795619640949961,0.0713121834628181,0.0594737270298864,0.0457272816769099,0.0283826986696994,0.00441312885828623,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-1.59476519061258e-05,-1.59223980761254e-05,-1.5906267481591e-05,-1.58886494715227e-05,-1.58694100815048e-05,-1.58483330808443e-05,-1.58253083059548e-05,-1.58010461521949e-05,-1.57774550111364e-05,-1.57529666318041e-05,-1.57294492283843e-05,-1.57053125366305e-05,-1.56752586347044e-05,-1.56390371368599e-05,-1.56045719386199e-05,-1.55629728894712e-05,-1.55178647873223e-05,-1.54651341521699e-05,-1.54120946151878e-05,-1.53507070746025e-05,-1.52850476031335e-05,-1.52174242197088e-05,-1.51431754799764e-05,-1.50671298324182e-05,-1.50282660292657e-05,-1.49843782144145e-05,-1.49961123927138e-05,-1.49743861224601e-05,-1.49123258675397e-05,-1.48322164375366e-05,-1.47033328059765e-05,-1.46298985733335e-05,-1.45451932729795e-05,-1.44711321627144e-05,-1.44212388044145e-05,-1.43960585358536e-05,-1.43688408797204e-05,-1.43526409171638e-05,-1.43651801362565e-05,-1.43815376751479e-05,-1.43665838651878e-05,-1.35113719976366e-05,-1.25007323563653e-05,-1.11503940782909e-05,-9.70443977490758e-06,-8.12833711391968e-06,-6.39773561761597e-06,-4.5462925062729e-06,-2.59877520289968e-06,-5.37508421038185e-07,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.000380287504290705,-0.000379049099390708,-0.000377448029639883,-0.000375697132202879,-0.000373784403333705,-0.000371695231157986,-0.000369413924372945,-0.000366900250696798,-0.000364066968940606,-0.000360986634701206,-0.000357617705923236,-0.000353966895242107,-0.000350073063278072,-0.000345911130868172,-0.000341304235104312,-0.00033636932307045,-0.000330975533924766,-0.000325170204363159,-0.000318846000534596,-0.000312035752459564,-0.000304672878495822,-0.000296717677819449,-0.000288113979735436,-0.000278639664234007,-0.000268996623621174,-0.00025896068492812,-0.000248203875088576,-0.000237556141280021,-0.000227405296408634,-0.000216861071214132,-0.00020583694755304,-0.00019447738973839,-0.000182687899099433,-0.000170663788642626,-0.000158363492720348,-0.00014551680594294,-0.000132249344968577,-0.000119072722731024,-0.000105675827604575,-9.22915597251068e-05,-7.90821224477542e-05,-6.57279444068654e-05,-5.31168241340954e-05,-4.14326386534757e-05,-3.10241993649002e-05,-2.15339473285194e-05,-1.31024177335986e-05,-5.5000912913249e-06,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.000100158031516349,-0.000100085207090165,-9.97550422696631e-05,-9.93927412718807e-05,-9.89961698622147e-05,-9.85599446775748e-05,-9.80802008360455e-05,-9.75407578405538e-05,-9.69429791213375e-05,-9.63188815967872e-05,-9.56084760230401e-05,-9.48326337525953e-05,-9.40654829716518e-05,-9.33012745068831e-05,-9.23788675808151e-05,-9.1444948373011e-05,-9.03849679083713e-05,-8.92780788522348e-05,-8.800896312771e-05,-8.66660979564522e-05,-8.51845681854747e-05,-8.35302764893439e-05,-8.17639832368346e-05,-7.98141429171859e-05,-7.77063940373351e-05,-7.54394393914511e-05,-7.24426489123567e-05,-6.94534935446601e-05,-6.63394028777043e-05,-6.30853981553513e-05,-5.99781139877848e-05,-5.60981627935853e-05,-5.19843474664139e-05,-4.74224302761756e-05,-4.22326583153026e-05,-3.65092425397578e-05,-3.02829855835614e-05,-2.36938485523831e-05,-1.63940092109487e-05,-8.51482788860006e-06,-8.77106716332791e-08,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-1.34656632603684e-07,-1.34542995950353e-07,-1.34378281763105e-07,-1.34194961632387e-07,-1.33990328797096e-07,-1.33769942984407e-07,-1.33529782241333e-07,-1.33229603268896e-07,-1.32562294076327e-07,-1.31684073779819e-07,-1.30743932286691e-07,-1.29753412803543e-07,-1.28704189763553e-07,-1.27566539587649e-07,-1.26289700378719e-07,-1.24945224553136e-07,-1.23449853931869e-07,-1.21858440475496e-07,-1.20088043002787e-07,-1.18214338832955e-07,-1.16152895283045e-07,-1.13857238927536e-07,-1.11397187088852e-07,-1.08620867069479e-07,-1.05378605243629e-07,-1.02669820701495e-07,-9.98841346144656e-08,-9.72202616067165e-08,-9.37558646979314e-08,-8.99847675738296e-08,-8.58044384069598e-08,-8.14711420313726e-08,-7.67604952531773e-08,-7.17654982178128e-08,-6.6748157141511e-08,-6.13489485249733e-08,-5.57242712294431e-08,-4.92620617679098e-08,-4.25953025396244e-08,-3.5654567396602e-08,-2.84626472137499e-08,-1.24271172778385e-08,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[4.1373318904285e-06,4.13470450169872e-06,4.12924025385064e-06,4.12322174021809e-06,4.11659343097707e-06,4.10932807183005e-06,4.10134873118392e-06,4.09241377615464e-06,4.08252550030244e-06,4.07163176355728e-06,4.05863316748903e-06,4.04336822363916e-06,4.02671853327036e-06,4.00899334264743e-06,3.98864040695578e-06,3.96687367105363e-06,3.9431796510258e-06,3.91737079728146e-06,3.88890781076126e-06,3.85761531613602e-06,3.82343907542554e-06,3.78622295233747e-06,3.74458309627524e-06,3.69562312557374e-06,3.63763923407575e-06,3.56927635143351e-06,3.48768738241575e-06,3.40932824505387e-06,3.34062453660487e-06,3.26384453869211e-06,3.18353600559749e-06,3.09170705761011e-06,2.99277493856927e-06,2.88516224145268e-06,2.76616209335224e-06,2.63311056747506e-06,2.48997965870276e-06,2.33715354454931e-06,2.17332672499114e-06,1.99614329910418e-06,1.79432362493025e-06,1.61254210307287e-06,1.44585735108405e-06,1.32622110946816e-06,1.19603247073337e-06,1.05083694614646e-06,9.06613161404714e-07,7.76595871713336e-07,6.57308481225434e-07,5.61388358482484e-07,5.17025966943212e-07,4.75014641833248e-07,4.3336001281417e-07,3.84816003763601e-07,3.35438156856685e-07,2.82116123513829e-07,2.24416209041932e-07,1.55980999557498e-07,7.56736383335653e-08,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.0683847970352286,-0.0682910133399163,-0.0679705945196887,-0.0676177302784381,-0.0672318430613752,-0.0668074853960946,-0.0663395920982323,-0.0657616284715637,-0.0647921805907766,-0.0637590442489387,-0.0627290302806083,-0.061731673368004,-0.060773938859366,-0.0597882228884916,-0.0586325118810653,-0.0574475626123085,-0.0560549769055784,-0.0546079278050436,-0.0529429707781045,-0.0512148862617095,-0.0492777753874216,-0.0470706423968494,-0.0447616022586216,-0.0423548662429554,-0.0402654625084291,-0.0394519176675481,-0.0386957803449649,-0.0376313557499414,-0.0362570588012229,-0.0347958262450317,-0.0333362056332076,-0.031662928792063,-0.0298716560790532,-0.027924828394528,-0.0257911495812151,-0.0234835396781984,-0.0208847507445089,-0.0180364852163231,-0.0148970952014657,-0.0114872132445536,-0.00791649286953216,-0.00445718465712107,-0.00111315380368444,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.463555740295291,0.462442397910289,0.46281094808791,0.463209987061253,0.463612126285606,0.464093692902086,0.464641042780624,0.4652845143525,0.46521627523048,0.462649700931383,0.45886143470845,0.454766553232151,0.450877519439553,0.447097903339844,0.442346103683553,0.437429669676414,0.431238913663176,0.425122507631686,0.417770062752722,0.410461602235625,0.402405898161231,0.393289294077172,0.383696317038614,0.375107520678147,0.363972602213026,0.346527110358473,0.321666127796812,0.296475473411668,0.268728034284038,0.241927595147969,0.21394332363342,0.177003504048058,0.136279137066601,0.090202018921609,0.0385639825347193,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.0450846418954491,-0.0450335965175493,-0.044965817471431,-0.0448914621264551,-0.0448099238942153,-0.0447200866096567,-0.044621198907112,-0.0445131751449064,-0.0444035470613232,-0.0442864526171206,-0.0441552603249358,-0.0440122495222643,-0.0438625811506031,-0.043704799487715,-0.0435246120126248,-0.0433316968683609,-0.0431118501351977,-0.042873313613455,-0.0426053731982982,-0.0423128644574275,-0.0419884730701126,-0.0416282797346686,-0.0412368420970224,-0.0408327394849804,-0.0404443585075865,-0.0399748976998242,-0.0394242617624114,-0.0388077694983922,-0.0381536027532027,-0.0374733218411176,-0.0367772918481058,-0.0359738162581832,-0.0351031225707569,-0.0341438152607745,-0.0330748988116511,-0.0319535055564148,-0.0307813880767847,-0.029485708931522,-0.0280266890511339,-0.0263970005362787,-0.0246164349138277,-0.0235600042210904,-0.0221843648230311,-0.0202689320713605,-0.0183011931496865,-0.0161418377428845,-0.0137448305459182,-0.0111365883398099,-0.00840410307066212,-0.0056009997225485,-0.00287093489450817,-3.08423251085886e-05,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-1.47500638468259,-1.47116104420926,-1.4657611513825,-1.45985822385439,-1.45340114703886,-1.44635545265792,-1.43866734871234,-1.43024266730902,-1.41990743066513,-1.40764400504574,-1.39432097737419,-1.38039653878409,-1.36603831440571,-1.35094621959223,-1.33385330746005,-1.31581295799314,-1.2955010231333,-1.27388149179386,-1.24968331140021,-1.22388051460415,-1.19539715330776,-1.16377047909875,-1.13004248794718,-1.09741657642785,-1.07242136042159,-1.04405019319748,-1.01470700798437,-0.980660359387567,-0.944894391602111,-0.903679757709298,-0.860921789355135,-0.814493863136036,-0.764657431486055,-0.710799161584543,-0.651723905040487,-0.590117010597086,-0.526018683098024,-0.455061373869944,-0.371560726497457,-0.279501167588276,-0.175647534692135,-0.0629052368056209,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.000690623060213959,-0.000690778554140578,-0.000689763286581991,-0.000688704381528156,-0.000687407712437739,-0.000685930267388564,-0.000684548548540662,-0.000691522010686553,-0.000740046768868643,-0.000796112761524615,-0.00085068646299659,-0.000895712863636499,-0.000927735812094618,-0.000954650537435785,-0.000993323098194041,-0.00102720367471876,-0.00108715686910029,-0.00114225668966539,-0.00121291998709341,-0.00127699169871119,-0.00135197032232569,-0.00144214828729205,-0.00152072663222147,-0.00158322534296025,-0.001673801144777,-0.00164698690350512,-0.00161417240862827,-0.00171411745476349,-0.00199936548201153,-0.00228379984870107,-0.00254106054637899,-0.00285786456575293,-0.00319239215065699,-0.0035594857669229,-0.00397401788139571,-0.00440862558243403,-0.00491317291823961,-0.00549097009859225,-0.006113712805821,-0.00677777207587922,-0.00733491644458103,-0.00758904798256639,-0.00787646221731849,-0.0077900516569842,-0.00734052971977174,-0.00677001691035897,-0.00607701998244642,-0.00536661599055833,-0.00465212902769691,-0.00386472260265318,-0.00295961390010333,-0.00196886947524619,-0.000836450244371119,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.0445528688505679,0.0462020900233977,0.046334359232878,0.0464213883188733,0.0464894211517475,0.0464449350791676,0.0462703289827308,0.0457393010184801,0.0465282523693085,0.0500430083162226,0.0541860590554895,0.0581315312480647,0.0618037094579577,0.0653993569591423,0.0695013659672488,0.0737313077858447,0.0789602390685343,0.0838363957128169,0.0894775817199588,0.0945136470366861,0.099966357741094,0.105916766168766,0.109620336201893,0.102579340267929,0.0801157364103451,0.0545160091659633,0.0198855154596188,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.0700705962840074,0.0700088557174875,0.070426775376118,0.0708844519009911,0.0713808871332393,0.0719288492360124,0.0725324170597011,0.0732465533024671,0.0742464880830102,0.0751789986625824,0.0760041710254911,0.0767307457575014,0.0774006501734766,0.0780916176067273,0.0788800536296463,0.0796508136857078,0.0804767815503285,0.0813039225599523,0.0822857003482137,0.0832502073482452,0.0843379824275555,0.0856414821517273,0.0869770755878023,0.0883844471300282,0.0897638105310335,0.0894484182030556,0.0892390994735434,0.0884286866446942,0.0870669346146547,0.0850545402708935,0.0825725787579423,0.0798069513406721,0.0768000862985355,0.0735071916207801,0.0703313987540223,0.0659379779392575,0.0586286828910808,0.0505581357125191,0.0410543310622393,0.0310055527777745,0.0208939179902223,0.00982838563680671,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.0613180298406043,0.0611412197969695,0.0601169799846077,0.0589917641953381,0.0577630636095672,0.0564049345316363,0.0549060414652836,0.0531736300595219,0.0509054274281135,0.0486860715527561,0.0465196994625905,0.0444213943881345,0.042394819181862,0.0403113367326068,0.0378890390869425,0.0354088310015775,0.0325939557496362,0.0296375013459283,0.0262274894587532,0.0226403592896254,0.0185991869874379,0.0139573731216746,0.00902131529193277,0.00399653830959799,0.000296304745846334,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.246795630215744,-0.243826878945496,-0.238500271233707,-0.232671425297243,-0.2262864203203,-0.21925539829206,-0.211546903977496,-0.203716076994867,-0.198204810184159,-0.19242074519545,-0.185491569599278,-0.177367893772692,-0.168558411951113,-0.159342711179674,-0.14883602157399,-0.137642013219317,-0.125594733299639,-0.112168685536609,-0.0971830226024885,-0.0802367338474127,-0.0614056827308404,-0.0404134841864309,-0.017058365162223,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-6.74374765425195,-6.74446393936143,-6.74587784023647,-6.74737647082223,-6.74890472708336,-6.75055402601054,-6.75233876304676,-6.7552445960579,-6.76184883739136,-6.76730883028798,-6.77159538225356,-6.77418897374304,-6.77483468596539,-6.77435609970982,-6.77425592515645,-6.77299443268717,-6.77406654063159,-6.77366008893836,-6.7736414431978,-6.77158015382494,-6.76888731469425,-6.76565144974868,-6.76025216951939,-6.75650795937233,-6.75545354691653,-6.74083661420981,-6.72511703087251,-6.69693422532975,-6.65724705582593,-6.60491652206856,-6.53686916272617,-6.45475130052725,-6.36130881544219,-6.25541497002173,-6.13606116214055,-6.00779410160221,-5.86328505351532,-5.70200182148539,-5.52033076738909,-5.31936781723106,-5.09937048009964,-4.85660328818855,-4.59490348731332,-4.29141650878077,-3.94254138608274,-3.55124906982691,-3.11710992759519,-2.6674553948326,-2.2097763355644,-1.73547934431102,-1.28112908610648,-0.814325187873766,-0.344707429096892,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[3.38406895305989,3.38948049464339,3.37969426102957,3.36907108762932,3.35761847918157,3.34503202595519,3.33132315619666,3.3165589998428,3.296494808942,3.27753332811791,3.26457931137563,3.25790720443997,3.25342512831125,3.24919741091616,3.24632397626855,3.24365492450805,3.24413799681067,3.24612926300535,3.2518284481671,3.26129081934062,3.27473138703017,3.29451408695911,3.31936534388167,3.34951881851642,3.38155325227364,3.37599876679102,3.40311377491312,3.43212958718146,3.5208773733471,3.61702888817271,3.688940393931,3.78497062391271,3.87908646119145,3.98375705412908,4.16894886294044,4.35061594257788,4.56419140930074,4.81727982417366,5.06203049208639,5.35050475114296,5.70321005072835,6.00917294524455,6.3490104420422,6.62704218981049,6.79599287395955,6.78621070058707,6.7074664724142,6.71340794444153,6.52069092291552,6.27180795682669,6.0388762213708,5.77766112873057,5.58481107667047,5.3501225819964,5.10585085050542,4.84397532971611,4.5674347989686,3.94046175191089,3.02991346033923,1.2720476539504,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-87.551125587034,-87.467242423204,-87.3283703127353,-87.1761919443562,-87.0095406947046,-86.8263111924957,-86.6249638971218,-86.4043885429038,-86.1812999019882,-85.9473733164978,-85.6865896806615,-85.4000069398449,-85.0948338391159,-84.7685719280632,-84.4046806803836,-84.0175140832912,-83.5966425477265,-83.1438966503287,-82.6355833173147,-82.0863885038072,-81.4793904012435,-80.8044279335721,-80.0679572674291,-79.2650685415087,-78.3531068431887,-77.5017559615029,-76.4974223206095,-75.2796642938869,-73.5737764398276,-71.7755466539822,-69.9498123824608,-67.8021202386701,-65.4702873542963,-62.9168451594203,-60.1236300983559,-57.1074009028131,-53.8432724280418,-50.3083670134031,-46.4286532313189,-42.086455498658,-37.4859307334696,-32.7326204199415,-27.8306025236291,-22.9366222602376,-17.7279686628825,-12.327936791016,-6.98013367608783,-1.27867940181255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.00041322070469487,-0.000412852675961729,-0.000412251505681791,-0.000411592507644871,-0.000410869129880529,-0.000410076850910164,-0.00040920941240653,-0.000408280582946818,-0.000407422548112024,-0.00040649445273365,-0.000405375403756899,-0.000404036805013084,-0.000402533040843392,-0.000400897761098975,-0.000399080715499281,-0.000397067154999262,-0.000394918091457566,-0.000392545099999131,-0.000389974934978068,-0.000387126589631002,-0.000384039775085376,-0.000380717168617108,-0.000377050768395991,-0.000373037616525801,-0.000368492596105683,-0.000362872800688934,-0.000356371082471064,-0.000349258960910522,-0.000341228576436201,-0.000332647603789098,-0.000323640623673665,-0.000313395944293605,-0.000302316609166659,-0.00029023554425607,-0.000277006545236896,-0.000262403934069352,-0.000246857249121907,-0.000230109897125304,-0.000212052783161236,-0.000192957658048324,-0.000172435942381098,-0.000152191186364007,-0.000131883451981778,-0.000114340591341634,-9.541009560776e-05,-7.44796822189632e-05,-5.30649759345678e-05,-3.13696663192179e-05,-1.18577255782013e-05,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-0.39496938547231,-0.39042040713722,-0.383927027399452,-0.376974363589494,-0.369546627413556,-0.361624473083242,-0.353196703896524,-0.344258390863695,-0.334876732821361,-0.324993142159062,-0.314521120501496,-0.303570617303885,-0.292331014144363,-0.280776892944438,-0.268617523880473,-0.256132989100959,-0.243217873514823,-0.229999108285522,-0.216494297831495,-0.202784079677054,-0.18894044341276,-0.175069369129786,-0.161227868222569,-0.147499682911894,-0.134025599159737,-0.120828848875428,-0.108095500483978,-0.0958463786393922,-0.0842134806827658,-0.0737960194908083,-0.0644750514042782,-0.0556862571710122,-0.0475860218027314,-0.0401836883514133,-0.033512855090476,-0.0274747426440938,-0.0220821794946478,-0.0174860578324353,-0.013752080553104,-0.0108801212266788,-0.00873680824113594,-0.00709553496266875,-0.00577917352237104,-0.00470227566395844,-0.00380934282256871,-0.00305004609304187,-0.00237995753382842,-0.00175890574256253,-0.00119694355658388,-0.000674425056732169,-0.000175899042053998,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0.00402119493017643,0.00398126037506296,0.00393978192171021,0.00389441736827583,0.00384478002867693,0.00379047966548033,0.00373108945009134,0.00366657837304351,0.00359760441409226,0.00352082171601842,0.00343582550028079,0.00334234308280742,0.003240125429781,0.00312832520311673,0.00300524656526092,0.00287054185591836,0.00271981750304936,0.00255360336531672,0.00236909378956688,0.00216460475709771,0.00193665745656912,0.00168136856933386,0.0013961222653222,0.00107688073672736,0.000703290446323505,0.000271403971556329,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[60.6291495500168,60.7863243145987,62.0937259969987,63.5139509662806,65.0449612064413,66.7544125058769,68.6442356326203,70.6163288492893,71.4813838992765,71.2686809945083,70.7248804388496,70.0479068632362,69.3135423852894,68.4871190380681,67.4172459083555,66.2398159877079,64.8209894471886,63.3550507543485,61.6755447037606,59.9422897262294,58.0056227435741,55.7759812953308,53.3420184771344,50.5979383823237,47.5394073369115,44.3321274560232,40.725177677888,38.1719333280212,36.2174703246252,34.0185704205073,31.8035783806357,29.0638438658647,26.1323246813809,22.8694253985334,19.2093827152058,15.4579848136383,11.5768242767457,7.46856750573393,3.25089855113521,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[85.2209426887244,85.3538188567046,86.4450894735656,87.634155752066,88.921064522918,90.3538619385245,91.9364289217802,93.6195883900326,94.7610283621559,95.2766394238542,95.5782421131619,95.7535076469499,95.843492041454,95.8747684150326,95.8474729444969,95.7493854462518,95.5631959920242,95.3661882446535,95.1458793532172,94.9114959562355,94.6553086485063,94.3761391432425,94.0783128598298,93.7508489136531,93.4137500525695,93.1382753552404,92.83180340142,92.535012859262,92.3340103461606,92.0071497937149,91.6629481142839,91.2788436064318,90.8734225096472,90.4480592071779,90.0142032171795,89.5837589063776,89.2408659952594,88.8728199572934,88.4587850263315,87.9547612959565,86.9682469181738,85.6513256748298,84.2421785146695,82.905974790282,81.3995507139763,79.4782867409862,77.6303517641592,75.9407515748699,74.0948399299731,72.1610597299245,69.9846741278144,67.549191638859,64.7558148722989,61.5305189737661,57.905215910268,53.8723810422241,49.3725731500049,44.8690718252304,40.1550410684039,34.3569940769232,27.3964596409166,19.5991239975132,10.8952890067966,0.620075678137024,0,0,0,0,0,0,0,0,0,0,0],[-159.121190075737,-158.805128772951,-157.31194958642,-155.683604978006,-153.917540536989,-151.959336009622,-149.801808421558,-147.492505483617,-145.702918070133,-144.539369006929,-143.554166914495,-142.647368473561,-141.775274250778,-140.902132380135,-140.005181439243,-139.098062876826,-138.202633277911,-137.213042897102,-136.155483808307,-134.988961212331,-133.727911875101,-132.374656513618,-130.909661876454,-129.317677165807,-127.639555799148,-125.816629001035,-123.95903241726,-121.764674430793,-119.092514556514,-116.240325337516,-113.256594742975,-110.155673690812,-106.831940685185,-103.28720296722,-99.8662282366939,-96.3148726586917,-92.480797675602,-88.3418275777969,-83.9790022632873,-79.5026527392268,-74.938395705283,-70.1224087682273,-64.8441065111473,-58.4099629242834,-51.6218157134726,-44.4370964255053,-36.6420864103809,-28.8592449035457,-20.752420283165,-12.455654948168,-3.81980655155323,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-20.3852696124717,-20.2402316038598,-19.0844042122985,-17.8245923992,-16.4602554603978,-14.9425707090527,-13.2669241473569,-11.4791979367517,-10.2039020742072,-9.53337479648035,-9.06878720674507,-8.72911345901035,-8.47466563899014,-8.27110907713466,-8.10002902445707,-7.98686121642107,-7.93324814026473,-7.87638763679051,-7.80760917055439,-7.73657659833196,-7.65323583362313,-7.55003913009638,-7.44593646743869,-7.3795361708383,-7.31914472867803,-7.226374807236,-7.10228048982108,-6.84146536110757,-6.33374664342868,-5.65927632293942,-4.90981737743193,-4.06148506682451,-3.13442359276318,-2.10725530663627,-1.09776309440525,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[16.811198959935,16.8250908809684,18.1201499731801,19.5325497809643,21.0614042680718,22.7633151321184,24.6430911259959,26.6603239234941,28.1390641705942,28.860670897914,29.2969173721488,29.5667560354301,29.737604307617,29.851788049557,29.9020749063056,29.874914131288,29.6808440375955,29.4807609054379,29.2212778026727,28.9466676294152,28.623114237955,28.2341465645724,27.8404997461998,27.6790409913766,27.8035895546438,27.8010131297016,27.6443378345155,27.1175297698902,26.2643928188948,25.4262721946265,24.6389394300392,23.7324061253421,22.7572097282097,21.6830454809784,20.3919449607622,19.1245298297039,17.7934754103042,16.0802910576959,13.8080033699587,10.8394358319804,7.05645454499084,5.04330931574629,2.94527839751131,1.22773956388416,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-478.877328214262,-478.300152504705,-476.441287310137,-474.41112321813,-472.202261206504,-469.761420448084,-467.076092138339,-464.180392544252,-461.667154125451,-459.557596554718,-457.442120155838,-455.288099337848,-453.144131146813,-450.972897230048,-448.55156667511,-446.06999369442,-443.370864054759,-440.483067559891,-437.278797460978,-433.826118741254,-430.050070520058,-425.918805922813,-421.432479765286,-416.491475308635,-411.175222149252,-405.295814269747,-398.736020366857,-391.99685670988,-385.421850739666,-378.113660695875,-370.208579009046,-361.546797407158,-352.180961481084,-342.026645445688,-332.265980179164,-322.11679343109,-311.804168842146,-300.793099638072,-288.930155316909,-276.058742070485,-261.880354925449,-246.607944912282,-230.753876653652,-215.225357676885,-198.751736082646,-180.928695889302,-162.610727243171,-145.869051805805,-129.211737998883,-112.614903609207,-97.6659199117344,-82.6490288281333,-67.5804152476722,-52.8063802022709,-38.656093226857,-24.7383807962121,-11.221402701666,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[41.4695726227675,41.4758687359603,42.4881157485596,43.59020007436,44.7805842931903,46.1096360768486,47.5797958669106,49.1364800465972,50.0678369633743,50.2810151366982,50.2316509269964,50.0220415431634,49.6992849623644,49.2841493883677,48.7559465410233,48.1178469474003,47.3291233698954,46.4875583539529,45.5494083225263,44.5480286582505,43.4484725947641,42.2329057998025,40.9210022938248,39.5677429359449,38.3245526126357,37.0948344190865,35.7537714847678,34.4569360453448,33.232287689384,32.0072797921193,30.8504439936244,29.819333026866,28.697895372306,27.490637585268,26.1100804169243,24.5492528210911,22.9890933847672,21.2402136780774,19.5884052880472,17.9758564167754,16.2285252900278,14.3944094130039,12.3792624099385,10.20215521588,7.47575698494717,3.89256035887021,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-71.3576198228902,-71.086945422349,-69.7649256887919,-68.3236904827709,-66.7630644630979,-65.0273938847629,-63.110656178324,-61.0484574881303,-59.5423013085622,-58.7467689492053,-58.1632395239073,-57.6915926601091,-57.3064956719152,-56.98069940998,-56.6773989051952,-56.4566213876779,-56.2526080424576,-56.0438136865486,-55.8004275367791,-55.5423727380934,-55.2542620190982,-54.9305763805773,-54.5380289884951,-53.9574985818512,-53.1883985881037,-52.3520087184142,-51.3731464040746,-50.8724606692475,-50.8574134544099,-50.7681951640361,-50.6593010750004,-50.5738711412279,-50.5011761810114,-50.4123020731365,-50.368459774656,-50.2547355430465,-50.045074204828,-49.8680872326254,-49.7522351389111,-49.2642573267046,-47.4911380687851,-44.6610901621491,-41.9272240268443,-40.2732030953244,-38.4848731728397,-36.0754162576149,-34.5523297137959,-33.196639589611,-31.6443704263632,-29.9069942408648,-27.9299762168689,-25.7066818879823,-23.1289801339569,-19.997753435703,-16.258758510463,-12.2983834684178,-8.15825483236493,-3.77128751014065,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[78.4323423899766,78.3896785338906,79.2555808216629,80.1978131494428,81.2148485321932,82.3503608774822,83.6057627642743,84.9225621802124,85.6014511641243,85.601888964461,85.3627825297236,85.0090613091605,84.6025511285294,84.139418586926,83.5321642773442,82.8535549282224,81.9534935786628,81.0238012549169,79.9375502245121,78.8096518509369,77.5357488046252,76.063669105993,74.4755506950267,72.8925136324888,71.5811323883244,70.3039593418366,68.8593005971083,67.6759811683034,66.6802064454259,65.5431302257605,64.4081439919316,63.029386047304,61.5594303718679,59.9182320376685,58.0130710390364,56.0714249109415,54.0552483822778,51.7358876465886,48.7963530776224,45.6435061238992,42.7467724208162,40.385054752573,38.2495106945088,37.7215805628922,37.3498896319918,36.791044765227,36.3089139674183,36.2549767743001,35.4549885233964,34.2880243034234,32.9813961781084,31.5095388266614,29.8217959200805,27.789277483087,25.2937859568476,22.619326478613,19.7708015022021,17.6807175546819,15.2964710690883,12.9393350844725,9.53731000813086,3.81918897670404,0,0,0,0,0,0,0,0,0,0,0,0,0],[-22.9412013307964,-22.7346729024765,-21.4836858672177,-20.1199438070745,-18.6426162020059,-17.0001410135452,-15.1871837195751,-13.2490751898663,-11.8443756140586,-11.0795994026112,-10.5181022786823,-10.0691434021132,-9.69725213201917,-9.37097051978595,-9.0685278471485,-8.82102884836572,-8.61603363043338,-8.39362435173611,-8.14927731138014,-7.88399892231374,-7.59359343701649,-7.27641638007767,-6.9251459822633,-6.54825218727051,-6.13835439488528,-5.72061700727398,-5.26492544756,-4.74144983153486,-4.14361048868146,-3.46646053938768,-2.70548358884007,-1.88505453676363,-0.984552559135889,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-118.859584125643,-118.453214010839,-117.05967111516,-115.537262706029,-113.881952398275,-112.046891396318,-110.022436317423,-107.816285813066,-105.766955658335,-104.053926258523,-102.435424680885,-100.929937100823,-99.5499703355688,-98.1961478671133,-96.6644351001069,-95.1486242796905,-93.3900953753816,-91.5655502106595,-89.4219669527676,-87.1935918485213,-84.6623350294047,-81.7019602487543,-78.4843236410616,-75.2651827429512,-72.8545738176391,-70.3331268884801,-67.4141202099369,-64.9551381914664,-62.6677016035762,-60.0510817223727,-57.3311652164755,-54.1422333620302,-50.7071512860866,-46.9016714874651,-42.752658077297,-38.5614853202179,-34.2108799210494,-29.1469249906826,-22.453146346425,-15.5071350196054,-9.63532375455693,-5.59787074372534,-1.48179026295744,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[18.3667465268209,18.3555610931763,19.2302700752962,20.1830468845378,21.2133936219586,22.3625979642995,23.632701177552,24.9696874387916,25.780872980857,26.0019358931229,26.0047796159002,25.8873390289707,25.7096290648653,25.4887064459578,25.1741461643755,24.8174566613449,24.2944639066935,23.7645965527683,23.1447847871715,22.5100096221414,21.8042281765078,20.9939334265414,20.1044053905423,19.0175224346228,17.7127538102366,15.960473522642,13.8089058509617,12.7076360757641,12.8353957821315,13.0964885681622,13.3268680307214,13.5555193493155,13.842773704045,14.2032582840539,14.7994724988498,15.2099997004381,15.4447992081499,15.668637983654,16.0631052284605,16.3793020352315,16.1740227634045,16.3094704148784,16.2154208730426,15.5834247573219,14.5719979922061,12.3493008709749,11.2234834661627,9.80183293921393,9.67181600453632,10.2902169426764,11.25485052787,12.4187304057558,13.2423309678713,13.3461464256094,12.3547380950584,11.1674029584796,9.72800367579044,5.28904519622391,0.808746591618897,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-114.908479983883,-114.328443598604,-112.6659676995,-110.850069820688,-108.875038509378,-106.686266736469,-104.273488925958,-101.686110256293,-99.6185231839226,-98.114411461223,-96.705899282745,-95.2752412478917,-93.7901796256075,-92.2345030375502,-90.5828255904123,-88.8534538629401,-87.0362007154687,-85.0273704066596,-82.8446136189889,-80.4339828978806,-77.8112680103431,-74.9795492258998,-71.9179440951869,-68.6015180933322,-64.9081065378513,-60.9320575380736,-56.6672997154394,-52.2217845034926,-48.0970129567176,-45.0494541088448,-41.9849715004774,-37.9933994572172,-33.5749210070543,-28.7968378685371,-23.9664010293183,-18.1806898623941,-10.9484221792019,-2.94664591965753,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[40.8560536483543,40.9795628773131,41.9576905260075,43.0225382750862,44.1736091086458,45.4566536947041,46.8745657299163,48.3768535203618,49.3303647981745,49.6546165492082,49.7464828594908,49.7009008751487,49.5758172582683,49.3981130139202,49.117178561396,48.7874063312188,48.3652930526253,47.9281604901504,47.4212994059389,46.8844372554461,46.285044674672,45.6069069238064,44.8388521415518,43.8634269255406,42.680825281243,41.417869598402,39.936711199846,38.9682386657809,38.6869739837789,38.690219704582,38.8255097340518,38.845440939637,38.8403839074863,38.8251427566865,38.3633793227706,37.7940689855266,37.0376266453251,36.2912810361762,35.5322024967922,34.459559450832,32.7590530237898,30.1853095180244,27.2467034271554,23.9116320636931,20.1974755734066,15.6735072584911,11.625362294676,8.44721339032222,4.68812779764147,0.5964044593416,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-12.4814692956479,-12.3260469626776,-11.1405705279693,-9.84856381095959,-8.44962212937854,-6.89340960941886,-5.17524462755711,-3.3418717614159,-2.05081383963265,-1.39937426742934,-0.956953930361653,-0.634111526865575,-0.394342512918067,-0.20865670760946,-0.061249196688782,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[35.3333844592658,35.2297842679121,36.0794102675699,37.004077089092,38.0014179900005,39.1170172136198,40.3521670346395,41.657732599431,42.3852201900925,42.4159780444735,42.1723355575134,41.7676163949478,41.2626635425901,40.6729111283035,39.9420049926757,39.1512662025627,38.1487478641253,37.0872841500824,35.8828864910997,34.5987847595024,33.1722678932405,31.5720816523852,29.8309531427963,27.9727539215958,26.0187696799353,23.8545280779041,21.3810841872918,18.9410079376803,16.5689524206717,14.1761094768666,11.7641289601907,9.10884990885327,6.17062444710243,2.92207816686991,0.544792507739653,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-29.4534299102214,-29.1504245159358,-27.9087463202572,-26.555227580699,-25.0882036387601,-23.4598833847924,-21.6646708450611,-19.7481072922229,-18.3431530528668,-17.5194911277326,-16.8628046810639,-16.2797625127837,-15.730510678084,-15.1920625615693,-14.6672104642667,-14.1776398941388,-13.7054105336418,-13.1791566403954,-12.623099879008,-12.0052149283983,-11.3434548479281,-10.6452664776523,-9.87615566731304,-9.04419515763797,-8.0962138815234,-7.04475616189968,-5.96300782667997,-4.58568599213446,-2.83968253931683,-1.09296250977786,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[47.8938906416248,48.1580823787366,49.5670177601552,51.1030001264986,52.7675488226923,54.6152840146695,56.6526058736138,58.8347604271737,60.4934222570839,61.5201349266813,62.3568770759591,63.0797625552793,63.7090050051564,64.2818950052965,64.8750169046597,65.3934716255077,65.9090650093125,66.4459176081839,67.044410675233,67.6728749643679,68.3568382729548,69.1067373202299,69.9019875287235,70.813282211581,71.8392873397256,72.8792713931903,74.021601718423,75.1002182248419,76.1031826580345,76.5911001221098,76.7425633680234,76.9060884489765,77.1264647131512,77.4169149232886,77.3584495107468,77.3144744182121,77.2236489860082,77.1397311721427,77.0935028683599,76.8616156710163,76.3958292176202,75.9431083653549,75.3057741163773,73.4170698255221,70.9059808101282,67.4342378859167,62.6192027733495,58.3575275193375,54.7264739513073,50.4500741031305,46.0878828189586,41.2428261933585,35.8654675146069,29.332017376091,20.9268655094596,11.2628429351164,0.0631538635056247,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-84.4022951654992,-84.2771369834507,-83.0794989712021,-81.7737530310273,-80.3587798961156,-78.7850806386265,-77.0477769465827,-75.199874351793,-73.9111579820453,-73.2373178459133,-72.7522958522835,-72.3700467096575,-72.0595052718825,-71.7946087593469,-71.5544699176414,-71.3698217889242,-71.2497960204236,-71.105896292754,-70.9605648866626,-70.7870376338074,-70.6041545904876,-70.4220706396046,-70.2365778061436,-70.0559903876387,-69.8280200804339,-69.5241220565431,-69.2055776163141,-68.5701787694623,-67.6423008140834,-66.7758613969625,-65.8962265112283,-64.9043669888058,-63.8023344827226,-62.5946410827955,-61.4226993193167,-60.1321668723752,-58.2585343787672,-56.2402542434405,-54.0885329536863,-52.0154983874922,-50.2699502279992,-48.6597328573505,-46.8926864369939,-44.9302676759536,-43.0788366206278,-41.4144665438806,-39.3956369037394,-36.47161691633,-33.7839745180204,-31.1094249975587,-27.6180581886661,-23.865497299733,-20.2062894831707,-16.5097240377795,-12.809498916161,-8.90536782947044,-4.76886381338838,-1.4641549028751,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-25.6285079902562,-25.4086541903581,-24.1655041822223,-22.8103589492723,-21.3422534273646,-19.7103770250817,-17.9094237548926,-15.9844888518309,-14.5784170986268,-13.7968373551869,-13.2183119566838,-12.7537857217389,-12.3612987915762,-12.0064067911787,-11.6775779680641,-11.3423840575922,-11.0587376500171,-10.7474937772323,-10.4125669177215,-10.0494968305207,-9.65698354166286,-9.23311594176222,-8.79345378435941,-8.41579704610628,-8.15346922270317,-7.93355713981937,-7.74058783098682,-7.37602035349583,-6.85162576842258,-6.21801220968212,-5.44696514442286,-4.6502278346741,-3.78140288180566,-2.84463541892969,-2.01671852688752,-1.07000836708251,-0.0599483002421891,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-121.658845071121,-121.003421399177,-119.138800385114,-117.103001490624,-114.889330640393,-112.438049640207,-109.736524401677,-106.812123348121,-104.171390400748,-101.92759426163,-99.7596016757707,-97.6918246919477,-95.7579451070086,-93.8517874188094,-91.7008712670704,-89.6347328301687,-87.1888043273598,-84.6468719283158,-81.6718163818246,-78.5703247895419,-75.0649658432978,-70.9967584642678,-66.6371324277918,-62.4994449343377,-59.5911470672444,-56.3824512823277,-52.5717191639282,-49.543760203907,-47.3956665036024,-45.1072593939012,-42.784780426616,-40.0606435547251,-37.1284803689215,-33.8948722694853,-30.4473174957477,-26.8729969419127,-23.2522616434171,-18.7986929929709,-12.4237322781407,-5.54373449313678,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[14.8924219272924,15.0177094125662,16.1210117177658,17.323070244256,18.6237491011742,20.0719140905861,21.6714942977571,23.3750930121685,24.5290615489797,25.0374094827365,25.325263848975,25.4860261237954,25.5633746806377,25.5832866260981,25.5393724460714,25.4141497019504,25.2033780674245,24.9815730350232,24.7292862237197,24.4613155487306,24.1643969395198,23.8319117087178,23.4643383249127,23.0599356120019,22.5950866812959,22.1196188398281,21.565558577713,21.0218536973533,20.4709106650611,19.8510608227531,19.1941665663512,18.417361642783,17.5607502629052,16.6059428017679,16.0441285963981,15.3815334902465,14.6555414192823,13.842367176853,12.9580947756175,11.9586078509151,10.908140492878,9.63679263973024,8.21297591495485,6.45740856321061,4.51140885893463,2.63459142459483,0.345925015683176,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[8.80666765614738,8.83422553548906,9.82225050484223,10.8972154523298,12.0580735754741,13.3532238481763,14.7846771606633,16.2873926772261,17.1033692984648,17.2074872115182,17.0701981322887,16.804360292481,16.4609214300283,16.0452449658998,15.4977748339401,14.8648398020885,14.0566331449358,13.2141206504324,12.2470644399458,11.235246559181,10.1042359576477,8.81321978813087,7.41251897482324,5.87323797756365,4.31888476244794,2.79088310084438,1.06322354852807,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-0.311368860086484,-1.24170491430098,-1.49594108528084,-1.38572078045903,-1.38798626373593,-1.28373768692682,-0.806307633605106,-0.257908679843116,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-42.2552737784716,-42.0383359791716,-40.7670452588125,-39.38063164529,-37.8776998263573,-36.208113032151,-34.3659965729528,-32.3930358551968,-30.9066332482734,-30.014480712877,-29.3076619728973,-28.7033491224328,-28.1654580551737,-27.656941525134,-27.1484125374859,-26.6669227907876,-26.2091088752242,-25.7103453565715,-25.1634492125911,-24.5689137670105,-23.9166270921184,-23.197246118137,-22.4246175388663,-21.5903822240028,-20.8216275317855,-19.9845659370839,-19.1100034853939,-18.1739415897542,-17.1043169414339,-15.8704789082418,-14.5255636957943,-13.0893467393664,-11.5517659839176,-9.90935850905894,-8.28527207998595,-6.54149761796789,-4.61177549372017,-2.54311440199639,-0.191487531793926,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-44.8252704490385,-44.6535811637035,-43.3783562244495,-41.9882849318574,-40.4823378413429,-38.8087110598916,-36.9619410721254,-34.9890496377838,-33.5588145533064,-32.7546245662037,-32.1421774870186,-31.6426191381995,-31.2308525201426,-30.8749892231298,-30.525952699559,-30.2517366039113,-30.0114757592228,-29.7579954378331,-29.4752600353385,-29.1737366554109,-28.8459999249832,-28.4913929402115,-28.1096056264605,-27.6917002521418,-27.2049025005578,-26.6834774742026,-26.0960049472024,-25.4173500485652,-24.5768983302747,-23.6567224983207,-22.7276522618599,-21.7051115813086,-20.611558459794,-19.4340341168853,-17.8059771757735,-16.030392246419,-14.3872675704741,-12.6694875076583,-10.9030034736042,-9.13629785557325,-7.23368807014291,-5.21445697837309,-3.17413635408419,-1.47772121328786,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[20.49724852649,20.7001615839107,21.8736134028284,23.1521295184664,24.5361578714931,26.076173167552,27.7766259988589,29.5865858509806,30.8306756091164,31.4298798234378,31.8179526441114,32.0838878692775,32.2655680424105,32.3901474580232,32.4642100528109,32.4541968634362,32.394155061453,32.3310742858962,32.2608282877662,32.1860588882357,32.1042044279706,32.0116389711464,31.9093432628175,31.7395325145526,31.5436008130257,31.3950324258981,31.2325854241346,31.1638428265193,31.1565109170795,31.1555678799044,31.162366474749,31.1648570196991,31.1623481113341,31.1562864270505,31.0792501047289,31.0213588776769,30.9864514779031,30.9777136971445,30.9962801401066,31.0059463503695,30.9543299159516,30.4732583121351,29.8425323597422,28.6575975814043,27.3040360267934,25.8049787349971,23.9548714271344,22.1700086009303,20.8046339273029,19.5302117423197,18.5412511323381,17.4619075407041,16.1939085919898,14.7738373780966,13.2321468371699,11.5859297802023,9.83921371888051,7.34834868731672,4.95061709762904,2.72654363358918,0.646818135178435,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[35.0587700833248,34.7624884331955,35.791694283722,36.909558003963,38.1095829505807,39.4551757072609,40.9443512888151,42.4857331052792,42.942773036156,42.1898044064339,40.9909899513859,39.6060924739068,38.1381005688769,36.5299828304655,34.5653397820104,32.3395848524816,29.5220891351029,26.5577213836227,23.0639916223998,19.3851044164366,15.1806974524716,10.2407790305095,4.92736089703879,0.124001633818789,0,0,0,0,0,0,0,0,0,0,0,0,0,-1.10676510469034,-7.84638697429795,-15.5328436452862,-22.0199846141191,-20.8109959361062,-18.9686265586611,-13.3479514056072,-5.91542838225709,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[24.462354408451,24.2961571674008,25.0600803533915,25.8932463163985,26.7930097479251,27.8071850861687,28.9372799244155,30.1353250656421,30.7581481998066,30.6804157957336,30.3139958018077,29.7443273329864,29.0174814055536,28.1649932305823,27.1824330196366,26.0376191924329,24.7445475004943,23.349358642635,21.8626434822223,20.2601444164001,18.5567589690819,16.7632002278696,14.841487911388,12.7417873680062,10.5949752600276,8.33753446501052,5.98117156800696,3.50530958239267,1.04380838031684,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-19.2344661916362,-19.0621097893204,-17.8715439520769,-16.5743130390779,-15.170061920204,-13.6082063588911,-11.8842067554421,-10.0465116621268,-8.76268375126433,-8.12575452369558,-7.70149746813329,-7.39865706873641,-7.1781768826754,-7.01167190224983,-6.89044288684131,-6.84500067063355,-6.86270786073821,-6.87874036468535,-6.90313528109497,-6.92658646345128,-6.9564457453521,-6.99741155187983,-7.03852934232481,-7.0982764538062,-7.15132491922663,-7.2026498122682,-7.27857035310213,-7.25810084383217,-7.13537391600711,-7.03807619692804,-6.94647130208445,-6.86865125615437,-6.76567987484862,-6.6405943897047,-6.59539205361777,-6.54324327934848,-6.41201004661069,-6.25386896858249,-6.05493366344023,-5.83608073345488,-5.65938094551702,-5.59349656476626,-5.51496186087649,-5.44631009897168,-5.36453812494144,-5.29290228858827,-5.30317732256089,-4.94692136088931,-4.43898055482454,-3.79260277267975,-2.78858522254363,-1.6954923782249,-0.660915359765329,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[-433.262664884984,-432.954321734684,-431.250152108104,-429.392120747669,-427.375600047945,-425.148982593174,-422.702444885305,-420.064570623393,-417.838866162206,-416.119274592221,-414.452070306408,-412.776882604969,-411.121911321942,-409.448473906531,-407.61762513683,-405.695351350521,-403.683960191021,-401.509420881782,-399.158460891371,-396.609632015553,-393.860737476885,-390.919445484871,-387.692037061312,-384.217260531197,-380.318249096624,-375.904332830046,-371.193949471208,-366.061014276287,-360.989391324865,-356.64281069759,-352.866552299316,-348.710218695312,-344.062290145108,-338.88851348095,-333.167283995375,-326.723874939059,-319.001744468589,-310.539180577284,-301.286772807217,-290.707038923924,-279.537274620715,-267.64582887334,-255.488354029974,-243.841428861706,-231.607428508665,-213.885723542135,-196.373256114726,-178.382028707816,-158.554253542651,-138.818911622938,-117.942695513201,-97.0291418585542,-76.9296288443928,-57.4469679513795,-37.6090638528096,-18.7986469428726,-1.19844422863632,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"plugins":{"Unzoom":"{}"}},"evals":["attrs.interactionModel","plugins.Unzoom"],"jsHooks":[]}</script>

---

## What's under the hood?

```r
# It shows from left to right the number of nonzero coefficients (Df),
# the percent of deviance explained (%dev) and the value of Lambda.
# Deviance is the stat for maximum likelihood estimation
# Although by default glmnet calls for 100 values of lambda the program stops
# early if %dev doesn't change sufficiently from one lambda to the next.
print(fit_LASSO)
```

```
## 
## Call:  glmnet(x = x, y = y, family = "binomial", alpha = 1) 
## 
##    Df  %Dev    Lambda
## 1   0  0.00 0.0143300
## 2   1  0.81 0.0130500
## 3   1  1.46 0.0118900
## 4   1  2.00 0.0108400
## 5   2  2.47 0.0098740
## 6   2  3.22 0.0089970
## 7   2  3.85 0.0081970
## 8   2  4.37 0.0074690
## 9   2  4.81 0.0068060
## 10  3  5.22 0.0062010
## 11  3  5.59 0.0056500
## 12  4  5.91 0.0051480
## 13  4  6.25 0.0046910
## 14  5  6.57 0.0042740
## 15  7  6.89 0.0038940
## 16  8  7.22 0.0035480
## 17 10  7.52 0.0032330
## 18 12  7.83 0.0029460
## 19 15  8.16 0.0026840
## 20 15  8.49 0.0024460
## 21 15  8.78 0.0022290
## 22 15  9.03 0.0020310
## 23 18  9.26 0.0018500
## 24 20  9.48 0.0016860
## 25 22  9.69 0.0015360
## 26 25  9.89 0.0014000
## 27 28 10.10 0.0012750
## 28 30 10.33 0.0011620
## 29 31 10.54 0.0010590
## 30 33 10.74 0.0009647
## 31 36 10.93 0.0008790
## 32 37 11.11 0.0008009
## 33 39 11.26 0.0007297
## 34 42 11.42 0.0006649
## 35 43 11.57 0.0006058
## 36 44 11.71 0.0005520
## 37 47 11.85 0.0005030
## 38 48 11.97 0.0004583
## 39 48 12.08 0.0004176
## 40 48 12.18 0.0003805
## 41 51 12.27 0.0003467
## 42 52 12.35 0.0003159
## 43 54 12.43 0.0002878
## 44 54 12.49 0.0002623
## 45 54 12.55 0.0002390
## 46 55 12.60 0.0002177
## 47 56 12.65 0.0001984
## 48 58 12.70 0.0001808
## 49 60 12.74 0.0001647
## 50 61 12.79 0.0001501
## 51 62 12.82 0.0001367
## 52 63 12.86 0.0001246
## 53 65 12.89 0.0001135
## 54 66 12.92 0.0001034
## 55 66 12.94 0.0000942
## 56 66 12.97 0.0000859
## 57 66 12.98 0.0000782
## 58 66 13.00 0.0000713
## 59 66 13.02 0.0000650
## 60 66 13.03 0.0000592
## 61 67 13.04 0.0000539
## 62 67 13.05 0.0000491
## 63 67 13.06 0.0000448
## 64 67 13.06 0.0000408
## 65 67 13.07 0.0000372
## 66 67 13.08 0.0000339
## 67 67 13.08 0.0000309
## 68 67 13.09 0.0000281
## 69 67 13.09 0.0000256
## 70 67 13.09 0.0000233
## 71 67 13.10 0.0000213
## 72 67 13.10 0.0000194
## 73 67 13.10 0.0000177
## 74 67 13.10 0.0000161
## 75 67 13.10 0.0000147
```

---

## **One** of the 100 models

```r
# I randomly pick a lambda and show the features
# You may try a different lambda from the previous slide
coefplot(fit_LASSO, lambda = 0.002031, sort = 'magnitude')
```

```r
# coef(fit_LASSO, s=0.002031) #to check the magnitude of coefficients
```

---

## How does this perform?

```r
# na.pass has model.matrix retain NA values (so the # of rows is constant)
xp <- model.matrix(BCE_eq, data = df, na.action = 'na.pass')[ , -1]
# s= specifies the version of the model to use
pred <- predict(fit_LASSO, xp, type = "response", s = 0.002031)
```

```
##     In sample AUC Out of sample AUC 
##         0.7593828         0.7239785
```

---

## Automating model selection

- LASSO seems nice, but picking between the 100 models is tough!
- It also contains a method of *k-fold cross validation* (default, `$k=10$`)
    1. Randomly split the data into `$k$` groups
    2. Hold out one group and run model on the rest data
    3. Determine the best model using the hold out group
    4. Repeat steps 2 and 3 `$k-1$` more times
    5. Uses the best overall model across all `$k$` hold out samples
- It gives 2 model options:
    - `"lambda.min"`: The best performing model that gives minimum mean cross-validated error.
    - `"lambda.1se"`: the most regularized model such that error is within one standard error of the `"lambda.min"`
        - This is the better choice if you are concerned about overfitting

.center[<img src = "../../../Figures/validation-set.png">]

---

## Running a cross validated model

```r
# Cross validation
set.seed(2021)  #for reproducibility
cvfit = cv.glmnet(x = x, y = y,family = "binomial", alpha = 1,
                  type.measure = "auc")
```

.pull-left[

```r
plot(cvfit)
```

<img src="Session_9s_files/figure-html/unnamed-chunk-28-1.png" width="100%" style="display: block; margin: auto;" />
]
.pull-right[

```r
cvfit$lambda.min
```

```
## [1] 0.001685798
```

```r
cvfit$lambda.1se
```

```
## [1] 0.004690834
```
- The left figure also shows the upper and lower standard deviation of AUC
]

---

## Models

.pull-left[
`lambda.min`
<img src="Session_9s_files/figure-html/unnamed-chunk-30-1.png" width="100%" style="display: block; margin: auto;" />
]
.pull-right[
`lambda.1se`
<img src="Session_9s_files/figure-html/unnamed-chunk-31-1.png" width="100%" style="display: block; margin: auto;" />
]

---

## CV LASSO performance

```r
# s= specifies the version of the model to use
pred <- predict(cvfit, xp, type = "response", s = "lambda.min")
pred2 <- predict(cvfit, xp, type = "response", s = "lambda.1se")
```

```
##     In sample AUC, lambda.min Out of sample AUC, lambda.min 
##                     0.7631710                     0.7290185 
##     In sample AUC, lambda.1se Out of sample AUC, lambda.1se 
##                     0.7337026                     0.6906418
```

---

## Drawbacks of LASSO

1. No p-values on coefficients
    - Simple solution -- run the resulting model with [`glm()`](https://rdrr.io/r/stats/glm.html)
    - Solution only if using `family = "gaussian"`:
        - Run the lasso use the `package:lars` package
            - `m <- lars(x = x, y = y, type = "lasso")`
        - Then test coefficients using the `package:covTest` package
            - `covTest(m, x, y)`
2. Generally worse in sample performance
3. Sometimes worse out of sample performance (short run)
    - BUT: predictions will be more stable

---

## Logistic with lambda.min variables

```r
fit_glm_lambda.min <- glm(AAER ~ logtotasset + soft_assets + merger + restruct +
                 processedsize + paralen_s + sentlen_s + passive_p +
                 lm_negative_p + Topic_2_n_oI + Topic_6_n_oI +
                 Topic_8_n_oI + Topic_9_n_oI + Topic_12_n_oI + Topic_18_n_oI +
                 Topic_19_n_oI + Topic_23_n_oI + Topic_26_n_oI +
                 Topic_29_n_oI + Topic_30_n_oI,
               data = df[df$Test == 0, ], family = binomial)
tidy(fit_glm_lambda.min)
```

```
## # A tibble: 21 x 5
##    term              estimate    std.error statistic  p.value
##    <chr>                <dbl>        <dbl>     <dbl>    <dbl>
##  1 (Intercept)   -7.51         0.455         -16.5   2.49e-61
##  2 logtotasset    0.379        0.0376         10.1   5.61e-24
##  3 soft_assets    2.43         0.359           6.77  1.26e-11
##  4 merger         0.391        0.144           2.72  6.53e- 3
##  5 restruct       0.175        0.151           1.16  2.46e- 1
##  6 processedsize  0.000000957  0.000000854     1.12  2.62e- 1
##  7 paralen_s     -0.0369       0.0244         -1.51  1.31e- 1
##  8 sentlen_s     -0.0139       0.0155         -0.897 3.70e- 1
##  9 passive_p     -7.17         3.95           -1.81  6.96e- 2
## 10 lm_negative_p  8.48        11.8             0.716 4.74e- 1
## # ... with 11 more rows
```

---

## Logistic with lambda.1se variables

```r
fit_glm_lambda.1se <- glm(AAER ~ logtotasset + soft_assets + merger
                          + Topic_2_n_oI,
                          data = df[df$Test == 0, ], family = binomial)
tidy(fit_glm_lambda.1se)
```

```
## # A tibble: 5 x 5
##   term         estimate std.error statistic   p.value
##   <chr>           <dbl>     <dbl>     <dbl>     <dbl>
## 1 (Intercept)    -7.22     0.316     -22.8  1.51e-115
## 2 logtotasset     0.315    0.0327      9.64 5.35e- 22
## 3 soft_assets     2.26     0.324       6.99 2.72e- 12
## 4 merger          0.421    0.141       2.98 2.89e-  3
## 5 Topic_2_n_oI   82.6     23.1         3.58 3.42e-  4
```

---

## Compare with Ridge and Elastic Net

```r
# It takes 3 minutes on my Xeon W-2133 32G workstation
list.of.fits <- list() # To store the results
for (i in 0:10) {
  ## First, make a variable name that we can use later to refer
  ## to the model optimized for a specific alpha.
  ## For example, when alpha = 0, we will be able to refer to 
  ## that model with the variable name "alpha0".
  fit.name <- paste0("alpha", i/10)
  
  ## Now fit a model (i.e. optimize lambda) and store it in a list that 
  ## uses the variable name we just created as the reference.
  list.of.fits[[fit.name]] <-
    cv.glmnet(x = x, y = y, family = "binomial", alpha = i/10,
              type.measure = "auc")
}
```

---

## Which alpha?

```r
## Now we see which alpha (0, 0.1, ... , 0.9, 1) does the best job
## predicting the values in the Testing dataset.
results <- data.frame()
for (i in 0:10) {
  fit.name <- paste0("alpha", i/10)
  
  ## Use each model to predict 'y' given the Testing dataset
  pred <- 
    predict(list.of.fits[[fit.name]], xp, type = "response",
            s = list.of.fits[[fit.name]]$lambda.1se)

ROCpred_out <- prediction(as.numeric(pred[df$Test == 1 & !is.na(pred)]),
                            as.numeric(df[df$Test == 1 & !is.na(pred), ]$AAER))
  auc_out <- performance(ROCpred_out, measure = "auc")

## Store the results
  temp <- data.frame(alpha=i/10, auc = auc_out@y.values[[1]],
                     fit.name = fit.name)
  results <- rbind(results, temp)
}
```

---

## Which alpha?

<table class="table table-striped table-hover" style="width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> alpha </th>
   <th style="text-align:center;"> auc </th>
   <th style="text-align:center;"> fit.name </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 0.0 </td>
   <td style="text-align:center;"> 0.7263743 </td>
   <td style="text-align:center;"> alpha0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.1 </td>
   <td style="text-align:center;"> 0.7145268 </td>
   <td style="text-align:center;"> alpha0.1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.2 </td>
   <td style="text-align:center;"> 0.7139278 </td>
   <td style="text-align:center;"> alpha0.2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.3 </td>
   <td style="text-align:center;"> 0.6957328 </td>
   <td style="text-align:center;"> alpha0.3 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.4 </td>
   <td style="text-align:center;"> 0.6988006 </td>
   <td style="text-align:center;"> alpha0.4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.5 </td>
   <td style="text-align:center;"> 0.6904372 </td>
   <td style="text-align:center;"> alpha0.5 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.6 </td>
   <td style="text-align:center;"> 0.7123501 </td>
   <td style="text-align:center;"> alpha0.6 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.7 </td>
   <td style="text-align:center;"> 0.7122478 </td>
   <td style="text-align:center;"> alpha0.7 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.8 </td>
   <td style="text-align:center;"> 0.7123574 </td>
   <td style="text-align:center;"> alpha0.8 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 0.9 </td>
   <td style="text-align:center;"> 0.7169080 </td>
   <td style="text-align:center;"> alpha0.9 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 1.0 </td>
   <td style="text-align:center;"> 0.7013864 </td>
   <td style="text-align:center;"> alpha1 </td>
  </tr>
</tbody>
</table>

---
class: middle, center, inverse

# Supervised Learning with Boosted Trees

---

## Decision trees

- We can make prediction through decision trees.
- Here's an example of a decision tree which classifies whether someone will like computer games.
- Decision tree is prone to overfitting (high variance) as it can keep growing until it has exactly one leaf node for every single observation (low bias).

.center[<img src="../../../Figures/cart.png" width="800px">]

---

## Decision tree ensembles

- To mitigate the overfitting, we can limit the tree depth and creat more trees (of course resulting in higher bias).
- Here is an example of a tree ensemble of two trees. The prediction scores of each individual tree are summed up to get the final score. An important benefit is that the two trees try to complement each other.

---

## Random Forest

- A popular tree ensembles is the random forest
    - Random sampling of training data points when building trees
    - Random subsets of features considered when splitting nodes
- The random forest combines hundreds or thousands of decision trees, trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features.
    - It grows trees independently in parallel
- The final predictions of the random forest are made by averaging the predictions of each individual trees.
- This process is known as [bootstrap aggregating (or bagging)](https://en.wikipedia.org/wiki/Bootstrap_aggregating)
- The benefit is to reduce variance, ie, mitigate overfitting problem

.center[<img src = "../../../Figures/Ensemble_Bagging.svg" height = "200px">]

---

## Gradient boosting through XGBoost

- *Boosting*: we use an additive strategy: fix what we have learned, and add one new tree at a time (ie, grow trees sequentially).
    - lower variance as we use multiple models (bagging)
    - lower bias by training subsequent trees using what errors the previous trees made (boosting).
- There are two main algorithms:
    - *Adaboost*: original, subsequent trees to punish more heavily observations mistaken by previous trees
    - *Gradient boosting*: train each subsequent tree using residuals
- [`package:xgboost`](https://github.com/dmlc/xgboost) is a library which provides a gradient boosting framework for C++, Java, Python, R, and Julia.

---

## What is XGBoost

- e**X**treme **G**radient **B**oosting
- A simple explanation:
    1. Start with 1 or more decision trees & check error
    2. Make more decision trees & check error
    3. Use the difference in error to guess another model
    4. Repeat #2 and #3 until the model's error is stable

.center[<img src="../../../Figures/XGBoost.png" width="600px">]

---

## What is Gradient Descent

- [Gradient descent](https://towardsdatascience.com/understanding-the-mathematics-behind-gradient-descent-dde5dc9be06e) algorithm is an iterative process that takes us to the minimum of a loss function (such as the SSE in a linear model)
    - min: slope of the tangent line (gradient or partial derivatives) = 0
    - in fact OLS is to estimate coeffcients which equal PD of SSE to zero
- Then why not OLS directly (or why iterate the gradient process?)
    - Many loss functions have no analytical solutions (ie, cannot solve PD = 0 equation such as `$\frac{dy}{dx}=\frac{1}{x}$`)
    - Have to rely on numerical solutions (by giving input values)

.center[<img src="../../../Figures/gradientdescent.png" width="600px">]

---

## What is Learning Rate

- Gradient descent algorithms multiply the gradient by a scalar known as the [learning rate](https://developers.google.com/machine-learning/crash-course/reducing-loss/gradient-descent) (also called step size) to determine the next point.
    - eg, if the gradient magnitude is 2.5 and the learning rate is 0.01, then the gradient descent algorithm will pick the next point 0.025 away from the previous point.
    - an optimal learning rate can increase the model efficiency to converge, [try the game](https://developers.google.com/machine-learning/crash-course/fitter/graph)

.center[<img src="../../../Figures/GradientDescentGradientStep.svg" height="300px">]

---

## Steps of gradient descent

1. Take the derivative (gradient) of the loss function
2. Pick a random value of the parameters
3. Plug parameter values and calcuate gradient values (slope)
4. Step size = slope * learning rate
5. New parameter = old parameter - step size
6. Repeat 3-5 until step size is very small or reach max number of steps

> Stochastic Gradient Descent: use random subset at every step rather than the full dataset (to save time)

---

## XGBoost model setup

```r
# XGBoost Model Setup
library(xgboost)
set.seed(2021)

# These params take some work to pin down
params <- list(max_depth = 5, # Maximum depth of a tree
               eta = 0.2, # learning rate or step size for each boosting
               gamma = 10, # Min loss reduction to make a further partition
               min_child_weight = 20, # min obs needed to be in each node
               objective = "binary:logistic") # "reg:linear" for simple OLS
# The cross-validation function of xgboost
xgbCV <- xgb.cv(params = params, data = x, label = y,
                nrounds = 100, # The number of rounds/iterations for boosting
                eval_metric = "auc",
                # randomly partitioned into nfold equal size subsamples
                nfold = 10,
                # stratified sampling by the values of outcome labels
                # ie, same proportion of outcomes for subsamples
                stratified = TRUE)
# Boost at minimum number of iterations with the max AUC
# which(): position of the elements in a logical vector which are TRUE
numRounds <- min(which(xgbCV$evaluation_log$test_auc_mean ==
                        max(xgbCV$evaluation_log$test_auc_mean)))
```

---

## Run XGBoost

```r
fit_XGB <- xgboost(params = params,
                data = x,
                label = y,
                nrounds = numRounds,
                eval_metric = "auc")
```

- Refer to the [XGBoost Parameters Manual](https://xgboost.readthedocs.io/en/latest/parameter.html)
- Refer to the [xgb.cv manual](https://www.rdocumentation.org/packages/xgboost/versions/0.81.0.1/topics/xgb.cv)
- Refer to the Assignment for [parameter tuning](https://cran.r-project.org/web/packages/ParBayesianOptimization/vignettes/tuningHyperparameters.html)
- You can plot the trees from your model using [`xgb.plot.tree()`](https://xgboost.readthedocs.io/en/latest/R-package/xgboostPresentation.html)

---

## Relative importance of variables

```r
# Display relative importance of variables for prediction
xtest <- model.matrix(BCE_eq, data = df[df$Test == 1, ])[ , -1]
ytest <- model.frame(BCE_eq, data = df[df$Test == 1, ])[ , "AAER"]
xgb.train.data = xgb.DMatrix(x, label = y, missing = NA)
xgb.test.data = xgb.DMatrix(xtest, label = ytest, missing = NA)
col_names = attr(xgb.train.data, ".Dimnames")[[2]]
imp = xgb.importance(col_names, fit_XGB)
```

---

## Plot relative importance of variables

```r
# the bar represents the improvement in accuracy brought by a feature
xgb.plot.importance(imp)
```

---

## Model performance

```
##             1990s              2011             2000s      2000s + 2011 
##         0.7292981         0.6849225         0.6295414         0.7147021 
##               BCE LASSO, lambda.min       XGBoost AUC 
##         0.7599594         0.7290185         0.7817700
```

---

## Explaining XGBoost

```r
# Use the following code to install the xgboostExplainer package
# You may need to launch the RStudio "as an administrator"
# It is also advised to launch a fresh RStudio without any startup files
# install.packages("devtools")
# library(devtools)
# install_github("AppliedDataSciencePartners/xgboostExplainer", force = T)
library(xgboostExplainer)
explainer = buildExplainer(fit_XGB, xgb.train.data, type = "binary",
                           base_score = 0.5, trees_idx = NULL)
pred.breakdown = explainPredictions(fit_XGB, explainer, xgb.test.data)
```

---

## How about Enron in 2000

```r
# See what XGBoost is thinking of Enron Corp (gvkey 6127) in year 2000
# enron_row is the position of record for Enron (row number 2237 in the data)
# the prediction is -2.9 (the last black bar), so how much is the probability?
enron_row = 2237
showWaterfall(fit_XGB, explainer, xgb.test.data, data.matrix(df[df$Test == 1, ]),
              enron_row, type = "binary") +
  ggtitle("Enron Corp in 2000 -- what's probability to have fraudulent report?")
```

---

## The cutoff

- Note that logistic regression does not predict 1 or 0 directly
- Pick a cutoff value, ie, fraud (positive) classification if >= cutoff%
- The optimal cutoff depends on data and context
- Check a more [detailed explanation](http://ethen8181.github.io/machine-learning/unbalanced/unbalanced.html)

---

## Sesentivity vs. Cutoff

- For imbalanced data, we care more about Sensitivity (TP), rather than Accuracy (TP + TN)
- The following plot shows the relation between Sensitivity and Cutoff values

---

## Confusion Matrix at Cutoff 0.05

---

## The impact on logodds

---
class: inverse, center, middle

# Summary of Session 9

---

## For next week

- Try to replicate the code
- Continue your Datacamp career track
- Try the new models with your project data
- Submit project by the dateline and prepare for presentation

---

## Supplementary readings

- [Many Ways to Lasso](https://jaredlander.com/content/2018/11/ManyWaysToLasso2.html)
- [Tune XGBoost with tidymodels](https://juliasilge.com/blog/xgboost-tune-volleyball/)
- [Using XGBoost with Tidymodels](https://www.tychobra.com/posts/2020-05-19-xgboost-with-tidymodels/)
- [Choosing Cutoff Value for Unbalanced Dataset](http://ethen8181.github.io/machine-learning/unbalanced/unbalanced.html#content)
- [XGBoost](https://finnstats.com/index.php/2021/04/17/gradient-boosting-in-r/)

---

## R Coding Style Guide

Style is subjective and arbitrary but it is important to follow a generally accepted style if you want to share code with others. I suggest the [The tidyverse style guide](https://style.tidyverse.org/) which is also adopted by [Google](https://google.github.io/styleguide/Rguide.html) with some modification
- Highlights of **the tidyverse style guide**:
    - *File names*: end with .R
    - *Identifiers*: variable_name, function_name, try not to use "." as it is reserved by Base R's S3 objects
    - *Line length*: 80 characters
    - *Indentation*: two spaces, no tabs (RStudio by default converts tabs to spaces and you may change under global options)
    - *Spacing*: x = 0, not x=0, no space before a comma, but always place one after a comma
    - *Curly braces {}*: first on same line, last on own line
    - *Assignment*: use `<-`, not `=` nor `->`
    - *Semicolon(;)*: don't use, I used once for the interest of space
    - *return()*: Use explicit returns in functions: default function return is the last evaluated expression
    - *File paths*: use [relative file path](https://www.w3schools.com/html/html_filepaths.asp) "../../filename.csv" rather than absolute path "C:/mydata/filename.csv". Backslash needs `\\`

---

## R packages used in this slide

This slide was prepared on 2021-09-08 from Session_9s.Rmd with R version 4.1.1 (2021-08-10) Kick Things on Windows 10 x64 build 18362 😀.

The attached packages used in this slide are:

```
##       data.table         ggthemes xgboostExplainer          xgboost 
##         "1.14.0"          "4.2.4"            "0.1"        "1.4.1.1" 
##           glmnet           Matrix             ROCR         coefplot 
##          "4.1-2"          "1.3-4"         "1.0-11"          "1.2.7" 
##            broom          forcats          stringr            dplyr 
##          "0.7.9"          "0.5.1"          "1.4.0"          "1.0.7" 
##            purrr            readr            tidyr           tibble 
##          "0.3.4"          "2.0.1"          "1.1.3"          "3.1.3" 
##          ggplot2        tidyverse       kableExtra            knitr 
##          "3.3.5"          "1.3.1"          "1.3.4"           "1.33"
```