question exercise

1. We perform best subset, forward stepwise, and backward stepwise
selection on a single data set. For each approach, we obtain p + 1
models, containing 0, 1, 2,…,p predictors. Explain your answers:

(a) Which of the three models with k predictors has the smallest
training RSS?

(b) Which of the three models with k predictors has the smallest
test RSS?

(c) True or False:

i. The predictors in the k-variable model identified by forward
stepwise are a subset of the predictors in the (k+1)-variable
model identified by forward stepwise selection.

ii. The predictors in the k-variable model identified by backward stepwise are a subset of the predictors in the (k + 1)-
variable model identified by backward stepwise selection.

iii. The predictors in the k-variable model identified by backward stepwise are a subset of the predictors in the (k + 1)-
variable model identified by forward stepwise selection.

iv. The predictors in the k-variable model identified by forward
stepwise are a subset of the predictors in the (k+1)-variable
model identified by backward stepwise selection.

v. The predictors in the k-variable model identified by best
subset are a subset of the predictors in the (k + 1)-variable
model identified by best subset selection.

2. For parts (a) through (c), indicate which of i. through iv. is correct.
Justify your answer.

(a) The lasso, relative to least squares, is:

i. More flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in
variance.

ii. More flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease
in bias.

iii. Less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in
variance.

iv. Less flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease
in bias.

(b) Repeat (a) for ridge regression relative to least squares.

(c) Repeat (a) for non-linear methods relative to least squares

3. Suppose we estimate the regression coefficients in a linear regression
model by minimizing
n
i=1
⎛
⎝yi − β0 −p
j=1
βjxij
⎞
⎠
2
subject to p
j=1
|βj | ≤ s
for a particular value of s. For parts (a) through (e), indicate which
of i. through v. is correct. Justify your answer.

(a) As we increase s from 0, the training RSS will:

i. Increase initially, and then eventually start decreasing in an
inverted U shape.

ii. Decrease initially, and then eventually start increasing in a
U shape.

iii. Steadily increase.

iv. Steadily decrease.

v. Remain constant.

(b) Repeat (a) for test RSS.

(c) Repeat (a) for variance.

(d) Repeat (a) for (squared) bias.

(e) Repeat (a) for the irreducible error.

Submit a .doc Remember to include the following

* Answers and justifications

* R script and outputs if any