# Test Process for Non-Gradient Filter Pipeline

For each attack, the following tests are to be evaluated. The performance of each attack should be evaluated using cross validation with $k=5$.

| Training | Test                    |
|----------|-------------------------|
| Clean    | Clean                   |
| Clean    | Attacked                |
| Clean    | Filtered (Not Attacked) |
| Clean    | Filtered (Attacked)     |
| Filtered | Filtered (Not Attacked) |
| Filtered | Filtered (Attacked)     |

## Testing on Pretrained Model Trained on Unfiltered Data
Epsilon: 0.05
Original Accuracy = 9912 / 10000 = 0.9912
Attacked Accuracy = 9605 / 10000 = 0.9605
Filtered Accuracy = 9522 / 10000 = 0.9522

Epsilon: 0.1
Original Accuracy = 9912 / 10000 = 0.9912
Attacked Accuracy = 8743 / 10000 = 0.8743
Filtered Accuracy = 9031 / 10000 = 0.9031

Epsilon: 0.15000000000000002
Original Accuracy = 9912 / 10000 = 0.9912
Attacked Accuracy = 7107 / 10000 = 0.7107
Filtered Accuracy = 8138 / 10000 = 0.8138

Epsilon: 0.2
Original Accuracy = 9912 / 10000 = 0.9912
Attacked Accuracy = 4876 / 10000 = 0.4876
Filtered Accuracy = 6921 / 10000 = 0.6921

Epsilon: 0.25
Original Accuracy = 9912 / 10000 = 0.9912
Attacked Accuracy = 2714 / 10000 = 0.2714
Filtered Accuracy = 5350 / 10000 = 0.535

Epsilon: 0.3
Original Accuracy = 9912 / 10000 = 0.9912
Attacked Accuracy = 1418 / 10000 = 0.1418
Filtered Accuracy = 3605 / 10000 = 0.3605

### Observations

| $\epsilon$ | Attacked Accuracy | Filtered Accuracy | Ratio  |
|------------|-------------------|-------------------|--------|
| 0.05       | 0.9605            | 0.9522            | 0.9914 |
| 0.1        | 0.8743            | 0.9031            | 1.0329 |
| 0.15       | 0.7107            | 0.8138            | 1.1451 |
| 0.2        | 0.4876            | 0.6921            | 1.4194 |
| 0.25       | 0.2714            | 0.5350            | 1.9713 |
| 0.3        | 0.1418            | 0.3605            | 2.5423 |

- Filter seems to consitently increase accuracy
	- When epsilon is too low to have a significant imact on the accuracy, the filter is seems to be counterproductive
		- This may be avoidable by training on filtered data
		- Low values of epsilon will be tested on filtered model to test this hypothesis

## Testing on Model Trained with Filtered Data
CNN classifier trained on MNIST dataset with 14 epochs. Kuwahara filter applied at runtime for each batch of training and test data.

### Hypothesis
Adding a denoising filter will increase accuracy against FGSM attack

### Results
Epsilon: 0.05
Original Accuracy = 9793 / 10000 = 0.9793
Attacked Accuracy = 7288 / 10000 = 0.7288
Filtered Accuracy = 9575 / 10000 = 0.9575
Filtered:Attacked = 0.9575 / 0.7288 = 1.3138035126234906

Epsilon: 0.1
Original Accuracy = 9793 / 10000 = 0.9793
Attacked Accuracy = 2942 / 10000 = 0.2942
Filtered Accuracy = 8268 / 10000 = 0.8268
Filtered:Attacked = 0.8268 / 0.2942 = 2.8103331067301154

Epsilon: 0.15000000000000002
Original Accuracy = 9793 / 10000 = 0.9793
Attacked Accuracy = 1021 / 10000 = 0.1021
Filtered Accuracy = 5253 / 10000 = 0.5253
Filtered:Attacked = 0.5253 / 0.1021 = 5.144955925563173

Epsilon: 0.2
Original Accuracy = 9793 / 10000 = 0.9793
Attacked Accuracy = 404 / 10000 = 0.0404
Filtered Accuracy = 2833 / 10000 = 0.2833
Filtered:Attacked = 0.2833 / 0.0404 = 7.012376237623762

Epsilon: 0.25
Original Accuracy = 9793 / 10000 = 0.9793
Attacked Accuracy = 234 / 10000 = 0.0234
Filtered Accuracy = 1614 / 10000 = 0.1614
Filtered:Attacked = 0.1614 / 0.0234 = 6.897435897435897

Epsilon: 0.3
Original Accuracy = 9793 / 10000 = 0.9793
Attacked Accuracy = 161 / 10000 = 0.0161
Filtered Accuracy = 959 / 10000 = 0.0959
Filtered:Attacked = 0.0959 / 0.0161 = 5.956521739130435

### Observations
- Model is more susceptable to FGSM than pretrained model
- Model repsonds much better to filtered data than pretrained model
- Even for $\epsilon = 0.25$, the model does better than random guessing (10 classes)
	- Potential for boost algorithm
- Filter is proportionally more effective for higher values of $\epsilon$ until $\epsilon=0.3$

## Testing on Model Trained with Unfiltered Data
CNN classifier, same as above, trained on 14 epochs of MNIST dataset without Kuwahara filtering. 

### Hypothesis
Given how the attacked model trained on filtered data performed against the FGSM attack, we expect that the model trained on unfiletered data will pereform poorly.

### Results
Epsilon: 0.05
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 9600 / 10000 = 0.96
Filtered Accuracy = 8700 / 10000 = 0.87
Filtered:Attacked = 0.87 / 0.96 = 0.90625

Epsilon: 0.1
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 8753 / 10000 = 0.8753
Filtered Accuracy = 8123 / 10000 = 0.8123
Filtered:Attacked = 0.8123 / 0.8753 = 0.9280246772535131

Epsilon: 0.15000000000000002
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 7229 / 10000 = 0.7229
Filtered Accuracy = 7328 / 10000 = 0.7328
Filtered:Attacked = 0.7328 / 0.7229 = 1.013694840226864

Epsilon: 0.2
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 5008 / 10000 = 0.5008
Filtered Accuracy = 6301 / 10000 = 0.6301
Filtered:Attacked = 0.6301 / 0.5008 = 1.2581869009584663

Epsilon: 0.25
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 2922 / 10000 = 0.2922
Filtered Accuracy = 5197 / 10000 = 0.5197
Filtered:Attacked = 0.5197 / 0.2922 = 1.7785763175906915

Epsilon: 0.3
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 1599 / 10000 = 0.1599
Filtered Accuracy = 3981 / 10000 = 0.3981
Filtered:Attacked = 0.3981 / 0.1599 = 2.4896810506566607

### Observations
- The ratio of filtered to attacked performance is stricty increasing
- The unfiltered model seems to be less susceptable to the FGSM attack