157 lines
5.6 KiB
Markdown
157 lines
5.6 KiB
Markdown
# Test Process for Non-Gradient Filter Pipeline
|
|
|
|
For each attack, the following tests are to be evaluated. The performance of each attack should be evaluated using cross validation with $k=5$.
|
|
|
|
| Training | Test |
|
|
|----------|-------------------------|
|
|
| Clean | Clean |
|
|
| Clean | Attacked |
|
|
| Clean | Filtered (Not Attacked) |
|
|
| Clean | Filtered (Attacked) |
|
|
| Filtered | Filtered (Not Attacked) |
|
|
| Filtered | Filtered (Attacked) |
|
|
|
|
## Testing on Pretrained Model Trained on Unfiltered Data
|
|
Epsilon: 0.05
|
|
Original Accuracy = 9912 / 10000 = 0.9912
|
|
Attacked Accuracy = 9605 / 10000 = 0.9605
|
|
Filtered Accuracy = 9522 / 10000 = 0.9522
|
|
|
|
Epsilon: 0.1
|
|
Original Accuracy = 9912 / 10000 = 0.9912
|
|
Attacked Accuracy = 8743 / 10000 = 0.8743
|
|
Filtered Accuracy = 9031 / 10000 = 0.9031
|
|
|
|
Epsilon: 0.15000000000000002
|
|
Original Accuracy = 9912 / 10000 = 0.9912
|
|
Attacked Accuracy = 7107 / 10000 = 0.7107
|
|
Filtered Accuracy = 8138 / 10000 = 0.8138
|
|
|
|
Epsilon: 0.2
|
|
Original Accuracy = 9912 / 10000 = 0.9912
|
|
Attacked Accuracy = 4876 / 10000 = 0.4876
|
|
Filtered Accuracy = 6921 / 10000 = 0.6921
|
|
|
|
Epsilon: 0.25
|
|
Original Accuracy = 9912 / 10000 = 0.9912
|
|
Attacked Accuracy = 2714 / 10000 = 0.2714
|
|
Filtered Accuracy = 5350 / 10000 = 0.535
|
|
|
|
Epsilon: 0.3
|
|
Original Accuracy = 9912 / 10000 = 0.9912
|
|
Attacked Accuracy = 1418 / 10000 = 0.1418
|
|
Filtered Accuracy = 3605 / 10000 = 0.3605
|
|
|
|
### Observations
|
|
|
|
| $\epsilon$ | Attacked Accuracy | Filtered Accuracy | Ratio |
|
|
|------------|-------------------|-------------------|--------|
|
|
| 0.05 | 0.9605 | 0.9522 | 0.9914 |
|
|
| 0.1 | 0.8743 | 0.9031 | 1.0329 |
|
|
| 0.15 | 0.7107 | 0.8138 | 1.1451 |
|
|
| 0.2 | 0.4876 | 0.6921 | 1.4194 |
|
|
| 0.25 | 0.2714 | 0.5350 | 1.9713 |
|
|
| 0.3 | 0.1418 | 0.3605 | 2.5423 |
|
|
|
|
- Filter seems to consitently increase accuracy
|
|
- When epsilon is too low to have a significant imact on the accuracy, the filter is seems to be counterproductive
|
|
- This may be avoidable by training on filtered data
|
|
- Low values of epsilon will be tested on filtered model to test this hypothesis
|
|
|
|
## Testing on Model Trained with Filtered Data
|
|
CNN classifier trained on MNIST dataset with 14 epochs. Kuwahara filter applied at runtime for each batch of training and test data.
|
|
|
|
### Hypothesis
|
|
Adding a denoising filter will increase accuracy against FGSM attack
|
|
|
|
### Results
|
|
Epsilon: 0.05
|
|
Original Accuracy = 9793 / 10000 = 0.9793
|
|
Attacked Accuracy = 7288 / 10000 = 0.7288
|
|
Filtered Accuracy = 9575 / 10000 = 0.9575
|
|
Filtered:Attacked = 0.9575 / 0.7288 = 1.3138035126234906
|
|
|
|
Epsilon: 0.1
|
|
Original Accuracy = 9793 / 10000 = 0.9793
|
|
Attacked Accuracy = 2942 / 10000 = 0.2942
|
|
Filtered Accuracy = 8268 / 10000 = 0.8268
|
|
Filtered:Attacked = 0.8268 / 0.2942 = 2.8103331067301154
|
|
|
|
Epsilon: 0.15000000000000002
|
|
Original Accuracy = 9793 / 10000 = 0.9793
|
|
Attacked Accuracy = 1021 / 10000 = 0.1021
|
|
Filtered Accuracy = 5253 / 10000 = 0.5253
|
|
Filtered:Attacked = 0.5253 / 0.1021 = 5.144955925563173
|
|
|
|
Epsilon: 0.2
|
|
Original Accuracy = 9793 / 10000 = 0.9793
|
|
Attacked Accuracy = 404 / 10000 = 0.0404
|
|
Filtered Accuracy = 2833 / 10000 = 0.2833
|
|
Filtered:Attacked = 0.2833 / 0.0404 = 7.012376237623762
|
|
|
|
Epsilon: 0.25
|
|
Original Accuracy = 9793 / 10000 = 0.9793
|
|
Attacked Accuracy = 234 / 10000 = 0.0234
|
|
Filtered Accuracy = 1614 / 10000 = 0.1614
|
|
Filtered:Attacked = 0.1614 / 0.0234 = 6.897435897435897
|
|
|
|
Epsilon: 0.3
|
|
Original Accuracy = 9793 / 10000 = 0.9793
|
|
Attacked Accuracy = 161 / 10000 = 0.0161
|
|
Filtered Accuracy = 959 / 10000 = 0.0959
|
|
Filtered:Attacked = 0.0959 / 0.0161 = 5.956521739130435
|
|
|
|
### Observations
|
|
- Model is more susceptable to FGSM than pretrained model
|
|
- Model repsonds much better to filtered data than pretrained model
|
|
- Even for $\epsilon = 0.25$, the model does better than random guessing (10 classes)
|
|
- Potential for boost algorithm
|
|
- Filter is proportionally more effective for higher values of $\epsilon$ until $\epsilon=0.3$
|
|
|
|
## Testing on Model Trained with Unfiltered Data
|
|
CNN classifier, same as above, trained on 14 epochs of MNIST dataset without Kuwahara filtering.
|
|
|
|
### Hypothesis
|
|
Given how the attacked model trained on filtered data performed against the FGSM attack, we expect that the model trained on unfiletered data will pereform poorly.
|
|
|
|
### Results
|
|
Epsilon: 0.05
|
|
Original Accuracy = 9920 / 10000 = 0.992
|
|
Attacked Accuracy = 9600 / 10000 = 0.96
|
|
Filtered Accuracy = 8700 / 10000 = 0.87
|
|
Filtered:Attacked = 0.87 / 0.96 = 0.90625
|
|
|
|
Epsilon: 0.1
|
|
Original Accuracy = 9920 / 10000 = 0.992
|
|
Attacked Accuracy = 8753 / 10000 = 0.8753
|
|
Filtered Accuracy = 8123 / 10000 = 0.8123
|
|
Filtered:Attacked = 0.8123 / 0.8753 = 0.9280246772535131
|
|
|
|
Epsilon: 0.15000000000000002
|
|
Original Accuracy = 9920 / 10000 = 0.992
|
|
Attacked Accuracy = 7229 / 10000 = 0.7229
|
|
Filtered Accuracy = 7328 / 10000 = 0.7328
|
|
Filtered:Attacked = 0.7328 / 0.7229 = 1.013694840226864
|
|
|
|
Epsilon: 0.2
|
|
Original Accuracy = 9920 / 10000 = 0.992
|
|
Attacked Accuracy = 5008 / 10000 = 0.5008
|
|
Filtered Accuracy = 6301 / 10000 = 0.6301
|
|
Filtered:Attacked = 0.6301 / 0.5008 = 1.2581869009584663
|
|
|
|
Epsilon: 0.25
|
|
Original Accuracy = 9920 / 10000 = 0.992
|
|
Attacked Accuracy = 2922 / 10000 = 0.2922
|
|
Filtered Accuracy = 5197 / 10000 = 0.5197
|
|
Filtered:Attacked = 0.5197 / 0.2922 = 1.7785763175906915
|
|
|
|
Epsilon: 0.3
|
|
Original Accuracy = 9920 / 10000 = 0.992
|
|
Attacked Accuracy = 1599 / 10000 = 0.1599
|
|
Filtered Accuracy = 3981 / 10000 = 0.3981
|
|
Filtered:Attacked = 0.3981 / 0.1599 = 2.4896810506566607
|
|
|
|
### Observations
|
|
- The ratio of filtered to attacked performance is stricty increasing
|
|
- The unfiltered model seems to be less susceptable to the FGSM attack
|