Directory structure overhaul, poster almost done

2024-05-01 01:26:25 -04:00
parent bf2440ef86
commit 008ef906d7
106 changed files with 58 additions and 20 deletions
--- a/wiki/Tests.md
+++ b/wiki/Tests.md
@ -0,0 +1,156 @@
+# Test Process for Non-Gradient Filter Pipeline
+
+For each attack, the following tests are to be evaluated. The performance of each attack should be evaluated using cross validation with $k=5$.
+
+| Training | Test                    |
+|----------|-------------------------|
+| Clean    | Clean                   |
+| Clean    | Attacked                |
+| Clean    | Filtered (Not Attacked) |
+| Clean    | Filtered (Attacked)     |
+| Filtered | Filtered (Not Attacked) |
+| Filtered | Filtered (Attacked)     |
+
+## Testing on Pretrained Model Trained on Unfiltered Data
+Epsilon: 0.05
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 9605 / 10000 = 0.9605
+Filtered Accuracy = 9522 / 10000 = 0.9522
+
+Epsilon: 0.1
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 8743 / 10000 = 0.8743
+Filtered Accuracy = 9031 / 10000 = 0.9031
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 7107 / 10000 = 0.7107
+Filtered Accuracy = 8138 / 10000 = 0.8138
+
+Epsilon: 0.2
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 4876 / 10000 = 0.4876
+Filtered Accuracy = 6921 / 10000 = 0.6921
+
+Epsilon: 0.25
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 2714 / 10000 = 0.2714
+Filtered Accuracy = 5350 / 10000 = 0.535
+
+Epsilon: 0.3
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 1418 / 10000 = 0.1418
+Filtered Accuracy = 3605 / 10000 = 0.3605
+
+### Observations
+
+| $\epsilon$ | Attacked Accuracy | Filtered Accuracy | Ratio  |
+|------------|-------------------|-------------------|--------|
+| 0.05       | 0.9605            | 0.9522            | 0.9914 |
+| 0.1        | 0.8743            | 0.9031            | 1.0329 |
+| 0.15       | 0.7107            | 0.8138            | 1.1451 |
+| 0.2        | 0.4876            | 0.6921            | 1.4194 |
+| 0.25       | 0.2714            | 0.5350            | 1.9713 |
+| 0.3        | 0.1418            | 0.3605            | 2.5423 |
+
+- Filter seems to consitently increase accuracy
+	- When epsilon is too low to have a significant imact on the accuracy, the filter is seems to be counterproductive
+		- This may be avoidable by training on filtered data
+		- Low values of epsilon will be tested on filtered model to test this hypothesis
+
+## Testing on Model Trained with Filtered Data
+CNN classifier trained on MNIST dataset with 14 epochs. Kuwahara filter applied at runtime for each batch of training and test data.
+
+### Hypothesis
+Adding a denoising filter will increase accuracy against FGSM attack
+
+### Results
+Epsilon: 0.05
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 7288 / 10000 = 0.7288
+Filtered Accuracy = 9575 / 10000 = 0.9575
+Filtered:Attacked = 0.9575 / 0.7288 = 1.3138035126234906
+
+Epsilon: 0.1
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 2942 / 10000 = 0.2942
+Filtered Accuracy = 8268 / 10000 = 0.8268
+Filtered:Attacked = 0.8268 / 0.2942 = 2.8103331067301154
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 1021 / 10000 = 0.1021
+Filtered Accuracy = 5253 / 10000 = 0.5253
+Filtered:Attacked = 0.5253 / 0.1021 = 5.144955925563173
+
+Epsilon: 0.2
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 404 / 10000 = 0.0404
+Filtered Accuracy = 2833 / 10000 = 0.2833
+Filtered:Attacked = 0.2833 / 0.0404 = 7.012376237623762
+
+Epsilon: 0.25
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 234 / 10000 = 0.0234
+Filtered Accuracy = 1614 / 10000 = 0.1614
+Filtered:Attacked = 0.1614 / 0.0234 = 6.897435897435897
+
+Epsilon: 0.3
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 161 / 10000 = 0.0161
+Filtered Accuracy = 959 / 10000 = 0.0959
+Filtered:Attacked = 0.0959 / 0.0161 = 5.956521739130435
+
+### Observations
+- Model is more susceptable to FGSM than pretrained model
+- Model repsonds much better to filtered data than pretrained model
+- Even for $\epsilon = 0.25$, the model does better than random guessing (10 classes)
+	- Potential for boost algorithm
+- Filter is proportionally more effective for higher values of $\epsilon$ until $\epsilon=0.3$
+
+## Testing on Model Trained with Unfiltered Data
+CNN classifier, same as above, trained on 14 epochs of MNIST dataset without Kuwahara filtering. 
+
+### Hypothesis
+Given how the attacked model trained on filtered data performed against the FGSM attack, we expect that the model trained on unfiletered data will pereform poorly.
+
+### Results
+Epsilon: 0.05
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 9600 / 10000 = 0.96
+Filtered Accuracy = 8700 / 10000 = 0.87
+Filtered:Attacked = 0.87 / 0.96 = 0.90625
+
+Epsilon: 0.1
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 8753 / 10000 = 0.8753
+Filtered Accuracy = 8123 / 10000 = 0.8123
+Filtered:Attacked = 0.8123 / 0.8753 = 0.9280246772535131
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 7229 / 10000 = 0.7229
+Filtered Accuracy = 7328 / 10000 = 0.7328
+Filtered:Attacked = 0.7328 / 0.7229 = 1.013694840226864
+
+Epsilon: 0.2
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 5008 / 10000 = 0.5008
+Filtered Accuracy = 6301 / 10000 = 0.6301
+Filtered:Attacked = 0.6301 / 0.5008 = 1.2581869009584663
+
+Epsilon: 0.25
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 2922 / 10000 = 0.2922
+Filtered Accuracy = 5197 / 10000 = 0.5197
+Filtered:Attacked = 0.5197 / 0.2922 = 1.7785763175906915
+
+Epsilon: 0.3
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 1599 / 10000 = 0.1599
+Filtered Accuracy = 3981 / 10000 = 0.3981
+Filtered:Attacked = 0.3981 / 0.1599 = 2.4896810506566607
+
+### Observations
+- The ratio of filtered to attacked performance is stricty increasing
+- The unfiltered model seems to be less susceptable to the FGSM attack