Tested models with filtered and unfiltered training data

Filter_Analysis/wiki/Approach.md

@@ -1,4 +1,4 @@
-= The Approach =
+# The Approach
 
 The goal is to use a filtering algorithm such as the [[https://en.wikipedia.org/wiki/Kuwahara_filter#|Kuwahara Filter]] to 
 

Filter_Analysis/wiki/FilterAnalysis.md

@@ -0,0 +1,8 @@
+# Mitigating Gradient Attacks using Denoising Filters
+
+## Contents
+- [[Tests]]
+- [[Approach]]
+- [[Rationale]]
+- [[Notes]]
+- [[Timeline]]

Filter_Analysis/wiki/FilterAnalysis.wiki

@@ -1,7 +0,0 @@
-= Halting Gradient Attacks with Non-Gradient Defenses =
-
-== Contents ==
-- [[Tests]]
-- [[Approach]]
-- [[Rationale]]
-- [[Notes]]

Filter_Analysis/wiki/Notes.md

@@ -0,0 +1,32 @@
+# Notes on Filter-Based Defenses
+
+## Engineering Design Principles
+1. Clearly defined problem
+	- Defending gradient-based attacks using denoising filters as a buffer between an attacked image and a classifier
+2. Requirements
+3. Constraints
+	- Computing power
+4. Engineering standards
+	- [[https://peps.python.org/pep-0008/|PEP 8]]
+5. Cite applicable references
+	- [[https://pytorch.org/tutorials/beginner/fgsm_tutorial.html|FGSM Attack]]
+	- [[https://github.com/pytorch/examples/blob/main/mnist/main.py|MNIST Model]]
+	- [[https://www.cs.toronto.edu/~kriz/cifar.html|CIFAR-10]]
+6. Considered alternatives
+	a) Iterate on the design
+		i) Advantages
+			- Potentially more computationally efficient than an ML approach
+		ii) Disadvantages
+			- Potentially less effective than than an ML approach
+		iii) Risks
+			- Conventional algorithm may be more vulnerable to reverse engineering
+7. Evaluation process 
+	- Cross validation
+	- Effectiveness will be measured as the percent of correct classifications
+	- Testing clean vs. filtered training data
+	- Ablation variables:
+		- Different models
+		- Different datasets
+		- Different filters
+		- 
+8. Deliverables and timeline

Filter_Analysis/wiki/Notes.wiki

@@ -1,18 +0,0 @@
-= Notes on Filter-Based Defenses =
-
-== Engineering Design Principles ==
-1. Clearly defined problem
-	a) Defending gradient-based attacks using denoising filters as a buffer between an attacked image and a classifier
-2. Requirements
-3. Constraints
-4. Engineering standards
-5. Cite applicable references
-6. Considered alternatives
-	a) Iterate on the design
-		i) Advantages
-		ii) Disadvantages
-		iii) Risks
-7. Evaluation process 
-	a) Validation
-8. Deliverables and timeline
-9. 

Filter_Analysis/wiki/Tests.md

@@ -0,0 +1,156 @@
+# Test Process for Non-Gradient Filter Pipeline
+
+For each attack, the following tests are to be evaluated. The performance of each attack should be evaluated using cross validation with $k=5$.
+
+| Training | Test                    |
+|----------|-------------------------|
+| Clean    | Clean                   |
+| Clean    | Attacked                |
+| Clean    | Filtered (Not Attacked) |
+| Clean    | Filtered (Attacked)     |
+| Filtered | Filtered (Not Attacked) |
+| Filtered | Filtered (Attacked)     |
+
+## Testing on Pretrained Model Trained on Unfiltered Data
+Epsilon: 0.05
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 9605 / 10000 = 0.9605
+Filtered Accuracy = 9522 / 10000 = 0.9522
+
+Epsilon: 0.1
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 8743 / 10000 = 0.8743
+Filtered Accuracy = 9031 / 10000 = 0.9031
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 7107 / 10000 = 0.7107
+Filtered Accuracy = 8138 / 10000 = 0.8138
+
+Epsilon: 0.2
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 4876 / 10000 = 0.4876
+Filtered Accuracy = 6921 / 10000 = 0.6921
+
+Epsilon: 0.25
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 2714 / 10000 = 0.2714
+Filtered Accuracy = 5350 / 10000 = 0.535
+
+Epsilon: 0.3
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 1418 / 10000 = 0.1418
+Filtered Accuracy = 3605 / 10000 = 0.3605
+
+### Observations
+
+| $\epsilon$ | Attacked Accuracy | Filtered Accuracy | Ratio  |
+|------------|-------------------|-------------------|--------|
+| 0.05       | 0.9605            | 0.9522            | 0.9914 |
+| 0.1        | 0.8743            | 0.9031            | 1.0329 |
+| 0.15       | 0.7107            | 0.8138            | 1.1451 |
+| 0.2        | 0.4876            | 0.6921            | 1.4194 |
+| 0.25       | 0.2714            | 0.5350            | 1.9713 |
+| 0.3        | 0.1418            | 0.3605            | 2.5423 |
+
+- Filter seems to consitently increase accuracy
+	- When epsilon is too low to have a significant imact on the accuracy, the filter is seems to be counterproductive
+		- This may be avoidable by training on filtered data
+		- Low values of epsilon will be tested on filtered model to test this hypothesis
+
+## Testing on Model Trained with Filtered Data
+CNN classifier trained on MNIST dataset with 14 epochs. Kuwahara filter applied at runtime for each batch of training and test data.
+
+### Hypothesis
+Adding a denoising filter will increase accuracy against FGSM attack
+
+### Results
+Epsilon: 0.05
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 7288 / 10000 = 0.7288
+Filtered Accuracy = 9575 / 10000 = 0.9575
+Filtered:Attacked = 0.9575 / 0.7288 = 1.3138035126234906
+
+Epsilon: 0.1
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 2942 / 10000 = 0.2942
+Filtered Accuracy = 8268 / 10000 = 0.8268
+Filtered:Attacked = 0.8268 / 0.2942 = 2.8103331067301154
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 1021 / 10000 = 0.1021
+Filtered Accuracy = 5253 / 10000 = 0.5253
+Filtered:Attacked = 0.5253 / 0.1021 = 5.144955925563173
+
+Epsilon: 0.2
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 404 / 10000 = 0.0404
+Filtered Accuracy = 2833 / 10000 = 0.2833
+Filtered:Attacked = 0.2833 / 0.0404 = 7.012376237623762
+
+Epsilon: 0.25
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 234 / 10000 = 0.0234
+Filtered Accuracy = 1614 / 10000 = 0.1614
+Filtered:Attacked = 0.1614 / 0.0234 = 6.897435897435897
+
+Epsilon: 0.3
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 161 / 10000 = 0.0161
+Filtered Accuracy = 959 / 10000 = 0.0959
+Filtered:Attacked = 0.0959 / 0.0161 = 5.956521739130435
+
+### Observations
+- Model is more susceptable to FGSM than pretrained model
+- Model repsonds much better to filtered data than pretrained model
+- Even for $\epsilon = 0.25$, the model does better than random guessing (10 classes)
+	- Potential for boost algorithm
+- Filter is proportionally more effective for higher values of $\epsilon$ until $\epsilon=0.3$
+
+## Testing on Model Trained with Unfiltered Data
+CNN classifier, same as above, trained on 14 epochs of MNIST dataset without Kuwahara filtering. 
+
+### Hypothesis
+Given how the attacked model trained on filtered data performed against the FGSM attack, we expect that the model trained on unfiletered data will pereform poorly.
+
+### Results
+Epsilon: 0.05
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 9600 / 10000 = 0.96
+Filtered Accuracy = 8700 / 10000 = 0.87
+Filtered:Attacked = 0.87 / 0.96 = 0.90625
+
+Epsilon: 0.1
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 8753 / 10000 = 0.8753
+Filtered Accuracy = 8123 / 10000 = 0.8123
+Filtered:Attacked = 0.8123 / 0.8753 = 0.9280246772535131
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 7229 / 10000 = 0.7229
+Filtered Accuracy = 7328 / 10000 = 0.7328
+Filtered:Attacked = 0.7328 / 0.7229 = 1.013694840226864
+
+Epsilon: 0.2
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 5008 / 10000 = 0.5008
+Filtered Accuracy = 6301 / 10000 = 0.6301
+Filtered:Attacked = 0.6301 / 0.5008 = 1.2581869009584663
+
+Epsilon: 0.25
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 2922 / 10000 = 0.2922
+Filtered Accuracy = 5197 / 10000 = 0.5197
+Filtered:Attacked = 0.5197 / 0.2922 = 1.7785763175906915
+
+Epsilon: 0.3
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 1599 / 10000 = 0.1599
+Filtered Accuracy = 3981 / 10000 = 0.3981
+Filtered:Attacked = 0.3981 / 0.1599 = 2.4896810506566607
+
+### Observations
+- The ratio of filtered to attacked performance is stricty increasing
+- The unfiltered model seems to be less susceptable to the FGSM attack

Filter_Analysis/wiki/Tests.wiki

@@ -1,42 +0,0 @@
-= Test Process for Non-Gradient Filter Pipeline =
-
-For each attack, the following tests are to be evaluated. The performance of each attack should be evaluated using cross validation with $k=5$.
-
-| Training | Test                    |
-|----------|-------------------------|
-| Clean    | Clean                   |
-| Clean    | Attacked                |
-| Clean    | Filtered (Not Attacked) |
-| Clean    | Filtered (Attacked)     |
-| Filtered | Filtered (Not Attacked) |
-| Filtered | Filtered (Attacked)     |
-
-Epsilon: 0.05
-Original Accuracy = 9912 / 10000 = 0.9912
-Attacked Accuracy = 9605 / 10000 = 0.9605
-Filtered Accuracy = 9522 / 10000 = 0.9522
-
-Epsilon: 0.1
-Original Accuracy = 9912 / 10000 = 0.9912
-Attacked Accuracy = 8743 / 10000 = 0.8743
-Filtered Accuracy = 9031 / 10000 = 0.9031
-
-Epsilon: 0.15000000000000002
-Original Accuracy = 9912 / 10000 = 0.9912
-Attacked Accuracy = 7107 / 10000 = 0.7107
-Filtered Accuracy = 8138 / 10000 = 0.8138
-
-Epsilon: 0.2
-Original Accuracy = 9912 / 10000 = 0.9912
-Attacked Accuracy = 4876 / 10000 = 0.4876
-Filtered Accuracy = 6921 / 10000 = 0.6921
-
-Epsilon: 0.25
-Original Accuracy = 9912 / 10000 = 0.9912
-Attacked Accuracy = 2714 / 10000 = 0.2714
-Filtered Accuracy = 5350 / 10000 = 0.535
-
-Epsilon: 0.3
-Original Accuracy = 9912 / 10000 = 0.9912
-Attacked Accuracy = 1418 / 10000 = 0.1418
-Filtered Accuracy = 3605 / 10000 = 0.3605

Filter_Analysis/wiki/Timeline.md

@@ -0,0 +1,26 @@
+= Timeline of Progress =
+
+== Tuesday, February 27th, 2024 ==
+- Determined that lack of effectiveness for low values of epsilon for the FGSM attack is normal ([[https://pytorch.org/tutorials/beginner/fgsm_tutorial.html#accuracy-vs-epsilon|PyTorch Example Results]]). 
+- Finished trainable, saveable MNIST model
+- Working on manipulating the MNIST dataset for cross validation and filtering
+- Looking into implementing [[https://www.cs.toronto.edu/~kriz/cifar.html|CIFAR-10]] due to the model architecture and the nature of the images being classified
+
+== Thursday, February 29th, 2024 ==
+- Created functionality for Kuwahara filtering of batches of 64 images at runtime
+- Encountering crash in last batch
+
+== Monday, March 4th, 2024 ==
+- Last batch of epoch doesn't have 64 images, batch size now variable
+- Encountered crash when testing at end of epoch
+	- Fixed crash, testing required specifying batch size
+- All 14 epochs train successfully on filtered data
+- Added `--filter` option to enable filtering on training and test data
+	- Encountered crash, `args` not passed to `test` function
+	- `args` variable now passed to `test` function
+	- Filtered and unfiltered models saved to different files
+- Tested filtered model with FGSM attack
+	- Got results inline with unfiltered model
+	- Realized that I forgot to save the filtered model
+	- Tested actually filtered model with FGSM attack
+	- Got really good results inline with hypothesis