Directory structure overhaul, poster almost done

wiki/Approach.md

@@ -0,0 +1,7 @@
+# The Approach
+
+Attacking classifier models essentially boils down to adding precisely calculated noise to the input image, thereby tricking the classifier into selecting an incorrect class. The goal is to understand the efficacy of an array of denoising algorithms as adversarial machine learning defenses.
+
+## Requirements
+For a given filter to be beneficial to th e
+1. The filter 

wiki/DesignImpact.md

@@ -0,0 +1,36 @@
+# Engineering Design Principles
+1. Clearly defined problem
+	- Assess the efficacy of various denoising filters in preserving the accuracy of image classifier models under a noise-based attack.
+2. Requirements
+	- Only algorithmic approach for defense
+	- Must be faster than auto-encoder
+3. Constraints
+	- Computing power
+	- Memory usage
+	- Impossible to know who and how a model will be attacked
+4. Engineering standards
+	- [[https://peps.python.org/pep-0008/|PEP 8]]
+	- 
+5. Cite applicable references
+	- [[https://pytorch.org/tutorials/beginner/fgsm_tutorial.html|FGSM Attack]]
+	- [[https://github.com/pytorch/examples/blob/main/mnist/main.py|MNIST Model]]
+	- [[https://www.cs.toronto.edu/~kriz/cifar.html|CIFAR-10]]
+6. Considered alternatives
+	a) Iterate on the design
+		i) Advantages
+			- Potentially more computationally efficient than an ML approach
+			- Will likely use less memory than a model used to clean inputs
+			- No training (very computationally intense) stage
+		ii) Disadvantages
+			- Potentially less effective than than an ML approach
+		iii) Risks
+			- Conventional algorithm may be more vulnerable to reverse engineering
+7. Evaluation process 
+	- Cross validation
+	- Effectiveness will be measured as the percent of correct classifications
+	- Testing clean vs. filtered training data
+	- Ablation variables:
+		- Different models
+		- Different datasets
+		- Different filters
+8. Deliverables and timeline

wiki/FilterAnalysis.md

@@ -0,0 +1,8 @@
+# Mitigating Gradient Attacks using Denoising Filters
+
+## Contents
+- [[Tests]]
+- [[Approach]]
+- [[Rationale]]
+- [[DesignImpact]]
+- [[Timeline]]

wiki/Tests.md

@@ -0,0 +1,156 @@
+# Test Process for Non-Gradient Filter Pipeline
+
+For each attack, the following tests are to be evaluated. The performance of each attack should be evaluated using cross validation with $k=5$.
+
+| Training | Test                    |
+|----------|-------------------------|
+| Clean    | Clean                   |
+| Clean    | Attacked                |
+| Clean    | Filtered (Not Attacked) |
+| Clean    | Filtered (Attacked)     |
+| Filtered | Filtered (Not Attacked) |
+| Filtered | Filtered (Attacked)     |
+
+## Testing on Pretrained Model Trained on Unfiltered Data
+Epsilon: 0.05
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 9605 / 10000 = 0.9605
+Filtered Accuracy = 9522 / 10000 = 0.9522
+
+Epsilon: 0.1
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 8743 / 10000 = 0.8743
+Filtered Accuracy = 9031 / 10000 = 0.9031
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 7107 / 10000 = 0.7107
+Filtered Accuracy = 8138 / 10000 = 0.8138
+
+Epsilon: 0.2
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 4876 / 10000 = 0.4876
+Filtered Accuracy = 6921 / 10000 = 0.6921
+
+Epsilon: 0.25
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 2714 / 10000 = 0.2714
+Filtered Accuracy = 5350 / 10000 = 0.535
+
+Epsilon: 0.3
+Original Accuracy = 9912 / 10000 = 0.9912
+Attacked Accuracy = 1418 / 10000 = 0.1418
+Filtered Accuracy = 3605 / 10000 = 0.3605
+
+### Observations
+
+| $\epsilon$ | Attacked Accuracy | Filtered Accuracy | Ratio  |
+|------------|-------------------|-------------------|--------|
+| 0.05       | 0.9605            | 0.9522            | 0.9914 |
+| 0.1        | 0.8743            | 0.9031            | 1.0329 |
+| 0.15       | 0.7107            | 0.8138            | 1.1451 |
+| 0.2        | 0.4876            | 0.6921            | 1.4194 |
+| 0.25       | 0.2714            | 0.5350            | 1.9713 |
+| 0.3        | 0.1418            | 0.3605            | 2.5423 |
+
+- Filter seems to consitently increase accuracy
+	- When epsilon is too low to have a significant imact on the accuracy, the filter is seems to be counterproductive
+		- This may be avoidable by training on filtered data
+		- Low values of epsilon will be tested on filtered model to test this hypothesis
+
+## Testing on Model Trained with Filtered Data
+CNN classifier trained on MNIST dataset with 14 epochs. Kuwahara filter applied at runtime for each batch of training and test data.
+
+### Hypothesis
+Adding a denoising filter will increase accuracy against FGSM attack
+
+### Results
+Epsilon: 0.05
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 7288 / 10000 = 0.7288
+Filtered Accuracy = 9575 / 10000 = 0.9575
+Filtered:Attacked = 0.9575 / 0.7288 = 1.3138035126234906
+
+Epsilon: 0.1
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 2942 / 10000 = 0.2942
+Filtered Accuracy = 8268 / 10000 = 0.8268
+Filtered:Attacked = 0.8268 / 0.2942 = 2.8103331067301154
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 1021 / 10000 = 0.1021
+Filtered Accuracy = 5253 / 10000 = 0.5253
+Filtered:Attacked = 0.5253 / 0.1021 = 5.144955925563173
+
+Epsilon: 0.2
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 404 / 10000 = 0.0404
+Filtered Accuracy = 2833 / 10000 = 0.2833
+Filtered:Attacked = 0.2833 / 0.0404 = 7.012376237623762
+
+Epsilon: 0.25
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 234 / 10000 = 0.0234
+Filtered Accuracy = 1614 / 10000 = 0.1614
+Filtered:Attacked = 0.1614 / 0.0234 = 6.897435897435897
+
+Epsilon: 0.3
+Original Accuracy = 9793 / 10000 = 0.9793
+Attacked Accuracy = 161 / 10000 = 0.0161
+Filtered Accuracy = 959 / 10000 = 0.0959
+Filtered:Attacked = 0.0959 / 0.0161 = 5.956521739130435
+
+### Observations
+- Model is more susceptable to FGSM than pretrained model
+- Model repsonds much better to filtered data than pretrained model
+- Even for $\epsilon = 0.25$, the model does better than random guessing (10 classes)
+	- Potential for boost algorithm
+- Filter is proportionally more effective for higher values of $\epsilon$ until $\epsilon=0.3$
+
+## Testing on Model Trained with Unfiltered Data
+CNN classifier, same as above, trained on 14 epochs of MNIST dataset without Kuwahara filtering. 
+
+### Hypothesis
+Given how the attacked model trained on filtered data performed against the FGSM attack, we expect that the model trained on unfiletered data will pereform poorly.
+
+### Results
+Epsilon: 0.05
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 9600 / 10000 = 0.96
+Filtered Accuracy = 8700 / 10000 = 0.87
+Filtered:Attacked = 0.87 / 0.96 = 0.90625
+
+Epsilon: 0.1
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 8753 / 10000 = 0.8753
+Filtered Accuracy = 8123 / 10000 = 0.8123
+Filtered:Attacked = 0.8123 / 0.8753 = 0.9280246772535131
+
+Epsilon: 0.15000000000000002
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 7229 / 10000 = 0.7229
+Filtered Accuracy = 7328 / 10000 = 0.7328
+Filtered:Attacked = 0.7328 / 0.7229 = 1.013694840226864
+
+Epsilon: 0.2
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 5008 / 10000 = 0.5008
+Filtered Accuracy = 6301 / 10000 = 0.6301
+Filtered:Attacked = 0.6301 / 0.5008 = 1.2581869009584663
+
+Epsilon: 0.25
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 2922 / 10000 = 0.2922
+Filtered Accuracy = 5197 / 10000 = 0.5197
+Filtered:Attacked = 0.5197 / 0.2922 = 1.7785763175906915
+
+Epsilon: 0.3
+Original Accuracy = 9920 / 10000 = 0.992
+Attacked Accuracy = 1599 / 10000 = 0.1599
+Filtered Accuracy = 3981 / 10000 = 0.3981
+Filtered:Attacked = 0.3981 / 0.1599 = 2.4896810506566607
+
+### Observations
+- The ratio of filtered to attacked performance is stricty increasing
+- The unfiltered model seems to be less susceptable to the FGSM attack

wiki/Timeline.md

@@ -0,0 +1,26 @@
+= Timeline of Progress =
+
+== Tuesday, February 27th, 2024 ==
+- Determined that lack of effectiveness for low values of epsilon for the FGSM attack is normal ([[https://pytorch.org/tutorials/beginner/fgsm_tutorial.html#accuracy-vs-epsilon|PyTorch Example Results]]). 
+- Finished trainable, saveable MNIST model
+- Working on manipulating the MNIST dataset for cross validation and filtering
+- Looking into implementing [[https://www.cs.toronto.edu/~kriz/cifar.html|CIFAR-10]] due to the model architecture and the nature of the images being classified
+
+== Thursday, February 29th, 2024 ==
+- Created functionality for Kuwahara filtering of batches of 64 images at runtime
+- Encountering crash in last batch
+
+== Monday, March 4th, 2024 ==
+- Last batch of epoch doesn't have 64 images, batch size now variable
+- Encountered crash when testing at end of epoch
+	- Fixed crash, testing required specifying batch size
+- All 14 epochs train successfully on filtered data
+- Added `--filter` option to enable filtering on training and test data
+	- Encountered crash, `args` not passed to `test` function
+	- `args` variable now passed to `test` function
+	- Filtered and unfiltered models saved to different files
+- Tested filtered model with FGSM attack
+	- Got results inline with unfiltered model
+	- Realized that I forgot to save the filtered model
+	- Tested actually filtered model with FGSM attack
+	- Got really good results inline with hypothesis