Adog64/Adversarial-Machine-Learning-Clinic
1
0
Fork 0
Code Issues 3 Pull Requests Packages Projects 1 Releases Wiki Activity

Various filters defending unmodified mnist cnn classifier

Browse Source
This commit is contained in:
Aidan Sharpe
2024-04-04 15:12:14 -04:00
parent d0a09e839f
commit 47b14362de
9 changed files with 32091 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
Filter_Analysis/wiki/Approach.md
View File

@ -1,4 +1,9 @@
# The Approach
The goal is to use a filtering algorithm such as the [[https://en.wikipedia.org/wiki/Kuwahara_filter#|Kuwahara Filter]] to
Attacking classifier models essentially boils down to adding precisely calculated noise to the input image, thereby tricking the classifier into selecting an incorrect class. The goal is to understand the efficacy of an array of denoising algorithms as adversarial machine learning defenses.
## Individual Denoising Algorithms
## An Ensemble Approach
## Training the Model on Filtered Data

32
Filter_Analysis/wiki/Notes.md
View File

@ -1,32 +0,0 @@
# Notes on Filter-Based Defenses
## Engineering Design Principles
1. Clearly defined problem
- Defending gradient-based attacks using denoising filters as a buffer between an attacked image and a classifier
2. Requirements
3. Constraints
- Computing power
4. Engineering standards
- [[https://peps.python.org/pep-0008/|PEP 8]]
5. Cite applicable references
- [[https://pytorch.org/tutorials/beginner/fgsm_tutorial.html|FGSM Attack]]
- [[https://github.com/pytorch/examples/blob/main/mnist/main.py|MNIST Model]]
- [[https://www.cs.toronto.edu/~kriz/cifar.html|CIFAR-10]]
6. Considered alternatives
a) Iterate on the design
i) Advantages
- Potentially more computationally efficient than an ML approach
ii) Disadvantages
- Potentially less effective than than an ML approach
iii) Risks
- Conventional algorithm may be more vulnerable to reverse engineering
7. Evaluation process
- Cross validation
- Effectiveness will be measured as the percent of correct classifications
- Testing clean vs. filtered training data
- Ablation variables:
- Different models
- Different datasets
- Different filters
-
8. Deliverables and timeline

100
Filter_Analysis/wiki/Results.md Normal file
View File

@ -0,0 +1,100 @@
# Experimental Results
## Models Trained on Various Filters
**NOTE**: The results in this section contain an oversight in the defense strategy. While models were trained using different filters, they were all defended from FGSM using a Kuwahara filter.
### Model Trained on Unfiltered MNIST Dataset
| $\epsilon$ | Accuracy |
|---------|----------|
| 0.05 | 0.9600 |
| 0.10 | 0.8753 |
| 0.15 | 0.7228 |
| 0.20 | 0.5008 |
| 0.25 | 0.2922 |
| 0.30 | 0.1599 |
### Model Trained on Kuwahara (R=5) Filtered MNIST Dataset
| $\epsilon$ | Attacked Accuracy | Filtered Accuracy | Ratio |
|---------|-------------------|-------------------|--------|
| 0.05 | 0.9605 | 0.9522 | 0.9914 |
| 0.1 | 0.8743 | 0.9031 | 1.0329 |
| 0.15 | 0.7107 | 0.8138 | 1.1451 |
| 0.2 | 0.4876 | 0.6921 | 1.4194 |
| 0.25 | 0.2714 | 0.5350 | 1.9713 |
| 0.3 | 0.1418 | 0.3605 | 2.5423 |
### Model Trained on Gaussian Blurred (K-Size=5x5) MNIST Dataset
| $\epsilon$ | Attacked Accuracy | Filtered Accuracy | Ratio |
|---------|-------------------|-------------------|-------|
| 0.05 | 0.9192 | 0.9325 | 1.014 |
| 0.10 | 0.7629 | 0.8802 | 1.154 |
| 0.15 | 0.4871 | 0.7865 | 1.615 |
| 0.20 | 0.2435 | 0.6556 | 2.692 |
| 0.25 | 0.1093 | 0.5024 | 4.596 |
| 0.30 | 0.0544 | 0.3522 | 6.474 |
### Model Trained on Bilateral Filtered (d=5) MNIST Dataset
| $\epsilon$ | Attacked Accuracy | Filtered Accuracy | Ratio |
|---------|-------------------|-------------------|-------|
| 0.05 | 0.9078 | 0.9287 | 1.023 |
| 0.10 | 0.7303 | 0.8611 | 1.179 |
| 0.15 | 0.4221 | 0.7501 | 1.777 |
| 0.20 | 0.1927 | 0.6007 | 3.117 |
| 0.25 | 0.0873 | 0.4433 | 5.078 |
| 0.30 | 0.0525 | 0.3023 | 5.758 |
## Models Defended with Various Filters
### Tabulated Results
| $\epsilon$ | FGSM | Kuwahara | Bilateral | Gaussian Blur | Random Noise |
|------------|--------|----------|-----------|---------------|--------------|
| 0.05 | 0.9600 | 0.8700 | 0.8902 | 0.9271 | 0.9603 |
| 0.10 | 0.8753 | 0.8123 | 0.8133 | 0.8516 | 0.8677 |
| 0.15 | 0.7229 | 0.7328 | 0.7098 | 0.7415 | 0.7153 |
| 0.20 | 0.5008 | 0.6301 | 0.5683 | 0.5983 | 0.4941 |
| 0.25 | 0.2922 | 0.5197 | 0.4381 | 0.4591 | 0.2843 |
| 0.30 | 0.1599 | 0.3981 | 0.3364 | 0.3481 | 0.1584 |
### Raw Program Output
Epsilon: 0.05
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 9600 / 10000 = 0.96
Kuwahara Accuracy = 8700 / 10000 = 0.87
Bilateral Accuracy = 8902 / 10000 = 0.8902
Gaussian Blur Accuracy = 9271 / 10000 = 0.9271
Noisy Accuracy = 9603 / 10000 = 0.9603
Epsilon: 0.1
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 8753 / 10000 = 0.8753
Kuwahara Accuracy = 8123 / 10000 = 0.8123
Bilateral Accuracy = 8133 / 10000 = 0.8133
Gaussian Blur Accuracy = 8516 / 10000 = 0.8516
Noisy Accuracy = 8677 / 10000 = 0.8677
Epsilon: 0.15000000000000002
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 7229 / 10000 = 0.7229
Kuwahara Accuracy = 7328 / 10000 = 0.7328
Bilateral Accuracy = 7098 / 10000 = 0.7098
Gaussian Blur Accuracy = 7415 / 10000 = 0.7415
Noisy Accuracy = 7153 / 10000 = 0.7153
Epsilon: 0.2
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 5008 / 10000 = 0.5008
Kuwahara Accuracy = 6301 / 10000 = 0.6301
Bilateral Accuracy = 5683 / 10000 = 0.5683
Gaussian Blur Accuracy = 5983 / 10000 = 0.5983
Noisy Accuracy = 4941 / 10000 = 0.4941
Epsilon: 0.25
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 2922 / 10000 = 0.2922
Kuwahara Accuracy = 5197 / 10000 = 0.5197
Bilateral Accuracy = 4381 / 10000 = 0.4381
Gaussian Blur Accuracy = 4591 / 10000 = 0.4591
Noisy Accuracy = 2843 / 10000 = 0.2843
Epsilon: 0.3
Original Accuracy = 9920 / 10000 = 0.992
Attacked Accuracy = 1599 / 10000 = 0.1599
Kuwahara Accuracy = 3981 / 10000 = 0.3981
Bilateral Accuracy = 3364 / 10000 = 0.3364
Gaussian Blur Accuracy = 3481 / 10000 = 0.3481
Noisy Accuracy = 1584 / 10000 = 0.1584

42
Filter_Analysis/wiki/Results.wiki
View File

@ -1,42 +0,0 @@
= Experimental Results =
== Model Trained on Unfiltered MNIST Dataset ==
| Epsilon | Accuracy |
|---------|----------|
| 0.05 | 0.9600 |
| 0.10 | 0.8753 |
| 0.15 | 0.7228 |
| 0.20 | 0.5008 |
| 0.25 | 0.2922 |
| 0.30 | 0.1599 |
== Model Trained on Kuwahara (R=5) Filtered MNIST Dataset ==
| Epsilon | Attacked Accuracy | Filtered Accuracy | Ratio |
|---------|-------------------|-------------------|--------|
| 0.05 | 0.9605 | 0.9522 | 0.9914 |
| 0.1 | 0.8743 | 0.9031 | 1.0329 |
| 0.15 | 0.7107 | 0.8138 | 1.1451 |
| 0.2 | 0.4876 | 0.6921 | 1.4194 |
| 0.25 | 0.2714 | 0.5350 | 1.9713 |
| 0.3 | 0.1418 | 0.3605 | 2.5423 |
== Model Trained on Gaussian Blurred (K-Size=5x5) MNIST Dataset ==
| Epsilon | Attacked Accuracy | Filtered Accuracy | Ratio |
|---------|-------------------|-------------------|-------|
| 0.05 | 0.9192 | 0.9325 | 1.014 |
| 0.10 | 0.7629 | 0.8802 | 1.154 |
| 0.15 | 0.4871 | 0.7865 | 1.615 |
| 0.20 | 0.2435 | 0.6556 | 2.692 |
| 0.25 | 0.1093 | 0.5024 | 4.596 |
| 0.30 | 0.0544 | 0.3522 | 6.474 |
== Model Trained on Bilateral Filtered (d=5) MNIST Dataset ==
| Epsilon | Attacked Accuracy | Filtered Accuracy | Ratio |
|---------|-------------------|-------------------|-------|
| 0.05 | 0.9078 | 0.9287 | 1.023 |
| 0.10 | 0.7303 | 0.8611 | 1.179 |
| 0.15 | 0.4221 | 0.7501 | 1.777 |
| 0.20 | 0.1927 | 0.6007 | 3.117 |
| 0.25 | 0.0873 | 0.4433 | 5.078 |
| 0.30 | 0.0525 | 0.3023 | 5.758 |