Computional Intelligence Algorithms

Table of Contents

1. Neural Networks Projects

   • NN1: Base Implementation
   • NN2: Backpropagation
   • NN3: Momentum and RMSProp
   • NN4: Softmax Classification
   • NN5: Activation Functions
   • NN6: Regularization

2. Kohonen Self-Organizing Maps

   • Synthetic Datasets
   • Real-World Datasets

3. Genetic Algorithms Projects

   • Function Optimization
   • Rectangle Packing
   • Neural Network Optimization

Neural Networks Projects

NN1: Base Implementation

Goal: Manual weight adjustment for regression
Key Insights:

• Specialized neuron tuning for steps-large dataset
• Data standardization essential for non-linear modeling
  Results:
  | Dataset | Architecture | Test MSE |
  |----------------|-------------|----------|
  | square-simple | [10] | 4.4 |
  | steps-large | [5] | 0.00 |

NN2: Backpropagation

Goal: Batch vs online update comparison
Key Insights:

• Batch updates accelerate convergence
• High learning rates (0.5) needed for complex data
  Top Result: 3.0 MSE on square-simple
  
  Figure: Loss convergence for MPG dataset

NN3: Momentum and RMSProp

Goal: Optimization algorithm analysis
Key Findings:

• RMSProp superior for discontinuous functions
• Momentum better for multimodal landscapes
  Result Highlight: RMSProp achieved 2.9 MSE vs Momentum's 37.7 on steps

NN4: Softmax Classification

Goal: Softmax impact evaluation
Key Insight: Stabilizes learning in classification
Result: Improved xor3 F-measure from 0.42 to 0.98

Figure: Iris classification confusion matrix

NN5: Activation Functions

Goal: Activation function comparison
Key Insight: Performance varies by problem type
Visual Comparison:

Figure: ReLU on steps-large

Figure: Sigmoid on steps-large

NN6: Regularization

Goal: Regularization method evaluation
Key Findings:

• L2 most effective for complex problems
• Dropout too aggressive for small networks
  Top Result: L2 achieved 0.9394 F1 on rings5-sparse

---

Kohonen Self-Organizing Maps

Synthetic Datasets

Hexagon Dataset:

Figure: 6-neuron cluster organization (97% accuracy)

Cube Dataset:

Figure: 8-neuron organization (82% accuracy)

Key Insights:

• Neuron count should match true clusters
• Topology type (hex vs square) has minimal impact
• Neurons position at cluster centers

Real-World Datasets

MNIST:

Figure: Partial digit organization

Human Activity:

Figure: Activity pattern grouping

Findings:

• Requires larger architectures for complex data
• Maintains high per-neuron purity (100% accuracy common)

---

Genetic Algorithms Projects

Function Optimization

Quadratic Function:

Figure: Minimization in 100 iterations

Rastrigin Function:

Figure: Global minimum found in 200 iterations

Key Strength: Handles multimodal landscapes effectively

Rectangle Packing

Solutions:

Radius	Value	Rectangles	Visualization
800	30,800	74
1200	28,160	86

Innovation: Effective space utilization in circular constraints

Neural Network Optimization

Iris Classification:

• Achieved 100% accuracy
• Training convergence:
  

MPG Regression:

• Reached 0.47 MSE
• Outperformed gradient-based methods

---

Cross-Project Insights

1. Algorithm Selection:

   • NN: Precise function approximation
   • SOM: Intuitive data exploration
   • GA: Global optimization capabilities

2. Parameter Sensitivity:

   • Learning rates critical for NN
   • Neuron count decisive for SOM
   • Mutation rates vital for GA

3. Visualization Value:

   • Reveals patterns beyond numerical metrics
   • Critical for model interpretation

4. Practical Applications:

   • NN: Complex pattern recognition
   • SOM: Customer segmentation
   • GA: Resource allocation problems

"The right tool for the right problem" - Optimal results come from matching algorithm strengths to problem characteristics.