feat: Add "Perplexity in AI Models" to theory section

src/_quarto.yml

@@ -29,6 +29,13 @@ website:
           - href: about.qmd
             text: About this Cookbook
 
+      - section: "Thoughts"
+        contents: 
+          - href: theory/dont_mess_with_kittens.qmd
+            text: "High Stakes, and... Kittens"
+          - href: theory/good_enough.qmd
+            text: "What is Good Enough ?"
+
       - section: "Theory"
         contents:
           - href: theory/activations.qmd
@@ -52,19 +59,27 @@ website:
             text: "Optimizers"
           - href: theory/quantization.qmd
             text: "Quantization"
+          - href: theory/perplexity_in_ai.qmd
+            text: "Perplexity and Quantization"
           - href: theory/regularization.qmd
             text: "Regularization"
-          - href: theory/training.qmd
-            text: "Training"
+          - section: "Training"
+            href: theory/training.qmd
+            contents:
+            - href: theory/training.qmd
+              text: "Training"
+            - href: theory/dying_neurons.qmd
+              text: "Dying Neurons"
+            - href: theory/overfitting.qmd
+              text: "Overfitting"
+
+          - href: theory/perplexity_in_ai.qmd
+            text: "Perplexity and Quantization"
           - href: theory/mixture_of_models.qmd
             text: "Mixture of Models"
-          - href: theory/dont_mess_with_kittens.qmd
-            text: "High Stakes, and... Kittens"
 
       - section: "Large Language Models"
         contents:
-          - href: theory/good_enough.qmd
-            text: "What is Good Enough ?"
           - href: llms/tasks.qmd
             text: "Tasks"
           - href: llms/tokenizers.qmd
@@ -81,8 +96,6 @@ website:
           - href: llms/rag_systems.qmd
             text: "Retrival Augmented Generation"
 
-
-
       - section: "Computer Vision Models"
         contents:
             - href: vision/tasks.qmd

src/theory/dying_neurons.qmd

@@ -0,0 +1,84 @@
+# Detecting Dying Neurons in Artificial Intelligence Modnels
+
+## Introduction
+
+In the field of artificial intelligence (AI), particularly deep learning, "dying" neurons refer to a phenomenon where certain neurons within an AI model stop contributing to the network's output. This can occur due to various reasons such as improper initialization or extreme weight updates during training. Detecting and addressing dying neurons is crucial for maintaining optimal performance in deep learning models.
+
+## Understanding Dying Neurons
+
+During the training process, AI models learn by adjusting their weights based on input data. However, if a neuron's output remains consistently close to zero or its gradient becomes negligible, it is considered "dead" or "dying." This can lead to suboptimal performance and reduced model accuracy.
+
+## Detecting Dying Neurons in AI Models
+
+Detecting dying neurons involves analyzing the activations of each layer within a deep learning model during training. By monitoring these activations, we can identify if any neuron is not contributing to the network's output and take appropriate measures to address it.
+
+### Step 1: Set Up Your Environment
+
+Firstly, ensure that you have installed all necessary libraries for your project. For this example, we will use TensorFlow as our deep learning framework. Install TensorFlow using pip:
+
+```bash
+pip install tensorflow
+```
+
+Next, import the required modules in your Python script:
+
+```python
+import tensorflow as tf
+from tensorflow.keras.models import Sequential
+from tensorflow.keras.layers import Dense
+```
+
+### Step 2: Create a Sample Model
+
+For demonstration purposes, let's create a simple feedforward neural network with one hidden layer and an output layer using the Keras API in TensorFlow:
+
+```python
+model = Sequential([
+    Dense(64, activation='relu', input_shape=(784,)),  # Hidden Layer
+    Dense(10, activation='softmax')                   # Output Layer
+])
+```
+
+### Step 3: Monitoring Neuron Activations During Training
+
+To detect dying neurons during training, we need to monitor the activations of each layer. We can achieve this by creating a custom callback in Keras that logs the mean activation value for each layer after every epoch:
+
+```python
+class DeadNeuronDetector(tf.keras.callbacks.Callback):
+    def on_epoch_end(self, epoch, logs=None):
+        print("\nEpoch {} --------------------------------------".format(epoch))
+        
+        for layer in self.model.layers:
+            if 'activation' in layer.get_config():
+                activations = layer.output[:5]  # Get the first five samples of output from this layer
+                
+                mean_activation = tf.reduce_mean(tf.abs(activations))
+                print("Mean activation for {} is: {:.4f}".format(layer.name, mean_activation.numpy()))
+```
+
+### Step 4: Train the Model and Detect Dying Neurons
+
+Now that we have our custom callback set up, let's train our model using a sample dataset (e.g., MNIST) while monitoring neuron activations:
+
+```python
+(x_train, y_n_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
+x_train, x_test = x_train / 255.0, x_test / 255.0
+
+model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
+dead_neuron_detector = DeadNeuronDetector()
+history = model.fit(x_train, y_n_train, epochs=10, validation_data=(x_test, y_test), callbacks=[dead_neuron_detector])
+```
+
+### Step 5: Addressing Dying Neurons
+
+If you detect dying neurons during training, consider the following approaches to address this issue:
+
+1. **Weight Initialization**: Use different weight initialization techniques such as He or Glorot initialization instead of default ones like Xavier or random normal.
+2. **Learning Rate Adjustment**: Try using a smaller learning rate or implementing adaptive learning rates (e.g., Adam, AdaGrad) to prevent extreme updates that may cause neurons to die.
+3. **Regularization Techniques**: Apply regularization techniques like dropout or L1/L2 regularization to encourage the model to learn more robust features and reduce overfitting.
+4. **Batch Normalization**: Incorporate batch normalization layers in your network architecture, which can help maintain stable activations throughout training.
+5. **Revive Dead Neurons**: If a neuron dies during training, you may try to revive it by reinitializing its weights and continuing the training process.
+
+## Conclusion
+
+Detecting dying neurons in AI models is essential for maintaining optimal performance and accuracy. By monitoring layer activations during training using custom callbacks, we can identify dead or dying neurons and take appropriate measures to address them. Implementing techniques such as weight initialization adjustments, learning rate tuning, regularization methods, batch normalization, and reviving dead neurons can help mitigate this issue in deep learning models.

src/theory/overfitting.qmd

@@ -0,0 +1,94 @@
+# Understanding Overfitting in Machine Learning Models
+
+## Introduction
+
+Overfitting occurs when a machine learning model learns to perform well on its training data but fails to generalize and make accurate predictions on new, unseen data. This phenomenon can lead to poor performance of the model in real-world scenarios. In this article, we will discuss overfamming, how to detect it using training metrics, and provide code examples with plots that illustrate the concept.
+
+## Detecting Overfitting Using Training Metrics
+
+To identify if a machine learning model is suffering from overfitting, you can monitor its performance on both the training set and validation set during the training process. The key indicators of overfitting are:
+
+1. High accuracy or low error rate on the training data but poor performance on the validation data.
+2. A large gap between the model's performance metrics (e.g., accuracy, precision, recall) for the training and validation sets.
+
+### Code Example
+
+Here is a Python code example using scikit-learn to train a logistic regression classifier with overfitting:
+
+```{python}
+import numpy as np
+from sklearn.datasets import make_classification
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import accuracy_score, confusion_matrix
+from sklearn.model_selection import train_test_split
+
+# Generate synthetic data for demonstration purposes
+X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
+
+# Split the dataset into training and validation sets
+X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=42)
+
+# Train a logistic regression classifier with overfitting
+clf = LogisticRegression(max_iter=100).fit(X_train, y_train)
+
+# Evaluate the model on training and validation sets
+y_pred_train = clf.predict(X_train)
+y_pred_val = clf.predict(X_val)
+
+print("Training accuracy:", accuracy_score(y_train, y_pred_train))
+print("Validation accuracy:", accuracy_score(y_val, y_pred_val))
+```
+
+## Visualizing Overfitting with Plots
+
+To better understand overfitting and its impact on model performance, we can visualize the training metrics using plots. Here are two examples of code blocks that generate plots for illustrating overfitting:
+
+### Plot 1: Training vs Validation Accuracy
+
+```{python}
+import matplotlib.pyplot as plt
+
+train_accuracies = [0.95, 0.96, 0, 0.97] # Example training accuracies for different epochs
+val_accuracies = [0.75, 0.72, 0.71, 0.70] # Corresponding validation accuracies
+
+plt.plot(train_accuracies, label="Training Accuracy")
+plt.plot(val_accuracies, label="Validation Accuracy")
+plt.xlabel("Epoch")
+plt.ylabel("Accuracy")
+plt.title("Overfitting: Training vs Validation Accuracy")
+plt.legend()
+plt.show()
+```
+
+### Plot 2: Learning Curves for Overfitting Detection
+
+Learning curves are a powerful tool to visualize the relationship between training and validation performance as more data is used during model training. Here's an example of generating learning curves using scikit-learn:
+
+```{python}
+from sklearn.model_selection import learning_curve
+import matplotlib.pyplot as plt
+
+train_sizes, train_scores, val_scores = learning_curve(clf, X, y, cv=5)
+
+# Calculate mean and standard deviation of training set scores
+train_mean = np.mean(train_scores, axis=1)
+train_std = np.std(train_scores, axis=1)
+
+# Calculate mean and standard deviation of validation set scores
+val_mean = np.mean(val_scores, axis=1)
+val_std = np.std(val_scores, axis=1)
+
+plt.fill_between(train_sizes, train_mean - train_std, train_mean + train_std, alpha=0.1, color="r")
+plt.title(label="Training Score", color="r")
+plt.fill_between(train_sizes, val_mean - val_std, val_mean + val_std, alpha=0.1, color="g")
+plt.plot(train_sizes, val_mean, label="Cross-validation Score", color="g")
+plt.xlabel("Training examples used")
+plt.ylabel("Score")
+plt.title("Learning Curves for Overfitting Detection")
+plt.legend()
+plt.show()
+```
+
+## Conclusion
+
+Overfitting is a common challenge in machine learning, and it can lead to poor model performance on unseen data. By monitoring training metrics such as accuracy or error rates and visualizing the results using plots like training vs validation accuracy graphs and learning curves, you can detect overfitting early during the model development process. This allows for timely interventions, such as regularization techniques or adjusting hyperparameters to improve your model's generalization capabilities.

src/theory/perplexity_in_ai.qmd

@@ -0,0 +1,27 @@
+# Perplexity in AI Models
+
+Quantization, as detailed in the [Quantization](https://sebdg-ai-cookbook.hf.space/theory/quantization.html) page, reduces the memory footprint of neural networks by using lower-precision formats. This technique is vital for deploying models on devices with limited computational power.
+
+## Introducing the Perplexity Metric
+
+Perplexity is a key metric used to evaluate language models, measuring their effectiveness in predicting the next word in a sequence. It essentially indicates the model's uncertainty; a lower perplexity means better predictive performance.
+
+## What is Perplexity?
+
+Perplexity is defined as the exponentiation of the entropy of the model's probability distribution. For language models, it is computed as:
+
+\[ \text{Perplexity}(P) = \exp \left( -\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_1, w_2, \ldots, w_{i-1}) \right) \]
+
+Here, \( w_i \) represents the \(i\)-th word in the sequence, and \( P(w_i | w_1, w_2, \ldots, w_{i-1}) \) is the conditional probability of the \(i\)-th word given the previous words.
+
+## Importance of Perplexity in AI
+
+Perplexity provides a single scalar value that summarizes how well a language model predicts test data, facilitating comparisons between models or versions of the same model.
+
+## Relating Perplexity to Quantization
+
+While quantization itself doesn’t directly affect perplexity, the reduction in model precision can impact overall performance, potentially increasing perplexity if errors are introduced. Balancing memory efficiency from quantization with maintaining low perplexity is crucial.
+
+## Conclusion
+
+Quantization optimizes AI models for deployment on resource-constrained devices. Understanding perplexity helps in evaluating model effectiveness. For a deeper dive into quantization, visit the [Quantization](https://sebdg-ai-cookbook.hf.space/theory/quantization.html) page.