feat: Update website format and remove unused files

src/_quarto.yml

@@ -108,8 +108,6 @@ website:
         contents:
             - href: vision/tasks.qmd
               text: "Tasks"
-            - href: math.qmd
-              text: "Math"
       
       - section: "Tools and Frameworks"
         contents:

src/theory/hyperparameter_tuning.qmd

@@ -74,6 +74,93 @@ model = SomeModel()
 bayes_search = BayesSearchCV(estimator=model, search_spaces=param_space)
 ```
 
+### 4. Gradient-based Optimization
+Gradient-based optimization techniques leverage the gradients of the loss function with respect to hyperparameters to find the optimal values. These methods are often more efficient than exhaustive search methods as they follow the direction of steepest descent, making them suitable for continuous hyperparameters.
+
+One popular approach within this category is the Gradient Descent algorithm, which iteratively adjusts hyperparameters in the direction that reduces the loss function. Another method is Hypergradient Descent, which extends this idea to also adjust learning rates during the optimization process.
+
+```python
+import tensorflow as tf
+from tensorflow.keras.optimizers import Adam
+
+# Example of gradient-based optimization using TensorFlow
+model = SomeModel()
+optimizer = Adam(learning_rate=0.01)
+
+# Define a training step
+@tf.function
+def train_step(data, labels):
+    with tf.GradientTape() as tape:
+        predictions = model(data)
+        loss = compute_loss(labels, predictions)
+    gradients = tape.gradient(loss, model.trainable_variables)
+    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
+```
+
+### 5. Evolutionary Algorithms
+Evolutionary algorithms (EAs) draw inspiration from natural evolution to optimize hyperparameters. These algorithms use mechanisms such as selection, mutation, and crossover to evolve a population of candidate solutions over several generations. Techniques like Genetic Algorithms (GA) and Covariance Matrix Adaptation Evolution Strategy (CMA-ES) are prominent examples.
+
+EAs can efficiently explore complex hyperparameter spaces and are particularly useful when the objective function is noisy or when there are multiple local optima.
+
+```python
+from deap import base, creator, tools, algorithms
+
+# Define fitness function and individual
+creator.create("FitnessMax", base.Fitness, weights=(1.0,))
+creator.create("Individual", list, fitness=creator.FitnessMax)
+
+# Register evolutionary operations
+toolbox = base.Toolbox()
+toolbox.register("attr_float", random.random)
+toolbox.register("individual", tools.initRepeat, creator.Individual, toolbox.attr_float, n=10)
+toolbox.register("population", tools.initRepeat, list, toolbox.individual)
+toolbox.register("evaluate", evaluate_model)
+toolbox.register("mate", tools.cxTwoPoint)
+toolbox.register("mutate", tools.mutGaussian, mu=0, sigma=1, indpb=0.2)
+toolbox.register("select", tools.selTournament, tournsize=3)
+
+# Perform the evolutionary algorithm
+population = toolbox.population(n=50)
+algorithms.eaSimple(population, toolbox, cxpb=0.5, mutpb=0.2, ngen=40, stats=None, halloffame=None, verbose=True)
+```
+
+### 6. Population Based Training (PBT)
+Population Based Training (PBT) is a hybrid optimization technique that combines elements of evolutionary algorithms and hyperparameter tuning. PBT maintains a population of models with different hyperparameters, periodically replacing poor-performing models with better-performing ones and mutating hyperparameters to explore new configurations.
+
+PBT is particularly effective for large-scale models and tasks that require significant computational resources, as it allows for parallel evaluation and optimization of multiple models.
+
+```python
+import tensorflow as tf
+from tensorflow.keras.optimizers import Adam
+
+# Define a simple PBT loop
+population_size = 10
+generations = 5
+population = [SomeModel() for _ in range(population_size)]
+optimizers = [Adam(learning_rate=0.01) for _ in range(population_size)]
+
+for generation in range(generations):
+    # Train each model in the population
+    for i in range(population_size):
+        for data, labels in dataset:
+            with tf.GradientTape() as tape:
+                predictions = population[i](data)
+                loss = compute_loss(labels, predictions)
+            gradients = tape.gradient(loss, population[i].trainable_variables)
+            optimizers[i].apply_gradients(zip(gradients, population[i].trainable_variables))
+    
+    # Evaluate performance and apply selection/mutation
+    scores = [evaluate_model(model) for model in population]
+    best_indices = np.argsort(scores)[-population_size//2:]
+    worst_indices = np.argsort(scores)[:population_size//2]
+    
+    for i in range(len(worst_indices)):
+        population[worst_indices[i]] = population[best_indices[i % len(best_indices)]]
+        optimizers[worst_indices[i]].learning_rate = optimizers[best_indices[i % len(best_indices)]].learning_rate * (0.8 + 0.4 * np.random.rand())
+```
+
+### Conclusion
+Hyperparameter tuning is an essential step in building effective machine learning models. By using techniques like grid search, random search, Bayesian optimization, gradient-based optimization, evolutionary algorithms, and Population Based Training (PBT), we can find the best hyperparameters for our models and improve their performance on unseen data. Each method has its strengths and is suitable for different types of problems and computational constraints, making it crucial to choose the appropriate technique for a given task.
 ### Conclusion
 
 Hyperparameter tuning is an essential step in building effective machine learning models. By using techniques like grid search, random search, or Bayesian optimization, we can find the best hyperparameters for our model and improve its performance on unseen data.