Update README.md

README.md

@@ -3,6 +3,7 @@ license: apache-2.0
 
 ---
 
+
 # Use in Colab
 
 from huggingface_hub import snapshot_download
@@ -143,6 +144,160 @@ python main_menu.py --task advanced_inference --query "Analyze the economic effe
 ```
 
 
+# Explanation:
+
+World Model Optimisation:
+-------------------------------------------------------------------
+Input: I_i
+-------------------------------------------------------------------
+Rotary Positional Encoding:
+
+emb_i = RoPE (Input)
+-------------------------------------------------------------------
+Token_i = t_i transformer(, k_beams = k, n_tokens = j)
+
+CE_Loss = CE_loss(token_i , true tokens)
+
+-------------------------------------------------------------------
+Variance of the next token + Entropy of the sequence =  State Score
+-------------------------------------------------------------------
+Representation Network: 
+GAN/VAE/SAE (o_t -> s_t)
+
+If the final hidden layer of the transformer outputs o_t of size S
+
+h_t = GELU(sum(W.o_t + b))
+
+Reconstruction Loss (o_t , h_t)
+
+
+-------------------------------------------------------------------
+Dynamics Network (s_t -> s_t+1)
+
+... -> LSTM(s_t)  -> LSTM(s_t+1) -> ...
+
+min MSE (s_t+1 , z_t+1 )
+
+State mapping
+-------------------------------------------------------------------
+Utilise dynamics influence:
+
+Action_i = a_i = t_1 , ... , t_n
+
+Prediction Network : mcts( Q(s,a) , gamma * LSTM(s_t) , delta * State Score (s_t),  tree_depth = m, num_simulations) -> Q(s_t+1)
+
+action search / selection
+-------------------------------------------------------------------
+Optimise the KL divergence between the policy of actions (and the tokens that were selected in those actions) and the actual sequences in the training data.
+
+Policy_i = p_i =  a_1, ... ,a_n
+
+min - KL(p_i / true_sequences)
+
+-------------------------------------------------------------------
+Inference: 
+
+Thought_i = p_i , ... , p_n
+
+Tree of Thought : 
+Example:
+
+-----------------
+1
+-----------------
+121
+122
+123
+-----------------
+12131
+12132
+12133
+
+12231
+12232
+12233
+
+12331
+12332
+12333
+---------------
+
+= Graph(system prompt, children = 3, depth = 4, min - KL(p_i / true_sequences))
+
+Graph(Thought_i -> Thought i+1)
+
+Min ThoughtLoss()
+
+-------------------------------------------------------------------
+
+Backpropagate back through each section, get gradients for:
+-------------------------------------------------------------------
+for thought batch size = b_t:
+
+d ThoughtLoss
+______________
+d Graph(Thought_i -> Thought_i+1)
+
+-------------------------------------------------------------------
+for policy batch size = b_p:
+
+d KL(p_i / true_sequences)
+_______________________
+d Prediction_Network
+
+-------------------------------------------------------------------
+for state batch size: b_s:
+
+d MSE(s_t+1 , z_t+1 )
+_________________
+d Dynamics Network
+
+-------------------------------------------------------------------
+for state batch size: b_s:
+
+d Contrastive Loss
+________________
+d Representation Network
+
+-------------------------------------------------------------------
+for token batch_size: b_to
+
+d Multi token beam search Transformer CE Loss
+__________________________________
+d transformer
+
+
++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++
+
+Inference:
+
+1. Input User Query
+2. The model's goal is to generate a thought, which contains a set of policies, which contains sequences of actions, and an action is a sequence of tokens. 
+3. The sequence of tokens is chosen using multi token prediction.
+4. The Thought size is defined based on the user prompt, if the user prompt is in depth, then given the text in the input query, a larger output tree of thought. 
+5. Perform the multi token beam search, depending on the action size, for each action will contain a multi token beam search (so an action will contain the state score of k beams for n tokens each time step, for a batch size of b_to).
+6. PPO agent selects the actions given a mcts over actions using their state scores
+7. Based on the tree of thought prompt tree, and given the sequence of actions selected for the policy, feed the chosen policy into the tree of thought, and get the Transformer Language Model to output token sequences based on the tree of thought prompts. There is an actor critic RL agent that selects the next child node in the tree of thought that is used, therefore learning to control how it responds to different user queries. The tree of thought should contain logic for decision making or solving problems in different ways. 
+8. Update world model given external evaluation datasets
+
++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++
+
+
+Web Search Agent:
+
+1. Given a user prompt, search N websites, using the input search query.
+2. Given meta charactistics of he webpages, use FFN to rank the web pages
+3. Utilise RAG to retrieve and summarize the content from the k highest ranking web pages given the user search query.
+4. Extract and formulate the retrieved information into a custom dataset.
+5. Feed the LLM and World Model the custom search dataset.
+
+
 ## Citation
 
 If you use LightBulb in your research, please cite the author:
@@ -158,6 +313,7 @@ If you use LightBulb in your research, please cite the author:
 ```
 
 
+
 ## License
 
 This project is licensed under the Apache 2.0 License.