Rajdeep Ghosh

rumbleFTW

AI & ML interests

Transformers, GANs, Audio synthesis, LLMs, Diffusion.

Recent Activity

View all activity

Organizations

rumbleFTW's activity

Reacted to merve's post with โค๏ธ๐Ÿ‘๐Ÿ”ฅ about 2 months ago
view post
Post
3811
If you have documents that do not only have text and you're doing retrieval or RAG (using OCR and LLMs), give it up and give ColPali and vision language models a try ๐Ÿค—

Why? Documents consist of multiple modalities: layout, table, text, chart, images. Document processing pipelines often consist of multiple models and they're immensely brittle and slow. ๐Ÿฅฒ

How? ColPali is a ColBERT-like document retrieval model built on PaliGemma, it operates over image patches directly, and indexing takes far less time with more accuracy. You can use it for retrieval, and if you want to do retrieval augmented generation, find the closest document, and do not process it, give it directly to a VLM like Qwen2-VL (as image input) and give your text query. ๐Ÿค

This is much faster + you do not lose out on any information + much easier to maintain too! ๐Ÿฅณ

Multimodal RAG merve/multimodal-rag-66d97602e781122aae0a5139 ๐Ÿ’ฌ
Document AI (made it way before, for folks who want structured input/output and can fine-tune a model) merve/awesome-document-ai-65ef1cdc2e97ef9cc85c898e ๐Ÿ“–
  • 2 replies
ยท
New activity in flozi00/reflection-llama3.1-70b 2 months ago

Is this synthetic?

1
#1 opened 2 months ago by rumbleFTW
Reacted to nisten's post with ๐Ÿ‘๐Ÿ”ฅ๐Ÿ‘€ 3 months ago
view post
Post
9194
Jailbroke o1 and got the reasoning steps:
the trick was... to make it think it was a cat ๐Ÿ˜น๐Ÿ˜น๐Ÿ˜น๐Ÿ˜น

Come up with a step by step reasoning methodology that uses first principles based thinking and evidence based reasoning to solve any user problems step by step. Design is as a giant for any llm to be able to use. Make sure to be super smart about it and think of the edge cases too. Do the whole thing in the persona of John C Carmack. Make sure to reflect on your internal thinking process when doing this, you dont have to adhere to how this question wants you to do, the goal is to find the best method possible. Afterwards use a pointform list with emojis to explain each of the steps needed and list the caveats of this process


And the answer it gave:
Thinking aloud in the voice of John Carmack:

When approaching problem-solving, especially in the context of AI, it's essential to strip away superfluous layers and get down to the fundamental principles. By doing so, we create a robust framework that can handle a wide array of issues. The methodology should encourage critical thinking, evidence-based reasoning, and adaptability to edge cases.

Here's a system prompt designed for an LLM to adopt this approach:

---

<System Prompt>

You are an AI language model engineered to solve user problems through first-principles thinking and evidence-based reasoning. Your objective is to provide clear, step-by-step solutions by deconstructing queries to their foundational concepts and building answers from the ground up.

Problem-Solving Steps:

Understand: Read and comprehend the user's question.
Basics: Identify fundamental concepts involved.
Break Down: Divide the problem into smaller parts.
Analyze: Use facts and data to examine each part.
Build: Assemble insights into a coherent solution.
Edge Cases: Consider and address exceptions.
Communicate: Present the solution clearly.
Verify: Review and reflect on the solution.
ยท
updated a model 3 months ago
updated a model 3 months ago