{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Minerva: AI Guardian for Scam Protection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook implements a multi-agent system for analyzing images (screenshots) to identify scam attempts, and provide personalized scam prevention. It uses [AutoGen](https://github.com/microsoft/autogen/) to orchestrate various specialized agents that work together.\n", "\n", "Benefits:\n", "- Automates the process of identifying suspicious scam patterns.\n", "- Prevents Financial Loss\n", "- Saves Time: Early scam detection reduces the number of claims filed by end-users." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Install Dependencies" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -q autogen-agentchat==0.4.0.dev11 autogen-ext[openai]==0.4.0.dev11 pillow pytesseract pyyaml " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Model Initialization" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "from dotenv import load_dotenv, find_dotenv\n", "from autogen_ext.models.openai import OpenAIChatCompletionClient\n", "\n", "load_dotenv(find_dotenv())\n", "\n", "model = OpenAIChatCompletionClient(\n", " model=\"gpt-4o\",\n", " api_key=os.getenv(\"OPENAI_API_KEY\")\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tools Creation" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "from autogen_core.tools import FunctionTool\n", "from tools import Tools\n", "\n", "tools = Tools()\n", "\n", "ocr_tool = FunctionTool(\n", " tools.ocr, description=\"Extracts text from an image\"\n", ")\n", "\n", "url_checker_tool = FunctionTool(\n", " tools.is_url_safe, description=\"Checks if a URL is safe\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Agents Creation" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "import yaml\n", "\n", "with open('config/agents.yaml', 'r') as file:\n", " config = yaml.safe_load(file)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from autogen_agentchat.agents import AssistantAgent\n", "from autogen_agentchat.teams import RoundRobinGroupChat\n", "\n", "ocr_agent = AssistantAgent(\n", " name=\"OCR_Specialist\",\n", " description=\"Extracts text from an image\",\n", " system_message=config['ocr_agent']['assignment'],\n", " model_client=model,\n", " #tools=[ocr_tool]\n", ")\n", "\n", "url_checker_agent = AssistantAgent(\n", " name=\"URL_Checker\",\n", " description=\"Checks if a URL is safe\",\n", " system_message=config['url_checker_agent']['assignment'],\n", " model_client=model,\n", " tools=[url_checker_tool]\n", ")\n", "\n", "content_agent = AssistantAgent(\n", " name=\"Content_Analyst\",\n", " description=\"Analyzes the text for scam patterns\",\n", " system_message=config['content_agent']['assignment'],\n", " model_client=model,\n", " #tools=[url_checker_tool]\n", ")\n", "\n", "decision_agent = AssistantAgent(\n", " name=\"Decision_Maker\",\n", " description=\"Synthesizes the analyses and make final determination\",\n", " system_message=config['decision_agent']['assignment'],\n", " model_client=model\n", ")\n", "\n", "summary_agent = AssistantAgent(\n", " name=\"Summary_Agent\",\n", " description=\"Generate a summary of the final determination\",\n", " system_message=config['summary_agent']['assignment'],\n", " model_client=model\n", ")\n", "\n", "language_translation_agent = AssistantAgent(\n", " name=\"Language_Translation_Agent\",\n", " description=\"Translate the summary to the user language, which is the language of the extracted text\",\n", " system_message=config['language_translation_agent']['assignment'],\n", " model_client=model\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Team Creation" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "team = RoundRobinGroupChat([ocr_agent, url_checker_agent, content_agent, decision_agent, summary_agent, language_translation_agent], max_turns=6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running the Team" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from autogen_agentchat.messages import MultiModalMessage\n", "from autogen_core import Image as AGImage\n", "from PIL import Image\n", "\n", "image_path = \"./samples/02.giftcard.message.scam.png\"\n", "# image_path = \"./samples/scam.spanish.png\"\n", "\n", "pil_image = Image.open(image_path)\n", "img = AGImage(pil_image)\n", "multi_modal_message = MultiModalMessage(content=[img], source=\"User\")\n", "img" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "#team.reset()\n", "stream = team.run_stream(task=multi_modal_message)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---------- User ----------\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "---------- OCR_Specialist ----------\n", "Congratulations! You've won a $1,000 Walmart gift card. Go to http://bit.ly/123456 tp claim now.\n", "[Prompt tokens: 330, Completion tokens: 26]\n", "---------- URL_Checker ----------\n", "[FunctionCall(id='call_fj1nCVYiksMkytwKw6jeThkj', arguments='{\"target_url\":\"http://bit.ly/123456\"}', name='is_url_safe')]\n", "[Prompt tokens: 384, Completion tokens: 22]\n", "---------- URL_Checker ----------\n", "[FunctionExecutionResult(content='(True, [])', call_id='call_fj1nCVYiksMkytwKw6jeThkj')]\n", "---------- URL_Checker ----------\n", "(True, [])\n", "---------- Content_Analyst ----------\n", "This message shows several signs of being a potential scam:\n", "\n", "1. **Unsolicited Message:** You received a notification out of the blue about winning a prize.\n", "\n", "2. **Shortened URL:** The use of a link shortener (bit.ly) could be hiding the actual destination, which is often used for malicious purposes.\n", "\n", "3. **Urgency:** The message prompts immediate action (\"claim now\"), pressuring you to visit the link without thinking.\n", "\n", "4. **Spelling Error:** There's a typo, \"tp\" instead of \"to,\" which is common in scam messages.\n", "\n", "5. **Large Prize Claim:** Offers of large sums or prizes often lure recipients into clicking or providing personal information.\n", "\n", "Always verify through official channels before acting on such messages. Do not click on the link or provide any personal information.\n", "[Prompt tokens: 391, Completion tokens: 163]\n", "---------- Decision_Maker ----------\n", "This message is likely a scam. Here’s why:\n", "\n", "1. **Unexpected Win:** Legitimate companies rarely give out significant prizes randomly.\n", "\n", "2. **Shortened Link:** The use of a URL shortener can conceal malicious sites.\n", "\n", "3. **Sense of Urgency:** It creates pressure to act quickly, a common tactic in scams.\n", "\n", "4. **Typo:** The presence of errors (\"tp\" instead of \"to\") is typical in fraudulent messages.\n", "\n", "5. **Generic Greeting:** Scams often use impersonal or generic messages.\n", "\n", "Always verify with the company directly if you receive such messages. Do not click the link or share personal information.\n", "[Prompt tokens: 521, Completion tokens: 129]\n", "---------- Summary_Agent ----------\n", "The text message is likely a scam. It claims you won a large prize unexpectedly, uses a shortened URL to conceal a potentially harmful site, and includes a typo (\"tp\" instead of \"to\"). These are common red flags. Additionally, it urges immediate action, a tactic often used by scammers. To stay safe, avoid clicking the link and don't share any personal information. Verify through official channels if you're unsure.\n", "[Prompt tokens: 695, Completion tokens: 83]\n", "---------- Language_Translation_Agent ----------\n", "The message claims you've won a $1,000 Walmart gift card, but it shows signs of being a scam. It uses a shortened URL, contains a typo, and urges quick action, which are common in fraudulent messages. Avoid clicking the link or sharing personal information. Always verify with the company directly if unsure.\n", "[Prompt tokens: 778, Completion tokens: 63]\n", "---------- Summary ----------\n", "Number of messages: 9\n", "Finish reason: Maximum number of turns 6 reached.\n", "Total prompt tokens: 3099\n", "Total completion tokens: 486\n", "Duration: 17.81 seconds\n" ] }, { "data": { "text/plain": [ "TaskResult(messages=[MultiModalMessage(source='User', models_usage=None, content=[], type='MultiModalMessage'), TextMessage(source='OCR_Specialist', models_usage=RequestUsage(prompt_tokens=330, completion_tokens=26), content=\"Congratulations! You've won a $1,000 Walmart gift card. Go to http://bit.ly/123456 tp claim now.\", type='TextMessage'), ToolCallMessage(source='URL_Checker', models_usage=RequestUsage(prompt_tokens=384, completion_tokens=22), content=[FunctionCall(id='call_fj1nCVYiksMkytwKw6jeThkj', arguments='{\"target_url\":\"http://bit.ly/123456\"}', name='is_url_safe')], type='ToolCallMessage'), ToolCallResultMessage(source='URL_Checker', models_usage=None, content=[FunctionExecutionResult(content='(True, [])', call_id='call_fj1nCVYiksMkytwKw6jeThkj')], type='ToolCallResultMessage'), TextMessage(source='URL_Checker', models_usage=None, content='(True, [])', type='TextMessage'), TextMessage(source='Content_Analyst', models_usage=RequestUsage(prompt_tokens=391, completion_tokens=163), content='This message shows several signs of being a potential scam:\\n\\n1. **Unsolicited Message:** You received a notification out of the blue about winning a prize.\\n\\n2. **Shortened URL:** The use of a link shortener (bit.ly) could be hiding the actual destination, which is often used for malicious purposes.\\n\\n3. **Urgency:** The message prompts immediate action (\"claim now\"), pressuring you to visit the link without thinking.\\n\\n4. **Spelling Error:** There\\'s a typo, \"tp\" instead of \"to,\" which is common in scam messages.\\n\\n5. **Large Prize Claim:** Offers of large sums or prizes often lure recipients into clicking or providing personal information.\\n\\nAlways verify through official channels before acting on such messages. Do not click on the link or provide any personal information.', type='TextMessage'), TextMessage(source='Decision_Maker', models_usage=RequestUsage(prompt_tokens=521, completion_tokens=129), content='This message is likely a scam. Here’s why:\\n\\n1. **Unexpected Win:** Legitimate companies rarely give out significant prizes randomly.\\n\\n2. **Shortened Link:** The use of a URL shortener can conceal malicious sites.\\n\\n3. **Sense of Urgency:** It creates pressure to act quickly, a common tactic in scams.\\n\\n4. **Typo:** The presence of errors (\"tp\" instead of \"to\") is typical in fraudulent messages.\\n\\n5. **Generic Greeting:** Scams often use impersonal or generic messages.\\n\\nAlways verify with the company directly if you receive such messages. Do not click the link or share personal information.', type='TextMessage'), TextMessage(source='Summary_Agent', models_usage=RequestUsage(prompt_tokens=695, completion_tokens=83), content='The text message is likely a scam. It claims you won a large prize unexpectedly, uses a shortened URL to conceal a potentially harmful site, and includes a typo (\"tp\" instead of \"to\"). These are common red flags. Additionally, it urges immediate action, a tactic often used by scammers. To stay safe, avoid clicking the link and don\\'t share any personal information. Verify through official channels if you\\'re unsure.', type='TextMessage'), TextMessage(source='Language_Translation_Agent', models_usage=RequestUsage(prompt_tokens=778, completion_tokens=63), content=\"The message claims you've won a $1,000 Walmart gift card, but it shows signs of being a scam. It uses a shortened URL, contains a typo, and urges quick action, which are common in fraudulent messages. Avoid clicking the link or sharing personal information. Always verify with the company directly if unsure.\", type='TextMessage')], stop_reason='Maximum number of turns 6 reached.')" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from autogen_agentchat.ui import Console\n", "await Console(stream)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "from pprint import pprint\n", "\n", "streams = []\n", "async for s in stream:\n", " streams.append(s)\n", "\n", "pprint(streams[-1].messages[-1].content)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.1" } }, "nbformat": 4, "nbformat_minor": 2 }