Lin Tan's picture

14 8

Lin Tan

lin-tan

·

https://www.cs.purdue.edu/homes/lintan/

AI & ML interests

AI-Software Synergy. LLM4Code (binary and source code). Mary J. Elmore New Frontiers Professor Purdue University

Recent Activity

liked a dataset 2 days ago

lt-asset/REPOCOD_Lite

Reacted to their post with 🔥 4 days ago

Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. - Leaderboard https://lt-asset.github.io/REPOCOD/ - Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD @jiang719 @shanchao @Yiran-Hu1007 Compared to #SWEBench, RepoCod tasks are - General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues. - With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8). Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with - Whole function generation - Repository-level context - Validation with test cases, and - Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00) Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with: - 67 repository-level, 67 file-level, and 66 self-contains tasks - Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens) - GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. - Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD_Lite #LLM4code #LLM #CodeGeneration #Security

Reacted to their post with 🤗 4 days ago

Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. - Leaderboard https://lt-asset.github.io/REPOCOD/ - Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD @jiang719 @shanchao @Yiran-Hu1007 Compared to #SWEBench, RepoCod tasks are - General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues. - With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8). Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with - Whole function generation - Repository-level context - Validation with test cases, and - Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00) Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with: - 67 repository-level, 67 file-level, and 66 self-contains tasks - Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens) - GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. - Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD_Lite #LLM4code #LLM #CodeGeneration #Security

View all activity

Organizations

lin-tan's activity

upvoted 2 papers about 1 month ago

SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models

Paper • 2409.19471 • Published Sep 28 • 2

WAFFLE: Multi-Modal Model for Automated Front-End Development

Paper • 2410.18362 • Published Oct 24 • 11

upvoted 2 papers about 2 months ago

ENCORE: Ensemble Learning using Convolution Neural Machine Translation for Automatic Program Repair

Paper • 1906.08691 • Published Jun 20, 2019 • 1

LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement

Paper • 2409.14201 • Published Sep 21 • 2

upvoted 10 papers 2 months ago

CaBaGe: Data-Free Model Extraction using ClAss BAlanced Generator Ensemble

Paper • 2409.10643 • Published Sep 16 • 1

Symbol Preference Aware Generative Models for Recovering Variable Names from Stripped Binary

Paper • 2306.02546 • Published Jun 5, 2023 • 1

KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair

Paper • 2302.01857 • Published Feb 3, 2023 • 1

DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functions

Paper • 2109.01002 • Published Sep 2, 2021 • 1

CURE: Code-Aware Neural Machine Translation for Automatic Program Repair

Paper • 2103.00073 • Published Feb 26, 2021 • 1

Automatically Extracting Web API Specifications from HTML Documentation

Paper • 1801.08928 • Published Jan 26, 2018 • 1

Impact of Code Language Models on Automated Program Repair

Paper • 2302.05020 • Published Feb 10, 2023 • 1

How Effective Are Neural Networks for Fixing Security Vulnerabilities

Paper • 2305.18607 • Published May 29, 2023 • 2

Impact of Large Language Models on Generating Software Specifications

Paper • 2306.03324 • Published Jun 6, 2023 • 2

Nova^+: Generative Language Models for Binaries

Paper • 2311.13721 • Published Nov 22, 2023 • 2