Amy Unruh, PhD

Catching up on improving LLMs by scaling inference-time compute, new LLM scaling laws for precision, and more! "Noteworthy AI Research Papers 2024, Part 2" is now formatted and published. It was certainly an eventful year for this fast-moving field: https://lnkd.in/gciNkD5U In this second part, I cover: 1. Some of the major open-weight LLM releases from last year, featuring more sophisticated pre- and post-training pipelines 2. Techniques for improving LLMs by scaling inference-time compute 3. The different multimodal LLM paradigms 4. Replicating OpenAI's reasoning capabilities in the o1 model 5. New LLM scaling laws for precision 6. Insights from learning with synthetic data I hope you had a great start to the year and find these insights useful: Happy reading!

694

27 Comments

Happy to announce AlignVLM 📏 – a novel approach to bridging vision and language latent spaces for multimodal understanding in Vision-Language Models (VLMs) 🌍📄🖼 🔗 Read the paper: https://lnkd.in/dw9v4i7P 🧐 What’s the challenge? Aligning visual features with language embeddings remains a major bottleneck in VLMs. Existing connectors such as Multi-layer perceptron (MLPs) often introduce noise that degrades performance. ❌ 🎯 Our Solution: ALIGN Connector We propose AlignVLM, a method that maps vision features into a weighted average of LLM text embeddings, ensuring they remain in a space that the LLM can effectively interpret. ✅ 🔬 How does it perform? We compared ALIGN against common connectors like MLPs, Perceiver Resampler, and Ovis trained under similar configurations. The results? ALIGN outperforms them all 🏆 on diverse document understanding tasks 📄. 📊 Meet the AlignVLM Model Family! We trained Llama 3.1 (1B, 3B, 8B) using our connector and benchmarked them against various models. The results: ✅ AlignVLM surpasses all Base VLMs trained under similar configurations. ✅ Our models also perform competitively against Instruct VLMs such as Qwen2-VL and InternVL-2.5 🚀. 🤔 What about robustness to noise? We injected Gaussian noise (μ=0, σ=3) into the vision encoder’s outputs before feeding them to the connector: ✅ ALIGN Connector: Minimal drop (↓1.67%) – proving its high robustness! ❌ MLP Connector: Severe degradation (↓25.54%) – struggling with noisy inputs. Code & model weights coming soon! Stay tuned! 🔥 Huge thanks to our amazing collaborators! 🙌 ServiceNow Research York University Mila - Quebec Artificial Intelligence Institute #AlignVLM #VisionLanguageModels #AI #MultimodalAI #MachineLearning #LLMs #DeepLearning #Research #DocumentUnderstanding

183

5 Comments

I just found the fastest serving engine for LLMs: (fully open-source & makes vLLM 7x faster) LMCache is an LLM serving engine designed to reduce time-to-first-token and increase throughput, especially under long-context scenarios. At a high level, LMCache is like a CDN of KV caches. It sits between the LLM serving engine, like vLLM, and the external data by storing the KV caches of reusable texts across various locations like GPU, CPU, and Local Disk. This way, it can boost vLLM with 7x faster access to 100x more KV caches. Find a link to the GitHub repo in the comments! ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

1,240

37 Comments