Abu Dhabi Digital Authority (ADDA)

Lead Technical Specialist

VuePay

Technical Director cum CTO

Xische & Co

Engineering Technical Lead

Pocket App

Leading development team based in UK and India. My role is to directly manage projects worth of £250k+ (Tier 1). My current focus is scaling Pocket App's business to increase agility, bring improved product to market, and manage a well-diversified technical team. I manage technical quality and staffing for the firm in addition to spotting and seeding business development opportunities to accelerate company growth.

Head of Technology

Managing 40+ developers along with guidance to developers on various projects which is diversified on various platforms like Android, iOS, Hybrid, PHP, AWS etc. Along with this my job role was to handle clients, deliverables for various projects, project quotations and training the existing team on ios, android and html5 apps.

Technical Manager

Hurix Systems

Managing a Team of 10 junior developers assigning them work, managing deliverables and also developing on Android and HTML5 during this tenure.

Android,HTML5 Developer

Quinnox Consultancy services

I had worked on multiple projects for clients Coca- Cola, HNI Corp, BookTrack. I was developing natively on Platforms Android, iOS and was also developing on technology like Sencha touch, PhoneGap.

Android ,iOS and blackberry Developer

With over 14 years of development expertise and 10 years of management experience, I specialize in creating modern, scalable, and resilient products for various domains, such as web, mobile, backend, and blockchain. As a Lead Technical Specialist at Abu Dhabi Digital Authority (ADDA), I oversee the technical architecture and delivery of digital solutions that enhance the lives of citizens and residents in Abu Dhabi.

I am passionate about developing high-performing teams that foster a culture of innovation, collaboration, and learning. I empower my engineers with the necessary business context, a psychologically safe environment, and goals that align their work with the success of the organization and their professional growth. I also leverage my skills and credentials as an AWS Champion, a Certified Scrum Master, and a React & React Native Nanodegree holder to design and implement cutting-edge technologies and best practices.

So RAG systems are slow. Like, really slow.

You hear this always....

And here is the kicker most of that slowness? Completely unnecessary.

I just read through Meta's REFRAG paper, and honestly, it's one of those "why didn't we think of this sooner" moments. You know the ones.

Think about how RAG works right now. You retrieve 80 passages. The LLM dutifully processes every. Single. Token. Even though, and this is the crucial part maybe only 3 of those passages actually matter for your query.

The rest? Noise. Filler. Computational waste.

Here's what people will relate to

When you concatenate a bunch of retrieved passages, they don't really talk to each other. They are diverse (by design, thanks to deduplication). 

The attention patterns look block diagonal, meaning tokens within a passage attend to each other. Still, there is basically zero cross-attention between different passages.

We are doing quadratic-complexity attention calculations on stuff that doesn't need it.

REFRAG's approach is pretty straightforward (the best ideas often are) compress chunks of text into single embeddings. Use a lightweight encoder—like RoBERTa, to create these compressed representations. Feed those to the decoder instead of thousands of individual tokens.

The results are... wild

  - 30.85× faster time-to-first-token
  - No accuracy loss
  - Actually extends context window by 16×
 
Let me repeat that: 30× faster with no performance hit.

They even added an RL policy that learns which chunks are essential enough to expand back to full tokens. So you get selective decompression where it matters. Smart.

Why this matters for practitioners

If you are building RAG systems (and who is not these days?), you are probably hitting latency walls, especially at scale. Users don't wait around for slow responses; it doesn't matter how accurate they are.

The beautiful part? REFRAG does not require changing your LLM architecture. No new decoder parameters. It is a plug-in optimisation.

The broader lesson here

Sometimes optimisation is not about making the algorithm faster it is about recognising what work you don't need to do in the first place.

RAG has a unique structure. 
Block-diagonal attention. 
Sparse relevance. 

We have been treating it like generic long-context inference, but it's not. It needs specialised treatment.

Has anyone else experimented with chunk-level compression for RAG? Or hit similar latency bottlenecks?

Code should be dropping on GitHub soon apparently

#MachineLearning #RAG #LLMs #AIEngineering #NLP
---
P.S. - The paper also shows this works for multi-turn conversations and document summarisation, not just RAG. So if you are dealing with long contexts anywhere... yeah, pay attention to this one.

Sangam Pandey

 for comparison, here's the REFRAG approach, showcasing the compression and selective expansion for a much faster process

No alternative text description for this image

Taplio

Sangam Pandey's Linkedin Analytics

Sangam Pandey

Performance ScoreSangam Pandey's LinkedIn Performance

What is Sangam talking about?

Who is engaging with Sangam

Sangam Pandey's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?

Taplio

.css-6ytdur{background-image:linear-gradient(284deg, rgba(39, 43, 72, 1) 0%, rgba(33, 47, 183, 1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Sangam Pandey's Linkedin Analytics

Sangam Pandey

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Sangam Pandey's LinkedIn Performance

What is Sangam talking about?

Who is engaging with Sangam

Sangam Pandey's Best Posts .css-1emoo84{font-size:var(--chakra-fontSizes-md);font-weight:400;color:var(--chakra-colors-gray-600);}(last 30 days)

Ready to grow your LinkedIn presence?

Sangam Pandey's Linkedin Analytics

Performance ScoreSangam Pandey's LinkedIn Performance

Sangam Pandey's Best Posts (last 30 days)