feat: In-memory rag by ilyabo · Pull Request #272 · sqlrooms/sqlrooms

ilyabo · 2025-12-20T16:52:33Z

No description provided.

ilyabo · 2026-01-15T10:09:31Z

greptile-apps · 2026-01-15T10:13:14Z

Greptile Summary

Adds in-memory RAG capabilities to @sqlrooms/ai-rag with browser-side document preparation including PDF parsing, markdown-aware chunking, and embedding generation

Key Changes:

New prepare module with utilities for chunking (markdown-aware and size-based), database schema initialization, metadata tracking, and PDF text extraction
RagSlice now supports in-memory DuckDB databases for user-uploaded documents alongside existing file-based databases
Example app updated with document upload UI and RAG search dialog
Added pdfjs-dist dependency for client-side PDF processing

Critical Issues:

SQL injection vulnerabilities in multiple locations where databaseName, embedding arrays, and databaseFilePathOrUrl are interpolated directly into SQL queries without proper sanitization or parameterization
CDN-based worker loading for PDF.js creates external dependency risk

Confidence Score: 1/5

This PR has critical security vulnerabilities that must be addressed before merging
Multiple SQL injection vulnerabilities exist in database operations where user-controlled or configuration parameters (databaseName, embedding arrays, file paths) are directly interpolated into SQL queries without sanitization. While the functionality is well-designed, these security issues pose significant risk.
Pay close attention to packages/ai-rag/src/prepare/database.ts, packages/ai-rag/src/RagSlice.ts, and packages/ai-rag/src/prepare/metadata.ts - all contain SQL injection vulnerabilities that must be fixed

Important Files Changed

Filename	Overview
packages/ai-rag/src/prepare/database.ts	Added database operations with SQL injection vulnerabilities in `databaseName` and `embedding` parameters
packages/ai-rag/src/RagSlice.ts	Major refactoring with in-memory RAG support, but contains critical SQL injection vulnerabilities in database operations
packages/ai-rag/src/prepare/metadata.ts	Metadata creation and storage utilities, contains SQL injection vulnerability in `databaseName` parameter
packages/ai-rag/src/prepare/pdf.ts	PDF text extraction using pdfjs-dist with CDN worker source

Sequence Diagram

sequenceDiagram
    participant User
    participant UI as UI Component
    participant Store as RagSlice Store
    participant Prepare as Prepare Module
    participant DB as DuckDB (In-Memory)
    participant Embedding as Embedding Provider

    Note over Store,DB: Initialization Flow
    User->>UI: Upload PDF/Markdown/Text
    UI->>Store: rag.initialize()
    Store->>DB: ATTACH DATABASE ':memory:' AS user_docs
    Store->>Prepare: initializeRagSchema(connector, 'user_docs', 1536)
    Prepare->>DB: CREATE TABLE documents
    Prepare->>DB: CREATE TABLE source_documents
    Prepare->>DB: CREATE TABLE embedding_metadata
    DB-->>Store: Schema initialized

    Note over Store,Embedding: Document Processing Flow
    UI->>Store: rag.addPdfDocument(file) or rag.addDocument(doc)
    Store->>Prepare: extractTextFromPDF(file) [if PDF]
    Prepare-->>Store: extracted text
    Store->>Prepare: chunkMarkdown(text) or chunkBySize(text)
    Prepare-->>Store: chunks[]
    Store->>Prepare: validateAndSplitChunks(chunks)
    Prepare-->>Store: validated chunks[]
    
    loop For each chunk
        Store->>Embedding: embeddingProvider(chunk.text)
        Embedding-->>Store: embedding vector (number[])
        Store->>Prepare: insertDocument(connector, dbName, {nodeId, text, embedding, metadata})
        Prepare->>DB: INSERT INTO documents (node_id, text, embedding, metadata_)
        DB-->>Store: chunk stored
    end
    
    Store->>Prepare: insertSourceDocument(connector, dbName, {docId, text, fileName})
    Prepare->>DB: INSERT INTO source_documents
    Store->>Prepare: updateMetadataStats(connector, dbName, chunks, [document])
    Prepare->>DB: UPDATE embedding_metadata
    Store-->>UI: nodeIds[] (success)

    Note over Store,Embedding: Query Flow
    User->>UI: Search query
    UI->>Store: rag.queryByText(query, {topK: 5})
    Store->>Embedding: embeddingProvider(query)
    Embedding-->>Store: query embedding vector
    Store->>DB: SELECT with array_cosine_similarity(embedding, query_embedding)
    DB-->>Store: top K results
    Store-->>UI: EmbeddingResult[] with scores
    UI-->>User: Display search results

greptile-apps

_{11 files reviewed, 7 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-01-15T10:13:09Z

packages/ai-rag/src/prepare/database.ts

+  embeddingDimensions: number,
+): Promise<void> {
+  await connector.query(`
+    CREATE TABLE IF NOT EXISTS ${databaseName}.documents (


logic: databaseName parameter is directly interpolated into SQL without sanitization, creating SQL injection risk

Suggested change

CREATE TABLE IF NOT EXISTS ${databaseName}.documents (

CREATE TABLE IF NOT EXISTS "${databaseName}".documents (

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/ai-rag/src/prepare/database.ts Line: 24:24 Comment: **logic:** `databaseName` parameter is directly interpolated into SQL without sanitization, creating SQL injection risk ```suggestion CREATE TABLE IF NOT EXISTS "${databaseName}".documents ( ``` How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2026-01-15T10:13:09Z

packages/ai-rag/src/prepare/database.ts

+  databaseName: string,
+  doc: InsertDocumentParams,
+): Promise<void> {
+  const embeddingLiteral = `[${doc.embedding.join(', ')}]`;


logic: embedding array values are joined directly without validation, allowing SQL injection if non-numeric values are provided

Suggested change

const embeddingLiteral = `[${doc.embedding.join(', ')}]`;

const embeddingLiteral = `[${doc.embedding.map(n => {

if (typeof n !== 'number' || !isFinite(n)) throw new Error('Invalid embedding value');

return n;

}).join(', ')}]`;

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/ai-rag/src/prepare/database.ts Line: 101:101 Comment: **logic:** `embedding` array values are joined directly without validation, allowing SQL injection if non-numeric values are provided ```suggestion const embeddingLiteral = `[${doc.embedding.map(n => { if (typeof n !== 'number' || !isFinite(n)) throw new Error('Invalid embedding value'); return n; }).join(', ')}]`; ``` How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2026-01-15T10:13:10Z

packages/ai-rag/src/RagSlice.ts

+              if (!databaseFilePathOrUrl) {
+                // Create in-memory attached database
+                await connector.query(
+                  `ATTACH DATABASE ':memory:' AS ${databaseName}`,


logic: databaseName is directly interpolated into SQL query without sanitization, creating SQL injection vulnerability

Suggested change

`ATTACH DATABASE ':memory:' AS ${databaseName}`,

`ATTACH DATABASE ':memory:' AS "${databaseName}"`,

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/ai-rag/src/RagSlice.ts Line: 262:262 Comment: **logic:** `databaseName` is directly interpolated into SQL query without sanitization, creating SQL injection vulnerability ```suggestion `ATTACH DATABASE ':memory:' AS "${databaseName}"`, ``` How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2026-01-15T10:13:11Z

packages/ai-rag/src/RagSlice.ts

+
              // ATTACH DATABASE 'path/to/file.duckdb' AS database_name (READ_ONLY)
              await connector.query(
                `ATTACH DATABASE '${databaseFilePathOrUrl}' AS ${databaseName} (READ_ONLY)`,


logic: databaseFilePathOrUrl and databaseName are directly interpolated without sanitization, creating SQL injection risk

Suggested change

`ATTACH DATABASE '${databaseFilePathOrUrl}' AS ${databaseName} (READ_ONLY)`,

`ATTACH DATABASE '${databaseFilePathOrUrl.replace(/'/g, "''")}' AS "${databaseName}" (READ_ONLY)`,

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/ai-rag/src/RagSlice.ts Line: 280:280 Comment: **logic:** `databaseFilePathOrUrl` and `databaseName` are directly interpolated without sanitization, creating SQL injection risk ```suggestion `ATTACH DATABASE '${databaseFilePathOrUrl.replace(/'/g, "''")}' AS "${databaseName}" (READ_ONLY)`, ``` How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2026-01-15T10:13:12Z

packages/ai-rag/src/RagSlice.ts

@@ -302,214 +368,258 @@ export function createRagSlice({
          const embeddingDim = queryEmbedding.length;
          const embeddingLiteral = `[${queryEmbedding.join(', ')}]`;


logic: queryEmbedding array is joined without validation, allowing SQL injection if non-numeric values are provided

Suggested change

const embeddingLiteral = `[${queryEmbedding.join(', ')}]`;

const embeddingLiteral = `[${queryEmbedding.map(n => {

if (typeof n !== 'number' || !isFinite(n)) throw new Error('Invalid embedding value');

return n;

}).join(', ')}]`;

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/ai-rag/src/RagSlice.ts Line: 369:369 Comment: **logic:** `queryEmbedding` array is joined without validation, allowing SQL injection if non-numeric values are provided ```suggestion const embeddingLiteral = `[${queryEmbedding.map(n => { if (typeof n !== 'number' || !isFinite(n)) throw new Error('Invalid embedding value'); return n; }).join(', ')}]`; ``` How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2026-01-15T10:13:13Z

packages/ai-rag/src/prepare/pdf.ts

+import * as pdfjsLib from 'pdfjs-dist';
+
+// Set worker source - uses the same version as the library
+pdfjsLib.GlobalWorkerOptions.workerSrc = `//cdnjs.cloudflare.com/ajax/libs/pdf.js/${pdfjsLib.version}/pdf.worker.min.mjs`;


style: Loading worker from external CDN creates security and reliability risks (CDN unavailability, tampering)

Consider bundling the worker locally or using a versioned URL with integrity checks

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/ai-rag/src/prepare/pdf.ts Line: 9:9 Comment: **style:** Loading worker from external CDN creates security and reliability risks (CDN unavailability, tampering) Consider bundling the worker locally or using a versioned URL with integrity checks How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2026-01-15T10:13:14Z

packages/ai-rag/src/prepare/metadata.ts

+
+  for (const [key, value] of Object.entries(flatMetadata)) {
+    await connector.query(`
+      INSERT INTO ${databaseName}.embedding_metadata (key, value)


logic: databaseName is directly interpolated without sanitization, creating SQL injection vulnerability

Suggested change

INSERT INTO ${databaseName}.embedding_metadata (key, value)

INSERT INTO "${databaseName}".embedding_metadata (key, value)

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/ai-rag/src/prepare/metadata.ts Line: 168:168 Comment: **logic:** `databaseName` is directly interpolated without sanitization, creating SQL injection vulnerability ```suggestion INSERT INTO "${databaseName}".embedding_metadata (key, value) ``` How can I resolve this? If you propose a fix, please make it concise.

stprnvsh added 4 commits December 2, 2025 10:41

add prepare module to handle document chunking

01a0cef

add inmemory database to prepare module in rag

c458f92

prepare pdf / chunk / embeddings genearation

e6dcdad

final steps of adding it to Ragstore and Ragslice

7094acc

ilyabo requested a review from lixun910 December 20, 2025 16:53

ilyabo mentioned this pull request Dec 20, 2025

feat: RAG Python embedding preparation and querying modules #167

Merged

greptile-apps bot reviewed Jan 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: In-memory rag#272

feat: In-memory rag#272
ilyabo wants to merge 4 commits intosqlrooms:mainfrom
stprnvsh:inmemory-rag

ilyabo commented Dec 20, 2025

Uh oh!

ilyabo commented Jan 15, 2026

Uh oh!

greptile-apps bot commented Jan 15, 2026

Uh oh!

greptile-apps bot left a comment

Uh oh!

greptile-apps bot Jan 15, 2026

Uh oh!

greptile-apps bot Jan 15, 2026

Uh oh!

greptile-apps bot Jan 15, 2026

Uh oh!

greptile-apps bot Jan 15, 2026

Uh oh!

greptile-apps bot Jan 15, 2026

Uh oh!

greptile-apps bot Jan 15, 2026

Uh oh!

greptile-apps bot Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	CREATE TABLE IF NOT EXISTS ${databaseName}.documents (
	CREATE TABLE IF NOT EXISTS "${databaseName}".documents (

- const embeddingLiteral = `[${doc.embedding.join(', ')}]`;
+ const embeddingLiteral = `[${doc.embedding.map(n => {
+ if (typeof n !== 'number' || !isFinite(n)) throw new Error('Invalid embedding value');
+ return n;
+ }).join(', ')}]`;

	`ATTACH DATABASE ':memory:' AS ${databaseName}`,
	`ATTACH DATABASE ':memory:' AS "${databaseName}"`,

	`ATTACH DATABASE '${databaseFilePathOrUrl}' AS ${databaseName} (READ_ONLY)`,
	`ATTACH DATABASE '${databaseFilePathOrUrl.replace(/'/g, "''")}' AS "${databaseName}" (READ_ONLY)`,

		@@ -302,214 +368,258 @@ export function createRagSlice({
		const embeddingDim = queryEmbedding.length;
		const embeddingLiteral = `[${queryEmbedding.join(', ')}]`;

	INSERT INTO ${databaseName}.embedding_metadata (key, value)
	INSERT INTO "${databaseName}".embedding_metadata (key, value)

Conversation

ilyabo commented Dec 20, 2025

Uh oh!

ilyabo commented Jan 15, 2026

Uh oh!

greptile-apps bot commented Jan 15, 2026

Greptile Summary

Confidence Score: 1/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants