Semantic Drift Detection: Using Vector Embeddings for LLM Validation

Vector embeddings transform text into high-dimensional vectors that capture semantic meaning. This enables precise drift detection by measuring how far LLM outputs deviate from intended meaning.

Understanding Vector Embeddings

Embeddings map text to points in a high-dimensional space where:

• Similar meanings are close together
• Different meanings are far apart
• Semantic relationships are preserved

Example: "dog" and "puppy" are close; "dog" and "airplane" are far.

Cosine Similarity

The most common similarity measure:

function cosineSimilarity(vecA: number[], vecB: number[]): number {
  const dotProduct = vecA.reduce((sum, a, i) => sum + a * vecB[i], 0)
  const magnitudeA = Math.sqrt(vecA.reduce((sum, a) => sum + a * a, 0))
  const magnitudeB = Math.sqrt(vecB.reduce((sum, b) => sum + b * b, 0))
  
  return dotProduct / (magnitudeA * magnitudeB)
  // Returns value between -1 and 1
  // 1 = identical, 0 = orthogonal, -1 = opposite
}

Angular Distance

More intuitive than cosine similarity:

function angularDistance(vecA: number[], vecB: number[]): number {
  const similarity = cosineSimilarity(vecA, vecB)
  const angle = Math.acos(Math.max(-1, Math.min(1, similarity)))
  return angle // Returns radians (0 to π)
}

Interpretation:

• 0 radians (0°) = Identical meaning
• π/6 radians (30°) = Very similar
• π/3 radians (60°) = Related but different
• π/2 radians (90°) = Orthogonal (unrelated)
• π radians (180°) = Opposite meaning

Drift Detection Workflow

Step 1: Generate Embeddings

import { OpenAIEmbeddingProvider } from '@verdic/sdk'

const embedder = new OpenAIEmbeddingProvider(apiKey)

// Embed the global intent (once, cache it)
const intentEmbedding = await embedder.embed(
  "Software development and code generation assistance"
)

// Embed each output (or batch for efficiency)
const outputEmbedding = await embedder.embed(llmOutput)

Step 2: Calculate Distance

function calculateDrift(
  intentEmbedding: number[],
  outputEmbedding: number[]
): number {
  // Calculate angular distance
  const angle = angularDistance(intentEmbedding, outputEmbedding)
  
  // Convert to drift score (0-1, higher = more drift)
  const driftScore = angle / Math.PI
  
  return driftScore
}

Step 3: Apply Threshold

const driftScore = calculateDrift(intentEmbedding, outputEmbedding)
const threshold = 0.76 // Configurable per use case

if (driftScore > threshold) {
  // Drift detected - block or warn
  return { decision: "BLOCK", driftScore, threshold }
} else {
  // Within acceptable range
  return { decision: "ALLOW", driftScore, threshold }
}

Threshold Selection

Choosing the right threshold is critical:

Too Strict (Low Threshold)

• Blocks legitimate variations
• High false positive rate
• Poor user experience

Too Loose (High Threshold)

• Allows drift and violations
• High false negative rate
• Safety/compliance issues

Recommended Thresholds

| Use Case | Threshold | Rationale | |----------|-----------|-----------| | Customer Support | 0.70-0.75 | Balance flexibility with consistency | | Code Generation | 0.75-0.80 | Strict to prevent off-topic code | | Content Moderation | 0.65-0.70 | Strict to catch violations | | Creative Writing | 0.80-0.85 | Allow more creative variation |

Advanced Techniques

Rotation-Based Detection

Measure not just distance, but rotation:

// Rotation angle indicates direction of drift
const rotationAngle = calculateRotation(intentEmbedding, outputEmbedding)

// High rotation = significant semantic shift
if (rotationAngle > Math.PI / 4) { // 45 degrees
  // Major drift detected
}

Multi-Intent Validation

Compare against multiple reference intents:

const intents = [
  "customer support",
  "technical documentation",
  "product information"
]

const distances = await Promise.all(
  intents.map(intent => 
    calculateDrift(
      await embedder.embed(intent),
      outputEmbedding
    )
  )
)

// Output must be close to at least one intent
const minDistance = Math.min(...distances)
if (minDistance > threshold) {
  // Doesn't match any intent
  return { decision: "BLOCK" }
}

Batch Processing

Process multiple outputs efficiently:

// Batch embedding generation
const outputs = ["output1", "output2", "output3"]
const embeddings = await embedder.embedBatch(outputs)

// Compare all against intent
const driftScores = embeddings.map(emb => 
  calculateDrift(intentEmbedding, emb)
)

Performance Optimization

Caching Embeddings

Cache intent embeddings (they don't change):

const embeddingCache = new Map()

async function getCachedEmbedding(text: string): Promise<number[]> {
  if (embeddingCache.has(text)) {
    return embeddingCache.get(text)
  }
  
  const embedding = await embedder.embed(text)
  embeddingCache.set(text, embedding)
  
  return embedding
}

Approximate Similarity

For high-throughput systems, use approximate methods:

// Use dimensionality reduction for faster comparison
import { PCA } from 'ml-pca'

const pca = new PCA(embeddings)
const reduced = pca.predict(embedding) // Reduce from 1536 to 128 dims

// Faster comparison on reduced dimensions

Limitations and Considerations

Embedding Quality

Not all embeddings are equal:

• OpenAI text-embedding-3-small: 1536 dims, high quality
• Custom embeddings: May need fine-tuning
• Local embeddings: Trade quality for privacy

Language Support

Most embeddings work best for English:

• Non-English text may have lower accuracy
• Consider language-specific models
• Multilingual embeddings available but less precise

Context Length

Embeddings have context limits:

• OpenAI: 8192 tokens
• Longer texts need chunking
• Consider average pooling for long texts

Real-World Example

import { verdic } from '@verdic/sdk'

async function validateOutput(output: string, intent: string) {
  const validation = await verdic.validate({
    projectId: "my-project",
    output: output,
    config: {
      globalIntent: intent,
      threshold: 0.76,
      enableV5: true, // Enable semantic drift detection
      enableV8: true  // Enable rotation-based detection
    }
  })
  
  console.log(`Drift Score: ${validation.drift}`)
  console.log(`Decision: ${validation.decision}`)
  console.log(`Rotation Angle: ${validation.rotationMetrics?.angle}`)
  
  return validation
}

// Usage
const result = await validateOutput(
  "Here's how to implement authentication in React...",
  "Software development and code generation assistance"
)

// Result: { decision: "ALLOW", drift: 0.15, ... }

Conclusion

Vector embeddings provide a powerful foundation for semantic drift detection. By measuring angular distance between intent and output embeddings, you can accurately detect when LLM outputs deviate from expected meaning.

Combined with multi-dimensional analysis and proper threshold tuning, this approach enables production-ready LLM validation that balances accuracy with performance.