🤔
Now, tell me which result is more reasonable according to you?
What Semantic Sara did was the foundation of Semantic Routing: **Analysis.**
### Definition:
Semantic routing is an intelligent routing technique that makes decisions based on the **meaning and context** of queries rather than just keyword matching. Unlike logical routing, which uses predefined rules and patterns, semantic routing understands the **intent, context, and semantic relationships** within the text.
## Deep Dive:
### How does it work?
Let’s understand this by the question: “***Help with web interactivity***\*”\*
1. **Query Preprocessing:**
1. **Pre-processing:**
* Text normalization (lowercase, remove special characters)
* Tokenization: \["help", "with", "web", "interactivity"\]
* **Stop word removal** (remove common words like `i`, `need`, `help`, `with`):
Result: `["web", "interactivity"]`
Stop words do not contribute much to semantic search
2. **Embedding Generation:**
* Uses pre-trained language models (BERT, Sentence-BERT, OpenAI embeddings)
* Converts text to a high-dimensional vector: `[0.2, 0.8, -0.1, 0.4, ...]`
* Captures semantic meaning, not just keywords
3. **Context Analysis:**
* **Intent Detection:** "Help-seeking" + "Learning-oriented"
* **Domain Analysis:** Web development
* **Complexity Assessment:** Beginner level (simple language, broad request)
2. **Knowledge Base (KB) Representation:**
Each knowledge base is represented as a collection of semantic vectors that capture the meaning and context of its content. It serves as the central repository against which incoming queries are semantically evaluated. Generally, it is set up at the first of the system. Look below for more clarification:
```markdown
# HTML/CSS KB Vector: [0.4, 0.8, 0.3, ...]
- High values for: "visual", "interaction", "beginner", "elements"
- Represents: Basic web interactivity, styling, simple events
# JavaScript KB Vector: [0.1, 0.9, 0.2, ...]
- High values for: "programming", "functions", "advanced", "logic"
- Represents: Programming concepts, complex interactions
# React KB Vector: [0.3, 0.7, 0.1, ...]
- High values for: "components", "framework", "interactive", "state"
- Represents: Framework-specific interactivity
```
3. **Similarity Calculation:**
* **Cosine Similarity Calculation:**
$$\text{similarity} = \frac{\vec{A} \cdot \vec{B}}{\|\vec{A}\| \times \|\vec{B}\|}$$
* **Results:**
* HTML/CSS KB: **0.87** (highest - captures "basic web interactivity")
* React KB: **0.62** (good match for "interactivity" but more advanced)
* JavaScript KB: **0.45** (relevant but too programming-focused)
* Node.js KB: **0.33** (server-side, not directly interactive)
(Dummy values for simulation.)
4. **Intelligent Decision Making:**
* **Ranking System:**
1. Sort by Similarity Scores
2. Apply Confidence Thresholds (minimum 0.5)
3. Check Score Gaps (clear winner vs ambiguous)
4. Validate with Context (beginner vs advanced)
* **Final Decision:** Route to **HTML/CSS KB** because:
* Highest semantic similarity (0.87)
* Matches beginner intent
* Covers basic web interactivity concepts
* Appropriate starting point for the user's journey
So, the final decision would be **“*For the query ‘Help with web interactivity’, the semantic routing system would route to the HTML/CSS Knowledge Base because it has the highest semantic similarity (0.87) and best matches the beginner-level intent for learning basic web interactivity concepts.*“**
### When to Semantic Routing?
When the query is kind of abstract and needs some validation, then sematic query excels. Like these:
* **Natural Language Queries:** “*How do I make my website respond to user clicks?*“
* **Ambiguous or Paraphrased Queries:** *"Fix broken authentication"* vs *"Login not working"* vs *"User verification issues"*
* **Cross Domain Queries:** *"Best practices for securing user data in web apps"*
* **Beginner-Friendly Routing:** *"Help me understand how websites work"*
* **Intent Heavy Queries:** *"I'm struggling with responsive design on mobile."*
* **Synonym and variation handling:** *"API endpoints"* vs *"REST services"* vs *"web services"*
### When not to?
When the query consists of highly technical or specific technologies, or the main concept is already given in the query, using Semantic Query does not help much.
## Let’s do some code:
1. **Preprocessing:**
```python
# It is part of another function.
# For simplicity, I have only used tokenization
query_vector = self.model.encode([query.strip()])[0] # Tokenizing
similarities = {}
query_norm = np.linalg.norm(query_vector)
if query_norm == 0:
return {
"error": "Invalid query vector",
"routed_to": "general",
"confidence": 0.0
}
```
2. **Knowledge Base setup:**
```python
# Setting up the knowledge base when initializing the respective class.
# The knowlwdge base should be big files stored in the database.
# For simplicity, I have only used a dictionary.
def __init__(self):
self.model = SentenceTransformer('all-MiniLM-L6-v2')
self.knowledge_bases = {
"html_css": "HTML CSS styling layout beginner web design interactive elements",
"javascript": "JavaScript programming functions DOM events advanced coding",
"react": "React components hooks state JSX frontend framework",
"nodejs": "Node.js server backend API express database",
"authentication": "login security JWT tokens password user auth"
}
print("Semantic Router Starting")
self.kb_vectors = {}
for name, description in self.knowledge_bases.items():
vector = self.model.encode([description])[0]
self.kb_vectors[name] = vector
print("Semantic Router Ready")
```
3. **Similarity Calculation:**
```python
def route_query(self, query):
"""Route a query to the best knowledge base"""
# Some error handling
if not query or not query.strip():
return {
"error": "Query cannot be empty",
"routed_to": "general",
"confidence": 0.0
}
if not self.kb_vectors:
return {
"error": "No knowledge bases available",
"routed_to": "general",
"confidence": 0.0
}
try:
# This part is from query-preprocessing
# Skip it now
query_vector = self.model.encode([query.strip()])[0]
similarities = {}
query_norm = np.linalg.norm(query_vector)
if query_norm == 0:
return {
"error": "Invalid query vector",
"routed_to": "general",
"confidence": 0.0
}
# Similarity Search starts:
for kb_name, kb_vector in self.kb_vectors.items():
kb_norm = np.linalg.norm(kb_vector)
if kb_norm == 0:
similarities[kb_name] = 0.0
# Used Cosine Similarity formula
else:
similarity = np.dot(query_vector, kb_vector) / (query_norm * kb_norm)
similarities[kb_name] = similarity
if not similarities:
return {
"error": "No similarities calculated",
"routed_to": "general",
"confidence": 0.0
}
best_kb = max(similarities, key=similarities.get)
best_score = similarities[best_kb]
threshold = 0.3
if best_score < threshold:
best_kb = "general"
confidence = best_score
else:
confidence = best_score
return {
"query": query.strip(),
"routed_to": best_kb,
"confidence": confidence,
"all_scores": similarities
}
except Exception as e:
return {
"error": f"Routing failed: {str(e)}",
"routed_to": "general",
"confidence": 0.0
}
# Similarity search ends
```
4. **Decision Making (Demo Function):**
```python
def demo(router, query):
"""Test the Semantic Routing"""
result = router.route_query(query)
print(result)
print(f"\n📰 Query: {result['query']}")
print(f"➡️ Routed to: {result['routed_to']}")
print(f"🙌 Confidence: {result['confidence']}")
print("\nAll Scores: ")
for name, score in result['all_scores'].items():
print(f"{name}: score-> {score}")
```
5. **Demo Input-Output:**
```powershell
Semantic Router Starting
Semantic Router Ready
🧪Initiating Test:
> What is js?
{'query': 'What is js?', 'routed_to': 'javascript', 'confidence': np.float32(0.43704033), 'all_scores': {'html_css': np.float32(0.17255959), 'javascript': np.float32(0.43704033), 'react': np.float32(0.31851408), 'nodejs': np.float32(0.22914742), 'authentication': np.float32(0.18344694)}}
📰 Query: What is js?
➡️ Routed to: javascript
🙌 Confidence: 0.4370403289794922
All Scores:
html_css: score-> 0.17255958914756775
javascript: score-> 0.4370403289794922
react: score-> 0.3185140788555145
nodejs: score-> 0.22914741933345795
authentication: score-> 0.1834469437599182
> Javascript interactivity tutorial
{'query': 'Javascript interactivity tutorial', 'routed_to': 'javascript', 'confidence': np.float32(0.5200579), 'all_scores': {'html_css': np.float32(0.40155196), 'javascript': np.float32(0.5200579), 'react': np.float32(0.1519815), 'nodejs': np.float32(0.0759646), 'authentication': np.float32(0.040414095)}}
📰 Query: Javascript interactivity tutorial
➡️ Routed to: javascript
🙌 Confidence: 0.5200579166412354
All Scores:
html_css: score-> 0.4015519618988037
javascript: score-> 0.5200579166412354
react: score-> 0.15198150277137756
nodejs: score-> 0.07596459984779358
authentication: score-> 0.04041409492492676
> javascript authentication
{'query': 'javascript authentication', 'routed_to': 'authentication', 'confidence': np.float32(0.51601857), 'all_scores': {'html_css': np.float32(0.16553222), 'javascript': np.float32(0.4025679), 'react': np.float32(0.14441179), 'nodejs': np.float32(0.20198642), 'authentication': np.float32(0.51601857)}}
📰 Query: javascript authentication
➡️ Routed to: authentication
🙌 Confidence: 0.5160185694694519
All Scores:
html_css: score-> 0.1655322164297104
javascript: score-> 0.40256789326667786
react: score-> 0.14441178739070892
nodejs: score-> 0.2019864171743393
authentication: score-> 0.5160185694694519
> What is python?
{'query': 'What is python?', 'routed_to': 'general', 'confidence': np.float32(0.07313205), 'all_scores': {'html_css': np.float32(0.06834164), 'javascript': np.float32(0.07313205), 'react': np.float32(0.043729357), 'nodejs': np.float32(0.04890592), 'authentication': np.float32(-0.04147682)}}
📰 Query: What is python?
➡️ Routed to: general
🙌 Confidence: 0.07313205301761627
All Scores:
html_css: score-> 0.06834164261817932
javascript: score-> 0.07313205301761627
react: score-> 0.04372935742139816
nodejs: score-> 0.04890592023730278
authentication: score-> -0.04147681966423988
```
This is it. The “Semantic Routing”/