Travel Recommendation System - Evaluation Framework

📊 Overview

This evaluation framework assesses the quality and performance of the Travel Recommendation System using real production data from your Neo4j knowledge graph.

Evaluation Strategy

PRIMARY (80%): Real data from production Neo4j graph

• 352 actual cities
• 19,883 real attractions
• 13,519+ real relationships
• Actual user query patterns (if available)

SECONDARY (20%): Synthetic test cases for edge cases

• Hypothetical scenarios
• Stress testing
• Future features

---

📁 Folder Structure

evaluation/
├── kg_quality/                      # Knowledge Graph Quality Evaluation
│   ├── entity_popularity_REAL.py   # Entity popularity analysis (REAL data)
│   └── [future scripts]
│
├── results/                         # Evaluation Results
│   ├── figures/                    # Generated plots and visualizations
│   │   └── REAL_entity_popularity.png
│   └── data/                       # CSV exports and data files
│       ├── REAL_entity_popularity.csv
│       ├── REAL_high_priority_entities.csv
│       └── evaluation_targets.csv
│
└── README.md                        # This file

---

🚀 Quick Start

Prerequisites

pip install pandas matplotlib seaborn neo4j numpy

Configuration

The scripts automatically try to load credentials from:

1. Streamlit secrets (.streamlit/secrets.toml) - Recommended
2. Environment variables - Fallback

Option 1: Use Existing Streamlit Secrets (Recommended)

No additional configuration needed! The script will use your existing .streamlit/secrets.toml:

[neo4j]
uri = "neo4j+s://your-instance.databases.neo4j.io"
username = "neo4j"
password = "your-password"

Option 2: Set Environment Variables

# Windows PowerShell
$env:NEO4J_URI="neo4j+s://your-instance.databases.neo4j.io"
$env:NEO4J_USER="neo4j"
$env:NEO4J_PASSWORD="your-password"

# Linux/Mac
export NEO4J_URI="neo4j+s://your-instance.databases.neo4j.io"
export NEO4J_USER="neo4j"
export NEO4J_PASSWORD="your-password"

Run Entity Popularity Analysis

cd evaluation/kg_quality
python entity_popularity_REAL.py

---

📊 Script 1: Entity Popularity Analysis

File: kg_quality/entity_popularity_REAL.py

What It Does

1. Extracts ALL entities from your Neo4j graph (19,883+ attractions)
2. Simulates query patterns based on entity type, city importance, and discoverability
3. Generates visualizations:
   • Entity popularity scatter plot
   • Top cities by query volume
   • Entity type distribution
   • Top 20 most-queried entities
4. Identifies evaluation priorities using Pareto principle (80/20 rule)
5. Exports evaluation targets for manual quality assessment

Output Files

File	Description
results/figures/REAL_entity_popularity.png	4-panel visualization of entity popularity
results/data/REAL_entity_popularity.csv	All entities with simulated query counts
results/data/REAL_high_priority_entities.csv	Top 100 entities to evaluate first
results/data/evaluation_targets.csv	Manual evaluation checklist (Excel-ready)

Key Metrics

• Pareto Analysis: Top 20% of entities account for ~80% of queries
• City Priority: Major cities (Boston, Cambridge, Salem) get 3x boost
• Discoverability: Entities with websites get 1.5x boost
• Entity Type Weights:
  • Cities: 50 (highest)
  • Museums: 30
  • Restaurants: 25
  • Parks/Beaches: 20
  • Historic sites: 15
  • Cafes: 10
  • Bars/Pubs: 8
  • Fast Food: 5 (lowest)

---

📋 Manual Evaluation Workflow

Step 1: Run the Script

python kg_quality/entity_popularity_REAL.py

Step 2: Review Visualizations

Open results/figures/REAL_entity_popularity.png to understand:

• Which entities are most popular
• Geographic distribution of queries
• Entity type breakdown
• Top attractions to prioritize

Step 3: Manual Quality Check

Open results/data/evaluation_targets.csv in Excel/Google Sheets:

Column	What to Check
accuracy_check	Is the entity information correct? (Yes/No/Partial)
completeness_check	Is all important info present? (Complete/Missing/Partial)
notes	What's wrong or missing?

Example:

entity_name: Museum of Fine Arts
accuracy_check: Yes
completeness_check: Partial
notes: Missing opening hours for holidays

Step 4: Prioritize Fixes

Focus on:

1. Top 20 entities (highest impact)
2. Major cities (Boston, Cambridge, Salem)
3. Entities with websites (more discoverable)
4. Museums and restaurants (most queried types)

---

🔍 Understanding the Results

Pareto Principle (80/20 Rule)

The script calculates how many entities account for 80% of queries:

Top 20% entities = ~3,977 attractions
These account for ~80% of all queries

Implication: Focus evaluation efforts on these high-priority entities first.

Query Count Simulation

Since you may not have user query logs yet, the script simulates realistic patterns:

query_count = base_weight × random_factor × city_boost × website_boost

• Base weight: Entity type importance (City=50, Museum=30, etc.)
• Random factor: Log-normal distribution to simulate real-world variance
• City boost: 3x for major cities (Boston, Cambridge, Salem, Worcester, Springfield)
• Website boost: 1.5x for entities with websites

When You Have Real Logs

If you implement user query logging in Snowflake:

CREATE TABLE user_search_logs (
    log_id VARCHAR(255) PRIMARY KEY,
    user_id VARCHAR(255),
    entity_searched VARCHAR(255),
    search_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP()
);

The script will automatically use real data instead of simulations.

---

🎯 Evaluation Priorities

Based on the Pareto analysis, prioritize evaluation in this order:

Priority 1: Top 100 Entities (~80% of queries)

• Manual quality check using evaluation_targets.csv
• Verify accuracy of all properties
• Ensure completeness (hours, website, accessibility)

Priority 2: Major Cities

• Boston, Cambridge, Salem, Worcester, Springfield
• Verify all attractions in these cities
• Check for missing popular attractions

Priority 3: High-Value Entity Types

• Museums (cultural significance)
• Restaurants (food tourism)
• Historic sites (tourist interest)
• Parks/Beaches (outdoor activities)

Priority 4: Long Tail

• Remaining entities (20% of queries)
• Spot-check for major errors
• Focus on entities with websites (more visible)

---

📈 Future Evaluation Scripts

Planned Additions

1. RAG Quality Evaluation

   • Measure response accuracy
   • Test with real user queries
   • Compare against ground truth

2. Cypher Generation Accuracy

   • Natural language → Cypher conversion success rate
   • Syntax error rate
   • Semantic correctness

3. Graph Completeness

   • Missing relationships
   • Orphaned nodes
   • Data quality metrics

4. Performance Benchmarks

   • Query latency
   • LLM response time
   • Database connection overhead

---

🛠️ Troubleshooting

Connection Errors

Error: Failed to connect to Neo4j

Solution:

1. Check your .streamlit/secrets.toml credentials
2. Verify Neo4j instance is running
3. Test connection: python -c "from neo4j import GraphDatabase; driver = GraphDatabase.driver('uri', auth=('user', 'pass')); driver.verify_connectivity()"

Import Errors

Error: ModuleNotFoundError: No module named 'pandas'

Solution:

pip install pandas matplotlib seaborn neo4j numpy

Empty Results

Error: Extracted 0 REAL entities from YOUR graph

Solution:

1. Check your Cypher query matches your graph schema
2. Verify node labels (City, Restaurant, Museum, etc.)
3. Run test query in Neo4j Browser:

   MATCH (n) WHERE n:City OR n:Restaurant RETURN count(n)

---

📞 Support

For questions or issues:

1. Check the main project README.md
2. Review TECHNICAL_DOCUMENTATION.md
3. Contact the development team

---

📝 License

Same as main project license.

---

Last Updated: December 12, 2024
Version: 1.0.0
Status: ✅ Ready for Use