I build data pipelines, lakehouse solutions, and production-oriented applications using Azure, Databricks, PySpark, SQL, and Python.
My interests sit at the intersection of data engineering and software engineering — from distributed data processing and Spark optimization to building maintainable Python applications around data platforms.
- Azure Data Factory
- Azure Data Lake Storage Gen2
- Azure Functions
- Azure Service Bus
- Azure Key Vault
- Azure Synapse
- Databricks
- PySpark
- Spark SQL
- SQL
- Delta Lake
- Unity Catalog
- ETL / ELT
- Batch & Streaming Pipelines
- Medallion Architecture
- Data Quality
- Spark Performance Optimization
- Python
- Pandas
- NumPy
- Flask
- Streamlit
- Object-Oriented Programming
- Software Design Patterns
- Git
- GitHub
- GitHub Actions
- Docker
- Configuration-driven applications
- Dependency Injection
- Repository / Service architecture
Private Project
A production-oriented Streamlit + Databricks application I am building to manage my learning, projects, interview preparation, certifications, and other career-development workflows.
The project is designed with software-engineering principles rather than as a simple Streamlit prototype.
Engineering highlights:
- Modular Python application architecture
- Repository / Service / Factory patterns
- Dependency Injection
- Databricks SQL backend
- Gold-layer data model
- Configuration-driven table management
- CRUD workflows
- Dirty-row detection to update only modified records
- Session-level caching
- Cache invalidation after writes
- Query optimization
- Reduced unnecessary Streamlit reruns
- Databricks App deployment
The repository is currently private because of the Databricks Apps deployment setup.
A collection of practical experiments and examples focused on understanding how Spark actually executes workloads and how seemingly small implementation choices can affect performance.
Topics include:
- Partitioning
repartition()vscoalesce()- Broadcast joins
- Data skew
- Caching and persistence
- Join optimization
- Unnecessary shuffles
- Query execution and performance trade-offs
End-to-end data engineering exercises using Apache Airflow for workflow orchestration, covering pipeline scheduling, task dependencies, ingestion, transformation, and data workflow design.
Projects and experiments covering modern Azure data engineering patterns using services such as:
- Azure Data Factory
- ADLS Gen2
- Databricks
- Azure Functions
- Service Bus
- Synapse
A collection of SQL projects and experiments focused on writing, understanding, and optimizing SQL for real-world data engineering scenarios.
Topics include:
- Advanced SQL
- Window functions
- CTEs
- Query execution order
- Joins
- Performance considerations
- Data transformation
- Advanced Databricks development
- PySpark performance engineering
- Lakehouse architecture
- Unity Catalog
- Spark Declarative Pipelines
- Data engineering system design
- Production-grade Python applications
- Building and deploying data applications with Databricks Apps
I create technical content around Data Engineering, SQL, PySpark, Databricks, and Azure, focusing on practical examples, performance optimization, and real-world engineering decisions.
On my channel, I cover topics such as:
- Advanced SQL and SQL internals
- PySpark performance optimization
- Databricks engineering
- Azure Data Engineering
- Productionizing data workloads
- Data Engineering architecture
- Practical engineering projects and walkthroughs
📺 YouTube: Shubham Shrimant
I'm particularly interested in building systems that combine:
Cloud + Data Platforms + Distributed Processing + Software Engineering
My long-term focus is on designing reliable and scalable data platforms while writing code that is maintainable, testable, and production-ready.
⭐️ Building, learning, and sharing Data Engineering solutions.