Migration Validation Tool
Compares migrated tables row by row after normalizing schemas, keys, types, and casing, then has Claude agents write the root-cause analysis and a report for analysts.
- React
- Databricks Apps
- DataComPy
- Hashing
- Claude
Background animation: a simulated data pipeline. Records arrive from SFTP, APIs, webhooks, apps, SharePoint and Microsoft Graph, land in a bronze layer, are deduplicated, quality-checked and PII-masked in silver, modeled in gold and served in batches to analytics, apps and reverse ETL, with simulated incidents, automatic retries and deploys.
What the animation shows
- id,amt · {"id":7} · POST
- Raw records arriving in each source's own format
- Bronze
- Landed as-is: mixed sizes, still loose
- Silver
- Duplicates fade out; records with a red corner failed a quality rule and get fixed (✓) or quarantined; PII is masked (🔒 ███-██-6789)
- Gold
- Modeled into rows and shipped to consumers in batches
- Chat thread
- Pops up every so often, bottom right: an incident (alert, retry, recovery), a deploy with passing tests, or a reconcile run replaying quarantined records into bronze
- Hover
- A source or destination to trace its lineage
Simulated. Each behavior mirrors work I've done on production platforms.
Jonathan Thomas
Principal Data Engineer · Data Platform Architect
- digitalsocal@gmail.com
- linkedin.com/in/jmtdata
- Costa Mesa, CA
Jonathan Thomas
Principal Data Engineer · Data Platform Architect
- 2006 Operations
- 2019 Analytics
- 2022 Engineering
- 2026 Principal
I design and own enterprise data platforms end to end, from ingestion and modeling to CI/CD, governance, observability, and the dashboards leaders act on. Today that means a lakehouse unifying 40+ systems at Mutual of Omaha Mortgage, built on Microsoft Fabric and now extending onto Databricks.
- Focus
- Technical lead on an enterprise Databricks migration spanning SQL Server, Snowflake, and Microsoft Fabric
- Based
- Costa Mesa, CA
- Certified
- Microsoft Certified: Fabric Analytics Engineer Associate (through 2027) · CompTIA A+
-
Systems integrated
40+
Unified into governed medallion layers
-
Saved per year
$120K
Offshore team replaced in 90 days
-
SQL Server data
10 TB+
Managed through the move to a lakehouse
-
Power BI dashboards
40+
Executive, operational, and migration reporting
Hands-on, every week4,005 GitHub contributions in the last year · active 53 of 53 weeks, 213 days
Mostly private work repositories, so GitHub shows counts only. Snapshot Sep 26.
Selected work
Platform-scale builds where the engineering changed how the business works with data. Select one for detail.
Job-Fit Scoring Engine
Scores job postings against my own criteria. A local Gemma model on one GPU reports each posting’s facts with word-for-word evidence, and rules in code decide, sorting every posting into a band with its reason. Tested against blind Claude reviewer labels on 135 held-out postings: all 62 good fits surfaced, 5 of 73 rejects still shown, no API cost.
- Python
- Ollama
- Gemma
- Claude Agents
- Evaluation
Enterprise Lakehouse Modernization
A Microsoft Fabric lakehouse integrating 40+ enterprise systems into governed Bronze, Silver, Gold, and Corporate layers, now extending onto Databricks as the platform scales under an expanded program.
- Microsoft Fabric
- Databricks
- PySpark
- Delta Lake
- Direct Lake
Analytics Engineering Platform
CI/CD, source control, environment promotion, and engineering standards across Fabric, Databricks, SQL Server, and Power BI, replacing manual deployments.
- Azure DevOps
- GitHub
- CI/CD
- SQL Server
Cross-Platform Snowflake Data Hub
The estate's largest dataset (~250 tables, 10+ TB), built on Snowflake from concept to production. Merges a vendor secure data share with SQL Server history, then serves two consumption paths: CDC and query federation into Databricks, and mirroring into Fabric. Provisioned as infrastructure as code with Terraform and deployed through GitHub Actions, with the full operating model of SSO, RBAC, streams, tasks, and stored procedures, plus budgets and spend alerting.
- Snowflake
- Terraform
- GitHub Actions
- CDC
- Data Sharing
- Federation
- Databricks
- Fabric
Platform Observability & Cost
Departmental reporting on deployment statistics, pipeline runs, item-level spend by department, and SQL Server performance, plus a tenant-wide Power BI data dictionary.
- Fabric
- Azure DevOps
- Power BI
- Governance
Decisions
The calls behind the platform: what I chose, what it cost, and what it bought. Select one for detail.
Add a Corporate layer above Gold
- Context
- 40+ operational systems feed one lakehouse, and many teams build on it. Stopping at Gold would leave each team to define its own metrics.
- What I chose
- Extended the Bronze/Silver/Gold medallion with a Corporate serving layer that enforces semantic consistency and governance, backed by a tenant-wide data dictionary.
- What it cost
- One more layer to model and maintain, and changes to a shared definition go through one owner instead of each team.
- What it bought
- Downstream teams consume one definition of a metric rather than negotiating their own.
One Snowflake hub, two consumption paths
- Context
- The estate's largest dataset (~250 tables, 10+ TB) had to serve both Databricks and Microsoft Fabric.
- What I chose
- Built it once on Snowflake and served each platform natively: CDC and query federation into Databricks, mirroring into Fabric. Provisioned users, roles and environments as code with Terraform and GitHub Actions.
- What it cost
- Three platforms in the path instead of one, so cost and reliability had to be made visible on purpose.
- What it bought
- One build of the dataset serves both platforms, so there is one set of definitions to govern instead of two. Environments and access are versioned and reproducible, and spend, job failures and daily runs are reported in Fabric rather than discovered on an invoice.
Direct Lake instead of scheduled refresh
- Context
- Import-mode reports meant refresh windows, stale-data windows, and refresh failures to triage at 6 a.m.
- What I chose
- Moved the semantic layer to Direct Lake so reports read straight from Delta tables.
- What it cost
- Model design has to stay within what Direct Lake supports, and data quality upstream matters more because there is no refresh step to catch problems.
- What it bought
- No refresh cycles to run or fail. The number a leader sees is the number the platform has.
From billing desks to data platforms
One bar per employer; each segment is a role I held there. Select an employer to see what I delivered.
- Operations
- Analytics
- Engineering
- Each segment is a role
| Role | Start | End | Duration |
|---|---|---|---|
| Mutual of Omaha Mortgage | 4 yr 3 mo | ||
| Principal Data Engineer | Mar 2026 | Present | 6 mo |
| Data Warehouse Manager | Mar 2025 | Mar 2026 | 1 yr |
| Data Warehouse Administrator | Mar 2024 | Mar 2025 | 1 yr |
| Marketing Data & Analytics Manager | Jun 2022 | Mar 2024 | 1 yr 9 mo |
| Ceridian | 3 yr 1 mo | ||
| Technical Advanced Support Analyst, Sr. | Feb 2022 | Jun 2022 | 4 mo |
| Senior Application Specialist | Jan 2021 | Feb 2022 | 1 yr 1 mo |
| Application Specialist | May 2019 | Jan 2021 | 1 yr 8 mo |
| Informative Research | 4 mo | ||
| Production Analyst | Jan 2019 | May 2019 | 4 mo |
| Hallmark Rehabilitation → Genesis HealthCare · Genesis HealthCare acquired Hallmark Rehabilitation in February 2015. | 12 yr 8 mo | ||
| Reimbursement Analyst | Feb 2015 | Jan 2019 | 3 yr 11 mo |
| Financial Operations Supervisor | Jan 2014 | Feb 2015 | 1 yr 1 mo |
| Finance Operations Assistant | Mar 2010 | Jan 2014 | 3 yr 10 mo |
| Billing / Accounts Receivable | May 2006 | Mar 2010 | 3 yr 10 mo |
Mutual of Omaha Mortgage
-
Principal Data Engineer Mar 2026 – Present · 6 mo
- Technical lead for the enterprise Databricks migration, directing three data engineers in a 10+ person program.
- Own Unity Catalog governance for the Databricks platform, including catalog and schema design, grants, and data access and security design.
- Keep one set of architecture, pipeline, and governance standards across four production platforms: legacy SQL Server, Snowflake, Microsoft Fabric, and Databricks.
- Still own the Snowflake platform after the re-org, including its Terraform infrastructure and GitHub Actions deployments.
- Built a reconciliation app (React, Databricks Apps, DataComPy, row hashing, Claude agents) that automated migration validation and root-cause analysis.
- Wrote reusable validation code that normalizes schemas, composite keys, data types, strings, and casing so tables compare row by row.
- Set team standards for PySpark, Git workflow, CI/CD, notebook structure, and data quality.
- Databricks
- Snowflake
- Delta Lake
- PySpark
- Spark SQL
- Microsoft Fabric
- SQL Server
- React
- DataComPy
- Claude
-
Data Warehouse Manager Mar 2025 – Mar 2026 · 1 yr
- Led the Microsoft Fabric data warehouse build with a two-person team while running the legacy reporting platform.
- Took a Snowflake platform from concept to production (~250 tables, 10+ TB, the largest dataset in the environment), merging a vendor data share with SQL Server history and feeding Databricks through CDC and query federation and Microsoft Fabric through mirroring.
- Set up Snowflake users, SSO, RBAC, streams, tasks, and stored procedures, plus budgets, spend alerts, and a metadata feed into Fabric that tracks spend, job failures, and daily runs.
- Managed Snowflake environments and access as code with Terraform, deployed through GitHub Actions.
- Architected a lakehouse integrating 40+ operational systems into Bronze, Silver, Gold, and Corporate data products on Microsoft Fabric, Delta Lake, and PySpark.
- Designed PySpark pipelines for financial, CRM, marketing, telephony, identity, and security systems.
- Wrote Airflow DAGs in Python for internal and vendor file exports, Power BI snapshot reports, and data freshness checks.
- Added near real-time pipeline monitoring and anomaly alerts using Azure Log Analytics.
- Introduced source control, environment promotion, and CI/CD for Fabric, SQL Server, and Power BI using Azure DevOps and GitHub.
- Designed Direct Lake semantic models that need no scheduled refreshes.
- Tuned workloads and SQL Server performance to meet SLAs, and reported platform spend by department.
- Owned Power BI governance, including workspace structure, Entra ID access, lifecycle management, and a tenant-wide data dictionary.
- Replaced an offshore team within 90 days, saving about $120K a year.
- Managed 10+ TB of SQL Server data through the move to the lakehouse.
- Microsoft Fabric
- Snowflake
- PySpark
- Delta Lake
- SQL Server
- Power BI
- Apache Airflow
- Azure Data Factory
- Azure DevOps
- GitHub
- Azure Log Analytics
-
Data Warehouse Administrator Mar 2024 – Mar 2025 · 1 yr
- Same scope as the Data Warehouse Manager role above; title changed March 2025.
- Microsoft Fabric
- Snowflake
- PySpark
- Delta Lake
- SQL Server
- Power BI
- Apache Airflow
- Azure Data Factory
- Azure DevOps
- GitHub
- Azure Log Analytics
-
Marketing Data & Analytics Manager Jun 2022 – Mar 2024 · 1 yr 9 mo
- Led an analytics modernization that improved data process efficiency by roughly 300%.
- Architected ELT and Reverse ETL pipelines with Python, PySpark, Azure Data Factory, SQL Server, and Microsoft Fabric to centralize operational and marketing data and sync it back to business systems.
- Hired and mentored a data team while staying hands-on.
- Delivered 40+ Power BI dashboards and semantic models for executive reporting, operations, and migration validation.
- Built real-time operational reporting on Salesforce and other systems for monitoring and security.
- Set team practices for data quality, modeling, version control, and automation.
- Python
- PySpark
- Azure Data Factory
- SQL Server
- Power BI
- Salesforce
- Azure DevOps
- Reverse ETL
Ceridian
-
Technical Advanced Support Analyst, Sr. Feb 2022 – Jun 2022 · 4 mo
- Served as senior escalation engineer for high-severity production incidents on enterprise HR and payroll platforms.
- Created an internal knowledge base that cut resolution time.
- SQL
- Dayforce
-
Senior Application Specialist Jan 2021 – Feb 2022 · 1 yr 1 mo
- Advised larger clients' HR teams on Dayforce payroll, benefits, and workforce management setup, and handled the SQL troubleshooting behind it.
- Built operational reports in SQL, Salesforce, and Power BI on platform health and customer operations.
- Built a custom C# desktop application for support analysts that kept documentation current, stored and recalled frequently used SQL scripts, and launched each tool in the browser it required.
- SQL
- C#
- Salesforce
- Power BI
- Dayforce
-
Application Specialist May 2019 – Jan 2021 · 1 yr 8 mo
- Supported smaller clients on Dayforce setup and SQL troubleshooting.
- SQL
- Dayforce
Informative Research
-
Production Analyst Jan 2019 – May 2019 · 4 mo
- Managed consumer credit data integrations in compliance with FCRA.
- Optimized ETL processes and ran ad hoc analysis to reduce manual handling.
- SQL
- ETL
- FCRA
Hallmark Rehabilitation → Genesis HealthCare
-
Reimbursement Analyst Feb 2015 – Jan 2019 · 3 yr 11 mo
- Integrated the appeals processes of two companies after Genesis HealthCare acquired Hallmark Rehabilitation, meeting appeal requirements from ADR through Administrative Law Judge stages.
- Consolidated appeals data from SQL Server, Access, and Excel to make appeals performance measurable.
- Designed a consolidated billing database and built denial tracking and reporting for compliance.
- Databases
- Reporting
- Compliance
-
Financial Operations Supervisor Jan 2014 – Feb 2015 · 1 yr 1 mo
- Ran day-to-day billing operations and owned AR and aging against roughly $12MM in monthly revenue while supervising a team of six.
- Wrote SOX procedures for auditing and reconciliation.
- Built ad hoc reports combining database queries, application reports, and user spreadsheets.
- Automation
- Compliance
- SOX
-
Finance Operations Assistant Mar 2010 – Jan 2014 · 3 yr 10 mo
- Worked with IT to automate recurring finance reports in SQL Server Reporting Services.
- Implemented paperless billing, cutting paper and shipping costs.
- Owned analysis, reconciliation, and maintenance of aging reports and trial balances, and ran the contract process from sales lead to finalized contract.
- SSRS
- Reporting
- Excel
-
Billing / Accounts Receivable May 2006 – Mar 2010 · 3 yr 10 mo
- Biller and client-support contact for the company's largest client, over $2MM in monthly revenue.
- Reconciled problem aging and GL accounts and supported cash flow reporting across 200+ facilities.
- Administered the billing software, including user and location management.
- Billing Systems
- Reconciliation
- Administration
Stack
What I build with, and the systems I've integrated.
- Data Platforms
- Microsoft Fabric, Databricks, Delta Lake, SQL Server, Azure SQL, Snowflake, ADLS Gen2
- Languages
- Python, SQL, T-SQL, PySpark, Snowpark, Spark SQL, DAX, Power Query M, C#, TypeScript, PowerShell, Pandas, VBA
- Pipelines & Orchestration
- Apache Airflow, Azure Data Factory, Fabric Data Pipelines, SSIS, SQL Server Replication, Snowflake Streams & Tasks, Change Data Capture (CDC), Batch & Streaming, Azure Event Hubs
- Architecture & Modeling
- Lakehouse Architecture, Medallion Architecture, Dimensional Modeling, SCD Type 1 & Type 2, Source-to-Target Mapping, Data Modeling, Data Warehousing, Distributed Processing, ETL / ELT, Data Products, Materialized Lake Views, Corporate Serving Layer
- Governance & Compliance
- Data Governance, Data Quality Engineering, Metadata Management, Data Cataloging, Entra ID RBAC, Snowflake RBAC, Unity Catalog, SSO / SAML, PII / NPI Handling, FCRA, SOX, HIPAA, Data Dictionary, Data Quality Rules
- Cloud & DevOps
- Azure, Azure DevOps, GitHub, Git, CI/CD, Environment Promotion, Release Governance, Terraform, Infrastructure as Code, GitHub Actions, YAML, Entra ID
- Integration
- REST APIs, SOAP APIs, Webhooks, SFTP / File Ingestion, Microsoft Graph API, OAuth2 / Service Principals, Reverse ETL, Salesforce Integration, Secure Data Sharing, Query Federation, Database Mirroring, Power Automate, Zapier, XML, SharePoint
- BI & Analytics
- Power BI, Direct Lake, Semantic Modeling, Row-Level Security, Executive Reporting, Self-Service BI, Power Apps, SSRS, Databricks Apps, Self-Service Apps
- Observability & Reliability
- Azure Log Analytics, Pipeline Monitoring, Anomaly Alerting, Performance Tuning, SLA Management, Cost & Capacity Management
- AI Engineering
- Claude, LLM Workflows, AI Agents, Prompt Engineering, DataComPy
- Leadership & Delivery
- Hiring, Mentorship, Engineering Standards, Vendor Management, Jira, Confluence
- Systems integrated · 40+
- Encompass, nCino, SimpleNexus, LoanCare, DMI, CoreLogic, TransUnion, NMLS, Salesforce, Relcu CRM, Velocify, Total Expert, Five9, ADP, Emburse / Certify, Microsoft Defender, Microsoft Entra, Cofense, JAMF, Azure DevOps · sample of 20
- Managed ELT & iPaaS (POC and short-term production)
- Fivetran, Airbyte, Estuary, Talend, TIBCO
Let's build data people trust.
Always glad to talk data platforms, lakehouse modernization, and hard data-quality problems.
- ✓ Microsoft Certified: Fabric Analytics Engineer Associate (through 2027)
- ✓ CompTIA A+