Field notes on data engineering, applied AI, and the decisions behind them — for engineers who ship pipelines and the leaders who bet budgets on them.
Five ways to run dbt from an Airflow DAG, the configuration on both sides that makes them cooperate, and the concurrency, retry, and CI decisions nobody writes down until something breaks at 3am.
Metadata layers, hidden partitioning, snapshots, branches, and the copy-on-write versus merge-on-read decision — what Iceberg actually does to your files, and the maintenance jobs that keep tables fast.
Partition pruning, join design, incremental predicates, clustering, and DAG-level fixes — the techniques that keep dbt incremental runs fast as data volume grows, not just on day one.
Data warehouse, data lake, data mart, and data mesh explained side by side — what each one actually is, the use cases each one wins at, and the real trade-offs behind picking one over another.
Generic tests, singular tests, custom tests, unit tests, and source freshness — and how they map onto Technical Data Quality (TDQ) and Business Data Quality (BDQ), with real code.
INNER, LEFT, RIGHT, FULL OUTER, CROSS, and SELF joins — what each one actually does to your rows, when to reach for it, and where people get it wrong.
Views, tables, ephemeral models, and every incremental strategy dbt supports — what each one actually does under the hood, when to reach for it, and where teams get it wrong.
AI can write a SQL query that runs perfectly, returns a plausible number, and is completely wrong — with no error message anywhere. A field guide to the five failure modes and how to catch them.
Every roadmap, every job posting, every hiring manager circles back to the same three letters. Here's why SQL isn't optional — it's the entry fee.
A tool built almost entirely out of SQL and YAML somehow became the single most-loved piece of the modern data stack. Here's what dbt actually does, and why it earned that reputation.
Data Analyst, Data Engineer, Data Scientist, Analytics Engineer, ML Engineer, AI Engineer — what each one actually pays, actually demands, and how to pick the right door in.
A short note on tools.techedge.in, what SQL formatting actually does, and what we're building next for data teams.