Imran Shaik

software engineer / , data pipelines / , backend systems / , AI

Hey, I'm Imran, a software engineer in Hyderabad, India. I build data pipelines and data platforms for a living, mostly with Python, PySpark, Databricks and a whole lot of AWS. My favourite example is a financial data pipeline I built: it swallowed 14TB of historical data in its initial load, then switched to CDC mode to replicate live changes as they happen. Big backfill first, then it keeps up in near real time.

What I care about most is data and backend systems that are fast and that people can trust. If the numbers are wrong or late, nobody cares how elegant the code is. A lot of my work is unglamorous, and honestly that's where the fun is. One of my data push jobs moves half a terabyte into an Iceberg lakehouse in 20 minutes. Another chews through 250GB in about 6. Nobody throws a party when a job finishes on time.

I'm a sucker for data models and architecture patterns. I like figuring out how data should be shaped before anyone writes a single query against it. Have I mentioned I designed a partition strategy for an Iceberg warehouse based on how people actually query it, and it improved data delivery timeliness by 94%? Well, now I have. Twice, if you count this sentence.

Then there's data QC, which I take personally. During a migration I audited data across a source database, the old sink database and the new lakehouse. I checked source against new sink, then new sink against old sink, so every row had to show up where it was supposed to. Trust is earned one reconciliation at a time.

These days it's AI (a lot of AI, actually ;)). I've built MCP servers and reusable Claude skills that a bunch of engineers now use in their own work. Turns out building tools that other engineers want to use is its own kind of satisfaction.

I started my career building APIs and never really left. Low-latency, high-performance REST APIs are still my thing, and I'll happily argue about response times with anyone who'll listen.

Oh, almost forgot: I also handle the security side of things, using Burp Suite and Checkmarx to find holes before someone else does.