shubham srivastava

At Amazon, I have spearheaded the transformation of data architecture, establishing a robust, data-driven culture that supports strategic decision-making at scale. By architecting and deploying cloud-native AWS data lake solutions, I reduced data processing times by 30% and improved query performance by 45%, significantly advancing analytics capabilities across the organization. Over the past five years, I have specialized in designing and optimizing real-time and batch data pipelines, achieving an exceptional 99.99% data availability that ensures seamless analytics for global operations.

My leadership has been instrumental in deploying self-service analytics tools, driving a 25% increase in user adoption among over 1,200 business users. These efforts empower stakeholders to derive actionable insights, streamlining decision-making processes independently. Beyond analytics, I have developed Python-based CI/CD frameworks that automate workflows, reduce deployment time, and elevate engineering standards.

I am deeply committed to operational excellence, consistently improving processes, and fostering innovation. By aligning strategic objectives with cutting-edge data solutions, I have enhanced productivity and ensured that the infrastructure scales efficiently to meet Amazon's growing demands. My ability to translate complex technical challenges into impactful solutions has driven measurable success and empowered teams with reliable, high-performance data systems.

Senior Data Engineer

Partitioning + Clustering triggers cheatsheet for Data Engineers, I wish I had when I was a junior.

1) When most queries filter on the same column

When you see

- Every dashboard starts with WHERE event_date BETWEEN ...
- Almost every job filters by region, tenant_id or event_type
- Query text looks identical across teams, only ranges change

Then do

- Pick that column as your primary partition key.
- Keep the key simple: event_date, country, tenant_id etc.
- Check cardinality: a few hundred or a few thousand partitions is fine.

Avoid

- Partitioning by 3 or 4 columns at once just because the tool allows it.
- Changing partition keys every quarter. Stable keys keep pipelines boring.

2) When the workload is time series

When you see

- Events, logs, metrics or clickstream arriving all day.
- Queries like “last 7 days”, “last 90 days”, “this month versus last month”.
- Backfills that always target specific dates or hours.

Then do

- Partition by time first: start with daily partitions.
- Only move to hourly partitions if a single day is very large or queries are extremely tight on time.
- Put clustering or sort keys on the next most common filter, such as customer_id or service_name.

Avoid

- Jumping to hourly or minutely partitions just to “match the data frequency”.
- Mixing time zones in partition keys. Pick one canonical time zone and stick to it.

3) When time and one more key always travel together

When you see

- Queries that always filter like WHERE event_date BETWEEN ... AND tenant_id = X.
- Dashboards per tenant or per region for the same time range.
- One tenant or region that often becomes a hotspot.

Then do

- Use composite partitioning: (event_date, region) or (event_date, tenant_id).
- Keep the second key low cardinality: regions, tenants, platforms, not user id.
- Inside each partition, cluster by the high cardinality key if needed, such as user_id.

Avoid

- Composite partitions where both keys have huge cardinality.
- Partitioning directly on user_id or order_id. It only looks smart on day one.

4) When joins dominate the workload

When you see

- Big fact tables that always join to the same dimension keys.
- Pipelines where shuffle time in joins is bigger than everything else.
- Star schemas in analytics systems.

Then do

- Cluster both fact and dimension tables on the join key (customer_id, account_id, etc.).
- For heavy, repeated joins in Spark or similar engines, consider bucketing on the join key for both sides.
- Keep bucket count tied to scale: enough buckets to parallelize, not so many that files explode.

Avoid

- Leaving large tables unclustered while trying to tune Spark configs. Layout beats configuration.
- Bucketing on different keys on the two sides of a join. That defeats the purpose.

Continued in Comments:

Shubham Srivastava

Principal Data Engineer @ Amazon | Data Engineering

I am a Principal Data Engineer at Amazon. I see a lot of talk about using AI to write code, but IMHO, for Data Engineers, the real value is in optimization and validation.

Because once a pipeline is in production, nobody cares how pretty the code is. They care about skew, cost, SLAs, and silent data bugs.

Instead of “write this ETL for me”, I use AI like this.

1. Turn AI into an execution plan reviewer

Paste:

- A slow Spark job
- A nasty join-heavy SQL query
- Your current cluster or session config

Then ask things like:

- “Where can skew happen in this join strategy?”
- “Which stage of this plan looks like a bottleneck?”
- “How can I reduce shuffle or memory pressure in this job?”

AI is very good at spotting:

- Broadcast vs shuffle mistakes
- Useless sorts and scans
- Wrong partitioning strategies
- Configs that look safe, but kill performance at scale

I still decide what to change.

AI just surfaces blind spots faster than manual inspection.

2. Use it for pattern conversion, not copy paste coding

Most migrations are 80 percent boring translation work.

Examples:

- Legacy warehouse SQL to PySpark or Trino
- Vendor specific functions to standard SQL
- Old UDF heavy logic to newer built in functions

Give AI:

- A clear source query
- The target engine or dialect
- Any limits, for example, “no UDFs, prefer window functions”

Then treat the output as a draft:

- Check that semantics are preserved
- Check that joins and filters happen in the right order
- Check that it uses features your engine actually supports and that you can monitor

You free up your brain to think about correctness and edge cases.

3. Let AI attack your pipeline with fake data

Most pipelines break on the data you did not imagine.

Instead of “write this ETL”, ask:

- “Generate 5 input datasets that should make this job fail”
- “Create test cases with nulls, extreme values, wrong enums, late arriving facts”
- “Give me sample records that would trigger every branch in this transformation logic”

Then:

- Turn those into unit tests and integration tests
- Wire them into CI for your ETL or Spark jobs
- Use them as regression tests when you refactor

You are using AI to think like a malicious user.

The goal for Data Engineers is not to ship code faster.

The goal is to ship data systems that break less, cost less, and are easier to debug when something goes wrong.

Use AI to:

- Review execution plans
- Translate patterns across engines
- Generate edge cases that attack your assumptions

Senior Data Engineer @ Amazon | Data Engineering

taplio

Shubham Srivastava's Linkedin Analytics