BENlabs

The ML Platform team exists to empower data scientists to rapidly develop and own AI services end to end by making the DevOps and software aspects trivial.

I am accountable for 
• the ML Platform team meeting our quarterly and annual KRs
• having clear ML Platform architecture and ownership/support model
• platform outages
• sprint planning
• interviewing; writing job descriptions
• acting as an individual contributor--second to ensuring the team is productive and unblocked

ML Platform Technical Lead

Some projects:

• Championing a company-wide "constructs library". Like https://constructs.dev/, but just for BENlabs. 

This is a great balance of deploying infrastructure to AWS in a consistent, secure way, but still gives individuals and teams freedom to deviate for new architectures.

Sidenote: we set a goal to have all of our engineers and many data scientists become "IaC-literate". It turned out to be doable. Great boost in speed & quality. Very exciting. Exploring LocalStack to make testing of IaC faster and easier. That's been a pitfall of developing IaC.

• Championing a project, boilerplate code tool (with projen, https://github.com/projen/projen). Goal: data scientists and engineers should be able to "launch" new experiments and products in minutes instead of weeks. 

I made these projects as open-source POCs before continuing them closed-source at BEN and digging deeper. POC for creating repos with IaC: https://www.youtube.com/watch?v=7TDrysCv5xg; POC of Python-projen library https://github.com/phitoduck/phito-projen

• Doing a POC of BentoML as opposed to FastAPI, Seldon Core, etc. for model serving. Gave a meetup talk on it at Utah's MLOps.community chapter. Being active in the BentoML community. 

• Technically, I'm a co-organizer of the MLOps Utah meetup, but I need to contribute more to claim credit for that.

• I'm one of the "local experts" on Auth0, NewRelic, DNS, Pulumi, projen, AWS CDK, packaging, and BentoML.

• Supporting an MLFlow POC, mostly led by Jon Jones.

• Pushing hard for support plans: AWS Support, BentoML dedicated Slack channel, NewRelic support. 

• Seeking ways to form relationships with AI teams at "sister tech companies", maybe with a mutual-NDA. 

• Organized a hackathon with some co-workers and friends outside of work. We made a self-hostable, serverless Minecraft Platform-as-a-Service using AWS CDK. ("pip install awscdk-minecraft") https://github.com/mlops-club/awscdk-minecraft

Senior MLOps Engineer

• Write/extend tools to make DS models trivially easy to "deploy" as REST APIs and in batch jobs on AWS. 

• Lead "hand-offs" of AI products from DS to DE.

• Research/implement MLOps practices: experiment tracking, shadow deployments, catching model drift, etc

• Worked with Steve Jackson and Daniel Clark to build an important JSON:API (web service) whose purpose I probably shouldn't say here. We've continued to use the framework we established here for new AI services. Now, heavily researching observability vendors for correlated metrics, logs, and traces (which trigger alerts). 

• Established patterns for testing applications in docker. Currently developing CI framework to run a battery of tests against services *after* they have been deployed to the cloud, and then on a schedule thereafter.

• Working on a modeling project with DS. I enjoy this and the learnings should feed back into the ML platform tools.

• Did/doing heavy research into no-downtime deployments of REST APIs--particularly when the underlying transactional database schema needs to change with the REST API version.

• Write loads of infrastructure as code. Heavily researching pro's/con's of AWS CDK vs CDKtf vs Pulumi. Also, establishing standards for those.

• Wrote a CLI-tool wrapping Sphinx to generate versioned documentation for all DE/DS repos. The docs build and are published in CI. Docs can be viewed in PRs as well :)

• Extended a project template framework: DE/DS engineers can generate boilerplate Python projects, ready with CI, docs, Dockerfiles, packaging configuration, tests, semantic versioning for internal PyPI server, and other nice things. Every team should have this!

• Actively mentor.

• Actively participate in recruiting and interviewing. Went to career fairs at BYU and UVU. Holding an info session at BYU promoting BEN and offering career advice.

• Participate in our internal AI paper club. Presented on the transformer architecture.

MLOps Engineer

Elevated Analytics

We offered Salesforce analytics via SaaS--as well as mentorship and training on best practices for tracking your sales process. This is ideal for groups who have not hired a full-time analyst but want affordable, high-quality visibility into their sales funnel.

Platform Engineer, Cofounder

rootski

Building an AI startup “in the open”, using the project as a mentorship platform for people wanting to get experience in all sorts of engineering and engineering-adjacent roles.

See the explainer video at https://www.rootski.io (looks awful on mobile 🙂)

Learn about the project and how to contribute here: https://github.com/rootski-io/rootski. Hope to see you on Slack!

Mentor/founder

Karat

Conduct technical interviews on behalf of client organizations.  My goal is to make interviews an uplifting experience for the candidates－whatever the outcome.

Interview Engineer

XANT

• Wrote many data pipelines and support tools using Apache Airflow. I also "Docker-ized" Apache Airflow for our team, managed its deployment in Kubernetes, and worked with the AWS support team to resolve mysterious Airflow/Kubernetes issues.

• Researched "Infrastructure as Code", then led an initiative to standardize deployments of all new services using AWS CloudFormation templates; later leveraged that work to spin up all data engineering services in a staging environment (Terraform, CloudFormation, pretty diagrams)

• Led migration and rearchitecting of a REST API for machine learning inferences, which was consumed by our core product. I used my data science background to reduce the HTTP request latency from 2+ minutes to <20 seconds in the worst case (Flask, python, Airflow, DynamoDB)

• Led an initiative to create and standardize unit testing and CI/CD pipelines for all Python projects (Bitbucket/Bamboo, AWS Container Registry)

• Assisted in maintaining massive data ingest pipelines using Kafka and other tools to land data in S3, Snowflake, and curated relational/non-relational databases

• Made lifelong friends and awesome software with brilliant people

Data Engineer

Data Engineer Intern

InsideSales.com

Wrote code to develop data infrastructure including SQL and NoSQL databases, Kafka, RabbitMQ, Airflow, and many services within AWS

• Led trainings 2x/week for analytics team on Python, Docker, Linux, and AWS which enabled multiple team members to independently write and deploy scripts to production.

• Developed a customer churn model from Salesforce and customer usage data. Gathered data SQL and a 3rd-party JavaScript-based query language (JQL)

• Deployed Airflow on a VM to run data pipelines daily. Helped another engineer dockerize and deploy an R script with Airflow that sent alerted account managers of accounts they hadn't reached out to in over a week.

• Learned SQL and DOMO. Also learned to make BI dashboards with a lot of handholding from my manager.

Lessons, looking back years later:

(1) After I left, Airflow fell over and the analysts stopped using it. It was unrealistic to expect a team of traditional B.I. analysts to maintain such a software-engineering-oriented tool. 

Positive note: some Airflow pipelines proved valuable, and were reimplemented in simpler tools.

(2) The customer churn model was not useful so it was abandoned. I think the sad reality is most A.I. projects are not worth building, and it's easy to get distracted to complexity.

Analytics Intern

InnovAsia, Inc.

- Worked with strategy team to identify and implement software solutions that eliminate bottlenecks in 2 million parts per month global supply chain
- Developed application for systems administration of U.S. and Chinese accounts (Python, Bash)

Contract Full-stack Developer

Brigham Young University

Started a rewrite of the lab's landing page in AngularJS. Never finished, but was a valuable learning experience.

Research Assistant in IDeA Labs

Helped a few hundred beginner CS students with clean code and OOP fundamentals in C++

Teaching Assistant (Intro. to Programming)

The Church of Jesus Christ of Latter-day Saints

— Created an internal support program in 3 cities—training Russian church members to visit, teach, and care for each other
— Lead training meetings with 21 volunteers to adjust to a major legal policy change in 2016
— Served as a Russian translator for top church leaders during regional conferences

Full-time Volunteer

Hi! My favorite area in tech is building data-powered apps. I do a lot of personal projects and focus my work experience to learn about all the buzzword aspects of that process: 

- services in the cloud
- AI models and their lifecycle
- databases, warehouses, lakes, pipelines for making data consumable
- websites, apps, etc. for users to interface with the data

As an interviewer, I look for immaculate code and give lots of credit for style even if the candidate doesn't solve the problem. I know several others who do, too.

This may be the only article you need about how to write clean Python code.

On top of this, if you read the (beautifully written) `mypy docs` and the `Google Python Style Guide`, you'll really level up your clean code game.

I've seen some interview questions that do nothing but have a candidate refactor some already written code. 

It's really insightful to see if they're able to simplify a code block that's difficult to reason about.

Plug: I've got a bunch of videos in my that cover these principles and include an exercise where we refactor a text-based RPG together. Most of the videos are free. If you like, there's a course link on my profile page.

------

[edit] Tomorrow: I'll post an actual interview question I give people as well as an example solution. If that does well, I'll share some more of those.

 Linkedin has a feature (🛎️ bell icon) to get alerted on posts for people you follow. You could use that or just check my posts if you'd like to see it.

🎧 Eric Riddoch

testdriven.io • 10 min read

Clean Code in Python

I used to feel impostor syndrome. 

But then Demitri Swan helped me realize:

I *am* an impostor. That was helpful, b/c it's not impostor syndrome if you're a real impostor. #cured

"I used to think Leetcode interviews were ridiculous, and then I hired someone who didn't know how to code and it was a disaster."

^^^ Paraphrasing a friend I spoke with today.

The candidate

- had a senior engineer title
- said all the right things
- had written a book on using Python
- had public code on GitHub

and couldn't create a virtual environment. I guess they'd been a manger for a while and had become nontechnical and out of date :/

Another time, my friend did a interview himself once for a senior+ position. They had him solve Fizz Buzz.

Funnily enough, he had the reverse reaction: 

"You're having me solve Fizz Buzz??? This problem is ridiculous. What does this say about the engineers who already work here?"

His interviewer: "All the problems are ridiculous. But you'd be surprised. 80% of people fail Fizz Buzz."

My take: my interview questions typically focus more on SWE fundamentals than algorithms. Something simple, combined with a live discussion to hear what they think. I think this is a really helpful approach.

But you know what? I'm definitely reassured if I know they have learned their algorithms at some point, too--and if they could produce a BFS when asked, that helps!

It's all data. It all helps paint a picture. I say collect it! 

But try to make it streamlined and comfortable for the candidate.

I just did a "Hello World" exercise in kubernetes! See? Yes, I turned on the Kubernetes dashboard. Is this where I post and say that I'm an expert now?

Or do people only do that for AI and web3?

I never understood why people mock their database in their tests...

...I mean, if you hurt its feelings, why would you think it'd work in prod?

As an interviewer, I look for immaculate code (part 2) 

Here's a sample interview question.

It's a refactoring exercise.

Here's a code-block that defines a turn-based, text RPG game where you fight a dragon-cat. 

In my opinion, this code is hard to read. 

What kind of changes would you make to refactor this code and why?

---

Plug: This exercise actually comes from my "Taking Python to Production" course. In the comments, I'll link to the course article that goes over a solution I came up with and my reasoning. 

Plug 2: There's a famous Martin Fowler book called "Refactoring" as well (albeit the examples are in JS, but the concepts are widely applicable).

In the course, we spend 23 videos--most of which are free--on `Python typing`, crafting libraries with perfect autocompletion, refactoring, the `Google Python Style Guide`, measures of code complexity, e.g. `cyclomatic complexity`, formatting, and linting.

I think part of my personal brand is I’m a “kubernetes denier” 🤣

In a shocking turn of events, I’m actually trying to “learn kubernetes” again for the first time since 2019, so that I can better understand BentoML and Metaflow.

Side note: It’s actually a uniquely good time to contribute to BentoML right now! They’re redoing their serving platform and licensing it under Apache 2.0 🎉

I have two questions I’d love advice on:

- what’s the cheapest/best way to play with EKS? I feel it needs to be EKS specifically because K8s in a cloud means learning how to set up IAM/OIDC, CNI with VPC networking, etc. Maybe LocalStack?

- what tooling do you recommend to do K8s “right”? I’m seeing CDK8s, Karpenter, Pulumi, helm, ArgoCD, … 

Any standout resources for learning these? 

👾 Ryan Russon, Joseph Szendre, Peter Halliday, Daniel Clark, Dana Engebretson, Jacob Lyman, Maria Vechtomova, Raphaël Hoogvliets

Return to office? Hybrid work? Full remote? Conspiracy to prop up real estate market? 

The answer is clear. Happy Thursday!

One of these will be the death of me 🤣

Taplio

🎧 Eric Riddoch's Linkedin Analytics

🎧 Eric Riddoch

What is 🎧 Eric talking about?

🎧 Eric Riddoch's Best Posts (last 30 days)

Want to drive more opportunities from LinkedIn?

Taplio

.css-1jgreyo{background-image:linear-gradient(284deg, rgba(39,43,72,1) 0%, rgba(33,47,183,1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}🎧 Eric Riddoch's Linkedin Analytics

🎧 Eric Riddoch

What is 🎧 Eric talking about?

🎧 Eric Riddoch's Best Posts .css-qqfgvy{font-size:var(--chakra-fontSizes-md);}(last 30 days)

Want to drive more opportunities from LinkedIn?

🎧 Eric Riddoch's Linkedin Analytics

🎧 Eric Riddoch's Best Posts (last 30 days)