Creating or Finding Value as a Data Scientist

Creating or Finding Value as a Data Scientist

Join our youtube channel to see more detailed content. 

By Jose Quesada

Hi, this is Jose Quesada for Data Science Retreat.

Today let’s talk about a word with no clear definition but has much meaning: Value. The word in itself has a more of a ‘fluffy’ or a vaguer meaning attached to it, so let’s give it a more concrete definition.

Everyone has ideas, and they want to make products out of these ideas. But what makes these ideas unique would be the value that it creates. It is this value that differentiates every entrepreneur from the rest.

So, what is Value? Personally, it means something which solves a real problem. Ideally, there are several audiences who have problems that they do not know how to solve. The concept of Machine Learning is something that gives a Data Scientist a tremendous amount of leverage over others to solve problems. 

I have made a video on this topic for those of you who prefer watching a video than reading, see below.

Play Video

So, what can you do with this unique leverage that creates value? Let’s navigate this issue through some examples.

  1. Suppose you have a child who is Dyslexic. Dyslexia is a disease that affects about 25% of the population, according to some studies. Dyslexic children have a hard time trying to read. For a parent, this is worrisome because, of course the child suffers, but also, the parent will have to spend more trying to solve the child’s problem. Since this is a solution that affects a large population, and you find a solution to this using Machine Learning, you create value.
  2. Another example is if ‘You’ have a rare condition. This is not an issue that most of the population faces, but creating a solution out of it will generate much value to you and others who have the same condition.
  3. There are small problems that companies face, like wanting to create a catalog of things they purchase to keep track and reduce the company’s cost. If you use Machine Learning to find a solution to this issue, a lot of companies can cut their costs with the value that you have generated.

Now, if you are a Data Scientist who is comfortable in his/ her company and thinks that you don’t want to look out for problems and solve them, though you may be able to survive in the long run, it is not something I personally recommend. Why?  Let me elaborate with an example:

Suppose there are 2 Data Scientists- Andreas and Vlad. Andreas is very good with people and wants to find solutions to the problems that various departments in his company face. He is well-liked in the company since he actively looks out for problems and solves them using Machine Learning, which other departments did not even know could be solved. Vlad, on the other hand, is a very productive employee too. If someone gives him a problem, he will find a solution to it. Everyone knows Vlad will be able to get to the bottom of any situation.

But who do you think makes more money? Andreas. Why? He finds problems and produces value for the company. He is an irreplaceable asset to the company; hence they will pay him more to keep him. He also has the ideal Product/ Market fit, which is very important in every company.

https://images.app.goo.gl/ebC6vsqV7HMCEiz39
How can you solve problems? By merely talking to people- be it on LinkedIn or real life.
In the book ‘The Lessons School Forgot’ by Steve Sammartino, he introduces a concept called ‘10 meetings’.
 

Suppose you are entirely new to the market, and you want to know more about it. You reach out to someone from the industry and ask to meet them over a cup of coffee to get to learn more. At the end of the conversation, you would have picked up certain keywords from the discussion that you can go home and lookup. Now, when you meet another person, your questions will be much more specific. And by the time you reach the 10th meeting, you possess good knowledge about not just the industry but also the problems in the industry, which you may be able to solve with Machine Learning. Then you can approach someone with a valuable solution to their problem, and you’ve secured the job as well!

The next idea is from the book ‘The Mom Test’ by Rob Fitzpatrick. Suppose you have a solution to an idea, and you go to your mom and present it. When you do, most probably, your mom will tell you that it is a great idea, even if it may not. Why? Because your mom loves you and wants to keep you in happy spirits. But what you learn from this is that people will lie to you.

So how do you get a reasonable opinion about what you think is a valuable idea? There are three things which you can do here.

1. Talk about their life instead of your idea:

Suppose you have an idea to make an app that gives you recipes based on the things you have in your pantry. You think this would be a successful app, so you test it with your mom, who has been cooking for a long time. When you tell her this, she seems enthusiastic at first and gives it a try. But eventually, you realize that she isn’t using the app since it isn’t really a necessity to her. Her way of cooking is a process whereby she plans much ahead, makes a list, buys things, and makes it. Such an app may not be a necessity to her or for many others.

2. Ask about specifics in their past instead of generics or opinions about the future:

If you ask your mom a generic question like ‘Do people like recipes?’, she will definitely say Yes! If you ask her a question like ‘How often do you check your recipe book’? She may say that the last she checked was five years ago as she has been cooking for 20 years.

Since what you want to know is that if people would be interested in an app that automatically generates recipes, you can derive ideas from your own circle by asking specific questions to them, which gives you a glimpse of their life and their needs.

3. Talk less, Listen more.

You can go from guessing the problem to be in a spot where you know the problem really well. You could be the person who knows about the problem the most in the world, and you have the solution. That is a wonderful place to be. So, you want to go in that direction.

This is very important when you want to know whether your idea is good enough or not. You may be very consumed in your idea that you may have forgotten other issues that come with it, it’s a necessity, or it’s practicality. When you talk to someone, just listen and you would get to know more about the actual issue and how often the product will be used- giving you the idea of what you actually need to do rather than you had planned to do.

 

To conclude, what makes every idea a class apart is the value that it creates.

To create more value, it is necessary to actively lookout for problems that require solutions and work towards creating the best solution from the rest.

 

transcription credit : Neeraja Nair

What do you want to achieve with your Machine Learning project

What do you want to achieve with your Machine Learning project?

Join our youtube channel to see more detailed content. 

By Jose Quesada

Hi, this is Jose Quesada for Data Science Retreat.

Today I’m going to talk to you about one thing that I’ve been thinking about for a while which is, what do you want to achieve with a project in AI or Machine Learning or Data Science. People all care about having projects having something to show. But there are different levels, this is something that I really want to go deeper.

Most people only want to learn a skill set with that project, which is fine.

But there are like different levels which we will discuss here and I would personally opt for the last option which is to create a product which you can charge for because if you fail you’re going to have kind of a fall back on one of the others.

I have made a video on this topic for those of you who prefer watching a video than reading, see below.

Play Video

Learning a skillset that’s a given. No matter what project you’re going to do, you are going to learn something, not a big deal. Now, one other goal would be to look competent and the real goal is to find a job. This is also very common “oh I’m doing this just because I want to put it on my LinkedIn and my github and I’m going to get the attention of recruiters or hiring managers. You want to get a job, that’s totally fine. So, there’s a difference between these two, right? It could be that, the thing that you do for running a skillset is not the same thing when you move to look higher now.

 

Now we’re getting into deeper things. So, you want to solve a real problem with your projects, that’s not very common. I would like it to be more common, but it’s not. I see people saying “oh I want to have a social impact, I want to help people, I want to solve your problems” but it’s very unlikely that you find people doing something about it with machine learning. And it’s totally doable, there’s plenty of things that you can do in machine learning today to solve a real problem that real people have. It’s kind of a shame that we are not doing more of it. So that would be one more goal that is kind of deeper than looking competent to get a job, right?

 

There is one more level which is to create a product. A product is not only a project it’s a product because you can charge for it. This is something that is packaged in a way that these people that have their problem and maybe can pay for it. They are willing to do something about it, that is a product.  Most people, I would say 99% of the people that I deal with today never get to the point where they take a project and build it to a point where they can charge for it. This is not a criticism, there’s a real reason for this. Traditional companies, they all want it, but they don’t get there mainly because of how much money and effort real big companies spend trying to figure out what will work and if they build it, how to market it and so on.

 

It’s totally fine, if you as an individual cannot get to this point and build something that you can charge for. But it’s a big part. All these companies spend a lot of money in doing market research, focus group, trying to get data from their users and using that data to sell them back something like netflix does. Netflix produces series that are influenced by the data they collect. Sometimes you see a series and you can see “oh this was targeted to teenage women who are overweight, and they would love this series”. How do they find out that there is a market that would want to consume that series because they have data and they know things that the person is watching? So, the big difference between you and these big corporations is that, they need to solve a problem that affects millions ideally billions of people.

 

That’s cool for you, example they will never start a new project if it doesn’t affect billions of people. So that limits their options quite a bit. They are kind of the king of the jungle for machine learning, there is no other company doing better with machine learning so far than google. They are probably good at products as well and you can question that because they give a lot of products right, but they have a problem which is they need to be products for billions of people. You don’t have that problem; you can go to a tiny little narrow group of people that you know very well and help them. You may know them so well that you may meet one of them who will identify with your idea and you might identify with their problems. It will be a beautiful match within the two of you.

 

So, here there are some ways to think if you are going to go for the product part of it where you want to create a product that you want to charge for. I’m not saying that you need to do that, but if you want to go that way then there are like different scales.

You pay People >>>>>>>>>>> They pay you

Sometimes you come up with an idea, and you want people to use it, but they actually don’t and give you reason such as I don’t have time now. Then you pay people to use it, or you promise them a favour. Imagine that your product is a questionnaire. You tell them if you fill this questionnaire then I’m going to give you a chance to win a Macbook. So to pay people to do something for you, that’s kind of one extreme.  The other extreme is where they pay you and they will be sad if your product doesn’t exist anymore. You want to be moving from you paying to they are paying you. I’m not saying that this is easy. This is something that companies big and small like startups as well struggles with right now.

You are undifferentiated >>>>>>>>> You are the first solution they think about

Sometimes you come up with an idea, and you want people to use it, but they actually don’t and give you reason such as I don’t have time now. Then you pay people to use it, or you promise them a favour. Imagine that your product is a questionnaire. You tell them if you fill this questionnaire then I’m going to give you a chance to win a Macbook. So to pay people to do something for you, that’s kind of one extreme.  The other extreme is where they pay you and they will be sad if your product doesn’t exist anymore. You want to be moving from you paying to they are paying you. I’m not saying that this is easy. This is something that companies big and small like startups as well struggles with right now.

You are guessing >>>>>>>>> you have validated the problem and solution

The last one is, you are consuming, you read a lot, you see blog posts about companies being built, about people making money with products maybe somebody just came up with something as simple as a newsletter.They’re making money with it, you read the newsletter you consume. You must aspire to go to the point where you produce, when you write. People should follow you because of what you write.

You consume, read >>>>>>>>> You produce, write

You can go from guessing the problem to be in a spot where you know the problem really well. You could be the person who knows about the problem the most in the world, and you have the solution. That is a wonderful place to be. So, you want to go in that direction.

What is that one hack to get all these scales much faster to move from one extreme to other?

It is

Let’s just look at the scales now, assume that you are in a niche. It’s easier to get people to pay you if you are targeting the right people and they have a problem that is very narrow to themselves. It’s a problem that’s worth paying for.

You’re probably the first solution to think about if you are the one person talking about this. Nobody else cares about it but you do, and you care for it very much. You know the problem very well, and you’re building a solution for that niche. You can be producing and writing about that niche and then it’s easy for people to follow you to find you. So basically, all these things become easier if you have a niche.

This is actually the big advantage that you have over google. You don’t need to create a product for a billionpeople, but a product that’s probably for few thousands. You’re going to be looking competent. So, like I mentioned earlier with this product, you’re going to learn a lot, solve your problem and you may be able to charge for it.

Therefore, the most important message I wanted to convey is that, it’s much easier for you in every possible way to aim to create a product that you could charge for and then degrade into solving a problem, finding job and lastly learning a skill. Companies are going to be impressed if you show up and show a product that you build with your hands that solves a problem in the market which you could charge for.

This is something that I would hire if I ever see in my doorstop, two data scientists with similar profiles but one of them have built a bankable project then he/she is way out of the league of anybody else and I would hire them.

To conclude “Aim to work on building a product rather than a mere project”

Is it too late for me to move into Data Science?

Is it too late for me to move into Data Science?

Join our youtube channel to see more detailed content. 

By Jose Quesada

Hi, this is Jose Quesada for Data Science Retreat. I want to cover a topic today that I find very interesting. Many people come to DSR, thinking that:

They are too late

They could have started earlier, and now they are late in their careers…

How are they going to make this happen…

They really want to be data scientists, but they think that they are maybe late at their late 30s or maybe even 40s..

So, I’m here to tell you that no, I don’t think it’s too late for you. I’m going to give you some reasons and maybe examples too. I have made a video on this topic for those of you who prefer watching a video than reading, see below.

Play Video

 

So, the first thing that you need to understand is that data science is totally a lateral move for everybody. It’s not like there is a career path that you go to university and get some kind of certificate (certificates in general of even master programs, I don’t see them working in the market).

 

I’ve interviewed many people with master’s in data science no, they didn’t pass the interview, and looking at the curriculums in their programs, they’re not going to do any better on their own, so that’s good news for you. That’s a level playing field.

 

If you’re coming from a different industry, so what? Everybody is doing the same. So that’s an advantage, and that comes with the territory when you are experienced. You have already produced something in your life that is valuable to somebody.

"Remember to produce something that is to somebody"

So good news for you! You’re an experienced person you may have a leg up just because of your experience, so you may say yeah that is true, I agree but other people who are younger than me have advantages over me.

Let me tell you one more thing that I think could benefit you, which is your network. What happens when you are new in any task, you don’t know people in that particular domain then everything becomes much harder. But when you’ve been around in one industry, it doesn’t have to be data science you magically have other people that you can ask things, and every person in your network is

a potential source of problems, which is very important. Everything in the life of a data scientist is to find a good problem, where you can provide a solution with this new technology, which is data science.

How do you do that? It’s much harder with your experience when you haven’t seen much in the industry.

I’m sure that I cannot produce problems to say the golf industry because I’ve never played golf. But if you have 30 years of golf experience, then I’m sure you exactly know what a good solution for a problem that exists in golf is.

Photo by Courtney Cook on Unsplash

Let’s say that you look at style transfer or keypoint detection like the skeletons for your pose like dance pose, and you say wow, this would be fantastic to learn and doing something with golf. You come up with that idea, and it may be a great idea, I cannot because Idon’t have any experience there and I don’t have a network either. 

So the same moment you have the idea you can call three people and tell them the idea and people will be like ‘Oh my god I will definitely use that if you create a system so I can learn the x,x,y golf moves they do because I don’t even know the names.’

 

They are going to love you, and you can test that because you have a network. More so, you can go and sell your things to that network before you build them, which is the best validation possible. If this idea resonates with people and they tell you Yes, I would definitely use it and you tell them okay how about 30

bucks per month, and they say No. Then maybe the idea is not that valuable, maybe it’s not so powerful for them to solve this problem.

But you have a network, and you can use it to validate and fine-tune problems. This network is also going to be valuable to the company that hires you as a data scientist. You may not expect that, but imagine that you’re a medical doctor, and you are actually learning machine learning on the side, and you are

becoming good at this. So, you apply for a job at a company that is doing something with x-rays, and you are 40. Do you think that they’re going to look at you and say well this guy is 40, We don’t want this type of profiles? No, they’re going to look at the medical doctors, and they are in the domain of x-rays, so you are going to see things that they don’t see because they don’t have domainknowledge, but you do.

This is fantastic; you have domain knowledge, it’s going to open doors for you. So, make sure that both your network and your domain knowledge is valuable to the company that you’re joining, and you may say but Jose all companies that I’ve worked for in the past they were boring, and I didn’t learn anything there.

 

That’s not true you probably know more things than anybody else in a room that is trying to hire and this is a really good match because vertical AI companies are really really valuable when they have domain expertise domain knowledge that nobody else has, and it’s the mixture is really hard to find then so much more valuable the companies

 

So remember network domain expertise, and you must have some management experience because if you’ve been around for a while, you may go well, I actually have only three people in my team, but  I was the most senior of them so they looked up to me when we had to make decisions or maybe I could interface better with the decision-makers because I was kind of in the same age range or I knew them for a long time. This is valuable definitely play that up in your LinkedIn in your applications and describe how you solve problems because you had management experience. When you are interviewed, remember not all the interviews are super deeply technical some are really like oh this is not going to fit with your culture.

 

Also, you must have some experience period, not just management experience, but you must have done something that the left trace does.

 

So let’s say that you are an athlete, you were an athlete and you were swimming competitively for many years or even something more radical you were a martial artist and you may say well that experience is completely worthless to anybody trying to hire me as a data scientist. Well, No, you have experience of how it is to really try hard at getting something and failing multiple times. You have the experience of training and improving and measuring your performance. You have experience of trying different things and finding one thing that works better than the others and doubling down that. This is totally transferable to the things that you want to do as a data scientist. You may go in a direction that is something related to AB testing or bandits. That’s exactly the same use case that I have described.

 

 

But there are disadvantages to being experienced or being older. What are those?

You are more expensive than others. There’s no way around. You’re not going to get paid the same as a fresh out of university person. You may not be able to want to pull long week hours like 60 or 80 or

typical startup business hours. That is personal, but maybe you can do as much with fewer hours. That’s the issue of culture, so if anybody in the company is really of a particular mindset, being the elder outsider may not help you, and maybe this is the biggest showstopper.

You cannot really do much about that if you have been trained if you’ve been working for years in big corporations, maybe that you don’t fit straight in at a sloppy culture where things happen very fast, and people make jokes about things that you may not even understand. That is a problem, and there is no way around it unless you spend months really trying to enter that culture. I particularly love that to find a new culture and see how you can understand it better that is not only countries or big cultures like that, but it could be just the culture of people who love a board game that you’ve never played. I don’t play both games; I don’t know anything about this. They have a culture, right?

Can I understand their culture can I understand why they like what they like? So, this is a wonderful thing for me to do and to learn;it’s valuable anywhere you go. All this is fine, but there’s one more thing that makes a big difference, and that is “Business Value.” This is super important, and there is no way that you can tell me, ‘oh I’m more serious, I’m older now, and I don’t know how to evaluate business value better than when I was younger’ I don’t believe you. If anything, now that you’ve been around, you’ve seen projects go up and down. You’ve seen companies going to market and failing and coming back. You’ve seen that this is very hard to communicate.

We try to teach this at Data Science Retreat. It’s not easy to get people to produce projects that have business value or value for any definition of value. This is a huge added advantage that you have; you can see it when it pops in front of you, you say ‘yep this is business value.’ This happens only when you have the experience; when you have been around the block. I think this is a very important skill that it’s totally underestimated. There are too many people doing things in data science that don’t necessarily translate to value, and the market is taking notice. So, we don’t want to be known with Data scientists as the people who never care about value and produce technically sophisticated projects but don’t add business value.

That’s it for this session, see you in the next topic where we talk about creating or finding value as a Data Scientist.

How to pick a successful AI project, part 1: Finding the problem and collecting data

This post is part of a series on ‘how to pick a successful AI project’. Part 2, Part 3.

Imagine that you have 200 hrs of your life to improve your career prospects in ML. What is the best use of this time? I’m betting on doing a portfolio project.

Let’s compare different options:

1. You could go to meetups

2. You could do tutorials, follow template code and be the Github user # 2245 who has this same exercise

3. You could take classes or MOOCs

Option 1 relies on serendipity. Assume you meet a potential job opportunity every 5 meetups (which could be a bit optimistic!) You may well spend a year going to meetups and getting exactly zero job opportunities if:

  • you are not looking amazing on paper/online, and
  • you don’t communicate very well in person

Assuming commuting to and attending the meetup takes 5hrs, How many meetups can you do? 40. That’s 20 conversations worth having. Still, without some strong signal that you can perform (such as work experience, or a tangible ML project you built from scratch), you will not convert these random conversations into job offers.

Options 2 and 3 (tutorials and MOOCs) don’t differentiate you enough from the masses. They are a prerequisite to reaching the level that would make a hiring manager take notice. Because work experience is hard to get when you are trying to get into a new field (chicken and egg problem) that leaves you with my preferred option, and the reason I wrote this series of posts: show the world what you can do by having an AI portfolio project.

A substantial project strongly dominates the other options

Once I asked Ted Dunning: “What is the one thing you care about as an interviewer?”. His answer: “I only care about one thing: What you have done when nobody told you what to do.”

Having a portfolio of projects shows creativity, which is extremely important in data problems.

Data cleaning, even model fitting, is somewhat grunt work. We will likely automate this in the future.

What is not automatable (and where you want to excel) is at:

• finding a problem

• That is worth solving (produces business value; helps someone)

• and is solvable with current tech

Ok, so you want to do a good AI project

The rest of this post is what I’ve learned after mentoring more than 150 AI projects over five years at the companies I’ve founded.

I’ve grouped what I know into four classes:

• Finding the problem

• Collecting data

• Working with data

• working with models

Finding the Problem

What human task is your project helping or displacing? If the task’s decisions take longer than a second for a human, pick another one

Machines are good at helping humans with boring tasks. The rule of thumb (from Andrew Ng) is that you want to tackle tasks that take a human less than one second. Driving, for example, is a long series of sub-second decisions. There’s little deep thinking there. Writing a novel is not like that. While NLP progress may look like we are getting closer and closer, writing a novel with AI is not a good project.

Myth: you need a lot of data. you need to be google to get value out of machine learning

When looking for problems, you are looking for data too. Often you feel there’s no data to address the problem you found. Or not enough data. Data is valuable, and those who have data don’t make them public. There’s plenty of public data, but nothing that matches what you need.

You come up with hacks, of course. Maybe you can combine different sources? Maybe you generate and label the data yourself?

Then you read online that you need giant datasets. You are screwed. You are not going to label millions of images just by yourself.

It’s true that for very complex deep learning models, you need those giant datasets. An excellent boost for ML in the 21st century is that now we have lots more data, and compute power, to train more complex models.

Many of the revolutionary results in computer vision and NLP come from having a lot of data.

Does this mean that you cannot do productive machine learning if you have no big compute clusters or datasets in the Terabyte range? Of course not.

Libraries contain pretrained models (like Inception V3, ResNet, AlexNet). You can use then to save computation time.

Take, for example, YOLO (you only look once). Anyone with an off the shelf GPU can do real-time object detection and segmentation in video. That is amazing. How many projects can you conceive with this feature? Make it an exercise: list 10 projects that you could build based on YOLO. By the time you read this, it’s likely there’s a new algorithm that does the task better. It’s a beautiful time to be alive. You don’t need terabyte-sized datasets to get value off of machine learning.

Pick a problem that passes ‘the eyebrow test’

The skill of picking a problem is as useful as the skill to solve it. Some problems will capture your imagination, and some will leave you unmoved. Pick the first.

And then there are those projects that make people think that what you are saying is impossible. You want one of those.

How do you know you picked the right problem? Use ‘the eyebrow test.’ You want to see the eyebrows of the other person going up when they hear it. If they don’t, keep looking.

Some problems are mere curiosities and can suck your resources. Don’t go for those. There’s tremendous potential for impact using ML right now. Why spend your time creating a translation algorithm from Klingon to Dothraki (both invented languages) when you can make people’s lives easier?

At Data Science Retreat we believe that because there are plenty of ‘good problems,’ there’s no excuse to pick one that doesn’t pass the eyebrow test. It takes time to find a good problem. How long should you spend? As much as needed to pass the eyebrow test.

It helps if you are in the surroundings of people who have found a good problem before. Sometimes, those with deep domain knowledge in ML are not as useful as you may think to pick up ideas.

You are selling the value of AI. When you do ‘eyebrow test’ projects, you help not only the people-who-have-that-problem (always keep them in mind) but… your fellow data scientists. The more impressive your projects are, the more the general public will appreciate how transformative AI is for society. And the more opportunities you create for everyone else in the field.

You only need to be in the ballpark of a good idea

One participant came to me saying that he was running late to pick up an idea and that he had exhausted all his possibilities. He had nothing.

“Ok, let’s look at what you are passionate about. What’s alive in you?”

“Well,… I hate waste.”

So we googled ‘waste trash machine learning.’ Nothing too obvious, nor too exciting, came about. After some more search, we found a Stanford project that categorized trash. The students had a trash dataset, painfully labeled. Still, this was not exactly an idea that passes the ‘eyebrow test.’

“What if instead of categorizing trash, we could build something that picks up trash?” (iterating on the idea)

“You mean something that goes around autonomously?”

“Yes, a self-driving toy car. There was one project in batch 08 of DSR that did exactly that. The car ran laps on a circuit. You would have to modify it to identify trash, get close to it, and pick it.”

“That sounds amazing!” (eyebrow test passed)

With time, this ballpark idea morphed from general trash to picking up cigarette butts, which are terrible for the environment. Details about how to pick the cigarette butts improved with time: stabbing them, instead of trying to grab them with a robotic arm. We will talk more about this project later in this series.

Pick a problem that moves you. If nothing does, use ‘watering holes’ to listen to problems that move people

If you have the problem you want to tackle, you have a significant advantage. You understand the need. You can build the solution and know how well it works by applying it to yourself. You can tell between ‘nice to have’ and ‘pain point.’ At this point, what we are doing is no different from what startup founders and product managers do.

You have in your hands the closest thing to a shortcut: have the problem yourself. You will save time but not going into detours that don’t help. You will have a keen sense of what to build.

In 2014 I was building a company that eventually failed doing customer lifetime value (CLV) predictions for e-commerce stores. The product CLV predictions, though, was something I could deliver myself as a consultant. So I became a ‘CLV consultant.’ One of my clients was hiring a data scientist, and they hired me fulltime, so I magically transitioned into data science.

Many others had the same problem: they had tech skills, maybe a Ph.D., and they wanted to become data scientists, but they didn’t know how. Remember, this was 2014, before the web started boiling with advice on how to become a data scientist. I built a business around helping others solving this problem, and it’s been working well for the last five years.

I knew well what the problem is; too much information, unclear guidelines, interviewers who don’t know how to recognize talent. Every step of the way, I felt I knew what I was doing with this business; a feeling that is extremely valuable. Transitioning to data science is an excellent problem; 5 years later, people still struggle with it.

I don’t play golf. If I wanted to build a product for golfers, I’d be lost entirely. I would build features nobody need; I would miss the pain points. Even if I ran interviews and listened to the market, I would at a disadvantage to a golfer.

So my advice to pick a portfolio project: pick a problem you know well. Even better if it’s a problem that moves you. If you lost three friends to suicide, build something to prevent suicide. In this case, you don’t have the problem yourself, but you have a strong motivation to solve it.

What if you have no problems whatsoever? You have been in the same industry forever (say Oil and Gas), and all valuable problems there got taken care of!

I don’t believe you. There’s no industry so mature that all problems are solved. But, ok, you cannot come up with something that moves you, some problem that you have.

Then observe what problems other people have. Large groups of people. They tend to congregate in public spaces and bitch about their problems; every time you see people bitching about something… turn that into an opportunity to do a project.

Which public spaces? I call these ‘watering holes’ (HT to Amy Hoy). Online, you can have obscure forums, but Reddit and twitter are the easiest. Just sit there and ‘listen’ to people discuss the problem. Learn every detail of it. Is it a real problem, or a ‘nice to have’?

For example, you may join a gaming subreddit to see if gamers care about having stronger AI in videogames. Or if they care for VR. These ideas are too abstract to be good project ideas, but you get where I’m going.

Collecting data

Integrate distinct data sources

Often companies are so focused on getting value out of the data they have they forget they can increase value using the data not in their company but publicly available.

There’s plenty of open access data. And APIs for data that changes frequently. There’s no reason not to use multiple data sources. You can solve a more interesting problem (one that was not obvious before) by integrating APIs.

For a collection of APIs, check https://www.programmableweb.com/.

Problems that seemed impossible with a single source of data become solvable when you add a new data source. Boring projects come alive. Thankless tasks become a joy to work with if you manage to find a twist that shows more value.

When using multiple data sources, you have to stitch them together using a shared key (a column that is present in both datasets.) You cannot combine data sources that don’t have a shared key, and that tends to be a showstopper for many ideas.

Instead of collecting or reusing data, produce your own data

You don’t need to find data, or own it. Thanks to pretrained models (see later section on them), you don’t need all that much data, which means you can produce it yourself. Removing “I don’t much/any data” as an obstacle opens up the space of problems you want to solve.

To produce original data, I found one big hack: use hardware. Sensors are cheap, and they give you data that you own.

JD.com’s Shanghai fulfillment center uses automated warehouse robotics to organize, pick, and ship 200k orders per day. 4 human workers tend the facility. JD.com grew its warehouse count and surface area 45% YoY. AI affects manufacturing too. It’s easier than ever to produce things en masse, and this means there are a lot more hardware ‘toys’ in the market. Things that you would never consider to be affordable like microscopes, spectroscopes are reaching the mass consumer market and are eminently hackable. These are wonderful data sources!

Because of Shenzhen, Kickstarter, etc. hardware is evolving far faster than before. It’s never going to be as fast to iterate on as software, but we are getting there. Have you checked what’s available on Ali express? There are multiple sensors you can buy for under 100 bucks. Attach one of these to a phone running your code, and you have a portable purpose-specific machine.

For example, you can buy a microscope and use it to detect Malaria without a human doctor. Deep learning running on a phone is good enough to count parasites on blood.

Ali express is full of cheap hardware that you can attach to a phone. You can add a lot of value to someone’s life using a mixture of phones (that run the ML code) and

Example: our Malaria Microscope.

Eduardo, AIscope’s founder, after reaching 1000x the first time

Malaria kills about 400k people per year, mostly children. It’s curable, but detecting it is not trivial, and it happens in parts of the world where hospitals and doctors are not very accessible. Malaria parasites are quite big, and a simple microscope can show them; the standard diagnostic method involves a doctor counting them.

It turns out you can attach a USB microscope to a mobile phone, and run DL code on the phone that counts parasites with accuracy comparable to a human. Eduardo Peire, DSR alumni, started with a pubic Malaria dataset. While small, this was enough to demonstrate value to people, and his crowdfunding campaign got him enough funds to fly to the Amazon and collect more samples. You can follow their progress here: http://aiscope.net.

For another example, you can buy a spectroscope that pointed at any material, and it’d tell you its composition. It’s small enough to attach to a phone for a hand-held scanner. Can you detect traces of peanuts in food? Yes! There you go, a solution to a problem real people have. If you are allergic to peanuts, this will buy you a certain quality of life.

Sensors are cheap nowadays, and they will help you get unique data. You can turn a phone into a microscope, a spectroscope, or any other tool. The built-in camera and accelerometer are excellent sources of data too.

Next on this series: what I’ve learned about working with data and how this can help you pick successful AI projects.

How to pick a machine learning project that would get you hired

I’ve mentored about 165 projects during the last five years at the companies where I work. The people who built those project all got jobs (mostly). The more I talk to companies interviewing today, the more apparent it is: A portfolio project is decisive when making hiring judgments. Jeremy Howard recommends it. Andrew Ng recommends it. Why? It’s far better at discriminating talent than any other proxy (CVs don’t work; pedigree doesn’t work; puzzles don’t work).

And yet, the proportion of people who finish MOOCs AND have a portfolio project is tiny. What’s the catch?

Well, it could be that these are orthogonal skills. Finishing something as demanding as a deep learning MOOC (or a uni degree) show that you have grit. You can start and finish difficult, demanding projects. But what hiring managers want to know is: “What did you do when nobody told you what to do?”, “Can you find ways to generate value (that we may have overlooked)?”, “Can you work with other people?”. Many people can prepare for, and pass, the ‘typical developer interview.’ That one with the whiteboard, and the typical questions that you never encounter in real life. However, what everyone, on both sides, hopes for is to meet someone who can work well beyond ‘the standard.’ What separates you from the rest is a substantial portfolio project. Not that many people can display the creativity and grit to get a killer portfolio project off the ground.

Because this post is about how to pick an ML project, you have to make a decision: you need to focus on machine learning (ML) or on data management to start your career. The second option is plenty lucrative and doesn’t require you to know much ML. But then you will be stuck with the dreaded ‘data cleaning’ (A data scientist spends 80% of her time cleaning data, and 20% of her time bitching about cleaning data). You may realize that after working extra-hard to become competent in machine learning, you are not doing all that much of it once you get a job.

There’s a trick to avoid too much data cleaning: go to a company that has a good team of data engineers. These are rare, though. So it’s safe to assume that in a ‘normal’ company you will have to do some data cleaning, even if you want to stay away from it. I have more to say about companies; the last bit of this post will cover how to choose which companies to apply for. But now let’s focus on the project.

How do you pick a good project? (Proof Of Concept)

Morten T. Hansen’s book ‘great at work’ shows that exceptional performers tend to ‘Do less, then obsess.’ He shows pretty convincing empirical evidence. So I’m going to recommend: Do one project, not many that are half-assed. If the hiring manager looks at your GitHub, there must be a clear ‘wow’ story there. Something she still remembers after an entire day looking at resumes.

You want to have something to show that is a Proof of Concept (POC). While you will not have resources to build an entire product, your GitHub should have something that demonstrates that such product is feasible. The interviewer must think, ‘ok, if he came to my company and did something like this, he would be adding serious value.’

Good problem

Picking the right problem is a great skill, and it takes years of practice to do it reliably. It’s a mixture of ‘Fermi-style’ estimations (‘how many piano tuners there are in a city?’) time management, and general technical know-how.

The good news is that in ML there’s lots of low hanging fruit. There’s lots of code already, so many pretrained networks, .and so few solved-with-ML problems! No matter how many papers per year we get (too many to track even in a reduced domain), there’s plenty of opportunity in ML right now. Applications of technology this complex have never been more accessible! If you want to practice this skill, tweet 2-3 project-worthy problems a day, and ask people for feedback. If you tweet them at me (@quesada), I’ll do my best to rate them with a boolean for plausibility (in case lots of people take up this suggestion!)

We data scientists have the luxury to pick problems. And you should practice that skill! Knowing what is possible today is the first step. Twitter is a great place to pick up what’s happening in ML. You must have heard of most of the breakthroughs, and have an internal ranking of methods and applications. For example, everybody seems to have watched David Ha’s ‘Everybody dances’ demo. If you didn’t, stop what you are doing and watch it now. Your immediate reaction should be: ‘how can I apply this in a way that generates business value?’

Over a year, at least a couple of dozen papers make a big splash in the scene. Your job is to keep a catalog and match it to business problems. Medium has several writers that are amazing at implementing the latest algos fast. If you want the most exhaustive collection of implementations, try https://paperswithcode.com. To navigate arXiV, try http://www.arxiv-sanity.com (this is good to pick up trends, I don’t recommend you to make paper reading a priority if you want to be a practitioner.) About videos: https://nips.cc has now videos for most talks. ‘Processing’ NeurIPS is a serious job, so it’s easier to read summaries from people soon after they attended. Remember, you are doing this to pick up “what’s possible,” not to become a learned scholar. It’s tempting to get lost in the rabbit hole of awesomeness.

Which problem you pick says a lot about your maturity as a data scientist and your creativity. You are demonstrating your taste and your business acumen. Passion projects are ok if your passion is not too obscure (for example, I wouldn’t work on detecting fake Cohiba cigars, no matter how passionate you are about those). If knowing the domain can give you an unfair advantage, by all means, use it. A good idea gets people saying ‘I would use that’ or ‘How can I help?’ If you don’t get these reactions, keep looking.

Deep learning (DL) has two good advantages over ‘classical’ machine learning: 1/ it can use unstructured data (images, audio, text) which is both flashier and more abundant than tabular data, and 2/ You can use pretrained networks! If there’s a pretrained network that does something remotely similar to what you want to do, you have just saved yourself a lot of time. Because of transfer learning and lots of pretrained networks, demonstrating value should be easy. Note that taking what you build to production would be problematic if you depend on GPUs, but this is beyond the scope of your proof of concept. Just be prepared to defend your choices in an interview.

If you don’t want to do DL, there’s another useful trick: pick a topic companies care about. For example, If you want to impress companies in the EU, you can do a POC that uses the company’s proprietary datasets while protecting privacy. Impossible, until recently: Andrew Trask is building a great framework to train models in data you cannot access at once. His book ‘Grokking deep learning’ (great stuff!) has an entire chapter on privacy, secure aggregation and homomorphic encryption. The keyword you want is ‘federated learning.’ Google is doing quite a bit of research about it because it could save the company from a frontal attach coming from regulators.

How long should you work on a ‘good problem’? At least 1.5 months. This length seems to work fine at the companies I work for. Anything shorter can look like a ‘weekend project’; you want to avoid that at all costs.

Original

Rather than having one of the popular datasets, for example, ‘mammograms,’ or worse ‘Titanic,’ pick some original data and problem. Remember you are trying to impress a hiring manager that sees hundreds of CVs and many have the same old datasets. For example, MOOCs and bootcamps that ask everybody to do a project on the same topic produce reams of CVs that all look the same. Now imagine you are the hiring manager, and you have seen dozens in the last month; will you be impressed?

You are going to do all that alone (or in a small team of two). In a company, accepting a project is something that goes through a more substantial group:

  • Product defines the problem and creates a user story that describes the need.
  • The research team reads relevant articles and looks for similar problems which other data science teams have faced.
  • The data team collects relevant data, and someone (internal or external) needs to label it.
  • The team tries some algorithms and approaches and returns with some kind of baseline.

Because you are working on a project to showcase your skills, you would have to do this without a large team. But is it impossible? No. Here are some examples:

Malaria Microscope

Malaria kills about 400k people per year, mostly children. It’s curable, but detecting it is not trivial, and it happens in parts of the world where hospitals and doctors are not very accessible. Malaria parasites are quite big and a simple microscope can show them; the standard diagnostic method involves a doctor counting them.

It turns out you can attach a USB microscope to a mobile phone, and run DL code on the phone that counts parasites with accuracy comparable to a human. Eduardo Peire, DSR alumni, started with a pubic Malaria dataset. While small, this was enough to demonstrate value to people, and his crowdfunding campaign got him enough funds to fly to the Amazon and collect more samples. This is a story for another day; you can follow their progress here: http://aiscope.net.

Wheelchair helper

Information about pedestrian road surface type and its roughness are essential for vehicles that need smoothness to operate. Especially for wheelchair users, scooters, or bicycle riders, road surface roughness significantly affects the comfortability of the riding. A large number of street images are available on the web, but for the pedestrian road, they typically do not have roughness information or surface labels.

Masanori Kanazu and Dmitry Efimenko classify road surface type and predict roughness value using only a smartphone camera image. They mount a smartphone to a wheeled device and record video and accelerometer data simultaneously. From the accelerometer data, they calculate a roughness index and train a classifier to predict the roughness of the road and surface type.

Their system can be utilized to annotate crowdsourcing street maps like OpenStreetMap with roughness values and surface types. Also, their approach demonstrates that one can map the road surface with an off-the-shelf device such as a smartphone camera and accelerometer.

These two projects are original. They demonstrate what a small team can do with modern DL and … well, cheap sensors that already come with a phone or you can buy online. If I were a hiring manager, these projects would make me pay attention.

Relevant

A project can be original, but useless. Example; translate Klingon to Dothraki language (Languages used on TV series that don’t exist in reality). You may laugh, but after > 165 projects I’ve mentored, I’ve seen people generating ideas that fail to meet the relevant criterion.

How do you know what is relevant?

If you are part of an industry, you should know of pain points. What is something trivial that looks automatable? What is something everyone in your sector loves to complain about?

Imagine you are not part of any industry; If you have a hobby, you can do a portfolio project on it IF:
1/ It’s damn interesting and memorable even if way outside the range of interest of the industry. Example: “recommender system for draft beer.” Many people will pay attention to beer.

2/ It showcases your skills: You found or created an exceptionally good dataset. You integrated multiple data sources. You made something that on first thought, most people wouldn’t believe it’s possible. You showed serious technical chops.

Good data

You may complain that there’s not that much data available out there, or that the massive, nice datasets are all inside companies.

The good news is that you don’t need big data to deliver value with ML! No matter how much hype there’s around the importance of large datasets. Take it from Andrew Ng:

Ok, so where do you get data?
Generate it yourself (see previous examples; they both generated their own data; or begged anyone who had data to contribute it (hospitals that deal with infectious diseases for the Malaria microscope).

Some hiring managers may not care much about how you fitted which model; they have seen it before. But if the dataset you generated or found is new to them, that may impress them.

Clean code

Have a clean project on GitHub, something the interviewer can run on his computer. If you are lucky enough to be dealing with an interviewer who checks code, the last thing you want is to disappoint her with something that is hard to run or doesn’t run at all.

Most people have ‘academic quality’ code on their GitHubs, i.e., something that works just enough to get you a plot. It’s damn hard to replicate, even putting in some effort (something a hiring manager will not do).

If you have clean code, it will feel refreshing. A good readme, appropriate comments, modules in python, good commit messages, a requirements.txt file, and … a simple way to replicate your results, such as ‘python train.py.’ That’s all it takes to be above 90% of the code most people learning data science have on their GitHub. If you can defend your library or model choices in an interview, that’s moving you up the ranking too.

Apply Now

Do not obsess with performance

I call this the “Kaggle Mirage.” You have seen Kagglers making real money and raking in the prestige just by moving a decimal point. Then you may think “I need to get to 90% accuracy because my manager (not a data scientist) set it as a goal.” Don’t let that happen to you without asking some important questions. What would happen if you only got to 85%? Are there nonalgorithmic ways to compensate for the missing 5%? What is the baseline? What is the algorithm? What is the current accuracy? Is 90% a reasonable goal? Is it worth the effort if it’s significantly harder than getting to 85%?

In all but a few real projects, moving performance a decimal point may not bring that much business value.

Now, if you have been working on a problem for a week and performance didn’t change… is it time to stop? Maybe your peers on a standup notice the pattern and are encouraging you to drop that project and join another one (coincidentally, theirs). Expectation management for data scientists would be a great post.

In data science, progress is not linear. That is you could spend days working on something that doesn’t move the needle, and then, one day, ‘boom,’ you have gained 5 points in accuracy. Companies not used to work with data science will have a hard time to understand that.

I recommend not to do Kaggle (or not only!) if your goal is to build a portfolio project to get hired.

Deciding whether to apply and preparing for an interview

Even if you do have a great project to show off, you still have to (1) find good companies to apply to, and (2) prepare. I estimate about 10% of companies are a slam dunk: They have a team that writes blog posts, they are using reasonable algos for their problems, they have a good team already, and AI is a core component of their product. Startups with funding and big N companies land in this box. For the remaining 90% of companies, it’s far harder to estimate whether they are a good match for someone building an AI career. I have quite a few heuristics, but this is worth a different post.

What doesn’t discriminate excellent and bad AI-career-building companies? The job ad.

It is difficult to write up a job description for a data scientist role; more so if there isn’t already a DS team in the company to write it! Even when there is one, HR could have mangled the job ad by adding criteria.

As a candidate, you can use this rule of thumb: ignore requirements and apply; If you only have 50% of the bullet points, that might be better than most other applicants. Knowing this would add some companies to your list, ones in which you could have disqualified yourself.

When companies don’t have an easy way to tell if a candidate is any good, they revert to ‘correlates’: pedigree, paper certifications, titles. Terrible choice, because so many great data scientists today are self-taught. Those companies asking for a Ph.D. or Masters are missing out on great talent that cut their teeth on fast.ai and Kaggle.

Before you start carpet bombing LinkedIn, … Have you prepared for interviews? Because it’s not a walk in the park! Don’t try to go interviewing without preparing; your competitors will.

The standard software developer interview in the Valley takes about three months of preparation. If you don’t believe me, go to this Reddit. That goes equally for fresh CS graduates (who should still remember all those ‘algorithms and data structures’ classes) and for more senior people (who have all the rights in the world to have forgotten them).

In less competitive places, it may not be that important to know all these CS tricks. In fact, preparing for a DS and CS interview at the same time may be overkill. Some positions (machine learning engineer) may require you to be a senior engineer on top of knowing machine learning.

For data science positions today, you don’t necessarily need to prepare ‘algorithms and data structures’ questions. But the trend is that companies ask for more and more engineering from their data scientists. You can know in advance by checking the profiles of company data scientists in LinkedIn. Are they very engineer-y? Then get the book ‘cracking the coding interview,’ and do a few exercises.

Summary

Work on one project. Spend 1.5 months on it. Make sure it’s impressive to a hiring manager that sees hundreds of CVs for each position. Make sure you pick a ‘good problem’, one that is, original, relevant, and uses good data.

Make sure you produce clean code, pick decent companies before you mass-apply, and prepare for interviews.

Apply Now

Training your empathy as a cure for cofounder problems, the #1 statup killer

I’m Jose; I run Data Science Retreat and AI Deep Dive, a 3-month school that takes pretty advanced machine learners and gets them to work together with mentors to produce a killer deep learning portfolio project. We want these projects to have social impact (example: a malaria microscope).

I’ve been doing startups for 10 years. I’m a solo founder now, and successful (bootstrapped two companies). I was miserable when I cofounded. I’ve experienced my share of hard conversations, cofounders leaving, jealousy, bickering for equity, etc. I’m happy to risk a drop in happiness cofounding again because I want to do the most good I can, and cofounders are the surest way of scaling impact.

I’m still scared of having cofounders, a single point of failure in your life. Or I was! I think I’ve found a hack that makes me feel confident that I will deal with any disagreement in a non-stressful way.

In this post, I’ll tell you about ‘Non-Violent communication’ (NVC), a technique that makes you better at ’empathy.’ This is what I did, warts and all. It may, or may not work for you. It’s not a surefire solution, but it is a beautiful hack, and I’m ecstatic to see if it works for anyone else.

The hack

How could I prevent any stress that comes from human interactions in a startup? It feels like an impossible task. But I’ve been here before (we all have!): a task that feels impossible blown wide open by a simple, elegant solution. The true definition of a hack. Last time I can remember finding a hack this powerful was when I was suffering from RSI. I thought my career in front of a keyboard was over. A horrible thought. Nothing standard medicine knew worked. Then, I found a solution (that may deserve a different post, ‘how a tennis ball in a socket saved my career’).

This book: The Trigger Point Therapy Workbook (Clair and Amber Davis) was enough to help me understand the way out. There are trigger points, and you have the ability to massage yourself. You know better where it hurts. Boom. Solved problem.

When you find something that effective (and rare), you want to tell others. This is exactly why I’m writing now. I’ve found another hack that is as effective, solving a problem as large (or larger!). You can train your empathy, and have better relationships with everyone around you. Including difficult people. Including cofounders (non-overlapping sets, with any luck).

It’s all in a single book, Marshall Rosenberg’s ‘Non-Violent communication,’ aka NVC.

What is empathy?

Empathy is hard to define. For me, it’s about

1- Being present, in the moment

2- Deep curiosity for what’s going on in the other person’s life

Empathy is a language. The components are simple (like a grammar), but the implementation is not. It takes lots of practice to learn a new language, and this is also true for empathy, at least the NVC variety.

Empathy is not compassion. You can have compassion for a group of people on a different continent who are suffering. Empathy happens one on one, with a person in close proximity. This is my definition, and I’m not an expert. Fortunately, there’s a much better definition of this language that is extremely actionable: NVC. This is how it works.

Empathy, the NVC way

Rosenberg’s method is simple and elegant, like lisp. it has the beauty of internal consistency.

An empathic conversation should go through four steps: observation, feeling, need, and request. On this post, I’m going to use a simpler version without the request at the end.

Observation

Just observe what the other person is doing that is causing you trouble, and state it. Do not provide evaluation. Otherwise, it feels like a judgment. Avoid judging people at all costs. Judging and blaming has become second nature in our culture, and you can see how it’s extremely toxic. You get intuitively that judging is bad, yet we do it all the time.

Example:

‘I have noticed that you haven’t  been able to work very long the last two days because you went to a festival.’

When we judge others, we contribute to violence. Intellectual analysis is often received as criticism. Stop the urge of providing it. The ‘because’ part may be already too much.

Rosenberg has a really nice rule. Do not use ‘but.’ “Rather than putting your ‘but’ in the face of an angry person, empathize.”

Feeling

Express your feelings. This is hard; some cultures make it harder than needed. There’s only about a hundred emotion words, make sure you use one of these. Many sentences using ‘I feel’ are not about emotions at all. You can use simple words, like ‘sad,’ ‘angry,’ ‘disappointed.’ No need for poetry here.

Example:

‘I feel disappointed because I’m working harder than you.’

Make it clear to the other person how you feel, but don’t make her responsible for your feelings. Only you are.

Rosenberg says ‘speak your pain nakedly without blame.’ Nobody said NVC is easy.

That should be enough for this step; outside an NVC convo, take a guess on someone else’s feelings, and tell them to see if you are right. Many people work hard hiding their feelings and may feel exposed an uncomfortable talking about them. Others will feel elated and impressed that you nailed them. Learn to identify feelings from the stream of communications that goes around you.

Need

Express your need. What can is ‘alive’ in you that the other person could satisfy? Note that needs create emotions. The other person doesn’t create the feeling, you do (through your needs). The other person may not be aware of your needs if you never communicated them. Rosenberg says ‘If we express our needs, we have a better chance of getting them.’ So simple, so true, so uncommon.

Example:

“I need to feel reassured that we are both totally committed to the success of our company.”

Unlike feelings, needs come in all forms; there is no shortlist. This one is a need to feel something. it could also be something more mundane, like a need for cleanliness (together with a request that you do the dishes).

Most people never communicate their needs. Clearly, they ask you to infer them. This is ineffective, don’t be that person.

In the example, you were the one communicating a need. But you could be the listener. Stay in the moment, no matter how uncomfortable the silence may be, no matter how much the other person is struggling, till she has been heard.

You will ‘see’ it when they felt heard: there’s a body change, shoulder drop, maybe a sigh, the flow of words stops. The conversation tone changes. You feel closer.

Don’t worry about the last part, the request. If you manage to see that body change you are already on your way to an agreement. Just make sure that your request is not a demand, is actionable, and worded in positive terms (i.e., don’t use ‘not’).

The more we hear them, the more they’ll hear us.

‘Ok, but does it work at all? Does it work for my particular conflict, at least?’ you may be thinking.

The challenge

I read voraciously, so I consumed all of Rosenberg’s books. I tried it with my six-year-old son and my girlfriend, to different degrees of success.

Son: crying because his football cards felt and are out of order

Me: Are you crying because you have a need for order and the cards are on the floor?

Son: nods, feeling heard. Crying stops in a too-good-to-be-true way

One other story:

Me: (some request). To make sure you understood, could you repeat what I said to me? (part of the method)

Girlfriend: Are you running this NVC thing on me? (Angry)

You can see it didn’t really work on the second story. This was a bad experience I didn’t want to repeat. As much as I had read about it, I was not any good at NVC. I was still not any better at empathy.

People write a lot about how NVC in the hands of a newbie sounds robotic as hell as not at all like receiving empathy. This robotic delivery may backfire. There are variants, for example, ‘street NVC’ that make it more informal. But the point is that you need to be present and curious about the other person. No scaffolding can turn a dull, uninspiring conversation into an emphatic one. Don’t try to give empathy if you don’t really care about the other person.

The practice

So how do you get out of ‘robotic NVC’? Practice. The best way I’ve found is to get together with others who are also trying to get better. A meetup is ideal. In Berlin, where I live, there was none, so I created one. Meeting at the DSR office, which makes for fascinating contrast (people door to door discussing hardcore algorithms for deep learning vs. people practicing empathy).

There are lots of meetups all over, but if there’s none where you live, I suggest you start one. It was extremely beneficial for me, even on my tight schedule.

NVC works excellent in ‘pair programming’ mode: one person is the one who receives empathy, one gives empathy (the one with hands in the keyboard so to say), and one observes and gives hints. Then you rotate.

It’s not impossible to practice NVC while alone. Self-empathy is a thing. We are terribly judgmental with ourselves (self-criticisms). For example, try to avoid the word ‘should’ (to yourself or two others). This word has enormous power to create blame. “Don’t do anything that isn’t play” says Rosenberg.

What else can you practice alone?

1- Observation. Practice translating judgment into something neutral.

2- Connect your feeling with your need: “I feel… because I need …”

3- Saying thank you in NVC: “This is what you did; this is how I felt; this is the need of mine that was felt.” Compliments are not great. They are actually judgments, positive ones but judgments. It’s much better to know how you enriched the life of the person praising you so that you can do more of it.

4 – Monitor how you talk, stop yourself if you are using judgments, ‘buts’ or ‘shoulds.’ “From the moment people begin talking about what they need rather than what’s wrong with one another, the possibility of finding ways to meet everybody’s needs is greatly increased,” says Rosenberg.

So, to answer the ‘does it work’ question… I’m a total beginner. I can tell you they are not kidding when they say NVC is like a language.

I need to practice more.

Have I experienced any benefits? I became Vegan around the same time I started learning NVC. That may mean I’m improving my health. I have a much deeper appreciation of who my outstanding girlfriend is; I ‘see’ her in ways that I didn’t before. I truly love this woman. I became involved with the ‘effective altruism’ movement. I constantly think about what is violent and how we can remove violence from our communication. I come up with better ideas to use deep learning for social impact. I feel more connected to the world around me. I meet more people by chance that are in the same mental space (the DSR scholarships we ran are an example).

But to the point: do I feel capable of starting something with cofounders? Yes, for the first time in many years. I’m confident any difficult conversation will be solved, even with my primitive empathy training.

NVC feels as effective as that tennis ball in a socket that saved my career once.

Getting your first job as a Deep Learning engineer: The current state

This post should give you an insider view of how it feels to be in the market for a deep learning engineer job. I have interviewed thousands of people in machine learning in the last five years; for deep learning, only a few dozens in the last year; I’ve been paying attention to the market, who goes where, salaries etc. It’s enough for me to form an impression.

How to pick your first job or company: intelligence compounds

If you go and take a relaxed, well paid, 9–5 early in your career, you are getting paid to forgo the growth in your capabilities. Ok, that’s fine, many people take that deal. More so in Germany, the number one country in the risk aversion scale (well, it would be off the scale if there was one). You may have better things to do with your time than just getting smarter.

But then, if intelligence compounds (as in ‘compound interest’)… Your choice has a dramatic effect mid or long term. Your employer should pay you a lot more for that opportunity that you are forgoing. If you take a job at say Deutsche Bank (I could not think of a shittier company) and stay say seven years you are virtually accepting you are never going to be competitive, even if you may have been at the beginning of these seven years.

Someone who used these seven years in situations with high growth opportunities will be very well positioned after that to solve really significant problems.

intelligence compounds, it’s essential to pick the right starting job in your career. Or the right startup to found if you are planning your life as an entrepreneur.

Because we spend so much of our time at work, one of the most powerful leverage points for increasing our learning rate is our choice of a work environment. How can we translate this observation into tactics?

Optimize for growth

Everyone who is any good at anything optimized for growth early in their career. Sometimes without knowing.

Small, but constant, daily growth leads to dramatic changes long term. The good news is that you don’t have to think about the long-term plan. Just make sure that today was not wasted, and that you are learning at say a 1% rate per week. That is enough. It’s vital that you measure your improvement. Track the time it takes you to finish simple programming tasks. Then plot progress every year or so. Measuring progress doesn’t work well for creative things like designing architecture or debugging some difficult bug. But it’s worth trying. You can also ask others for honest feedback. Am I getting better at this? Often people won’t tell you to your face when you are getting ‘kinda good’, but they talk to each other. It’s only by chance that you realize you are growing a reputation.

Culture

A work environment that iterates quickly provides a faster feedback cycle and enables you to learn at a quicker rate. Lengthy release cycles, formalized product approvals, and indecisive leadership slow down iteration speed; automation tools, lightweight approval processes, and a willingness to experiment accelerate progress. Anything that works at ‘Deutsche bank speed’ is to be avoided.

Do people help each other to get better? Or is the culture one of ‘one-up-manship’? Do the guys that have mastery share what they know? Are there others in your team with complementary skillsets?

People

Empathy is a skill like any other; it improves with practice. Reading fiction improves empathy. But the best book I can think of is Marshall Rosenberg’s Nonviolent Communication (NVC). If you can speak NVC you will be a fantastic manager; probably a tremendous father, partner, anything. Bradford Cross recommends it, and I can see why. I’ve read it, and I’m extremely impressed by Rosenberg’s ingenuity. It does take practice to get it to work. Another technocrat that recommends improving empathy is Chad Fowler.

Bosses: make sure you have a good one, and that you understand what each other needs. People leave bosses, not companies. If you have a bad feeling during interviews, just say no. This single person can have an outsized effect on your growth and happiness.

Toxicity

Do good ideas get killed in front of everyone by management?

Do bad ideas get the green light by sheer politics?

Are you treated with respect when you deliver results? Do people admire that you are effective?

Is incompetence tolerated? If so, run away. Having the freedom to fuck up and not been crucified is fantastic, but there are different types of fuck ups. Trying innovative things is bound to produce plenty of failures. But executing at world class level and failing is one thing. Failing to perform because of sheer incompetence: No. Everyone in the company should be able to tell the difference. The problem with tolerating incompetence is that it deters good people from doing their best. In fact, very likely they will leave. So finding the incompetent and ousting him or her should be everyone’s priority.

One exception to this rule: polymaths, or well-balanced generalists. They are not world-class at any one single skill. A specialist may consider their competence not up to scratch. Generalists abound in early-stage companies. In fact, without them, there would be no startups, as everyone tends to wear many hats. The guy who can code, sell, and do strategy is worth his weight in gold even if you wince when you see his commit messages. Eventually, generalists get replaced as the company grows. This is painful for the generalist, so be gentle.

People who can think strategically (e.g., find the intersection of which products can be built with current tech, and what the market is willing to pay for) are enormously valuable. More so in areas like machine learning, where everyone is trying to figure out what is plausible. If your company culture seems to be toxic for these profiles, it’s time to run.