When Tracking Data Distorts Behaviors

And a bonus photo of me as a teenager

I was born in New York City in the 1980’s. At that time, New York was crime-ridden and unsafe. Rolling Stone magazine described Times Square as the “sleaziest block in America.” And nothing was sketchier than taking the subway.

By the time I was in high school, things changed dramatically. Times Square became a tourist hub where families traveled via subway to see performances of the Lion King.

Picture of a picture of me as a teenager in Queens, getting ready to jump in my mom’s minivan to go to high school. (She dropped me off and I took the public bus home.)

So what happened in the early 90s to change NYC? One popular explanation is the “broken windows” theory, where the NYPD pursued petty criminals, such as turnstile hoppers, under the assumption that if you addressed the petty crimes, then everything else would follow.

My friend and former colleague James Underhill recommended a podcast that provided a different perspective from the broken windows theory. The podcast tells the story of Jack Maple. Jack was a transit cop. Now, I mentioned I grew up in NYC. I also grew up around a lot of cops. My grandfather was a police sergeant and my uncle was a homicide detective. Two of my cousins are (now retired) cops too. The transit police are not at the top of the pecking order within the NYPD. And when Jack was in the NYPD, the crime in the subway was terrifying.

Jack took a very simple approach to tackling this problem. He created a giant map of the city, and he started using color-coded dots to track crimes on the subway to identify patterns. Through this approach, he was able to pinpoint the places where criminals were more active, and deploy cops there. According to this podcast, he believed in the 80/20 rule: that 80% of the crime was generated by 20% of the criminals.

This approach worked so well that Jack was promoted to deputy commissioner. As deputy commissioner, he instituted something revolutionary, a system to capture and digitize crime stats all around the city, called COMPSTAT. Every week, he would bring the leaders of each precinct downtown to review their metrics and ask them what they were doing to improve crime in their respective neighborhoods.

When I heard that, my first thought was, that sounds a lot like a QBR! He was:

• Using data to make the invisible visible

• Inspecting leaders

• Holding people accountable to results

And what happened? Crime went down across the city. Jack was invited to introduce COMPSTAT to police departments all over the country. And we all lived happily ever after.

Not quite. Over the years since COMPSTAT was introduced in NYC, crime just keeps going down. Statistically. But is crime truly going down? The COMPSTAT system had some side effects. NYPD started measuring not just crime, but activity. Over time, quotas for activities — like arrests — were introduced. Jack Maple wanted to find the outlier criminals and eliminate them, but this approach became distorted to focus on activity and arrests.

Not only that, but the measurement began to impact the inputs and the process for collecting crime data. Police started downgrading crimes because they didn’t want their stats to look bad. “Assault becomes harassment, robbery becomes grand larceny, grand larceny becomes petit larceny, burglary becomes criminal trespass.” (NYTimes)

Why am I writing about the NYPD? Because there are many parallels in RevOps. We create processes, like logging and categorizing meetings or tracking deals in a CRM. We aggregate data about those processes, like the number of meetings and the size of opportunities. Then, we derive insight from that data to figure out how to improve the processes.

It’s possible (and common) for the relentless focus on data to result in unexpected consequences. For example, after having tracked meetings and deals, the RevOps leader surfaces data on pipeline generation in a shiny dashboard. We instruct sales leaders to hold AEs accountable to driving pipeline. But all too often, a leader uses those dashboards to “scream at the scoreboard:” they ask their AEs for more activity. The micromanage instead of coach, asking for more pipeline but without helping them figure out how to generate it. The result? Lots of low-quality meetings and opportunities that doesn’t convert to sales.

The NYPD tale reminds me of a story I always loved about sales compensation. There is a village that is overrun with cobras. So the mayor says, I’ll offer $1 for every dead cobra. So what is the result? People start breeding cobras and killing them. It’s a great example of how incentives drive behaviors, not outcomes.