Born in Chicago in 1933, Jones would cite the hymns his mother sang around the house as the first music he could remember. But he looked back sadly on his childhood, telling Oprah Winfrey that “There are two kinds of people: those who have nurturing parents or caretakers, and those who don’t. Nothing’s in between.” Jones’ mother suffered from emotional problems and was eventually institutionalized, a loss that made the world seem “senseless” for Quincy. He spent much of his time in Chicago on the streets, with gangs, stealing and fighting.

Music was his passion, and, almost literally, his salvation. As a boy, he learned that a Chicago neighbor owned a piano and he soon played it constantly himself. His father moved to Washington state when Quincy was 10 and his world changed at a neighborhood recreation center. Jones and some friends had broken into the kitchen and helped themselves to lemon meringue pie when Jones noticed a small room nearby with a stage. On the stage was a piano.

“I went up there, paused, stared, and then tinkled on it for a moment,” he wrote in his autobiography. “That’s where I began to find peace. I was 11. I knew this was it for me. Forever.”

Within a few years he was playing trumpet and befriending a young blind musician named Ray Charles, who became a lifelong friend. He was gifted enough to win a scholarship at the Berklee College of Music in Boston, but dropped out when Hampton invited him to tour with his band. Jones went on to work as a freelance composer, conductor, arranger and producer. As a teen, he backed Billie Holiday. By his mid-20s, he was touring with his own band.

“We had the best jazz band on the planet, and yet we were literally starving,” Jones later told Musician magazine. “That’s when I discovered that there was music, and there was the music business. If I were to survive, I would have to learn the difference between the two.”

His survivors include actor Rashida Jones and five other daughters: Jolie Jones Levine, Rachel Jones, Martina Jones, Kidada Jones and Kenya Kinski-Jones; son Quincy Jones III; brother Richard Jones and sisters Theresa Frank and Margie Jay.