"Puerto Rico has lost a great Puerto Rican, a very valiant person who aside from being a great athlete, was a great human being," David Bernier, president of the US territory's Olympic committee, told radio station WKAQ.
Mr. Torres began fighting when he joined the US Army as a teenager. He won a silver medal as a light middleweight at the 1956 Melbourne Olympics while competing for the United States. He lost the title bout to Hungary's Laszlo Papp.
After retiring in 1969, he became a representative of New York's Puerto Rican community, chaired the New York State Athletic Commission in the 1980s, and served as supervisor for the World Boxing Organization.
Early in his pro career, according to The New York Times, Mr. Torres became friendly with writers and began writing a column in the New York Post, often on Hispanic community affairs.
He also wrote two biographies, "Sting Like a Bee" on Muhammad Ali and "Fire and Fear" on Mike Tyson.
Norman Mailer wrote the preface to the biography of Ali. They remained friends, and as late as 1984, Mr. Torres was regularly sparring three rounds on Saturdays with Mailer, who was 61, the Times reported.
"Through his boxing, writing, and speaking out on the important issues of our time, Jose was an inspiration to millions of people across the country and around the world," Governor David Paterson of New York said in a statement, pointing out that Mr. Torres trained in New York City and was a New York Golden Gloves champion in the 1950s.
As the first Puerto Rican to serve as commissioner of the New York Athletic Commission, Mr. Torres "was instrumental in strengthening boxing in New York," Paterson said.
Mr. Torres will be buried Thursday in Ponce.