Brooks Calbert Robinson Jr. was born on May 18, 1937, in Little Rock, Arkansas. He was the son of Brooks Calbert Robinson Sr., a fireman and electrician, and Ethel Mae Denker Robinson, a homemaker. He had two sisters, Betty and Doris. He grew up as a fan of the St. Louis Cardinals and idolized their star player Stan Musial.
Brooks Robinson attended Little Rock High School, where he excelled in baseball and basketball. He was a member of the M.M. Eberts Post No. 1 Doughboys, one of the best American Legion teams in the region. He also received a full scholarship offer from the University of Arkansas for his basketball skills, but he chose to pursue a career in baseball instead.
Robinson was signed by the Baltimore Orioles as an amateur free agent in 1955, shortly after graduating from high school. He made his major league debut later that year at the age of 18. He spent most of his first three seasons in the minor leagues, playing for teams such as the York White Roses, the San Antonio Missions, and the Vancouver Mounties. Brooks also attended two winter semesters at Little Rock University during the offseasons and majored in commerce.