The Freer Gallery of Art Library was created around 1921 and housed publications deposited by the Smithsonian Institution, a large library collected by Charles Lang Freer himself and by donations from others, all relating to Asian art and culture and the Freer collection of art in particular. It is used by Smithsonian staff, Freer and Sackler galleries' staff especially; private researchers; and students. The Freer Gallery of Art and Arthur M. Sackler Gallery Library is a branch of the Smithsonian Institution Libraries network and currently reports to the Deputy Director of the Freer and Sackler galleries.

One of the first librarians in charge of the Freer Gallery of Art Library was Grace A. Parler (Grace Aesen before marriage in 1931) from at least 1931 to 1932. Grace C. Griffin was appointed Librarian for one year, 1943-1944, and was followed by Bertha M. Usilton, Librarian, 1944-1964. From 1964 to 1966, Constance B. Olsen held the position. Priscilla P. Smith was Librarian from about 1966 to 1981. Ellen Anne Nollman was Librarian, 1981-1986, and Head Librarian, 1986-1987. Lily Kecskes was Head Librarian from 1987 until 1992 when the title changed to Head, Library and Archives. Kecskes still holds the title of Head, Library and Archives, to the present.