The library of the Mathematical Institute is the successor of the library of the Mathematical Seminar, which originated from the first mathematical library, which was founded in 1871 by Dimitrije Nešić, professor of mathematics at the Great School. The initiator of the Mathematical Seminar was Mihailo Petrović. Although it was founded before 1905, the Seminar began its work after the transformation of the Higher School into a University, in the room where the first mathematical library was located.

The library of the Mathematical Seminar was destroyed in a fire in 1944, when the occupiers, while fleeing Belgrade, set fire to the University buildings and the Kolarac Еndowment. According to Milutin Milanković, "that library burned down to the last paper, and from that time it could only be renewed to an insufficient extent. But next to it another library was created in the Mathematical Institute of the Serbian Academy of Sciences, as a supplement to the one at the University." The library of the Mathematical Institute was founded in 1946, the same year as the Institute.

The Library's book collection was without a single publication from the beginning. The first library material was procured with the help of funds allocated by the Academy, but also with the funds of members of the Institute, and correspondence, procurement and exchange were performed personally by the Director and associates of the Institute, as there were no employees in the Library at that time. Before the employment of the first librarian, the work of arranging the material was performed mainly by younger associates of the Institute, for whom it was a unique opportunity to get acquainted with the literature.

The book fund of the Library was gradually enriched, mostly through exchanges with domestic and foreign institutions. The first agreement on the exchange of publications was reached in 1949 with Professor Fehr, the editor of the Swiss mathematical journal L’Enseignement Mathématique. The oldest volume (II - III) of this journal available in the Library is from 1956/7. It is important to note that today, 75 years later, the tradition of exchanging publications with many mathematical institutions is still nurtured. In this way, it was possible for the Institute's publications, especially “Publications de l'institut Mathématique”, to be presented to the world.

It is considered that one of the oldest books in the Library is “Elementorum Universae Matheseos” by Ruđer Bošković from 1757. The Library also houses a collection of about 260 old books in the fields of mathematics and mechanics, dating from 1790 up to 1950. On some books it is possible to find the seal of the institution from which it was purchased, while others were most likely received as a gift. The books are mostly in German and French. A few, no more than 30, are in Russian, while one, registered under number 1, is in Czech.

The first acting librarian in 1950 was Boško Tomić, secretary of the Institute, while the first librarian of the Institute was Vojislav Popović, a translator, employed in that position since 1952. Under his leadership, in 1955 a card catalog was made and library fund continued to increase. Vojislav Popović retired in 1963, when he was succeeded by Ilija Bratić. Ilija was the librarian of the Institute until his retirement in 1992. For a very short period, until 1993, the librarian of the Institute was Rajko Petronijević. In July 1993, Branka Bubonja became a librarian, where she remained until the end of August 2020. Since September 2020 librarian of the Institut is Kristina Milojević, a graduate librarian.

In the period from 1995 to 1997, the card catalog, the inventory book as well as the debt book were completely transferred to a digital format, using the PERGAM system.

Today, the library of the Mathematical Institute is a central and scientific library that collects, processes according to established principles, preserves material from all areas of mathematics, mechanics and computer science and gives it for use to employed researchers. It contains monographs and serial publications in its collection. In most cases, monographs are purchased, at the request of the researcher. It is most often foreign literature of some of the leading publishers in the field of (natural) sciences. The material is placed on shelves according to the principle of numerus currens, ie according to the number of arrivals. The current number of monographs available in the Library is 15452. The library also has 618 titles of serial publications, which amounts to over 65000 issues. Out of the total number of journals, only 18 magazines were purchased in 2020. The rest is procured, since the founding of the Institute by an established exchange.

Both monographs and journals can be searched through the electronic catalog which is available on the Institute's website (link).

In addition to monographs and serial publications, the Library provides access to some of the most important databases of scientific information specific in that they store data on exclusively mathematical disciplines, such as e.g. Zentralblatt MATH or MathSciNet.

Also, at the beginning of 2020, the base of research results of the employees of the Institute was started at the Institute (link). The database currently contains 3967 records/papers.