The tumultuous past of Karlobag is shown by the remains of the medieval fortress emerging above the town – Fortice and some churches. There is a library and art treasure in the Capuchin monastery with the church of St. Joseph containing the particularly valuable Cristoforo Tasca painting of the Last Supper.

The Roman settlement called Vegium was of importance for transport and trade and existed up to the 4th and 5th century when, due to attacks from the Goths, it was abandoned by its inhabitants who ran off to the island of Pag. But abandoning the settlement did not mean abandoning the memories of its existence, and due to the similarity to the island of Pag, the locality was named Bag.

During the time of the Croatian King Zvonimir, the area belonged to the coastal parish that belonged to the diocese of Rab. During Venetian rule there were caravans from the natural port of Bag (Baška draga) to the central hinterland of Lika, crossing important Velebit passes and Baške Oštarije.

In 1480 it was taken over by King Matija Korvin and connected to the Senj military district. After peace was concluded in Sremski Karlovci in 1699, when Lika was joined to Croatia, to the Croatian Military Border respectively, Karlobag was, for some time, the center of the entire Lika area. In 1733 the administrative center for the entire newly conquered area was transferred to the newly founded center of Gospić .