The masonry customs building erected in Tauragė between 1844 and 1847 is now known as Tauragė Castle. The entire architectural ensemble of the castle exhibits features of the Romantic Renaissance style. The phased project for the customs building was developed in 1844, and the two-story masonry building intended for border detainees was constructed in 1847. In 1852, the building was enclosed by a brick wall, a sauna was built in the courtyard, utility buildings were added, and a well was dug.

In 1866, the entire ensemble was transformed into a castle prison, featuring corner towers and shooting loopholes during the renovation. The "romantic" Renaissance castle character was imparted to the entire architectural ensemble. During that period, the castle acquired the overall appearance and layout that has been preserved to this day, with only two new buildings added to the courtyard in 1971 and 1986 after the war.