Agarose gels are viscoelastic soft solids that display a porous microstructure filled with water at 90% w/w or more. Despite an extensive use in food industry and microbiology, little is known about the drying kinetics of such squishy solids, which suffers from a lack of time-resolved local measurements. Moreover, only scattered empirical observations are available on the role of the gel composition on the drying kinetics. Here we study by in-situ interferometry the drying of agarose gels of various compositions cast in Petri dishes. The gel thinning is associated with the displacement of interference fringes that are analyzed using an efficient spatiotemporal filtering method, which allows us to assess local thinning rates as low as 10 nm/s with high accuracy. The gel thinning rate measured at the center of the dish appears as a robust observable to quantify the role of additives on the gel drying kinetics and compare the drying speed of agarose gels loaded with various non-gelling saccharides of increasing molecular weights. Our work shows that saccharides systematically decrease the agarose gel thinning rate up to a factor two, and exemplifies interferometry as a powerful tool to quantify the impact of additives on the drying kinetics of polymer gels.