Image and Signal Processing for Art Investigation

Ghissi Altarpiece Project

the new Caspers panel
The new Caspers panel

The Ghissi altarpiece had been removed from its church over a century ago, and dismantled. The 9 individual scenes (one central scene crucifixion, and 8 smaller pictures featuring St. John the Evangelist) had been sawn apart; the resulting panels ended up in different collections. Three of the four panels from the Kress Foundation are now at NCMA; the fourth is in the Portland Art Museum. Three other small panels are with the Metropolitan Museum in New York; the larger central panel is in the Art Institute in Chicago. But 3 + 1 + 3 + 1 adds up to only 8, not 9 — the ninth panel (the last of the 8 smaller scenes) is lost.

To complete the altarpiece, NCMA commissioned the Dutch artist and art reconstruction expert Charlotte Caspers to “reconstruct” the lost panel. Together with NCMA curator David Steel she designed a composition in Ghissi’s style; the scene likely represented in the ninth panel could be determined from the Golden Legend, a medieval bestseller chronicling lives of saints, which was the source for the first 7 small panels.

Learn more about the project. 

Platypus

PlaytpusPlatypus is a software solution that comes both as a standalone application and a Photoshop plugin. It is specifically designed to digitally remove cradling artifacts in X-ray images of paintings on panel. This project was made possible thanks to the financial support of the Kress foundation. Read more and download the software.

ARTICT - Pigment Identification using Machine Learning

Perseus and Andromeda
Perseus and Andromeda , Titian, 1554-1556, Wallace Collection, London

As a painting ages, parts of its pigment layers fall off or lose their original colors, due to chemical interactions with the surrounding environment.To maintain the completeness and vividness of the paintings, art conservators and restorers fill in areas of paint loss or repaint discolored areas.

In this project, we use a wavelets-based representation to take advantage of the similarity between the maps for different elements in the spatial domain (joint sparsity in the wavelet representation) to provide pigment-mixture maps. These maps can be used by the investigators to both “read” the pigments (each of which is typically already a compound of several elements) used by the artist to create the painting, and identify retouches or fills in an automatic fashion.

Learn more about the Pigment Identification using Machine Learning project.

ARTICT - Image Separation for Art Investigation

Ghent Altarpiece

Two double-sided panels from the Ghent Altarpiece closed. Images in this figure, used with permission of copyright holder, Saint-Bavo’s Cathedral – Art in Flanders; photo Dominique Provost.

X-radiographs (X-ray images) are a particularly valuable tool during the examination and restoration of paintings because these can help establish the condition of a painting (e.g., whether there are losses and damages that may not be apparent at the surface, perhaps because of obscuring varnish, overpainted layers, structural issues, or cracks in the paint) and the status of different paint passages (e.g., help to identify retouchings or fills).

However, interpreting X-ray images can be problematic because – due to the penetration ability of x-rays – these can contain features appearing on the front, back, or even within the painting.

Our team has developed a suite of entirely new self-supervised deep learning based approaches to tackle this X-ray image separation problem [1,2]. Our approach leverages readily available visible (RGB) images of the paintings on each side of the panel in order to decompose the mixed X-ray image onto its constituent (imagined) X-ray images (see Fig. 4).

Learn more about the Image Separation for Art Investigation project.

 

 

Members

Current
  • Prof. Ingrid Daubechies
  • Barak Sober, Assistant Research Professor
  • Ashley Kwon, undergrad Art & CS)
  • Wallace Peasley, undergrad Math & CS
  • Noelle Ocon, NC Museum of Art
  • Shira Faigenbaum-Golovin, Assistant Research Professor
Alumni

Publications

  • Pu, W., Sober, B., Daly, N., Higgitt, C., Daubechies, I., & Rodrigues, M. R. (2020, May). A Connected Auto-Encoders Based Approach for Image Separation with Side Information: With Applications to Art Investigation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2213-2217). IEEE.‏
  • Sabetsarvestani, Z., Sober, B., Higgitt, C., Daubechies, I., & Rodrigues, M. R. D. (2019). Artificial intelligence for art investigation: Meeting the challenge of separating X-ray images of the Ghent Altarpiece. Science advances, 5(8), eaaw7416.‏
  • Platisa, L., Cornelis, B., Fodor, G., Huang, S., Fesus, A., David, J., Liao, W., Ruzic, T., Dooms, A., Martens, M., Daubechies, I., (2016). Big Data Processing in Artwork Analysis, SuperMinds 2016.
  • Yin, R., Cornelis, B., Fodor, G., Ocon, N., Dunson, D., & Daubechies, I. (2016). Removing Cradle Artifacts in X-ray images of paintings. SIAM Journal on Imaging Sciences, 9(3), 1247-1272.
  • Cornelis, B., Yang, H., Goodfriend, A., Ocon, N., Lu, J., & Daubechies, I. (2016). Removal of canvas patterns in digital acquisitions of paintings. IEEE Transactions on Image Processing, 26(1), 160-171.‏
  • Deligiannis, N., Mota, J. F., Cornelis, B., Rodrigues, M. R., & Daubechies, I. (2016). Multi-modal dictionary learning for image separation with application in art investigation. IEEE Transactions on Image Processing, 26(2), 751-764.‏
  • Deligiannis, N., Mota, J. F., Cornelis, B., Rodrigues, M. R., & Daubechies, I. (2016, September). X-ray image separation via coupled dictionary learning. In 2016 IEEE International Conference on Image Processing (ICIP) (pp. 3533-3537). IEEE.‏
  • Pizurica, A., Platisa, L., Ruzic, T., Cornelis, B., Dooms, A., Martens, M., Dubois, H., Devolder, B., De Mey, M., Daubechies, I. (2015). Digital image processing of the Ghent Altarpiece: Supporting the painting's study and conservation treatment. IEEE Signal Processing Magazine, 32(4), 112-122.‏
  • Yin, R., Dunson, D., Cornelis, B., Brown, B., Ocon, N., & Daubechies, I. (2014, October). Digital cradle removal in X-ray images of art paintings. In 2014 IEEE International Conference on Image Processing (ICIP) (pp. 4299-4303). IEEE.‏
  • Cornelis, B., Ružić, T., Gezels, E., Dooms, A., Pižurica, A., Platiša, L., Cornelis, J., Martens, M., De Mey, M., Daubechies, I. (2013). Crack detection and inpainting for virtual restoration of paintings: The case of the Ghent Altarpiece. Signal Processing, 93(3), 605-619.‏ ‏
  • Cornelis, B., Yang, Y., Vogelstein, J. T., Dooms, A., Daubechies, I., & Dunson, D. (2013, July). Bayesian crack detection in ultra high resolution multimodal images of paintings. In 2013 18th International Conference on Digital Signal Processing (DSP) (pp. 1-8). IEEE.‏
  • Ružić, T., Cornelis, B., Platiša, L., Pižurica, A., Dooms, A., Philips, W., Martens, M., and De Mey, M., Daubechies, I. (2011, August). Virtual restoration of the Ghent Altarpiece using crack detection and inpainting. In International Conference on Advanced Concepts for Intelligent Vision Systems (pp. 417-428). Springer, Berlin, Heidelberg.‏