banner

Using neural networks to solve image problems through artificial intelligence

Huda Lafta Majeed

University of Wasit /college of agriculture/ Iraq

hulafta@uowasit.edu.iq

Orcid :https://orcid.org/0009-0001-7220-8076

Abstract

    Deep neural networks may be utilized to handle a wide range of inverse issues that arise in computational imaging, according to recent machine learning research. We examine the key recurring themes in this developing field and offer a taxonomy that can be applied to group various issues and reconstruction approaches. Our taxonomy is arranged along two main axes in which first includes that if a forward model is known and how much it is utilized in training and testing; and other that whether the learning is supervised or unsupervised, that is, whether the training depends on having access to matched ground truth picture and measurement pairs. The manuscript discusses trade-offs with these various rebuilding strategies, cautions, and typical failure scenarios with potential future research directions in imaging with inverse problems. In addition, the implementation patterns and aspects are integrated with the use of deep convolutional networks in deep learning for inverse problems in imaging.

Keywords: Imaging, Inverse Problems in Imaging, Deep Convolutional Networks

Introduction

Inverse issues, or reconstructing an unseen signal, picture, or multidimensional volume from data, are the topic of this study. A forward procedure, which is often non-invertible, is used to extract the observations from the unknown data. This framework encompasses numerous imaging tasks, such as image contrast enhancement, deconvolution, inpainting, compressed perception, superresolution, and many more. Without some prior understanding of the data, recreating a singular solution that conforms to the observation for these forward processes is challenging or impossible [1, 2]. The data-fit term, which assesses how well the reconstructed picture fits the observations, and the regularizer, which takes into account past information and favours images with desirable characteristics like smoothness, are the two components of the cost function that are often minimized [3, 4].

Deep neural networks may directly compute regularised reconstructions across a variety of computational imaging applications using vast amounts of training data, as recently shown by machine learning research [5]. By requiring the rebuilt picture x to remain on a trained manifold, work on unsupervised approaches shown how deep neural network models may regularise [6, 7]. We examine the key recurring themes in this developing field and offer a taxonomy that can be applied to group various issues and reconstruction approaches. We also go through the trade-offs related to various reconstruction strategies and outline potential directions for further research.

https://media.springernature.com/lw685/springer-static/image/art%3A10.1186%2Fs40537-021-00444-8/MediaObjects/40537_2021_444_Fig29_HTML.png

Figure 1 : Deep Neural Networks and Application Areas

In contrast to the well-posed issues that are generally encountered in mathematical modelling, inverse questions are frequently ill-posed. The durability of the solution or remedies is the most frequently broken of Jacques Hadamard’s three requirements for a well-posed issue (presence, distinctiveness, and stability of the solution or solutions).

The inverse issue is represented as a mapping between metric spaces in the sense of systematic study. Although inverse problems are frequently defined in infinite dimensional spaces, constraints on the number of observations and the practical concern of only retrieving a small number of random variables may force the issues to be reformulated in discrete form [8].

Key Perspectives and Challenges

Medical and Health Sciences

Deep learning has shown tremendous promise in the last five years for resolving a variety of imaging inverse issues; for instance. The fundamental comprehension of the application of deep learning techniques and their limits, however, is still developing. This opens up new possibilities for in-depth scientific analysis, further investigation, and fundamental comprehension [9, 10].

imaging in medicine Many imaging modalities, including MRI, CT, PET, SPECT, and others, require the reconstruction of pictures from projective data. While performing well, traditional approaches might be computationally taxing.

Figure 2 : Deep Learning and Imaging

Computational photography

Computational photography aims to produce photos that are aesthetically pleasing and substantially accurate representations of the scenes they depict. Deep learning is a great choice for computational photograph reconstruction issues because of these circumstances [11].

Deep learning, for instance, offers outstanding low-light imaging. The work shows how deep learning makes it possible to estimate the depths of various objects in a scene from a snapshot. The production version of Google’s most recent smartphone photography systems now use deep learning for white balancing [12, 13].

Figure 3 : Integration Aspects

Computational Microscopy

Solving a reconstructions problem has become an essential component of microscopy with the rise in popularity of computer methods like ptychography. As a result, there has been an increase in interest in using deep learning in microscopy, leading to the development of novel methods for both image reconstruction and the design of the lighting patterns and optical components of microscopes.

Figure 4 : Computational Microscopy

Geophysical Imaging

By simulating the actual physical dispersion of seismic waves, seismic inversion and imagery entail the reconstruction of the Earth’s interior. The formulations of these ill-posed inverse issues can be adjusted by comparing simulated synthetic readings to true acoustic measurements of reflected waves. These issues have recently been addressed by deep learning techniques, including approaches that depend on generative models [14] limited by partial differential equation.

Assorted Computational Imaging Applications

Figure 5 : Monocular depth estimation

Deep learning has demonstrated great potential in a variety of additional difficult computational inverse issues, including as ou pas imaging, lensless imaging, high – resolution, ghost tomography, and imaging in scattering medium, all of which are still in the research and development stages.

Unsupervised and Supervised Aspects in Inversion

We begin by describing a key dichotomy seen in the literature as well as in our suggested taxonomy of methods for solving inverse situations. We refer to the supervised inversions used by the first (and best-known) deep learning inversion algorithm family as inversions. The next step is to train a network that receives measurements y and reproduces the picture x, or learns an inverse mapping, from the measurements. Although these supervised approaches frequently yield excellent results, they are sensitive to modifications or ambiguity in the forward operator A.

Additionally, each time the measuring procedure changes, a communication system needs to be trained. A matched dataset of pictures x and measurements y is not used in the second family of approaches we discuss. In our taxonomy, unsupervised techniques are divided into three categories: (1) methods that combine measurements y with unpaired dataset pictures x; (2) methods that only use ground truth photos x; and (3) methods that only use data y.

Deep Convolutional Networks

This work’s primary experimental emphasis is X-ray CT reconstruction (though we stress that the presented method is general and should apply to several modalities). Both direct and iterative approaches have a long history in X-ray reconstruction.

Regularized iterative techniques have been the focus of recent study on the issue. For instance, one method [13] makes use of the Fair potential function to encourage sparse gradients, and another method makes use of a nonlocal regularizer to encourage reconstruction patches to resemble one another. The researchers utilise a regularisation term that encourages regions of the reconstruction to be sparse in a learnt vocabulary. Learning has also been investigated for X-ray CT reconstruction.

Additionally, CNNs are now being used for X-ray CT reconstruction. A weighted mix of FBP reconstructions with learnt filters and a proven improvement over normal FBP for low-views reconstruction. The comparison of regularised iterative approaches was not included in this paper. Recent research examines the use of a CNN to postprocess a single reconstruction, while employs a CNN to learn to fuse many reconstructions in the low-dose situation.

Figure 6 : Deep Convolutional Neural Networks

Over the past few decades, regularised iterative algorithms have been the de facto method for solving ill-posed inverse problems. These techniques yield great results, but their practical application might be problematic due to issues like the expensive forward and curve fitting operators’ computations and the complex hyper parameter selection [13].

The discovery that unrolled iterative algorithms have the shape of a CNN when the advance model’s normal operator (H*H, the adjoint of H times H), is a convolution, serves as the basis for our research.

Figure 7 : Feed-Forward Neural Networks

The mapping of input image(s) to output image(s) is a common feature of inverse issues in image processing, phase imaging, and computer vision; nevertheless, these problems are often resolved by various application-specific techniques. Although deep convolutional networks have demonstrated considerable promise for a wide range of image-based applications, their underlying non-linearity can occasionally make training them difficult [14].

Caveats and Assorted Aspects

The robustness to a variable forward model during testing than during train operations. In certain circumstances, the forward model used for training and testing are two separate models. Consider learning how to rebuild MRI pictures for one clinic’s scanning and then attempting to utilise that knowledge to recreate MRI images for another clinic’s (slightly different) scanner. The degree to which the various techniques outlined are resistant to changes in the forward models between training and testing will vary.

The fundamental premise of all machine continuing to learn image reconstruction techniques is that learning algorithm should be indicative of test data. It is unknown how true that assumption is in other applications, including medical imaging. One may picture people who have tumours or atypical anatomy that isn’t mirrored in the training set.

Deep learning models’ flexibility and capacity [15] have the side consequence of being challenging to comprehend and assess. Because of this, our knowledge of some techniques that provide cutting-edge results is now relatively limited.

Figure 8 : Robustness to forward model perturbations

In the past several years, generative modelling has advanced significantly, and the perceived quality of produced pictures is now nearly lifelike. Earlier GANs had trouble processing pictures with complex semantic structures, but more recent GANs have been able to do so. Despite recent advancements in creating better generative models, there are still numerous aberrations and distortions in the pictures that are produced. There is some debate about deep learning models for CT scans that automatically map measurements to pictures. Even when the characteristics in the photo are not there, deep learning has an amazing capacity to produce images that appear realistic [16].

It has been suggested and shown how to solve inverse issues involving picture to image translations in many domains using an application-neutral framework. Three examples in three quite distinct fields have been used to illustrate the general capacity of this framework. With the help of this framework, the challenges of a difficult application, tough experimental setups, and challenging inverse algorithm implementation have been reduced. The frustration of DNN hyper-parameter adjustment is reduced by matching extra output layers to corresponding low-frequency characteristics, if adequate datasets are available from either numerical simulation or direct measurement. We anticipate that this sturdy, all-purpose design will spur the development of additional systems that address various inverse issue subtypes, extending the range of inverse problem applications.

Conclusion

The process of determining the causes of a set of observations is known as an inverse problem in science. Examples of inverse problems include calculating an image in X-ray imaging techniques, source reconstruction in sounds, or determining the density of the Earth from dimensions of its gravity field. The reason it begins with the consequences and then assesses the causes is why it is known as an inverted issue. It is the opposite of a forward issue, which determines causes first before calculating effects. We only had access to small amounts of training data in many contexts, such as medical imaging; in other contexts, such as cosmology, we could only have access to any “actual” training pictures, but we may nevertheless create simulated training data. In these circumstances, the objective is to enhance inverse problem solvers in our target domain by using data from simulations or data from a different application domain. The term “transfer learning” or “area adaptation” is often used to describe this difficulty. The minimal empirical work in transfer learning for imaging’s inversion issues is encouraging and points to the need for more research.

References

[1]  Chambolle, “An algorithm for total variation minimization and applications”, J. Math. Imag. Vis., vol. 20, no. 1, pp. 89-97, 2004.https://doi.org/10.1023/B:JMIV.0000011325.36760.1e.

[2]  S. G. Chang, B. Yu and M. Vetterli, “Adaptive wavelet thresholding for image denoising and compression”, IEEE Trans. Image Process., vol. 9, no. 9, pp. 1532-1546, Sep. 2000.DOI: 10.1109/83.862633

[3]  T. F. Chan and C.-K. Wong, “Total variation blind deconvolution”, IEEE Trans. Image Process., vol. 7, no. 3, pp. 370-375, Mar. 1998.DOI: 10.1109/83.661187

[4]  L. I. Rudin, S. Osher and E. Fatemi, “Nonlinear total variation based noise removal algorithms”, Phys. D Nonlinear Phenomena, vol. 60, no. 1, pp. 259-268, 1992. https://doi.org/10.1016/0167-2789(92)90242-F.

[5]  D. Krishnan and R. Fergus, “Fast image deconvolution using hyper-Laplacian priors”, Proc. Adv. Neural Inf. Process. Syst., pp. 1033-1041, 2009.

[6]  M. Bertalmio, G. Sapiro, V. Caselles and C. Ballester, “Image inpainting”, Proc. 27th Annu. Conf. Comput. Graph. Interactive Tech., pp. 417-424, 2000.

[7]  H. Jung, K. Sung, K. S. Nayak, E. Y. Kim and J. C. Ye, “k-t FOCUSS: A general compressed sensing framework for high resolution dynamic MRI”, Magn. Reson. Med., vol. 61, no. 1, pp. 103-116, 2009.

[8]  M. Guerquin-Kern, M. Häberlin, K. P. Pruessmann and M. Unser, “A fast wavelet-based reconstruction method for magnetic resonance imaging”, IEEE Trans. Med. Imag., vol. 30, no. 9, pp. 1649-1660, Sep. 2011.

[9]  E. Y. Sidky and X. Pan, “Image reconstruction in circular cone-beam computed tomography by constrained total-variation minimization”, Phys. Med. Biol., vol. 53, no. 17, pp. 4777, Sep. 2008.

[10]  M. G. McGaffin and J. A. Fessler, “Alternating dual updates algorithm for X-ray CT reconstruction on the GPU”, IEEE Trans. Comput. Imag., vol. 1, no. 3, pp. 186-199, Sep. 2015.

[11]  M. T. McCann, M. Nilchian, M. Stampanoni and M. Unser, “Fast 3D reconstruction method for differential phase contrast X-ray CT”, Opt. Exp., vol. 24, no. 13, pp. 14564-14581, Jun. 2016.

[12]  Y. LeCun, Y. Bengio and G. Hinton, “Deep learning”, Nature, vol. 521, pp. 436-444, May 2015.

[13]  A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet classification with deep convolutional neural networks”, Proc. Adv. Neural Inf. Process. Syst., pp. 1097-1105, 2012.

[14]  E. J. Candès, J. Romberg and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information”, IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489-509, Feb. 2006.

[15]  T. Goldstein and S. Osher, “The split Bregman method for L1-regularized problems”, SIAM J. Imag. Sci., vol. 2, no. 2, pp. 323-343, 2009.

[16]  M. Lustig, D. Donoho and J. M. Pauly, “Sparse MRI: The application of compressed sensing for rapid MR imaging”, Magn. Reson. Med., vol. 58, no. 6, pp. 1182-1195, 2007.

https://powderencouraged.com/zht6b3bs66?key=8523542c4aaf6f083ce56f74bd271319
https://powderencouraged.com/w3m6c9b5?key=af8b4909c0965493e9682d74bb50a646
https://powderencouraged.com/ksq5rgwdmh?key=b5b47a0d83f8742dcf71d4de8ef00494
https://powderencouraged.com/bkcduddx?key=0252cc21d0f4e0ae5f53245940611301
×
error: Content is protected !!