Please enable JavaScript.

Coggle requires JavaScript to display documents.

Hannes Style Transfer Project (IDÈER (Som StyLit fast med flera olika…

- - - - "We take advantage of existing renderers' ability to render these LPE buffers with little computational overhead (We use NVIDIA Iray SDK)"
      - Light paths used:
        
        direct diffuse (LDE)
        
        direct specular (LSE)
        
        Additional channels can be added as necessitated by the given scene (e.g., the caustics channel LS*DE)
        
        diffuse interreflection (L.*DDE)
        
        first two diffuse bounces (LD{1,2}E)
        
        Det är RGB-kanalerna hos dessa, plus hos den färdiga bilden som utgör feature-vektorn för varje pixel
    - - Take 3 multi-channel images
        
        A
        
        B
        
        A'
      - Synthesize a new target image B' such that
        
        A : A' :: B : B'
      - Hertzmann's original algorithm: #
        
        For each resolution level and each pixel q in B' in scan-line order, a best matching pixel p is found in the source A' such that
        \(E(\mathbf{A},\mathbf{B},p,q,\mu)=\left|\left|A'(p)-B'(q)\right|\right|^{2}+\mu\left|\left|A(p)-B(q)\right|\right|^{2}\)
        is minimized.
        
        \(\mu\) - weight that controls the influence of a guidance
        
        For any image I, we use \(I(p)\) to denote a feature vector at a pixel \(p\).
        The vector \(I(p)\) is a concatenation of all pixels in a small square patch of width \(w\) centered at the pixel \(p\), where each pixel can have multiple feature channels.
        
        For features, Hertzmann et al. use the intensity value and an output from a steerable filter, whereas Bénard et al. (Stylizing animation by example) augment the RGB colors with several additional guidance channels, including temporal coherence and a distance transform.
        In our case, the feature vector contains colors of the full rendered image (RGB) and four LPE channels (each stored as an RGB image):
        { A, B } \(=\) ( FULL, LDE, LSE, L.*DDE, LD{1,2}E )
        
        dvs maximera likheten mellan pixeln p och q både mellan 3d-modellen och 3d-bollen, och mellan bilden vi genererar och den målade bilden.
        Så om vi har börjat på ett visst sätt i B' så fortsätter vi förmodligen på det vi börjat på typ?
        
        bold A och B är båda varianterna
      - Herzmann's original algorithm suffers from its greediness and can fail to preserve high-level structures.
        Subsequent work, [Fiser et al. 2014; Barnes et al. 2015] showed that better results can be obtained using the optimization scheme described by Kwatra et al. [2005] # and Wexler et al. [2007] #, which minimizes the following energy:
        \(\sum_{q\in\mathbf{B}}\textrm{min}_{p\in\mathbf{A}}E(\mathbf{A},\mathbf{B},p,q,\mu)\)
        using EM-like # iteration executed multiple times from coarse to fine resolution:
        
        (Algorithm 1)
        \(B'_k\) = de aktuella pixelvärden i \(B'\)
        \(B'_{k+1}\) = de uppdaterade värdena
        \(NNF\) = the nearest negihbor field that assigns source patch to each target patch.
        Funktionen \(Average\) beräknar genomsnittlig färg av colocated pixels i neighbour patches
        
        This approach produces notably better results, but as described in Section 2, it frequently leads to the wash-out effect. Kaspar et al. [2015] # and Jamriska et al. [2015] # mitigate this problem by encouraging uniform source patch usage. However, this restriction is suitable only when each randomly picked sub-region of the source texture is perceived similarly. Enforcing uniform patch usage in our scenario would create artifacts. For example, if the target has comparatively more highlight region than the source, uniform patch usage would force the highlight in the target to be filled with non-highlight source patches. In our scenario we need a different scheme that avoids the excessive use of certain patches while still permitting non-uniform utilization.
        
        A possible solution would be for each source patch to estimate its optimal usage and then augment the original uniformity-preserving approaches to handle this non-uniform case. However, it is difficult to estimate optimal patch utilization in advance unless we run the actual synthesis. To overcome this chicken-and-egg problem we propose a different approach that inspects the actual matching errors during the synthesis and detects cases when the algorithm starts to force patches into inappropriate locations.
        
        Our solution is based on the idea of reversed NNF retrieval [Rosenberger et al. 2009; Jamriska et al. 2015 #], in which a best matching target patch is retrieved for each source patch. The advantage of this approach is that it can enforce uniform usage of source patches. However, we must avoid forcing source patches to inappropriate locations in the target. We observe that, in practice, when we sort all matching error values and plot them with normalized axes, the resulting graph has a distinct, hyperbolic shape.
        
        Figure 6: Estimation of the error budget T: the sorted matching
        errors of all potential source-to-target patch assignments can be
        approximated by a hyperbolic curve \(f(x)\) on which a knee point \(k\)
        is detected and used to distinguish between feasible \(A^*\) and erroneous
        \(A^\times\) assignments. The integral of the errors in \(A^*\) gives an
        estimate of the error budget T.
        
        It starts with small error values, corresponding to feasible assignments (\(\mathbf{A}^{*}\) in Fig. 6). There is a knee point \(k\) where the error starts to increase rapidly. We estimate \(k\) by fitting a hyperbolic function \(f(x)=(a-bx)^{-1}\) to the data and retrieving the point where \(f'(x)=1\), i.e., \(k=\sqrt{1/b}+a/b\). Patches with indices above \(k\) are probably erroneous assignments (\(\mathbf{A}^{\times}\) in Fig. 6) that need to be avoided. We thus set a feasible error budget \(T\) that is an integral of all patch errors with indices below \(k\) and modify the original source-to-target assignment in a wat that maximizes the number of used source patches \(\left|\mathbf{A}^{*}\right|\) while satisfying an additional feasibility constraint:
        \(\sum_{p\in\mathbf{A}^{*}}\textrm{min}_{q\in\mathbf{B}} E(\mathbf{A}^{*},\mathbf{B},p,q,\mu)< T\)
        Such a constrained assignment leads to the desired situation with some target patches remaining unassigned because assigning them would introduce artifacts.
        
        We can repeat the retrieval
        
        and reuse good source patches to cover remaining positions in the target.
        
        This iterative scheme stops when all target patches are covered. In practice, it is feasible to stop even earlier (e.g., when 95% are covered) and use standard target-to-source nearest-neighbor retrieval to speed up the process. The number of iterations depends on the structure of the target scene and the complexity of the exemplar. Typically, after 10 iterations more than 80% of the target patches have been covered. In the general case, the number of iterations is roughly proportional to the ratio of the areas of corresponding illumination effects in the source and target images. For example, if the source contains one highlight and the target has four of similar size, then at least four iterations will be necessary to cover them. In practice, the number of iterations is typically slightly higher due to different structures of individual effects.
        To complete the algorithm we plug our modified patch assignment process into the original EM iteration (Algorithm 1) by replacing the step where the nearest neighbor field NNF is created. We then run the standard coarse-to-fine synthesis.
    - - "We implemented our techinque in C++ and CUDA. To accelerate the retrieval of nearest neighbours, we use PatchMatch with integrated support for masking. [PatchMatch paper] # To further accelerate the processing we exploit multicore processing using parallel tiling on the CPU [Generalized PatchMatch paper] and jump flooding on the GPU [Jump flooding in GPU with applications to Voronoi diagram and distance transform, 2009].
        We use fixed patch size \(w=5\) and guidance influence \(\mu=2\). Our pyramid uses 2 for the downsampling ratio, and for a one-megapixel image, we run the synthesis on 6 levels with 6 optimization iterations on each level. The NNF retrieval is accelerated with 6 PatchMatch iterations.
        Synthesizing a one-megapixel image takes about 15 minutes on a 3GHz CPU with 4 cores or 3 minutes on the GPU (GeForce GTX Titan Black). In addition to high quality synthesis, we also implemented a preview mode of the algorithm that uses half resolution, compresses LPE channels using PCA, and stores all textures as integers instead of the floating-point numbers used by the high-quality version. This achieves interactive response within 3-6 seconds on the GPU, enabling the applications discussed in Section 4.2."
- - - - https://www.reddit.com/r/CUDA/comments/80dk6p/how_to_install_cudas_toolkit_on_google_colab/
      - https://stackoverflow.com/questions/50560395/how-to-install-cuda-in-google-colab-gpus