Please enable JavaScript.

Coggle requires JavaScript to display documents.

Copy projects - Coggle Diagram

- - - - Move to Blob (Vit) :check:
      - We don't need to copy revisions bc. it's already stored per project container
    - - Taxonomy repo (asset folders) [Table API]
      - FileReference repo [Table API]
      - Asset repo [Table API]
      - Binaries [Blob storage]
    - - :!: Old storage [Table API] (won't be removed sooner than November)
      - New storage [SQL API]
  - - - Prepare some guidelines for teams
      - Enforcing in source code
    - - DB Triggers / Procedures supported in Change Feed [yes!]
      - Solutions
        
        Migrate also Blob storage :red_cross:
        
        add support to DataMigrations tool
        
        Do not migrate Blob storage but provide adapter :check:: -> Preferred by Vit. We should move the entity version to the blob properties/metadata so that we can know what adapter/migration to use before deserializing.
  - - - Initialization/Start
      - Restart on Fail
        
        Deliver
        
        Draft
    - - Deliver
      - Draft
    - - Restore failed
        
        on-project-data-processing-failed
        
        on-project-copy-stuck (=ProjectCopyStuck)
        
        on-project-data-copy-failed (=ProjectDataCopyFailed)
  - - - PRECO?
- - - - :!: Only the most recent change for a given item is included in the change log. Intermediate changes may not be available. If there are multiple updates to the same item in a short period of time, it is possible to miss processing intermediate updates. Does it need to be mitigated by some persistent storage of change feed data? link link2 -- Discussed with Ales & Vit and we acknowledge that this can happen, but it's not a blocker ✅
      - CFP - "at least once" guarantee link :check:
      - Error handling docs :check:
        
        Recommended to use dead-letter queue for failed consuming
        
        In what circumstances a message could not be processed and needs to be sent to DLQ? (maybe DAO mismatch?) -> DQL won't be used (see below)
        
        Failed items
        
        Differences based on the consumer
        
        CFP: retries the same item
        
        Azure Functions: skip & process next item
        
        What should we do in general?
        
        When blob write retry policy (e.g. 10 retries) fails, we should stop CFP and trigger alert for 3rd level support. They can restart WebJob and if it fails, manual intervention is needed (probably a problem with the code)
        
        DLQ :green_cross:
        
        Discussed with Vit and this seems to be overkill since there's not much else to fail other than a) reading from Change Feed b) storing to Blob. In both cases there's no need to continue processing other messages from the feed.
      - Monitoring
        
        Lease container RUs
        
        Source container RUs (expected an increase bc. of CFP)
        
        CFP monitoring
        
        Change feed estimator link -- seems like good way to monitor the change feed, lets use it :check:
        
        we should be able to identify that some event is not able to be handled and call 3rd level support :!:
        
        we can also log processing time of the message into AI
        
        what exactly should we monitor? :check:
        
        SPOZDENI CHANGE FEEDU
    - - :check: CosmosDB SQL API
      - :red_cross: CosmosDB Table API
        
        ❌ Not supported based on https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed
        
        Alternatives / Solutions
        
        :check: Move everything to SQL API details
        
        WEST can prepare their part of migration to run with ours :check:
        
        :red_cross: implement custom solution for Change Feed
        
        :red_cross: Use live data for Table API & Event source for SQL API
        
        Solutions with pros & cons
      - Storage account
        
        Blob storage (assets / revisions)
        
        :check: immutable - use original storage without change feed
    - - Is there any need for parallel consumers? CFP should handle that automatically, but needs to be checked -- discussed slightly with Vit and since bridge uses CFP already and has more-less the same amount of changes to process, there are no indices that we need more throughput :check:
      - WebJob (+ CFP) :check: vs Azure Functions :red_cross:
        
        AFs work differently with failed items (they skip & process next item) which makes them unsuitable in our case I believe. Therefore I suggest CFP.
        
        Also, cost-wise the WebJob should be much better
    - - Insert :check:
      - Update :warning:
        
        Intermediate updates are not tracked, only the latest is stored. :warning:
      - Delete :red_cross: (not supported - deleted item is removed from change feed)
        
        In most cases we use soft-delete :check:
        
        Hard-delete cases :question:
        
        Name placeholder
        
        Published variants & create new version
        
        GDPR data cleanup (LRT)
        
        RoleSettings
        
        soft + TTL :question:
        
        historical deletes should not be in Change feed :!: NEEDS TO BE VERIFIED https://kentico.atlassian.net/browse/DEL-3474
    - - Synapse link / Analytical store
        
        This is meant to be used for analytics operations on big data, not for warehouse scenarios
  - - - similar to deliver blob storage = blobs in folders
    - - cycle through all blobs and keep the last one in memory
    - - the same migration should be used to init new snapshot in Deliver
      - migration using CFP with separate processor & .WithStartTime(DateTime.MinValue.ToUniversalTime())
      - watch out for duplicities (the feed processor will eventually catch on live data already processed) :warning:
      - Needs to contain Application Timestamp (our .NET datetime timestamp, not the CosmosDB internal). Needs to be added to our DTOs because its not there and could be lost in mappings. :!:
      - pre-release tasks :check:
        
        update repositories to use CachedValue (to have better timestamps)
        
        new projects use New project migration status
      - extend DTOs with timestamp so that it's correctly stored in Blob
      - could we use CFP webjob? :
        
        can we “allow” webjob only some RUs? Should check whether CFP SDK allows to read RUs, Task.Wait() if > threshold
        
        Deliver - skip dispatcher?
    - - .json suffix?
    - - Check with DevOps about the AFs that they use -> can it be used for this one as well? --> Incremental backup :check:
      - disaster recovery scenarios :question: -> disucssed in other line
    - - https://kentico.atlassian.net/wiki/spaces/KMeerkat/pages/3442212885/Grooming+vNext#%3Acheck_mark%3A-Storage-account-for-event-store-(Draft)
      - New!
    - - https://docs.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview?tabs=azure-portal#delete-data-with-blob-index-tags + nejaky cleanup scheduled job?
      - use retention policy for all blobs, use fallback to cosmos for latest blob? But if there's no blob, how can I know I should fallback to CosmosDB?
      - na zaklade CFP by som si mohol ukladat "touchnute" projekty, a pre ne potom raz za cas vymazavat stare bloby?
      - the job/function could only cleanup projects that were recently (maybe daily) touched. This "touch" info is needed for CC, therefore it could be available. We would need to probably move this "touch" info from Deliver (where it resides at the moment) to Draft Change feed, so that we know about EVERY change in the project.
- - - - Immutable data reuse (per Project Container)
        
        Assets
        
        Revisions