Please enable JavaScript.
Coggle requires JavaScript to display documents.
DAG Story Boarding (Test Cluster (Supportability (Can TS help?, How far do…
DAG Story Boarding
Test Cluster
Dev Owned
Proposal in place
Build over next few weeks
Monitoring
Visualization
Pipelining
Logging
Influx concerns - can we handle this
Infrastructure impacts
Deployment
CICD
CLient core - what can we leverage?
Adding Tables
DB Version control (SPIKE)
Flyaway?
Liquibase?
Vertica Upgrades
Testing Strategy
Pipeline strategy
ETL
What transformations?
How often do we want to refresh?
Materialized Views
Hourly is current - could drop to batching
How fast can vertica refresh projections
SLowly chaning dimensions
Should we continue using OOzie
Airflow and perhaps others (SPIKE)
Blackhole as pipeline - not doing everything in there
Loading Vertica from Blackhole
Monitor for completness
How do we connect the two? (SPIKE)
Network connections to BH
Data Encoding/Compression
Spike: Do we need to encrypt?
SLI/SLO/SLA model
SLA agreement with Product (define these!)
Better than DOMO
SLO - Collection of SLIs (when will we alert)
SLI - Indicators (more aggressive than SLA) [what do we need to measure)
Access
Integrate with Jumpcloud or Keycloak
SMart Layer
Translate languages
Vocabularies
Table Language
User Language
Categrories/Flaggign expensive queries/Queues
Vertica has some bultin things (SPIKE)
Don't let them blow us up!
Regular processes
Other user's queries
Transparency: Let them know about scheduling/delays
What limits
won't be "everything over all time"
Product to explore limits
API
Versioning/Upgrades
Can we use a more recent version of SQL?
Supportability
Can TS help?
How far do we go into issues before sendign to Eng
Roles/Versions/Users
Don't be a noisy system!
Servicability
Documentation/Runbooks!
Provisioning
SQL Access
What are analysts going to use?
Which one first
API infrastructure
Inside or outside Pandora
Likely re-use existing
Is galera up to date?
Can we move up as Keycloak is moved to v9
Outside deliveries
Google/Microsoft/Amazon
Email
Prod Cluster
Partnership (OPS.Dev)
Read Only access
Disaster Recovery
Disasters
Restores
SHarding
IS count hardcoded?
Sub clusters
Server failure
Cluster offline
Offsite copy
Backup
Preventions
Rack awareness
VERTICA TRAINING!
Victor? What did you lean via the POC.
Vertica Engineer/Training
NOT! Pythian
Staging Cluster
Same spec machines as Prod but smaller cluster
Read Only access
Reports
Scheduled and Ad-hoc