Please enable JavaScript.
Coggle requires JavaScript to display documents.
DS Internal Documentation (Audience Targeted Documentation (User support…
DS Internal Documentation
Audience Targeted Documentation
User support "How do I help users use the datastore? How do I help them get data in/out?"
user permissions "weirdness" with iRODS / Agave?
How to help users configure appropriate permissions for their project?
Is datastore suitable for sensitive data and data protected by HIPAA/FERPA/ITAR/CUI? Pre-publication data?
I want to add my own reserved/private capacity to datastore with my own hardware. How do?
How to audit user permissions / user actions?
How do I know that no one else can access my data?
When to use iCommands vs. iRODS FUSE vs. Cyberduck vs. WebDAV vs. Cyberduck vs. ??? / etc.
How to use client tools (e.g. iCommands) on Windows?
How to efficiently upload 10E6+ tiny files?
How to deal with user claiming datastore "ate my homework"? Auditing previous user mistakes
How to use it
iCommands
help
ihelp
ierror
environment
ienv
imiscsvrinfo
iinit
irods_environment.json
.irodsA
environment variables
IRODS_ENVIRONMENT_FILE
others ...
password='kraqM3' iinit "$password"
iexit
ilsresc
list resources
list resource details
navigation
ipwd
icd
ils
list collection contents
list zone level details
list zone and grid level details
list recursively
file transfer
iput
streaming, parallel, RBUDP
choosing resource
bulk, redirected
overwrite, overwrite all
restarting
with ticket
write lock
checksum validation
atomically assigned metadata
iget
streaming, parallel, RBUDP
choosing resource/replica
redirected
restarting
with ticket
read lock
chksum validation
file manipulation
imkdir
create collection
create collection with parents
irm
move to trash
delete
icp
streaming, parallel, RBUDP
choosing destination resource
restarting
checksum validation
irmtrash
access control
ichmod
iticket
metadata
isysmeta
ichksum
imeta
iquest
ips
iqstat
irepl
itrim
irsync
iphymv
iRODS FUSE client
kanki-irodsclient
WebDAV
Cyberduck
Rules
Backend support and troubleshooting (DS-243)
Issues and Resolutions
Historical performance issues and how to mitigate them
Any time a box is not SSHable?
What are most common issues and resolutions?
Operations runbook (DS-244)
Deployment (and upgrade?) process
How to "initialize" datastore (if manual intervention is needed)?
How to restart iRODS? What will break when we do? (Checklist!)
What NOT to ever do?
Maintenance procedures and suggested schedule
ansible
issue reporting
documentation
maintenance host
What to do for each type of error message / email (DS-552)
How to detect corrupted data? What does datastore do about bitrot? Autonomic periodic scrubbing/healing?
Disaster Recovery (DS-594)
Emergency Contact List
architectural description of Data Store (DS-250)
How is CyVerse's datastore different from "vanilla" iRODS? (DS-254)
Project-specific domain logic (rulesets) for SERNEC, Agave, etc
What is the iCAT database? How does it work?
Underlying DB engine for iCAT
Replication
What part of this lives at TACC, how and why? (DS-255)
iRODS ACLs
How are underlying servers configured?
Firewall?
Underlying storage?
RAID?
Are we optimizing for capacity, speed, or some "compromise"?
TLS/other certificates?
Load balancing?
WebDAV
What does "normal" look like? How do we monitor the datastore and know that it is happy?
Where do logs live?
Where do configs live?
cron scripts?
How are metrics collected?
What responsibilities lie with Andy E / UITS?
Which component services are most critical, which are ancillary?
What is the Data Store?
iRODS
Data Virtualization
Grid
Catalog Server/ICAT-Enabled Server (IES)
Resource Server
File/Replica
Zone
Data Object
Resource
Storage Resource
unix file system
more ...
Coordinating Resource
random
replication
passthru
more ...
Collection
Metadatum
User
rodsuser
rodsadmin
rodsgroup
custom type
groupadmin
anonymous
public
Authorization
Permissions
Tickets
Workflow Automation/Policies
Microservice
Rule
Policy Execution Point (PEP)
Connection
Agent
Reconection
File Transfer
Streaming
Parallel/Chunked
RBUDP
Communication
Client-Server
Server-Server
Grid Level
Zone Level/Clerver
Federation
Authentication
Native
PAM
Kerberos
Extensibility
Specific Query
Command Script
Plugin
API
Microservice
Resource
more ...
Administrative iCommands
iadmin
help
interactive mode
user management
info
adding
modifying
removing
scramble/descramble password
resource management
make resource
modify resource
remove resource
modify paths of files on resource
add/remove child to resource
zone management
zone info
connect to remote zone
update remote zone connection
disconnect from remote zone
special queries
quotas
listing
for users or groups
limit to a particular resource or global
always restricted to zone
tokens
trim unused AVUs
izonereport
ifsck
ips
iqstat
irepl
itrim
irsync
iphymv
ireg
igroupadmin
irods-grid
server configuration
runtime processes
irodsServer
irodsAgent
irodsReServer (IES only)
layout
/var/lib/irods
.odbc.ini (IES only)
.irods
.pgpass (IES only)
iRODS
plugins
/etc/irods
variable mappings
hosts_config.json
host_access_control_config.json
service_account.config
server_config.json
database_config.json (IES only)
server rules
function mappings
Vault (resource server only)
ICAT DB
iRODS Rules and Command Scripts
RabbitMQ (AMQP)
Apache with davrods (WebDAV)
NetCDF plugin
iCommands/FUSE installers
Custom iRODS admin scripts
HAProxy
Technology champion / recommender / CIO
What is our roadmap? What is on Tony's todo/wish list?
We are currently sending people with "sensitive" data elsewhere, is there a plan to support these use cases? VERSSA-R?
When to recommend datastore vs. other solutions?
What workload does datastore do a great job of (archival? temp storage for data-intensive compute?
What workload does datastore not do a great job of? (Too many files? Too large files? In-place compute?)
What are CyVerse's "Lessons learned" from operating datastore? What have we done right, what could we have done / do better? Good and bad tech decisions in hindsight
How to run incident response when Something Bad happens? auditing possibly malicious user actions?
What are our policies?
What is retention policy for un-accessed private data?
What is user storage quota? What is "official word" and what is actually enforced? ;)
Developer support "How do I extend / integrate with datastore?"
How do I make a development environment?
Orientation for developers and integrators
WBS
Jira
What client libraries and APIs are available for integrating with datastore? When to use which ones?
How do I manage user credentials in my app that connects to datastore?
Can iRODS listen for events in my application and do things with them?
APIs
Jargon Java API
C API
python-irodsclient python API
WebDAV
Process Execution Points (PEPs)
Rules
Command Scripts
Microservices
Data curators "How do I manage/curate my data?"
What metadata to use?
Which standards to use?
How to organize metadata?
Which platform to use to manage metadata?
DE?
iRODS?
Agave?
Bisque?
How long will my data last? What is long-term archival strategy for published data?
What happens to my data if CyVerse goes away?
Organization of Documentation
Proximity to Artifacts
Link to upstream docs at iRODS/etc where we can (DRY)
Bidirectional traceability
Miscellanea
Who "owns" the datastore?
Does datastore predate Tony???