Credential Assets

Docs

Credential assets provide secure access to external services and sensitive data. They standardize authentication patterns and ensure secrets are managed safely across workloads.

What are Credential Assets?

Credential assets manage: - API Keys: Access tokens for external services - Database Credentials: Username/password combinations - SSH Keys: Git repository and server access - Cloud Credentials: AWS, GCP, Azure authentication - Registry Access: Docker image registry credentials

Types of Credentials

1. Environment Variable Credentials

API Keys and Tokens:

Name: wandb-credentials
Type: Environment Variables
Variables:
  WANDB_API_KEY: <api-key-value>
  WANDB_PROJECT: ml-experiments
  WANDB_ENTITY: company-team

Cloud Storage Access:

Name: aws-s3-credentials
Type: Environment Variables
Variables:
  AWS_ACCESS_KEY_ID: <access-key>
  AWS_SECRET_ACCESS_KEY: <secret-key>
  AWS_DEFAULT_REGION: us-west-2

2. File-based Credentials

SSH Keys:

Name: git-ssh-credentials
Type: File Mount
Mount Path: /root/.ssh/
Files:
  id_rsa: <private-key-content>
  id_rsa.pub: <public-key-content>
  known_hosts: <known-hosts-content>

Configuration Files:

Name: cloud-config-credentials
Type: File Mount
Mount Path: /root/.aws/
Files:
  credentials: <aws-credentials-file>
  config: <aws-config-file>

Creating Credential Assets

Method 1: Using the UI

1. Navigate to Assets → Credentials
2. Click "+ NEW CREDENTIAL"
3. Configure the credential:

API Token Credential:

Name: huggingface-token
Description: HuggingFace API access token
Type: Environment Variables
Scope: Project
Variables:
  HUGGINGFACE_HUB_TOKEN: hf_xxxxxxxxxxxxxxxxxx

Method 2: Using Kubernetes Secrets

Create secret first:

# Create API key secret
kubectl create secret generic wandb-secret \
    --from-literal=WANDB_API_KEY=your-actual-api-key \
    --namespace=runai-<project-name>

# Create SSH key secret
kubectl create secret generic git-ssh-secret \
    --from-file=id_rsa=/path/to/private/key \
    --from-file=id_rsa.pub=/path/to/public/key \
    --namespace=runai-<project-name>

Create credential asset YAML:

apiVersion: run.ai/v1
kind: Credential
metadata:
  name: wandb-credentials
  namespace: runai-<project-name>
spec:
  secretRef:
    name: wandb-secret
  type: environment

Apply with:

kubectl apply -f wandb-credentials.yaml

Common Credential Examples

1. Machine Learning Services

Weights & Biases:

Name: wandb-access
Type: Environment Variables
Variables:
  WANDB_API_KEY: <your-wandb-api-key>
  WANDB_PROJECT: deep-learning-experiments
  WANDB_ENTITY: research-team
Usage: Experiment tracking and model monitoring

MLflow:

Name: mlflow-credentials
Type: Environment Variables  
Variables:
  MLFLOW_TRACKING_URI: https://mlflow.company.com
  MLFLOW_TRACKING_USERNAME: ml-user
  MLFLOW_TRACKING_PASSWORD: <password>
Usage: Model registry and experiment tracking

HuggingFace Hub:

Name: huggingface-access
Type: Environment Variables
Variables:
  HUGGINGFACE_HUB_TOKEN: hf_<token>
  HF_HOME: /workspace/.cache/huggingface
Usage: Download private models and datasets

2. Cloud Storage Services

AWS S3:

Name: aws-s3-access
Type: Environment Variables
Variables:
  AWS_ACCESS_KEY_ID: AKIA...
  AWS_SECRET_ACCESS_KEY: <secret>
  AWS_DEFAULT_REGION: us-west-2
  AWS_SESSION_TOKEN: <session-token>  # If using temporary credentials

Google Cloud Storage:

Name: gcp-storage-access
Type: File Mount
Mount Path: /workspace/.config/gcloud/
Files:
  service-account.json: <service-account-key>
Environment Variables:
  GOOGLE_APPLICATION_CREDENTIALS: /workspace/.config/gcloud/service-account.json

Azure Blob Storage:

Name: azure-storage-access
Type: Environment Variables
Variables:
  AZURE_STORAGE_ACCOUNT: mystorageaccount
  AZURE_STORAGE_KEY: <storage-key>
  AZURE_STORAGE_CONNECTION_STRING: <connection-string>

3. Database Access

PostgreSQL:

Name: postgres-credentials
Type: Environment Variables
Variables:
  POSTGRES_HOST: db.company.com
  POSTGRES_PORT: "5432"
  POSTGRES_DB: ml_experiments
  POSTGRES_USER: ml_user
  POSTGRES_PASSWORD: <password>

MongoDB:

Name: mongodb-credentials
Type: Environment Variables
Variables:
  MONGODB_URI: mongodb://username:password@mongo.company.com:27017/ml_data
  MONGODB_DATABASE: experiments

4. Container Registry Access

Docker Hub:

Name: dockerhub-credentials
Type: Docker Registry
Registry: docker.io
Username: company-user
Password: <password>

Private Registry:

Name: private-registry-access
Type: Docker Registry
Registry: registry.company.com
Username: ml-team
Password: <access-token>

Using Credential Assets

In Workload Creation

Via UI: 1. Create new workload 2. Credentials section → Add credential assets 3. Verify no conflicts with environment variables

Via CLI:

runai submit "ml-experiment" \
    --credential wandb-credentials \
    --credential aws-s3-access \
    --image pytorch/pytorch:latest

Multiple Credentials

Combine Multiple Services:

runai submit "comprehensive-training" \
    --credential wandb-credentials \
    --credential huggingface-access \
    --credential aws-s3-access \
    --credential git-ssh-credentials \
    --gpu 1

Override Credential Variables

Temporary Override:

runai submit "experiment-override" \
    --credential wandb-credentials \
    --env WANDB_PROJECT=special-experiment \
    --env WANDB_RUN_GROUP=test-group \
    --gpu 1

Secure Credential Management

1. Creating Secrets Securely

From Command Line:

# Create secret from stdin (safer than command line)
echo -n 'your-api-key' | kubectl create secret generic api-secret \
    --from-file=API_KEY=/dev/stdin \
    --namespace=runai-<project>

# Create from file
kubectl create secret generic config-secret \
    --from-file=config.json=/path/to/config.json \
    --namespace=runai-<project>

From YAML (Base64 encoded):

apiVersion: v1
kind: Secret
metadata:
  name: encoded-secret
  namespace: runai-<project>
type: Opaque
data:
  API_KEY: eW91ci1hcGkta2V5LWhlcmU=  # Base64 encoded
  USERNAME: dXNlcm5hbWU=              # Base64 encoded

2. SSH Key Management

Generate and Store SSH Keys:

# Generate SSH key pair
ssh-keygen -t rsa -b 4096 -f ~/.ssh/runai_key -N ""

# Create secret from key files
kubectl create secret generic git-ssh-key \
    --from-file=id_rsa=~/.ssh/runai_key \
    --from-file=id_rsa.pub=~/.ssh/runai_key.pub \
    --namespace=runai-<project>

# Create known_hosts for Git servers
ssh-keyscan github.com gitlab.com > known_hosts
kubectl create secret generic git-known-hosts \
    --from-file=known_hosts=known_hosts \
    --namespace=runai-<project>

3. Credential Rotation

Update Existing Credentials:

# Update secret with new API key
kubectl patch secret api-secret \
    --type='json' \
    -p='[{"op": "replace", "path": "/data/API_KEY", "value": "'$(echo -n 'new-api-key' | base64)'"}]' \
    --namespace=runai-<project>

# Restart workloads to pick up new credentials
runai delete <workload-name>
runai submit <workload-name> --credential updated-credentials

Using Credentials in Code

1. Environment Variable Access

Python Example:

import os
import wandb

# Access credentials from environment variables
api_key = os.getenv('WANDB_API_KEY')
project = os.getenv('WANDB_PROJECT', 'default-project')

# Initialize service
wandb.init(project=project)

print("Successfully authenticated with W&B")

AWS S3 Access:

import boto3
import os

# Credentials loaded from environment variables
s3_client = boto3.client('s3')

# List buckets (requires valid credentials)
response = s3_client.list_buckets()
print(f"Found {len(response['Buckets'])} buckets")

2. File-based Credential Access

SSH Git Clone:

import subprocess
import os

# SSH key mounted at /root/.ssh/id_rsa
git_repo = "git@github.com:company/ml-project.git"
clone_path = "/workspace/code"

# Clone repository using SSH key
result = subprocess.run([
    'git', 'clone', git_repo, clone_path
], capture_output=True, text=True)

if result.returncode == 0:
    print("Repository cloned successfully")
else:
    print(f"Clone failed: {result.stderr}")

Google Cloud Authentication:

from google.cloud import storage
import os

# Service account key mounted as file
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/workspace/.config/gcloud/service-account.json'

# Initialize client
client = storage.Client()

# List buckets
buckets = list(client.list_buckets())
print(f"Found {len(buckets)} buckets")

Best Practices

1. Security Guidelines

Principle of Least Privilege:

# Create role-specific credentials
readonly-s3-access:
  Variables:
    AWS_ACCESS_KEY_ID: <readonly-key>
    AWS_SECRET_ACCESS_KEY: <readonly-secret>

admin-s3-access:
  Variables:
    AWS_ACCESS_KEY_ID: <admin-key>
    AWS_SECRET_ACCESS_KEY: <admin-secret>

Credential Scoping:

# Project-scoped for team use
Scope: Project
Usage: Team experiments and shared resources

# Cluster-scoped for system services  
Scope: Cluster
Usage: Infrastructure and monitoring

2. Naming Conventions

Descriptive Names:

# Good examples
wandb-research-team-access
aws-s3-training-data-readonly
github-ssh-deployment-key
postgres-experiment-db-credentials

# Avoid generic names
api-key
secret1
credentials
token

3. Credential Organization

Service-based Grouping:

# Group by service
ml-services/
├── wandb-credentials
├── mlflow-credentials
└── huggingface-credentials

cloud-storage/
├── aws-s3-credentials
├── gcp-storage-credentials
└── azure-blob-credentials

databases/
├── postgres-credentials
├── mongodb-credentials
└── redis-credentials

Monitoring and Auditing

1. Access Monitoring

Track Credential Usage:

# Check which workloads use credentials
kubectl get pods -n runai-<project> -o jsonpath='{.items[*].spec.containers[*].envFrom[*].secretRef.name}'

# Monitor secret access in audit logs
kubectl logs -n kube-system audit-logs | grep "secrets"

2. Credential Health Checks

Validate Credentials:

# Test API key validity
def test_wandb_credentials():
    import wandb
    try:
        wandb.init(project="test-project", mode="disabled")
        return True
    except Exception as e:
        print(f"W&B credentials invalid: {e}")
        return False

# Test AWS credentials
def test_aws_credentials():
    import boto3
    try:
        sts = boto3.client('sts')
        sts.get_caller_identity()
        return True
    except Exception as e:
        print(f"AWS credentials invalid: {e}")
        return False

Troubleshooting

Common Issues

Credential Not Found:

# Check if secret exists
kubectl get secret <secret-name> -n runai-<project>

# Verify credential asset
kubectl get credential <credential-name> -n runai-<project>

Permission Denied:

# Check secret permissions
kubectl describe secret <secret-name> -n runai-<project>

# Verify environment variables in pod
runai exec <workload-name> -- env | grep API

Authentication Failures:

# Test credential access from workload
runai exec <workload-name> -- python -c "import os; print(os.getenv('API_KEY'))"

# Verify file-based credentials
runai exec <workload-name> -- ls -la /root/.ssh/

Debugging Commands

Inspect Credentials in Workload:

# Check environment variables (be careful with sensitive data)
runai exec <workload-name> -- env | grep -E "(KEY|TOKEN|SECRET)"

# Verify mounted credential files
runai exec <workload-name> -- find /workspace -name "*credential*" -o -name "*key*"

# Test service connectivity
runai exec <workload-name> -- curl -H "Authorization: Bearer $API_TOKEN" https://api.service.com/test

Next Steps

1. Audit Current Credentials: Review existing secret management
2. Implement Credential Standards: Create organization-wide credential patterns
3. Set Up Rotation: Establish regular credential update procedures
4. Monitor Usage: Track credential access and validate regularly

Related Assets

• Environments - Include credential-dependent tools
• Data Sources - Secure access to private data
• Compute Resources - Match security requirements with resources