Moving Data & Transfer Methods

Into the Cluster

When transferring data between Berkelium cluster storage and external storage (including your local workstation or remote servers), the best approach depends on your data volume, file count, and whether the external storage is accessible from inside the cluster.

Because Kubernetes pods use internal cluster addresses and are not directly accessible from outside, your main options are accessing cluster-provided object storage (S3) or pulling data from inside a running pod with a mounted persistent storage volume (CephFS).

Using S3 Object Storage

Using S3 object storage (ceph.lbl.gov) is the most scalable method and supports transferring the largest data volumes and file counts. You can use standard S3 clients (such as rclone, aws-cli, or s5cmd) from both outside and inside the cluster to upload and download datasets.

Pulling Data from Inside a Pod

If your dataset is hosted on external storage reachable over the network (such as a cloud provider, HTTP server, or remote SSH/SFTP host), you can pull the data directly into your pod’s mounted CephFS PVC. Depending on the dataset size, you can either:

• Run an interactive pod, kubectl exec into it, and manually run your transfer command after configuring credentials.
• Run a batch Job to perform the transfer automatically. A Job is recommended for large datasets because you do not need to keep a terminal session open, and Kubernetes will automatically retry the pod if the transfer is interrupted.

Common tools you can run inside your pod include:

• rclone: Supports dozens of cloud and institutional storage backends (you can copy or mount the config file generated on your local machine).
• scp / rsync: For pulling over SSH (using an SSH key mounted from a Kubernetes Secret).
• wget / curl: For downloading datasets from HTTP/HTTPS servers.

Providing Credentials via Kubernetes Secrets

If your transfer tool requires credentials or an SSH/API key file, store them in a Kubernetes Secret rather than hardcoding them in your manifest or container image.

Create a secret in your namespace from a local file:

kubectl create -n <your-namespace> secret generic my-secret --from-file=secret.key

Verify that the secret was created:

kubectl get -n <your-namespace> secrets

Mount the secret into your pod as a directory:

    volumeMounts:
      - mountPath: /secrets
        name: sec
  volumes:
    - name: sec
      secret:
        secretName: my-secret

Or mount a single key file using subPath:

    volumeMounts:
      - mountPath: /secrets/secret.key
        subPath: secret.key
        name: sec
  volumes:
    - name: sec
      secret:
        secretName: my-secret

Or expose a secret value as an environment variable:

kubectl create -n <your-namespace> secret generic my-secret --from-literal=pass=my-super-pass

    env:
      - name: DB_PASS
        valueFrom:
          secretKeyRef:
            key: pass
            name: my-secret

kubectl cp Command

For small configuration files or scripts, the most straightforward way to copy files to or from a running pod is kubectl cp:

# Copy a local file into a pod
kubectl -n <your-namespace> cp ~/tmp/file.dat my-pod:/mnt/shared/file.dat

# Copy a file from a pod to your local workstation
kubectl -n <your-namespace> cp my-pod:/mnt/shared/file.dat ~/tmp/file.dat

Do Not Use kubectl cp for Large Datasets

You should NOT use kubectl cp for large amounts of data. All traffic transferred via kubectl cp flows through the Kubernetes API management server, which is not designed for bulk data transfer and will degrade cluster API performance if used for more than a few megabytes.

---

Inside the Cluster

To copy data from one PersistentVolumeClaim (PVC) to another, you can use an interactive pod or a batch Job.

Below is an example of a Job using the gsutil image to synchronize data in parallel between two mounted PVCs:

apiVersion: batch/v1
kind: Job
metadata:
  name: copy-data
  namespace: <your-namespace>
spec:
  template:
    spec:
      containers:
        - name: backup
          image: gitlab-registry.nrp-nautilus.io/prp/gsutil:latest
          imagePullPolicy: Always
          command:
            - bash
            - -c
            - "gsutil -m rsync -e -r -P /from /to/"
          volumeMounts:
            - mountPath: /from
              name: source
            - mountPath: /to
              name: target
          resources:
            limits:
              cpu: "4"
              memory: 4Gi
            requests:
              cpu: "4"
              memory: 4Gi
      restartPolicy: Never
      volumes:
        - name: target
          persistentVolumeClaim:
            claimName: <target-pvc>
        - name: source
          persistentVolumeClaim:
            claimName: <source-pvc>
            readOnly: true
  backoffLimit: 1