Ensure that you have:

1. curl installed
2. Sufficient free disk space for the selected dataset files

Download a file for testing

curl -L -C - -O "<DOWNLOAD_URL from download_links.txt for a file>" 

Step 1: Create a download folder

Open a terminal and run:

 mkdir -p dataset_download
 cd dataset_download

Step 2: Save download_links.txt in the download folder

Download download_links.txt from the CAIB Exploration Portal and save it in the dataset_download folder created above.

Step 3: Download the bulk files

Inside the dataset_download folder, save the code below into a file named download.sh:

#!/bin/bash
















while read -r line; do
  url=$(echo "$line" | sed -n 's/^url: \(.*\) size: .*/\1/p')
  [ -z "$url" ] && continue
  filename=$(basename "$url" | cut -d'?' -f1)
  
  # 1. Skip completely if the final file already exists
  if [ -f "$filename" ]; then
    echo "Already downloaded: $filename"
    continue
  fi
  
  echo "Downloading $filename ..."
  
  # 2. Download to a .part file, then rename when successful
  curl -L -C - -o "${filename}.part" "$url" && mv "${filename}.part" "$filename"
















done < download_links.txt

Open your terminal inside the dataset_download folder and run the following commands:

 chmod +x download.sh
./download.sh

If the process is interrupted, run the same command again.

./download.sh

Troubleshooting

1. Authorization, access-denied, or expiry error: The temporary URL has expired. Generate and download a new download_links.txt file from the CAIB Exploration Portal, then rerun the command.
2. curl: command not found: Install curl using your Linux distribution’s package manager. For Ubuntu or Debian-based systems:

sudo apt install curl

1. Insufficient disk space: Free sufficient storage space before restarting the download.
2. Network interruption: Run the same command again. Completed files will be skipped and partial downloads will resume where supported.

Data-access notice:download_links.txt contains temporary links to CAIB data. Store and share this file only in accordance with the applicable CAIB data-access and data-use requirements.

Ensure that you have:

1. Windows PowerShell
2. curl.exe available (included with most current Windows installations)
3. Sufficient free disk space for the selected dataset files

Download a file for testing

curl.exe -L -C - -O "<DOWNLOAD_URL from download_links.txt for a file>" 

Step 1: Create a download folder

Open PowerShell and run:

New-Item -ItemType Directory -Force -Path "$HOME\dataset_download" | Out-Null
Set-Location "$HOME\dataset_download"

Step 2: Save download_links.txt in the download folder

Download download_links.txt from the CAIB Exploration Portal and save it in the dataset_download folder created above.

Step 3: Download the dataset files

Inside the dataset_download folder, save the code below into a file named download.ps1`:

Get-Content "download_links.txt" | ForEach-Object {
    if ($_ -match '^url:\s*(.+?)\s+size:') {
        $url = $matches[1]
        $filename = ($url.Split('?')[0] -split '/')[ -1 ]
        Write-Host "Downloading $filename ..."
































        curl.exe -L -C - -o $filename $url
    }
}

In the PowerShell go to dataset_download folder and run the following commands:

.\download.ps1

If the process is interrupted, run the same command again.

.\download.ps1

Resuming or repeating a download

If the process is interrupted, run the same command again.

1. Successfully completed files will be skipped.
2. Partially downloaded .part files will resume where possible, provided that the download server supports resumable downloads.
3. If resuming is not supported, delete the affected .part file and rerun the command to restart that file.

Troubleshooting

1. Authorization, access-denied, or expiry error: The temporary URL has expired. Generate and download a new download_links.txt file from the CAIB Exploration Portal, then rerun the command.
2. curl.exe is not recognized: Use the Python download method, or ask your local IT support to install or enable curl.
3. Insufficient disk space: Free sufficient storage space before restarting the download.
4. Network interruption: Run the same command again. Completed files will be skipped and partial downloads will resume where supported.

Ensure that you have:

1. Python 3 installed
2. Sufficient free disk space for the selected dataset files
3. A stable internet connection
4. The required Python packages installed. Open your terminal or PowerShell and run:

pip install requests tqdm

Step 1: Create a download folder

1. Create a folder named dataset_download at a suitable location on your computer.
2. Save both of the following files in this folder:
3. download_links.txt, downloaded from the CAIB Exploration Portal.
4. The Python script created in Step 2.
5. Note: download_links.txt contains one download per line, formatted with the temporary download URL and file size.

Step 2: Create the Python download script

Open a text editor, copy the following code, and save it as download_caib.py:

import os
import requests
import re
from tqdm import tqdm
















# Open and read the links file
try:
    with open('download_links.txt', 'r') as file:
        lines = file.readlines()
except FileNotFoundError:
    print("Error: download_links.txt not found. Ensure it is in the same folder as this script.")
    exit()
















for line in lines:
    line = line.strip()
    if not line:
        continue
    
    # Extract URL using regular expression
    match = re.search(r'url:\s*(.+?)\s+size:', line)
    if not match:
        print(f"Skipping invalid row: {line}")
        continue
        
    url = match.group(1)
    # Extract filename from the URL, ignoring query parameters
    filename = url.split('?')[0].split('/')[-1]
















    # Setup resume byte position if the file already exists locally
    resume_byte_pos = os.path.getsize(filename) if os.path.exists(filename) else 0
    headers = {"Range": f"bytes={resume_byte_pos}-"} if resume_byte_pos else {}
















    try:
        with requests.get(url, headers=headers, stream=True) as r:
            # Handle standard HTTP responses
            if r.status_code not in (200, 206):
                if r.status_code == 416:
                    # HTTP 416 means Requested Range Not Satisfiable (file is completely downloaded)
                    print(f"Already downloaded: {filename}")
                else:
                    print(f"Error accessing {filename}: HTTP {r.status_code}")
                continue
















            total_size = int(r.headers.get("Content-Length", 0)) + resume_byte_pos
















            # Double-check if the existing file is already exactly the target size
            if resume_byte_pos == total_size and total_size > 0:
                print(f"Already downloaded: {filename}")
                continue
















            print(f"Starting download: {filename} ({total_size / (1024**2):.2f} MB)")
            
            # 'ab' appends to existing file (resume), 'wb' writes a new file
            mode = "ab" if resume_byte_pos else "wb"
            
            with open(filename, mode) as f, tqdm(
                total=total_size,
                unit='B',
                unit_scale=True,
                unit_divisor=1024,
                initial=resume_byte_pos,
                desc=filename
            ) as bar:
                for chunk in r.iter_content(1024*1024):
                    if chunk:
                        f.write(chunk)
                        bar.update(len(chunk))
                        
    except Exception as e:
        print(f"Network error or interruption for {filename}: {e}")

Step 3: Run the download script Open a terminal or PowerShell window in the dataset_download folder.

python download_caib.py

(If this does not work, try python3 download_caib.py)

The script downloads files directly into the dataset_download folder.

Resuming or repeating a download If the process is interrupted, run the same command again.

1. Successfully completed files will be skipped automatically.
2. Partially downloaded files will resume exactly where they left off, provided that the download server supports resumable downloads.
3. If resuming is not supported, the script restarts that file automatically.

Troubleshooting

1. Authorization, access-denied, or expiry error: The temporary URL has expired. Generate and download a new download_links.txt file from the CAIB Exploration Portal, then rerun the script.
2. python command not found: Install Python 3, then reopen the terminal or PowerShell window and run the command again.
3. ModuleNotFoundError: Ensure you have installed the required dependencies by running pip install requests tqdm.
4. Insufficient disk space: Free sufficient storage space before restarting the download.
5. Network interruption: Run the same command again. Completed files will be skipped and partial downloads will resume where supported.
6. download_links.txt was not found: Ensure that download_links.txt and download_caib.py are saved in the exact same dataset_download folder.

The intended audience is medical personnel, scientists, bioinformaticians and other technical individuals who would be involved in either downloading, managing, using or analyzing the data they download from CAIB.

Organization → Patient → Cancer Type → Episode/Timeline → Study (Pathology/Radiology) → Image/Report → Annotation/Label

 [CAIB] - [T] [00004390] [OC] [01] [R] [02] [P] [02] [02] [HE]

┌──────────────────────────────────────────────────────────────────────────────┐

│ SLIDE NAME (Decoded)                            │

├──────────────────────────────────────────────────────────────────────────────┤

│ Raw Slide ID : CAIB-T00004390OC01R02P0202HE                 │

│ Patient_ID  : CAIB-T00004390OC     (CAIB → Project/Site)        │

│ Path_ID   : CAIB-T00004390OC01R02   (CAIB → Procedure+PathSerial)    │

├──────────────────────────────────────────────────────────────────────────────┤

│ [CAIB] - [T] [00004390] [OC] [01] [R] [02] [P] [02] [02] [HE]            │

│  │    │     │     │  │ │ │ │ │ │ │ └── Stain         │

│  │    │     │     │  │ │ │ │ │ │ └── BlockNo (00-99)      │

│  │    │     │     │  │ │ │ │ │ └──SlideNo (00-99)      │

│  │    │     │     │  │ │ │ │ └──TissueClass (P)     │

│  │    │     │     │  │ │ │ │ └──P = Primary    │

│  │    │     │     │  │ │ │ │ └──R = Regional    │

│  │    │     │     │  │ │ │ │ └──N = Non-Regional Node │

│  │    │     │     │  │ │ │ │ └──D = Distant metastasis   │

│  │    │     │     │  │ │ │ │ └──S = Second Primary   │

│  │    │     │     │  │ │ │ │ └──O = Others (mostly referring to non-neoplastic pathologies)    │

│  │    │     │     │  │ │ │ └──Serial number(00–99)  │

│  │    │     │     │  │ │ └──Type of Specimen

│  │    │     │     │  │ │ └──R = Resection

│  │    │     │     │  │ │ └──B = Biopsy

│  │    │     │     │  │ └──Presentation

│  │    │     │     │  │ └──01 = First Presentation │

│  │    │     │     │  │ └──02 = Second Presentation │

│  │    │     │     │  │ └──........

│  │    │     │     │  │ └── n = nth Presentation │

│  │    │     │     │  └── Project/Cancer Type│

│  │    │     │     │  └── OC = Oral Cancer     │

│  │    │     │     │  └── LC = Lung Cancer     │

│  │    │     │     └── Patient serial       │

│  │    │     └── Org code:

│           └── T → TMC                      │

│           └── Org code: A → AIIMS                     │

│     └── Org code: P → PGIMER                     │

│         └── Org code: R → Rajiv Gandhi                  │

└── Program/Consortium: CAIB │

Pathology WSI: format (SVS/TIFF/DICOM-WSI), pyramid levels, scanner metadata retained/stripped

 (e.g., H&N: tumor/normal/dysplasia; lymph node: metastasis/ENE/pN)