pe.data package

class pe.data.Camelyon17(split='train', root_dir='data', res=64)

Bases: Data

The Camelyon17 dataset.

__init__(split='train', root_dir='data', res=64)

Constructor.

Parameters:

• split (str, optional) – The split of the dataset. It should be either “train”, “val”, or “test”, defaults to “train”

• root_dir (str, optional) – The root directory to save the dataset, defaults to “data”

• res (int, optional) – The resolution of the images, defaults to 64

Raises:

ValueError – If the split is invalid

class pe.data.Cat(root_dir='data', res=512)

Bases: Data

The Cat dataset.

URL = 'https://www.kaggle.com/api/v1/datasets/download/fjxmlzn/cat-cookie-doudou'

The URL of the dataset

__init__(root_dir='data', res=512)

Constructor.

Parameters:

• root_dir (str, optional) – The root directory to save the dataset, defaults to “data”

• res (int, optional) – The resolution of the images, defaults to 512

_download()

Download the dataset if it does not exist.

_read_data()

Read the data from the zip file.

class pe.data.CelebA(root_dir, res=32, attr_name='Male', split='train')

Bases: Data

The CelebA dataset.

__init__(root_dir, res=32, attr_name='Male', split='train')

Constructor.

Parameters:

• root_dir (str) – The root directory of the CelebA dataset

• res (int, optional) – The resolution of the image, defaults to 32

• attr_name (str, optional) – The attribute name to use as the label, defaults to “Male”

• split (str, optional) – The split of the dataset, default is “train”

class pe.data.Cifar10(split='train')

Bases: Data

The CIFAR10 dataset.

__init__(split='train')

Constructor.

Parameters:

split (str, optional) – The split of the dataset. It should be either “train” or “test”, defaults to “train”

Raises:

ValueError – If the split is invalid

class pe.data.Data(data_frame=None, metadata={})

Bases: object

The class that holds the private data or synthetic data from PE.

__init__(data_frame=None, metadata={})

Constructor.

Parameters:

• data_frame (pandas.DataFrame, optional) – A pandas dataframe that holds the data, defaults to None

• metadata (dict, optional) – the metadata of the data, defaults to {}

classmethod concat(data_list, metadata=None)

Concatenate the data frames of a list of data objects

Parameters:

• data_list (list[pe.data.Data]) – The list of data objects to concatenate

• metadata (dict, optional) – The metadata of the concatenated data. When None, the metadata of the list of data objects must be the same and will be used. Defaults to None

Raises:

ValueError – If the metadata of the data objects are not the same

Returns:

The concatenated data object

Return type:

pe.data.Data

filter(filter_criteria)

Filter the data object according to a filter criteria

Parameters:

filter_criteria (dict) – The filter criteria. None means no filter

Returns:

The filtered data object

Return type:

pe.data.Data

filter_label_id(label_id)

Filter the data frame according to a label id

Parameters:

label_id (int) – The label id that is used to filter the data frame

Returns:

pe.data.Data object with the filtered data frame

Return type:

pe.data.Data

load_checkpoint(path)

Load data from a checkpoint

Parameters:

path (str) – The folder that contains the checkpoint

Returns:

Whether the checkpoint is loaded successfully

Return type:

bool

merge(data)

Merge the data object with another data object

Parameters:

data (pe.data.Data) – The data object to merge

Raises:

ValueError – If the metadata of data is not the same as the metadata of the current object

Returns:

The merged data object

Return type:

pe.data.Data

random_split(num_samples_list, seed=0)

Randomly split the data frame into multiple data frames

Parameters:

• num_samples_list (list[int]) – The list of numbers of samples for each data frame

• seed (int, optional) – The seed for the random number generator, defaults to 0

Raises:

ValueError – If the sum of num_samples_list is not equal to the number of samples

Returns:

The list of pe.data.Data objects with the splited data

Return type:

list[pe.data.Data]

random_truncate(num_samples)

Randomly truncate the data frame to a certain number of samples

Parameters:

num_samples (int) – The number of samples to randomly truncate

Returns:

A new pe.data.Data object with the randomly truncated data frame

Return type:

pe.data.Data

reset_index(**kwargs)

Reset the index of the data frame

Parameters:

kwargs (dict) – The keyword arguments to pass to the pandas reset_index function

Returns:

A new pe.data.Data object with the reset index data frame

Return type:

pe.data.Data

save_checkpoint(path)

Save the data to a checkpoint.

Parameters:

path (str) – The folder to save the checkpoint

Raises:

• ValueError – If the path is None

• ValueError – If the data frame is empty

set_label_id(label_id)

Set the label id for the data frame

Parameters:

label_id (int) – The label id to set

split_by_client()

Split the data frame by client ID

Raises:

ValueError – If the client ID column is not in the data frame

Returns:

The list of data objects with the splited data

Return type:

list[pe.data.Data]

split_by_index()

Split the data frame by index

Returns:

The list of data objects with the splited data

Return type:

list[pe.data.Data]

truncate(num_samples)

Truncate the data frame to a certain number of samples

Parameters:

num_samples (int) – The number of samples to truncate

Returns:

A new pe.data.Data object with the truncated data frame

Return type:

pe.data.Data

class pe.data.DigiFace1M(root_dir, res=32, num_processes=50, chunksize=1000)

Bases: Data

The DigiFace1M dataset (https://github.com/microsoft/DigiFace1M/).

__init__(root_dir, res=32, num_processes=50, chunksize=1000)

Constructor.

Parameters:

• root_dir (str, optional) – The root directory of the dataset, defaults to “data”

• res (int, optional) – The resolution of the images, defaults to 32

• num_processes (int, optional) – The number of processes to use for parallel processing, defaults to 50

• chunksize (int, optional) – The chunk size to use for parallel processing, defaults to 1000

Raises:

ValueError – If the number of images in root_dir is not 1,219,995

_read_and_process_image(path)

Read and process an image.

Parameters:

path (str) – The path to the image

Returns:

The processed image

Return type:

np.ndarray

class pe.data.MNIST(split='train')

Bases: Data

The MNIST dataset.

__init__(split='train')

Constructor.

Parameters:

split (str, optional) – The split of the dataset. It should be either “train” or “test”, defaults to “train”

Raises:

ValueError – If the split is invalid

class pe.data.OpenReview(root_dir='data', split='train', **kwargs)

Bases: TextCSV

The OpenReview dataset in the ICML 2024 Spotlight paper, “Differentially Private Synthetic Data via Foundation Model APIs 2: Text” (https://arxiv.org/abs/2403.01749).

DOWNLOAD_INFO_DICT = {'test': ('https://raw.githubusercontent.com/AI-secure/aug-pe/bca21c90921bd1151aa7627e676c906165e205a0/data/openreview/iclr23_reviews_test.csv', 'direct'), 'train': ('https://raw.githubusercontent.com/AI-secure/aug-pe/bca21c90921bd1151aa7627e676c906165e205a0/data/openreview/iclr23_reviews_train.csv', 'direct'), 'val': ('https://raw.githubusercontent.com/AI-secure/aug-pe/bca21c90921bd1151aa7627e676c906165e205a0/data/openreview/iclr23_reviews_val.csv', 'direct')}

The download information for the OpenReview dataset.

__init__(root_dir='data', split='train', **kwargs)

Constructor.

Parameters:

• root_dir (str, optional) – The root directory of the dataset. If the dataset is not there, it will be downloaded automatically. Defaults to “data”

• split (str, optional) – The split of the dataset. It should be either “train”, “val”, or “test”, defaults to “train”

_download(download_info, data_path, processed_data_path)

Download the dataset.

Parameters:

• download_info (pe.data.text.openreview.DownloadInfo) – The download information

• data_path (str) – The path to the raw data

• processed_data_path (str) – The path to the processed data

Raises:

ValueError – If the download type is unknown

class pe.data.PubMed(root_dir='data', split='train', **kwargs)

Bases: TextCSV

The PubMed dataset in the ICML 2024 Spotlight paper, “Differentially Private Synthetic Data via Foundation Model APIs 2: Text” (https://arxiv.org/abs/2403.01749).

DOWNLOAD_INFO_DICT = {'test': ('https://raw.githubusercontent.com/AI-secure/aug-pe/bca21c90921bd1151aa7627e676c906165e205a0/data/pubmed/test.csv', 'direct'), 'train': ('https://drive.google.com/uc?id=12-zV93MQNPvM_ORUoahZ2n4odkkOXD-r', 'gdown'), 'val': ('https://raw.githubusercontent.com/AI-secure/aug-pe/bca21c90921bd1151aa7627e676c906165e205a0/data/pubmed/dev.csv', 'direct')}

The download information for the PubMed dataset.

__init__(root_dir='data', split='train', **kwargs)

Constructor.

Parameters:

• root_dir (str, optional) – The root directory of the dataset. If the dataset is not there, it will be downloaded automatically. Defaults to “data”

• split (str, optional) – The split of the dataset. It should be either “train”, “val”, or “test”, defaults to “train”

_download(download_info, data_path)

Download the dataset.

Parameters:

• download_info (pe.data.text.pubmed.DownloadInfo) – The download information

• data_path (str) – The path to the raw data

Raises:

ValueError – If the download type is unknown

class pe.data.TextCSV(csv_path, label_columns=[], text_column='text', num_samples=None)

Bases: Data

The text dataset in CSV format.

__init__(csv_path, label_columns=[], text_column='text', num_samples=None)

Constructor.

Parameters:

• csv_path (str) – The path to the CSV file

• label_columns (list, optional) – The names of the columns that contain the labels, defaults to []

• text_column (str, optional) – The name of the column that contains the text data, defaults to “text”

• num_samples (int, optional) – The number of samples to load from the CSV file. If None, load all samples. Defaults to None

Raises:

ValueError – If the label columns or text column does not exist in the CSV file

class pe.data.Yelp(root_dir='data', split='train', **kwargs)

Bases: TextCSV

The Yelp dataset in the ICML 2024 Spotlight paper, “Differentially Private Synthetic Data via Foundation Model APIs 2: Text” (https://arxiv.org/abs/2403.01749).

DOWNLOAD_INFO_DICT = {'test': ('https://raw.githubusercontent.com/AI-secure/aug-pe/bca21c90921bd1151aa7627e676c906165e205a0/data/yelp/test.csv', 'direct'), 'train': ('https://drive.google.com/uc?id=1epLuBxCk5MGnm1GiIfLcTcr-tKgjCrc2', 'gdown'), 'val': ('https://raw.githubusercontent.com/AI-secure/aug-pe/bca21c90921bd1151aa7627e676c906165e205a0/data/yelp/dev.csv', 'direct')}

The download information for the Yelp dataset.

__init__(root_dir='data', split='train', **kwargs)

Constructor.

Parameters:

• root_dir (str, optional) – The root directory of the dataset. If the dataset is not there, it will be downloaded automatically. Defaults to “data”

• split (str, optional) – The split of the dataset. It should be either “train”, “val”, or “test”, defaults to “train”

_download(download_info, data_path, processed_data_path)

Download the dataset.

Parameters:

• download_info (pe.data.text.yelp.DownloadInfo) – The download information

• data_path (str) – The path to the raw data

• processed_data_path (str) – The path to the processed data

Raises:

ValueError – If the download type is unknown

pe.data.load_image_folder(path, image_size, class_cond=True, num_images=-1, num_workers=10, batch_size=1000)

Load a image dataset from a folder that contains image files. The folder can be nested arbitrarily. The image file names must be in the format of “{class_name without ‘_’}_{suffix in any string}.ext”. The “ext” can be “jpg”, “jpeg”, “png”, or “gif”. The class names will be extracted from the file names before the first “_”. If class_cond is False, the class names will be ignored and all images will be treated as the same class with class name “None”.

Parameters:

• path (str) – The path to the root folder that contains the image files

• image_size (int) – The size of the images. Images will be resized to this size

• class_cond (bool, optional) – Whether to treat the loaded dataset as class conditional, defaults to True

• num_images (int, optional) – The number of images to load. If -1, load all images. Defaults to -1

• num_workers (int, optional) – The number of workers to use for loading the images, defaults to 10

• batch_size (int, optional) – The batch size to use for loading the images, defaults to 1000

Returns:

The loaded data

Return type:

pe.data.Data

Subpackages

• pe.data.image package
  • Submodules
    • pe.data.image.camelyon17 module
      • Camelyon17
        • Camelyon17.__init__()
    • pe.data.image.cat module
      • Cat
        • Cat.URL
        • Cat.__init__()
        • Cat._download()
        • Cat._read_data()
    • pe.data.image.celeba module
      • CelebA
        • CelebA.__init__()
      • target_transform
    • pe.data.image.cifar10 module
      • Cifar10
        • Cifar10.__init__()
    • pe.data.image.digiface1m module
      • DigiFace1M
        • DigiFace1M.__init__()
        • DigiFace1M._read_and_process_image()
    • pe.data.image.image module
      • ImageDataset
      • _list_image_files_recursively()
      • load_image_folder()
    • pe.data.image.mnist module
      • MNIST
        • MNIST.__init__()
• pe.data.text package
  • Submodules
    • pe.data.text.openreview module
      • DownloadInfo
      • OpenReview
        • OpenReview.DOWNLOAD_INFO_DICT
        • OpenReview.__init__()
        • OpenReview._download()
    • pe.data.text.pubmed module
      • DownloadInfo
      • PubMed
        • PubMed.DOWNLOAD_INFO_DICT
        • PubMed.__init__()
        • PubMed._download()
    • pe.data.text.text_csv module
      • TextCSV
        • TextCSV.__init__()
    • pe.data.text.yelp module
      • DownloadInfo
      • Yelp
        • Yelp.DOWNLOAD_INFO_DICT
        • Yelp.__init__()
        • Yelp._download()

Submodules

• pe.data.data module
  • Data
    • Data.__init__()
    • Data.concat()
    • Data.filter()
    • Data.filter_label_id()
    • Data.load_checkpoint()
    • Data.merge()
    • Data.random_split()
    • Data.random_truncate()
    • Data.reset_index()
    • Data.save_checkpoint()
    • Data.set_label_id()
    • Data.split_by_client()
    • Data.split_by_index()
    • Data.truncate()