merge_batch_csv Module

The merge_batch_csv module provides functionality for merging CSV files generated from batch processes into a single consolidated CSV file. This is particularly useful for analyzing results that are split across multiple batch output files.

This module provides functionality to merge CSV files named ‘result.csv’ found in directories with specific batch sizes (e.g., folders ending in ‘sz100’, ‘sz200’). The merged result is saved to a specified output file.

QhX.merge_batch_csv.check_endings(root, all_sizes)[source]

Checks if a directory path ends with any of the specified batch size suffixes.

Parameters:

  • root (str) – The directory path to check.

  • all_sizes (list of int) – The list of batch sizes to look for in directory endings.

Returns:

True if the directory ends with one of the specified sizes, False otherwise.

Return type:

bool

QhX.merge_batch_csv.merge_batch_csv(all_sizes=None, directory='.', output_file='merged_result.csv')[source]

Merges CSV files named ‘result.csv’ found in directories ending with szX, where X is a number from all_sizes, into a single CSV file.

Parameters:

  • all_sizes (list of int, optional) – List of all batch sizes whose folders we need to merge. Defaults to [100, 200].

  • directory (str) – The root directory to search for CSV files. Defaults to the current directory.

  • output_file (str) – The name of the output file where the merged results will be saved. Defaults to ‘merged_result.csv’.

Returns:

The number of files successfully merged.

Return type:

int

Usage

The merge_batch_csv function can be executed directly as a standalone script or imported into another Python script.

As a Standalone Script:

Navigate to the package directory and execute:

cd path/to/QhX
python -m merge_batch_csv

Importing in Python Scripts:

from QhX.merge_batch_csv import merge_batch_csv

directory_to_search = "./data"
output_file_name = "final_merged_results.csv"

merge_batch_csv(directory=directory_to_search, output_file=output_file_name)

print("CSV files have been merged successfully.")

Parameters

  • directory (str): The root directory to search for CSV files. Defaults to the current directory.

  • output_file (str): The filename for the merged CSV. Defaults to ‘merged_result.csv’.