Example of using Parallelization module

# Import everything from parallelization_solver submodule
from QhX.parallelization_solver import *

# Import data manager
from QhX.data_manager import DataManager

# Create a data manager instance
data_manager = DataManager()

# Load local copy of data from "https://zenodo.org/record/6878414/files/ForcedSourceTable.parquet"

fs_df = data_manager.load_fs_df("https://zenodo.org/record/6878414/files/ForcedSourceTable.parquet")

# Group the data
fs_gp = data_manager.group_fs_df()

INFO:root:Forced source data loaded successfully.
INFO:root:Forced source data grouped successfully.

Solver Modes: Fixed and Dynamical

This notebook demonstrates how to set up and use the solver in either Fixed or Dynamical mode.

The solver can be configured to process data in two modes: - Fixed Mode: For data with predefined filters and static sampling rates. - Dynamical Mode: For data with variable filters and time-dependent configurations. Note: Please ensure that the correct imports and initializations are used for each mode, as described in the sections below.

Fixed Mode Setup

In Fixed Mode, you can use the default DataManager class to manage the data. This mode does not require additional configuration for time-dependent filters.

Instructions

1. Import the DataManager class from the QhX.data_manager module.

2. Initialize the DataManager as shown in the example below.

Run the code cell below to set up Fixed Mode.

# Example : 2 set IDs , files logged, parallel arithmetic not used
setids = ['1384177', '0458387']

# Create instance of parallel solver without parallel arithmetic and passing ngrid and provided fq values (ntau=80 by default)
solver = ParallelSolver(data_manager=data_manager, delta_seconds=12.0, num_workers=2, log_files=True, parallel_arithmetic=False, ntau=80,ngrid = 100, provided_minfq = 500, provided_maxfq = 10, mode='fixed')

# Run the solver & store results into results.csv
solver.process_ids(setids, 'results.csv')

# Example : 1 ID with parallel arithmetic
setids = ['0458387']

# Create instance of parallel solver with parallel arithmetic and provide parameters
solver = ParallelSolver(data_manager=data_manager, delta_seconds=12.0, num_workers=1, log_files=False, parallel_arithmetic=True, ntau=80,ngrid = 100, provided_minfq = 500, provided_maxfq = 10,mode='fixed')

# Run & store into results2.csv
solver.process_ids(setids, 'results2.csv')

Starting time for ID 0458387 : 2024-04-03 16:12:31.848847

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.9 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  31
Pseudo sample frequency (median) is  0.095
largest tau window is  74.996
0.12 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  31
Pseudo sample frequency (median) is  0.111
largest tau window is  74.996
0.1 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  30
Pseudo sample frequency (median) is  0.111
largest tau window is  77.581
0.1 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.43 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.41 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.54 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.4 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.49 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.39 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.41 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
Processing 0458387
PID 98003
Duration: 1.0 ticks
12.0 seconds each
Time so far 12.0s

0.36 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.36 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.36 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.36 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.36 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.36 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.36 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.36 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.39 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.4 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.4 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
Processing 0458387
PID 98003
Duration: 2.0 ticks
12.0 seconds each
Time so far 24.0s

0.39 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.39 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.38 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

*** Starting Weighted Wavelet Z-transform ***

adjusted time_divisions to:  33
Pseudo sample frequency (median) is  0.111
largest tau window is  70.308
0.37 seconds has passed to complete Weighted Wavelet Z-transform

End time for ID 0458387 : 2024-04-03 16:12:59.790757
Total time : 0:00:27.941910

# Show content of results.csv file - no parallel arithmetic; sets 1384177, 0458387
with open('results.csv') as results_file:
    print(results_file.read())

ID,Sampling_1,Sampling_2,Common period (Band1 & Band1),Upper error bound,Lower error bound,Significance,Band1-Band2
1384177,22.7388695652174,21.9744537815126,335.570469798658,-1,-1,0.98,0-1
1384177,22.7388695652174,19.3330769230769,335.570469798658,-1,-1,1,0-2
1384177,22.7388695652174,22.5428448275862,nan,nan,nan,nan,0-3
1384177,21.9744537815126,19.3330769230769,335.570469798658,-1,-1,1,1-2
1384177,21.9744537815126,22.5428448275862,nan,nan,nan,nan,1-3
1384177,19.3330769230769,22.5428448275862,nan,nan,nan,nan,2-3
458387,70.308125,74.9956666666665,nan,nan,nan,nan,0-1
458387,70.308125,74.9956666666665,nan,nan,nan,nan,0-2
458387,70.308125,77.5813793103449,nan,nan,nan,nan,0-3
458387,74.9956666666665,74.9956666666665,nan,nan,nan,nan,1-2
458387,74.9956666666665,77.5813793103449,nan,nan,nan,nan,1-3
458387,74.9956666666665,77.5813793103449,44.2869796279894,41.6265844303889,0.914820284696262,1,2-3

# Show content of results2.csv file - used parallel arithmetic, set 0458387
with open('results2.csv') as results_file:
    print(results_file.read())

ID,Sampling_1,Sampling_2,Common period (Band1 & Band1),Upper error bound,Lower error bound,Significance,Band1-Band2
0458387,70.30812500000002,74.99566666666651,nan,nan,nan,nan,0-1
0458387,70.30812500000002,74.99566666666651,nan,nan,nan,nan,0-2
0458387,70.30812500000002,77.58137931034484,nan,nan,nan,nan,0-3
0458387,74.99566666666651,74.99566666666651,nan,nan,nan,nan,1-2
0458387,74.99566666666651,77.58137931034484,nan,nan,nan,nan,1-3
0458387,74.99566666666651,77.58137931034484,44.28697962798937,41.626584450805616,0.9148202831698526,0.98,2-3

Dynamical Mode Setup

In Dynamical Mode, the solver can handle datasets with varying filters and dynamic sampling rates. This mode requires the use of DynamicalDataManager from the dynamical_mode module.

For more information on configuring this mode, refer to the Dynamical Mode section in the documentation.