prediction-20241217-160934-selective.txt

I’m encountering an error in my script, and I’ve included the output along with all related project scripts below. Please review and provide a complete fix. Ensure your response includes a fully functional, error-free, and deployment-ready script, with no placeholders or omissions. Additionally, any revised code should be as compact as possible, without remarks or docstrings, while maintaining full functionality, compatibility, and interoperability. If no changes are needed, there’s no need to include the script in your response.

See the error I receive here:

====================

2024-12-17 16:00:48,847 - INFO - Preparing data (scaling features, etc.)...
2024-12-17 16:00:48,866 - INFO - Data preparation done in 0.02 seconds. Features count: 100
2024-12-17 16:00:48,866 - INFO - Identifying original indicators...
2024-12-17 16:00:48,892 - INFO - Original indicators count: 95
2024-12-17 16:00:48,892 - INFO - Max lag calculated as 2541
2024-12-17 16:00:48,893 - INFO - Loading or calculating correlations. This may take a while...
2024-12-17 16:00:48,893 - INFO - [INFO] Calculating correlations for symbol='BTCUSDT', timeframe='1d'...
2024-12-17 16:00:48,894 - INFO - [INFO] Processing indicator 1/95: 'TSI_13_25_13'
2024-12-17 16:00:48,894 - INFO - [ERROR] Exception occurred during correlation calculations: no such column: indicator_id
2024-12-17 16:00:48,894 - INFO - Traceback (most recent call last):
2024-12-17 16:00:48,894 - INFO - File "c:\code\prediction\launch.py", line 43, in <module>
2024-12-17 16:00:48,894 - INFO - runpy.run_path("start.py", run_name="__main__")
2024-12-17 16:00:48,899 - INFO - File "C:\Users\jkoka\.pyenv\pyenv-win\versions\3.10.0\lib\runpy.py", line 269, in run_path
2024-12-17 16:00:48,899 - INFO - return _run_module_code(code, init_globals, run_name,
2024-12-17 16:00:48,899 - INFO - File "C:\Users\jkoka\.pyenv\pyenv-win\versions\3.10.0\lib\runpy.py", line 96, in _run_module_code
2024-12-17 16:00:48,901 - INFO - _run_code(code, mod_globals, init_globals,
2024-12-17 16:00:48,901 - INFO - File "C:\Users\jkoka\.pyenv\pyenv-win\versions\3.10.0\lib\runpy.py", line 86, in _run_code
2024-12-17 16:00:48,902 - INFO - exec(code, run_globals)
2024-12-17 16:00:48,903 - INFO - File "start.py", line 484, in <module>
2024-12-17 16:00:48,903 - INFO - main()
2024-12-17 16:00:48,903 - INFO - File "start.py", line 333, in main
2024-12-17 16:00:48,903 - INFO - load_or_calculate_correlations(
2024-12-17 16:00:48,905 - INFO - File "c:\code\prediction\correlation_utils.py", line 159, in load_or_calculate_correlations
2024-12-17 16:00:48,906 - INFO - cursor.execute("""
2024-12-17 16:00:48,907 - INFO - sqlite3
2024-12-17 16:00:48,907 - INFO - .
2024-12-17 16:00:48,907 - INFO - OperationalError
2024-12-17 16:00:48,907 - INFO - :
2024-12-17 16:00:48,907 - INFO - no such column: indicator_id


====================

I've listed all the scripts in this project here:

====================


config.py:
# config.py
import os
SCRIPT_DIR=os.path.dirname(os.path.abspath(__file__))
DATABASE_DIR=os.path.join(SCRIPT_DIR,'database')
DB_FILENAME='klines.db'
DB_PATH=os.path.join(DATABASE_DIR,DB_FILENAME)
os.makedirs(DATABASE_DIR,exist_ok=True)
====================
correlation_database.py:
# correlation_database.py
import sqlite3

class CorrelationDatabase:
    def __init__(self,db_path:str):
        self.db_path=db_path
        self.connection=sqlite3.connect(self.db_path)
        self.create_table()

    def create_table(self)->None:
        create_table_query="""
        CREATE TABLE IF NOT EXISTS correlations (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            symbol_id INTEGER NOT NULL,
            timeframe_id INTEGER NOT NULL,
            indicator_id INTEGER NOT NULL,
            lag INTEGER NOT NULL,
            correlation_value REAL NOT NULL,
            FOREIGN KEY (symbol_id) REFERENCES symbols(id),
            FOREIGN KEY (timeframe_id) REFERENCES timeframes(id),
            FOREIGN KEY (indicator_id) REFERENCES indicators(id),
            UNIQUE(symbol_id, timeframe_id, indicator_id, lag)
        );
        """
        create_index_queries=["CREATE INDEX IF NOT EXISTS idx_correlations_symbol_timeframe_indicator_lag ON correlations (symbol_id, timeframe_id, indicator_id, lag);"]
        cursor=self.connection.cursor()
        cursor.execute(create_table_query)
        for query in create_index_queries:
            cursor.execute(query)
        self.connection.commit()

    def insert_correlation(self,symbol:str,timeframe:str,indicator_name:str,lag:int,correlation_value:float)->None:
        cursor=self.connection.cursor()
        cursor.execute("SELECT id FROM symbols WHERE symbol = ?",(symbol,))
        result=cursor.fetchone()
        if result:
            symbol_id=result[0]
        else:
            cursor.execute("INSERT INTO symbols (symbol) VALUES (?)",(symbol,))
            symbol_id=cursor.lastrowid
        cursor.execute("SELECT id FROM timeframes WHERE timeframe = ?",(timeframe,))
        result=cursor.fetchone()
        if result:
            timeframe_id=result[0]
        else:
            cursor.execute("INSERT INTO timeframes (timeframe) VALUES (?)",(timeframe,))
            timeframe_id=cursor.lastrowid
        cursor.execute("SELECT id FROM indicators WHERE name = ?",(indicator_name,))
        result=cursor.fetchone()
        if result:
            indicator_id=result[0]
        else:
            cursor.execute("INSERT INTO indicators (name) VALUES (?)",(indicator_name,))
            indicator_id=cursor.lastrowid
        insert_query="""
        INSERT OR REPLACE INTO correlations (
            symbol_id, timeframe_id, indicator_id, lag, correlation_value
        ) VALUES (?, ?, ?, ?, ?);
        """
        try:
            cursor.execute(insert_query,(symbol_id,timeframe_id,indicator_id,lag,correlation_value))
            self.connection.commit()
        except sqlite3.Error:
            pass

    def get_correlation(self,symbol:str,timeframe:str,indicator_name:str,lag:int):
        cursor=self.connection.cursor()
        cursor.execute("SELECT id FROM symbols WHERE symbol = ?",(symbol,))
        result=cursor.fetchone()
        if result:
            symbol_id=result[0]
        else:
            return None
        cursor.execute("SELECT id FROM timeframes WHERE timeframe = ?",(timeframe,))
        result=cursor.fetchone()
        if result:
            timeframe_id=result[0]
        else:
            return None
        cursor.execute("SELECT id FROM indicators WHERE name = ?",(indicator_name,))
        result=cursor.fetchone()
        if result:
            indicator_id=result[0]
        else:
            return None
        select_query="""
        SELECT correlation_value FROM correlations
        WHERE symbol_id = ? AND timeframe_id = ? AND indicator_id = ? AND lag = ?;
        """
        cursor.execute(select_query,(symbol_id,timeframe_id,indicator_id,lag))
        result=cursor.fetchone()
        return result[0] if result else None

    def get_all_correlations(self):
        select_query="""
        SELECT symbols.symbol, timeframes.timeframe, indicators.name, correlations.lag, correlations.correlation_value
        FROM correlations
        JOIN symbols ON correlations.symbol_id = symbols.id
        JOIN timeframes ON correlations.timeframe_id = timeframes.id
        JOIN indicators ON correlations.indicator_id = indicators.id;
        """
        cursor=self.connection.cursor()
        cursor.execute(select_query)
        return cursor.fetchall()

    def close(self)->None:
        self.connection.close()
====================
correlation_utils.py:
# correlation_utils.py

import os
import pandas as pd
import numpy as np
import sqlite3
from typing import List
from sklearn.preprocessing import StandardScaler
from config import DB_PATH

def calculate_correlation(data: pd.DataFrame, indicator_name: str, lag: int, is_reverse_chronological: bool) -> float:
    """
    Calculate the Pearson correlation between the specified indicator and the 'Close' price,
    shifted by the specified lag.

    Args:
        data (pd.DataFrame): The DataFrame containing indicator and price data.
        indicator_name (str): The name of the indicator column.
        lag (int): The lag period.
        is_reverse_chronological (bool): If True, indicates data is in reverse chronological order.

    Returns:
        float: The Pearson correlation coefficient.
    """
    if indicator_name.lower() == 'close':
        return np.nan

    try:
        shift_value = lag if is_reverse_chronological else -lag
        shifted_col = data[indicator_name].shift(shift_value)

        valid_data = pd.concat([shifted_col, data['Close']], axis=1).dropna()

        if not valid_data.empty:
            corr = valid_data[indicator_name].corr(valid_data['Close'])
            return corr
        else:
            return np.nan
    except:
        return np.nan

def is_valid_indicator(series: pd.Series) -> bool:
    """
    Validate if the indicator series is suitable for correlation calculation.

    Args:
        series (pd.Series): The indicator data series.

    Returns:
        bool: True if valid, False otherwise.
    """
    if series.isna().all():
        return False
    if series.nunique() <= 1:
        return False
    return True

def get_symbol_id(conn: sqlite3.Connection, symbol: str) -> int:
    """
    Retrieve the symbol ID from the database, inserting it if it doesn't exist.

    Args:
        conn (sqlite3.Connection): The SQLite connection object.
        symbol (str): The trading symbol.

    Returns:
        int: The symbol ID.
    """
    cursor = conn.cursor()
    cursor.execute("SELECT id FROM symbols WHERE symbol = ?", (symbol,))
    result = cursor.fetchone()
    if result:
        return result[0]
    else:
        cursor.execute("INSERT INTO symbols (symbol) VALUES (?)", (symbol,))
        conn.commit()
        return cursor.lastrowid

def get_timeframe_id(conn: sqlite3.Connection, timeframe: str) -> int:
    """
    Retrieve the timeframe ID from the database, inserting it if it doesn't exist.

    Args:
        conn (sqlite3.Connection): The SQLite connection object.
        timeframe (str): The timeframe string.

    Returns:
        int: The timeframe ID.
    """
    cursor = conn.cursor()
    cursor.execute("SELECT id FROM timeframes WHERE timeframe = ?", (timeframe,))
    result = cursor.fetchone()
    if result:
        return result[0]
    else:
        cursor.execute("INSERT INTO timeframes (timeframe) VALUES (?)", (timeframe,))
        conn.commit()
        return cursor.lastrowid

def get_indicator_id(conn: sqlite3.Connection, indicator_name: str) -> int:
    """
    Retrieve the indicator ID from the database, inserting it if it doesn't exist.

    Args:
        conn (sqlite3.Connection): The SQLite connection object.
        indicator_name (str): The name of the indicator.

    Returns:
        int: The indicator ID.
    """
    cursor = conn.cursor()
    cursor.execute("SELECT id FROM indicators WHERE name = ?", (indicator_name,))
    result = cursor.fetchone()
    if result:
        return result[0]
    else:
        cursor.execute("INSERT INTO indicators (name) VALUES (?)", (indicator_name,))
        conn.commit()
        return cursor.lastrowid

def load_or_calculate_correlations(
    data: pd.DataFrame,
    original_indicators: List[str],
    max_lag: int,
    is_reverse_chronological: bool,
    symbol: str,
    timeframe: str
) -> None:
    """
    Load existing correlations from the database or calculate them if they don't exist.

    Args:
        data (pd.DataFrame): The DataFrame containing indicator and price data.
        original_indicators (List[str]): List of indicator names.
        max_lag (int): The maximum number of lag periods.
        is_reverse_chronological (bool): If True, data is in reverse chronological order.
        symbol (str): The trading symbol.
        timeframe (str): The timeframe string.

    Returns:
        None
    """
    print(f"[INFO] Calculating correlations for symbol='{symbol}', timeframe='{timeframe}'...")
    try:
        conn = sqlite3.connect(DB_PATH)
        cursor = conn.cursor()

        symbol_id = get_symbol_id(conn, symbol)
        timeframe_id = get_timeframe_id(conn, timeframe)

        for i, indicator_name in enumerate(original_indicators, start=1):
            print(f"[INFO] Processing indicator {i}/{len(original_indicators)}: '{indicator_name}'")
            indicator_id = get_indicator_id(conn, indicator_name)

            if not is_valid_indicator(data[indicator_name]):
                print(f"[WARN] Indicator '{indicator_name}' is invalid. Skipping.")
                continue

            cursor.execute("""
                SELECT lag FROM correlations
                WHERE symbol_id = ? AND timeframe_id = ? AND indicator_id = ?
            """, (symbol_id, timeframe_id, indicator_id))
            existing_lags = set(row[0] for row in cursor.fetchall())

            lags_to_calculate = [lag for lag in range(1, max_lag + 1) if lag not in existing_lags]
            print(f"[INFO] {len(lags_to_calculate)} lags to calculate for '{indicator_name}'")

            for lag in lags_to_calculate:
                corr_value = calculate_correlation(data, indicator_name, lag, is_reverse_chronological)
                try:
                    cursor.execute("""
                        INSERT INTO correlations (symbol_id, timeframe_id, indicator_id, lag, correlation_value)
                        VALUES (?, ?, ?, ?, ?)
                    """, (symbol_id, timeframe_id, indicator_id, lag, corr_value))
                except sqlite3.Error as e:
                    print(f"[ERROR] Failed to insert correlation for '{indicator_name}', lag {lag}: {e}")

            conn.commit()
            print(f"[INFO] Completed correlations for indicator '{indicator_name}'")

        conn.close()
        print("[INFO] All correlations processed successfully.")
    except Exception as e:
        print(f"[ERROR] Exception occurred during correlation calculations: {e}")
        if conn:
            conn.close()
        raise

def get_all_correlations(
    conn: sqlite3.Connection,
    symbol_id: int,
    timeframe_id: int,
    indicator_id: int,
    max_lag: int
) -> List[float]:
    """
    Retrieve all correlation values for a given indicator and symbol/timeframe up to max_lag.

    Args:
        conn (sqlite3.Connection): The SQLite connection object.
        symbol_id (int): The symbol ID.
        timeframe_id (int): The timeframe ID.
        indicator_id (int): The indicator ID.
        max_lag (int): The maximum lag to retrieve.

    Returns:
        List[float]: List of correlation values ordered by lag.
    """
    cursor = conn.cursor()
    cursor.execute("""
        SELECT lag, correlation_value FROM correlations
        WHERE symbol_id = ? AND timeframe_id = ? AND indicator_id = ?
        AND lag BETWEEN 1 AND ?
        ORDER BY lag ASC
    """, (symbol_id, timeframe_id, indicator_id, max_lag))
    rows = cursor.fetchall()
    correlations = [row[1] for row in rows]
    print(f"[DEBUG] get_all_correlations: Retrieved {len(correlations)} correlations for indicator_id={indicator_id}")
    return correlations
====================
data_utils.py:
# data_utils.py
import os
import sys
import pandas as pd
from sklearn.preprocessing import StandardScaler
from typing import Tuple,List
from datetime import datetime
from joblib import Parallel,delayed
import numpy as np
from load_data import load_data
from indicators import compute_all_indicators

def clear_screen()->None:
    os.system('cls' if os.name=='nt' else 'clear')

def prepare_data(data:pd.DataFrame)->Tuple[pd.DataFrame,List[str]]:
    feature_columns=data.columns.difference(['date','open','high','low','close','volume'])
    X=data[feature_columns]
    numeric_features=X.select_dtypes(include=[np.number]).columns.tolist()
    non_numeric_features=set(feature_columns)-set(numeric_features)
    if non_numeric_features:
        feature_columns=numeric_features
        X=X[feature_columns]
    scaler=StandardScaler()
    X_scaled_array=scaler.fit_transform(X)
    X_scaled_df=pd.DataFrame(X_scaled_array,columns=feature_columns,index=data.index)
    return X_scaled_df,feature_columns

def determine_time_interval(data:pd.DataFrame)->str:
    date_col='Date' if 'Date' in data.columns else 'open_time' if 'open_time' in data.columns else None
    if date_col is None:
        raise ValueError("No 'Date' or 'open_time' column found in data.")
    data[date_col]=pd.to_datetime(data[date_col],errors='coerce')
    data.dropna(subset=[date_col],inplace=True)
    if not data[date_col].is_monotonic_increasing:
        raise ValueError(f"The '{date_col}' column is not monotonically increasing.")
    time_diffs=data[date_col].diff().dropna().dt.total_seconds()
    if time_diffs.empty:
        raise ValueError("No time differences found.")
    most_common_diff=time_diffs.mode().iloc[0]
    if most_common_diff<60:
        return 'second'
    elif most_common_diff<3600:
        return 'minute'
    elif most_common_diff<86400:
        return 'hour'
    elif most_common_diff<604800:
        return 'day'
    else:
        return 'week'

def get_original_indicators(feature_names:List[str],data:pd.DataFrame)->List[str]:
    original_indicators=[col for col in feature_names if col.lower()not in['open','high','low','close','volume']and data[col].notna().any()and data[col].var()>1e-6]
    return original_indicators

def handle_missing_indicators(original_indicators:List[str],data:pd.DataFrame,expected_indicators:List[str])->List[str]:
    missing_indicators=[indicator for indicator in expected_indicators if indicator not in data.columns]
    for indicator in missing_indicators:
        original_indicators=[col for col in original_indicators if col!=indicator]
    return original_indicators
====================
indicators.py:
# indicators.py
import pandas as pd
import numpy as np
import talib as ta
import pandas_ta as pta

def compute_obv_price_divergence(data,method="Difference",obv_method="SMA",obv_period=14,price_input_type="OHLC/4",price_method="SMA",price_period=14,bearish_threshold=-0.8,bullish_threshold=0.8,smoothing=0.01):
    if price_input_type.lower()=="close":
        selected_price=data['close']
    elif price_input_type.lower()=="open":
        selected_price=data['open']
    elif price_input_type.lower()=="high":
        selected_price=data['high']
    elif price_input_type.lower()=="low":
        selected_price=data['low']
    elif price_input_type.lower()=="hl/2":
        selected_price=(data['high']+data['low'])/2
    elif price_input_type.lower()=="ohlc/4":
        selected_price=(data['open']+data['high']+data['low']+data['close'])/4
    else:
        raise ValueError(f"Unsupported price input type: {price_input_type}")
    obv=ta.OBV(data['close'],data['volume'])
    if obv_method=="SMA":
        obv_ma=ta.SMA(obv,timeperiod=obv_period)
    elif obv_method=="EMA":
        obv_ma=ta.EMA(obv,timeperiod=obv_period)
    else:
        obv_ma=obv
    if price_method=="SMA":
        price_ma=ta.SMA(selected_price,timeperiod=price_period)
    elif price_method=="EMA":
        price_ma=ta.EMA(selected_price,timeperiod=price_period)
    else:
        price_ma=selected_price
    obv_change_percent=(obv_ma - obv_ma.shift(1))/obv_ma.shift(1)*100
    price_change_percent=(price_ma - price_ma.shift(1))/price_ma.shift(1)*100
    if method=="Difference":
        metric=obv_change_percent - price_change_percent
    elif method=="Ratio":
        metric=obv_change_percent/np.maximum(smoothing,np.abs(price_change_percent))
    elif method=="Log Ratio":
        metric=np.log(np.maximum(smoothing,np.abs(obv_change_percent))/np.maximum(smoothing,np.abs(price_change_percent)))
    else:
        raise ValueError(f"Unsupported method: {method}")
    data['obv_price_divergence']=metric
    return data

def compute_all_indicators(data):
    indicators={}
    indicators['bbands_upper'],indicators['bbands_middle'],indicators['bbands_lower']=ta.BBANDS(data['close'],timeperiod=5,nbdevup=2,nbdevdn=2,matype=0)
    indicators['dema']=ta.DEMA(data['close'],timeperiod=30)
    indicators['ema']=ta.EMA(data['close'],timeperiod=30)
    indicators['ht_trendline']=ta.HT_TRENDLINE(data['close'])
    indicators['kama']=ta.KAMA(data['close'],timeperiod=30)
    indicators['ma']=ta.MA(data['close'],timeperiod=30,matype=0)
    indicators['mama'],indicators['fama']=ta.MAMA(data['close'],fastlimit=0.5,slowlimit=0.05)
    indicators['midpoint']=ta.MIDPOINT(data['close'],timeperiod=14)
    indicators['midprice']=ta.MIDPRICE(data['high'],data['low'],timeperiod=14)
    indicators['sar']=ta.SAR(data['high'],data['low'],acceleration=0.02,maximum=0.2)
    indicators['sma']=ta.SMA(data['close'],timeperiod=30)
    indicators['t3']=ta.T3(data['close'],timeperiod=5,vfactor=0.7)
    indicators['tema']=ta.TEMA(data['close'],timeperiod=30)
    indicators['trima']=ta.TRIMA(data['close'],timeperiod=30)
    indicators['wma']=ta.WMA(data['close'],timeperiod=30)
    indicators['adx']=ta.ADX(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['adxr']=ta.ADXR(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['apo']=ta.APO(data['close'],fastperiod=12,slowperiod=26,matype=0)
    indicators['aroon_down'],indicators['aroon_up']=ta.AROON(data['high'],data['low'],timeperiod=14)
    indicators['aroonosc']=ta.AROONOSC(data['high'],data['low'],timeperiod=14)
    indicators['bop']=ta.BOP(data['open'],data['high'],data['low'],data['close'])
    indicators['cci']=ta.CCI(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['cmo']=ta.CMO(data['close'],timeperiod=14)
    indicators['dx']=ta.DX(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['macd'],indicators['macd_signal'],indicators['macd_hist']=ta.MACD(data['close'],fastperiod=12,slowperiod=26,signalperiod=9)
    indicators['macdext'],indicators['macdext_signal'],indicators['macdext_hist']=ta.MACDEXT(data['close'],fastperiod=12,fastmatype=0,slowperiod=26,slowmatype=0,signalperiod=9,signalmatype=0)
    indicators['macdfix'],indicators['macdfix_signal'],indicators['macdfix_hist']=ta.MACDFIX(data['close'],signalperiod=9)
    indicators['minus_di']=ta.MINUS_DI(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['minus_dm']=ta.MINUS_DM(data['high'],data['low'],timeperiod=14)
    indicators['mom']=ta.MOM(data['close'],timeperiod=10)
    indicators['plus_di']=ta.PLUS_DI(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['plus_dm']=ta.PLUS_DM(data['high'],data['low'],timeperiod=14)
    indicators['ppo']=ta.PPO(data['close'],fastperiod=12,slowperiod=26,matype=0)
    indicators['roc']=ta.ROC(data['close'],timeperiod=10)
    indicators['rocp']=ta.ROCP(data['close'],timeperiod=10)
    indicators['rocr']=ta.ROCR(data['close'],timeperiod=10)
    indicators['rocr100']=ta.ROCR100(data['close'],timeperiod=10)
    indicators['rsi']=ta.RSI(data['close'],timeperiod=14)
    indicators['stoch_slowk'],indicators['stoch_slowd']=ta.STOCH(data['high'],data['low'],data['close'],fastk_period=5,slowk_period=3,slowk_matype=0,slowd_period=3,slowd_matype=0)
    indicators['stochf_fastk'],indicators['stochf_fastd']=ta.STOCHF(data['high'],data['low'],data['close'],fastk_period=5,fastd_period=3,fastd_matype=0)
    indicators['stochrsi_fastk'],indicators['stochrsi_fastd']=ta.STOCHRSI(data['close'],timeperiod=14,fastk_period=5,fastd_period=3,fastd_matype=0)
    indicators['trix']=ta.TRIX(data['close'],timeperiod=30)
    indicators['ultosc']=ta.ULTOSC(data['high'],data['low'],data['close'],timeperiod1=7,timeperiod2=14,timeperiod3=28)
    indicators['willr']=ta.WILLR(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['ad']=ta.AD(data['high'],data['low'],data['close'],data['volume'])
    indicators['adosc']=ta.ADOSC(data['high'],data['low'],data['close'],data['volume'],fastperiod=3,slowperiod=10)
    indicators['obv']=ta.OBV(data['close'],data['volume'])
    indicators['volume_osc']=(data['volume']-data['volume'].rolling(window=20).mean())/data['volume'].rolling(window=20).mean()
    indicators['vwap']=(data['close']*data['volume']).cumsum()/data['volume'].cumsum()
    indicators['pvi']=data['volume'].diff().apply(lambda x:1 if x>0 else 0).cumsum()*data['close']
    indicators['nvi']=data['volume'].diff().apply(lambda x:1 if x<0 else 0).cumsum()*data['close']
    indicators['atr']=ta.ATR(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['natr']=ta.NATR(data['high'],data['low'],data['close'],timeperiod=14)
    indicators['trange']=ta.TRANGE(data['high'],data['low'],data['close'])
    indicators['ht_dcperiod']=ta.HT_DCPERIOD(data['close'])
    indicators['ht_dcpphase']=ta.HT_DCPHASE(data['close'])
    indicators['ht_phasor_inphase'],indicators['ht_phasor_quadrature']=ta.HT_PHASOR(data['close'])
    indicators['ht_sine_sine'],indicators['ht_sine_leadsine']=ta.HT_SINE(data['close'])
    indicators['ht_trendmode']=ta.HT_TRENDMODE(data['close'])
    indicators['beta']=ta.BETA(data['high'],data['low'],timeperiod=5)
    indicators['correl']=ta.CORREL(data['high'],data['low'],timeperiod=30)
    indicators['linearreg']=ta.LINEARREG(data['close'],timeperiod=14)
    indicators['linearreg_angle']=ta.LINEARREG_ANGLE(data['close'],timeperiod=14)
    indicators['linearreg_intercept']=ta.LINEARREG_INTERCEPT(data['close'],timeperiod=14)
    indicators['linearreg_slope']=ta.LINEARREG_SLOPE(data['close'],timeperiod=14)
    indicators['stddev']=ta.STDDEV(data['close'],timeperiod=5,nbdev=1)
    indicators['tsf']=ta.TSF(data['close'],timeperiod=14)
    indicators['var']=ta.VAR(data['close'],timeperiod=5,nbdev=1)
    try:
        ao=pta.ao(data['high'],data['low'])
        indicators['ao']=ao
    except AttributeError:
        pass
    try:
        fi=pta.fi(data['close'],data['volume'])
        indicators['fi']=fi
    except AttributeError:
        data['fi']=(data['close']-data['close'].shift(1))*data['volume']
        indicators['fi']=data['fi']
    try:
        ichimoku=data.ta.ichimoku(append=False)
        expected_columns=['isa_9','isb_26','its_9','iks_26']
        for col in expected_columns:
            if col not in ichimoku.columns:
                raise KeyError(col)
        indicators['ichimoku_conversion']=ichimoku['isa_9']
        indicators['ichimoku_base']=ichimoku['isb_26']
        indicators['ichimoku_span_a']=ichimoku['its_9']
        indicators['ichimoku_span_b']=ichimoku['iks_26']
    except:
        pass
    try:
        kc=data.ta.kc(append=False)
        expected_columns=['kcu_20_2.0','kcm_20_2.0','kcl_20_2.0']
        for col in expected_columns:
            if col not in kc.columns:
                raise KeyError(col)
        indicators['kc_upper']=kc['kcu_20_2.0']
        indicators['kc_middle']=kc['kcm_20_2.0']
        indicators['kc_lower']=kc['kcl_20_2.0']
    except:
        pass
    try:
        mfi=pta.mfi(data['high'],data['low'],data['close'],data['volume'])
        indicators['mfi']=mfi
    except:
        pass
    try:
        rvi=pta.rvi(data['close'])
        indicators['rvi']=rvi
    except:
        pass
    try:
        stochrsi=data.ta.stochrsi(append=False)
        expected_columns=['stochrsi_14_5_3_slowk','stochrsi_14_5_3_slowd']
        for col in expected_columns:
            if col not in stochrsi.columns:
                raise KeyError(col)
        indicators['stochrsi_slowk']=stochrsi['stochrsi_14_5_3_slowk']
        indicators['stochrsi_slowd']=stochrsi['stochrsi_14_5_3_slowd']
    except:
        pass
    try:
        tsi=pta.tsi(data['close'])
        for col in tsi.columns:
            indicators[col]=tsi[col]
    except:
        pass
    try:
        vortex=data.ta.vortex(append=False)
        expected_columns=['vi+_14','vi-_14']
        for col in expected_columns:
            if col not in vortex.columns:
                raise KeyError(col)
        indicators['vi_plus']=vortex['vi+_14']
        indicators['vi_minus']=vortex['vi-_14']
    except:
        pass
    data=compute_obv_price_divergence(data)
    for key,value in indicators.items():
        data[key]=value
    data.dropna(inplace=True)
    return data
====================
indicator_config_parser.py:
# indicator_config_parser.py
import json
from typing import Dict, List, Optional

def parse_indicators_json(indicator_params_path: str = 'indicator_params.json') -> Dict[str, Dict]:
    with open(indicator_params_path, 'r') as f:
        indicator_params = json.load(f)
    return indicator_params

def get_configurable_indicators(indicator_params_path: str = 'indicator_params.json') -> List[str]:
    indicator_params = parse_indicators_json(indicator_params_path)
    return sorted(indicator_params.get("indicators", {}).keys())

def get_indicator_parameters(indicator_name: str, indicator_params_path: str = 'indicator_params.json') -> Optional[Dict]:
    indicator_params = parse_indicators_json(indicator_params_path)
    return indicator_params.get("indicators", {}).get(indicator_name)

====================
indicator_params.json:
{
    "indicators": {
        "macd": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "fastperiod": { "default": 12 },
                "slowperiod": { "default": 26 },
                "signalperiod": { "default": 9 },
                "matype": { "default": 0 }
            },
            "conditions": [
                {
                    "fastperiod": { "gte": 2 }
                },
                {
                    "slowperiod": { "gte": 2 }
                },
                {
                    "fastperiod": { "lt": "slowperiod" }
                }
            ]
        },
        "minus_di": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "minus_dm": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "mom": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 10 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "plus_di": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "plus_dm": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "ppo": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "fastperiod": { "default": 12 },
                "slowperiod": { "default": 26 },
                "matype": { "default": 0 }
            },
            "conditions": [
                {
                    "fastperiod": { "gte": 2 }
                },
                {
                    "slowperiod": { "gte": 2 }
                },
                {
                    "fastperiod": { "lt": "slowperiod" }
                }
            ]
        },
        "roc": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 10 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "rocp": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 10 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "rocr": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 10 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "rocr100": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 10 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "stochf": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "fastk_period": { "default": 5 },
                "slowk_period": { "default": 3 },
                "slowk_matype": { "default": 0 },
                "slowd_period": { "default": 3 },
                "slowd_matype": { "default": 0 }
            },
            "conditions": [
                {
                    "fastk_period": { "gte": 2 }
                },
                {
                    "slowk_period": { "gte": 2 }
                },
                {
                    "slowd_period": { "gte": 2 }
                }
            ]
        },
        "stochrsi": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 14 },
                "fastk_period": { "default": 5 },
                "fastd_period": { "default": 3 },
                "fastd_matype": { "default": 0 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                },
                {
                    "fastk_period": { "gte": 2 }
                },
                {
                    "fastd_period": { "gte": 2 }
                }
            ]
        },
        "trix": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 30 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "ultosc": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "timeperiod1": { "default": 7 },
                "timeperiod2": { "default": 14 },
                "timeperiod3": { "default": 28 }
            },
            "conditions": [
                {
                    "timeperiod1": { "gte": 2 }
                },
                {
                    "timeperiod2": { "gte": 2 }
                },
                {
                    "timeperiod3": { "gte": 2 }
                },
                {
                    "timeperiod1": { "lt": "timeperiod2" }
                },
                {
                    "timeperiod2": { "lt": "timeperiod3" }
                }
            ]
        },
        "willr": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "adosc": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close", "volume"],
            "parameters": {
                "fastperiod": { "default": 3 },
                "slowperiod": { "default": 10 }
            },
            "conditions": [
                {
                    "fastperiod": { "gte": 2 }
                },
                {
                    "slowperiod": { "gte": 2 }
                },
                {
                    "fastperiod": { "lt": "slowperiod" }
                }
            ]
        },
        "atr": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "natr": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "linearreg": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "stddev": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 5 },
                "nbdev": { "default": 1 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "var": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 5 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "mfi": {
            "type": "ta-lib",
            "required_inputs": ["high", "low", "close", "volume"],
            "parameters": {
                "timeperiod": { "default": 14 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "vortex": {
            "type": "pandas-ta",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "length": { "default": 14 }
            },
            "conditions": [
                {
                    "length": { "gte": 2 }
                }
            ]
        },
        "ao": {
            "type": "pandas-ta",
            "required_inputs": ["high", "low"],
            "parameters": {
                "fast": { "default": 5 },
                "slow": { "default": 34 }
            },
            "conditions": [
                {
                    "fast": { "gte": 2 }
                },
                {
                    "slow": { "gte": 2 }
                },
                {
                    "fast": { "lt": "slow" }
                }
            ]
        },
        "kc": {
            "type": "pandas-ta",
            "required_inputs": ["high", "low", "close"],
            "parameters": {
                "length": { "default": 20 },
                "mamode": { "default": "ema" },
                "multup": { "default": 2 },
                "multdn": { "default": 2 }
            },
            "conditions": [
                {
                    "length": { "gte": 2 }
                },
                {
                    "multup": { "gte": 0.1 }
                },
                {
                    "multdn": { "gte": 0.1 }
                }
            ]
        },
        "rvi": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "timeperiod": { "default": 10 }
            },
            "conditions": [
                {
                    "timeperiod": { "gte": 2 }
                }
            ]
        },
        "tsi": {
            "type": "ta-lib",
            "required_inputs": ["close"],
            "parameters": {
                "long": { "default": 25 },
                "short": { "default": 13 }
            },
            "conditions": [
                {
                    "long": { "gte": 2 }
                },
                {
                    "short": { "gte": 2 }
                },
                {
                    "short": { "lt": "long" }
                }
            ]
        },
        "EyeX MFV Volume": {
            "type": "custom",
            "required_inputs": [],
            "parameters": {
                "ranges": {
                    "default": [50, 75, 100, 200]
                }
            },
            "conditions": []
        },
        "EyeX MFV S/R Bull": {
            "type": "custom",
            "required_inputs": [],
            "parameters": {
                "ranges": {
                    "default": [50, 75, 100, 200]
                },
                "pivot_lookback": {
                    "default": 5
                },
                "price_proximity": {
                    "default": 0.00001
                }
            },
            "conditions": [
                {
                    "pivot_lookback": { "gte": 1 }
                },
                {
                    "price_proximity": { "gte": 0.000001 }
                }
            ]
        }
    }
}

====================
launch.py:
# launch.py
import os
import sys
from pathlib import Path
from datetime import datetime
import logging
import runpy

for f in Path.cwd().glob('*.log'):
    f.unlink()

timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
working_dir_name = Path.cwd().name
log_filename = f"{working_dir_name}_{timestamp}.log"

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
for h in logger.handlers[:]:
    logger.removeHandler(h)
file_handler = logging.FileHandler(log_filename, mode='w')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
logger.addHandler(file_handler)

class DoubleWriter:
    def __init__(self, stdout, stderr, logger):
        self.stdout = stdout
        self.stderr = stderr
        self.logger = logger
    def write(self, msg):
        if msg.strip():
            self.logger.info(msg.strip())
        self.stdout.write(msg)
    def flush(self):
        self.stdout.flush()
    def isatty(self):
        return self.stdout.isatty()

sys.stdout = DoubleWriter(sys.__stdout__, sys.__stderr__, logger)
sys.stderr = DoubleWriter(sys.__stderr__, sys.__stderr__, logger)

try:
    runpy.run_path("start.py", run_name="__main__")
except SystemExit as e:
    sys.exit(e.code)
except:
    raise
====================
load_data.py:
# load_data.py

import os
import sys
import sqlite3
import logging
import pandas as pd
from dateutil import parser
import numpy as np

from config import DB_PATH

from sqlite_data_manager import create_connection, create_tables

def load_data(symbol, timeframe):
    """
    Loads and preprocesses data from the SQLite database for a specific symbol and timeframe.

    Args:
        symbol (str): The trading symbol (e.g., 'BTCUSDT').
        timeframe (str): The timeframe (e.g., '1d').

    Returns:
        tuple: A tuple containing the processed DataFrame, a boolean indicating if the data is in reverse chronological order, and the database filename.
    """
    logging.basicConfig(level=logging.INFO)

    db_path = DB_PATH

    if not os.path.exists(db_path):
        logging.warning(f"SQLite database '{db_path}' not found. Creating a new database with all required tables.")
        conn = create_connection(db_path)
        if conn:
            create_tables(conn)
            conn.close()
        else:
            logging.error(f"Failed to create connection to '{db_path}'. Exiting.")
            sys.exit(1)

    try:
        conn = sqlite3.connect(db_path)
        logging.info(f"Connected to SQLite database at: {db_path}")
    except sqlite3.Error as e:
        logging.error(f"Failed to connect to database '{db_path}'. Error: {e}")
        sys.exit(1)

    query = """
    SELECT klines.* FROM klines
    JOIN symbols ON klines.symbol_id = symbols.id
    JOIN timeframes ON klines.timeframe_id = timeframes.id
    WHERE symbols.symbol = ? AND timeframes.timeframe = ?
    ORDER BY open_time ASC
    """

    try:
        data = pd.read_sql_query(query, conn, params=(symbol, timeframe), parse_dates=['open_time', 'close_time'])
        logging.info("Data loaded from SQLite database.")
    except pd.io.sql.DatabaseError as e:
        logging.error(f"Failed to execute query. Error: {e}")
        conn.close()
        sys.exit(1)

    conn.close()

    if data.empty:
        logging.warning("No data available in the database.")
        is_reverse_chronological = False
        return data, is_reverse_chronological, os.path.basename(db_path)
    else:
        time_column = 'open_time'
        is_reverse_chronological = data[time_column].is_monotonic_decreasing

        if is_reverse_chronological:
            data = data.sort_values(time_column)
            data.reset_index(drop=True, inplace=True)
            logging.info("Data sorted in chronological order.")

        data.dropna(inplace=True)

        logging.info("Columns in the data: %s", list(data.columns))
        logging.info("First few rows of the data:\n%s", data.head())

        data['open_time'] = data['open_time'].dt.tz_localize(None)
        data['close_time'] = data['close_time'].dt.tz_localize(None)
        logging.info("Removed timezone information from 'open_time' and 'close_time'.")

        if not pd.api.types.is_datetime64_any_dtype(data['open_time']):
            data['open_time'] = pd.to_datetime(data['open_time'], errors='coerce')

        try:
            data['TimeDiff'] = data['open_time'].diff().dt.total_seconds()
            if data['TimeDiff'].isna().all():
                logging.error("Failed to determine time interval: All computed time differences are NaN.")
                sys.exit(1)
            else:
                logging.info("First few time differences (in seconds):\n%s", data['TimeDiff'].head())
                data['TimeDiff'].fillna(0, inplace=True)
        except Exception as e:
            logging.error(f"Failed to compute time differences: {e}")
            sys.exit(1)

    return data, is_reverse_chronological, os.path.basename(db_path)

if __name__ == "__main__":
    symbol = input("Enter the symbol (e.g., 'BTCUSDT'): ").strip().upper()
    timeframe = input("Enter the timeframe (e.g., '1d'): ").strip()
    df, is_rev, filename = load_data(symbol, timeframe)

====================
logging_setup.py:
# logging_setup.py
import logging
import sys
from pathlib import Path
from datetime import datetime
import inspect

class TaskAwareFormatter(logging.Formatter):
    def format(self, record):
        for frame in inspect.stack():
            module = inspect.getmodule(frame[0])
            if module and not module.__name__.startswith('logging'):
                record.filename = Path(module.__file__).name
                record.funcName = frame.function
                record.lineno = frame.lineno
                break
        return super().format(record)

class StreamToLogger:
    def __init__(self, stream):
        self.stream = stream

    def write(self, msg):
        if msg.strip():
            self.stream.write(msg + "\n")
            self.stream.flush()

    def flush(self):
        self.stream.flush()

class ExcludeFilter(logging.Filter):
    def __init__(self, exclude_substrings):
        super().__init__()
        self.exclude_substrings = exclude_substrings

    def filter(self, record):
        message = record.getMessage()
        module = record.module
        pathname = record.pathname
        return not any(
            substring in (message or '') or substring in (module or '') or substring in (pathname or '')
            for substring in self.exclude_substrings
        )

_configured = False

def configure_logging(log_file_prefix='predictions'):
    global _configured
    if _configured:
        return

    logger = logging.getLogger()
    if logger.hasHandlers():
        logger.handlers.clear()

    logger.setLevel(logging.DEBUG)
    log_path = Path.cwd() / f"{log_file_prefix}_{datetime.now().strftime('%Y%m%d-%H%M%S')}.log"

    try:
        exclude_filter = ExcludeFilter([
            'copyscripts', 'COPYSCRIPTS_SELECTIVE',
            'eucjpprober', 'mbcharsetprober', 'charsetgroupprober'
        ])

        file_handler = logging.FileHandler(log_path, 'w')
        file_handler.setLevel(logging.DEBUG)
        file_handler.setFormatter(
            TaskAwareFormatter('%(asctime)s - %(levelname)s - [%(filename)s:%(lineno)d(%(funcName)s)]: %(message)s')
        )
        file_handler.addFilter(exclude_filter)

        stream_handler = logging.StreamHandler()
        stream_handler.setLevel(logging.INFO)
        stream_handler.setFormatter(logging.Formatter('%(message)s'))
        stream_handler.addFilter(exclude_filter)

        logger.addHandler(file_handler)
        logger.addHandler(stream_handler)

        sys.stdout = StreamToLogger(sys.stdout)
        sys.stderr = StreamToLogger(sys.stderr)

        def exception_handler(exc_type, exc_value, exc_traceback):
            if not issubclass(exc_type, KeyboardInterrupt):
                logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))

        sys.excepthook = exception_handler
        logger.info(f"Logging configured. Log file at: {log_path.resolve()}")
        _configured = True
        return log_path

    except Exception as e:
        print(f"Logging setup failed: {e}")
        sys.exit(1)

====================
requirements.txt:
certifi==2024.8.30
chardet==5.2.0
charset-normalizer==3.4.0
colorama==0.4.6
contourpy==1.3.1
cycler==0.12.1
fonttools==4.55.3
idna==3.10
joblib==1.4.2
kiwisolver==1.4.7
matplotlib==3.9.3
numpy==1.24.4
packaging==24.2
pandas==2.2.3
pandas_ta==0.3.14b0
pillow==11.0.0
pyparsing==3.2.0
python-dateutil==2.9.0.post0
pytz==2024.2
requests==2.32.3
scikit-learn==1.6.0
scipy==1.14.1
seaborn==0.13.2
six==1.17.0
ta-lib==0.5.1
threadpoolctl==3.5.0
tqdm==4.67.1
tzdata==2024.2
urllib3==2.2.3
xgboost==2.1.3

====================
revert-to-gpt-scripts.py:
# REVERT-to-GPT-scripts.py
import os
import sys
import re
import logging
from pathlib import Path

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

EXCLUDED_FILES = {
    '.gitignore',
    'copyscripts.py',
    'repair-remarks.py',
    'REVERT-to-GPT-scripts.py'
}

MANAGED_DIRECTORIES = [
    Path.cwd(),
    Path.cwd() / 'scripts'
]

def list_gptbak_files(directory: Path) -> list:
    """
    Lists all .GPTBAK files in the specified directory.

    Parameters:
    - directory: Path object of the directory to search.

    Returns:
    - List of Path objects for each .GPTBAK file found.
    """
    gptbak_files = list(directory.glob('*.GPTBAK'))
    return gptbak_files

def prompt_user_to_select_gptbak(gptbak_files: list) -> tuple:
    """
    Prompts the user to select a GPTBAK file from the list.

    Parameters:
    - gptbak_files: List of Path objects for GPTBAK files.

    Returns:
    - Tuple containing the Path object of the selected GPTBAK file and its selection number.
    """
    if not gptbak_files:
        logging.error("No .GPTBAK files found in the working directory.")
        sys.exit(1)
    
    if len(gptbak_files) == 1:
        selected = gptbak_files[0]
        logging.info(f"Only one GPTBAK file found: '{selected.name}'. Selecting it by default.")
        return (selected, 1)
    
    logging.info("Multiple GPTBAK files found:")
    for idx, gptbak in enumerate(gptbak_files, start=1):
        logging.info(f"{idx}) {gptbak.name}")
    
    while True:
        try:
            choice = input(f"Enter the number of the GPTBAK file you wish to restore (1-{len(gptbak_files)}): ").strip()
            if not choice.isdigit():
                logging.warning("Invalid input. Please enter a number.")
                continue
            choice = int(choice)
            if 1 <= choice <= len(gptbak_files):
                selected = gptbak_files[choice - 1]
                logging.info(f"Selected GPTBAK file: '{selected.name}'.")
                return (selected, choice)
            else:
                logging.warning(f"Please enter a number between 1 and {len(gptbak_files)}.")
        except Exception as e:
            logging.error(f"Error during selection: {e}")

def confirm_overwrite() -> None:
    """
    Prompts the user to confirm overwriting scripts by typing "I am sure".
    Repeats until the user inputs the exact phrase.
    """
    prompt = 'Are you sure you want to overwrite the existing scripts with the ones in the selected GPTBAK file? Type "I am sure" to proceed: '
    while True:
        confirmation = input(prompt).strip()
        if confirmation == "I am sure":
            logging.info("Confirmation received. Proceeding with script replacement.")
            break
        else:
            logging.warning('Confirmation failed. Please type "I am sure" to proceed.')

def parse_gptbak_file(gptbak_filepath: Path):
    """
    Parses the GPTBAK file and extracts scripts with their filenames and target directories.

    Parameters:
    - gptbak_filepath: Path object pointing to the GPTBAK file.

    Returns:
    - List of dictionaries with 'filename', 'directory', and 'content' keys.
    """
    if not gptbak_filepath.exists():
        logging.error(f"GPTBAK file '{gptbak_filepath}' does not exist.")
        sys.exit(1)
    
    with gptbak_filepath.open('r', encoding='utf-8') as file:
        content = file.read()
    
    sections = re.split(r'={5,}', content)
    
    scripts = []
    
    script_header_pattern = re.compile(
        r'^\s*\d+\)\s+([^\s]+)\s+\(located in the (working directory|\'([^\']+)\' subdirectory)\):\s*\n',
        re.MULTILINE
    )
    
    for section in sections:
        match = script_header_pattern.search(section)
        if match:
            filename = match.group(1).strip()
            location = match.group(2).strip()
            subdirectory = match.group(3)
            
            if location == 'working directory':
                target_dir = Path.cwd()
            elif subdirectory and subdirectory == 'scripts':
                target_dir = Path.cwd() / 'scripts'
            else:
                logging.warning(f"Skipping script '{filename}' with unsupported location: '{location}'.")
                continue
            
            if filename in EXCLUDED_FILES:
                logging.info(f"Skipping excluded file: '{filename}'.")
                continue
            
            start_pos = match.end()
            script_content = section[start_pos:].strip()
            
            script_content = script_content.strip('`').strip()
            
            scripts.append({
                'filename': filename,
                'directory': target_dir,
                'content': script_content
            })
        else:
            continue
    
    return scripts

def sanitize_filename_comment(comment: str) -> str:
    """
    Sanitizes the user's comment to ensure it's safe for filenames.

    Parameters:
    - comment: The comment string entered by the user.

    Returns:
    - A sanitized version of the comment string.
    """
    invalid_chars = r'<>:"/\|?*'
    sanitized = ''.join(['_' if c in invalid_chars else c for c in comment])
    return sanitized

def rename_gptbak_file(gptbak_file: Path, comment: str):
    """
    Renames the GPTBAK file by appending the user's comment.

    Parameters:
    - gptbak_file: Path object of the original GPTBAK file.
    - comment: The comment string to append.
    """
    sanitized_comment = sanitize_filename_comment(comment)
    new_filename = f"{gptbak_file.stem} - {sanitized_comment}{gptbak_file.suffix}"
    new_filepath = gptbak_file.parent / new_filename

    if new_filepath.exists():
        logging.warning(f"Cannot rename '{gptbak_file.name}' to '{new_filename}' because the destination file already exists.")
        return

    try:
        gptbak_file.rename(new_filepath)
        logging.info(f"Renamed GPTBAK file to: '{new_filename}'.")
    except Exception as e:
        logging.error(f"Failed to rename '{gptbak_file}' to '{new_filename}': {e}")

def find_existing_scripts(managed_dirs: list, filename: str) -> list:
    """
    Searches only the managed directories for existing scripts with the given filename.

    Parameters:
    - managed_dirs: List of Path objects representing directories to search.
    - filename: Name of the script file to search for.

    Returns:
    - List of Path objects where the script is found.
    """
    existing_scripts = []
    for directory in managed_dirs:
        for file_path in directory.glob(filename):
            if file_path.name in EXCLUDED_FILES:
                continue
            existing_scripts.append(file_path)
    return existing_scripts

def replace_scripts(scripts: list, managed_dirs: list):
    """
    Replaces existing scripts with the provided scripts.

    Parameters:
    - scripts: List of dictionaries with 'filename', 'directory', and 'content' keys.
    - managed_dirs: List of Path objects representing directories to manage.
    """
    for script in scripts:
        filename = script['filename']
        target_dir = script['directory']
        content = script['content']
        
        if not target_dir.is_absolute():
            target_dir = Path.cwd() / target_dir
        
        try:
            target_dir.mkdir(parents=True, exist_ok=True)
            logging.info(f"Ensured directory exists: {target_dir}")
        except Exception as e:
            logging.error(f"Failed to create directory '{target_dir}': {e}")
            continue
        
        existing_scripts = find_existing_scripts(managed_dirs, filename)
        for existing_script in existing_scripts:
            try:
                existing_script.unlink()
                logging.info(f"Deleted existing script: {existing_script}")
            except Exception as e:
                logging.error(f"Failed to delete '{existing_script}': {e}")
        
        new_script_path = target_dir / filename
        
        try:
            with new_script_path.open('w', encoding='utf-8') as f:
                f.write(content)
            logging.info(f"Created/Updated script: {new_script_path}")
        except Exception as e:
            logging.error(f"Failed to write to '{new_script_path}': {e}")

def prompt_amend_filename(gptbak_file: Path, selection_number: int):
    """
    Prompts the user to amend the GPTBAK file's filename with comments.

    Parameters:
    - gptbak_file: Path object of the GPTBAK file.
    - selection_number: The number corresponding to the GPTBAK file selection.
    """
    print(f"You just used the following GPTBAK file: {gptbak_file.name}.")
    print(f"You selected option number {selection_number}.")
    comment = input('Would you like to amend the GPTBAK\'s filename with any comments? Type your comments, otherwise just hit enter to return to the file page: ').strip()
    
    if comment:
        rename_gptbak_file(gptbak_file, comment)
    else:
        logging.info("No comments entered. Returning to GPTBAK file selection.")

def main():
    """
    Main function to execute the script restoration.
    """
    working_dir = Path.cwd()
    scripts_dir = working_dir / 'scripts'
    managed_dirs = [working_dir, scripts_dir]
    
    while True:
        gptbak_files = list_gptbak_files(working_dir)
        
        selected_gptbak, selection_number = prompt_user_to_select_gptbak(gptbak_files)
        
        confirm_overwrite()
        
        logging.info(f"Parsing GPTBAK file: {selected_gptbak}")
        
        scripts = parse_gptbak_file(selected_gptbak)
        
        if not scripts:
            logging.warning("No valid scripts found in the GPTBAK file.")
            prompt_amend_filename(selected_gptbak, selection_number)
            continue
        
        logging.info(f"Found {len(scripts)} scripts to replace.")
        
        replace_scripts(scripts, managed_dirs)
        
        prompt_amend_filename(selected_gptbak, selection_number)
        
        remaining_gptbak_files = list_gptbak_files(working_dir)
        if not remaining_gptbak_files:
            logging.info("No more GPTBAK files to process. Exiting.")
            break
        else:
            proceed = input("Would you like to restore another GPTBAK file? (y/n): ").strip().lower()
            if proceed not in ['y', 'yes']:
                logging.info("Exiting script restoration process.")
                break

if __name__ == "__main__":
    main()

====================
sqlite_data_manager.py:
# sqlite_data_manager.py
import os
import sys
import sqlite3
import pandas as pd
from pathlib import Path
from config import DB_PATH

def create_connection(db_file):
    conn=None
    try:
        conn=sqlite3.connect(db_file)
    except:
        pass
    return conn

def create_tables(conn):
    try:
        cursor=conn.cursor()
        create_symbols_table="""
        CREATE TABLE IF NOT EXISTS symbols (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            symbol TEXT UNIQUE
        );
        """
        cursor.execute(create_symbols_table)
        create_timeframes_table="""
        CREATE TABLE IF NOT EXISTS timeframes (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            timeframe TEXT UNIQUE
        );
        """
        cursor.execute(create_timeframes_table)
        create_indicators_table="""
        CREATE TABLE IF NOT EXISTS indicators (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            name TEXT UNIQUE
        );
        """
        cursor.execute(create_indicators_table)
        create_klines_table="""
        CREATE TABLE IF NOT EXISTS klines (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            symbol_id INTEGER,
            timeframe_id INTEGER,
            open_time TEXT,
            open REAL,
            high REAL,
            low REAL,
            close REAL,
            volume REAL,
            close_time TEXT,
            quote_asset_volume REAL,
            number_of_trades INTEGER,
            taker_buy_base_asset_volume REAL,
            taker_buy_quote_asset_volume REAL,
            correlation_computed BOOLEAN DEFAULT FALSE,
            FOREIGN KEY (symbol_id) REFERENCES symbols(id),
            FOREIGN KEY (timeframe_id) REFERENCES timeframes(id)
        );
        """
        cursor.execute(create_klines_table)
        create_correlations_table="""
        CREATE TABLE IF NOT EXISTS correlations (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            symbol_id INTEGER,
            timeframe_id INTEGER,
            indicator_id INTEGER,
            lag INTEGER,
            correlation_value REAL,
            FOREIGN KEY (symbol_id) REFERENCES symbols(id),
            FOREIGN KEY (timeframe_id) REFERENCES timeframes(id),
            FOREIGN KEY (indicator_id) REFERENCES indicators(id)
        );
        """
        cursor.execute(create_correlations_table)
        indexes=[
            "CREATE INDEX IF NOT EXISTS idx_open_time ON klines (open_time);",
            "CREATE INDEX IF NOT EXISTS idx_close_time ON klines (close_time);",
            "CREATE INDEX IF NOT EXISTS idx_correlation_computed ON klines (correlation_computed);",
            "CREATE INDEX IF NOT EXISTS idx_correlations_symbol_timeframe_indicator_lag ON correlations (symbol_id, timeframe_id, indicator_id, lag);"
        ]
        for index_query in indexes:
            cursor.execute(index_query)
        conn.commit()
    except:
        pass

def save_to_sqlite(df,db_path,symbol,timeframe):
    conn=None
    try:
        os.makedirs(os.path.dirname(db_path),exist_ok=True)
        conn=sqlite3.connect(db_path)
        cursor=conn.cursor()
        cursor.execute("INSERT OR IGNORE INTO symbols (symbol) VALUES (?)",(symbol,))
        cursor.execute("SELECT id FROM symbols WHERE symbol = ?",(symbol,))
        symbol_id=cursor.fetchone()[0]
        cursor.execute("INSERT OR IGNORE INTO timeframes (timeframe) VALUES (?)",(timeframe,))
        cursor.execute("SELECT id FROM timeframes WHERE timeframe = ?",(timeframe,))
        timeframe_id=cursor.fetchone()[0]
        df['open_time']=pd.to_datetime(df['open_time']).dt.strftime('%Y-%m-%d %H:%M:%S')
        df['close_time']=pd.to_datetime(df['close_time']).dt.strftime('%Y-%m-%d %H:%M:%S')
        insert_query="""
            INSERT INTO klines (
                symbol_id, timeframe_id, open_time, open, high, low, close, volume, close_time,
                quote_asset_volume, number_of_trades, taker_buy_base_asset_volume, taker_buy_quote_asset_volume
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        """
        for index,row in df.iterrows():
            cursor.execute(insert_query,(symbol_id,timeframe_id,row['open_time'],row['open'],row['high'],row['low'],row['close'],row['volume'],row['close_time'],row['quote_asset_volume'],row['number_of_trades'],row['taker_buy_base_asset_volume'],row['taker_buy_quote_asset_volume']))
        conn.commit()
    except:
        pass
    finally:
        if conn:
            conn.close()
====================
start.py:
# start.py
import os
import sys
import shutil
import re
from pathlib import Path
from datetime import datetime, timedelta
from typing import List
import numpy as np
import pandas as pd
import time
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import seaborn as sns
import dateutil.parser
from joblib import Parallel, delayed
from scipy.stats import t
from sklearn.preprocessing import StandardScaler

from linear_regression import perform_linear_regression
from advanced_analysis import advanced_price_prediction
from load_data import load_data
from indicators import compute_all_indicators
from data_utils import clear_screen, prepare_data, determine_time_interval, get_original_indicators, handle_missing_indicators
from correlation_utils import load_or_calculate_correlations, calculate_correlation
from visualization_utils import generate_combined_correlation_chart, visualize_data
from generate_heatmaps import generate_heatmaps
from backup_utils import run_backup_cleanup
from table_generation import generate_best_indicator_table, generate_statistical_summary, generate_correlation_csv
from binance_historical_data_downloader import download_binance_data
from correlation_database import CorrelationDatabase
from config import DB_PATH
import sqlite3

def parse_date_time_input(user_input: str, reference_datetime: datetime) -> datetime:
    print("[DEBUG] parse_date_time_input: start")
    user_input = user_input.strip()
    if not user_input:
        print("[DEBUG] parse_date_time_input: user_input blank, returning reference_datetime")
        return reference_datetime
    relative_time_pattern = r'^([+-]\d+)([smhdw])$'
    match = re.match(relative_time_pattern, user_input)
    if match:
        print("[DEBUG] parse_date_time_input: relative time pattern matched")
        amount = int(match.group(1))
        unit = match.group(2)
        delta_kwargs = {'s': 'seconds', 'm': 'minutes', 'h': 'hours', 'd': 'days', 'w': 'weeks'}
        result = reference_datetime + timedelta(**{delta_kwargs[unit]: amount})
        print(f"[DEBUG] parse_date_time_input: returning {result}")
        return result
    for fmt in ['%Y%m%d-%H%M', '%Y%m%d']:
        try:
            result = datetime.strptime(user_input, fmt)
            print(f"[DEBUG] parse_date_time_input: parsed with {fmt}, returning {result}")
            return result
        except ValueError:
            print(f"[DEBUG] parse_date_time_input: failed {fmt}")
            continue
    print("[DEBUG] parse_date_time_input: using dateutil.parser")
    result = dateutil.parser.parse(user_input, fuzzy=True)
    print(f"[DEBUG] parse_date_time_input: returning {result}")
    return result

def input_with_default(prompt: str, default: str) -> str:
    print(f"[DEBUG] input_with_default: prompt='{prompt}', default='{default}'")
    val = input(prompt).strip()
    if val == "":
        print(f"No input given. Defaulting to '{default}'")
        return default
    print(f"User input: '{val}'")
    return val

def input_yes_no(prompt: str, default: str = 'y') -> str:
    print(f"[DEBUG] input_yes_no: prompt='{prompt}', default='{default}'")
    val = input(prompt).strip().lower()
    if val not in ['y','n','yes','no','']:
        print(f"Invalid input '{val}', defaulting to '{default}'")
        return default
    if val == '':
        print(f"No input given, defaulting to '{default}'")
        return default
    choice = 'y' if val.startswith('y') else 'n'
    print(f"User input: '{choice}'")
    return choice

def input_yes_no_no_default(prompt: str) -> str:
    """
    Prompt the user with a yes/no question.
    There is NO default response.
    If blank or invalid, keep asking until 'y' or 'n' is provided.
    """
    while True:
        val = input(prompt).strip().lower()
        if val in ['y','yes']:
            print("User input: 'y'")
            return 'y'
        elif val in ['n','no']:
            print("User input: 'n'")
            return 'n'
        else:
            print("Invalid input, please enter 'y' or 'n'.")

def preview_database(db_path: str):
    print("[DEBUG] preview_database: start")
    try:
        conn = sqlite3.connect(db_path)
        cursor = conn.cursor()

        print("[DEBUG] preview_database: previewing symbols table")
        symbols_df = pd.read_sql_query("SELECT * FROM symbols", conn)
        print("Symbols table preview:")
        print(symbols_df.head())

        print("[DEBUG] preview_database: previewing timeframes table")
        timeframes_df = pd.read_sql_query("SELECT * FROM timeframes", conn)
        print("Timeframes table preview:")
        print(timeframes_df.head())

        print("[DEBUG] preview_database: previewing klines table")
        klines_df = pd.read_sql_query("SELECT * FROM klines LIMIT 5", conn)
        print("Klines table sample:")
        print(klines_df.head())

        conn.close()

        if symbols_df.empty or timeframes_df.empty or klines_df.empty:
            print("[DEBUG] preview_database: database seems empty or incomplete")
            return False
        print("[DEBUG] preview_database: database content looks valid")
        return True
    except Exception as e:
        print(f"Error previewing database: {e}")
        return False

def recreate_database(db_path: str):
    print("[DEBUG] recreate_database: deleting old DB and creating new empty DB")
    if os.path.exists(db_path):
        os.remove(db_path)
        print(f"[DEBUG] recreate_database: deleted existing database at {db_path}")
    from sqlite_data_manager import create_connection, create_tables
    conn = create_connection(db_path)
    if conn:
        create_tables(conn)
        conn.close()
        print("[DEBUG] recreate_database: new DB created with tables")
    else:
        print("[DEBUG] recreate_database: failed to create new database")
        sys.exit(1)

def main() -> None:
    print("Script start: Clearing screen...")
    clear_screen()

    print("Running backup cleanup...")
    run_backup_cleanup()
    print("Backup cleanup done.")

    new_timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
    reports_dir = 'reports'
    csv_dir = 'csv'
    print(f"Ensuring output directories exist: {reports_dir}, {csv_dir}...")
    for directory in [reports_dir, csv_dir]:
        try:
            os.makedirs(directory, exist_ok=True)
            print(f"Directory ensured: {directory}")
        except Exception as e:
            print(f"Error creating directory {directory}: {e}")
            sys.exit(1)
    print("Output directories are ready.")

    print("Do you want to delete all previously generated output? (y/n): ")
    delete_output = input_yes_no("Do you want to delete all previously generated output? (y/n): ", default='y')
    if delete_output == 'y':
        print("User chose to delete previously generated output. Cleaning...")
        for folder in ['indicator_charts', 'heatmaps', 'combined_charts', reports_dir]:
            folder_path = Path(folder).resolve()
            if folder_path.exists():
                print(f"Clearing folder: {folder_path}")
                for filename in os.listdir(folder_path):
                    file_path = folder_path / filename
                    try:
                        if file_path.is_file() or file_path.is_symlink():
                            file_path.unlink()
                            print(f"Deleted file: {file_path}")
                        elif file_path.is_dir():
                            shutil.rmtree(file_path)
                            print(f"Deleted directory: {file_path}")
                    except Exception as e:
                        print(f"Error deleting {file_path}: {e}")
        print("Output folders cleaned.")
    else:
        print("User chose not to delete previously generated output.")

    generate_charts = (input_yes_no("Do you want to generate individual indicator charts? (y/n): ", default='y') == 'y')
    print(f"Generate individual indicator charts: {'Yes' if generate_charts else 'No'}")

    generate_heatmaps_flag = (input_yes_no("Do you want to generate heatmaps? (y/n): ", default='y') == 'y')
    print(f"Generate heatmaps: {'Yes' if generate_heatmaps_flag else 'No'}")

    save_correlation_csv = (input_yes_no("Do you want to save a single CSV containing each indicator's correlation values for each lag point? (y/n): ", default='y') == 'y')
    print(f"Save correlation CSV: {'Yes' if save_correlation_csv else 'No'}")

    symbol = input_with_default("Enter the trading symbol (e.g., 'BTCUSDT'): ", "BTCUSDT").upper()
    timeframe = input_with_default("Enter the timeframe (e.g., '1d'): ", "1d")
    print(f"Using symbol={symbol}, timeframe={timeframe}")

    print("Loading data...")
    start_time = time.time()
    data, is_reverse_chronological, db_filename = load_data(symbol, timeframe)
    load_duration = time.time() - start_time
    print(f"Data load completed in {load_duration:.2f} seconds. Data empty: {data.empty}")

    print("Previewing database contents...")
    db_ok = preview_database(DB_PATH)
    if not db_ok:
        print("Database content seems invalid or incomplete.")
        while True:
            choice = input_yes_no_no_default("Database content seems invalid. Erase and create a new one? (y/n): ")
            if choice == 'y':
                recreate_database(DB_PATH)
                print("New database created. Attempting to load data again...")
                data, is_reverse_chronological, db_filename = load_data(symbol, timeframe)
                if data.empty:
                    print("Data still not found after recreating the database. Attempting to download data...")
                    download_choice = input_yes_no("Do you want to download Binance historical data now? (y/n): ", default='y')
                    if download_choice == 'y':
                        try:
                            download_binance_data(symbol, timeframe)
                            data, is_reverse_chronological, db_filename = load_data(symbol, timeframe)
                            if data.empty:
                                print("Data still not found after download. Exiting.")
                                sys.exit(1)
                            else:
                                print("Data successfully loaded from database after download.")
                                break
                        except Exception as e:
                            print(f"Error occurred during data download: {e}")
                            sys.exit(1)
                    else:
                        print("Exiting since no data and no download requested.")
                        sys.exit(1)
                else:
                    print("Data successfully loaded from the new database.")
                    break
            elif choice == 'n':
                print("User chose not to recreate the database. Exiting.")
                sys.exit(1)

    if data.empty:
        print("No data found in the database.")
        download_choice = input_yes_no("Do you want to download Binance historical data now? (y/n): ", default='y')
        if download_choice == 'y':
            print("Downloading data from Binance...")
            try:
                start_time = time.time()
                download_binance_data(symbol, timeframe)
                download_duration = time.time() - start_time
                print(f"Data download completed in {download_duration:.2f} seconds. Reloading data...")
                data, is_reverse_chronological, db_filename = load_data(symbol, timeframe)
                if data.empty:
                    print("Data still not found after download. Exiting.")
                    sys.exit(1)
                else:
                    print("Data successfully loaded from database after download.")
            except Exception as e:
                print(f"Error occurred during data download: {e}")
                sys.exit(1)
        else:
            print("Exiting since no data and no download requested.")
            sys.exit(0)
    else:
        print("Data found in database. Proceeding with analysis...")

    print("Computing indicators...")
    start_time = time.time()
    data = compute_all_indicators(data)
    indicator_duration = time.time() - start_time
    print(f"Indicators computed in {indicator_duration:.2f} seconds.")

    if 'Date' not in data.columns:
        if 'open_time' in data.columns:
            data['Date'] = pd.to_datetime(data['open_time'], errors='coerce').dt.tz_localize(None)
            print("Created 'Date' column from 'open_time'.")
        else:
            print("No 'Date' or 'open_time' column found. Exiting.")
            sys.exit(1)

    if 'Close' not in data.columns:
        if 'close' in data.columns:
            data.rename(columns={'close': 'Close'}, inplace=True)
            print("Renamed 'close' to 'Close'.")
        else:
            print("No 'Close' or 'close' column found. Exiting.")
            sys.exit(1)

    print("Determining time interval between data points...")
    try:
        time_interval = determine_time_interval(data)
        print(f"Time interval determined: {time_interval}")
    except Exception as e:
        print(f"Failed to determine time interval: {e}")
        sys.exit(1)

    print("Preparing data (scaling features, etc.)...")
    start_time = time.time()
    try:
        X_scaled, feature_names = prepare_data(data)
        prep_duration = time.time() - start_time
        print(f"Data preparation done in {prep_duration:.2f} seconds. Features count: {len(feature_names)}")
    except Exception as e:
        print(f"Data preparation failed: {e}")
        sys.exit(1)

    print("Identifying original indicators...")
    original_indicators = get_original_indicators(feature_names, data)
    expected_indicators = ['FI', 'ichimoku', 'KCU_20_2.0', 'STOCHRSI_14_5_3_slowk', 'VI+_14']
    original_indicators = handle_missing_indicators(original_indicators, data, expected_indicators)
    print(f"Original indicators count: {len(original_indicators)}")

    if not original_indicators:
        print("No valid indicators found. Exiting.")
        sys.exit(1)

    max_lag = len(data) - 51
    print(f"Max lag calculated as {max_lag}")
    if max_lag < 1:
        print("Not enough data to proceed (max_lag < 1). Exiting.")
        sys.exit(1)

    print("Loading or calculating correlations. This may take a while...")
    start_time = time.time()
    load_or_calculate_correlations(
        data=data,
        original_indicators=original_indicators,
        max_lag=max_lag,
        is_reverse_chronological=is_reverse_chronological,
        symbol=symbol,
        timeframe=timeframe
    )
    corr_duration = time.time() - start_time
    print(f"Correlations loaded/calculated in {corr_duration:.2f} seconds.")

    print("Retrieving correlations from database...")
    correlation_db = CorrelationDatabase(DB_PATH)
    correlations = {}
    for i, indicator in enumerate(original_indicators, start=1):
        print(f"Processing indicator {i}/{len(original_indicators)}: {indicator}")
        correlations[indicator] = []
        for lag in range(1, max_lag + 1):
            val = correlation_db.get_correlation(symbol, timeframe, indicator, lag)
            correlations[indicator].append(val)
        print(f"Completed retrieving correlations for indicator: {indicator}")
    correlation_db.close()
    print("Finished retrieving all correlations.")

    print("Generating statistical summary of correlations...")
    try:
        summary_df = generate_statistical_summary(correlations, max_lag)
        summary_csv = os.path.join(csv_dir, f"{new_timestamp}_{symbol}_{timeframe}_statistical_summary.csv")
        summary_df.to_csv(summary_csv, index=True)
        print(f"Statistical summary saved to: {summary_csv}")
    except Exception as e:
        print(f"Error generating statistical summary: {e}")

    print("Generating combined correlation chart...")
    try:
        generate_combined_correlation_chart(correlations, max_lag, time_interval, new_timestamp, f"{symbol}_{timeframe}")
        print("Combined correlation chart generated.")
    except Exception as e:
        print(f"Error generating combined correlation chart: {e}")

    if generate_charts:
        print("Generating individual indicator charts...")
        try:
            visualize_data(
                data=data,
                features=X_scaled,
                feature_columns=feature_names,
                timestamp=new_timestamp,
                is_reverse_chronological=is_reverse_chronological,
                time_interval=time_interval,
                generate_charts=generate_charts,
                cache=correlations,
                calculate_correlation_func=calculate_correlation,
                base_csv_filename=f"{symbol}_{timeframe}"
            )
            print("Individual indicator charts generated.")
        except Exception as e:
            print(f"Error generating individual charts: {e}")
    else:
        print("Skipping individual indicator charts as per user choice.")

    if generate_heatmaps_flag:
        print("Generating heatmaps...")
        try:
            generate_heatmaps(
                data=data,
                timestamp=new_timestamp,
                time_interval=time_interval,
                generate_heatmaps_flag=generate_heatmaps_flag,
                cache=correlations,
                calculate_correlation=calculate_correlation,
                base_csv_filename=f"{symbol}_{timeframe}"
            )
            print("Heatmaps generated.")
        except Exception as e:
            print(f"Error generating heatmaps: {e}")
    else:
        print("Skipping heatmaps as per user choice.")

    print("Generating best indicators table...")
    try:
        best_indicators_df = generate_best_indicator_table(correlations, max_lag)
        best_indicators_csv = os.path.join(csv_dir, f"{new_timestamp}_{symbol}_{timeframe}_best_indicators.csv")
        best_indicators_df.to_csv(best_indicators_csv, index=False)
        print(f"Best indicators table saved to: {best_indicators_csv}")
    except Exception as e:
        print(f"Error generating best indicators table: {e}")

    if save_correlation_csv:
        print("Generating correlation CSV for all indicators and lags...")
        try:
            generate_correlation_csv(correlations, max_lag, f"{symbol}_{timeframe}", csv_dir)
            print("Correlation CSV generated.")
        except Exception as e:
            print(f"Error generating correlation CSV: {e}")
    else:
        print("Skipping correlation CSV generation as per user choice.")

    print("Preparing future prediction steps...")
    data['Date'] = pd.to_datetime(data['Date']).dt.tz_localize(None)
    latest_date_in_data = data['Date'].max()
    current_datetime = datetime.now()
    time_interval_seconds_map = {'second': 1, 'minute': 60, 'hour': 3600, 'day': 86400, 'week': 604800}
    if time_interval not in time_interval_seconds_map:
        print("Unsupported time interval. Exiting.")
        sys.exit(1)

    time_diff_seconds = (current_datetime - latest_date_in_data).total_seconds()
    lag_periods_behind_current = int(time_diff_seconds / time_interval_seconds_map[time_interval])
    print(f"Latest data point is {lag_periods_behind_current} {time_interval}(s) behind current time.")

    future_input = input_with_default("Future date/time: ", "")
    print(f"User input for future date/time: '{future_input}'")
    try:
        future_datetime = parse_date_time_input(future_input, current_datetime) if future_input else current_datetime
        print(f"Using future datetime: {future_datetime}")
    except Exception as e:
        print(f"Error parsing future datetime: {e}")
        sys.exit(1)

    lag_seconds = (future_datetime - latest_date_in_data).total_seconds()
    print(f"Lag seconds: {lag_seconds}")
    if lag_seconds <= 0:
        print("Future date/time is not after the latest data point. Exiting.")
        sys.exit(1)
    lag_periods = int(lag_seconds / time_interval_seconds_map[time_interval])
    print(f"Calculated lag_periods for prediction: {lag_periods}")
    if lag_periods < 1:
        print("lag_periods < 1, cannot predict. Exiting.")
        sys.exit(1)
    if lag_periods > max_lag:
        print(f"Requested future lag ({lag_periods}) exceeds max_lag ({max_lag}). Exiting.")
        sys.exit(1)

    print("Performing linear regression prediction...")
    try:
        perform_linear_regression(data, correlations, max_lag, time_interval, new_timestamp, f"{symbol}_{timeframe}", future_datetime, lag_periods)
        print("Linear regression prediction done.")
    except Exception as e:
        print(f"Error performing linear regression prediction: {e}")

    print("Performing advanced analysis prediction...")
    try:
        advanced_price_prediction(data, correlations, max_lag, time_interval, new_timestamp, f"{symbol}_{timeframe}", future_datetime, lag_periods)
        print("Advanced analysis prediction done.")
    except Exception as e:
        print(f"Error performing advanced price prediction: {e}")

    print("Script completed successfully.")

if __name__ == "__main__":
    main()
====================
table_generation.py:
# table_generation.py

import os
import pandas as pd
import numpy as np
import logging

def generate_statistical_summary(correlations: dict, max_lag: int) -> pd.DataFrame:
    """
    Generates a statistical summary (mean, std, min, max) for each indicator's correlations.

    Args:
        correlations (dict): Dictionary with indicator names as keys and list of correlation values as values.
        max_lag (int): Maximum lag period.

    Returns:
        pd.DataFrame: DataFrame containing statistical summaries for each indicator.
    """
    summary = {}
    for indicator, values in correlations.items():
        clean_values = [v for v in values if v is not None and not pd.isna(v)]
        if not clean_values:
            logging.warning(f"No valid correlation values for indicator '{indicator}'. Skipping.")
            continue
        summary[indicator] = {
            'mean': np.mean(clean_values),
            'std': np.std(clean_values),
            'min': np.min(clean_values),
            'max': np.max(clean_values)
        }
    summary_df = pd.DataFrame(summary).T
    return summary_df

def generate_best_indicator_table(correlations: dict, max_lag: int) -> pd.DataFrame:
    """
    Generates a table of best indicators based on the highest mean correlation.

    Args:
        correlations (dict): Dictionary with indicator names as keys and list of correlation values as values.
        max_lag (int): Maximum lag period.

    Returns:
        pd.DataFrame: DataFrame containing the best indicators.
    """
    summary_df = generate_statistical_summary(correlations, max_lag)
    if summary_df.empty:
        logging.warning("No data available to generate best indicators table.")
        return pd.DataFrame()
    
    summary_df_sorted = summary_df.sort_values(by='mean', ascending=False)
    return summary_df_sorted.head(10)

def generate_correlation_csv(correlations: dict, max_lag: int, base_filename: str, csv_dir: str) -> None:
    """
    Generates a CSV file containing all correlation values for each indicator and lag.

    Args:
        correlations (dict): Dictionary with indicator names as keys and list of correlation values as values.
        max_lag (int): Maximum lag period.
        base_filename (str): Base name for the CSV file.
        csv_dir (str): Directory where the CSV should be saved.

    Returns:
        None
    """
    correlation_df = pd.DataFrame(correlations)
    correlation_df.index = range(1, max_lag + 1)
    os.makedirs(csv_dir, exist_ok=True)
    file_path = os.path.join(csv_dir, f"{base_filename}_correlations.csv")
    correlation_df.to_csv(file_path, index_label='Lag')
====================
tweak_indicator.py:
# tweak_indicator.py
import sys
import itertools
import logging
from pathlib import Path
from typing import Dict, List, Optional
import numpy as np
import json
from config import DB_PATH
from indicator_config_parser import get_configurable_indicators, get_indicator_parameters
from sqlite_data_manager import insert_indicator_configs, create_connection, initialize_database
from logging_setup import configure_logging

logger = logging.getLogger(__name__)

def generate_configurations(parameter_keys: List[str], parameter_definitions: Dict) -> List[Dict]:
    param_alternatives = {}
    for param in parameter_keys:
        param_def = parameter_definitions[param]
        default = param_def.get('default')
        if default is None:
            logger.warning(f"Parameter '{param}' does not have a default value. Skipping.")
            continue
        if isinstance(default, int):
            if param in ['fast', 'slow', 'fastperiod', 'slowperiod', 'signalperiod', 'timeperiod', 'timeperiod1', 'timeperiod2', 'timeperiod3', 'pivot_lookback']:
                start = max(2, default - 5)
            else:
                start = max(1, default - 5)
            param_alternatives[param] = [start + i for i in range(11)]
        elif isinstance(default, float):
            if param in ['fast', 'slow', 'fastperiod', 'slowperiod', 'signalperiod', 'timeperiod', 'timeperiod1', 'timeperiod2', 'timeperiod3', 'pivot_lookback']:
                start = max(2.0, default - 5.0)
                param_alternatives[param] = [round(start + i, 1) for i in range(11)]
            elif 0 < default < 1:
                start = max(0.1, default * 0.9)
                param_alternatives[param] = [round(start + 0.02 * i, 4) for i in range(11)]
            else:
                param_alternatives[param] = [default]
        else:
            logger.warning(f"Parameter '{param}' is not a numeric type. Using default only.")
            param_alternatives[param] = [default]

    if not param_alternatives:
        return []

    keys, values = zip(*param_alternatives.items())
    configurations = [dict(zip(keys, v)) for v in itertools.product(*values)]

    valid_combinations = []
    for combo in configurations:
        valid_combinations.append(combo)
    return valid_combinations

def fetch_available_indicators() -> List[str]:
    return get_configurable_indicators('indicator_params.json')

def parse_indicator_parameters(indicator_name: str) -> Optional[Dict]:
    return get_indicator_parameters(indicator_name, 'indicator_params.json')

def insert_tweaked_configs(conn, indicator_name: str, configurations: List[Dict]):
    try:
        insert_indicator_configs(conn, indicator_name, configurations)
    except Exception as e:
        logger.error(f"Error inserting configurations for '{indicator_name}': {e}")
        raise e

def run_tweak_indicator():
    """
    Generates parameter configurations for configurable indicators and inserts them into the database.
    """
    initialize_database(DB_PATH)
    conn = create_connection(DB_PATH)
    if not conn:
        logger.error("Failed to connect to the database.")
        sys.exit(1)
    try:
        available_indicators = fetch_available_indicators()
        if not available_indicators:
            logger.error("No configurable indicators found in indicator_params.json.")
            print("No configurable indicators found. Exiting.")
            sys.exit(1)
    except Exception as e:
        logger.exception(f"Error fetching configurable indicators: {e}")
        print(f"Error fetching configurable indicators: {e}")
        sys.exit(1)

    selected_indicators = []
    print("\nAvailable Indicators for Tweak:")
    for idx, indicator in enumerate(available_indicators, 1):
        print(f"{idx}. {indicator}")
    choice = input(f"Select an indicator by number (1-{len(available_indicators)}): ").strip()
    if not choice.isdigit() or int(choice) < 1 or int(choice) > len(available_indicators):
        logger.error("Invalid choice. Exiting.")
        print("Invalid choice. Exiting.")
        sys.exit(1)

    selected_indicator = available_indicators[int(choice) - 1]
    logger.info(f"Selected indicator: {selected_indicator}")
    print(f"Selected indicator: {selected_indicator}")

    try:
        parameters = parse_indicator_parameters(selected_indicator)
        if not parameters:
            logger.error(f"No parameters found for '{selected_indicator}'. Using base indicator.")
            print(f"No parameters found for '{selected_indicator}'. Using base indicator.")
            return selected_indicator
        else:
            logger.info(f"Parameters for '{selected_indicator}': {parameters}")
            print(f"Parameters for '{selected_indicator}': {parameters}")
    except Exception as e:
        logger.error(f"Error retrieving parameters for '{selected_indicator}': {e}")
        print(f"Error retrieving parameters for '{selected_indicator}'. Exiting.")
        sys.exit(1)

    if 'parameters' in parameters:
        configurations = generate_configurations(list(parameters['parameters'].keys()), parameters['parameters'])
    else:
        configurations = []

    if not configurations:
        logger.error(f"No configurations generated for '{selected_indicator}'. Using base indicator.")
        print(f"No configurations generated for '{selected_indicator}'. Using base indicator.")
    else:
        logger.info(f"Generated {len(configurations)} configurations for '{selected_indicator}'.")
        print(f"Generated {len(configurations)} configurations for '{selected_indicator}'.")
        example_configs = configurations[:15]
        logger.info(f"Example configurations for '{selected_indicator}': {example_configs}")
        print(f"Example configurations for '{selected_indicator}': {example_configs}")

    try:
        insert_tweaked_configs(conn, selected_indicator, configurations)
        if configurations:
            logger.info("Configurations stored in database:")
            for config in configurations[:15]:
                logger.info(config)
                print(config)
        print(f"Configurations for '{selected_indicator}' have been added to the database.")
    except Exception as e:
        logger.error(f"Error inserting configurations into database for '{selected_indicator}': {e}")
        print(f"Error inserting configurations into database for '{selected_indicator}'. Exiting.")
        sys.exit(1)

    conn.close()
    logger.info(f"Configurations for '{selected_indicator}' have been added to the database.")
    print(f"Configurations for '{selected_indicator}' have been added to the database.")

if __name__ == "__main__":
    configure_logging(log_file_prefix="tweak_indicator")
    run_tweak_indicator()

====================
utils.py:
# utils.py
import os
import sys
import json
import shutil
import subprocess
import re
from pathlib import Path
from datetime import datetime,timedelta
from typing import List,Optional
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import seaborn as sns
from joblib import Parallel,delayed
from scipy.stats import t
from sklearn.preprocessing import StandardScaler
import dateutil.parser
from linear_regression import perform_linear_regression
from advanced_analysis import advanced_price_prediction
from load_data import load_data
from indicators import compute_all_indicators
from data_utils import clear_screen,prepare_data,determine_time_interval,get_original_indicators,handle_missing_indicators
from correlation_utils import load_or_calculate_correlations,calculate_correlation,calculate_and_save_indicator_correlations
from visualization_utils import generate_combined_correlation_chart,visualize_data,generate_heatmaps
from backup_utils import run_backup_cleanup
from table_generation import generate_best_indicator_table,generate_statistical_summary,generate_correlation_csv
from binance_historical_data_downloader import download_binance_data,fetch_klines,process_klines,save_to_sqlite
import warnings
warnings.filterwarnings('ignore')

def run_backup_cleanup():
    pass

def list_database_files(database_dir:str)->List[str]:
    return[f for f in os.listdir(database_dir)if f.endswith('.db')]

def select_existing_database(database_dir:str)->Optional[str]:
    db_files=list_database_files(database_dir)
    if not db_files:
        print("No existing databases found.")
        return None
    print("\nExisting Databases:")
    for idx,db in enumerate(db_files,1):
        print(f"{idx}. {db}")
    while True:
        selected=input(f"Enter the number of the database to select (1-{len(db_files)}) or type 'x' to go back: ").strip()
        if selected.lower()=='x':
            return None
        if selected.isdigit()and 1<=int(selected)<=len(db_files):
            selected_db=db_files[int(selected)-1]
            print(f"Selected Database: {selected_db}")
            return os.path.join(database_dir,selected_db)
        else:
            print("Invalid selection. Please try again.")

def preview_database(db_path:str)->None:
    try:
        data,is_reverse_chronological,_=load_data(db_path)
        if data.empty:
            print("The selected database is empty.")
        else:
            print(f"\nPreview of the latest data in '{os.path.basename(db_path)}':")
            print(data.tail())
    except:
        print(f"Failed to preview database '{db_path}'.")

def update_database(db_path:str)->None:
    try:
        base_filename=os.path.basename(db_path)
        symbol,interval=os.path.splitext(base_filename)[0].split('_')
    except ValueError:
        print(f"Database filename '{db_path}' does not follow the 'symbol_interval.db' format.")
        return
    print(f"Updating database for {symbol} with interval {interval}...")
    print("Please enter the date range.")
    start_date_input=input("Enter the start date (YYYY-MM-DD) or press Enter to use the latest date in the database: ").strip()
    end_date_input=input("Enter the end date (YYYY-MM-DD) or press Enter to use today's date: ").strip()
    try:
        data,is_reverse_chronological,_=load_data(db_path)
        if data.empty:
            print("The selected database is empty. Downloading full dataset.")
            download_binance_data(symbol,interval,db_path)
            return
    except:
        print(f"Failed to load data from '{db_path}'.")
        return
    if start_date_input:
        try:
            start_datetime=datetime.strptime(start_date_input,'%Y-%m-%d')
        except ValueError:
            print("Invalid start date format.")
            return
    else:
        latest_timestamp=data['Date'].max()
        start_datetime=latest_timestamp+timedelta(seconds=1)
        print(f"No start date entered. Using {start_datetime}")
    if end_date_input:
        try:
            end_datetime=datetime.strptime(end_date_input,'%Y-%m-%d')
        except ValueError:
            print("Invalid end date format.")
            return
    else:
        end_datetime=datetime.now()
        print(f"No end date entered. Using {end_datetime}")
    start_time=int(start_datetime.timestamp()*1000)
    end_time=int(end_datetime.timestamp()*1000)
    if start_time>=end_time:
        print("Start date must be before end date.")
        return
    try:
        klines=fetch_klines(symbol,interval,start_time,end_time)
        if not klines:
            print("No new data available.")
            return
        df=process_klines(klines)
        save_to_sqlite(df,db_path)
        print(f"Updated '{os.path.basename(db_path)}' with {len(df)} new records.")
    except:
        print(f"Failed to update database '{db_path}'.")

def download_new_dataset(database_dir:str,default_interval:str='1d')->Optional[str]:
    return create_new_database(database_dir,default_interval=default_interval)

def create_new_database(database_dir:str,default_interval:str='1d')->Optional[str]:
    symbol=input("Enter the trading symbol (e.g., BTCUSDT): ").strip().upper()
    if not symbol:
        print("Symbol cannot be empty.")
        return None
    interval=default_interval
    db_filename=f"{symbol}_{interval}.db"
    db_path=os.path.join(database_dir,db_filename)
    if os.path.exists(db_path):
        print(f"Database '{db_filename}' already exists.")
        return db_path
    else:
        download_binance_data(symbol,interval,db_path)
        return db_path

def perform_analysis(db_path:str,reports_dir:str,cache_dir:str,timestamp:str)->None:
    try:
        data,is_reverse_chronological,db_filename=load_data(db_path)
    except:
        print(f"Failed to load data: {db_path}")
        return
    if data.empty:
        print("No data found.")
        download_choice=input("Download now? (y/n): ").strip().lower()
        if download_choice=='y':
            try:
                symbol,interval=os.path.splitext(os.path.basename(db_path))[0].split('_')
            except ValueError:
                print("Filename invalid.")
                return
            download_binance_data(symbol,interval,db_path)
            try:
                data,is_reverse_chronological,db_filename=load_data(db_path)
                if data.empty:
                    print("DB empty.")
                    return
                else:
                    print("Data loaded.")
            except:
                print("Load fail.")
                return
        else:
            print("No data.")
            return
    try:
        data=compute_all_indicators(data)
    except:
        print("Fail.")
        return
    print("Data and indicators done.")
    try:
        time_interval=determine_time_interval(data)
    except:
        print("Fail.")
        return
    print(f"Interval: {time_interval}")
    print("Preparing data...")
    try:
        X_scaled,feature_names=prepare_data(data)
        print("Data prepared.")
    except:
        print("Prep fail.")
        return
    original_indicators=get_original_indicators(feature_names,data)
    expected_indicators=['FI','ichimoku','KCU_20_2.0','STOCHRSI_14_5_3_slowk','VI+_14']
    original_indicators=handle_missing_indicators(original_indicators,data,expected_indicators)
    if not original_indicators:
        print("No valid indicators.")
        return
    base_csv_filename=os.path.splitext(os.path.basename(db_filename))[0]
    max_lag=len(data)-51
    if max_lag<1:
        print("Insufficient data.")
        return
    try:
        correlations=load_or_calculate_correlations(data,original_indicators,max_lag,is_reverse_chronological,base_csv_filename,db_path=db_path)
    except ValueError as ve:
        print(str(ve))
        return
    except:
        print("Corr fail.")
        return
    try:
        summary_df=generate_statistical_summary(correlations,max_lag)
        summary_csv=os.path.join(reports_dir,f"{timestamp}_{base_csv_filename}_statistical_summary.csv")
        summary_df.to_csv(summary_csv,index=True)
        print(f"Summary: {summary_csv}")
    except:
        print("Fail.")
    try:
        generate_combined_correlation_chart(correlations,max_lag,time_interval,timestamp,base_csv_filename)
    except:
        print("Fail.")
    generate_charts=input("Charts? (y/n): ").strip().lower()=='y'
    generate_heatmaps_flag=input("Heatmaps? (y/n): ").strip().lower()=='y'
    save_correlation_csv=input("Save correlation CSV? (y/n): ").strip().lower()=='y'
    if generate_charts:
        try:
            visualize_data(data,X_scaled,feature_names,timestamp,is_reverse_chronological,time_interval,generate_charts,correlations,calculate_correlation,base_csv_filename)
            print("Viz done.")
        except:
            print("Fail.")
    if generate_heatmaps_flag:
        try:
            generate_heatmaps(data,timestamp,time_interval,generate_heatmaps_flag,correlations,calculate_correlation,base_csv_filename)
            print("Heatmaps done.")
        except:
            print("Fail.")
    try:
        best_indicators_df=generate_best_indicator_table(correlations,max_lag)
        best_indicators_csv=os.path.join(reports_dir,f"{timestamp}_{base_csv_filename}_best_indicators.csv")
        best_indicators_df.to_csv(best_indicators_csv,index=False)
        print(f"Best: {best_indicators_csv}")
    except:
        print("Fail.")
    if save_correlation_csv:
        try:
            generate_correlation_csv(correlations,max_lag,base_csv_filename,reports_dir)
        except TypeError as te:
            print(f"Error: {te}")
        except:
            print("Fail.")
    try:
        data['Date']=pd.to_datetime(data['Date'])
        latest_date_in_data=data['Date'].max()
        print(f"\nLatest: {latest_date_in_data}")
    except:
        print("Fail.")
        return
    current_datetime=datetime.now()
    time_interval_seconds={'1s':1,'1m':60,'5m':300,'15m':900,'30m':1800,'1h':3600,'4h':14400,'1d':86400,'1w':604800}
    if time_interval not in time_interval_seconds:
        print(f"Unsupported '{time_interval}'.")
        return
    time_diff_seconds=(current_datetime - latest_date_in_data).total_seconds()
    lag_periods_behind_current=int(time_diff_seconds/time_interval_seconds[time_interval])
    print("\nRelative time?")
    user_input=input("Future date/time: ").strip()
    try:
        future_datetime=parse_date_time_input(user_input,current_datetime)if user_input else current_datetime
        if user_input:
            print(f"Using {future_datetime}")
        else:
            print(f"Using current {future_datetime}")
    except:
        print("Parse err.")
        return
    lag_seconds=(future_datetime - latest_date_in_data).total_seconds()
    if lag_seconds<=0:
        print("Future after latest!")
        return
    lag_periods=int(lag_seconds/time_interval_seconds[time_interval])
    if lag_periods<1:
        print("Lag <1.")
        return
    if lag_periods>max_lag:
        print(f"Lag ≤ {max_lag}")
        return
    print(f"Lag: {lag_periods}")
    try:
        perform_linear_regression(data,correlations,max_lag,time_interval,timestamp,base_csv_filename,future_datetime,lag_periods)
        print("LR done.")
    except TypeError as te:
        print(f"LR err: {te}")
    except:
        print("LR fail.")
    try:
        advanced_price_prediction(data,correlations,max_lag,time_interval,timestamp,base_csv_filename,future_datetime,lag_periods)
        print("Adv done.")
    except TypeError as te:
        print(f"Adv err: {te}")
    except:
        print("Adv fail.")
    try:
        calculate_and_save_indicator_correlations(data,original_indicators,max_lag,is_reverse_chronological,db_path)
        print("Ind-ind corr done.")
    except:
        print("Fail.")
    print("All done.")

def parse_date_time_input(user_input:str,reference_datetime:datetime)->datetime:
    user_input=user_input.strip()
    if not user_input:
        return reference_datetime
    relative_time_pattern=r'^([+-]\d+)([smhdw])$'
    match=re.match(relative_time_pattern,user_input)
    if match:
        amount=int(match.group(1))
        unit=match.group(2)
        delta_kwargs={'s':'seconds','m':'minutes','h':'hours','d':'days','w':'weeks'}
        delta=timedelta(**{delta_kwargs[unit]:amount})
        return reference_datetime+delta
    for fmt in['%Y%m%d-%H%M','%Y%m%d']:
        try:
            return datetime.strptime(user_input,fmt)
        except ValueError:
            continue
    return dateutil.parser.parse(user_input,fuzzy=True)
====================
visualization_utils.py:
# visualization_utils.py
import os
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
from typing import Dict,List,Any,Callable
from datetime import datetime
from joblib import Parallel,delayed
from scipy.stats import t,zscore

def generate_combined_correlation_chart(correlations:Dict[str,List[float]],max_lag:int,time_interval:str,timestamp:str,base_csv_filename:str,output_dir:str='combined_charts')->None:
    os.makedirs(output_dir,exist_ok=True)
    max_positive_correlations=[]
    max_negative_correlations=[]
    max_absolute_correlations=[]
    for lag in range(1,max_lag+1):
        lag_correlations=[correlations[col][lag-1]for col in correlations if lag-1<len(correlations[col])]
        pos_correlations=[x for x in lag_correlations if x>0]
        max_pos=max(pos_correlations)if pos_correlations else 0
        neg_correlations=[x for x in lag_correlations if x<0]
        max_neg=min(neg_correlations)if neg_correlations else 0
        max_abs=max(max_pos,abs(max_neg))
        max_positive_correlations.append(max_pos)
        max_negative_correlations.append(max_neg)
        max_absolute_correlations.append(max_abs)
    plt.figure(figsize=(15,10))
    plt.plot(range(1,max_lag+1),max_positive_correlations,color='green',label='Max Positive Correlation')
    plt.plot(range(1,max_lag+1),max_negative_correlations,color='red',label='Max Negative Correlation')
    plt.plot(range(1,max_lag+1),max_absolute_correlations,color='blue',label='Max Absolute Correlation')
    plt.axhline(0,color='black',linewidth=0.5)
    plt.axvline(0,color='black',linewidth=0.5)
    plt.title('Maximum Positive, Negative, and Absolute Correlations at Each Lag Point',fontsize=14)
    plt.xlabel(f'Time Lag ({time_interval})',fontsize=12)
    plt.ylabel('Correlation',fontsize=12)
    plt.xticks(fontsize=10)
    plt.yticks(fontsize=10)
    plt.ylim(-1.0,1.0)
    plt.grid(True,which='both',linestyle='--',linewidth=0.5)
    plt.legend(loc='center left',bbox_to_anchor=(1,0.5),fontsize=10)
    plt.tight_layout()
    combined_filename=f"{timestamp}_{base_csv_filename}_max_correlation.png"
    combined_filepath=os.path.join(output_dir,combined_filename)
    plt.savefig(combined_filepath,bbox_inches='tight')
    plt.close()

def visualize_data(data:pd.DataFrame,features:pd.DataFrame,feature_columns:List[str],timestamp:str,is_reverse_chronological:bool,time_interval:str,generate_charts:bool,cache:Dict[str,Any],calculate_correlation_func:Callable[...,float],base_csv_filename:str)->None:
    if not generate_charts:
        return
    charts_dir='indicator_charts'
    os.makedirs(charts_dir,exist_ok=True)
    max_lag=len(data)-51
    if max_lag<=0:
        return
    correlations={}
    original_indicators=[col for col in feature_columns if not any(future in col for future in['future_1d','future_5d','future_10d','future_20d'])and col!='Close']
    original_indicators=[col for col in original_indicators if data[col].notna().any()and data[col].var()>1e-6]
    for col in original_indicators:
        if col not in cache:
            corr_list=Parallel(n_jobs=-1)(delayed(calculate_correlation_func)(data,col,lag,is_reverse_chronological)for lag in range(1,max_lag+1))
            cache[col]=corr_list
        else:
            corr_list=cache[col]
        correlations[col]=corr_list
        plt.figure(figsize=(10,4))
        plt.axhline(0,color='black',linewidth=0.5)
        plt.axvline(0,color='black',linewidth=0.5)
        plt.fill_between(range(1,max_lag+1),corr_list,where=np.array(corr_list)>0,color='blue',alpha=0.3)
        plt.fill_between(range(1,max_lag+1),corr_list,where=np.array(corr_list)<0,color='red',alpha=0.3)
        n=len(corr_list)
        if n>1:
            std_err=np.std(corr_list,ddof=1)/np.sqrt(n)
            margin_of_error=t.ppf(0.975,n-1)*std_err
            lower_bound=np.array(corr_list)-margin_of_error
            upper_bound=np.array(corr_list)+margin_of_error
            plt.fill_between(range(1,max_lag+1),lower_bound,upper_bound,color='gray',alpha=0.4,label='95% CI')
        plt.title(f'Average Correlation of {col} with Close Price',fontsize=10)
        plt.xlabel(f'Time Lag ({time_interval})',fontsize=8)
        plt.ylabel('Average Correlation',fontsize=8)
        plt.xticks(fontsize=6)
        plt.yticks(fontsize=6)
        plt.ylim(-1.0,1.0)
        plt.grid(True,which='both',linestyle='--',linewidth=0.5)
        plt.tight_layout()
        filename=f"{timestamp}_{base_csv_filename}_{col}_correlation.png"
        filepath=os.path.join(charts_dir,filename)
        plt.savefig(filepath,bbox_inches='tight')
        plt.close()
    combined_charts_dir='combined_charts'
    os.makedirs(combined_charts_dir,exist_ok=True)
    sorted_indicators=sorted(original_indicators,key=lambda col:correlations[col][-1]if len(correlations[col])>0 else 0,reverse=True)
    plt.figure(figsize=(15,10))
    colors=plt.cm.rainbow(np.linspace(0,1,len(sorted_indicators)))
    for col,color in zip(sorted_indicators,colors):
        plt.plot(range(1,max_lag+1),correlations[col],color=color,label=col)
    plt.axhline(0,color='black',linewidth=0.5)
    plt.axvline(0,color='black',linewidth=0.5)
    plt.title('Average Correlation of All Indicators with Close Price',fontsize=14)
    plt.xlabel(f'Time Lag ({time_interval})',fontsize=12)
    plt.ylabel('Average Correlation',fontsize=12)
    plt.xticks(fontsize=10)
    plt.yticks(fontsize=10)
    plt.ylim(-1.0,1.0)
    plt.grid(True,which='both',linestyle='--',linewidth=0.5)
    plt.legend(loc='center left',bbox_to_anchor=(1,0.5),fontsize=10)
    plt.tight_layout()
    combined_filename=f"{timestamp}_{base_csv_filename}_combined_correlation.png"
    combined_filepath=os.path.join(combined_charts_dir,combined_filename)
    plt.savefig(combined_filepath,bbox_inches='tight')
    plt.close()
    max_positive_correlations=[]
    max_negative_correlations=[]
    max_absolute_correlations=[]
    for lag in range(1,max_lag+1):
        lag_correlations=[correlations[col][lag-1]for col in original_indicators]
        pos_values=[x for x in lag_correlations if x>0]
        neg_values=[x for x in lag_correlations if x<0]
        max_pos=max(pos_values)if pos_values else 0
        max_neg=min(neg_values)if neg_values else 0
        max_abs=max(max_pos,abs(max_neg))
        max_positive_correlations.append(max_pos)
        max_negative_correlations.append(max_neg)
        max_absolute_correlations.append(max_abs)
    plt.figure(figsize=(15,10))
    plt.plot(range(1,max_lag+1),max_positive_correlations,color='green',label='Max Positive Correlation')
    plt.plot(range(1,max_lag+1),max_negative_correlations,color='red',label='Max Negative Correlation')
    plt.plot(range(1,max_lag+1),max_absolute_correlations,color='blue',label='Max Absolute Correlation')
    plt.axhline(0,color='black',linewidth=0.5)
    plt.axvline(0,color='black',linewidth=0.5)
    plt.title('Maximum Positive, Negative, and Absolute Correlations at Each Lag Point',fontsize=14)
    plt.xlabel(f'Time Lag ({time_interval})',fontsize=12)
    plt.ylabel('Correlation',fontsize=12)
    plt.xticks(fontsize=10)
    plt.yticks(fontsize=10)
    plt.ylim(-1.0,1.0)
    plt.grid(True,which='both',linestyle='--',linewidth=0.5)
    plt.legend(loc='center left',bbox_to_anchor=(1,0.5),fontsize=10)
    plt.tight_layout()
    combined_filename=f"{timestamp}_{base_csv_filename}_max_correlation.png"
    combined_filepath=os.path.join(combined_charts_dir,combined_filename)
    plt.savefig(combined_filepath,bbox_inches='tight')
    plt.close()
====================
visualize_data.py:
# visualize_data.py
import os
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from scipy.stats import t
from joblib import Parallel,delayed

def visualize_data(data,features,feature_columns,timestamp,is_reverse_chronological,time_interval,generate_charts,cache,calculate_correlation,base_csv_filename):
    if not generate_charts:
        return
    charts_dir='indicator_charts'
    if not os.path.exists(charts_dir):
        os.makedirs(charts_dir)
    else:
        existing_files=os.listdir(charts_dir)
        if existing_files:
            delete_choice=input(f"Do you want to delete them? (y/n): ").lower()
            if delete_choice=='y':
                for file in existing_files:
                    file_path=os.path.join(charts_dir,file)
                    if os.path.isfile(file_path):
                        os.remove(file_path)
    max_lag=len(data)-51
    correlations={}
    original_indicators=[col for col in feature_columns if not any(future in col for future in['future_1d','future_5d','future_10d','future_20d'])and col!='Close']
    original_indicators=[col for col in original_indicators if data[col].notna().any()and data[col].var()>1e-6]
    print(f"Original indicators in visualize_data.py: {original_indicators}")
    for col in original_indicators:
        if col not in cache:
            corr_list=Parallel(n_jobs=-1)(delayed(calculate_correlation)(data,col,lag,is_reverse_chronological)for lag in range(1,max_lag+1))
            cache[col]=corr_list
        else:
            corr_list=cache[col]
        correlations[col]=corr_list
        plt.figure(figsize=(10,4))
        plt.axhline(0,color='black',linewidth=0.5)
        plt.axvline(0,color='black',linewidth=0.5)
        plt.fill_between(range(1,max_lag+1),corr_list,where=np.array(corr_list)>0,color='blue',alpha=0.3)
        plt.fill_between(range(1,max_lag+1),corr_list,where=np.array(corr_list)<0,color='red',alpha=0.3)
        n=len(corr_list)
        if n>1:
            std_err=np.std(corr_list,ddof=1)/np.sqrt(n)
            margin_of_error=t.ppf(0.975,n-1)*std_err
            lower_bound=np.array(corr_list)-margin_of_error
            upper_bound=np.array(corr_list)+margin_of_error
            plt.fill_between(range(1,max_lag+1),lower_bound,upper_bound,color='gray',alpha=0.4,label='95% CI')
        plt.title(f'Average Correlation of {col} with Close Price',fontsize=10)
        plt.xlabel(f'Time Lag ({time_interval})',fontsize=8)
        plt.ylabel('Average Correlation',fontsize=8)
        plt.xticks(fontsize=6)
        plt.yticks(fontsize=6)
        plt.ylim(-1.0,1.0)
        plt.grid(True,which='both',linestyle='--',linewidth=0.5)
        plt.tight_layout()
        filename=f"{timestamp}_{base_csv_filename}_{col}_correlation.png"
        filepath=os.path.join(charts_dir,filename)
        plt.savefig(filepath,bbox_inches='tight')
        plt.close()
    combined_charts_dir='combined_charts'
    if not os.path.exists(combined_charts_dir):
        os.makedirs(combined_charts_dir)
    sorted_indicators=sorted(original_indicators,key=lambda col:correlations[col][-1] if len(correlations[col])>0 else 0,reverse=True)
    plt.figure(figsize=(15,10))
    colors=plt.cm.rainbow(np.linspace(0,1,len(sorted_indicators)))
    for col,color in zip(sorted_indicators,colors):
        plt.plot(range(1,max_lag+1),correlations[col],color=color,label=col)
    plt.axhline(0,color='black',linewidth=0.5)
    plt.axvline(0,color='black',linewidth=0.5)
    plt.title('Average Correlation of All Indicators with Close Price',fontsize=14)
    plt.xlabel(f'Time Lag ({time_interval})',fontsize=12)
    plt.ylabel('Average Correlation',fontsize=12)
    plt.xticks(fontsize=10)
    plt.yticks(fontsize=10)
    plt.ylim(-1.0,1.0)
    plt.grid(True,which='both',linestyle='--',linewidth=0.5)
    plt.legend(loc='center left',bbox_to_anchor=(1,0.5),fontsize=10)
    plt.tight_layout()
    combined_filename=f"{timestamp}_{base_csv_filename}_combined_correlation.png"
    combined_filepath=os.path.join(combined_charts_dir,combined_filename)
    plt.savefig(combined_filepath,bbox_inches='tight')
    plt.close()
    max_positive_correlations=[]
    max_negative_correlations=[]
    for lag in range(1,max_lag+1):
        lag_correlations=[correlations[col][lag-1]for col in original_indicators]
        max_positive_correlations.append(max(lag_correlations,key=lambda x:x if x>0 else -np.inf))
        max_negative_correlations.append(min(lag_correlations,key=lambda x:x if x<0 else np.inf))
    plt.figure(figsize=(15,10))
    plt.plot(range(1,max_lag+1),max_positive_correlations,color='green',label='Max Positive Correlation')
    plt.plot(range(1,max_lag+1),max_negative_correlations,color='red',label='Max Negative Correlation')
    plt.axhline(0,color='black',linewidth=0.5)
    plt.axvline(0,color='black',linewidth=0.5)
    plt.title('Maximum Positive and Negative Correlations at Each Lag Point',fontsize=14)
    plt.xlabel(f'Time Lag ({time_interval})',fontsize=12)
    plt.ylabel('Correlation',fontsize=12)
    plt.xticks(fontsize=10)
    plt.yticks(fontsize=10)
    plt.ylim(-1.0,1.0)
    plt.grid(True,which='both',linestyle='--',linewidth=0.5)
    plt.legend(loc='center left',bbox_to_anchor=(1,0.5),fontsize=10)
    plt.tight_layout()
    combined_filename=f"{timestamp}_{base_csv_filename}_max_correlation.png"
    combined_filepath=os.path.join(combined_charts_dir,combined_filename)
    plt.savefig(combined_filepath,bbox_inches='tight')
    plt.close()
====================