image

.gemini/checkpoint-20260425.md

@@ -0,0 +1,19 @@
+# fumen-analyze 진행 리포트 (2026-04-25)
+
+## 1. 현재 상태
+- **목표**: TJA 채보 기반 상수 예측 (Feature 및 Factor 방식)
+- **현재 상황**: 
+    - Feature 및 Factor 학습/추론 파이프라인 구축 완료.
+    - XGBoost 및 LightGBM 모델 비교 중.
+    - `script/compare_feature.ts` 실행 환경 점검 필요.
+- **데이터셋**: 1,000개 이상의 TJA 파일 파싱 및 피처화 성공.
+
+## 2. 주요 개선사항
+- **파이프라인 고도화**: TypeScript로 전처리(Parse, Featurize, Factorize)를 모듈화하여 일관성 확보.
+- **예측 도구**: Python(XGBoost/LightGBM) 기반 예측 및 시각화 도구(`compare_feature.ts`, `compare_factor.ts`) 구성.
+- **상수 정답지**: `datas/measure.csv`를 기준으로 모델 성능 정량적 평가 체계 마련.
+
+## 3. 남은 작업
+- `script/compare_feature.ts` 오류 디버깅 및 안정화.
+- 모델별(XGBoost vs LightGBM) 성능 최적화.
+- 에러 분포 시각화(`compare.png`)를 통한 모델 약점 보완.

.gemini/notes.md

@@ -3,19 +3,20 @@
 ## Technical Stack
 - **Runtime**: [Bun](https://bun.sh/)
 - **Language**: TypeScript (Preprocessing), Python (Machine Learning)
-- **ML Library**: [XGBoost](https://xgboost.readthedocs.io/), [scikit-learn](https://scikit-learn.org/)
+- **ML Library**: [XGBoost](https://xgboost.readthedocs.io/), [LightGBM](https://lightgbm.readthedocs.io/), [scikit-learn](https://scikit-learn.org/)
 - **TJA Parser**: [tja-parser](https://www.npmjs.com/package/tja-parser)
 
 ## Key Directories
-- `preprocess/`: TJA 파싱 및 피처 추출 로직 (TypeScript)
+- `preprocess/`: TJA 파싱 및 피처/팩터 추출 로직 (TypeScript)
-- `script/`: 전처리, 학습 제어 스크립트
+- `script/`: 전처리, 학습 제어 및 결과 비교 스크립트
-- `train/`: XGBoost 학습 엔진 (Python)
+- `train/`: 학습 엔진 (Python - XGBoost, LightGBM)
 - `predict/`: 추론 엔진 (Python)
 - `datas/tja/`: 원본 TJA 데이터셋
 - `datas/measure.csv`: 정답지 (상수 데이터)
-- `test/`: 학습 결과물 (model.pkl, scaler.pkl, features.json)
+- `output/`: 학습 모델(pkl/pkl), scaler, 결과 데이터(json/png)
 
 ## Data Flow
-1. `datas/tja/*.tja` → `script/preprocess.ts` → `test/features.json`
+1. `datas/tja/*.tja` → `preprocess/*.ts` → `temp.json` (features/factors)
-2. `test/features.json` + `datas/measure.csv` → `train/train_xgboost.py` → `test/model.pkl`
+2. `temp.json` + `datas/measure.csv` → `train/*/train_*.py` → `model.*`, `scaler.*`
-3. `test/model.pkl` + `test/features.json` → `predict/predict_xgboost.py` → Result
+3. `model.*` + `temp.json` → `predict/*/predict_*.py` → Prediction Result
+4. `script/compare_*.ts` → Evaluation (MAE) & Visualization (PNG)

predict/factor/predict_lightgbm.py

@@ -0,0 +1,68 @@
+import argparse
+import json
+import os
+import joblib
+import numpy as np
+from pathlib import Path
+
+# =========================================================
+# Configuration (Must match training)
+# =========================================================
+MAX_NOTES = 2000
+FACTOR_COUNT = 4
+INPUT_DIM = MAX_NOTES * FACTOR_COUNT
+
+MODEL_FILENAME = "model.pkl"
+SCALER_FILENAME = "scaler.pkl"
+
+def safe_float(value):
+    try: return float(value)
+    except: return 0.0
+
+def predict():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--workingDir", required=True)
+    parser.add_argument("--songno", required=True)
+    parser.add_argument("--factor", required=True)
+    args = parser.parse_args()
+
+    model_path = os.path.join(args.workingDir, MODEL_FILENAME)
+    scaler_path = os.path.join(args.workingDir, SCALER_FILENAME)
+
+    if not os.path.exists(model_path):
+        print(f"Model not found: {model_path}")
+        return
+
+    model = joblib.load(model_path)
+    scaler = joblib.load(scaler_path)
+
+    with open(args.factor or (Path(args.workingDir) / 'factors.json'), "r", encoding="utf-8") as f:
+        data = json.load(f)
+
+    targets = [item for item in data if str(item["songno"]) == str(args.songno)]
+    if not targets:
+        print(f"No data found for songno: {args.songno}")
+        return
+
+    results = []
+    for item in targets:
+        raw_factors = item["factors"]
+        vector = np.zeros(INPUT_DIM, dtype=np.float32)
+        
+        for i in range(min(len(raw_factors), MAX_NOTES)):
+            for j in range(FACTOR_COUNT):
+                vector[i * FACTOR_COUNT + j] = safe_float(raw_factors[i][j])
+        
+        X = scaler.transform([vector])
+        pred = model.predict(X)[0]
+        
+        results.append({
+            "songno": item["songno"],
+            "diff": item["difficulty"],
+            "predicted": float(pred)
+        })
+
+    print(json.dumps(results, indent=2))
+
+if __name__ == "__main__":
+    predict()

predict/factor/predict_xgboost.py

@@ -0,0 +1,70 @@
+import argparse
+import json
+import os
+import joblib
+import numpy as np
+from pathlib import Path
+
+# =========================================================
+# Configuration (Must match training)
+# =========================================================
+MAX_NOTES = 2000
+FACTOR_COUNT = 4
+INPUT_DIM = MAX_NOTES * FACTOR_COUNT
+
+MODEL_FILENAME = "model.pkl"
+SCALER_FILENAME = "scaler.pkl"
+
+def safe_float(value):
+    try: return float(value)
+    except: return 0.0
+
+def predict():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--workingDir", required=True)
+    parser.add_argument("--songno", required=True)
+    parser.add_argument("--factor", required=False) # Input factor JSON file
+    args = parser.parse_args()
+
+    model_path = os.path.join(args.workingDir, MODEL_FILENAME)
+    scaler_path = os.path.join(args.workingDir, SCALER_FILENAME)
+
+    if not os.path.exists(model_path):
+        print(f"Model not found: {model_path}")
+        return
+
+    model = joblib.load(model_path)
+    scaler = joblib.load(scaler_path)
+
+    with open(args.factor or (Path(args.workingDir) / 'factors.json'), "r", encoding="utf-8") as f:
+        data = json.load(f)
+
+    # Filter by songno
+    targets = [item for item in data if str(item["songno"]) == str(args.songno)]
+    if not targets:
+        print(f"No data found for songno: {args.songno}")
+        return
+
+    results = []
+    for item in targets:
+        raw_factors = item["factors"]
+        vector = np.zeros(INPUT_DIM, dtype=np.float32)
+        
+        for i in range(min(len(raw_factors), MAX_NOTES)):
+            for j in range(FACTOR_COUNT):
+                vector[i * FACTOR_COUNT + j] = safe_float(raw_factors[i][j])
+        
+        # Scale and Predict
+        X = scaler.transform([vector])
+        pred = model.predict(X)[0]
+        
+        results.append({
+            "songno": item["songno"],
+            "diff": item["difficulty"],
+            "predicted": float(pred)
+        })
+
+    print(json.dumps(results, indent=2))
+
+if __name__ == "__main__":
+    predict()

script/compare_factor.ts

@@ -0,0 +1,181 @@
+import Bun from 'bun';
+import path from 'node:path';
+import { parseArgs } from 'node:util';
+import fs from 'node:fs';
+
+const { values } = parseArgs({
+    args: Bun.argv,
+    options: {
+        workingDir: { type: "string" },
+        dataDir: { type: "string" },
+        script: { type: "string" },
+    },
+    strict: true,
+    allowPositionals: true,
+});
+
+if (!values.workingDir || !values.dataDir || !values.script) {
+    console.error("Usage: bun run script/compare_factor.ts --workingDir <dir> --dataDir <dir> --script <python_script>");
+    process.exit(1);
+}
+
+const workingDir = values.workingDir;
+const dataDir = values.dataDir;
+const predictScript = values.script;
+const tempFileName = "temp.json";
+const tempFilePath = path.join(workingDir, tempFileName);
+
+// 1. 전처리 실행 (Factor 기반)
+console.log("Step 1: Running factor preprocessing to temp.json...");
+const preprocessResult = Bun.spawnSync([
+    "bun", "run", "script/preprocess_factor.ts",
+    "--workingDir", workingDir,
+    "--dataDir", dataDir,
+    "--fileName", tempFileName
+]);
+
+if (!preprocessResult.success) {
+    console.error("Preprocessing failed");
+    console.error(preprocessResult.stderr.toString());
+    process.exit(1);
+}
+
+// 2. measure.csv 로드
+console.log("Step 2: Loading measure.csv...");
+const measurePath = path.join(dataDir, "measure.csv");
+const measureContent = fs.readFileSync(measurePath, "utf-8");
+const measureMap = new Map<string, number>();
+
+measureContent.split("\n").forEach((line, index) => {
+    if (index === 0 || !line.trim()) return;
+    const parts = line.split(",");
+    if (parts.length >= 3) {
+        const constant = parts[0];
+        const songno = parts[1];
+        const diff = parts[2];
+        measureMap.set(`${songno.trim()}_${diff.trim()}`, parseFloat(constant));
+    }
+});
+
+// 3. temp.json 로드하여 대상 곡 목록 추출
+const factors = JSON.parse(fs.readFileSync(tempFilePath, "utf-8"));
+const uniqueSongnos = Array.from(new Set(factors.map((f: any) => f.songno)));
+
+// 4. 예측 및 비교
+console.log(`Step 3: Predicting and comparing ${uniqueSongnos.length} songs (Factor-based)...`);
+const comparisonResults: any[] = [];
+let processedCount = 0;
+
+for (const songno of uniqueSongnos) {
+    try {
+        const predictProcess = Bun.spawnSync([
+            "python3", predictScript,
+            "--workingDir", workingDir,
+            "--songno", songno as string,
+            "--factor", tempFilePath // 파이썬 스크립트 인자명이 --factor로 고정되어 있는 경우를 가정
+        ]);
+
+        if (!predictProcess.success) {
+            console.error(`\n[ERROR] Failed to predict songno ${songno}`);
+            processedCount++;
+            continue;
+        }
+
+        const output = predictProcess.stdout.toString().trim();
+        const jsonStart = output.indexOf('[');
+        const jsonEnd = output.lastIndexOf(']') + 1;
+
+        if (jsonStart !== -1 && jsonEnd !== 0) {
+            const predictions = JSON.parse(output.substring(jsonStart, jsonEnd));
+            predictions.forEach((pred: any) => {
+                const key = `${pred.songno}_${pred.diff}`;
+                const actual = measureMap.get(key);
+                if (actual !== undefined) {
+                    comparisonResults.push({
+                        songno: pred.songno,
+                        diff: pred.diff,
+                        actual: actual,
+                        predicted: pred.predicted,
+                        error: Math.abs(actual - pred.predicted)
+                    });
+                }
+            });
+        }
+
+        processedCount++;
+        if (processedCount % 10 === 0 || processedCount === uniqueSongnos.length) {
+            const percent = ((processedCount / uniqueSongnos.length) * 100).toFixed(1);
+            process.stdout.write(`\rProgress: ${processedCount}/${uniqueSongnos.length} (${percent}%)   `);
+        }
+    } catch (err) {
+        console.error(`\nError processing songno ${songno}:`, err);
+        processedCount++;
+    }
+}
+console.log("\nPrediction finished.");
+
+const avgError = comparisonResults.reduce((acc, curr) => acc + curr.error, 0) / comparisonResults.length;
+const resultData = {
+    summary: {
+        total_compared: comparisonResults.length,
+        average_absolute_error: avgError,
+        timestamp: new Date().toISOString(),
+        script_used: predictScript,
+        type: "factor"
+    },
+    details: comparisonResults.sort((a, b) => b.error - a.error)
+};
+
+const comparePath = path.join(workingDir, "compare.json");
+fs.writeFileSync(comparePath, JSON.stringify(resultData, null, 2), "utf-8");
+
+console.log(`\nComparison complete! Results saved to: ${comparePath}`);
+
+// 6. 결과 시각화 (compare.png 생성)
+if (comparisonResults.length > 0) {
+    console.log("Step 4: Generating comparison plot (compare.png)...");
+    const plotPythonCode = `
+import pandas as pd
+import matplotlib.pyplot as plt
+import seaborn as sns
+import json
+import os
+
+with open('${comparePath}', 'r', encoding='utf-8') as f:
+    data = json.load(f)
+
+if not data['details']:
+    print("No details to plot.")
+    exit(0)
+
+df = pd.DataFrame(data['details'])
+df['abs_error'] = (df['actual'] - df['predicted']).abs()
+df = df.sort_values('abs_error', ascending=False).reset_index(drop=True)
+
+plt.figure(figsize=(12, 6))
+plt.switch_backend('Agg')
+sns.scatterplot(x=df.index, y=df['abs_error'], alpha=0.6, s=20, color='royalblue')
+
+plt.axhline(0.2, color='green', linestyle='--', linewidth=0.8, alpha=0.5, label='Target (0.2)')
+plt.axhline(0.5, color='blue', linestyle='--', linewidth=0.8, alpha=0.5)
+plt.axhline(1.0, color='red', linestyle='--', linewidth=0.8, alpha=0.5)
+
+plt.ylim(0, max(3.5, df['abs_error'].max() + 0.5) if not df.empty else 3.5)
+plt.title('Comparison Absolute Error Distribution (Feature-based)', fontsize=14)
+plt.xlabel('Samples (Sorted by Error Magnitude)', fontsize=12)
+plt.ylabel('Absolute Error', fontsize=12)
+plt.grid(True, axis='y', alpha=0.3)
+
+plot_path = os.path.join('${workingDir}', 'compare.png')
+plt.savefig(plot_path)
+print(f"Plot saved to: {plot_path}")
+`;
+
+    const plotProc = Bun.spawnSync(["python3", "-c", plotPythonCode]);
+    if (plotProc.success) {
+        console.log(plotProc.stdout.toString().trim());
+    } else {
+        console.error("Failed to generate plot:");
+        console.error(plotProc.stderr.toString());
+    }
+}

script/compare_feature.ts

@@ -15,7 +15,7 @@ const { values } = parseArgs({
 });
 
 if (!values.workingDir || !values.dataDir || !values.script) {
-    console.error("Usage: bun run script/compare.ts --workingDir <dir> --dataDir <dir> --script <python_script>");
+    console.error("Usage: bun run script/compare_feature.ts --workingDir <dir> --dataDir <dir> --script <python_script>");
     process.exit(1);
 }
 
@@ -25,10 +25,10 @@ const predictScript = values.script;
 const tempFileName = "temp.json";
 const tempFilePath = path.join(workingDir, tempFileName);
 
-// 1. 전처리 실행 (temp.json 생성하여 기존 features.json 보존)
+// 1. 전처리 실행 (Feature 기반)
-console.log("Step 1: Running preprocessing to temp.json...");
+console.log("Step 1: Running feature preprocessing to temp.json...");
 const preprocessResult = Bun.spawnSync([
-    "bun", "run", "script/preprocess.ts",
+    "bun", "run", "script/preprocess_feature.ts",
     "--workingDir", workingDir,
     "--dataDir", dataDir,
     "--fileName", tempFileName
@@ -62,7 +62,7 @@ const features = JSON.parse(fs.readFileSync(tempFilePath, "utf-8"));
 const uniqueSongnos = Array.from(new Set(features.map((f: any) => f.songno)));
 
 // 4. 예측 및 비교
-console.log(`Step 3: Predicting and comparing ${uniqueSongnos.length} songs...`);
+console.log(`Step 3: Predicting and comparing ${uniqueSongnos.length} songs (Feature-based)...`);
 const comparisonResults: any[] = [];
 let processedCount = 0;
 
@@ -77,39 +77,31 @@ for (const songno of uniqueSongnos) {
 
         if (!predictProcess.success) {
             console.error(`\n[ERROR] Failed to predict songno ${songno}`);
-            console.error(predictProcess.stderr.toString());
             processedCount++;
             continue;
         }
 
         const output = predictProcess.stdout.toString().trim();
-        // JSON 부분만 추출 (경고문 등이 섞여있을 경우 대비)
         const jsonStart = output.indexOf('[');
         const jsonEnd = output.lastIndexOf(']') + 1;
-        
+
-        if (jsonStart === -1 || jsonEnd === 0) {
+        if (jsonStart !== -1 && jsonEnd !== 0) {
-            console.error(`\n[ERROR] Invalid output format for songno ${songno}`);
+            const predictions = JSON.parse(output.substring(jsonStart, jsonEnd));
-            processedCount++;
+            predictions.forEach((pred: any) => {
-            continue;
+                const key = `${pred.songno}_${pred.diff}`;
+                const actual = measureMap.get(key);
+                if (actual !== undefined) {
+                    comparisonResults.push({
+                        songno: pred.songno,
+                        diff: pred.diff,
+                        actual: actual,
+                        predicted: pred.predicted,
+                        error: Math.abs(actual - pred.predicted)
+                    });
+                }
+            });
         }
 
-        const predictions = JSON.parse(output.substring(jsonStart, jsonEnd));
-
-        predictions.forEach((pred: any) => {
-            const key = `${pred.songno}_${pred.diff}`;
-            const actual = measureMap.get(key);
-
-            if (actual !== undefined) {
-                comparisonResults.push({
-                    songno: pred.songno,
-                    diff: pred.diff,
-                    actual: actual,
-                    predicted: pred.predicted,
-                    error: Math.abs(actual - pred.predicted)
-                });
-            }
-        });
-
         processedCount++;
         if (processedCount % 10 === 0 || processedCount === uniqueSongnos.length) {
             const percent = ((processedCount / uniqueSongnos.length) * 100).toFixed(1);
@@ -122,27 +114,68 @@ for (const songno of uniqueSongnos) {
 }
 console.log("\nPrediction finished.");
 
-// 5. 결과 분석 및 저장
-if (comparisonResults.length === 0) {
-    console.error("No comparison results were generated.");
-    process.exit(1);
-}
-
 const avgError = comparisonResults.reduce((acc, curr) => acc + curr.error, 0) / comparisonResults.length;
 const resultData = {
     summary: {
         total_compared: comparisonResults.length,
         average_absolute_error: avgError,
         timestamp: new Date().toISOString(),
-        script_used: predictScript
+        script_used: predictScript,
+        type: "feature"
     },
     details: comparisonResults.sort((a, b) => b.error - a.error)
 };
 
-const comparePath = path.join(workingDir, "compare.json");
+const comparePath = path.join(workingDir, "compare_feature.json");
 fs.writeFileSync(comparePath, JSON.stringify(resultData, null, 2), "utf-8");
 
-console.log(`\nComparison complete!`);
+console.log(`\nComparison complete! Results saved to: ${comparePath}`);
-console.log(`Total compared: ${comparisonResults.length}`);
+
-console.log(`Average Error: ${avgError.toFixed(4)}`);
+// 6. 결과 시각화 (compare.png 생성)
-console.log(`Results saved to: ${comparePath}`);
+if (comparisonResults.length > 0) {
+    console.log("Step 4: Generating comparison plot (compare.png)...");
+    const plotPythonCode = `
+import pandas as pd
+import matplotlib.pyplot as plt
+import seaborn as sns
+import json
+import os
+
+with open('${comparePath}', 'r', encoding='utf-8') as f:
+    data = json.load(f)
+
+if not data['details']:
+    print("No details to plot.")
+    exit(0)
+
+df = pd.DataFrame(data['details'])
+df['abs_error'] = (df['actual'] - df['predicted']).abs()
+df = df.sort_values('abs_error', ascending=False).reset_index(drop=True)
+
+plt.figure(figsize=(12, 6))
+plt.switch_backend('Agg')
+sns.scatterplot(x=df.index, y=df['abs_error'], alpha=0.6, s=20, color='royalblue')
+
+plt.axhline(0.2, color='green', linestyle='--', linewidth=0.8, alpha=0.5, label='Target (0.2)')
+plt.axhline(0.5, color='blue', linestyle='--', linewidth=0.8, alpha=0.5)
+plt.axhline(1.0, color='red', linestyle='--', linewidth=0.8, alpha=0.5)
+
+plt.ylim(0, max(3.5, df['abs_error'].max() + 0.5) if not df.empty else 3.5)
+plt.title('Comparison Absolute Error Distribution (Feature-based)', fontsize=14)
+plt.xlabel('Samples (Sorted by Error Magnitude)', fontsize=12)
+plt.ylabel('Absolute Error', fontsize=12)
+plt.grid(True, axis='y', alpha=0.3)
+
+plot_path = os.path.join('${workingDir}', 'compare.png')
+plt.savefig(plot_path)
+print(f"Plot saved to: {plot_path}")
+`;
+
+    const plotProc = Bun.spawnSync(["python3", "-c", plotPythonCode]);
+    if (plotProc.success) {
+        console.log(plotProc.stdout.toString().trim());
+    } else {
+        console.error("Failed to generate plot:");
+        console.error(plotProc.stderr.toString());
+    }
+}

script/predict_tja.ts

@@ -1,94 +0,0 @@
-import Bun from 'bun';
-import path from 'node:path';
-import { parseArgs } from 'node:util';
-import fs from 'node:fs';
-import { featurize } from '../preprocess/featurize';
-import { parseTja } from '../preprocess/parse';
-
-const { values } = parseArgs({
-    args: Bun.argv,
-    options: {
-        tja: {
-            type: "string"
-        },
-        workingDir: {
-            type: "string"
-        },
-        script: {
-            type: "string"
-        }
-    },
-    allowPositionals: true,
-});
-
-if (!values.tja || !values.workingDir || !values.script) {
-    console.error("Usage: bun script/predict_tja.ts --tja <path_to_tja> [--workingDir <dir>] [--script <python_script>]");
-    process.exit(1);
-}
-
-const tjaPath = values.tja;
-if (!fs.existsSync(tjaPath)) {
-    console.error(`File not found: ${tjaPath}`);
-    process.exit(1);
-}
-
-const workingDir = values.workingDir!;
-const pythonScript = values.script!;
-
-if (!fs.existsSync(pythonScript)) {
-    console.error(`Python script not found: ${pythonScript}`);
-    process.exit(1);
-}
-
-// 1. TJA 파싱 및 피처 추출
-const tjaContent = fs.readFileSync(tjaPath, 'utf-8');
-const songno = path.basename(tjaPath, '.tja');
-const parsed = parseTja(tjaContent);
-
-const features: any[] = [];
-if (parsed.oni) {
-    features.push({
-        songno,
-        difficulty: 'oni',
-        ...featurize(parsed.oni)
-    });
-}
-if (parsed.edit) {
-    features.push({
-        songno,
-        difficulty: 'ura',
-        ...featurize(parsed.edit)
-    });
-}
-
-if (features.length === 0) {
-    console.error("No Oni or Ura difficulty found in the TJA file.");
-    process.exit(1);
-}
-
-// 2. 임시 피처 파일 저장
-const tempFeaturePath = path.join(workingDir, `temp_predict_${Date.now()}.json`);
-fs.writeFileSync(tempFeaturePath, JSON.stringify(features, null, 2), 'utf-8');
-
-try {
-    // 3. Python 예측 스크립트 실행
-    const proc = Bun.spawnSync([
-        "python3",
-        pythonScript,
-        "--workingDir", workingDir,
-        "--feature", tempFeaturePath,
-        "--songno", songno
-    ]);
-
-    if (proc.success) {
-        console.log(proc.stdout.toString());
-    } else {
-        console.error("Prediction failed:");
-        console.error(proc.stderr.toString());
-    }
-} finally {
-    // 4. 임시 파일 삭제
-    if (fs.existsSync(tempFeaturePath)) {
-        fs.unlinkSync(tempFeaturePath);
-    }
-}

script/predict_tja_factor.ts

@@ -0,0 +1,68 @@
+import Bun from 'bun';
+import path from 'node:path';
+import { parseArgs } from 'node:util';
+import fs from 'node:fs';
+import { factorize } from '../preprocess/factorize';
+import { parseTja } from '../preprocess/parse';
+
+const { values } = parseArgs({
+    args: Bun.argv,
+    options: {
+        tja: { type: "string" },
+        workingDir: { type: "string" },
+        script: { type: "string" }
+    },
+    allowPositionals: true,
+});
+
+if (!values.tja || !values.workingDir || !values.script) {
+    console.error("Usage: bun script/predict_tja_factor.ts --tja <path_to_tja> --workingDir <dir> --script <python_script>");
+    process.exit(1);
+}
+
+const tjaPath = values.tja;
+const workingDir = values.workingDir!;
+const pythonScript = values.script!;
+
+const tjaContent = fs.readFileSync(tjaPath, 'utf-8');
+const songno = path.basename(tjaPath, '.tja');
+const parsed = parseTja(tjaContent);
+
+const factors: any[] = [];
+if (parsed.oni) {
+    factors.push({
+        songno,
+        difficulty: 'oni',
+        factors: factorize(parsed.oni)
+    });
+}
+if (parsed.edit) {
+    factors.push({
+        songno,
+        difficulty: 'ura',
+        factors: factorize(parsed.edit)
+    });
+}
+
+if (factors.length === 0) {
+    console.error("No Oni or Ura difficulty found.");
+    process.exit(1);
+}
+
+const tempFactorPath = path.join(workingDir, `temp_predict_fact_${Date.now()}.json`);
+fs.writeFileSync(tempFactorPath, JSON.stringify(factors, null, 2), 'utf-8');
+
+try {
+    const proc = Bun.spawnSync([
+        "python3",
+        pythonScript,
+        "--workingDir", workingDir,
+        "--feature", tempFactorPath, // 파이썬 스크립트 인자명이 --feature로 고정되어 있다고 가정
+        "--songno", songno
+    ]);
+
+    if (proc.success) console.log(proc.stdout.toString());
+    else console.error("Prediction failed:", proc.stderr.toString());
+} finally {
+    if (fs.existsSync(tempFactorPath)) fs.unlinkSync(tempFactorPath);
+}

script/predict_tja_feature.ts

@@ -0,0 +1,49 @@
+import Bun from 'bun';
+import path from 'node:path';
+import { parseArgs } from 'node:util';
+import fs from 'node:fs';
+import { featurize } from '../preprocess/featurize';
+import { parseTja } from '../preprocess/parse';
+
+const { values } = parseArgs({
+    args: Bun.argv,
+    options: {
+        tja: { type: "string" },
+        workingDir: { type: "string" },
+        script: { type: "string" }
+    },
+    allowPositionals: true,
+});
+
+if (!values.tja || !values.workingDir || !values.script) {
+    console.error("Usage: bun script/predict_tja_feature.ts --tja <path_to_tja> --workingDir <dir> --script <python_script>");
+    process.exit(1);
+}
+
+const tjaPath = values.tja;
+const workingDir = values.workingDir!;
+const pythonScript = values.script!;
+
+const tjaContent = fs.readFileSync(tjaPath, 'utf-8');
+const songno = path.basename(tjaPath, '.tja');
+const parsed = parseTja(tjaContent);
+
+const features: any[] = [];
+if (parsed.oni) features.push({ songno, difficulty: 'oni', ...featurize(parsed.oni) });
+if (parsed.edit) features.push({ songno, difficulty: 'ura', ...featurize(parsed.edit) });
+
+if (features.length === 0) {
+    console.error("No Oni or Ura difficulty found.");
+    process.exit(1);
+}
+
+const tempFeaturePath = path.join(workingDir, `temp_predict_feat_${Date.now()}.json`);
+fs.writeFileSync(tempFeaturePath, JSON.stringify(features, null, 2), 'utf-8');
+
+try {
+    const proc = Bun.spawnSync(["python3", pythonScript, "--workingDir", workingDir, "--feature", tempFeaturePath, "--songno", songno]);
+    if (proc.success) console.log(proc.stdout.toString());
+    else console.error("Prediction failed:", proc.stderr.toString());
+} finally {
+    if (fs.existsSync(tempFeaturePath)) fs.unlinkSync(tempFeaturePath);
+}

script/preprocess_factor.ts

@@ -0,0 +1,65 @@
+import Bun from 'bun';
+import path from 'node:path';
+import { parseArgs } from 'node:util';
+import fs, { mkdirSync } from 'node:fs';
+import { factorize } from '../preprocess/factorize';
+import { parseTja } from '../preprocess/parse'
+
+const { values } = parseArgs({
+    args: Bun.argv,
+    options: {
+        workingDir: {
+            type: "string"
+        },
+        dataDir: {
+            type: "string"
+        },
+        fileName: {
+            type: "string"
+        }
+    },
+    allowPositionals: true,
+})
+
+if (!values.dataDir || !values.workingDir) {
+    console.error("--workingDir --dataDir --fileName");
+    process.exit(1);
+}
+
+const workingDir = values.workingDir ?? '';
+if (!fs.existsSync(workingDir)) mkdirSync(workingDir)
+const dataDir = values.dataDir ?? '';
+
+const tjaDir = path.join(dataDir, 'tja');
+const files = fs.readdirSync(tjaDir);
+
+const results: any[] = [];
+for (const file of files) {
+    if (!file.endsWith('.tja')) continue;
+    const tja = fs.readFileSync(path.join(tjaDir, file), 'utf-8');
+    const songno = path.basename(file, '.tja');
+    try {
+        const parsed = parseTja(tja);
+        const courses = [
+            { diff: 'oni', data: parsed?.oni },
+            { diff: 'ura', data: parsed?.edit }
+        ];
+
+        for (const course of courses) {
+            if (course.data) {
+                results.push({
+                    songno,
+                    difficulty: course.diff,
+                    factors: factorize(course.data)
+                });
+            }
+        }
+    }
+    catch (err) {
+        console.error(`[Error] ${file}:`, err);
+    }
+}
+
+const outputPath = path.join(workingDir, values.fileName ?? 'factors.json');
+fs.writeFileSync(outputPath, JSON.stringify(results, null, 2), 'utf-8');
+console.log(`Successfully saved factors to ${outputPath}`);

script/train_factor.ts

@@ -0,0 +1,106 @@
+import Bun from 'bun';
+import { spawn } from 'node:child_process';
+import { parseArgs } from 'node:util';
+import fs from 'fs';
+import path from 'path';
+import { factorize } from '../preprocess/factorize';
+import { parseTja } from '../preprocess/parse'
+
+const { values } = parseArgs({
+    args: Bun.argv,
+    options: {
+        workingDir: {
+            type: "string"
+        },
+        dataDir: {
+            type: "string"
+        },
+        script: {
+            type: "string"
+        },
+        trainSize: {
+            type: 'string'
+        },
+        validSize: {
+            type: 'string'
+        }
+    },
+    allowPositionals: true,
+});
+
+if (!values.dataDir || !values.workingDir || !values.script) {
+    console.error("Usage: bun run script/train_factor.ts --workingDir <dir> --dataDir <dir> --script <python_script> --trainSize <num> --validSize <num>");
+    process.exit(1);
+}
+
+// 1. factors.json 생성 (없을 경우)
+generateFactors();
+
+// 2. 파이썬 학습 스크립트 실행
+console.log(`Starting training with factor-based script: ${values.script}`);
+const child = spawn("python3", [
+    values.script,
+    "--workingDir", values.workingDir,
+    "--dataDir", values.dataDir,
+    "--trainSize", (Number(values.trainSize) || 1000).toString(),
+    "--validSize", (Number(values.validSize) || 200).toString(),
+]);
+
+child.stdout.pipe(process.stdout);
+child.stderr.pipe(process.stderr);
+
+child.on("close", (code) => {
+    console.log(`Training process exited with code ${code}`);
+    process.exit(code || 0);
+});
+
+// functions
+function generateFactors() {
+    const workingDir = values.workingDir ?? '';
+    if (!fs.existsSync(workingDir)) fs.mkdirSync(workingDir, { recursive: true });
+    
+    const factorsPath = path.join(workingDir, 'factors.json');
+    if (fs.existsSync(factorsPath)) {
+        console.log('factors.json already exists, skipping generation.');
+        return;
+    }
+
+    const dataDir = values.dataDir ?? '';
+    const tjaDir = path.join(dataDir, 'tja');
+    if (!fs.existsSync(tjaDir)) {
+        console.error(`TJA directory not found: ${tjaDir}`);
+        return;
+    }
+
+    const files = fs.readdirSync(tjaDir).filter(f => f.endsWith('.tja'));
+    console.log(`Generating factors from ${files.length} TJA files...`);
+
+    const results: any[] = [];
+    for (const file of files) {
+        try {
+            const tja = fs.readFileSync(path.join(tjaDir, file), 'utf-8');
+            const songno = path.basename(file, '.tja');
+            const parsed = parseTja(tja);
+            
+            const courses = [
+                { diff: 'oni', data: parsed?.oni },
+                { diff: 'ura', data: parsed?.edit }
+            ];
+
+            for (const course of courses) {
+                if (course.data) {
+                    results.push({
+                        songno,
+                        difficulty: course.diff,
+                        factors: factorize(course.data)
+                    });
+                }
+            }
+        } catch (err) {
+            console.error(`Error processing ${file}:`, err);
+        }
+    }
+
+    fs.writeFileSync(factorsPath, JSON.stringify(results, null, 2), 'utf-8');
+    console.log(`factors.json generated at ${factorsPath}`);
+}

train/factor/train_lightgbm.py

@@ -0,0 +1,152 @@
+import argparse
+import csv
+import json
+import math
+import os
+import random
+import joblib
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+import seaborn as sns
+import lightgbm as lgb
+
+from sklearn.preprocessing import StandardScaler
+from sklearn.metrics import mean_absolute_error
+
+# =========================================================
+# Hyper Parameters
+# =========================================================
+
+MAX_NOTES = 2000
+FACTOR_COUNT = 4
+INPUT_DIM = MAX_NOTES * FACTOR_COUNT
+
+TRAIN_SIZE = 0
+VALID_SIZE = 0
+RANDOM_STATE = 42
+
+PARAMS = {
+    'objective': 'regression',
+    'metric': 'mae',
+    'verbosity': -1,
+    'boosting_type': 'gbdt',
+    'random_state': RANDOM_STATE,
+    'learning_rate': 0.02,
+    'num_leaves': 63,
+    'n_estimators': 2000
+}
+
+CONTINUE_TRAINING = True
+ERROR_TOLERANCE = 0.1
+
+# =========================================================
+# 파일명
+# =========================================================
+
+FACTORS_FILENAME = "factors.json"
+MEASURE_FILENAME = "measure.csv"
+MODEL_FILENAME = "model.pkl"
+SCALER_FILENAME = "scaler.pkl"
+
+def safe_float(value):
+    if value is None: return 0.0
+    x = float(value)
+    return x if math.isfinite(x) else 0.0
+
+def train_model(working_dir: str, data_dir: str):
+    random.seed(RANDOM_STATE)
+
+    factors_path = os.path.join(working_dir, FACTORS_FILENAME)
+    measure_path = os.path.join(data_dir, MEASURE_FILENAME)
+    model_path = os.path.join(working_dir, MODEL_FILENAME)
+    scaler_path = os.path.join(working_dir, SCALER_FILENAME)
+
+    with open(factors_path, "r", encoding="utf-8") as f:
+        factor_data = json.load(f)
+
+    feature_map = {(str(item["songno"]), str(item["difficulty"])): item["factors"] for item in factor_data}
+
+    dataset = []
+    with open(measure_path, "r", encoding="utf-8") as f:
+        reader = csv.reader(f)
+        next(reader, None)
+        for row in reader:
+            if len(row) < 3: continue
+            measure, songno, diff = safe_float(row[0]), str(row[1]), str(row[2])
+            key = (songno, diff)
+            
+            if key in feature_map:
+                raw_factors = feature_map[key]
+                vector = np.zeros(INPUT_DIM, dtype=np.float32)
+                for i in range(min(len(raw_factors), MAX_NOTES)):
+                    for j in range(FACTOR_COUNT):
+                        vector[i * FACTOR_COUNT + j] = safe_float(raw_factors[i][j])
+                dataset.append((vector, measure, songno, diff))
+
+    random.shuffle(dataset)
+    if len(dataset) < (TRAIN_SIZE + VALID_SIZE):
+        raise ValueError(f"Dataset size {len(dataset)} < required {TRAIN_SIZE + VALID_SIZE}")
+
+    train_ds = dataset[:TRAIN_SIZE]
+    valid_ds = dataset[TRAIN_SIZE:TRAIN_SIZE + VALID_SIZE]
+
+    X_train = np.array([x for x, _, _, _ in train_ds])
+    y_train = np.array([y for _, y, _, _ in train_ds])
+    X_valid = np.array([x for x, _, _, _ in valid_ds])
+    y_valid = np.array([y for _, y, _, _ in valid_ds])
+    valid_info = [(s, d) for _, _, s, d in valid_ds]
+
+    if CONTINUE_TRAINING and os.path.exists(scaler_path):
+        scaler = joblib.load(scaler_path)
+    else:
+        scaler = StandardScaler()
+        scaler.fit(X_train)
+
+    X_train = scaler.transform(X_train)
+    X_valid = scaler.transform(X_valid)
+
+    if CONTINUE_TRAINING and os.path.exists(model_path):
+        model = joblib.load(model_path)
+        model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], init_model=model, callbacks=[lgb.early_stopping(stopping_rounds=100)])
+    else:
+        model = lgb.LGBMRegressor(**PARAMS)
+        model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], callbacks=[lgb.early_stopping(stopping_rounds=100)])
+
+    pred = model.predict(X_valid)
+    mae = mean_absolute_error(y_valid, pred)
+    accuracy = np.sum(np.abs(pred - y_valid) <= ERROR_TOLERANCE) / len(y_valid)
+    print(f"MAE: {mae:.4f} | Accuracy: {accuracy:.4f}")
+
+    # Results save
+    validate_details = []
+    for i in range(len(y_valid)):
+        validate_details.append({"songno": valid_info[i][0], "diff": valid_info[i][1], "actual": float(y_valid[i]), "predicted": float(pred[i]), "error": float(y_valid[i] - pred[i])})
+    
+    validate_details.sort(key=lambda x: abs(x["error"]), reverse=True)
+    with open(os.path.join(working_dir, "validate.json"), "w", encoding="utf-8") as f:
+        json.dump({"summary": {"mae": float(mae), "accuracy": float(accuracy)}, "details": validate_details}, f, indent=2)
+
+    # Plot
+    plt.switch_backend('Agg')
+    df_plot = pd.DataFrame(validate_details)
+    df_plot['abs_error'] = df_plot['error'].abs()
+    df_plot = df_plot.sort_values('abs_error', ascending=False).reset_index(drop=True)
+    plt.figure(figsize=(12, 6))
+    sns.scatterplot(data=df_plot, x=df_plot.index, y='abs_error', color='teal')
+    plt.axhline(0.2, color='green', linestyle='--')
+    plt.ylim(0, 4)
+    plt.savefig(os.path.join(working_dir, "validate.png"))
+    
+    joblib.dump(model, model_path)
+    joblib.dump(scaler, scaler_path)
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--workingDir", required=True)
+    parser.add_argument("--dataDir", required=True)
+    parser.add_argument("--trainSize", required=True, type=int)
+    parser.add_argument("--validSize", required=True, type=int)
+    args = parser.parse_args()
+    TRAIN_SIZE, VALID_SIZE = args.trainSize, args.validSize
+    train_model(args.workingDir, args.dataDir)

train/factor/train_xgboost.py

@@ -0,0 +1,147 @@
+import argparse
+import csv
+import json
+import math
+import os
+import random
+import joblib
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+import seaborn as sns
+
+from xgboost import XGBRegressor
+from sklearn.preprocessing import StandardScaler
+from sklearn.metrics import mean_absolute_error
+
+# =========================================================
+# Hyper Parameters
+# =========================================================
+
+MAX_NOTES = 2000 # 분석할 최대 노트 수
+FACTOR_COUNT = 4 # [type, bpm, scroll, delta]
+INPUT_DIM = MAX_NOTES * FACTOR_COUNT
+
+TRAIN_SIZE = 0
+VALID_SIZE = 0
+RANDOM_STATE = 42
+
+N_ESTIMATORS = 500
+MAX_DEPTH = 6
+LEARNING_RATE = 0.05
+CONTINUE_TRAINING = True
+ERROR_TOLERANCE = 0.1
+
+# =========================================================
+# 파일명
+# =========================================================
+
+FACTORS_FILENAME = "factors.json"
+MEASURE_FILENAME = "measure.csv"
+MODEL_FILENAME = "model.pkl"
+SCALER_FILENAME = "scaler.pkl"
+
+def safe_float(value):
+    if value is None: return 0.0
+    x = float(value)
+    return x if math.isfinite(x) else 0.0
+
+def train_model(working_dir: str, data_dir: str):
+    random.seed(RANDOM_STATE)
+
+    factors_path = os.path.join(working_dir, FACTORS_FILENAME)
+    measure_path = os.path.join(data_dir, MEASURE_FILENAME)
+    model_path = os.path.join(working_dir, MODEL_FILENAME)
+    scaler_path = os.path.join(working_dir, SCALER_FILENAME)
+
+    with open(factors_path, "r", encoding="utf-8") as f:
+        factor_data = json.load(f)
+
+    # feature_map build: key -> list of factors
+    feature_map = {(str(item["songno"]), str(item["difficulty"])): item["factors"] for item in factor_data}
+
+    dataset = []
+    with open(measure_path, "r", encoding="utf-8") as f:
+        reader = csv.reader(f)
+        next(reader, None)
+        for row in reader:
+            if len(row) < 3: continue
+            measure, songno, diff = safe_float(row[0]), str(row[1]), str(row[2])
+            key = (songno, diff)
+            
+            if key in feature_map:
+                raw_factors = feature_map[key]
+                # 고정 길이 벡터로 변환 (Padding or Truncating)
+                vector = np.zeros(INPUT_DIM, dtype=np.float32)
+                for i in range(min(len(raw_factors), MAX_NOTES)):
+                    for j in range(FACTOR_COUNT):
+                        vector[i * FACTOR_COUNT + j] = safe_float(raw_factors[i][j])
+                
+                dataset.append((vector, measure, songno, diff))
+
+    random.shuffle(dataset)
+    if len(dataset) < (TRAIN_SIZE + VALID_SIZE):
+        raise ValueError(f"Dataset size {len(dataset)} < required {TRAIN_SIZE + VALID_SIZE}")
+
+    train_ds = dataset[:TRAIN_SIZE]
+    valid_ds = dataset[TRAIN_SIZE:TRAIN_SIZE + VALID_SIZE]
+
+    X_train = np.array([x for x, _, _, _ in train_ds])
+    y_train = np.array([y for _, y, _, _ in train_ds])
+    X_valid = np.array([x for x, _, _, _ in valid_ds])
+    y_valid = np.array([y for _, y, _, _ in valid_ds])
+    valid_info = [(s, d) for _, _, s, d in valid_ds]
+
+    if CONTINUE_TRAINING and os.path.exists(scaler_path):
+        scaler = joblib.load(scaler_path)
+    else:
+        scaler = StandardScaler()
+        scaler.fit(X_train)
+
+    X_train = scaler.transform(X_train)
+    X_valid = scaler.transform(X_valid)
+
+    if CONTINUE_TRAINING and os.path.exists(model_path):
+        model = joblib.load(model_path)
+        model.fit(X_train, y_train, xgb_model=model.get_booster())
+    else:
+        model = XGBRegressor(n_estimators=N_ESTIMATORS, max_depth=MAX_DEPTH, learning_rate=LEARNING_RATE, objective="reg:squarederror", random_state=RANDOM_STATE)
+        model.fit(X_train, y_train)
+
+    pred = model.predict(X_valid)
+    mae = mean_absolute_error(y_valid, pred)
+    accuracy = np.sum(np.abs(pred - y_valid) <= ERROR_TOLERANCE) / len(y_valid)
+    print(f"MAE: {mae:.4f} | Accuracy: {accuracy:.4f}")
+
+    # Results save
+    validate_details = []
+    for i in range(len(y_valid)):
+        validate_details.append({"songno": valid_info[i][0], "diff": valid_info[i][1], "actual": float(y_valid[i]), "predicted": float(pred[i]), "error": float(y_valid[i] - pred[i])})
+    
+    validate_details.sort(key=lambda x: abs(x["error"]), reverse=True)
+    with open(os.path.join(working_dir, "validate.json"), "w", encoding="utf-8") as f:
+        json.dump({"summary": {"mae": float(mae), "accuracy": float(accuracy)}, "details": validate_details}, f, indent=2)
+
+    # Plot
+    plt.switch_backend('Agg')
+    df_plot = pd.DataFrame(validate_details)
+    df_plot['abs_error'] = df_plot['error'].abs()
+    df_plot = df_plot.sort_values('abs_error', ascending=False).reset_index(drop=True)
+    plt.figure(figsize=(12, 6))
+    sns.scatterplot(data=df_plot, x=df_plot.index, y='abs_error', color='crimson')
+    plt.axhline(0.2, color='green', linestyle='--')
+    plt.ylim(0, 4)
+    plt.savefig(os.path.join(working_dir, "validate.png"))
+    
+    joblib.dump(model, model_path)
+    joblib.dump(scaler, scaler_path)
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--workingDir", required=True)
+    parser.add_argument("--dataDir", required=True)
+    parser.add_argument("--trainSize", required=True, type=int)
+    parser.add_argument("--validSize", required=True, type=int)
+    args = parser.parse_args()
+    TRAIN_SIZE, VALID_SIZE = args.trainSize, args.validSize
+    train_model(args.workingDir, args.dataDir)

train/feature/train_lightgbm.py

@@ -46,9 +46,9 @@ ERROR_TOLERANCE = 0.1
 
 FEATURES_FILENAME = "features.json"
 MEASURE_FILENAME = "measure.csv"
-MODEL_FILENAME = "model_lgbm.pkl"
+MODEL_FILENAME = "model.pkl"
-SCALER_FILENAME = "scaler_lgbm.pkl"
+SCALER_FILENAME = "scaler.pkl"
-FEATURE_NAMES_FILENAME = "features_lgbm.txt"
+FEATURE_NAMES_FILENAME = "features.txt"
 
 IGNORE_KEYS = {"songno", "difficulty"}