feat: AI 웹 스크래퍼 프로젝트의 초기 구조와 핵심 기능 구현
This commit is contained in:
18
AI_Web_Scraper/config.json
Normal file
18
AI_Web_Scraper/config.json
Normal file
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"model_name": "jxm/gpt-oss-20b-base",
|
||||
"model_local_path": "./models/gpt-oss-20b-base",
|
||||
"google_drive_folder_id": "YOUR_GOOGLE_DRIVE_FOLDER_ID",
|
||||
"google_credentials_path": "./credentials.json",
|
||||
"max_tokens": 2048,
|
||||
"temperature": 0.7,
|
||||
"web_scraping": {
|
||||
"max_pages": 100,
|
||||
"delay_between_requests": 2,
|
||||
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||
},
|
||||
"data_storage": {
|
||||
"local_storage_path": "./collected_data",
|
||||
"file_format": "json",
|
||||
"drive_mount_path": "/content/drive/MyDrive/AI_Data"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user