mrpoddaa commited on
Commit
3952353
Β·
verified Β·
1 Parent(s): 18b952c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +70 -335
README.md CHANGED
@@ -1,384 +1,119 @@
 
 
 
 
 
 
 
 
 
 
1
  # πŸš€ Telegram Multi-Part File Streamer
2
 
3
- A high-performance file upload and streaming service that uses Telegram as a backend storage system. Capable of handling files up to **1TB** with zero-disk buffering, optimized for low-RAM environments like Hugging Face Spaces.
4
 
5
  ## ✨ Features
6
 
7
- - **🎯 Zero-Disk Buffering**: Direct streaming from HTTP to Telegram without local storage
8
  - **πŸ“¦ Auto-Splitting**: Automatically splits large files into 2GB parts
9
- - **⚑ Multi-Session Load Balancing**: Rotates between multiple Telegram sessions for maximum bandwidth
10
- - **πŸ“‘ Full Range Request Support**: HTTP 206 partial content for seeking in 1TB files
11
- - **πŸ”„ Parallel Download Support**: Compatible with IDM and multi-threaded downloaders
12
- - **πŸ’Ύ Low Memory Footprint**: Optimized for <512MB RAM usage even with 1TB files
13
- - **πŸ›‘οΈ Production Ready**: Comprehensive error handling, retry logic, and logging
14
-
15
- ## πŸ—οΈ Architecture
16
-
17
- ```
18
- β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
19
- β”‚ Client β”‚
20
- β”‚ (Browser/ β”‚
21
- β”‚ IDM) β”‚
22
- β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
23
- β”‚
24
- β”‚ HTTP Stream
25
- β”‚
26
- β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
27
- β”‚ FastAPI Server β”‚
28
- β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
29
- β”‚ β”‚ /upload (POST) β”‚ β”‚
30
- β”‚ β”‚ - Stream from HTTP β”‚ β”‚
31
- β”‚ β”‚ - Auto-split into parts β”‚ β”‚
32
- β”‚ β”‚ - Upload to Telegram β”‚ β”‚
33
- β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
34
- β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
35
- β”‚ β”‚ /dl/{id} (GET) β”‚ β”‚
36
- β”‚ β”‚ - Multi-part concatenation β”‚ β”‚
37
- β”‚ β”‚ - Range request support β”‚ β”‚
38
- β”‚ β”‚ - Session load balancing β”‚ β”‚
39
- β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
40
- β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
41
- β”‚
42
- β”œβ”€β”€β”€β”€β”€β”€β–Ί Telegram Session 1
43
- β”œβ”€β”€β”€β”€β”€β”€β–Ί Telegram Session 2
44
- └──────► Telegram Session N
45
- (Load Balanced)
46
-
47
- β”‚
48
- β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”
49
- β”‚ MongoDB β”‚
50
- β”‚ (Metadata) β”‚
51
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
52
- ```
53
-
54
- ## πŸ“‹ Requirements
55
-
56
- - Python 3.10+
57
- - MongoDB (local or Atlas)
58
- - Telegram API credentials
59
- - At least 512MB RAM
60
-
61
- ## πŸš€ Quick Start
62
-
63
- ### 1. Clone and Setup
64
-
65
- ```bash
66
- git clone <repository>
67
- cd telegram-streamer
68
- pip install -r requirements.txt
69
- ```
70
-
71
- ### 2. Configure Environment
72
-
73
- ```bash
74
- cp .env.template .env
75
- # Edit .env with your credentials
76
- ```
77
-
78
- Required environment variables:
79
- - `API_ID`: Your Telegram API ID
80
- - `API_HASH`: Your Telegram API Hash
81
- - `BOT_TOKEN`: Telegram bot token
82
- - `MONGO_URI`: MongoDB connection string
83
- - `SESSION_STRINGS`: (Optional) Comma-separated Pyrogram session strings
84
-
85
- ### 3. Generate Session Strings (Optional but Recommended)
86
-
87
- For multi-session load balancing:
88
-
89
- ```bash
90
- python -c "from pyrogram import Client; \
91
- client = Client('my_session', api_id=YOUR_API_ID, api_hash='YOUR_API_HASH'); \
92
- client.start(); \
93
- print(client.export_session_string())"
94
- ```
95
 
96
- Add the output to `SESSION_STRINGS` in your `.env` file.
97
 
98
- ### 4. Run the Application
99
 
100
- ```bash
101
- # Development
102
- uvicorn main:app --reload
 
 
 
 
103
 
104
- # Production
105
- uvicorn main:app --host 0.0.0.0 --port 8000 --workers 1
106
- ```
107
 
108
- ### 5. Using Docker
 
 
 
109
 
110
- ```bash
111
- # Build
112
- docker build -t telegram-streamer .
 
113
 
114
- # Run
115
- docker run -p 8000:8000 --env-file .env telegram-streamer
116
- ```
117
 
118
  ## πŸ“‘ API Endpoints
119
 
120
  ### Upload File
121
-
122
- **POST** `/upload?filename=myfile.zip`
123
-
124
- Upload a file with zero-disk buffering:
125
-
126
  ```bash
127
- curl -X POST "http://localhost:8000/upload?filename=large_file.zip" \
128
- -H "Content-Type: application/octet-stream" \
129
- --data-binary "@large_file.zip"
130
- ```
131
 
132
- Response:
133
- ```json
134
- {
135
- "success": true,
136
- "unique_id": "a1b2c3d4e5f6g7h8",
137
- "filename": "large_file.zip",
138
- "total_size": 1099511627776,
139
- "parts": 550,
140
- "download_url": "/dl/a1b2c3d4e5f6g7h8"
141
- }
142
  ```
143
 
144
- ### Stream/Download File
145
-
146
- **GET** `/dl/{unique_id}`
147
-
148
- Stream a file with full range request support:
149
-
150
  ```bash
151
- # Full download
152
- curl "http://localhost:8000/dl/a1b2c3d4e5f6g7h8" -o output.zip
153
 
154
- # Range request (download bytes 1000000-2000000)
155
- curl "http://localhost:8000/dl/a1b2c3d4e5f6g7h8" \
156
- -H "Range: bytes=1000000-2000000" -o partial.zip
157
  ```
158
 
159
  ### Get File Info
160
-
161
- **GET** `/info/{unique_id}`
162
-
163
  ```bash
164
- curl "http://localhost:8000/info/a1b2c3d4e5f6g7h8"
165
  ```
166
 
167
- ### Delete File
168
-
169
- **DELETE** `/delete/{unique_id}`
170
-
171
  ```bash
172
- curl -X DELETE "http://localhost:8000/delete/a1b2c3d4e5f6g7h8"
173
- ```
174
 
175
- ## 🎯 Usage Examples
176
-
177
- ### Upload with Python
178
-
179
- ```python
180
- import requests
181
-
182
- url = "http://localhost:8000/upload"
183
- params = {"filename": "large_dataset.tar.gz"}
184
-
185
- with open("large_dataset.tar.gz", "rb") as f:
186
- response = requests.post(url, params=params, data=f, stream=True)
187
-
188
- print(response.json())
189
  ```
190
 
191
- ### Download with Python
192
-
193
- ```python
194
- import requests
195
-
196
- unique_id = "a1b2c3d4e5f6g7h8"
197
- url = f"http://localhost:8000/dl/{unique_id}"
198
-
199
- with requests.get(url, stream=True) as r:
200
- r.raise_for_status()
201
- with open("output.zip", "wb") as f:
202
- for chunk in r.iter_content(chunk_size=8192):
203
- f.write(chunk)
204
- ```
205
-
206
- ### Range Request Example
207
-
208
- ```python
209
- import requests
210
-
211
- unique_id = "a1b2c3d4e5f6g7h8"
212
- url = f"http://localhost:8000/dl/{unique_id}"
213
-
214
- # Download only bytes 1MB to 10MB
215
- headers = {"Range": "bytes=1048576-10485760"}
216
- response = requests.get(url, headers=headers)
217
-
218
- with open("partial.bin", "wb") as f:
219
- f.write(response.content)
220
- ```
221
-
222
- ## πŸ”§ Configuration
223
-
224
- ### Environment Variables
225
-
226
- | Variable | Description | Required | Default |
227
- |----------|-------------|----------|---------|
228
- | `API_ID` | Telegram API ID | βœ… | - |
229
- | `API_HASH` | Telegram API Hash | βœ… | - |
230
- | `BOT_TOKEN` | Telegram Bot Token | βœ… | - |
231
- | `MONGO_URI` | MongoDB connection string | βœ… | - |
232
- | `MONGO_DATABASE` | Database name | ❌ | telegram_streamer |
233
- | `SESSION_STRINGS` | Comma-separated session strings | ❌ | (uses bot only) |
234
- | `LOG_LEVEL` | Logging level | ❌ | INFO |
235
-
236
- ### Performance Tuning
237
-
238
- **For Maximum Upload Speed:**
239
- - Add multiple session strings to `SESSION_STRINGS`
240
- - Each session can handle ~10-15 MB/s upload speed
241
- - 5 sessions = ~50-75 MB/s combined
242
-
243
- **For Maximum Download Speed:**
244
- - Session rotation automatically balances load
245
- - Supports unlimited parallel connections
246
- - Compatible with IDM, aria2c, axel
247
-
248
- **Memory Optimization:**
249
- - Default chunk size: 1MB (adjustable via `CHUNK_SIZE`)
250
- - RAM usage stays constant regardless of file size
251
- - Recommended: 512MB RAM minimum
252
-
253
- ## πŸ† Performance Benchmarks
254
 
255
  | Metric | Value |
256
  |--------|-------|
257
  | Max File Size | 1TB+ |
258
  | Upload Speed | 10-15 MB/s per session |
259
- | Download Speed | 50-100 MB/s (multi-threaded) |
260
- | RAM Usage | <512MB (constant) |
261
- | Concurrent Uploads | Limited by sessions |
262
- | Concurrent Downloads | Unlimited |
263
-
264
- ## πŸ› Troubleshooting
265
 
266
- ### FloodWait Errors
267
-
268
- If you encounter `FloodWait` errors:
269
- 1. Add more session strings to `SESSION_STRINGS`
270
- 2. Session rotation will automatically handle flood waits
271
- 3. Consider adding delays between uploads
272
-
273
- ### MongoDB Connection Issues
274
-
275
- ```bash
276
- # Test MongoDB connection
277
- mongosh "YOUR_MONGO_URI"
278
-
279
- # Check if MongoDB is running (local)
280
- sudo systemctl status mongod
281
- ```
282
 
283
- ### Session Initialization Failures
 
 
 
284
 
285
- 1. Verify `API_ID` and `API_HASH` are correct
286
- 2. Ensure session strings are valid
287
- 3. Check Telegram API availability
288
-
289
- ## 🌐 Deployment
290
-
291
- ### Hugging Face Spaces
292
-
293
- 1. Create a new Space (Docker type)
294
- 2. Add Secrets in Space Settings:
295
- - `API_ID`
296
- - `API_HASH`
297
- - `BOT_TOKEN`
298
- - `MONGO_URI`
299
- - `SESSION_STRINGS`
300
- 3. Push code to Space repository
301
- 4. App will auto-deploy
302
-
303
- ### Railway/Render
304
-
305
- 1. Connect GitHub repository
306
- 2. Add environment variables
307
- 3. Deploy (automatically detects Dockerfile)
308
-
309
- ### VPS/Dedicated Server
310
-
311
- ```bash
312
- # Using systemd
313
- sudo nano /etc/systemd/system/telegram-streamer.service
314
-
315
- [Unit]
316
- Description=Telegram Multi-Part File Streamer
317
- After=network.target
318
-
319
- [Service]
320
- Type=simple
321
- User=www-data
322
- WorkingDirectory=/opt/telegram-streamer
323
- EnvironmentFile=/opt/telegram-streamer/.env
324
- ExecStart=/usr/bin/uvicorn main:app --host 0.0.0.0 --port 8000
325
- Restart=always
326
-
327
- [Install]
328
- WantedBy=multi-user.target
329
-
330
- # Enable and start
331
- sudo systemctl enable telegram-streamer
332
- sudo systemctl start telegram-streamer
333
- ```
334
-
335
- ## πŸ“Š Monitoring
336
-
337
- Check application health:
338
-
339
- ```bash
340
- curl http://localhost:8000/health
341
- ```
342
-
343
- Response:
344
- ```json
345
- {
346
- "status": "healthy",
347
- "sessions": 5,
348
- "database": "connected"
349
- }
350
- ```
351
-
352
- ## πŸ”’ Security Considerations
353
-
354
- 1. **Access Control**: Add authentication middleware for production
355
- 2. **Rate Limiting**: Implement rate limits on upload/download endpoints
356
- 3. **Input Validation**: Validate filenames and parameters
357
- 4. **HTTPS**: Always use HTTPS in production
358
- 5. **Session Security**: Keep session strings private
359
-
360
- ## πŸ“ License
361
-
362
- MIT License - See LICENSE file for details
363
-
364
- ## 🀝 Contributing
365
 
366
- Contributions are welcome! Please:
367
- 1. Fork the repository
368
- 2. Create a feature branch
369
- 3. Submit a pull request
370
 
371
- ## πŸ“ž Support
 
 
372
 
373
- - Issues: GitHub Issues
374
- - Discussions: GitHub Discussions
375
 
376
- ## πŸ™ Acknowledgments
377
 
378
- - Built with [Pyrogram](https://docs.pyrogram.org/)
379
- - Powered by [FastAPI](https://fastapi.tiangolo.com/)
380
- - Storage: [MongoDB](https://www.mongodb.com/)
381
 
382
  ---
383
 
384
- **⚠️ Disclaimer**: This tool uses Telegram as a storage backend. Ensure compliance with Telegram's Terms of Service. Not recommended for storing illegal or copyrighted content.
 
 
 
1
+ ---
2
+ title: Telegram File Streamer
3
+ emoji: πŸš€
4
+ colorFrom: blue
5
+ colorTo: purple
6
+ sdk: docker
7
+ pinned: false
8
+ license: mit
9
+ ---
10
+
11
  # πŸš€ Telegram Multi-Part File Streamer
12
 
13
+ A high-performance file upload and streaming service that uses Telegram as a backend storage system. Capable of handling files up to **1TB** with zero-disk buffering.
14
 
15
  ## ✨ Features
16
 
17
+ - **🎯 Zero-Disk Buffering**: Direct streaming from HTTP to Telegram
18
  - **πŸ“¦ Auto-Splitting**: Automatically splits large files into 2GB parts
19
+ - **⚑ Multi-Session Load Balancing**: Maximum bandwidth with session rotation
20
+ - **πŸ“‘ Range Request Support**: HTTP 206 for seeking in large files
21
+ - **πŸ”„ Parallel Downloads**: IDM and multi-threaded downloader compatible
22
+ - **πŸ’Ύ Low Memory**: Optimized for <512MB RAM usage
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
 
24
+ ## πŸ”§ Required Secrets
25
 
26
+ This Space requires the following environment variables (add in Settings β†’ Repository secrets):
27
 
28
+ | Variable | Description | Example |
29
+ |----------|-------------|---------|
30
+ | `API_ID` | Telegram API ID | `12345678` |
31
+ | `API_HASH` | Telegram API Hash | `abc123def456...` |
32
+ | `BOT_TOKEN` | Bot Token from @BotFather | `1234567890:ABC...` |
33
+ | `MONGO_URI` | MongoDB connection string | `mongodb+srv://...` |
34
+ | `SESSION_STRINGS` | Session strings (optional) | `BQAB...,BQAC...` |
35
 
36
+ ### How to Get Credentials:
 
 
37
 
38
+ **Telegram API (API_ID & API_HASH):**
39
+ 1. Visit https://my.telegram.org/apps
40
+ 2. Login and create application
41
+ 3. Copy api_id and api_hash
42
 
43
+ **Bot Token:**
44
+ 1. Search @BotFather on Telegram
45
+ 2. Send `/newbot` command
46
+ 3. Copy bot token
47
 
48
+ **MongoDB:**
49
+ - Free: MongoDB Atlas (https://www.mongodb.com/cloud/atlas)
50
+ - Create cluster β†’ Get connection string
51
 
52
  ## πŸ“‘ API Endpoints
53
 
54
  ### Upload File
 
 
 
 
 
55
  ```bash
56
+ POST /upload?filename=myfile.zip
 
 
 
57
 
58
+ curl -X POST "https://YOUR-SPACE.hf.space/upload?filename=file.zip" \
59
+ -H "Content-Type: application/octet-stream" \
60
+ --data-binary "@file.zip"
 
 
 
 
 
 
 
61
  ```
62
 
63
+ ### Download File
 
 
 
 
 
64
  ```bash
65
+ GET /dl/{unique_id}
 
66
 
67
+ curl "https://YOUR-SPACE.hf.space/dl/{unique_id}" -o output.zip
 
 
68
  ```
69
 
70
  ### Get File Info
 
 
 
71
  ```bash
72
+ GET /info/{unique_id}
73
  ```
74
 
75
+ ### Health Check
 
 
 
76
  ```bash
77
+ GET /health
 
78
 
79
+ curl "https://YOUR-SPACE.hf.space/health"
 
 
 
 
 
 
 
 
 
 
 
 
 
80
  ```
81
 
82
+ ## πŸ“Š Performance
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
83
 
84
  | Metric | Value |
85
  |--------|-------|
86
  | Max File Size | 1TB+ |
87
  | Upload Speed | 10-15 MB/s per session |
88
+ | Download Speed | 50-100 MB/s |
89
+ | RAM Usage | <512MB constant |
 
 
 
 
90
 
91
+ ## ⚠️ Important
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92
 
93
+ - Ensure Telegram TOS compliance
94
+ - Files stored in your Telegram account
95
+ - Add multiple SESSION_STRINGS for better performance
96
+ - MongoDB IP whitelist: 0.0.0.0/0
97
 
98
+ ## πŸ› Troubleshooting
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
99
 
100
+ **Build Failed:**
101
+ - Check all secrets are set correctly
102
+ - Verify MongoDB URI is valid
 
103
 
104
+ **FloodWait Errors:**
105
+ - Add more SESSION_STRINGS
106
+ - Use user sessions instead of bot only
107
 
108
+ **Out of Memory:**
109
+ - Upgrade Space hardware in Settings
110
 
111
+ ## πŸ“„ License
112
 
113
+ MIT License
 
 
114
 
115
  ---
116
 
117
+ **Made with ❀️ for the community**
118
+
119
+ *Disclaimer: Use responsibly and comply with Telegram's Terms of Service.*