Compare commits
1 Commits
9a3fa5a5af
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 max
|
0c77c19368 |
@@ -0,0 +1,97 @@
|
||||
name: "Bug Report"
|
||||
description: "Report a bug or unexpected behavior to help us improve"
|
||||
title: "[BUG]: "
|
||||
labels: ["bug"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## ⚠️ Before You Begin
|
||||
Please ensure you have:
|
||||
- [ ] Searched existing issues to avoid duplicates
|
||||
- [ ] Confirmed this is a bug, not a question or configuration problem
|
||||
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: "📋 Description"
|
||||
description: "Provide a clear and concise description of the bug"
|
||||
placeholder: "What happened? What did you expect to happen instead?"
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: reproduction
|
||||
attributes:
|
||||
label: "🔁 Steps to Reproduce"
|
||||
description: "Step-by-step instructions to reproduce the issue"
|
||||
placeholder: |
|
||||
1. Go to '...'
|
||||
2. Click on '....'
|
||||
3. Scroll down to '....'
|
||||
4. See error
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: "✅ Expected Behavior"
|
||||
description: "What you expected to happen"
|
||||
placeholder: "A clear description of what should happen..."
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: "❌ Actual Behavior"
|
||||
description: "What actually happened"
|
||||
placeholder: "Include error messages, stack traces, or unexpected outcomes..."
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: "📄 Logs / Screenshots"
|
||||
description: |
|
||||
Provide relevant logs, error messages, or screenshots.
|
||||
For logs, please use a pastebin and share the URL.
|
||||
**Remember to remove any sensitive information (API keys, passwords, etc.).**
|
||||
placeholder: "Paste logs here or provide a Gist URL..."
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
---
|
||||
## 🖥️ Environment Details
|
||||
Please fill out the relevant information below.
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: "📎 Additional Context"
|
||||
description: "Any other information that might be relevant"
|
||||
placeholder: |
|
||||
- Database type and version (e.g., PostgreSQL 15, SQLite)
|
||||
- Reverse proxy/CDN in use (e.g., Nginx, Cloudflare)
|
||||
- Any custom configuration
|
||||
- Related issues or PRs
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: checkboxes
|
||||
id: checklist
|
||||
attributes:
|
||||
label: "✅ Submission Checklist"
|
||||
description: "Please confirm the following before submitting"
|
||||
options:
|
||||
- label: "I have searched for existing issues (open and closed) that report the same problem"
|
||||
required: true
|
||||
- label: "I have provided clear steps to reproduce the issue"
|
||||
required: true
|
||||
- label: "I have included relevant logs or error messages (with sensitive info removed)"
|
||||
required: false
|
||||
@@ -0,0 +1,99 @@
|
||||
name: "Feature Request"
|
||||
description: "Suggest a new feature or enhancement for this project"
|
||||
title: "[FEATURE]: "
|
||||
labels: ["enhancement"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## ⚠️ Before You Begin
|
||||
Please ensure you have:
|
||||
- [ ] Searched existing issues (open and closed) to avoid duplicates
|
||||
- [ ] Reviewed the [project documentation](https://git.vmn.su/max/vmn-ansible/wiki) for existing functionality
|
||||
- [ ] Confirmed this is a feature request, not a bug or configuration question
|
||||
|
||||
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: "🎯 Problem Statement"
|
||||
description: "What problem does this feature solve? What can't you do today?"
|
||||
placeholder: |
|
||||
I'm always frustrated when...
|
||||
Currently, it's hard/impossible to...
|
||||
This feature would help me because...
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: "💡 Proposed Solution"
|
||||
description: "Describe the solution you'd like to see"
|
||||
placeholder: |
|
||||
A clear description of what you want to happen.
|
||||
If you have a specific API, UI, or implementation in mind, describe it here.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: screenshots
|
||||
attributes:
|
||||
label: "📸 Screenshots"
|
||||
description: "If you can, provide screenshots or diagrams of the proposed feature"
|
||||
placeholder: "Drag and drop images or paste links..."
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: technical
|
||||
attributes:
|
||||
label: "⚙️ Technical Approach"
|
||||
description: "If you have ideas about how to implement this, share them here"
|
||||
placeholder: |
|
||||
- Which components/modules would be affected?
|
||||
- Any database schema changes?
|
||||
- API design considerations?
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: acceptance
|
||||
attributes:
|
||||
label: "✅ Acceptance Criteria"
|
||||
description: "What does 'done' look like for this feature?"
|
||||
placeholder: |
|
||||
- [ ] Feature works in the UI
|
||||
- [ ] API endpoints are documented
|
||||
- [ ] Tests are added
|
||||
- [ ] Documentation is updated
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: dropdown
|
||||
id: contribution
|
||||
attributes:
|
||||
label: "🤝 Are you willing to contribute this feature?"
|
||||
description: "Knowing if you can help implement this helps us prioritize"
|
||||
options:
|
||||
- "Yes, I can contribute"
|
||||
- "Yes, but I need guidance"
|
||||
- "No, but I can test/review"
|
||||
- "No, I'm just requesting"
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: checkboxes
|
||||
id: checklist
|
||||
attributes:
|
||||
label: "✅ Submission Checklist"
|
||||
description: "Please confirm the following before submitting"
|
||||
options:
|
||||
- label: "I have searched for existing issues (open and closed) that request the same feature"
|
||||
required: true
|
||||
- label: "I have explained the problem this feature solves"
|
||||
required: true
|
||||
- label: "I have described a concrete solution or direction"
|
||||
required: true
|
||||
- label: "I have considered alternatives and explained why they fall short"
|
||||
required: false
|
||||
@@ -0,0 +1,134 @@
|
||||
name: "Refactor Request"
|
||||
description: "Propose a code restructuring or cleanup with no behavior change"
|
||||
title: "[REFACTOR]: "
|
||||
labels: ["refactoring"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## ⚠️ Before You Begin
|
||||
Please ensure you have:
|
||||
- [ ] Searched existing issues (open and closed) to avoid duplicates
|
||||
- [ ] Reviewed the [project documentation](https://git.vmn.su/max/vmn-ansible/wiki) for architecture guidelines
|
||||
- [ ] Confirmed this is a refactor (code restructuring with NO behavior change), not a bug fix or new feature
|
||||
|
||||
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: "🎯 Problem Statement"
|
||||
description: "What pain points does the current code cause? Why is this refactor needed?"
|
||||
placeholder: |
|
||||
The current implementation is difficult to maintain because...
|
||||
This code has accumulated technical debt due to...
|
||||
I'm constantly frustrated when working with this module because...
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: scope
|
||||
attributes:
|
||||
label: "📂 Scope"
|
||||
description: "Which files, modules, or components need refactoring? Be specific."
|
||||
placeholder: |
|
||||
- File paths: `src/auth/`, `internal/handler/`
|
||||
- Component: `UserService` class
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: current-state
|
||||
attributes:
|
||||
label: "🔍 Current State"
|
||||
description: "Describe the current code structure, its issues, and why it's problematic"
|
||||
placeholder: |
|
||||
- The function `processUser()` is 500+ lines long
|
||||
- Duplicated logic across 3 different files
|
||||
- Mixed concerns (business logic + HTTP handling)
|
||||
- Poor test coverage makes changes risky
|
||||
- Naming is unclear and inconsistent
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: proposed-changes
|
||||
attributes:
|
||||
label: "💡 Proposed Solution"
|
||||
description: "What should the code look like after the refactor?"
|
||||
placeholder: |
|
||||
- Extract `processUser()` into smaller, focused functions
|
||||
- Consolidate duplicated logic into a shared utility
|
||||
- Separate business logic from HTTP handlers
|
||||
- Introduce interfaces for better testability
|
||||
- Rename variables and functions for clarity
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: constraints
|
||||
attributes:
|
||||
label: "🚫 Constraints"
|
||||
description: "What must NOT change? Public APIs, behavior, backward compatibility, performance characteristics?"
|
||||
placeholder: |
|
||||
- All public APIs must remain identical
|
||||
- Database schema must not change
|
||||
- External behavior must be identical
|
||||
- Response formats must stay the same
|
||||
- Performance must not degrade
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: benefits
|
||||
attributes:
|
||||
label: "📈 Expected Benefits"
|
||||
description: "What improvements will this refactor bring?"
|
||||
placeholder: |
|
||||
- Improved maintainability
|
||||
- Better testability
|
||||
- Reduced code duplication
|
||||
- Clearer separation of concerns
|
||||
- Easier onboarding for new contributors
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: risks
|
||||
attributes:
|
||||
label: "⚠️ Risks & Mitigations"
|
||||
description: "What could go wrong, and how will you mitigate it?"
|
||||
placeholder: |
|
||||
- Risk: Regression bugs
|
||||
Mitigation: Comprehensive test coverage before and after
|
||||
- Risk: Large diff making code review difficult
|
||||
Mitigation: Break into smaller, incremental PRs
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: dropdown
|
||||
id: contribution
|
||||
attributes:
|
||||
label: "🤝 Are you willing to contribute this refactor?"
|
||||
description: "Knowing if you can help implement this helps us prioritize"
|
||||
options:
|
||||
- "Yes, I can contribute"
|
||||
- "Yes, but I need guidance"
|
||||
- "No, but I can review"
|
||||
- "No, I'm just suggesting"
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: checkboxes
|
||||
id: checklist
|
||||
attributes:
|
||||
label: "✅ Submission Checklist"
|
||||
description: "Please confirm the following before submitting"
|
||||
options:
|
||||
- label: "I have searched for existing issues (open and closed) that request the same refactor"
|
||||
required: true
|
||||
- label: "I have clearly explained the problem this refactor solves"
|
||||
required: true
|
||||
- label: "I have described the scope of the refactor (specific files/modules)"
|
||||
required: false
|
||||
- label: "I have identified what must NOT change (APIs, behavior, backward compatibility)"
|
||||
required: false
|
||||
-49
@@ -1,52 +1,3 @@
|
||||
.vscode
|
||||
__pycache__
|
||||
test_data
|
||||
|
||||
# Python
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Virtual environments
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Environment variables
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# Test coverage
|
||||
.htmlcov/
|
||||
.coverage
|
||||
.coverage.*
|
||||
coverage.xml
|
||||
|
||||
# Pytest
|
||||
.pytest_cache/
|
||||
|
||||
# Generated files
|
||||
*.png
|
||||
*.csv
|
||||
data/samples/*.csv
|
||||
@@ -1,78 +0,0 @@
|
||||
# Disk Checker
|
||||
|
||||
Disk benchmarking and latency visualization tool for fio and badblocks output.
|
||||
|
||||
## Features
|
||||
|
||||
- Parse fio JSON+ latency CSV files
|
||||
- Generate histogram and CDF plots
|
||||
- REST API for remote benchmarking
|
||||
- Async job management for long-running benchmarks
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
git clone <repository>
|
||||
cd disk-checker
|
||||
python -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
## CLI Usage
|
||||
|
||||
```bash
|
||||
# Basic usage
|
||||
python -m disk_checker scan_latency.csv
|
||||
|
||||
# With options
|
||||
python -m disk_checker scan_latency.csv --metric read_clat --unit ms --log-x
|
||||
|
||||
# Generate CDF
|
||||
python -m disk_checker scan_latency.csv --cdf
|
||||
|
||||
# View all metrics
|
||||
python -m disk_checker scan_latency.csv --all-metrics
|
||||
```
|
||||
|
||||
## API Usage
|
||||
|
||||
```bash
|
||||
# Start the server
|
||||
python -m disk_checker.scripts.serve --host 0.0.0.0 --port 8000
|
||||
|
||||
# Access API docs at http://localhost:8000/docs
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
|
||||
- `POST /api/v1/scan` - Upload CSV and get analysis
|
||||
- `POST /api/v1/plot` - Generate and return plot image
|
||||
- `POST /api/v1/benchmark` - Start a benchmark job
|
||||
- `GET /api/v1/jobs/{id}` - Check job status
|
||||
- `GET /health` - Health check
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
disk-checker/
|
||||
├── src/disk_checker/
|
||||
│ ├── core/ # Business logic (extractor, converter, helpers, orchestrator)
|
||||
│ ├── plotting/ # Visualization (matplotlib)
|
||||
│ ├── web/ # FastAPI backend
|
||||
│ └── scripts/ # Entry points (cli, serve)
|
||||
├── tests/ # Test suite
|
||||
└── data/ # Sample data
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
pytest
|
||||
pytest --cov=disk_checker
|
||||
pytest --cov=disk_checker --cov-report=html
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -1,67 +0,0 @@
|
||||
nsec, read_slat_ns_count, read_slat_ns_cumulative, read_slat_ns_percentile, read_clat_ns_count, read_clat_ns_cumulative, read_clat_ns_percentile, read_lat_ns_count, read_lat_ns_cumulative, read_lat_ns_percentile, write_slat_ns_count, write_slat_ns_cumulative, write_slat_ns_percentile, write_clat_ns_count, write_clat_ns_cumulative, write_clat_ns_percentile, write_lat_ns_count, write_lat_ns_cumulative, write_lat_ns_percentile, trim_slat_ns_count, trim_slat_ns_cumulative, trim_slat_ns_percentile, trim_clat_ns_count, trim_clat_ns_cumulative, trim_clat_ns_percentile, trim_lat_ns_count, trim_lat_ns_cumulative, trim_lat_ns_percentile,
|
||||
19791872, , , , 1, 1, 0.00016775708773695687, , , , , , , , , , , , , , , , , , , , , ,
|
||||
22413312, , , , 4, 5, 0.0008387854386847844, , , , , , , , , , , , , , , , , , , , , ,
|
||||
22675456, , , , 9, 14, 0.0023485992283173965, , , , , , , , , , , , , , , , , , , , , ,
|
||||
22937600, , , , 2, 16, 0.00268411340379131, , , , , , , , , , , , , , , , , , , , , ,
|
||||
23461888, , , , 2, 18, 0.003019627579265224, , , , , , , , , , , , , , , , , , , , , ,
|
||||
23724032, , , , 7, 25, 0.004193927193423922, , , , , , , , , , , , , , , , , , , , , ,
|
||||
23986176, , , , 14, 39, 0.006542526421741319, , , , , , , , , , , , , , , , , , , , , ,
|
||||
24248320, , , , 1, 40, 0.006710283509478276, , , , , , , , , , , , , , , , , , , , , ,
|
||||
24772608, , , , 2, 42, 0.007045797684952189, , , , , , , , , , , , , , , , , , , , , ,
|
||||
25034752, , , , 2, 44, 0.007381311860426103, , , , , , , , , , , , , , , , , , , , , ,
|
||||
25296896, , , , 4, 48, 0.00805234021137393, , , , , , , , , , , , , , , , , , , , , ,
|
||||
25559040, , , , 8, 56, 0.009394396913269586, , , , , , , , , , , , , , , , , , , , , ,
|
||||
25821184, , , , 9, 65, 0.010904210702902198, , , , , , , , , , , , , , , , , , , , , ,
|
||||
26083328, , , , 9, 74, 0.012414024492534809, , , , , , , , , , , , , , , , , , , , , ,
|
||||
26345472, , , , 16, 90, 0.01509813789632612, , , , , , , , , , , , , , , , , , , , , ,
|
||||
26607616, , , , 10, 100, 0.016775708773695688, , , , , , , , , , , , , , , , , , , , , ,
|
||||
26869760, , , , 34, 134, 0.022479449756752222, , , , , , , , , , , , , , , , , , , , , ,
|
||||
27131904, , , , 38, 172, 0.028854219090756583, , , , , , , , , , , , , , , , , , , , , ,
|
||||
27394048, , , , 49, 221, 0.037074316389867475, , , , , , , , , , , , , , , , , , , , , ,
|
||||
27656192, , , , 57, 278, 0.04663647039087401, , , , , , , , , , , , , , , , , , , , , ,
|
||||
27918336, , , , 90, 368, 0.061734608287200135, , , , , , , , , , , , , , , , , , , , , ,
|
||||
28180480, , , , 91, 459, 0.0770005032712632, , , , , , , , , , , , , , , , , , , , , ,
|
||||
28442624, , , , 105, 564, 0.09461499748364369, , , , , , , , , , , , , , , , , , , , , ,
|
||||
28704768, , , , 149, 713, 0.11961080355645026, , , , , , , , , , , , , , , , , , , , , ,
|
||||
28966912, , , , 152, 865, 0.1451098808924677, , , , , , , , , , , , , , , , , , , , , ,
|
||||
29229056, , , , 170, 1035, 0.17362858580775037, , , , , , , , , , , , , , , , , , , , , ,
|
||||
29491200, , , , 183, 1218, 0.20432813286361348, , , , , , , , , , , , , , , , , , , , , ,
|
||||
29753344, , , , 171, 1389, 0.23301459486663312, , , , , , , , , , , , , , , , , , , , , ,
|
||||
30015488, , , , 174, 1563, 0.2622043281328636, , , , , , , , , , , , , , , , , , , , , ,
|
||||
30277632, , , , 187, 1750, 0.2935749035396745, , , , , , , , , , , , , , , , , , , , , ,
|
||||
30539776, , , , 158, 1908, 0.32008052340211374, , , , , , , , , , , , , , , , , , , , , ,
|
||||
30801920, , , , 154, 2062, 0.3459151149136051, , , , , , , , , , , , , , , , , , , , , ,
|
||||
31064064, , , , 167, 2229, 0.3739305485656769, , , , , , , , , , , , , , , , , , , , , ,
|
||||
31326208, , , , 174, 2403, 0.4031202818319074, , , , , , , , , , , , , , , , , , , , , ,
|
||||
31588352, , , , 175, 2578, 0.43247777218587485, , , , , , , , , , , , , , , , , , , , , ,
|
||||
31850496, , , , 174, 2752, 0.46166750545210533, , , , , , , , , , , , , , , , , , , , , ,
|
||||
32112640, , , , 162, 2914, 0.4888441536654924, , , , , , , , , , , , , , , , , , , , , ,
|
||||
32374784, , , , 162, 3076, 0.5160208018788793, , , , , , , , , , , , , , , , , , , , , ,
|
||||
32636928, , , , 209, 3285, 0.5510820332159033, , , , , , , , , , , , , , , , , , , , , ,
|
||||
32899072, , , , 168, 3453, 0.5792652239557121, , , , , , , , , , , , , , , , , , , , , ,
|
||||
33161216, , , , 188, 3641, 0.6108035564502601, , , , , , , , , , , , , , , , , , , , , ,
|
||||
33423360, , , , 162, 3803, 0.637980204663647, , , , , , , , , , , , , , , , , , , , , ,
|
||||
33816576, , , , 324, 4127, 0.692333501090421, , , , , , , , , , , , , , , , , , , , , ,
|
||||
34340864, , , , 352, 4479, 0.7513839959738299, , , , , , , , , , , , , , , , , , , , , ,
|
||||
34865152, , , , 315, 4794, 0.8042274786109713, , , , , , , , , , , , , , , , , , , , , ,
|
||||
35389440, , , , 291, 5085, 0.8530447911424258, , , , , , , , , , , , , , , , , , , , , ,
|
||||
35913728, , , , 259, 5344, 0.8964938768662976, , , , , , , , , , , , , , , , , , , , , ,
|
||||
36438016, , , , 157, 5501, 0.9228317396409998, , , , , , , , , , , , , , , , , , , , , ,
|
||||
36962304, , , , 98, 5599, 0.9392719342392216, , , , , , , , , , , , , , , , , , , , , ,
|
||||
37486592, , , , 75, 5674, 0.9518537158194934, , , , , , , , , , , , , , , , , , , , , ,
|
||||
38010880, , , , 26, 5700, 0.9562154001006542, , , , , , , , , , , , , , , , , , , , , ,
|
||||
38535168, , , , 22, 5722, 0.9599060560308673, , , , , , , , , , , , , , , , , , , , , ,
|
||||
39059456, , , , 25, 5747, 0.9640999832242912, , , , , , , , , , , , , , , , , , , , , ,
|
||||
39583744, , , , 17, 5764, 0.9669518537158195, , , , , , , , , , , , , , , , , , , , , ,
|
||||
40108032, , , , 26, 5790, 0.9713135379969804, , , , , , , , , , , , , , , , , , , , , ,
|
||||
40632320, , , , 23, 5813, 0.9751719510149304, , , , , , , , , , , , , , , , , , , , , ,
|
||||
41156608, , , , 18, 5831, 0.9781915785941956, , , , , , , , , , , , , , , , , , , , , ,
|
||||
41680896, , , , 21, 5852, 0.9817144774366717, , , , , , , , , , , , , , , , , , , , , ,
|
||||
42205184, , , , 22, 5874, 0.9854051333668847, , , , , , , , , , , , , , , , , , , , , ,
|
||||
42729472, , , , 15, 5889, 0.9879214896829391, , , , , , , , , , , , , , , , , , , , , ,
|
||||
43253760, , , , 18, 5907, 0.9909411172622044, , , , , , , , , , , , , , , , , , , , , ,
|
||||
43778048, , , , 25, 5932, 0.9951350444556283, , , , , , , , , , , , , , , , , , , , , ,
|
||||
44302336, , , , 15, 5947, 0.9976514007716826, , , , , , , , , , , , , , , , , , , , , ,
|
||||
44826624, , , , 8, 5955, 0.9989934574735783, , , , , , , , , , , , , , , , , , , , , ,
|
||||
45350912, , , , 3, 5958, 0.9994967287367891, , , , , , , , , , , , , , , , , , , , , ,
|
||||
45875200, , , , 3, 5961, 1.0, , , , , , , , , , , , , , , , , , , , , ,
|
||||
|
Binary file not shown.
|
Before Width: | Height: | Size: 56 KiB |
@@ -1,68 +0,0 @@
|
||||
nsec, read_slat_ns_count, read_slat_ns_cumulative, read_slat_ns_percentile, read_clat_ns_count, read_clat_ns_cumulative, read_clat_ns_percentile, read_lat_ns_count, read_lat_ns_cumulative, read_lat_ns_percentile, write_slat_ns_count, write_slat_ns_cumulative, write_slat_ns_percentile, write_clat_ns_count, write_clat_ns_cumulative, write_clat_ns_percentile, write_lat_ns_count, write_lat_ns_cumulative, write_lat_ns_percentile, trim_slat_ns_count, trim_slat_ns_cumulative, trim_slat_ns_percentile, trim_clat_ns_count, trim_clat_ns_cumulative, trim_clat_ns_percentile, trim_lat_ns_count, trim_lat_ns_cumulative, trim_lat_ns_percentile,
|
||||
22151168, , , , 3, 3, 0.00025161452654533256, , , , , , , , , , , , , , , , , , , , , ,
|
||||
22413312, , , , 4, 7, 0.0005871005619391093, , , , , , , , , , , , , , , , , , , , , ,
|
||||
22675456, , , , 8, 15, 0.0012580726327266628, , , , , , , , , , , , , , , , , , , , , ,
|
||||
22937600, , , , 5, 20, 0.0016774301769688837, , , , , , , , , , , , , , , , , , , , , ,
|
||||
23461888, , , , 2, 22, 0.001845173194665772, , , , , , , , , , , , , , , , , , , , , ,
|
||||
23724032, , , , 5, 27, 0.002264530738907993, , , , , , , , , , , , , , , , , , , , , ,
|
||||
23986176, , , , 7, 34, 0.0028516313008471023, , , , , , , , , , , , , , , , , , , , , ,
|
||||
25034752, , , , 3, 37, 0.0031032458273924346, , , , , , , , , , , , , , , , , , , , , ,
|
||||
25296896, , , , 1, 38, 0.003187117336240879, , , , , , , , , , , , , , , , , , , , , ,
|
||||
25559040, , , , 4, 42, 0.003522603371634656, , , , , , , , , , , , , , , , , , , , , ,
|
||||
25821184, , , , 2, 44, 0.003690346389331544, , , , , , , , , , , , , , , , , , , , , ,
|
||||
26083328, , , , 10, 54, 0.004529061477815986, , , , , , , , , , , , , , , , , , , , , ,
|
||||
26345472, , , , 15, 69, 0.005787134110542649, , , , , , , , , , , , , , , , , , , , , ,
|
||||
26607616, , , , 22, 91, 0.007632307305208421, , , , , , , , , , , , , , , , , , , , , ,
|
||||
26869760, , , , 19, 110, 0.00922586597332886, , , , , , , , , , , , , , , , , , , , , ,
|
||||
27131904, , , , 35, 145, 0.012161368783024406, , , , , , , , , , , , , , , , , , , , , ,
|
||||
27394048, , , , 60, 205, 0.017193659313931058, , , , , , , , , , , , , , , , , , , , , ,
|
||||
27656192, , , , 53, 258, 0.0216388492828986, , , , , , , , , , , , , , , , , , , , , ,
|
||||
27918336, , , , 78, 336, 0.028180826973077246, , , , , , , , , , , , , , , , , , , , , ,
|
||||
28180480, , , , 90, 426, 0.03572926276943722, , , , , , , , , , , , , , , , , , , , , ,
|
||||
28442624, , , , 128, 554, 0.046464815902038076, , , , , , , , , , , , , , , , , , , , , ,
|
||||
28704768, , , , 174, 728, 0.061058458441667365, , , , , , , , , , , , , , , , , , , , , ,
|
||||
28966912, , , , 209, 937, 0.0785876037909922, , , , , , , , , , , , , , , , , , , , , ,
|
||||
29229056, , , , 222, 1159, 0.09720707875534682, , , , , , , , , , , , , , , , , , , , , ,
|
||||
29491200, , , , 250, 1409, 0.11817495596745785, , , , , , , , , , , , , , , , , , , , , ,
|
||||
29753344, , , , 310, 1719, 0.14417512371047556, , , , , , , , , , , , , , , , , , , , , ,
|
||||
30015488, , , , 289, 2008, 0.1684139897676759, , , , , , , , , , , , , , , , , , , , , ,
|
||||
30277632, , , , 325, 2333, 0.19567223014342028, , , , , , , , , , , , , , , , , , , , , ,
|
||||
30539776, , , , 341, 2674, 0.22427241466073974, , , , , , , , , , , , , , , , , , , , , ,
|
||||
30801920, , , , 338, 3012, 0.2526209846515139, , , , , , , , , , , , , , , , , , , , , ,
|
||||
31064064, , , , 352, 3364, 0.28214375576616624, , , , , , , , , , , , , , , , , , , , , ,
|
||||
31326208, , , , 368, 3732, 0.3130084710223937, , , , , , , , , , , , , , , , , , , , , ,
|
||||
31588352, , , , 310, 4042, 0.3390086387654114, , , , , , , , , , , , , , , , , , , , , ,
|
||||
31850496, , , , 314, 4356, 0.3653442925438229, , , , , , , , , , , , , , , , , , , , , ,
|
||||
32112640, , , , 370, 4726, 0.3963767508177472, , , , , , , , , , , , , , , , , , , , , ,
|
||||
32374784, , , , 377, 5103, 0.4279963096536107, , , , , , , , , , , , , , , , , , , , , ,
|
||||
32636928, , , , 346, 5449, 0.45701585171517234, , , , , , , , , , , , , , , , , , , , , ,
|
||||
32899072, , , , 349, 5798, 0.48628700830327937, , , , , , , , , , , , , , , , , , , , , ,
|
||||
33161216, , , , 334, 6132, 0.5143000922586597, , , , , , , , , , , , , , , , , , , , , ,
|
||||
33423360, , , , 313, 6445, 0.5405518745282227, , , , , , , , , , , , , , , , , , , , , ,
|
||||
33816576, , , , 740, 7185, 0.6026167910760715, , , , , , , , , , , , , , , , , , , , , ,
|
||||
34340864, , , , 724, 7909, 0.663339763482345, , , , , , , , , , , , , , , , , , , , , ,
|
||||
34865152, , , , 653, 8562, 0.7181078587603791, , , , , , , , , , , , , , , , , , , , , ,
|
||||
35389440, , , , 666, 9228, 0.7739662836534429, , , , , , , , , , , , , , , , , , , , , ,
|
||||
35913728, , , , 584, 9812, 0.8229472448209343, , , , , , , , , , , , , , , , , , , , , ,
|
||||
36438016, , , , 499, 10311, 0.864799127736308, , , , , , , , , , , , , , , , , , , , , ,
|
||||
36962304, , , , 399, 10710, 0.8982638597668372, , , , , , , , , , , , , , , , , , , , , ,
|
||||
37486592, , , , 298, 11008, 0.9232575694036735, , , , , , , , , , , , , , , , , , , , , ,
|
||||
38010880, , , , 192, 11200, 0.9393608991025748, , , , , , , , , , , , , , , , , , , , , ,
|
||||
38535168, , , , 108, 11308, 0.9484190220582068, , , , , , , , , , , , , , , , , , , , , ,
|
||||
39059456, , , , 71, 11379, 0.9543738991864463, , , , , , , , , , , , , , , , , , , , , ,
|
||||
39583744, , , , 57, 11436, 0.9591545751908077, , , , , , , , , , , , , , , , , , , , , ,
|
||||
40108032, , , , 47, 11483, 0.9630965361066846, , , , , , , , , , , , , , , , , , , , , ,
|
||||
40632320, , , , 46, 11529, 0.966954625513713, , , , , , , , , , , , , , , , , , , , , ,
|
||||
41156608, , , , 42, 11571, 0.9704772288853476, , , , , , , , , , , , , , , , , , , , , ,
|
||||
41680896, , , , 38, 11609, 0.9736643462215885, , , , , , , , , , , , , , , , , , , , , ,
|
||||
42205184, , , , 38, 11647, 0.9768514635578294, , , , , , , , , , , , , , , , , , , , , ,
|
||||
42729472, , , , 47, 11694, 0.9807934244737063, , , , , , , , , , , , , , , , , , , , , ,
|
||||
43253760, , , , 37, 11731, 0.9838966703010987, , , , , , , , , , , , , , , , , , , , , ,
|
||||
43778048, , , , 51, 11782, 0.9881741172523694, , , , , , , , , , , , , , , , , , , , , ,
|
||||
44302336, , , , 40, 11822, 0.9915289776063071, , , , , , , , , , , , , , , , , , , , , ,
|
||||
44826624, , , , 41, 11863, 0.9949677094690933, , , , , , , , , , , , , , , , , , , , , ,
|
||||
45350912, , , , 29, 11892, 0.9973999832256982, , , , , , , , , , , , , , , , , , , , , ,
|
||||
45875200, , , , 16, 11908, 0.9987419273672733, , , , , , , , , , , , , , , , , , , , , ,
|
||||
46399488, , , , 10, 11918, 0.9995806424557577, , , , , , , , , , , , , , , , , , , , , ,
|
||||
46923776, , , , 4, 11922, 0.9999161284911515, , , , , , , , , , , , , , , , , , , , , ,
|
||||
73924608, , , , 1, 11923, 1.0, , , , , , , , , , , , , , , , , , , , , ,
|
||||
|
@@ -1,53 +0,0 @@
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "disk-checker"
|
||||
version = "0.1.0"
|
||||
description = "Disk benchmarking and latency visualization tool"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"pandas>=2.0",
|
||||
"numpy>=1.24",
|
||||
"matplotlib>=3.7",
|
||||
"fastapi>=0.100",
|
||||
"uvicorn[standard]>=0.23",
|
||||
"python-multipart>=0.0.6",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.0",
|
||||
"pytest-asyncio>=0.21",
|
||||
"pytest-cov>=4.0",
|
||||
"black>=23.0",
|
||||
"ruff>=0.1.0",
|
||||
"mypy>=1.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
disk-checker = "disk_checker.scripts.cli:main"
|
||||
disk-checker-serve = "disk_checker.scripts.serve:main"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/disk_checker"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
python_functions = ["test_*"]
|
||||
|
||||
[tool.black]
|
||||
line-length = 100
|
||||
target-version = ["py310"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py310"
|
||||
|
||||
[tool.mypy]
|
||||
python_version = "3.10"
|
||||
warn_return_any = true
|
||||
warn_unused_configs = true
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
CLI entry point for disk-checker.
|
||||
|
||||
Usage:
|
||||
python -m disk_checker scan_latency.csv
|
||||
disk-checker scan_latency.csv --metric read_clat --unit ms
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from disk_checker.core.extractor import load_latency_csv
|
||||
from disk_checker.core.helpers import percentile_table
|
||||
from disk_checker.plotting.plots import generate_plot
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Plot fio json+ wide-format latency CSV."
|
||||
)
|
||||
parser.add_argument("csv", help="CSV from fio_jsonplus_clat2csv")
|
||||
parser.add_argument("--metric", default=None,
|
||||
help="Metric to plot (e.g. read_clat). "
|
||||
"If omitted, the metric with the most I/Os is used.")
|
||||
parser.add_argument("--all-metrics", action="store_true",
|
||||
help="Plot every metric that has data")
|
||||
parser.add_argument("--unit", choices=["ns", "us", "ms", "s"], default="us",
|
||||
help="X-axis latency unit (default: us)")
|
||||
parser.add_argument("--log-y", action="store_true")
|
||||
parser.add_argument("--log-x", action="store_true")
|
||||
parser.add_argument("--cdf", action="store_true",
|
||||
help="Add a CDF subplot below the histogram")
|
||||
parser.add_argument("--cdf-only", action="store_true")
|
||||
parser.add_argument("--normalize", action="store_true",
|
||||
help="Normalize histogram to a probability density")
|
||||
parser.add_argument("--percentiles", default="50,90,99,99.9,99.99")
|
||||
parser.add_argument("--no-percentiles", action="store_true")
|
||||
parser.add_argument("--output", "-o", default=None)
|
||||
parser.add_argument("--dpi", type=int, default=150)
|
||||
parser.add_argument("--title", default="")
|
||||
parser.add_argument("--show", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not Path(args.csv).is_file():
|
||||
sys.exit(f"error: file not found: {args.csv}")
|
||||
|
||||
df = load_latency_csv(args.csv)
|
||||
if df.empty:
|
||||
sys.exit("error: no latency data in CSV")
|
||||
|
||||
available = sorted(df["metric"].unique())
|
||||
print("Metrics with data:", ", ".join(available))
|
||||
|
||||
if args.all_metrics:
|
||||
metrics = available
|
||||
elif args.metric:
|
||||
if args.metric not in available:
|
||||
sys.exit(f"error: metric {args.metric!r} not in {available}")
|
||||
metrics = [args.metric]
|
||||
else:
|
||||
totals = df.groupby("metric")["count"].sum().sort_values(ascending=False)
|
||||
metrics = [totals.index[0]]
|
||||
print(f"Auto-selected metric: {metrics[0]}")
|
||||
|
||||
output = args.output or f"{Path(args.csv).stem}_latency.png"
|
||||
generate_plot(
|
||||
df=df,
|
||||
metrics=metrics,
|
||||
output_path=Path(output),
|
||||
unit=args.unit,
|
||||
log_y=args.log_y,
|
||||
log_x=args.log_x,
|
||||
normalize=args.normalize,
|
||||
show_cdf=args.cdf,
|
||||
cdf_only=args.cdf_only,
|
||||
title=args.title,
|
||||
dpi=args.dpi,
|
||||
)
|
||||
print(f"Saved plot to {output}")
|
||||
|
||||
if not args.no_percentiles:
|
||||
try:
|
||||
pcts = [float(x) for x in args.percentiles.split(",") if x.strip()]
|
||||
except ValueError:
|
||||
sys.exit("error: --percentiles must be comma-separated numbers")
|
||||
for m in metrics:
|
||||
print(percentile_table(df, m, args.unit, pcts))
|
||||
|
||||
if args.show:
|
||||
import matplotlib.pyplot as plt
|
||||
plt.show()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,32 +0,0 @@
|
||||
"""Startup script for the web server."""
|
||||
|
||||
import argparse
|
||||
import uvicorn
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point for the web server."""
|
||||
parser = argparse.ArgumentParser(description="Start Disk Checker API server")
|
||||
parser.add_argument("--host", default="127.0.0.1", help="Host to bind to")
|
||||
parser.add_argument("--port", type=int, default=8000, help="Port to bind to")
|
||||
parser.add_argument("--workers", type=int, default=1, help="Number of workers")
|
||||
parser.add_argument("--reload", action="store_true", help="Enable auto-reload")
|
||||
parser.add_argument("--work-dir", type=Path, default=Path("."), help="Work directory")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Set work directory in environment
|
||||
import os
|
||||
os.environ["DISK_CHECKER_WORK_DIR"] = str(args.work_dir)
|
||||
|
||||
uvicorn.run(
|
||||
"disk_checker.web.server:app",
|
||||
host=args.host,
|
||||
port=args.port,
|
||||
workers=args.workers,
|
||||
reload=args.reload,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,3 +0,0 @@
|
||||
"""Disk Checker - Disk benchmarking and latency visualization tool."""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
@@ -1,6 +0,0 @@
|
||||
"""Allow running disk-checker as a module: python -m disk_checker"""
|
||||
|
||||
from disk_checker.scripts.cli import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,14 +0,0 @@
|
||||
"""Core business logic module."""
|
||||
|
||||
from .extractor import load_latency_csv
|
||||
from .converter import to_unit, UNIT_FACTORS, UNIT_LABELS
|
||||
from .helpers import human_count, percentile_table
|
||||
|
||||
__all__ = [
|
||||
"load_latency_csv",
|
||||
"to_unit",
|
||||
"UNIT_FACTORS",
|
||||
"UNIT_LABELS",
|
||||
"human_count",
|
||||
"percentile_table",
|
||||
]
|
||||
@@ -1,32 +0,0 @@
|
||||
"""Unit conversion utilities for latency data."""
|
||||
|
||||
from typing import Dict
|
||||
import numpy as np
|
||||
|
||||
UNIT_FACTORS: Dict[str, float] = {
|
||||
"ns": 1.0,
|
||||
"us": 1e3,
|
||||
"ms": 1e6,
|
||||
"s": 1e9,
|
||||
}
|
||||
|
||||
UNIT_LABELS: Dict[str, str] = {
|
||||
"ns": "ns",
|
||||
"us": "µs",
|
||||
"ms": "ms",
|
||||
"s": "s",
|
||||
}
|
||||
|
||||
|
||||
def to_unit(values_ns, unit: str) -> np.ndarray:
|
||||
"""
|
||||
Convert nanosecond values to the specified unit.
|
||||
|
||||
Args:
|
||||
values_ns: Values in nanoseconds
|
||||
unit: Target unit (ns, us, ms, s)
|
||||
|
||||
Returns:
|
||||
Converted values as numpy array
|
||||
"""
|
||||
return np.asarray(values_ns, dtype=float) / UNIT_FACTORS[unit]
|
||||
@@ -1,88 +0,0 @@
|
||||
"""CSV data extraction module for fio latency data."""
|
||||
|
||||
from pathlib import Path
|
||||
import pandas as pd
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def load_latency_csv(path: str) -> pd.DataFrame:
|
||||
"""
|
||||
Parse wide-format fio JSON+ CSV into long-format DataFrame.
|
||||
|
||||
Transforms wide-format columns (e.g., read_clat_ns_count) into rows with
|
||||
a metric identifier, making it easier to group and analyze by operation type.
|
||||
|
||||
Args:
|
||||
path: Path to CSV file generated by fio_jsonplus_clat2csv
|
||||
|
||||
Returns:
|
||||
DataFrame with columns: metric, latency_ns, count, cumulative, percentile
|
||||
|
||||
Raises:
|
||||
ValueError: If CSV format is invalid or contains no valid data
|
||||
"""
|
||||
df = pd.read_csv(path, skipinitialspace=True)
|
||||
df.columns = [str(c).strip() for c in df.columns]
|
||||
|
||||
# Drop trailing unnamed / empty columns
|
||||
df = df.loc[:, ~df.columns.str.match(r"^Unnamed")]
|
||||
df = df.loc[:, df.columns != ""]
|
||||
|
||||
nsec_col = df.columns[0]
|
||||
if "nsec" not in nsec_col.lower():
|
||||
raise ValueError(
|
||||
f"expected first column to be 'nsec', got {nsec_col!r}"
|
||||
)
|
||||
|
||||
# Coerce the nsec column to numeric
|
||||
df[nsec_col] = pd.to_numeric(df[nsec_col], errors="coerce")
|
||||
df = df.dropna(subset=[nsec_col])
|
||||
|
||||
# Discover metric triples: <metric>_count / _cumulative / _percentile
|
||||
metrics = {}
|
||||
for col in df.columns:
|
||||
if col == nsec_col:
|
||||
continue
|
||||
for suffix in ("_count", "_cumulative", "_percentile"):
|
||||
if col.endswith(suffix):
|
||||
base = col[: -len(suffix)]
|
||||
metrics.setdefault(base, {})[suffix[1:]] = col
|
||||
|
||||
if not metrics:
|
||||
raise ValueError("no *_count columns found in CSV")
|
||||
|
||||
frames = []
|
||||
for base, cols in metrics.items():
|
||||
count_col = cols["count"]
|
||||
|
||||
# Coerce count to numeric
|
||||
counts = pd.to_numeric(df[count_col], errors="coerce")
|
||||
|
||||
# Keep only rows where the count column actually has a number
|
||||
mask = counts.notna()
|
||||
if not mask.any():
|
||||
continue
|
||||
|
||||
sub = pd.DataFrame({
|
||||
"latency_ns": df.loc[mask, nsec_col].values,
|
||||
"count": counts[mask].astype(int).values,
|
||||
})
|
||||
if "cumulative" in cols:
|
||||
sub["cumulative"] = pd.to_numeric(
|
||||
df.loc[mask, cols["cumulative"]], errors="coerce"
|
||||
).values
|
||||
if "percentile" in cols:
|
||||
sub["percentile"] = pd.to_numeric(
|
||||
df.loc[mask, cols["percentile"]], errors="coerce"
|
||||
).values
|
||||
|
||||
# read_clat_ns -> read_clat
|
||||
sub["metric"] = base.replace("_ns", "")
|
||||
frames.append(sub)
|
||||
|
||||
if not frames:
|
||||
raise ValueError("no metric had any numeric count data")
|
||||
|
||||
out = pd.concat(frames, ignore_index=True)
|
||||
out = out[out["count"] > 0].reset_index(drop=True)
|
||||
return out
|
||||
@@ -1,62 +0,0 @@
|
||||
"""Helper functions for formatting and display."""
|
||||
|
||||
from typing import List
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from matplotlib.ticker import FuncFormatter
|
||||
|
||||
from .converter import to_unit, UNIT_LABELS
|
||||
|
||||
|
||||
def human_count(x, _pos=None) -> str:
|
||||
"""
|
||||
Format large numbers with K/M/G suffixes for axis labels.
|
||||
|
||||
Args:
|
||||
x: Numeric value to format
|
||||
_pos: Matplotlib tick position (unused)
|
||||
|
||||
Returns:
|
||||
Formatted string (e.g., "1.5M", "3.2G")
|
||||
"""
|
||||
if x >= 1e9:
|
||||
return f"{x/1e9:.1f}G"
|
||||
if x >= 1e6:
|
||||
return f"{x/1e6:.1f}M"
|
||||
if x >= 1e3:
|
||||
return f"{x/1e3:.0f}k"
|
||||
return f"{x:.0f}"
|
||||
|
||||
|
||||
def percentile_table(df: pd.DataFrame, metric: str, unit: str, pcts: List[float]) -> str:
|
||||
"""
|
||||
Generate a formatted table showing latency values at specified percentiles.
|
||||
|
||||
Args:
|
||||
df: Long-format DataFrame from load_latency_csv
|
||||
metric: Metric name (e.g., 'read_clat')
|
||||
unit: Display unit (ns, us, ms, s)
|
||||
pcts: List of percentile values (e.g., [50, 90, 99])
|
||||
|
||||
Returns:
|
||||
Formatted string table
|
||||
"""
|
||||
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||
if g.empty:
|
||||
return f"(no data for metric {metric})"
|
||||
|
||||
counts = g["count"].values.astype(float)
|
||||
total = int(counts.sum())
|
||||
cdf = np.cumsum(counts) / total * 100.0
|
||||
lat = to_unit(g["latency_ns"].values, unit)
|
||||
|
||||
lines = [
|
||||
f"\nMetric: {metric} (total I/Os: {total:,})",
|
||||
f" {'percentile':>12} {'latency':>14}",
|
||||
]
|
||||
|
||||
for p in pcts:
|
||||
idx = min(np.searchsorted(cdf, p), len(lat) - 1)
|
||||
lines.append(f" {p:>11.2f}% {lat[idx]:>12.3f} {UNIT_LABELS[unit]}")
|
||||
|
||||
return "\n".join(lines)
|
||||
@@ -1,164 +0,0 @@
|
||||
"""Tool orchestration module for running disk benchmarks."""
|
||||
|
||||
import asyncio
|
||||
import subprocess
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
|
||||
class ToolStatus(Enum):
|
||||
"""Status of a benchmark job."""
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
@dataclass
|
||||
class JobResult:
|
||||
"""Result of a benchmark job."""
|
||||
job_id: str
|
||||
status: ToolStatus
|
||||
output_path: Optional[Path] = None
|
||||
error: Optional[str] = None
|
||||
metadata: Dict = field(default_factory=dict)
|
||||
|
||||
|
||||
class Orchestrator:
|
||||
"""
|
||||
Orchestrates disk benchmarking tools (fio, badblocks, etc.)
|
||||
|
||||
Provides async interfaces for running benchmarks and collecting results.
|
||||
"""
|
||||
|
||||
def __init__(self, work_dir: Path = Path(".")):
|
||||
self.work_dir = work_dir
|
||||
self._jobs: Dict[str, JobResult] = {}
|
||||
|
||||
async def run_fio(
|
||||
self,
|
||||
device: str,
|
||||
job_file: Optional[Path] = None,
|
||||
output_format: str = "jsonplus",
|
||||
**kwargs,
|
||||
) -> Path:
|
||||
"""
|
||||
Run fio benchmark and return path to output file.
|
||||
|
||||
Args:
|
||||
device: Block device path (e.g., '/dev/sda')
|
||||
job_file: Optional fio job file
|
||||
output_format: Output format (json, jsonplus, csv)
|
||||
**kwargs: Additional fio options
|
||||
|
||||
Returns:
|
||||
Path to output file
|
||||
|
||||
Raises:
|
||||
subprocess.CalledProcessError: If fio command fails
|
||||
"""
|
||||
job_id = f"fio_{asyncio.get_event_loop().time()}"
|
||||
output_path = self.work_dir / f"{job_id}.{output_format}"
|
||||
|
||||
cmd = ["fio", "--output-format=jsonplus"]
|
||||
if job_file:
|
||||
cmd.extend(["--filename", str(device)])
|
||||
else:
|
||||
cmd.extend(["--name=test", "--rw=randread", "--bs=4k"])
|
||||
|
||||
cmd.extend(["--output", str(output_path)])
|
||||
cmd.extend([f"--output-format={output_format}"])
|
||||
|
||||
for k, v in kwargs.items():
|
||||
cmd.extend([f"--{k}={v}"])
|
||||
|
||||
self._jobs[job_id] = JobResult(
|
||||
job_id=job_id,
|
||||
status=ToolStatus.RUNNING,
|
||||
output_path=output_path,
|
||||
)
|
||||
|
||||
try:
|
||||
await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
self._jobs[job_id] = JobResult(
|
||||
job_id=job_id,
|
||||
status=ToolStatus.COMPLETED,
|
||||
output_path=output_path,
|
||||
)
|
||||
return output_path
|
||||
except subprocess.CalledProcessError as e:
|
||||
self._jobs[job_id] = JobResult(
|
||||
job_id=job_id,
|
||||
status=ToolStatus.FAILED,
|
||||
error=str(e),
|
||||
)
|
||||
raise
|
||||
|
||||
async def run_badblocks(
|
||||
self,
|
||||
device: str,
|
||||
blocks_count: Optional[int] = None,
|
||||
**kwargs,
|
||||
) -> Path:
|
||||
"""
|
||||
Run badblocks scan on device.
|
||||
|
||||
Args:
|
||||
device: Block device path
|
||||
blocks_count: Number of blocks to scan
|
||||
**kwargs: Additional badblocks options
|
||||
|
||||
Returns:
|
||||
Path to badblocks output file
|
||||
"""
|
||||
job_id = f"badblocks_{asyncio.get_event_loop().time()}"
|
||||
output_path = self.work_dir / f"{job_id}.txt"
|
||||
|
||||
cmd = ["badblocks", "-w", "-s", "-p", "0", "-t", "RANDOM", "-o"]
|
||||
cmd.extend([str(output_path), str(device)])
|
||||
|
||||
if blocks_count:
|
||||
cmd.extend(["-b", str(blocks_count)])
|
||||
|
||||
for k, v in kwargs.items():
|
||||
cmd.extend([f"-{k}", str(v)])
|
||||
|
||||
self._jobs[job_id] = JobResult(
|
||||
job_id=job_id,
|
||||
status=ToolStatus.RUNNING,
|
||||
output_path=output_path,
|
||||
)
|
||||
|
||||
try:
|
||||
await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
self._jobs[job_id] = JobResult(
|
||||
job_id=job_id,
|
||||
status=ToolStatus.COMPLETED,
|
||||
output_path=output_path,
|
||||
)
|
||||
return output_path
|
||||
except subprocess.CalledProcessError as e:
|
||||
self._jobs[job_id] = JobResult(
|
||||
job_id=job_id,
|
||||
status=ToolStatus.FAILED,
|
||||
error=str(e),
|
||||
)
|
||||
raise
|
||||
|
||||
def get_job_status(self, job_id: str) -> Optional[JobResult]:
|
||||
"""Get status of a submitted job."""
|
||||
return self._jobs.get(job_id)
|
||||
|
||||
def list_jobs(self) -> Dict[str, JobResult]:
|
||||
"""List all jobs."""
|
||||
return self._jobs.copy()
|
||||
@@ -1,5 +0,0 @@
|
||||
"""Plotting module for disk latency visualization."""
|
||||
|
||||
from .plots import draw_histogram, draw_cdf, generate_plot
|
||||
|
||||
__all__ = ["draw_histogram", "draw_cdf", "generate_plot"]
|
||||
@@ -1,162 +0,0 @@
|
||||
"""Plotting module for disk latency visualization."""
|
||||
|
||||
import matplotlib
|
||||
matplotlib.use("Agg") # Non-interactive backend for server environments
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.ticker import FuncFormatter
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
from disk_checker.core.converter import to_unit, UNIT_LABELS
|
||||
from disk_checker.core.helpers import human_count
|
||||
|
||||
|
||||
def draw_histogram(
|
||||
ax,
|
||||
df,
|
||||
metrics: List[str],
|
||||
unit: str = "us",
|
||||
log_y: bool = False,
|
||||
log_x: bool = False,
|
||||
normalize: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Draw a histogram showing the distribution of latencies for each metric.
|
||||
|
||||
Uses a step plot (where="post") to show the discrete nature of the data.
|
||||
If normalize=True, Y-axis shows probability density instead of counts.
|
||||
|
||||
Args:
|
||||
ax: Matplotlib axis object
|
||||
df: Long-format DataFrame from load_latency_csv
|
||||
metrics: List of metric names to plot
|
||||
unit: X-axis unit (ns, us, ms, s)
|
||||
log_y: Use logarithmic Y scale
|
||||
log_x: Use logarithmic X scale
|
||||
normalize: Normalize to probability density
|
||||
"""
|
||||
for metric in metrics:
|
||||
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||
if g.empty:
|
||||
continue
|
||||
|
||||
x = to_unit(g["latency_ns"].values, unit)
|
||||
y = g["count"].values.astype(float)
|
||||
|
||||
if normalize:
|
||||
s = y.sum()
|
||||
if s > 0:
|
||||
y = y / s
|
||||
|
||||
ax.step(x, y, where="post", lw=1.4, label=metric)
|
||||
ax.fill_between(x, y, step="post", alpha=0.25)
|
||||
|
||||
ax.set_xlabel(f"Latency ({UNIT_LABELS[unit]})")
|
||||
ax.set_ylabel("Probability" if normalize else "Number of I/Os")
|
||||
ax.set_title("Completion latency distribution")
|
||||
ax.grid(True, which="both", ls=":", alpha=0.5)
|
||||
|
||||
if not normalize:
|
||||
ax.yaxis.set_major_formatter(FuncFormatter(human_count))
|
||||
if log_y:
|
||||
ax.set_yscale("log")
|
||||
if log_x:
|
||||
ax.set_xscale("log")
|
||||
if len(metrics) > 1:
|
||||
ax.legend(title="metric", fontsize=9)
|
||||
|
||||
|
||||
def draw_cdf(ax, df, metrics: List[str], unit: str = "us", log_x: bool = False) -> None:
|
||||
"""
|
||||
Draw a Cumulative Distribution Function (CDF).
|
||||
|
||||
Shows the percentage of I/Os completed within each latency threshold.
|
||||
Each metric gets its own line; Y-axis ranges from 0% to 100%.
|
||||
|
||||
Args:
|
||||
ax: Matplotlib axis object
|
||||
df: Long-format DataFrame
|
||||
metrics: List of metric names
|
||||
unit: X-axis unit
|
||||
log_x: Use logarithmic X scale
|
||||
"""
|
||||
for metric in metrics:
|
||||
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||
if g.empty:
|
||||
continue
|
||||
|
||||
x = to_unit(g["latency_ns"].values, unit)
|
||||
counts = g["count"].values.astype(float)
|
||||
total = counts.sum()
|
||||
if total <= 0:
|
||||
continue
|
||||
|
||||
cdf = np.cumsum(counts) / total * 100.0
|
||||
ax.plot(x, cdf, lw=1.6, label=f"{metric} (n={int(total):,})")
|
||||
|
||||
ax.set_xlabel(f"Latency ({UNIT_LABELS[unit]})")
|
||||
ax.set_ylabel("Cumulative % of I/Os")
|
||||
ax.set_title("Cumulative latency distribution")
|
||||
ax.set_ylim(0, 100)
|
||||
ax.grid(True, which="both", ls=":", alpha=0.5)
|
||||
ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:.0f}%"))
|
||||
|
||||
if log_x:
|
||||
ax.set_xscale("log")
|
||||
ax.legend(fontsize=9)
|
||||
|
||||
|
||||
def generate_plot(
|
||||
df,
|
||||
metrics: List[str],
|
||||
output_path: Path,
|
||||
unit: str = "us",
|
||||
log_y: bool = False,
|
||||
log_x: bool = False,
|
||||
normalize: bool = False,
|
||||
show_cdf: bool = False,
|
||||
cdf_only: bool = False,
|
||||
title: str = "",
|
||||
dpi: int = 150,
|
||||
) -> Path:
|
||||
"""
|
||||
Generate a complete plot with histogram and/or CDF.
|
||||
|
||||
Creates the appropriate subplot layout based on the options provided.
|
||||
|
||||
Args:
|
||||
df: Long-format DataFrame
|
||||
metrics: Metrics to plot
|
||||
output_path: Where to save the plot
|
||||
unit: X-axis unit
|
||||
log_y: Logarithmic Y scale
|
||||
log_x: Logarithmic X scale
|
||||
normalize: Normalize histogram
|
||||
show_cdf: Show CDF below histogram
|
||||
cdf_only: Show only CDF
|
||||
title: Plot title
|
||||
dpi: Output resolution
|
||||
|
||||
Returns:
|
||||
Path to saved plot
|
||||
"""
|
||||
if cdf_only:
|
||||
fig, ax = plt.subplots(figsize=(11, 6))
|
||||
draw_cdf(ax, df, metrics, unit, log_x)
|
||||
elif show_cdf:
|
||||
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(11, 10))
|
||||
draw_histogram(
|
||||
ax1, df, metrics, unit, log_y, log_x, normalize
|
||||
)
|
||||
draw_cdf(ax2, df, metrics, unit, log_x)
|
||||
else:
|
||||
fig, ax = plt.subplots(figsize=(11, 6))
|
||||
draw_histogram(ax, df, metrics, unit, log_y, log_x, normalize)
|
||||
|
||||
if title:
|
||||
fig.suptitle(title, fontsize=13, y=0.995)
|
||||
fig.tight_layout()
|
||||
fig.savefig(output_path, dpi=dpi, bbox_inches="tight")
|
||||
plt.close(fig)
|
||||
return output_path
|
||||
@@ -1 +0,0 @@
|
||||
"""Web backend module for disk-checker API."""
|
||||
@@ -1,95 +0,0 @@
|
||||
"""Pydantic models for the web API."""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import List, Optional, Dict, Any
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class ToolType(str, Enum):
|
||||
"""Available benchmarking tools."""
|
||||
FIO = "fio"
|
||||
BADBLOCKS = "badblocks"
|
||||
IO_URING = "io_uring"
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
"""Status of a benchmark job."""
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
class MetricInfo(BaseModel):
|
||||
"""Information about a metric."""
|
||||
name: str
|
||||
total_ios: int
|
||||
min_latency_ns: Optional[float] = None
|
||||
max_latency_ns: Optional[float] = None
|
||||
avg_latency_ns: Optional[float] = None
|
||||
|
||||
|
||||
class PercentileInfo(BaseModel):
|
||||
"""Latency value at a specific percentile."""
|
||||
percentile: float
|
||||
latency_ns: float
|
||||
latency_display: str
|
||||
|
||||
|
||||
class MetricResult(BaseModel):
|
||||
"""Complete result for a metric."""
|
||||
metric: str
|
||||
total_ios: int
|
||||
percentiles: List[PercentileInfo]
|
||||
min_latency_ns: float
|
||||
max_latency_ns: float
|
||||
avg_latency_ns: float
|
||||
|
||||
|
||||
class ScanResponse(BaseModel):
|
||||
"""Response from CSV scan endpoint."""
|
||||
metrics: List[MetricResult]
|
||||
summary: Dict[str, Any]
|
||||
|
||||
|
||||
class BenchmarkRequest(BaseModel):
|
||||
"""Request to start a benchmark job."""
|
||||
device: str = Field(..., description="Block device path (e.g., /dev/sda)")
|
||||
tool: ToolType = ToolType.FIO
|
||||
job_file: Optional[str] = None
|
||||
options: Dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
class Config:
|
||||
json_schema_extra = {
|
||||
"example": {
|
||||
"device": "/dev/sda",
|
||||
"tool": "fio",
|
||||
"options": {
|
||||
"rw": "randread",
|
||||
"bs": "4k",
|
||||
"size": "1G",
|
||||
"runtime": "60"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class JobStatusResponse(BaseModel):
|
||||
"""Response with job status."""
|
||||
job_id: str
|
||||
status: JobStatus
|
||||
output_path: Optional[str] = None
|
||||
error: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
completed_at: Optional[str] = None
|
||||
|
||||
|
||||
class PlotRequest(BaseModel):
|
||||
"""Request to generate a plot."""
|
||||
metric: Optional[str] = None
|
||||
unit: str = Field(default="us", pattern="^(ns|us|ms|s)$")
|
||||
log_x: bool = False
|
||||
log_y: bool = False
|
||||
cdf: bool = False
|
||||
normalize: bool = False
|
||||
title: Optional[str] = None
|
||||
@@ -1,209 +0,0 @@
|
||||
"""API routes for disk-checker web backend."""
|
||||
|
||||
import asyncio
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, UploadFile, File, HTTPException, Depends, Request
|
||||
from fastapi.responses import JSONResponse, FileResponse
|
||||
import pandas as pd
|
||||
|
||||
from disk_checker.core.extractor import load_latency_csv
|
||||
from disk_checker.core.helpers import percentile_table
|
||||
from disk_checker.plotting.plots import generate_plot
|
||||
from disk_checker.web.models import (
|
||||
ScanResponse,
|
||||
BenchmarkRequest,
|
||||
JobStatusResponse,
|
||||
PlotRequest,
|
||||
MetricInfo,
|
||||
PercentileInfo,
|
||||
MetricResult,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _calculate_metric_results(df: pd.DataFrame, metric: str, pcts: List[float]) -> MetricResult:
|
||||
"""Calculate detailed results for a metric."""
|
||||
metric_df = df[df["metric"] == metric]
|
||||
counts = metric_df["count"].values.astype(float)
|
||||
total_ios = int(counts.sum())
|
||||
|
||||
latencies_ns = metric_df["latency_ns"].values
|
||||
|
||||
# Calculate percentiles
|
||||
percentiles = []
|
||||
cdf = __import__('numpy').cumsum(counts) / total_ios * 100.0
|
||||
for p in pcts:
|
||||
idx = min(__import__('numpy').searchsorted(cdf, p), len(latencies_ns) - 1)
|
||||
percentiles.append(PercentileInfo(
|
||||
percentile=p,
|
||||
latency_ns=float(latencies_ns[idx]),
|
||||
latency_display=f"{latencies_ns[idx]:.3f} ns"
|
||||
))
|
||||
|
||||
return MetricResult(
|
||||
metric=metric,
|
||||
total_ios=total_ios,
|
||||
percentiles=percentiles,
|
||||
min_latency_ns=float(latencies_ns.min()),
|
||||
max_latency_ns=float(latencies_ns.max()),
|
||||
avg_latency_ns=float(latencies_ns.mean()),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/scan", response_model=ScanResponse)
|
||||
async def scan_csv(file: UploadFile = File(...)):
|
||||
"""
|
||||
Upload a fio JSON+ CSV and return analysis.
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".csv") as tmp:
|
||||
content = await file.read()
|
||||
tmp.write(content)
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
df = load_latency_csv(tmp_path)
|
||||
if df.empty:
|
||||
raise HTTPException(status_code=400, detail="No valid data in CSV")
|
||||
|
||||
# Calculate metrics
|
||||
metrics = []
|
||||
pcts = [50, 90, 99, 99.9, 99.99]
|
||||
for metric in sorted(df["metric"].unique()):
|
||||
metrics.append(_calculate_metric_results(df, metric, pcts))
|
||||
|
||||
summary = {
|
||||
"total_metrics": len(metrics),
|
||||
"total_ios": sum(m.total_ios for m in metrics),
|
||||
}
|
||||
|
||||
return ScanResponse(metrics=metrics, summary=summary)
|
||||
finally:
|
||||
Path(tmp_path).unlink(missing_ok=True)
|
||||
|
||||
|
||||
@router.post("/plot")
|
||||
async def generate_plot_endpoint(
|
||||
file: UploadFile = File(...),
|
||||
metric: Optional[str] = None,
|
||||
unit: str = "us",
|
||||
log_x: bool = False,
|
||||
log_y: bool = False,
|
||||
cdf: bool = False,
|
||||
normalize: bool = False,
|
||||
title: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
Generate and return a latency plot.
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".csv") as tmp:
|
||||
content = await file.read()
|
||||
tmp.write(content)
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
df = load_latency_csv(tmp_path)
|
||||
if df.empty:
|
||||
raise HTTPException(status_code=400, detail="No valid data in CSV")
|
||||
|
||||
# Determine metrics
|
||||
available = sorted(df["metric"].unique())
|
||||
if request.metric:
|
||||
if request.metric not in available:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Metric {request.metric} not found"
|
||||
)
|
||||
metrics = [request.metric]
|
||||
else:
|
||||
totals = df.groupby("metric")["count"].sum().sort_values(ascending=False)
|
||||
metrics = [totals.index[0]]
|
||||
|
||||
# Generate plot
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".png") as img:
|
||||
output_path = Path(img.name)
|
||||
|
||||
generate_plot(
|
||||
df=df,
|
||||
metrics=metrics,
|
||||
output_path=output_path,
|
||||
unit=unit,
|
||||
log_x=log_x,
|
||||
log_y=log_y,
|
||||
normalize=normalize,
|
||||
show_cdf=cdf,
|
||||
title=title or "",
|
||||
)
|
||||
|
||||
return FileResponse(output_path, media_type="image/png")
|
||||
finally:
|
||||
Path(tmp_path).unlink(missing_ok=True)
|
||||
|
||||
|
||||
@router.post("/benchmark", response_model=JobStatusResponse)
|
||||
async def start_benchmark(request: BenchmarkRequest):
|
||||
"""
|
||||
Start a disk benchmark job.
|
||||
"""
|
||||
orchestrator = request.app.state.orchestrator
|
||||
|
||||
job_id = f"bench_{asyncio.get_event_loop().time()}"
|
||||
|
||||
async def run_job():
|
||||
try:
|
||||
if request.tool == "fio":
|
||||
output_path = await orchestrator.run_fio(
|
||||
device=request.device,
|
||||
**request.options,
|
||||
)
|
||||
elif request.tool == "badblocks":
|
||||
output_path = await orchestrator.run_badblocks(
|
||||
device=request.device,
|
||||
**request.options,
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unknown tool: {request.tool}")
|
||||
return {
|
||||
"job_id": job_id,
|
||||
"status": "completed",
|
||||
"output_path": str(output_path),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"job_id": job_id, "status": "failed", "error": str(e)}
|
||||
|
||||
request.app.state.jobs[job_id] = {
|
||||
"status": "running",
|
||||
"task": asyncio.create_task(run_job()),
|
||||
}
|
||||
|
||||
return JobStatusResponse(job_id=job_id, status="running")
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}", response_model=JobStatusResponse)
|
||||
async def get_job_status(job_id: str, request: Request):
|
||||
"""
|
||||
Check the status of a benchmark job.
|
||||
"""
|
||||
jobs = request.app.state.jobs
|
||||
if job_id not in jobs:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
job = jobs[job_id]
|
||||
task = job["task"]
|
||||
|
||||
if task.done():
|
||||
result = task.result()
|
||||
return JobStatusResponse(**result)
|
||||
|
||||
return JobStatusResponse(job_id=job_id, status="running")
|
||||
|
||||
|
||||
@router.get("/jobs", response_model=List[str])
|
||||
async def list_jobs(request: Request):
|
||||
"""
|
||||
List all benchmark jobs.
|
||||
"""
|
||||
return list(request.app.state.jobs.keys())
|
||||
@@ -1,49 +0,0 @@
|
||||
"""FastAPI application for disk-checker web API."""
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from contextlib import asynccontextmanager
|
||||
import logging
|
||||
|
||||
from disk_checker.core.orchestrator import Orchestrator
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""Application lifespan handler."""
|
||||
# Startup: initialize orchestrator
|
||||
app.state.orchestrator = Orchestrator(work_dir=app.state.work_dir)
|
||||
app.state.jobs = {}
|
||||
logger.info("Disk Checker API started")
|
||||
yield
|
||||
# Shutdown: cleanup
|
||||
logger.info("Disk Checker API stopped")
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="Disk Checker API",
|
||||
description="API for disk benchmarking and latency visualization",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
"""Health check endpoint."""
|
||||
return {"status": "ok", "version": "0.1.0"}
|
||||
|
||||
|
||||
# Import and include routers
|
||||
from disk_checker.web.routes import router as api_router
|
||||
app.include_router(api_router, prefix="/api/v1")
|
||||
@@ -1 +0,0 @@
|
||||
"""Tests for disk-checker."""
|
||||
@@ -1,68 +0,0 @@
|
||||
"""Tests for the unit conversion module."""
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from disk_checker.core.converter import to_unit, UNIT_FACTORS, UNIT_LABELS
|
||||
|
||||
|
||||
class TestToUnit:
|
||||
"""Tests for to_unit function."""
|
||||
|
||||
def test_nanoseconds_to_microseconds(self):
|
||||
"""Test conversion from ns to us."""
|
||||
values = np.array([1000, 2000, 3000])
|
||||
result = to_unit(values, "us")
|
||||
expected = np.array([1.0, 2.0, 3.0])
|
||||
np.testing.assert_array_equal(result, expected)
|
||||
|
||||
def test_nanoseconds_to_milliseconds(self):
|
||||
"""Test conversion from ns to ms."""
|
||||
values = np.array([1000000, 2000000])
|
||||
result = to_unit(values, "ms")
|
||||
expected = np.array([1.0, 2.0])
|
||||
np.testing.assert_array_equal(result, expected)
|
||||
|
||||
def test_nanoseconds_to_seconds(self):
|
||||
"""Test conversion from ns to s."""
|
||||
values = np.array([1000000000, 2000000000])
|
||||
result = to_unit(values, "s")
|
||||
expected = np.array([1.0, 2.0])
|
||||
np.testing.assert_array_equal(result, expected)
|
||||
|
||||
def test_same_unit_returns_same(self):
|
||||
"""Test conversion to same unit."""
|
||||
values = np.array([100, 200, 300])
|
||||
result = to_unit(values, "ns")
|
||||
np.testing.assert_array_equal(result, values)
|
||||
|
||||
def test_empty_array(self):
|
||||
"""Test conversion with empty array."""
|
||||
values = np.array([])
|
||||
result = to_unit(values, "us")
|
||||
assert len(result) == 0
|
||||
|
||||
def test_single_value(self):
|
||||
"""Test conversion with single value."""
|
||||
result = to_unit(1000, "us")
|
||||
assert result == 1.0
|
||||
|
||||
|
||||
class TestConstants:
|
||||
"""Tests for module constants."""
|
||||
|
||||
def test_unit_factors(self):
|
||||
"""Test UNIT_FACTORS contains all expected units."""
|
||||
assert set(UNIT_FACTORS.keys()) == {"ns", "us", "ms", "s"}
|
||||
assert UNIT_FACTORS["ns"] == 1.0
|
||||
assert UNIT_FACTORS["us"] == 1e3
|
||||
assert UNIT_FACTORS["ms"] == 1e6
|
||||
assert UNIT_FACTORS["s"] == 1e9
|
||||
|
||||
def test_unit_labels(self):
|
||||
"""Test UNIT_LABELS contains all expected units."""
|
||||
assert set(UNIT_LABELS.keys()) == {"ns", "us", "ms", "s"}
|
||||
assert UNIT_LABELS["ns"] == "ns"
|
||||
assert UNIT_LABELS["us"] == "µs"
|
||||
assert UNIT_LABELS["ms"] == "ms"
|
||||
assert UNIT_LABELS["s"] == "s"
|
||||
@@ -1,72 +0,0 @@
|
||||
"""Tests for the data extraction module."""
|
||||
|
||||
import pytest
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
|
||||
from disk_checker.core.extractor import load_latency_csv
|
||||
|
||||
|
||||
class TestLoadLatencyCsv:
|
||||
"""Tests for load_latency_csv function."""
|
||||
|
||||
def test_load_valid_csv(self, sample_csv_path):
|
||||
"""Test loading a valid fio CSV file."""
|
||||
df = load_latency_csv(str(sample_csv_path))
|
||||
assert not df.empty
|
||||
assert "metric" in df.columns
|
||||
assert "latency_ns" in df.columns
|
||||
assert "count" in df.columns
|
||||
|
||||
def test_returns_long_format(self, sample_csv_path):
|
||||
"""Test that output is in long format with metric column."""
|
||||
df = load_latency_csv(str(sample_csv_path))
|
||||
metrics = df["metric"].unique()
|
||||
assert "read_clat" in metrics
|
||||
|
||||
def test_empty_csv_raises(self, empty_csv_path):
|
||||
"""Test that empty CSV raises ValueError."""
|
||||
with pytest.raises(ValueError):
|
||||
load_latency_csv(str(empty_csv_path))
|
||||
|
||||
def test_invalid_format_raises(self, invalid_csv_path):
|
||||
"""Test that invalid CSV format raises ValueError."""
|
||||
with pytest.raises(ValueError):
|
||||
load_latency_csv(str(invalid_csv_path))
|
||||
|
||||
def test_missing_nsec_column_raises(self, tmp_path):
|
||||
"""Test that missing nsec column raises ValueError."""
|
||||
csv_content = "col1,col2\n1,2\n"
|
||||
path = tmp_path / "invalid.csv"
|
||||
path.write_text(csv_content)
|
||||
with pytest.raises(ValueError, match="expected first column to be 'nsec'"):
|
||||
load_latency_csv(str(path))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_csv_path(tmp_path):
|
||||
"""Create a sample CSV file for testing."""
|
||||
csv_content = """nsec,read_clat_ns_count,read_clat_ns_cumulative,read_clat_ns_percentile
|
||||
19791872,1,1,0.00016775708773695687
|
||||
22413312,4,5,0.0008387854386847844
|
||||
22675456,9,14,0.0023485992283173965
|
||||
"""
|
||||
path = tmp_path / "test.csv"
|
||||
path.write_text(csv_content)
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def empty_csv_path(tmp_path):
|
||||
"""Create an empty CSV file for testing."""
|
||||
path = tmp_path / "empty.csv"
|
||||
path.write_text("nsec\n")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def invalid_csv_path(tmp_path):
|
||||
"""Create an invalid CSV file for testing."""
|
||||
path = tmp_path / "invalid.csv"
|
||||
path.write_text("col1,col2\n1,2\n")
|
||||
return path
|
||||
@@ -1,77 +0,0 @@
|
||||
"""Tests for helper functions."""
|
||||
|
||||
import pandas as pd
|
||||
import pytest
|
||||
|
||||
from disk_checker.core.helpers import human_count, percentile_table
|
||||
|
||||
|
||||
class TestHumanCount:
|
||||
"""Tests for human_count function."""
|
||||
|
||||
def test_small_number(self):
|
||||
"""Test formatting of small numbers."""
|
||||
assert human_count(100) == "100"
|
||||
assert human_count(50) == "50"
|
||||
|
||||
def test_thousands(self):
|
||||
"""Test formatting of thousands."""
|
||||
assert human_count(1500) == "2k"
|
||||
assert human_count(2500) == "2k"
|
||||
assert human_count(1000) == "1k"
|
||||
assert human_count(999) == "999"
|
||||
|
||||
def test_millions(self):
|
||||
"""Test formatting of millions."""
|
||||
assert human_count(1500000) == "1.5M"
|
||||
assert human_count(2750000) == "2.8M"
|
||||
|
||||
def test_billions(self):
|
||||
"""Test formatting of billions."""
|
||||
assert human_count(1500000000) == "1.5G"
|
||||
assert human_count(2750000000) == "2.8G"
|
||||
|
||||
def test_zero(self):
|
||||
"""Test formatting of zero."""
|
||||
assert human_count(0) == "0"
|
||||
|
||||
def test_position_argument_ignored(self):
|
||||
"""Test that _pos argument is ignored."""
|
||||
assert human_count(1000, 0) == "1k"
|
||||
|
||||
|
||||
class TestPercentileTable:
|
||||
"""Tests for percentile_table function."""
|
||||
|
||||
def test_basic_percentiles(self):
|
||||
"""Test basic percentile table generation."""
|
||||
df = pd.DataFrame({
|
||||
"metric": ["read_clat"] * 3,
|
||||
"latency_ns": [1000, 2000, 3000],
|
||||
"count": [1, 2, 1],
|
||||
})
|
||||
result = percentile_table(df, "read_clat", "us", [50, 90])
|
||||
assert "Metric: read_clat" in result
|
||||
assert "total I/Os: 4" in result
|
||||
assert "percentile" in result
|
||||
assert "latency" in result
|
||||
|
||||
def test_missing_metric(self):
|
||||
"""Test percentile table with missing metric."""
|
||||
df = pd.DataFrame({
|
||||
"metric": ["read_clat"],
|
||||
"latency_ns": [1000],
|
||||
"count": [1],
|
||||
})
|
||||
result = percentile_table(df, "write_clat", "us", [50])
|
||||
assert "(no data for metric write_clat)" in result
|
||||
|
||||
def test_unit_display(self):
|
||||
"""Test that correct unit is displayed."""
|
||||
df = pd.DataFrame({
|
||||
"metric": ["read_clat"],
|
||||
"latency_ns": [1000000], # 1ms in ns
|
||||
"count": [1],
|
||||
})
|
||||
result = percentile_table(df, "read_clat", "ms", [50])
|
||||
assert "ms" in result
|
||||
@@ -1,53 +0,0 @@
|
||||
"""Tests for the orchestrator module."""
|
||||
|
||||
import asyncio
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from disk_checker.core.orchestrator import Orchestrator, ToolStatus, JobResult
|
||||
|
||||
|
||||
class TestOrchestrator:
|
||||
"""Tests for Orchestrator class."""
|
||||
|
||||
@pytest.fixture
|
||||
def orchestrator(self, tmp_path):
|
||||
"""Create an orchestrator instance."""
|
||||
return Orchestrator(work_dir=tmp_path)
|
||||
|
||||
def test_initial_state(self, orchestrator):
|
||||
"""Test initial orchestrator state."""
|
||||
assert orchestrator.list_jobs() == {}
|
||||
|
||||
def test_run_fio_creates_job(self, orchestrator, tmp_path):
|
||||
"""Test that run_fio creates a job entry."""
|
||||
with patch("asyncio.create_subprocess_exec") as mock_exec:
|
||||
mock_exec.return_value = AsyncMock()
|
||||
result = asyncio.get_event_loop().run_until_complete(
|
||||
orchestrator.run_fio("/dev/sda")
|
||||
)
|
||||
|
||||
jobs = orchestrator.list_jobs()
|
||||
assert len(jobs) == 1
|
||||
job = list(jobs.values())[0]
|
||||
assert job.status == ToolStatus.COMPLETED
|
||||
# Note: output_path may not exist in test environment without actual fio
|
||||
|
||||
def test_get_job_status(self, orchestrator, tmp_path):
|
||||
"""Test retrieving job status."""
|
||||
with patch("asyncio.create_subprocess_exec") as mock_exec:
|
||||
mock_exec.return_value = AsyncMock()
|
||||
result = asyncio.get_event_loop().run_until_complete(
|
||||
orchestrator.run_fio("/dev/sda")
|
||||
)
|
||||
job_id = result.job_id
|
||||
|
||||
job = orchestrator.get_job_status(job_id)
|
||||
assert job is not None
|
||||
assert job.status == ToolStatus.COMPLETED
|
||||
|
||||
def test_get_nonexistent_job(self, orchestrator):
|
||||
"""Test getting status of non-existent job."""
|
||||
job = orchestrator.get_job_status("nonexistent")
|
||||
assert job is None
|
||||
@@ -1,134 +0,0 @@
|
||||
"""Tests for the plotting module."""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
from disk_checker.plotting.plots import draw_histogram, draw_cdf, generate_plot
|
||||
import pandas as pd
|
||||
|
||||
|
||||
class TestDrawHistogram:
|
||||
"""Tests for draw_histogram function."""
|
||||
|
||||
@pytest.fixture
|
||||
def sample_df(self):
|
||||
"""Create a sample DataFrame for testing."""
|
||||
return pd.DataFrame({
|
||||
"metric": ["read_clat", "read_clat", "write_clat"],
|
||||
"latency_ns": [1000, 2000, 1500],
|
||||
"count": [1, 2, 1],
|
||||
})
|
||||
|
||||
def test_draws_histogram(self, sample_df, tmp_path):
|
||||
"""Test that histogram is drawn without errors."""
|
||||
fig, ax = plt.subplots()
|
||||
draw_histogram(ax, sample_df, ["read_clat"])
|
||||
assert ax.get_title() == "Completion latency distribution"
|
||||
plt.close(fig)
|
||||
|
||||
def test_multiple_metrics(self, sample_df, tmp_path):
|
||||
"""Test drawing multiple metrics."""
|
||||
fig, ax = plt.subplots()
|
||||
draw_histogram(ax, sample_df, ["read_clat", "write_clat"])
|
||||
assert len(ax.collections) > 0
|
||||
plt.close(fig)
|
||||
|
||||
def test_log_scale(self, sample_df, tmp_path):
|
||||
"""Test logarithmic scale."""
|
||||
fig, ax = plt.subplots()
|
||||
draw_histogram(ax, sample_df, ["read_clat"], log_x=True, log_y=True)
|
||||
assert ax.get_xscale() == "log"
|
||||
assert ax.get_yscale() == "log"
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
class TestDrawCdf:
|
||||
"""Tests for draw_cdf function."""
|
||||
|
||||
@pytest.fixture
|
||||
def sample_df(self):
|
||||
"""Create a sample DataFrame for testing."""
|
||||
return pd.DataFrame({
|
||||
"metric": ["read_clat", "read_clat"],
|
||||
"latency_ns": [1000, 2000],
|
||||
"count": [1, 3],
|
||||
})
|
||||
|
||||
def test_draws_cdf(self, sample_df, tmp_path):
|
||||
"""Test that CDF is drawn without errors."""
|
||||
fig, ax = plt.subplots()
|
||||
draw_cdf(ax, sample_df, ["read_clat"])
|
||||
assert ax.get_title() == "Cumulative latency distribution"
|
||||
assert ax.get_ylim() == (0, 100)
|
||||
plt.close(fig)
|
||||
|
||||
def test_log_scale(self, sample_df, tmp_path):
|
||||
"""Test logarithmic X scale."""
|
||||
fig, ax = plt.subplots()
|
||||
draw_cdf(ax, sample_df, ["read_clat"], log_x=True)
|
||||
assert ax.get_xscale() == "log"
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
class TestGeneratePlot:
|
||||
"""Tests for generate_plot function."""
|
||||
|
||||
@pytest.fixture
|
||||
def sample_df(self):
|
||||
"""Create a sample DataFrame for testing."""
|
||||
return pd.DataFrame({
|
||||
"metric": ["read_clat", "read_clat"],
|
||||
"latency_ns": [1000, 2000],
|
||||
"count": [1, 3],
|
||||
})
|
||||
|
||||
def test_generates_histogram(self, sample_df, tmp_path):
|
||||
"""Test generating histogram plot."""
|
||||
output = tmp_path / "test.png"
|
||||
result = generate_plot(
|
||||
df=sample_df,
|
||||
metrics=["read_clat"],
|
||||
output_path=output,
|
||||
)
|
||||
assert result == output
|
||||
assert output.exists()
|
||||
|
||||
def test_generates_cdf_only(self, sample_df, tmp_path):
|
||||
"""Test generating CDF-only plot."""
|
||||
output = tmp_path / "test_cdf.png"
|
||||
result = generate_plot(
|
||||
df=sample_df,
|
||||
metrics=["read_clat"],
|
||||
output_path=output,
|
||||
cdf_only=True,
|
||||
)
|
||||
assert result == output
|
||||
assert output.exists()
|
||||
|
||||
def test_generates_histogram_with_cdf(self, sample_df, tmp_path):
|
||||
"""Test generating histogram with CDF below."""
|
||||
output = tmp_path / "test_both.png"
|
||||
result = generate_plot(
|
||||
df=sample_df,
|
||||
metrics=["read_clat"],
|
||||
output_path=output,
|
||||
show_cdf=True,
|
||||
)
|
||||
assert result == output
|
||||
assert output.exists()
|
||||
|
||||
def test_custom_dpi(self, sample_df, tmp_path):
|
||||
"""Test custom DPI setting."""
|
||||
output = tmp_path / "test_dpi.png"
|
||||
result = generate_plot(
|
||||
df=sample_df,
|
||||
metrics=["read_clat"],
|
||||
output_path=output,
|
||||
dpi=300,
|
||||
)
|
||||
assert result == output
|
||||
assert output.exists()
|
||||
@@ -1,88 +0,0 @@
|
||||
"""Tests for the web backend."""
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from pathlib import Path
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from disk_checker.web.server import app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_csv_content():
|
||||
"""Sample CSV content for testing."""
|
||||
return """nsec,read_clat_ns_count,read_clat_ns_cumulative,read_clat_ns_percentile
|
||||
19791872,1,1,0.00016775708773695687
|
||||
22413312,4,5,0.0008387854386847844
|
||||
22675456,9,14,0.0023485992283173965
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_csv_path(tmp_path, sample_csv_content):
|
||||
"""Create sample CSV file."""
|
||||
path = tmp_path / "test.csv"
|
||||
path.write_text(sample_csv_content)
|
||||
return path
|
||||
|
||||
|
||||
def test_health_check(client):
|
||||
"""Test health check endpoint."""
|
||||
response = client.get("/health")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "ok"
|
||||
|
||||
|
||||
def test_scan_csv(client, sample_csv_path):
|
||||
"""Test CSV scan endpoint."""
|
||||
with open(sample_csv_path, "rb") as f:
|
||||
response = client.post("/api/v1/scan", files={"file": f})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "metrics" in data
|
||||
assert len(data["metrics"]) > 0
|
||||
|
||||
|
||||
def test_plot_endpoint(client, sample_csv_path):
|
||||
"""Test plot generation endpoint."""
|
||||
with open(sample_csv_path, "rb") as f:
|
||||
response = client.post(
|
||||
"/api/v1/plot",
|
||||
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||
data={"unit": "us"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.headers["content-type"] == "image/png"
|
||||
|
||||
|
||||
def test_plot_with_options(client, sample_csv_path):
|
||||
"""Test plot generation with options."""
|
||||
with open(sample_csv_path, "rb") as f:
|
||||
response = client.post(
|
||||
"/api/v1/plot",
|
||||
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||
data={
|
||||
"metric": "read_clat",
|
||||
"unit": "ms",
|
||||
"log_x": "true",
|
||||
"cdf": "true",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
def test_invalid_metric(client, sample_csv_path):
|
||||
"""Test invalid metric returns error."""
|
||||
with open(sample_csv_path, "rb") as f:
|
||||
response = client.post(
|
||||
"/api/v1/plot",
|
||||
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||
data={"metric": "invalid_metric", "unit": "us"},
|
||||
)
|
||||
assert response.status_code == 400
|
||||
Reference in New Issue
Block a user