Compare commits
15 Commits
main
..
9a3fa5a5af
| Author | SHA1 | Date | |
|---|---|---|---|
| 9a3fa5a5af | |||
| fe31a8893b | |||
| 37a2384fde | |||
| a0d0b04e78 | |||
| ac64a95502 | |||
| 31562586d7 | |||
| f0452e6ef6 | |||
| d659ec7910 | |||
| 57dc4bdd26 | |||
| c933f4f9ca | |||
| 271502cccd | |||
| 29c02c27fb | |||
| 2eed2053a1 | |||
| dac3aeaafb | |||
| 352bc16ba5 |
@@ -1,97 +0,0 @@
|
|||||||
name: "Bug Report"
|
|
||||||
description: "Report a bug or unexpected behavior to help us improve"
|
|
||||||
title: "[BUG]: "
|
|
||||||
labels: ["bug"]
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## ⚠️ Before You Begin
|
|
||||||
Please ensure you have:
|
|
||||||
- [ ] Searched existing issues to avoid duplicates
|
|
||||||
- [ ] Confirmed this is a bug, not a question or configuration problem
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: description
|
|
||||||
attributes:
|
|
||||||
label: "📋 Description"
|
|
||||||
description: "Provide a clear and concise description of the bug"
|
|
||||||
placeholder: "What happened? What did you expect to happen instead?"
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: reproduction
|
|
||||||
attributes:
|
|
||||||
label: "🔁 Steps to Reproduce"
|
|
||||||
description: "Step-by-step instructions to reproduce the issue"
|
|
||||||
placeholder: |
|
|
||||||
1. Go to '...'
|
|
||||||
2. Click on '....'
|
|
||||||
3. Scroll down to '....'
|
|
||||||
4. See error
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: expected
|
|
||||||
attributes:
|
|
||||||
label: "✅ Expected Behavior"
|
|
||||||
description: "What you expected to happen"
|
|
||||||
placeholder: "A clear description of what should happen..."
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: actual
|
|
||||||
attributes:
|
|
||||||
label: "❌ Actual Behavior"
|
|
||||||
description: "What actually happened"
|
|
||||||
placeholder: "Include error messages, stack traces, or unexpected outcomes..."
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: logs
|
|
||||||
attributes:
|
|
||||||
label: "📄 Logs / Screenshots"
|
|
||||||
description: |
|
|
||||||
Provide relevant logs, error messages, or screenshots.
|
|
||||||
For logs, please use a pastebin and share the URL.
|
|
||||||
**Remember to remove any sensitive information (API keys, passwords, etc.).**
|
|
||||||
placeholder: "Paste logs here or provide a Gist URL..."
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
---
|
|
||||||
## 🖥️ Environment Details
|
|
||||||
Please fill out the relevant information below.
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: additional
|
|
||||||
attributes:
|
|
||||||
label: "📎 Additional Context"
|
|
||||||
description: "Any other information that might be relevant"
|
|
||||||
placeholder: |
|
|
||||||
- Database type and version (e.g., PostgreSQL 15, SQLite)
|
|
||||||
- Reverse proxy/CDN in use (e.g., Nginx, Cloudflare)
|
|
||||||
- Any custom configuration
|
|
||||||
- Related issues or PRs
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: checkboxes
|
|
||||||
id: checklist
|
|
||||||
attributes:
|
|
||||||
label: "✅ Submission Checklist"
|
|
||||||
description: "Please confirm the following before submitting"
|
|
||||||
options:
|
|
||||||
- label: "I have searched for existing issues (open and closed) that report the same problem"
|
|
||||||
required: true
|
|
||||||
- label: "I have provided clear steps to reproduce the issue"
|
|
||||||
required: true
|
|
||||||
- label: "I have included relevant logs or error messages (with sensitive info removed)"
|
|
||||||
required: false
|
|
||||||
@@ -1,99 +0,0 @@
|
|||||||
name: "Feature Request"
|
|
||||||
description: "Suggest a new feature or enhancement for this project"
|
|
||||||
title: "[FEATURE]: "
|
|
||||||
labels: ["enhancement"]
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## ⚠️ Before You Begin
|
|
||||||
Please ensure you have:
|
|
||||||
- [ ] Searched existing issues (open and closed) to avoid duplicates
|
|
||||||
- [ ] Reviewed the [project documentation](https://git.vmn.su/max/vmn-ansible/wiki) for existing functionality
|
|
||||||
- [ ] Confirmed this is a feature request, not a bug or configuration question
|
|
||||||
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: problem
|
|
||||||
attributes:
|
|
||||||
label: "🎯 Problem Statement"
|
|
||||||
description: "What problem does this feature solve? What can't you do today?"
|
|
||||||
placeholder: |
|
|
||||||
I'm always frustrated when...
|
|
||||||
Currently, it's hard/impossible to...
|
|
||||||
This feature would help me because...
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: solution
|
|
||||||
attributes:
|
|
||||||
label: "💡 Proposed Solution"
|
|
||||||
description: "Describe the solution you'd like to see"
|
|
||||||
placeholder: |
|
|
||||||
A clear description of what you want to happen.
|
|
||||||
If you have a specific API, UI, or implementation in mind, describe it here.
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: screenshots
|
|
||||||
attributes:
|
|
||||||
label: "📸 Screenshots"
|
|
||||||
description: "If you can, provide screenshots or diagrams of the proposed feature"
|
|
||||||
placeholder: "Drag and drop images or paste links..."
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: technical
|
|
||||||
attributes:
|
|
||||||
label: "⚙️ Technical Approach"
|
|
||||||
description: "If you have ideas about how to implement this, share them here"
|
|
||||||
placeholder: |
|
|
||||||
- Which components/modules would be affected?
|
|
||||||
- Any database schema changes?
|
|
||||||
- API design considerations?
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: acceptance
|
|
||||||
attributes:
|
|
||||||
label: "✅ Acceptance Criteria"
|
|
||||||
description: "What does 'done' look like for this feature?"
|
|
||||||
placeholder: |
|
|
||||||
- [ ] Feature works in the UI
|
|
||||||
- [ ] API endpoints are documented
|
|
||||||
- [ ] Tests are added
|
|
||||||
- [ ] Documentation is updated
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: dropdown
|
|
||||||
id: contribution
|
|
||||||
attributes:
|
|
||||||
label: "🤝 Are you willing to contribute this feature?"
|
|
||||||
description: "Knowing if you can help implement this helps us prioritize"
|
|
||||||
options:
|
|
||||||
- "Yes, I can contribute"
|
|
||||||
- "Yes, but I need guidance"
|
|
||||||
- "No, but I can test/review"
|
|
||||||
- "No, I'm just requesting"
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: checkboxes
|
|
||||||
id: checklist
|
|
||||||
attributes:
|
|
||||||
label: "✅ Submission Checklist"
|
|
||||||
description: "Please confirm the following before submitting"
|
|
||||||
options:
|
|
||||||
- label: "I have searched for existing issues (open and closed) that request the same feature"
|
|
||||||
required: true
|
|
||||||
- label: "I have explained the problem this feature solves"
|
|
||||||
required: true
|
|
||||||
- label: "I have described a concrete solution or direction"
|
|
||||||
required: true
|
|
||||||
- label: "I have considered alternatives and explained why they fall short"
|
|
||||||
required: false
|
|
||||||
@@ -1,134 +0,0 @@
|
|||||||
name: "Refactor Request"
|
|
||||||
description: "Propose a code restructuring or cleanup with no behavior change"
|
|
||||||
title: "[REFACTOR]: "
|
|
||||||
labels: ["refactoring"]
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## ⚠️ Before You Begin
|
|
||||||
Please ensure you have:
|
|
||||||
- [ ] Searched existing issues (open and closed) to avoid duplicates
|
|
||||||
- [ ] Reviewed the [project documentation](https://git.vmn.su/max/vmn-ansible/wiki) for architecture guidelines
|
|
||||||
- [ ] Confirmed this is a refactor (code restructuring with NO behavior change), not a bug fix or new feature
|
|
||||||
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: problem
|
|
||||||
attributes:
|
|
||||||
label: "🎯 Problem Statement"
|
|
||||||
description: "What pain points does the current code cause? Why is this refactor needed?"
|
|
||||||
placeholder: |
|
|
||||||
The current implementation is difficult to maintain because...
|
|
||||||
This code has accumulated technical debt due to...
|
|
||||||
I'm constantly frustrated when working with this module because...
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: scope
|
|
||||||
attributes:
|
|
||||||
label: "📂 Scope"
|
|
||||||
description: "Which files, modules, or components need refactoring? Be specific."
|
|
||||||
placeholder: |
|
|
||||||
- File paths: `src/auth/`, `internal/handler/`
|
|
||||||
- Component: `UserService` class
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: current-state
|
|
||||||
attributes:
|
|
||||||
label: "🔍 Current State"
|
|
||||||
description: "Describe the current code structure, its issues, and why it's problematic"
|
|
||||||
placeholder: |
|
|
||||||
- The function `processUser()` is 500+ lines long
|
|
||||||
- Duplicated logic across 3 different files
|
|
||||||
- Mixed concerns (business logic + HTTP handling)
|
|
||||||
- Poor test coverage makes changes risky
|
|
||||||
- Naming is unclear and inconsistent
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: proposed-changes
|
|
||||||
attributes:
|
|
||||||
label: "💡 Proposed Solution"
|
|
||||||
description: "What should the code look like after the refactor?"
|
|
||||||
placeholder: |
|
|
||||||
- Extract `processUser()` into smaller, focused functions
|
|
||||||
- Consolidate duplicated logic into a shared utility
|
|
||||||
- Separate business logic from HTTP handlers
|
|
||||||
- Introduce interfaces for better testability
|
|
||||||
- Rename variables and functions for clarity
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: constraints
|
|
||||||
attributes:
|
|
||||||
label: "🚫 Constraints"
|
|
||||||
description: "What must NOT change? Public APIs, behavior, backward compatibility, performance characteristics?"
|
|
||||||
placeholder: |
|
|
||||||
- All public APIs must remain identical
|
|
||||||
- Database schema must not change
|
|
||||||
- External behavior must be identical
|
|
||||||
- Response formats must stay the same
|
|
||||||
- Performance must not degrade
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: benefits
|
|
||||||
attributes:
|
|
||||||
label: "📈 Expected Benefits"
|
|
||||||
description: "What improvements will this refactor bring?"
|
|
||||||
placeholder: |
|
|
||||||
- Improved maintainability
|
|
||||||
- Better testability
|
|
||||||
- Reduced code duplication
|
|
||||||
- Clearer separation of concerns
|
|
||||||
- Easier onboarding for new contributors
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: risks
|
|
||||||
attributes:
|
|
||||||
label: "⚠️ Risks & Mitigations"
|
|
||||||
description: "What could go wrong, and how will you mitigate it?"
|
|
||||||
placeholder: |
|
|
||||||
- Risk: Regression bugs
|
|
||||||
Mitigation: Comprehensive test coverage before and after
|
|
||||||
- Risk: Large diff making code review difficult
|
|
||||||
Mitigation: Break into smaller, incremental PRs
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: dropdown
|
|
||||||
id: contribution
|
|
||||||
attributes:
|
|
||||||
label: "🤝 Are you willing to contribute this refactor?"
|
|
||||||
description: "Knowing if you can help implement this helps us prioritize"
|
|
||||||
options:
|
|
||||||
- "Yes, I can contribute"
|
|
||||||
- "Yes, but I need guidance"
|
|
||||||
- "No, but I can review"
|
|
||||||
- "No, I'm just suggesting"
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: checkboxes
|
|
||||||
id: checklist
|
|
||||||
attributes:
|
|
||||||
label: "✅ Submission Checklist"
|
|
||||||
description: "Please confirm the following before submitting"
|
|
||||||
options:
|
|
||||||
- label: "I have searched for existing issues (open and closed) that request the same refactor"
|
|
||||||
required: true
|
|
||||||
- label: "I have clearly explained the problem this refactor solves"
|
|
||||||
required: true
|
|
||||||
- label: "I have described the scope of the refactor (specific files/modules)"
|
|
||||||
required: false
|
|
||||||
- label: "I have identified what must NOT change (APIs, behavior, backward compatibility)"
|
|
||||||
required: false
|
|
||||||
+50
-1
@@ -1,3 +1,52 @@
|
|||||||
.vscode
|
.vscode
|
||||||
__pycache__
|
__pycache__
|
||||||
test_data
|
test_data
|
||||||
|
|
||||||
|
# Python
|
||||||
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
|
*.so
|
||||||
|
.Python
|
||||||
|
build/
|
||||||
|
develop-eggs/
|
||||||
|
dist/
|
||||||
|
downloads/
|
||||||
|
eggs/
|
||||||
|
.eggs/
|
||||||
|
lib/
|
||||||
|
lib64/
|
||||||
|
parts/
|
||||||
|
sdist/
|
||||||
|
var/
|
||||||
|
wheels/
|
||||||
|
*.egg-info/
|
||||||
|
.installed.cfg
|
||||||
|
*.egg
|
||||||
|
|
||||||
|
# Virtual environments
|
||||||
|
venv/
|
||||||
|
ENV/
|
||||||
|
env/
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
|
||||||
|
# Environment variables
|
||||||
|
.env
|
||||||
|
.env.local
|
||||||
|
|
||||||
|
# Test coverage
|
||||||
|
.htmlcov/
|
||||||
|
.coverage
|
||||||
|
.coverage.*
|
||||||
|
coverage.xml
|
||||||
|
|
||||||
|
# Pytest
|
||||||
|
.pytest_cache/
|
||||||
|
|
||||||
|
# Generated files
|
||||||
|
*.png
|
||||||
|
*.csv
|
||||||
|
data/samples/*.csv
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Disk Checker
|
||||||
|
|
||||||
|
Disk benchmarking and latency visualization tool for fio and badblocks output.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- Parse fio JSON+ latency CSV files
|
||||||
|
- Generate histogram and CDF plots
|
||||||
|
- REST API for remote benchmarking
|
||||||
|
- Async job management for long-running benchmarks
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone <repository>
|
||||||
|
cd disk-checker
|
||||||
|
python -m venv venv
|
||||||
|
source venv/bin/activate
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## CLI Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Basic usage
|
||||||
|
python -m disk_checker scan_latency.csv
|
||||||
|
|
||||||
|
# With options
|
||||||
|
python -m disk_checker scan_latency.csv --metric read_clat --unit ms --log-x
|
||||||
|
|
||||||
|
# Generate CDF
|
||||||
|
python -m disk_checker scan_latency.csv --cdf
|
||||||
|
|
||||||
|
# View all metrics
|
||||||
|
python -m disk_checker scan_latency.csv --all-metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start the server
|
||||||
|
python -m disk_checker.scripts.serve --host 0.0.0.0 --port 8000
|
||||||
|
|
||||||
|
# Access API docs at http://localhost:8000/docs
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Endpoints
|
||||||
|
|
||||||
|
- `POST /api/v1/scan` - Upload CSV and get analysis
|
||||||
|
- `POST /api/v1/plot` - Generate and return plot image
|
||||||
|
- `POST /api/v1/benchmark` - Start a benchmark job
|
||||||
|
- `GET /api/v1/jobs/{id}` - Check job status
|
||||||
|
- `GET /health` - Health check
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
disk-checker/
|
||||||
|
├── src/disk_checker/
|
||||||
|
│ ├── core/ # Business logic (extractor, converter, helpers, orchestrator)
|
||||||
|
│ ├── plotting/ # Visualization (matplotlib)
|
||||||
|
│ ├── web/ # FastAPI backend
|
||||||
|
│ └── scripts/ # Entry points (cli, serve)
|
||||||
|
├── tests/ # Test suite
|
||||||
|
└── data/ # Sample data
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest
|
||||||
|
pytest --cov=disk_checker
|
||||||
|
pytest --cov=disk_checker --cov-report=html
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
nsec, read_slat_ns_count, read_slat_ns_cumulative, read_slat_ns_percentile, read_clat_ns_count, read_clat_ns_cumulative, read_clat_ns_percentile, read_lat_ns_count, read_lat_ns_cumulative, read_lat_ns_percentile, write_slat_ns_count, write_slat_ns_cumulative, write_slat_ns_percentile, write_clat_ns_count, write_clat_ns_cumulative, write_clat_ns_percentile, write_lat_ns_count, write_lat_ns_cumulative, write_lat_ns_percentile, trim_slat_ns_count, trim_slat_ns_cumulative, trim_slat_ns_percentile, trim_clat_ns_count, trim_clat_ns_cumulative, trim_clat_ns_percentile, trim_lat_ns_count, trim_lat_ns_cumulative, trim_lat_ns_percentile,
|
||||||
|
19791872, , , , 1, 1, 0.00016775708773695687, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
22413312, , , , 4, 5, 0.0008387854386847844, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
22675456, , , , 9, 14, 0.0023485992283173965, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
22937600, , , , 2, 16, 0.00268411340379131, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
23461888, , , , 2, 18, 0.003019627579265224, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
23724032, , , , 7, 25, 0.004193927193423922, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
23986176, , , , 14, 39, 0.006542526421741319, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
24248320, , , , 1, 40, 0.006710283509478276, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
24772608, , , , 2, 42, 0.007045797684952189, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
25034752, , , , 2, 44, 0.007381311860426103, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
25296896, , , , 4, 48, 0.00805234021137393, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
25559040, , , , 8, 56, 0.009394396913269586, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
25821184, , , , 9, 65, 0.010904210702902198, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
26083328, , , , 9, 74, 0.012414024492534809, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
26345472, , , , 16, 90, 0.01509813789632612, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
26607616, , , , 10, 100, 0.016775708773695688, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
26869760, , , , 34, 134, 0.022479449756752222, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
27131904, , , , 38, 172, 0.028854219090756583, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
27394048, , , , 49, 221, 0.037074316389867475, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
27656192, , , , 57, 278, 0.04663647039087401, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
27918336, , , , 90, 368, 0.061734608287200135, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
28180480, , , , 91, 459, 0.0770005032712632, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
28442624, , , , 105, 564, 0.09461499748364369, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
28704768, , , , 149, 713, 0.11961080355645026, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
28966912, , , , 152, 865, 0.1451098808924677, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
29229056, , , , 170, 1035, 0.17362858580775037, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
29491200, , , , 183, 1218, 0.20432813286361348, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
29753344, , , , 171, 1389, 0.23301459486663312, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
30015488, , , , 174, 1563, 0.2622043281328636, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
30277632, , , , 187, 1750, 0.2935749035396745, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
30539776, , , , 158, 1908, 0.32008052340211374, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
30801920, , , , 154, 2062, 0.3459151149136051, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
31064064, , , , 167, 2229, 0.3739305485656769, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
31326208, , , , 174, 2403, 0.4031202818319074, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
31588352, , , , 175, 2578, 0.43247777218587485, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
31850496, , , , 174, 2752, 0.46166750545210533, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
32112640, , , , 162, 2914, 0.4888441536654924, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
32374784, , , , 162, 3076, 0.5160208018788793, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
32636928, , , , 209, 3285, 0.5510820332159033, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
32899072, , , , 168, 3453, 0.5792652239557121, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
33161216, , , , 188, 3641, 0.6108035564502601, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
33423360, , , , 162, 3803, 0.637980204663647, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
33816576, , , , 324, 4127, 0.692333501090421, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
34340864, , , , 352, 4479, 0.7513839959738299, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
34865152, , , , 315, 4794, 0.8042274786109713, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
35389440, , , , 291, 5085, 0.8530447911424258, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
35913728, , , , 259, 5344, 0.8964938768662976, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
36438016, , , , 157, 5501, 0.9228317396409998, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
36962304, , , , 98, 5599, 0.9392719342392216, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
37486592, , , , 75, 5674, 0.9518537158194934, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
38010880, , , , 26, 5700, 0.9562154001006542, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
38535168, , , , 22, 5722, 0.9599060560308673, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
39059456, , , , 25, 5747, 0.9640999832242912, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
39583744, , , , 17, 5764, 0.9669518537158195, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
40108032, , , , 26, 5790, 0.9713135379969804, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
40632320, , , , 23, 5813, 0.9751719510149304, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
41156608, , , , 18, 5831, 0.9781915785941956, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
41680896, , , , 21, 5852, 0.9817144774366717, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
42205184, , , , 22, 5874, 0.9854051333668847, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
42729472, , , , 15, 5889, 0.9879214896829391, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
43253760, , , , 18, 5907, 0.9909411172622044, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
43778048, , , , 25, 5932, 0.9951350444556283, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
44302336, , , , 15, 5947, 0.9976514007716826, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
44826624, , , , 8, 5955, 0.9989934574735783, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
45350912, , , , 3, 5958, 0.9994967287367891, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
45875200, , , , 3, 5961, 1.0, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
@@ -0,0 +1,68 @@
|
|||||||
|
nsec, read_slat_ns_count, read_slat_ns_cumulative, read_slat_ns_percentile, read_clat_ns_count, read_clat_ns_cumulative, read_clat_ns_percentile, read_lat_ns_count, read_lat_ns_cumulative, read_lat_ns_percentile, write_slat_ns_count, write_slat_ns_cumulative, write_slat_ns_percentile, write_clat_ns_count, write_clat_ns_cumulative, write_clat_ns_percentile, write_lat_ns_count, write_lat_ns_cumulative, write_lat_ns_percentile, trim_slat_ns_count, trim_slat_ns_cumulative, trim_slat_ns_percentile, trim_clat_ns_count, trim_clat_ns_cumulative, trim_clat_ns_percentile, trim_lat_ns_count, trim_lat_ns_cumulative, trim_lat_ns_percentile,
|
||||||
|
22151168, , , , 3, 3, 0.00025161452654533256, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
22413312, , , , 4, 7, 0.0005871005619391093, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
22675456, , , , 8, 15, 0.0012580726327266628, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
22937600, , , , 5, 20, 0.0016774301769688837, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
23461888, , , , 2, 22, 0.001845173194665772, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
23724032, , , , 5, 27, 0.002264530738907993, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
23986176, , , , 7, 34, 0.0028516313008471023, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
25034752, , , , 3, 37, 0.0031032458273924346, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
25296896, , , , 1, 38, 0.003187117336240879, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
25559040, , , , 4, 42, 0.003522603371634656, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
25821184, , , , 2, 44, 0.003690346389331544, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
26083328, , , , 10, 54, 0.004529061477815986, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
26345472, , , , 15, 69, 0.005787134110542649, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
26607616, , , , 22, 91, 0.007632307305208421, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
26869760, , , , 19, 110, 0.00922586597332886, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
27131904, , , , 35, 145, 0.012161368783024406, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
27394048, , , , 60, 205, 0.017193659313931058, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
27656192, , , , 53, 258, 0.0216388492828986, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
27918336, , , , 78, 336, 0.028180826973077246, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
28180480, , , , 90, 426, 0.03572926276943722, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
28442624, , , , 128, 554, 0.046464815902038076, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
28704768, , , , 174, 728, 0.061058458441667365, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
28966912, , , , 209, 937, 0.0785876037909922, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
29229056, , , , 222, 1159, 0.09720707875534682, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
29491200, , , , 250, 1409, 0.11817495596745785, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
29753344, , , , 310, 1719, 0.14417512371047556, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
30015488, , , , 289, 2008, 0.1684139897676759, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
30277632, , , , 325, 2333, 0.19567223014342028, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
30539776, , , , 341, 2674, 0.22427241466073974, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
30801920, , , , 338, 3012, 0.2526209846515139, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
31064064, , , , 352, 3364, 0.28214375576616624, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
31326208, , , , 368, 3732, 0.3130084710223937, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
31588352, , , , 310, 4042, 0.3390086387654114, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
31850496, , , , 314, 4356, 0.3653442925438229, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
32112640, , , , 370, 4726, 0.3963767508177472, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
32374784, , , , 377, 5103, 0.4279963096536107, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
32636928, , , , 346, 5449, 0.45701585171517234, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
32899072, , , , 349, 5798, 0.48628700830327937, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
33161216, , , , 334, 6132, 0.5143000922586597, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
33423360, , , , 313, 6445, 0.5405518745282227, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
33816576, , , , 740, 7185, 0.6026167910760715, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
34340864, , , , 724, 7909, 0.663339763482345, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
34865152, , , , 653, 8562, 0.7181078587603791, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
35389440, , , , 666, 9228, 0.7739662836534429, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
35913728, , , , 584, 9812, 0.8229472448209343, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
36438016, , , , 499, 10311, 0.864799127736308, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
36962304, , , , 399, 10710, 0.8982638597668372, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
37486592, , , , 298, 11008, 0.9232575694036735, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
38010880, , , , 192, 11200, 0.9393608991025748, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
38535168, , , , 108, 11308, 0.9484190220582068, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
39059456, , , , 71, 11379, 0.9543738991864463, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
39583744, , , , 57, 11436, 0.9591545751908077, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
40108032, , , , 47, 11483, 0.9630965361066846, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
40632320, , , , 46, 11529, 0.966954625513713, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
41156608, , , , 42, 11571, 0.9704772288853476, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
41680896, , , , 38, 11609, 0.9736643462215885, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
42205184, , , , 38, 11647, 0.9768514635578294, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
42729472, , , , 47, 11694, 0.9807934244737063, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
43253760, , , , 37, 11731, 0.9838966703010987, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
43778048, , , , 51, 11782, 0.9881741172523694, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
44302336, , , , 40, 11822, 0.9915289776063071, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
44826624, , , , 41, 11863, 0.9949677094690933, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
45350912, , , , 29, 11892, 0.9973999832256982, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
45875200, , , , 16, 11908, 0.9987419273672733, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
46399488, , , , 10, 11918, 0.9995806424557577, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
46923776, , , , 4, 11922, 0.9999161284911515, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
73924608, , , , 1, 11923, 1.0, , , , , , , , , , , , , , , , , , , , , ,
|
||||||
|
@@ -0,0 +1,53 @@
|
|||||||
|
[build-system]
|
||||||
|
requires = ["hatchling"]
|
||||||
|
build-backend = "hatchling.build"
|
||||||
|
|
||||||
|
[project]
|
||||||
|
name = "disk-checker"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Disk benchmarking and latency visualization tool"
|
||||||
|
readme = "README.md"
|
||||||
|
requires-python = ">=3.10"
|
||||||
|
dependencies = [
|
||||||
|
"pandas>=2.0",
|
||||||
|
"numpy>=1.24",
|
||||||
|
"matplotlib>=3.7",
|
||||||
|
"fastapi>=0.100",
|
||||||
|
"uvicorn[standard]>=0.23",
|
||||||
|
"python-multipart>=0.0.6",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
dev = [
|
||||||
|
"pytest>=7.0",
|
||||||
|
"pytest-asyncio>=0.21",
|
||||||
|
"pytest-cov>=4.0",
|
||||||
|
"black>=23.0",
|
||||||
|
"ruff>=0.1.0",
|
||||||
|
"mypy>=1.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.scripts]
|
||||||
|
disk-checker = "disk_checker.scripts.cli:main"
|
||||||
|
disk-checker-serve = "disk_checker.scripts.serve:main"
|
||||||
|
|
||||||
|
[tool.hatch.build.targets.wheel]
|
||||||
|
packages = ["src/disk_checker"]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
testpaths = ["tests"]
|
||||||
|
python_files = ["test_*.py"]
|
||||||
|
python_functions = ["test_*"]
|
||||||
|
|
||||||
|
[tool.black]
|
||||||
|
line-length = 100
|
||||||
|
target-version = ["py310"]
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 100
|
||||||
|
target-version = "py310"
|
||||||
|
|
||||||
|
[tool.mypy]
|
||||||
|
python_version = "3.10"
|
||||||
|
warn_return_any = true
|
||||||
|
warn_unused_configs = true
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
CLI entry point for disk-checker.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python -m disk_checker scan_latency.csv
|
||||||
|
disk-checker scan_latency.csv --metric read_clat --unit ms
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from disk_checker.core.extractor import load_latency_csv
|
||||||
|
from disk_checker.core.helpers import percentile_table
|
||||||
|
from disk_checker.plotting.plots import generate_plot
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Plot fio json+ wide-format latency CSV."
|
||||||
|
)
|
||||||
|
parser.add_argument("csv", help="CSV from fio_jsonplus_clat2csv")
|
||||||
|
parser.add_argument("--metric", default=None,
|
||||||
|
help="Metric to plot (e.g. read_clat). "
|
||||||
|
"If omitted, the metric with the most I/Os is used.")
|
||||||
|
parser.add_argument("--all-metrics", action="store_true",
|
||||||
|
help="Plot every metric that has data")
|
||||||
|
parser.add_argument("--unit", choices=["ns", "us", "ms", "s"], default="us",
|
||||||
|
help="X-axis latency unit (default: us)")
|
||||||
|
parser.add_argument("--log-y", action="store_true")
|
||||||
|
parser.add_argument("--log-x", action="store_true")
|
||||||
|
parser.add_argument("--cdf", action="store_true",
|
||||||
|
help="Add a CDF subplot below the histogram")
|
||||||
|
parser.add_argument("--cdf-only", action="store_true")
|
||||||
|
parser.add_argument("--normalize", action="store_true",
|
||||||
|
help="Normalize histogram to a probability density")
|
||||||
|
parser.add_argument("--percentiles", default="50,90,99,99.9,99.99")
|
||||||
|
parser.add_argument("--no-percentiles", action="store_true")
|
||||||
|
parser.add_argument("--output", "-o", default=None)
|
||||||
|
parser.add_argument("--dpi", type=int, default=150)
|
||||||
|
parser.add_argument("--title", default="")
|
||||||
|
parser.add_argument("--show", action="store_true")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if not Path(args.csv).is_file():
|
||||||
|
sys.exit(f"error: file not found: {args.csv}")
|
||||||
|
|
||||||
|
df = load_latency_csv(args.csv)
|
||||||
|
if df.empty:
|
||||||
|
sys.exit("error: no latency data in CSV")
|
||||||
|
|
||||||
|
available = sorted(df["metric"].unique())
|
||||||
|
print("Metrics with data:", ", ".join(available))
|
||||||
|
|
||||||
|
if args.all_metrics:
|
||||||
|
metrics = available
|
||||||
|
elif args.metric:
|
||||||
|
if args.metric not in available:
|
||||||
|
sys.exit(f"error: metric {args.metric!r} not in {available}")
|
||||||
|
metrics = [args.metric]
|
||||||
|
else:
|
||||||
|
totals = df.groupby("metric")["count"].sum().sort_values(ascending=False)
|
||||||
|
metrics = [totals.index[0]]
|
||||||
|
print(f"Auto-selected metric: {metrics[0]}")
|
||||||
|
|
||||||
|
output = args.output or f"{Path(args.csv).stem}_latency.png"
|
||||||
|
generate_plot(
|
||||||
|
df=df,
|
||||||
|
metrics=metrics,
|
||||||
|
output_path=Path(output),
|
||||||
|
unit=args.unit,
|
||||||
|
log_y=args.log_y,
|
||||||
|
log_x=args.log_x,
|
||||||
|
normalize=args.normalize,
|
||||||
|
show_cdf=args.cdf,
|
||||||
|
cdf_only=args.cdf_only,
|
||||||
|
title=args.title,
|
||||||
|
dpi=args.dpi,
|
||||||
|
)
|
||||||
|
print(f"Saved plot to {output}")
|
||||||
|
|
||||||
|
if not args.no_percentiles:
|
||||||
|
try:
|
||||||
|
pcts = [float(x) for x in args.percentiles.split(",") if x.strip()]
|
||||||
|
except ValueError:
|
||||||
|
sys.exit("error: --percentiles must be comma-separated numbers")
|
||||||
|
for m in metrics:
|
||||||
|
print(percentile_table(df, m, args.unit, pcts))
|
||||||
|
|
||||||
|
if args.show:
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
plt.show()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
"""Startup script for the web server."""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import uvicorn
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main entry point for the web server."""
|
||||||
|
parser = argparse.ArgumentParser(description="Start Disk Checker API server")
|
||||||
|
parser.add_argument("--host", default="127.0.0.1", help="Host to bind to")
|
||||||
|
parser.add_argument("--port", type=int, default=8000, help="Port to bind to")
|
||||||
|
parser.add_argument("--workers", type=int, default=1, help="Number of workers")
|
||||||
|
parser.add_argument("--reload", action="store_true", help="Enable auto-reload")
|
||||||
|
parser.add_argument("--work-dir", type=Path, default=Path("."), help="Work directory")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# Set work directory in environment
|
||||||
|
import os
|
||||||
|
os.environ["DISK_CHECKER_WORK_DIR"] = str(args.work_dir)
|
||||||
|
|
||||||
|
uvicorn.run(
|
||||||
|
"disk_checker.web.server:app",
|
||||||
|
host=args.host,
|
||||||
|
port=args.port,
|
||||||
|
workers=args.workers,
|
||||||
|
reload=args.reload,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
"""Disk Checker - Disk benchmarking and latency visualization tool."""
|
||||||
|
|
||||||
|
__version__ = "0.1.0"
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
"""Allow running disk-checker as a module: python -m disk_checker"""
|
||||||
|
|
||||||
|
from disk_checker.scripts.cli import main
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
"""Core business logic module."""
|
||||||
|
|
||||||
|
from .extractor import load_latency_csv
|
||||||
|
from .converter import to_unit, UNIT_FACTORS, UNIT_LABELS
|
||||||
|
from .helpers import human_count, percentile_table
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"load_latency_csv",
|
||||||
|
"to_unit",
|
||||||
|
"UNIT_FACTORS",
|
||||||
|
"UNIT_LABELS",
|
||||||
|
"human_count",
|
||||||
|
"percentile_table",
|
||||||
|
]
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
"""Unit conversion utilities for latency data."""
|
||||||
|
|
||||||
|
from typing import Dict
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
UNIT_FACTORS: Dict[str, float] = {
|
||||||
|
"ns": 1.0,
|
||||||
|
"us": 1e3,
|
||||||
|
"ms": 1e6,
|
||||||
|
"s": 1e9,
|
||||||
|
}
|
||||||
|
|
||||||
|
UNIT_LABELS: Dict[str, str] = {
|
||||||
|
"ns": "ns",
|
||||||
|
"us": "µs",
|
||||||
|
"ms": "ms",
|
||||||
|
"s": "s",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def to_unit(values_ns, unit: str) -> np.ndarray:
|
||||||
|
"""
|
||||||
|
Convert nanosecond values to the specified unit.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
values_ns: Values in nanoseconds
|
||||||
|
unit: Target unit (ns, us, ms, s)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Converted values as numpy array
|
||||||
|
"""
|
||||||
|
return np.asarray(values_ns, dtype=float) / UNIT_FACTORS[unit]
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
"""CSV data extraction module for fio latency data."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
import pandas as pd
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
|
||||||
|
def load_latency_csv(path: str) -> pd.DataFrame:
|
||||||
|
"""
|
||||||
|
Parse wide-format fio JSON+ CSV into long-format DataFrame.
|
||||||
|
|
||||||
|
Transforms wide-format columns (e.g., read_clat_ns_count) into rows with
|
||||||
|
a metric identifier, making it easier to group and analyze by operation type.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
path: Path to CSV file generated by fio_jsonplus_clat2csv
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
DataFrame with columns: metric, latency_ns, count, cumulative, percentile
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If CSV format is invalid or contains no valid data
|
||||||
|
"""
|
||||||
|
df = pd.read_csv(path, skipinitialspace=True)
|
||||||
|
df.columns = [str(c).strip() for c in df.columns]
|
||||||
|
|
||||||
|
# Drop trailing unnamed / empty columns
|
||||||
|
df = df.loc[:, ~df.columns.str.match(r"^Unnamed")]
|
||||||
|
df = df.loc[:, df.columns != ""]
|
||||||
|
|
||||||
|
nsec_col = df.columns[0]
|
||||||
|
if "nsec" not in nsec_col.lower():
|
||||||
|
raise ValueError(
|
||||||
|
f"expected first column to be 'nsec', got {nsec_col!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Coerce the nsec column to numeric
|
||||||
|
df[nsec_col] = pd.to_numeric(df[nsec_col], errors="coerce")
|
||||||
|
df = df.dropna(subset=[nsec_col])
|
||||||
|
|
||||||
|
# Discover metric triples: <metric>_count / _cumulative / _percentile
|
||||||
|
metrics = {}
|
||||||
|
for col in df.columns:
|
||||||
|
if col == nsec_col:
|
||||||
|
continue
|
||||||
|
for suffix in ("_count", "_cumulative", "_percentile"):
|
||||||
|
if col.endswith(suffix):
|
||||||
|
base = col[: -len(suffix)]
|
||||||
|
metrics.setdefault(base, {})[suffix[1:]] = col
|
||||||
|
|
||||||
|
if not metrics:
|
||||||
|
raise ValueError("no *_count columns found in CSV")
|
||||||
|
|
||||||
|
frames = []
|
||||||
|
for base, cols in metrics.items():
|
||||||
|
count_col = cols["count"]
|
||||||
|
|
||||||
|
# Coerce count to numeric
|
||||||
|
counts = pd.to_numeric(df[count_col], errors="coerce")
|
||||||
|
|
||||||
|
# Keep only rows where the count column actually has a number
|
||||||
|
mask = counts.notna()
|
||||||
|
if not mask.any():
|
||||||
|
continue
|
||||||
|
|
||||||
|
sub = pd.DataFrame({
|
||||||
|
"latency_ns": df.loc[mask, nsec_col].values,
|
||||||
|
"count": counts[mask].astype(int).values,
|
||||||
|
})
|
||||||
|
if "cumulative" in cols:
|
||||||
|
sub["cumulative"] = pd.to_numeric(
|
||||||
|
df.loc[mask, cols["cumulative"]], errors="coerce"
|
||||||
|
).values
|
||||||
|
if "percentile" in cols:
|
||||||
|
sub["percentile"] = pd.to_numeric(
|
||||||
|
df.loc[mask, cols["percentile"]], errors="coerce"
|
||||||
|
).values
|
||||||
|
|
||||||
|
# read_clat_ns -> read_clat
|
||||||
|
sub["metric"] = base.replace("_ns", "")
|
||||||
|
frames.append(sub)
|
||||||
|
|
||||||
|
if not frames:
|
||||||
|
raise ValueError("no metric had any numeric count data")
|
||||||
|
|
||||||
|
out = pd.concat(frames, ignore_index=True)
|
||||||
|
out = out[out["count"] > 0].reset_index(drop=True)
|
||||||
|
return out
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
"""Helper functions for formatting and display."""
|
||||||
|
|
||||||
|
from typing import List
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from matplotlib.ticker import FuncFormatter
|
||||||
|
|
||||||
|
from .converter import to_unit, UNIT_LABELS
|
||||||
|
|
||||||
|
|
||||||
|
def human_count(x, _pos=None) -> str:
|
||||||
|
"""
|
||||||
|
Format large numbers with K/M/G suffixes for axis labels.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
x: Numeric value to format
|
||||||
|
_pos: Matplotlib tick position (unused)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted string (e.g., "1.5M", "3.2G")
|
||||||
|
"""
|
||||||
|
if x >= 1e9:
|
||||||
|
return f"{x/1e9:.1f}G"
|
||||||
|
if x >= 1e6:
|
||||||
|
return f"{x/1e6:.1f}M"
|
||||||
|
if x >= 1e3:
|
||||||
|
return f"{x/1e3:.0f}k"
|
||||||
|
return f"{x:.0f}"
|
||||||
|
|
||||||
|
|
||||||
|
def percentile_table(df: pd.DataFrame, metric: str, unit: str, pcts: List[float]) -> str:
|
||||||
|
"""
|
||||||
|
Generate a formatted table showing latency values at specified percentiles.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
df: Long-format DataFrame from load_latency_csv
|
||||||
|
metric: Metric name (e.g., 'read_clat')
|
||||||
|
unit: Display unit (ns, us, ms, s)
|
||||||
|
pcts: List of percentile values (e.g., [50, 90, 99])
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted string table
|
||||||
|
"""
|
||||||
|
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||||
|
if g.empty:
|
||||||
|
return f"(no data for metric {metric})"
|
||||||
|
|
||||||
|
counts = g["count"].values.astype(float)
|
||||||
|
total = int(counts.sum())
|
||||||
|
cdf = np.cumsum(counts) / total * 100.0
|
||||||
|
lat = to_unit(g["latency_ns"].values, unit)
|
||||||
|
|
||||||
|
lines = [
|
||||||
|
f"\nMetric: {metric} (total I/Os: {total:,})",
|
||||||
|
f" {'percentile':>12} {'latency':>14}",
|
||||||
|
]
|
||||||
|
|
||||||
|
for p in pcts:
|
||||||
|
idx = min(np.searchsorted(cdf, p), len(lat) - 1)
|
||||||
|
lines.append(f" {p:>11.2f}% {lat[idx]:>12.3f} {UNIT_LABELS[unit]}")
|
||||||
|
|
||||||
|
return "\n".join(lines)
|
||||||
@@ -0,0 +1,164 @@
|
|||||||
|
"""Tool orchestration module for running disk benchmarks."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import subprocess
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from enum import Enum
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, Optional
|
||||||
|
|
||||||
|
|
||||||
|
class ToolStatus(Enum):
|
||||||
|
"""Status of a benchmark job."""
|
||||||
|
PENDING = "pending"
|
||||||
|
RUNNING = "running"
|
||||||
|
COMPLETED = "completed"
|
||||||
|
FAILED = "failed"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class JobResult:
|
||||||
|
"""Result of a benchmark job."""
|
||||||
|
job_id: str
|
||||||
|
status: ToolStatus
|
||||||
|
output_path: Optional[Path] = None
|
||||||
|
error: Optional[str] = None
|
||||||
|
metadata: Dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class Orchestrator:
|
||||||
|
"""
|
||||||
|
Orchestrates disk benchmarking tools (fio, badblocks, etc.)
|
||||||
|
|
||||||
|
Provides async interfaces for running benchmarks and collecting results.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, work_dir: Path = Path(".")):
|
||||||
|
self.work_dir = work_dir
|
||||||
|
self._jobs: Dict[str, JobResult] = {}
|
||||||
|
|
||||||
|
async def run_fio(
|
||||||
|
self,
|
||||||
|
device: str,
|
||||||
|
job_file: Optional[Path] = None,
|
||||||
|
output_format: str = "jsonplus",
|
||||||
|
**kwargs,
|
||||||
|
) -> Path:
|
||||||
|
"""
|
||||||
|
Run fio benchmark and return path to output file.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
device: Block device path (e.g., '/dev/sda')
|
||||||
|
job_file: Optional fio job file
|
||||||
|
output_format: Output format (json, jsonplus, csv)
|
||||||
|
**kwargs: Additional fio options
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Path to output file
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
subprocess.CalledProcessError: If fio command fails
|
||||||
|
"""
|
||||||
|
job_id = f"fio_{asyncio.get_event_loop().time()}"
|
||||||
|
output_path = self.work_dir / f"{job_id}.{output_format}"
|
||||||
|
|
||||||
|
cmd = ["fio", "--output-format=jsonplus"]
|
||||||
|
if job_file:
|
||||||
|
cmd.extend(["--filename", str(device)])
|
||||||
|
else:
|
||||||
|
cmd.extend(["--name=test", "--rw=randread", "--bs=4k"])
|
||||||
|
|
||||||
|
cmd.extend(["--output", str(output_path)])
|
||||||
|
cmd.extend([f"--output-format={output_format}"])
|
||||||
|
|
||||||
|
for k, v in kwargs.items():
|
||||||
|
cmd.extend([f"--{k}={v}"])
|
||||||
|
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.RUNNING,
|
||||||
|
output_path=output_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
await asyncio.create_subprocess_exec(
|
||||||
|
*cmd,
|
||||||
|
stdout=asyncio.subprocess.PIPE,
|
||||||
|
stderr=asyncio.subprocess.PIPE,
|
||||||
|
)
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.COMPLETED,
|
||||||
|
output_path=output_path,
|
||||||
|
)
|
||||||
|
return output_path
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.FAILED,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def run_badblocks(
|
||||||
|
self,
|
||||||
|
device: str,
|
||||||
|
blocks_count: Optional[int] = None,
|
||||||
|
**kwargs,
|
||||||
|
) -> Path:
|
||||||
|
"""
|
||||||
|
Run badblocks scan on device.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
device: Block device path
|
||||||
|
blocks_count: Number of blocks to scan
|
||||||
|
**kwargs: Additional badblocks options
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Path to badblocks output file
|
||||||
|
"""
|
||||||
|
job_id = f"badblocks_{asyncio.get_event_loop().time()}"
|
||||||
|
output_path = self.work_dir / f"{job_id}.txt"
|
||||||
|
|
||||||
|
cmd = ["badblocks", "-w", "-s", "-p", "0", "-t", "RANDOM", "-o"]
|
||||||
|
cmd.extend([str(output_path), str(device)])
|
||||||
|
|
||||||
|
if blocks_count:
|
||||||
|
cmd.extend(["-b", str(blocks_count)])
|
||||||
|
|
||||||
|
for k, v in kwargs.items():
|
||||||
|
cmd.extend([f"-{k}", str(v)])
|
||||||
|
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.RUNNING,
|
||||||
|
output_path=output_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
await asyncio.create_subprocess_exec(
|
||||||
|
*cmd,
|
||||||
|
stdout=asyncio.subprocess.PIPE,
|
||||||
|
stderr=asyncio.subprocess.PIPE,
|
||||||
|
)
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.COMPLETED,
|
||||||
|
output_path=output_path,
|
||||||
|
)
|
||||||
|
return output_path
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.FAILED,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def get_job_status(self, job_id: str) -> Optional[JobResult]:
|
||||||
|
"""Get status of a submitted job."""
|
||||||
|
return self._jobs.get(job_id)
|
||||||
|
|
||||||
|
def list_jobs(self) -> Dict[str, JobResult]:
|
||||||
|
"""List all jobs."""
|
||||||
|
return self._jobs.copy()
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
"""Plotting module for disk latency visualization."""
|
||||||
|
|
||||||
|
from .plots import draw_histogram, draw_cdf, generate_plot
|
||||||
|
|
||||||
|
__all__ = ["draw_histogram", "draw_cdf", "generate_plot"]
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
"""Plotting module for disk latency visualization."""
|
||||||
|
|
||||||
|
import matplotlib
|
||||||
|
matplotlib.use("Agg") # Non-interactive backend for server environments
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
from matplotlib.ticker import FuncFormatter
|
||||||
|
import numpy as np
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Optional
|
||||||
|
|
||||||
|
from disk_checker.core.converter import to_unit, UNIT_LABELS
|
||||||
|
from disk_checker.core.helpers import human_count
|
||||||
|
|
||||||
|
|
||||||
|
def draw_histogram(
|
||||||
|
ax,
|
||||||
|
df,
|
||||||
|
metrics: List[str],
|
||||||
|
unit: str = "us",
|
||||||
|
log_y: bool = False,
|
||||||
|
log_x: bool = False,
|
||||||
|
normalize: bool = False,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
Draw a histogram showing the distribution of latencies for each metric.
|
||||||
|
|
||||||
|
Uses a step plot (where="post") to show the discrete nature of the data.
|
||||||
|
If normalize=True, Y-axis shows probability density instead of counts.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
ax: Matplotlib axis object
|
||||||
|
df: Long-format DataFrame from load_latency_csv
|
||||||
|
metrics: List of metric names to plot
|
||||||
|
unit: X-axis unit (ns, us, ms, s)
|
||||||
|
log_y: Use logarithmic Y scale
|
||||||
|
log_x: Use logarithmic X scale
|
||||||
|
normalize: Normalize to probability density
|
||||||
|
"""
|
||||||
|
for metric in metrics:
|
||||||
|
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||||
|
if g.empty:
|
||||||
|
continue
|
||||||
|
|
||||||
|
x = to_unit(g["latency_ns"].values, unit)
|
||||||
|
y = g["count"].values.astype(float)
|
||||||
|
|
||||||
|
if normalize:
|
||||||
|
s = y.sum()
|
||||||
|
if s > 0:
|
||||||
|
y = y / s
|
||||||
|
|
||||||
|
ax.step(x, y, where="post", lw=1.4, label=metric)
|
||||||
|
ax.fill_between(x, y, step="post", alpha=0.25)
|
||||||
|
|
||||||
|
ax.set_xlabel(f"Latency ({UNIT_LABELS[unit]})")
|
||||||
|
ax.set_ylabel("Probability" if normalize else "Number of I/Os")
|
||||||
|
ax.set_title("Completion latency distribution")
|
||||||
|
ax.grid(True, which="both", ls=":", alpha=0.5)
|
||||||
|
|
||||||
|
if not normalize:
|
||||||
|
ax.yaxis.set_major_formatter(FuncFormatter(human_count))
|
||||||
|
if log_y:
|
||||||
|
ax.set_yscale("log")
|
||||||
|
if log_x:
|
||||||
|
ax.set_xscale("log")
|
||||||
|
if len(metrics) > 1:
|
||||||
|
ax.legend(title="metric", fontsize=9)
|
||||||
|
|
||||||
|
|
||||||
|
def draw_cdf(ax, df, metrics: List[str], unit: str = "us", log_x: bool = False) -> None:
|
||||||
|
"""
|
||||||
|
Draw a Cumulative Distribution Function (CDF).
|
||||||
|
|
||||||
|
Shows the percentage of I/Os completed within each latency threshold.
|
||||||
|
Each metric gets its own line; Y-axis ranges from 0% to 100%.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
ax: Matplotlib axis object
|
||||||
|
df: Long-format DataFrame
|
||||||
|
metrics: List of metric names
|
||||||
|
unit: X-axis unit
|
||||||
|
log_x: Use logarithmic X scale
|
||||||
|
"""
|
||||||
|
for metric in metrics:
|
||||||
|
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||||
|
if g.empty:
|
||||||
|
continue
|
||||||
|
|
||||||
|
x = to_unit(g["latency_ns"].values, unit)
|
||||||
|
counts = g["count"].values.astype(float)
|
||||||
|
total = counts.sum()
|
||||||
|
if total <= 0:
|
||||||
|
continue
|
||||||
|
|
||||||
|
cdf = np.cumsum(counts) / total * 100.0
|
||||||
|
ax.plot(x, cdf, lw=1.6, label=f"{metric} (n={int(total):,})")
|
||||||
|
|
||||||
|
ax.set_xlabel(f"Latency ({UNIT_LABELS[unit]})")
|
||||||
|
ax.set_ylabel("Cumulative % of I/Os")
|
||||||
|
ax.set_title("Cumulative latency distribution")
|
||||||
|
ax.set_ylim(0, 100)
|
||||||
|
ax.grid(True, which="both", ls=":", alpha=0.5)
|
||||||
|
ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:.0f}%"))
|
||||||
|
|
||||||
|
if log_x:
|
||||||
|
ax.set_xscale("log")
|
||||||
|
ax.legend(fontsize=9)
|
||||||
|
|
||||||
|
|
||||||
|
def generate_plot(
|
||||||
|
df,
|
||||||
|
metrics: List[str],
|
||||||
|
output_path: Path,
|
||||||
|
unit: str = "us",
|
||||||
|
log_y: bool = False,
|
||||||
|
log_x: bool = False,
|
||||||
|
normalize: bool = False,
|
||||||
|
show_cdf: bool = False,
|
||||||
|
cdf_only: bool = False,
|
||||||
|
title: str = "",
|
||||||
|
dpi: int = 150,
|
||||||
|
) -> Path:
|
||||||
|
"""
|
||||||
|
Generate a complete plot with histogram and/or CDF.
|
||||||
|
|
||||||
|
Creates the appropriate subplot layout based on the options provided.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
df: Long-format DataFrame
|
||||||
|
metrics: Metrics to plot
|
||||||
|
output_path: Where to save the plot
|
||||||
|
unit: X-axis unit
|
||||||
|
log_y: Logarithmic Y scale
|
||||||
|
log_x: Logarithmic X scale
|
||||||
|
normalize: Normalize histogram
|
||||||
|
show_cdf: Show CDF below histogram
|
||||||
|
cdf_only: Show only CDF
|
||||||
|
title: Plot title
|
||||||
|
dpi: Output resolution
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Path to saved plot
|
||||||
|
"""
|
||||||
|
if cdf_only:
|
||||||
|
fig, ax = plt.subplots(figsize=(11, 6))
|
||||||
|
draw_cdf(ax, df, metrics, unit, log_x)
|
||||||
|
elif show_cdf:
|
||||||
|
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(11, 10))
|
||||||
|
draw_histogram(
|
||||||
|
ax1, df, metrics, unit, log_y, log_x, normalize
|
||||||
|
)
|
||||||
|
draw_cdf(ax2, df, metrics, unit, log_x)
|
||||||
|
else:
|
||||||
|
fig, ax = plt.subplots(figsize=(11, 6))
|
||||||
|
draw_histogram(ax, df, metrics, unit, log_y, log_x, normalize)
|
||||||
|
|
||||||
|
if title:
|
||||||
|
fig.suptitle(title, fontsize=13, y=0.995)
|
||||||
|
fig.tight_layout()
|
||||||
|
fig.savefig(output_path, dpi=dpi, bbox_inches="tight")
|
||||||
|
plt.close(fig)
|
||||||
|
return output_path
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Web backend module for disk-checker API."""
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
"""Pydantic models for the web API."""
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from typing import List, Optional, Dict, Any
|
||||||
|
from enum import Enum
|
||||||
|
|
||||||
|
|
||||||
|
class ToolType(str, Enum):
|
||||||
|
"""Available benchmarking tools."""
|
||||||
|
FIO = "fio"
|
||||||
|
BADBLOCKS = "badblocks"
|
||||||
|
IO_URING = "io_uring"
|
||||||
|
|
||||||
|
|
||||||
|
class JobStatus(str, Enum):
|
||||||
|
"""Status of a benchmark job."""
|
||||||
|
PENDING = "pending"
|
||||||
|
RUNNING = "running"
|
||||||
|
COMPLETED = "completed"
|
||||||
|
FAILED = "failed"
|
||||||
|
|
||||||
|
|
||||||
|
class MetricInfo(BaseModel):
|
||||||
|
"""Information about a metric."""
|
||||||
|
name: str
|
||||||
|
total_ios: int
|
||||||
|
min_latency_ns: Optional[float] = None
|
||||||
|
max_latency_ns: Optional[float] = None
|
||||||
|
avg_latency_ns: Optional[float] = None
|
||||||
|
|
||||||
|
|
||||||
|
class PercentileInfo(BaseModel):
|
||||||
|
"""Latency value at a specific percentile."""
|
||||||
|
percentile: float
|
||||||
|
latency_ns: float
|
||||||
|
latency_display: str
|
||||||
|
|
||||||
|
|
||||||
|
class MetricResult(BaseModel):
|
||||||
|
"""Complete result for a metric."""
|
||||||
|
metric: str
|
||||||
|
total_ios: int
|
||||||
|
percentiles: List[PercentileInfo]
|
||||||
|
min_latency_ns: float
|
||||||
|
max_latency_ns: float
|
||||||
|
avg_latency_ns: float
|
||||||
|
|
||||||
|
|
||||||
|
class ScanResponse(BaseModel):
|
||||||
|
"""Response from CSV scan endpoint."""
|
||||||
|
metrics: List[MetricResult]
|
||||||
|
summary: Dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class BenchmarkRequest(BaseModel):
|
||||||
|
"""Request to start a benchmark job."""
|
||||||
|
device: str = Field(..., description="Block device path (e.g., /dev/sda)")
|
||||||
|
tool: ToolType = ToolType.FIO
|
||||||
|
job_file: Optional[str] = None
|
||||||
|
options: Dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
class Config:
|
||||||
|
json_schema_extra = {
|
||||||
|
"example": {
|
||||||
|
"device": "/dev/sda",
|
||||||
|
"tool": "fio",
|
||||||
|
"options": {
|
||||||
|
"rw": "randread",
|
||||||
|
"bs": "4k",
|
||||||
|
"size": "1G",
|
||||||
|
"runtime": "60"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class JobStatusResponse(BaseModel):
|
||||||
|
"""Response with job status."""
|
||||||
|
job_id: str
|
||||||
|
status: JobStatus
|
||||||
|
output_path: Optional[str] = None
|
||||||
|
error: Optional[str] = None
|
||||||
|
created_at: Optional[str] = None
|
||||||
|
completed_at: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
class PlotRequest(BaseModel):
|
||||||
|
"""Request to generate a plot."""
|
||||||
|
metric: Optional[str] = None
|
||||||
|
unit: str = Field(default="us", pattern="^(ns|us|ms|s)$")
|
||||||
|
log_x: bool = False
|
||||||
|
log_y: bool = False
|
||||||
|
cdf: bool = False
|
||||||
|
normalize: bool = False
|
||||||
|
title: Optional[str] = None
|
||||||
@@ -0,0 +1,209 @@
|
|||||||
|
"""API routes for disk-checker web backend."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Optional
|
||||||
|
|
||||||
|
from fastapi import APIRouter, UploadFile, File, HTTPException, Depends, Request
|
||||||
|
from fastapi.responses import JSONResponse, FileResponse
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from disk_checker.core.extractor import load_latency_csv
|
||||||
|
from disk_checker.core.helpers import percentile_table
|
||||||
|
from disk_checker.plotting.plots import generate_plot
|
||||||
|
from disk_checker.web.models import (
|
||||||
|
ScanResponse,
|
||||||
|
BenchmarkRequest,
|
||||||
|
JobStatusResponse,
|
||||||
|
PlotRequest,
|
||||||
|
MetricInfo,
|
||||||
|
PercentileInfo,
|
||||||
|
MetricResult,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
||||||
|
def _calculate_metric_results(df: pd.DataFrame, metric: str, pcts: List[float]) -> MetricResult:
|
||||||
|
"""Calculate detailed results for a metric."""
|
||||||
|
metric_df = df[df["metric"] == metric]
|
||||||
|
counts = metric_df["count"].values.astype(float)
|
||||||
|
total_ios = int(counts.sum())
|
||||||
|
|
||||||
|
latencies_ns = metric_df["latency_ns"].values
|
||||||
|
|
||||||
|
# Calculate percentiles
|
||||||
|
percentiles = []
|
||||||
|
cdf = __import__('numpy').cumsum(counts) / total_ios * 100.0
|
||||||
|
for p in pcts:
|
||||||
|
idx = min(__import__('numpy').searchsorted(cdf, p), len(latencies_ns) - 1)
|
||||||
|
percentiles.append(PercentileInfo(
|
||||||
|
percentile=p,
|
||||||
|
latency_ns=float(latencies_ns[idx]),
|
||||||
|
latency_display=f"{latencies_ns[idx]:.3f} ns"
|
||||||
|
))
|
||||||
|
|
||||||
|
return MetricResult(
|
||||||
|
metric=metric,
|
||||||
|
total_ios=total_ios,
|
||||||
|
percentiles=percentiles,
|
||||||
|
min_latency_ns=float(latencies_ns.min()),
|
||||||
|
max_latency_ns=float(latencies_ns.max()),
|
||||||
|
avg_latency_ns=float(latencies_ns.mean()),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/scan", response_model=ScanResponse)
|
||||||
|
async def scan_csv(file: UploadFile = File(...)):
|
||||||
|
"""
|
||||||
|
Upload a fio JSON+ CSV and return analysis.
|
||||||
|
"""
|
||||||
|
with tempfile.NamedTemporaryFile(delete=False, suffix=".csv") as tmp:
|
||||||
|
content = await file.read()
|
||||||
|
tmp.write(content)
|
||||||
|
tmp_path = tmp.name
|
||||||
|
|
||||||
|
try:
|
||||||
|
df = load_latency_csv(tmp_path)
|
||||||
|
if df.empty:
|
||||||
|
raise HTTPException(status_code=400, detail="No valid data in CSV")
|
||||||
|
|
||||||
|
# Calculate metrics
|
||||||
|
metrics = []
|
||||||
|
pcts = [50, 90, 99, 99.9, 99.99]
|
||||||
|
for metric in sorted(df["metric"].unique()):
|
||||||
|
metrics.append(_calculate_metric_results(df, metric, pcts))
|
||||||
|
|
||||||
|
summary = {
|
||||||
|
"total_metrics": len(metrics),
|
||||||
|
"total_ios": sum(m.total_ios for m in metrics),
|
||||||
|
}
|
||||||
|
|
||||||
|
return ScanResponse(metrics=metrics, summary=summary)
|
||||||
|
finally:
|
||||||
|
Path(tmp_path).unlink(missing_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/plot")
|
||||||
|
async def generate_plot_endpoint(
|
||||||
|
file: UploadFile = File(...),
|
||||||
|
metric: Optional[str] = None,
|
||||||
|
unit: str = "us",
|
||||||
|
log_x: bool = False,
|
||||||
|
log_y: bool = False,
|
||||||
|
cdf: bool = False,
|
||||||
|
normalize: bool = False,
|
||||||
|
title: Optional[str] = None,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Generate and return a latency plot.
|
||||||
|
"""
|
||||||
|
with tempfile.NamedTemporaryFile(delete=False, suffix=".csv") as tmp:
|
||||||
|
content = await file.read()
|
||||||
|
tmp.write(content)
|
||||||
|
tmp_path = tmp.name
|
||||||
|
|
||||||
|
try:
|
||||||
|
df = load_latency_csv(tmp_path)
|
||||||
|
if df.empty:
|
||||||
|
raise HTTPException(status_code=400, detail="No valid data in CSV")
|
||||||
|
|
||||||
|
# Determine metrics
|
||||||
|
available = sorted(df["metric"].unique())
|
||||||
|
if request.metric:
|
||||||
|
if request.metric not in available:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"Metric {request.metric} not found"
|
||||||
|
)
|
||||||
|
metrics = [request.metric]
|
||||||
|
else:
|
||||||
|
totals = df.groupby("metric")["count"].sum().sort_values(ascending=False)
|
||||||
|
metrics = [totals.index[0]]
|
||||||
|
|
||||||
|
# Generate plot
|
||||||
|
with tempfile.NamedTemporaryFile(delete=False, suffix=".png") as img:
|
||||||
|
output_path = Path(img.name)
|
||||||
|
|
||||||
|
generate_plot(
|
||||||
|
df=df,
|
||||||
|
metrics=metrics,
|
||||||
|
output_path=output_path,
|
||||||
|
unit=unit,
|
||||||
|
log_x=log_x,
|
||||||
|
log_y=log_y,
|
||||||
|
normalize=normalize,
|
||||||
|
show_cdf=cdf,
|
||||||
|
title=title or "",
|
||||||
|
)
|
||||||
|
|
||||||
|
return FileResponse(output_path, media_type="image/png")
|
||||||
|
finally:
|
||||||
|
Path(tmp_path).unlink(missing_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/benchmark", response_model=JobStatusResponse)
|
||||||
|
async def start_benchmark(request: BenchmarkRequest):
|
||||||
|
"""
|
||||||
|
Start a disk benchmark job.
|
||||||
|
"""
|
||||||
|
orchestrator = request.app.state.orchestrator
|
||||||
|
|
||||||
|
job_id = f"bench_{asyncio.get_event_loop().time()}"
|
||||||
|
|
||||||
|
async def run_job():
|
||||||
|
try:
|
||||||
|
if request.tool == "fio":
|
||||||
|
output_path = await orchestrator.run_fio(
|
||||||
|
device=request.device,
|
||||||
|
**request.options,
|
||||||
|
)
|
||||||
|
elif request.tool == "badblocks":
|
||||||
|
output_path = await orchestrator.run_badblocks(
|
||||||
|
device=request.device,
|
||||||
|
**request.options,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown tool: {request.tool}")
|
||||||
|
return {
|
||||||
|
"job_id": job_id,
|
||||||
|
"status": "completed",
|
||||||
|
"output_path": str(output_path),
|
||||||
|
}
|
||||||
|
except Exception as e:
|
||||||
|
return {"job_id": job_id, "status": "failed", "error": str(e)}
|
||||||
|
|
||||||
|
request.app.state.jobs[job_id] = {
|
||||||
|
"status": "running",
|
||||||
|
"task": asyncio.create_task(run_job()),
|
||||||
|
}
|
||||||
|
|
||||||
|
return JobStatusResponse(job_id=job_id, status="running")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/jobs/{job_id}", response_model=JobStatusResponse)
|
||||||
|
async def get_job_status(job_id: str, request: Request):
|
||||||
|
"""
|
||||||
|
Check the status of a benchmark job.
|
||||||
|
"""
|
||||||
|
jobs = request.app.state.jobs
|
||||||
|
if job_id not in jobs:
|
||||||
|
raise HTTPException(status_code=404, detail="Job not found")
|
||||||
|
|
||||||
|
job = jobs[job_id]
|
||||||
|
task = job["task"]
|
||||||
|
|
||||||
|
if task.done():
|
||||||
|
result = task.result()
|
||||||
|
return JobStatusResponse(**result)
|
||||||
|
|
||||||
|
return JobStatusResponse(job_id=job_id, status="running")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/jobs", response_model=List[str])
|
||||||
|
async def list_jobs(request: Request):
|
||||||
|
"""
|
||||||
|
List all benchmark jobs.
|
||||||
|
"""
|
||||||
|
return list(request.app.state.jobs.keys())
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
"""FastAPI application for disk-checker web API."""
|
||||||
|
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from disk_checker.core.orchestrator import Orchestrator
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI):
|
||||||
|
"""Application lifespan handler."""
|
||||||
|
# Startup: initialize orchestrator
|
||||||
|
app.state.orchestrator = Orchestrator(work_dir=app.state.work_dir)
|
||||||
|
app.state.jobs = {}
|
||||||
|
logger.info("Disk Checker API started")
|
||||||
|
yield
|
||||||
|
# Shutdown: cleanup
|
||||||
|
logger.info("Disk Checker API stopped")
|
||||||
|
|
||||||
|
|
||||||
|
app = FastAPI(
|
||||||
|
title="Disk Checker API",
|
||||||
|
description="API for disk benchmarking and latency visualization",
|
||||||
|
version="0.1.0",
|
||||||
|
lifespan=lifespan,
|
||||||
|
)
|
||||||
|
|
||||||
|
app.add_middleware(
|
||||||
|
CORSMiddleware,
|
||||||
|
allow_origins=["*"],
|
||||||
|
allow_credentials=True,
|
||||||
|
allow_methods=["*"],
|
||||||
|
allow_headers=["*"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
async def health_check():
|
||||||
|
"""Health check endpoint."""
|
||||||
|
return {"status": "ok", "version": "0.1.0"}
|
||||||
|
|
||||||
|
|
||||||
|
# Import and include routers
|
||||||
|
from disk_checker.web.routes import router as api_router
|
||||||
|
app.include_router(api_router, prefix="/api/v1")
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Tests for disk-checker."""
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Tests for the unit conversion module."""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from disk_checker.core.converter import to_unit, UNIT_FACTORS, UNIT_LABELS
|
||||||
|
|
||||||
|
|
||||||
|
class TestToUnit:
|
||||||
|
"""Tests for to_unit function."""
|
||||||
|
|
||||||
|
def test_nanoseconds_to_microseconds(self):
|
||||||
|
"""Test conversion from ns to us."""
|
||||||
|
values = np.array([1000, 2000, 3000])
|
||||||
|
result = to_unit(values, "us")
|
||||||
|
expected = np.array([1.0, 2.0, 3.0])
|
||||||
|
np.testing.assert_array_equal(result, expected)
|
||||||
|
|
||||||
|
def test_nanoseconds_to_milliseconds(self):
|
||||||
|
"""Test conversion from ns to ms."""
|
||||||
|
values = np.array([1000000, 2000000])
|
||||||
|
result = to_unit(values, "ms")
|
||||||
|
expected = np.array([1.0, 2.0])
|
||||||
|
np.testing.assert_array_equal(result, expected)
|
||||||
|
|
||||||
|
def test_nanoseconds_to_seconds(self):
|
||||||
|
"""Test conversion from ns to s."""
|
||||||
|
values = np.array([1000000000, 2000000000])
|
||||||
|
result = to_unit(values, "s")
|
||||||
|
expected = np.array([1.0, 2.0])
|
||||||
|
np.testing.assert_array_equal(result, expected)
|
||||||
|
|
||||||
|
def test_same_unit_returns_same(self):
|
||||||
|
"""Test conversion to same unit."""
|
||||||
|
values = np.array([100, 200, 300])
|
||||||
|
result = to_unit(values, "ns")
|
||||||
|
np.testing.assert_array_equal(result, values)
|
||||||
|
|
||||||
|
def test_empty_array(self):
|
||||||
|
"""Test conversion with empty array."""
|
||||||
|
values = np.array([])
|
||||||
|
result = to_unit(values, "us")
|
||||||
|
assert len(result) == 0
|
||||||
|
|
||||||
|
def test_single_value(self):
|
||||||
|
"""Test conversion with single value."""
|
||||||
|
result = to_unit(1000, "us")
|
||||||
|
assert result == 1.0
|
||||||
|
|
||||||
|
|
||||||
|
class TestConstants:
|
||||||
|
"""Tests for module constants."""
|
||||||
|
|
||||||
|
def test_unit_factors(self):
|
||||||
|
"""Test UNIT_FACTORS contains all expected units."""
|
||||||
|
assert set(UNIT_FACTORS.keys()) == {"ns", "us", "ms", "s"}
|
||||||
|
assert UNIT_FACTORS["ns"] == 1.0
|
||||||
|
assert UNIT_FACTORS["us"] == 1e3
|
||||||
|
assert UNIT_FACTORS["ms"] == 1e6
|
||||||
|
assert UNIT_FACTORS["s"] == 1e9
|
||||||
|
|
||||||
|
def test_unit_labels(self):
|
||||||
|
"""Test UNIT_LABELS contains all expected units."""
|
||||||
|
assert set(UNIT_LABELS.keys()) == {"ns", "us", "ms", "s"}
|
||||||
|
assert UNIT_LABELS["ns"] == "ns"
|
||||||
|
assert UNIT_LABELS["us"] == "µs"
|
||||||
|
assert UNIT_LABELS["ms"] == "ms"
|
||||||
|
assert UNIT_LABELS["s"] == "s"
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
"""Tests for the data extraction module."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import pandas as pd
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from disk_checker.core.extractor import load_latency_csv
|
||||||
|
|
||||||
|
|
||||||
|
class TestLoadLatencyCsv:
|
||||||
|
"""Tests for load_latency_csv function."""
|
||||||
|
|
||||||
|
def test_load_valid_csv(self, sample_csv_path):
|
||||||
|
"""Test loading a valid fio CSV file."""
|
||||||
|
df = load_latency_csv(str(sample_csv_path))
|
||||||
|
assert not df.empty
|
||||||
|
assert "metric" in df.columns
|
||||||
|
assert "latency_ns" in df.columns
|
||||||
|
assert "count" in df.columns
|
||||||
|
|
||||||
|
def test_returns_long_format(self, sample_csv_path):
|
||||||
|
"""Test that output is in long format with metric column."""
|
||||||
|
df = load_latency_csv(str(sample_csv_path))
|
||||||
|
metrics = df["metric"].unique()
|
||||||
|
assert "read_clat" in metrics
|
||||||
|
|
||||||
|
def test_empty_csv_raises(self, empty_csv_path):
|
||||||
|
"""Test that empty CSV raises ValueError."""
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
load_latency_csv(str(empty_csv_path))
|
||||||
|
|
||||||
|
def test_invalid_format_raises(self, invalid_csv_path):
|
||||||
|
"""Test that invalid CSV format raises ValueError."""
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
load_latency_csv(str(invalid_csv_path))
|
||||||
|
|
||||||
|
def test_missing_nsec_column_raises(self, tmp_path):
|
||||||
|
"""Test that missing nsec column raises ValueError."""
|
||||||
|
csv_content = "col1,col2\n1,2\n"
|
||||||
|
path = tmp_path / "invalid.csv"
|
||||||
|
path.write_text(csv_content)
|
||||||
|
with pytest.raises(ValueError, match="expected first column to be 'nsec'"):
|
||||||
|
load_latency_csv(str(path))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_csv_path(tmp_path):
|
||||||
|
"""Create a sample CSV file for testing."""
|
||||||
|
csv_content = """nsec,read_clat_ns_count,read_clat_ns_cumulative,read_clat_ns_percentile
|
||||||
|
19791872,1,1,0.00016775708773695687
|
||||||
|
22413312,4,5,0.0008387854386847844
|
||||||
|
22675456,9,14,0.0023485992283173965
|
||||||
|
"""
|
||||||
|
path = tmp_path / "test.csv"
|
||||||
|
path.write_text(csv_content)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def empty_csv_path(tmp_path):
|
||||||
|
"""Create an empty CSV file for testing."""
|
||||||
|
path = tmp_path / "empty.csv"
|
||||||
|
path.write_text("nsec\n")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def invalid_csv_path(tmp_path):
|
||||||
|
"""Create an invalid CSV file for testing."""
|
||||||
|
path = tmp_path / "invalid.csv"
|
||||||
|
path.write_text("col1,col2\n1,2\n")
|
||||||
|
return path
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
"""Tests for helper functions."""
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from disk_checker.core.helpers import human_count, percentile_table
|
||||||
|
|
||||||
|
|
||||||
|
class TestHumanCount:
|
||||||
|
"""Tests for human_count function."""
|
||||||
|
|
||||||
|
def test_small_number(self):
|
||||||
|
"""Test formatting of small numbers."""
|
||||||
|
assert human_count(100) == "100"
|
||||||
|
assert human_count(50) == "50"
|
||||||
|
|
||||||
|
def test_thousands(self):
|
||||||
|
"""Test formatting of thousands."""
|
||||||
|
assert human_count(1500) == "2k"
|
||||||
|
assert human_count(2500) == "2k"
|
||||||
|
assert human_count(1000) == "1k"
|
||||||
|
assert human_count(999) == "999"
|
||||||
|
|
||||||
|
def test_millions(self):
|
||||||
|
"""Test formatting of millions."""
|
||||||
|
assert human_count(1500000) == "1.5M"
|
||||||
|
assert human_count(2750000) == "2.8M"
|
||||||
|
|
||||||
|
def test_billions(self):
|
||||||
|
"""Test formatting of billions."""
|
||||||
|
assert human_count(1500000000) == "1.5G"
|
||||||
|
assert human_count(2750000000) == "2.8G"
|
||||||
|
|
||||||
|
def test_zero(self):
|
||||||
|
"""Test formatting of zero."""
|
||||||
|
assert human_count(0) == "0"
|
||||||
|
|
||||||
|
def test_position_argument_ignored(self):
|
||||||
|
"""Test that _pos argument is ignored."""
|
||||||
|
assert human_count(1000, 0) == "1k"
|
||||||
|
|
||||||
|
|
||||||
|
class TestPercentileTable:
|
||||||
|
"""Tests for percentile_table function."""
|
||||||
|
|
||||||
|
def test_basic_percentiles(self):
|
||||||
|
"""Test basic percentile table generation."""
|
||||||
|
df = pd.DataFrame({
|
||||||
|
"metric": ["read_clat"] * 3,
|
||||||
|
"latency_ns": [1000, 2000, 3000],
|
||||||
|
"count": [1, 2, 1],
|
||||||
|
})
|
||||||
|
result = percentile_table(df, "read_clat", "us", [50, 90])
|
||||||
|
assert "Metric: read_clat" in result
|
||||||
|
assert "total I/Os: 4" in result
|
||||||
|
assert "percentile" in result
|
||||||
|
assert "latency" in result
|
||||||
|
|
||||||
|
def test_missing_metric(self):
|
||||||
|
"""Test percentile table with missing metric."""
|
||||||
|
df = pd.DataFrame({
|
||||||
|
"metric": ["read_clat"],
|
||||||
|
"latency_ns": [1000],
|
||||||
|
"count": [1],
|
||||||
|
})
|
||||||
|
result = percentile_table(df, "write_clat", "us", [50])
|
||||||
|
assert "(no data for metric write_clat)" in result
|
||||||
|
|
||||||
|
def test_unit_display(self):
|
||||||
|
"""Test that correct unit is displayed."""
|
||||||
|
df = pd.DataFrame({
|
||||||
|
"metric": ["read_clat"],
|
||||||
|
"latency_ns": [1000000], # 1ms in ns
|
||||||
|
"count": [1],
|
||||||
|
})
|
||||||
|
result = percentile_table(df, "read_clat", "ms", [50])
|
||||||
|
assert "ms" in result
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
"""Tests for the orchestrator module."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import pytest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import AsyncMock, patch
|
||||||
|
|
||||||
|
from disk_checker.core.orchestrator import Orchestrator, ToolStatus, JobResult
|
||||||
|
|
||||||
|
|
||||||
|
class TestOrchestrator:
|
||||||
|
"""Tests for Orchestrator class."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def orchestrator(self, tmp_path):
|
||||||
|
"""Create an orchestrator instance."""
|
||||||
|
return Orchestrator(work_dir=tmp_path)
|
||||||
|
|
||||||
|
def test_initial_state(self, orchestrator):
|
||||||
|
"""Test initial orchestrator state."""
|
||||||
|
assert orchestrator.list_jobs() == {}
|
||||||
|
|
||||||
|
def test_run_fio_creates_job(self, orchestrator, tmp_path):
|
||||||
|
"""Test that run_fio creates a job entry."""
|
||||||
|
with patch("asyncio.create_subprocess_exec") as mock_exec:
|
||||||
|
mock_exec.return_value = AsyncMock()
|
||||||
|
result = asyncio.get_event_loop().run_until_complete(
|
||||||
|
orchestrator.run_fio("/dev/sda")
|
||||||
|
)
|
||||||
|
|
||||||
|
jobs = orchestrator.list_jobs()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
job = list(jobs.values())[0]
|
||||||
|
assert job.status == ToolStatus.COMPLETED
|
||||||
|
# Note: output_path may not exist in test environment without actual fio
|
||||||
|
|
||||||
|
def test_get_job_status(self, orchestrator, tmp_path):
|
||||||
|
"""Test retrieving job status."""
|
||||||
|
with patch("asyncio.create_subprocess_exec") as mock_exec:
|
||||||
|
mock_exec.return_value = AsyncMock()
|
||||||
|
result = asyncio.get_event_loop().run_until_complete(
|
||||||
|
orchestrator.run_fio("/dev/sda")
|
||||||
|
)
|
||||||
|
job_id = result.job_id
|
||||||
|
|
||||||
|
job = orchestrator.get_job_status(job_id)
|
||||||
|
assert job is not None
|
||||||
|
assert job.status == ToolStatus.COMPLETED
|
||||||
|
|
||||||
|
def test_get_nonexistent_job(self, orchestrator):
|
||||||
|
"""Test getting status of non-existent job."""
|
||||||
|
job = orchestrator.get_job_status("nonexistent")
|
||||||
|
assert job is None
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
"""Tests for the plotting module."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
import matplotlib
|
||||||
|
matplotlib.use("Agg")
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
|
||||||
|
from disk_checker.plotting.plots import draw_histogram, draw_cdf, generate_plot
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
|
||||||
|
class TestDrawHistogram:
|
||||||
|
"""Tests for draw_histogram function."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_df(self):
|
||||||
|
"""Create a sample DataFrame for testing."""
|
||||||
|
return pd.DataFrame({
|
||||||
|
"metric": ["read_clat", "read_clat", "write_clat"],
|
||||||
|
"latency_ns": [1000, 2000, 1500],
|
||||||
|
"count": [1, 2, 1],
|
||||||
|
})
|
||||||
|
|
||||||
|
def test_draws_histogram(self, sample_df, tmp_path):
|
||||||
|
"""Test that histogram is drawn without errors."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_histogram(ax, sample_df, ["read_clat"])
|
||||||
|
assert ax.get_title() == "Completion latency distribution"
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
def test_multiple_metrics(self, sample_df, tmp_path):
|
||||||
|
"""Test drawing multiple metrics."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_histogram(ax, sample_df, ["read_clat", "write_clat"])
|
||||||
|
assert len(ax.collections) > 0
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
def test_log_scale(self, sample_df, tmp_path):
|
||||||
|
"""Test logarithmic scale."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_histogram(ax, sample_df, ["read_clat"], log_x=True, log_y=True)
|
||||||
|
assert ax.get_xscale() == "log"
|
||||||
|
assert ax.get_yscale() == "log"
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
|
||||||
|
class TestDrawCdf:
|
||||||
|
"""Tests for draw_cdf function."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_df(self):
|
||||||
|
"""Create a sample DataFrame for testing."""
|
||||||
|
return pd.DataFrame({
|
||||||
|
"metric": ["read_clat", "read_clat"],
|
||||||
|
"latency_ns": [1000, 2000],
|
||||||
|
"count": [1, 3],
|
||||||
|
})
|
||||||
|
|
||||||
|
def test_draws_cdf(self, sample_df, tmp_path):
|
||||||
|
"""Test that CDF is drawn without errors."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_cdf(ax, sample_df, ["read_clat"])
|
||||||
|
assert ax.get_title() == "Cumulative latency distribution"
|
||||||
|
assert ax.get_ylim() == (0, 100)
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
def test_log_scale(self, sample_df, tmp_path):
|
||||||
|
"""Test logarithmic X scale."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_cdf(ax, sample_df, ["read_clat"], log_x=True)
|
||||||
|
assert ax.get_xscale() == "log"
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
|
||||||
|
class TestGeneratePlot:
|
||||||
|
"""Tests for generate_plot function."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_df(self):
|
||||||
|
"""Create a sample DataFrame for testing."""
|
||||||
|
return pd.DataFrame({
|
||||||
|
"metric": ["read_clat", "read_clat"],
|
||||||
|
"latency_ns": [1000, 2000],
|
||||||
|
"count": [1, 3],
|
||||||
|
})
|
||||||
|
|
||||||
|
def test_generates_histogram(self, sample_df, tmp_path):
|
||||||
|
"""Test generating histogram plot."""
|
||||||
|
output = tmp_path / "test.png"
|
||||||
|
result = generate_plot(
|
||||||
|
df=sample_df,
|
||||||
|
metrics=["read_clat"],
|
||||||
|
output_path=output,
|
||||||
|
)
|
||||||
|
assert result == output
|
||||||
|
assert output.exists()
|
||||||
|
|
||||||
|
def test_generates_cdf_only(self, sample_df, tmp_path):
|
||||||
|
"""Test generating CDF-only plot."""
|
||||||
|
output = tmp_path / "test_cdf.png"
|
||||||
|
result = generate_plot(
|
||||||
|
df=sample_df,
|
||||||
|
metrics=["read_clat"],
|
||||||
|
output_path=output,
|
||||||
|
cdf_only=True,
|
||||||
|
)
|
||||||
|
assert result == output
|
||||||
|
assert output.exists()
|
||||||
|
|
||||||
|
def test_generates_histogram_with_cdf(self, sample_df, tmp_path):
|
||||||
|
"""Test generating histogram with CDF below."""
|
||||||
|
output = tmp_path / "test_both.png"
|
||||||
|
result = generate_plot(
|
||||||
|
df=sample_df,
|
||||||
|
metrics=["read_clat"],
|
||||||
|
output_path=output,
|
||||||
|
show_cdf=True,
|
||||||
|
)
|
||||||
|
assert result == output
|
||||||
|
assert output.exists()
|
||||||
|
|
||||||
|
def test_custom_dpi(self, sample_df, tmp_path):
|
||||||
|
"""Test custom DPI setting."""
|
||||||
|
output = tmp_path / "test_dpi.png"
|
||||||
|
result = generate_plot(
|
||||||
|
df=sample_df,
|
||||||
|
metrics=["read_clat"],
|
||||||
|
output_path=output,
|
||||||
|
dpi=300,
|
||||||
|
)
|
||||||
|
assert result == output
|
||||||
|
assert output.exists()
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
"""Tests for the web backend."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import AsyncMock, patch
|
||||||
|
|
||||||
|
from disk_checker.web.server import app
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client():
|
||||||
|
"""Create test client."""
|
||||||
|
return TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_csv_content():
|
||||||
|
"""Sample CSV content for testing."""
|
||||||
|
return """nsec,read_clat_ns_count,read_clat_ns_cumulative,read_clat_ns_percentile
|
||||||
|
19791872,1,1,0.00016775708773695687
|
||||||
|
22413312,4,5,0.0008387854386847844
|
||||||
|
22675456,9,14,0.0023485992283173965
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_csv_path(tmp_path, sample_csv_content):
|
||||||
|
"""Create sample CSV file."""
|
||||||
|
path = tmp_path / "test.csv"
|
||||||
|
path.write_text(sample_csv_content)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def test_health_check(client):
|
||||||
|
"""Test health check endpoint."""
|
||||||
|
response = client.get("/health")
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json()["status"] == "ok"
|
||||||
|
|
||||||
|
|
||||||
|
def test_scan_csv(client, sample_csv_path):
|
||||||
|
"""Test CSV scan endpoint."""
|
||||||
|
with open(sample_csv_path, "rb") as f:
|
||||||
|
response = client.post("/api/v1/scan", files={"file": f})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert "metrics" in data
|
||||||
|
assert len(data["metrics"]) > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_plot_endpoint(client, sample_csv_path):
|
||||||
|
"""Test plot generation endpoint."""
|
||||||
|
with open(sample_csv_path, "rb") as f:
|
||||||
|
response = client.post(
|
||||||
|
"/api/v1/plot",
|
||||||
|
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||||
|
data={"unit": "us"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.headers["content-type"] == "image/png"
|
||||||
|
|
||||||
|
|
||||||
|
def test_plot_with_options(client, sample_csv_path):
|
||||||
|
"""Test plot generation with options."""
|
||||||
|
with open(sample_csv_path, "rb") as f:
|
||||||
|
response = client.post(
|
||||||
|
"/api/v1/plot",
|
||||||
|
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||||
|
data={
|
||||||
|
"metric": "read_clat",
|
||||||
|
"unit": "ms",
|
||||||
|
"log_x": "true",
|
||||||
|
"cdf": "true",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_metric(client, sample_csv_path):
|
||||||
|
"""Test invalid metric returns error."""
|
||||||
|
with open(sample_csv_path, "rb") as f:
|
||||||
|
response = client.post(
|
||||||
|
"/api/v1/plot",
|
||||||
|
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||||
|
data={"metric": "invalid_metric", "unit": "us"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 400
|
||||||
Reference in New Issue
Block a user