Compare commits
25 Commits
main
..
a831ec2546
| Author | SHA1 | Date | |
|---|---|---|---|
| a831ec2546 | |||
| e13d45401f | |||
| bcb696a666 | |||
| d2216f9eec | |||
| 415d8b3601 | |||
| e0c0da6def | |||
| cb25dc08d4 | |||
| e9da6b8a24 | |||
| bf5fa8a133 | |||
| a425291f5b | |||
| 9a3fa5a5af | |||
| fe31a8893b | |||
| 37a2384fde | |||
| a0d0b04e78 | |||
| ac64a95502 | |||
| 31562586d7 | |||
| f0452e6ef6 | |||
| d659ec7910 | |||
| 57dc4bdd26 | |||
| c933f4f9ca | |||
| 271502cccd | |||
| 29c02c27fb | |||
| 2eed2053a1 | |||
| dac3aeaafb | |||
| 352bc16ba5 |
@@ -1,97 +0,0 @@
|
|||||||
name: "Bug Report"
|
|
||||||
description: "Report a bug or unexpected behavior to help us improve"
|
|
||||||
title: "[BUG]: "
|
|
||||||
labels: ["bug"]
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## ⚠️ Before You Begin
|
|
||||||
Please ensure you have:
|
|
||||||
- [ ] Searched existing issues to avoid duplicates
|
|
||||||
- [ ] Confirmed this is a bug, not a question or configuration problem
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: description
|
|
||||||
attributes:
|
|
||||||
label: "📋 Description"
|
|
||||||
description: "Provide a clear and concise description of the bug"
|
|
||||||
placeholder: "What happened? What did you expect to happen instead?"
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: reproduction
|
|
||||||
attributes:
|
|
||||||
label: "🔁 Steps to Reproduce"
|
|
||||||
description: "Step-by-step instructions to reproduce the issue"
|
|
||||||
placeholder: |
|
|
||||||
1. Go to '...'
|
|
||||||
2. Click on '....'
|
|
||||||
3. Scroll down to '....'
|
|
||||||
4. See error
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: expected
|
|
||||||
attributes:
|
|
||||||
label: "✅ Expected Behavior"
|
|
||||||
description: "What you expected to happen"
|
|
||||||
placeholder: "A clear description of what should happen..."
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: actual
|
|
||||||
attributes:
|
|
||||||
label: "❌ Actual Behavior"
|
|
||||||
description: "What actually happened"
|
|
||||||
placeholder: "Include error messages, stack traces, or unexpected outcomes..."
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: logs
|
|
||||||
attributes:
|
|
||||||
label: "📄 Logs / Screenshots"
|
|
||||||
description: |
|
|
||||||
Provide relevant logs, error messages, or screenshots.
|
|
||||||
For logs, please use a pastebin and share the URL.
|
|
||||||
**Remember to remove any sensitive information (API keys, passwords, etc.).**
|
|
||||||
placeholder: "Paste logs here or provide a Gist URL..."
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
---
|
|
||||||
## 🖥️ Environment Details
|
|
||||||
Please fill out the relevant information below.
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: additional
|
|
||||||
attributes:
|
|
||||||
label: "📎 Additional Context"
|
|
||||||
description: "Any other information that might be relevant"
|
|
||||||
placeholder: |
|
|
||||||
- Database type and version (e.g., PostgreSQL 15, SQLite)
|
|
||||||
- Reverse proxy/CDN in use (e.g., Nginx, Cloudflare)
|
|
||||||
- Any custom configuration
|
|
||||||
- Related issues or PRs
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: checkboxes
|
|
||||||
id: checklist
|
|
||||||
attributes:
|
|
||||||
label: "✅ Submission Checklist"
|
|
||||||
description: "Please confirm the following before submitting"
|
|
||||||
options:
|
|
||||||
- label: "I have searched for existing issues (open and closed) that report the same problem"
|
|
||||||
required: true
|
|
||||||
- label: "I have provided clear steps to reproduce the issue"
|
|
||||||
required: true
|
|
||||||
- label: "I have included relevant logs or error messages (with sensitive info removed)"
|
|
||||||
required: false
|
|
||||||
@@ -1,99 +0,0 @@
|
|||||||
name: "Feature Request"
|
|
||||||
description: "Suggest a new feature or enhancement for this project"
|
|
||||||
title: "[FEATURE]: "
|
|
||||||
labels: ["enhancement"]
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## ⚠️ Before You Begin
|
|
||||||
Please ensure you have:
|
|
||||||
- [ ] Searched existing issues (open and closed) to avoid duplicates
|
|
||||||
- [ ] Reviewed the [project documentation](https://git.vmn.su/max/vmn-ansible/wiki) for existing functionality
|
|
||||||
- [ ] Confirmed this is a feature request, not a bug or configuration question
|
|
||||||
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: problem
|
|
||||||
attributes:
|
|
||||||
label: "🎯 Problem Statement"
|
|
||||||
description: "What problem does this feature solve? What can't you do today?"
|
|
||||||
placeholder: |
|
|
||||||
I'm always frustrated when...
|
|
||||||
Currently, it's hard/impossible to...
|
|
||||||
This feature would help me because...
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: solution
|
|
||||||
attributes:
|
|
||||||
label: "💡 Proposed Solution"
|
|
||||||
description: "Describe the solution you'd like to see"
|
|
||||||
placeholder: |
|
|
||||||
A clear description of what you want to happen.
|
|
||||||
If you have a specific API, UI, or implementation in mind, describe it here.
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: screenshots
|
|
||||||
attributes:
|
|
||||||
label: "📸 Screenshots"
|
|
||||||
description: "If you can, provide screenshots or diagrams of the proposed feature"
|
|
||||||
placeholder: "Drag and drop images or paste links..."
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: technical
|
|
||||||
attributes:
|
|
||||||
label: "⚙️ Technical Approach"
|
|
||||||
description: "If you have ideas about how to implement this, share them here"
|
|
||||||
placeholder: |
|
|
||||||
- Which components/modules would be affected?
|
|
||||||
- Any database schema changes?
|
|
||||||
- API design considerations?
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: acceptance
|
|
||||||
attributes:
|
|
||||||
label: "✅ Acceptance Criteria"
|
|
||||||
description: "What does 'done' look like for this feature?"
|
|
||||||
placeholder: |
|
|
||||||
- [ ] Feature works in the UI
|
|
||||||
- [ ] API endpoints are documented
|
|
||||||
- [ ] Tests are added
|
|
||||||
- [ ] Documentation is updated
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: dropdown
|
|
||||||
id: contribution
|
|
||||||
attributes:
|
|
||||||
label: "🤝 Are you willing to contribute this feature?"
|
|
||||||
description: "Knowing if you can help implement this helps us prioritize"
|
|
||||||
options:
|
|
||||||
- "Yes, I can contribute"
|
|
||||||
- "Yes, but I need guidance"
|
|
||||||
- "No, but I can test/review"
|
|
||||||
- "No, I'm just requesting"
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: checkboxes
|
|
||||||
id: checklist
|
|
||||||
attributes:
|
|
||||||
label: "✅ Submission Checklist"
|
|
||||||
description: "Please confirm the following before submitting"
|
|
||||||
options:
|
|
||||||
- label: "I have searched for existing issues (open and closed) that request the same feature"
|
|
||||||
required: true
|
|
||||||
- label: "I have explained the problem this feature solves"
|
|
||||||
required: true
|
|
||||||
- label: "I have described a concrete solution or direction"
|
|
||||||
required: true
|
|
||||||
- label: "I have considered alternatives and explained why they fall short"
|
|
||||||
required: false
|
|
||||||
@@ -1,134 +0,0 @@
|
|||||||
name: "Refactor Request"
|
|
||||||
description: "Propose a code restructuring or cleanup with no behavior change"
|
|
||||||
title: "[REFACTOR]: "
|
|
||||||
labels: ["refactoring"]
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## ⚠️ Before You Begin
|
|
||||||
Please ensure you have:
|
|
||||||
- [ ] Searched existing issues (open and closed) to avoid duplicates
|
|
||||||
- [ ] Reviewed the [project documentation](https://git.vmn.su/max/vmn-ansible/wiki) for architecture guidelines
|
|
||||||
- [ ] Confirmed this is a refactor (code restructuring with NO behavior change), not a bug fix or new feature
|
|
||||||
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: problem
|
|
||||||
attributes:
|
|
||||||
label: "🎯 Problem Statement"
|
|
||||||
description: "What pain points does the current code cause? Why is this refactor needed?"
|
|
||||||
placeholder: |
|
|
||||||
The current implementation is difficult to maintain because...
|
|
||||||
This code has accumulated technical debt due to...
|
|
||||||
I'm constantly frustrated when working with this module because...
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: scope
|
|
||||||
attributes:
|
|
||||||
label: "📂 Scope"
|
|
||||||
description: "Which files, modules, or components need refactoring? Be specific."
|
|
||||||
placeholder: |
|
|
||||||
- File paths: `src/auth/`, `internal/handler/`
|
|
||||||
- Component: `UserService` class
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: current-state
|
|
||||||
attributes:
|
|
||||||
label: "🔍 Current State"
|
|
||||||
description: "Describe the current code structure, its issues, and why it's problematic"
|
|
||||||
placeholder: |
|
|
||||||
- The function `processUser()` is 500+ lines long
|
|
||||||
- Duplicated logic across 3 different files
|
|
||||||
- Mixed concerns (business logic + HTTP handling)
|
|
||||||
- Poor test coverage makes changes risky
|
|
||||||
- Naming is unclear and inconsistent
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: proposed-changes
|
|
||||||
attributes:
|
|
||||||
label: "💡 Proposed Solution"
|
|
||||||
description: "What should the code look like after the refactor?"
|
|
||||||
placeholder: |
|
|
||||||
- Extract `processUser()` into smaller, focused functions
|
|
||||||
- Consolidate duplicated logic into a shared utility
|
|
||||||
- Separate business logic from HTTP handlers
|
|
||||||
- Introduce interfaces for better testability
|
|
||||||
- Rename variables and functions for clarity
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: constraints
|
|
||||||
attributes:
|
|
||||||
label: "🚫 Constraints"
|
|
||||||
description: "What must NOT change? Public APIs, behavior, backward compatibility, performance characteristics?"
|
|
||||||
placeholder: |
|
|
||||||
- All public APIs must remain identical
|
|
||||||
- Database schema must not change
|
|
||||||
- External behavior must be identical
|
|
||||||
- Response formats must stay the same
|
|
||||||
- Performance must not degrade
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: benefits
|
|
||||||
attributes:
|
|
||||||
label: "📈 Expected Benefits"
|
|
||||||
description: "What improvements will this refactor bring?"
|
|
||||||
placeholder: |
|
|
||||||
- Improved maintainability
|
|
||||||
- Better testability
|
|
||||||
- Reduced code duplication
|
|
||||||
- Clearer separation of concerns
|
|
||||||
- Easier onboarding for new contributors
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: textarea
|
|
||||||
id: risks
|
|
||||||
attributes:
|
|
||||||
label: "⚠️ Risks & Mitigations"
|
|
||||||
description: "What could go wrong, and how will you mitigate it?"
|
|
||||||
placeholder: |
|
|
||||||
- Risk: Regression bugs
|
|
||||||
Mitigation: Comprehensive test coverage before and after
|
|
||||||
- Risk: Large diff making code review difficult
|
|
||||||
Mitigation: Break into smaller, incremental PRs
|
|
||||||
validations:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
- type: dropdown
|
|
||||||
id: contribution
|
|
||||||
attributes:
|
|
||||||
label: "🤝 Are you willing to contribute this refactor?"
|
|
||||||
description: "Knowing if you can help implement this helps us prioritize"
|
|
||||||
options:
|
|
||||||
- "Yes, I can contribute"
|
|
||||||
- "Yes, but I need guidance"
|
|
||||||
- "No, but I can review"
|
|
||||||
- "No, I'm just suggesting"
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
|
|
||||||
- type: checkboxes
|
|
||||||
id: checklist
|
|
||||||
attributes:
|
|
||||||
label: "✅ Submission Checklist"
|
|
||||||
description: "Please confirm the following before submitting"
|
|
||||||
options:
|
|
||||||
- label: "I have searched for existing issues (open and closed) that request the same refactor"
|
|
||||||
required: true
|
|
||||||
- label: "I have clearly explained the problem this refactor solves"
|
|
||||||
required: true
|
|
||||||
- label: "I have described the scope of the refactor (specific files/modules)"
|
|
||||||
required: false
|
|
||||||
- label: "I have identified what must NOT change (APIs, behavior, backward compatibility)"
|
|
||||||
required: false
|
|
||||||
+50
-1
@@ -1,3 +1,52 @@
|
|||||||
.vscode
|
.vscode
|
||||||
__pycache__
|
__pycache__
|
||||||
test_data
|
test_data
|
||||||
|
|
||||||
|
# Python
|
||||||
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
|
*.so
|
||||||
|
.Python
|
||||||
|
build/
|
||||||
|
develop-eggs/
|
||||||
|
dist/
|
||||||
|
downloads/
|
||||||
|
eggs/
|
||||||
|
.eggs/
|
||||||
|
lib/
|
||||||
|
lib64/
|
||||||
|
parts/
|
||||||
|
sdist/
|
||||||
|
var/
|
||||||
|
wheels/
|
||||||
|
*.egg-info/
|
||||||
|
.installed.cfg
|
||||||
|
*.egg
|
||||||
|
|
||||||
|
# Virtual environments
|
||||||
|
venv/
|
||||||
|
ENV/
|
||||||
|
env/
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
|
||||||
|
# Environment variables
|
||||||
|
.env
|
||||||
|
.env.local
|
||||||
|
|
||||||
|
# Test coverage
|
||||||
|
.htmlcov/
|
||||||
|
.coverage
|
||||||
|
.coverage.*
|
||||||
|
coverage.xml
|
||||||
|
|
||||||
|
# Pytest
|
||||||
|
.pytest_cache/
|
||||||
|
|
||||||
|
# Generated files
|
||||||
|
*.png
|
||||||
|
*.csv
|
||||||
|
data/
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Disk Checker
|
||||||
|
|
||||||
|
Disk benchmarking and latency visualization tool for fio and badblocks output.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- Parse fio JSON+ latency CSV files
|
||||||
|
- Generate histogram and CDF plots
|
||||||
|
- REST API for remote benchmarking
|
||||||
|
- Async job management for long-running benchmarks
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone <repository>
|
||||||
|
cd disk-checker
|
||||||
|
python -m venv venv
|
||||||
|
source venv/bin/activate
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## CLI Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Basic usage
|
||||||
|
python -m disk_checker scan_latency.csv
|
||||||
|
|
||||||
|
# With options
|
||||||
|
python -m disk_checker scan_latency.csv --metric read_clat --unit ms --log-x
|
||||||
|
|
||||||
|
# Generate CDF
|
||||||
|
python -m disk_checker scan_latency.csv --cdf
|
||||||
|
|
||||||
|
# View all metrics
|
||||||
|
python -m disk_checker scan_latency.csv --all-metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start the server
|
||||||
|
python -m disk_checker.scripts.serve --host 0.0.0.0 --port 8000
|
||||||
|
|
||||||
|
# Access API docs at http://localhost:8000/docs
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Endpoints
|
||||||
|
|
||||||
|
- `POST /api/v1/scan` - Upload CSV and get analysis
|
||||||
|
- `POST /api/v1/plot` - Generate and return plot image
|
||||||
|
- `POST /api/v1/benchmark` - Start a benchmark job
|
||||||
|
- `GET /api/v1/jobs/{id}` - Check job status
|
||||||
|
- `GET /health` - Health check
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
disk-checker/
|
||||||
|
├── src/disk_checker/
|
||||||
|
│ ├── core/ # Business logic (extractor, converter, helpers, orchestrator)
|
||||||
|
│ ├── plotting/ # Visualization (matplotlib)
|
||||||
|
│ ├── web/ # FastAPI backend
|
||||||
|
│ └── scripts/ # Entry points (cli, serve)
|
||||||
|
├── tests/ # Test suite
|
||||||
|
└── data/ # Sample data
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest
|
||||||
|
pytest --cov=disk_checker
|
||||||
|
pytest --cov=disk_checker --cov-report=html
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
[build-system]
|
||||||
|
requires = ["hatchling"]
|
||||||
|
build-backend = "hatchling.build"
|
||||||
|
|
||||||
|
[project]
|
||||||
|
name = "disk-checker"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Disk benchmarking and latency visualization tool"
|
||||||
|
readme = "README.md"
|
||||||
|
requires-python = ">=3.10"
|
||||||
|
dependencies = [
|
||||||
|
"pandas>=2.0",
|
||||||
|
"numpy>=1.24",
|
||||||
|
"matplotlib>=3.7",
|
||||||
|
"fastapi>=0.100",
|
||||||
|
"uvicorn[standard]>=0.23",
|
||||||
|
"python-multipart>=0.0.6",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
dev = [
|
||||||
|
"pytest>=7.0",
|
||||||
|
"pytest-asyncio>=0.21",
|
||||||
|
"pytest-cov>=4.0",
|
||||||
|
"black>=23.0",
|
||||||
|
"ruff>=0.1.0",
|
||||||
|
"mypy>=1.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.scripts]
|
||||||
|
disk-checker = "disk_checker.scripts.cli:main"
|
||||||
|
disk-checker-serve = "disk_checker.scripts.serve:main"
|
||||||
|
|
||||||
|
[tool.hatch.build.targets.wheel]
|
||||||
|
packages = ["src/disk_checker"]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
testpaths = ["tests"]
|
||||||
|
python_files = ["test_*.py"]
|
||||||
|
python_functions = ["test_*"]
|
||||||
|
|
||||||
|
[tool.black]
|
||||||
|
line-length = 100
|
||||||
|
target-version = ["py310"]
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 100
|
||||||
|
target-version = "py310"
|
||||||
|
|
||||||
|
[tool.mypy]
|
||||||
|
python_version = "3.10"
|
||||||
|
warn_return_any = true
|
||||||
|
warn_unused_configs = true
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
"""Disk Checker - Disk benchmarking and latency visualization tool."""
|
||||||
|
|
||||||
|
__version__ = "0.1.0"
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
"""Allow running disk-checker as a module: python -m disk_checker"""
|
||||||
|
|
||||||
|
from disk_checker.scripts.cli import main
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
"""Core business logic module."""
|
||||||
|
|
||||||
|
from .extractor import load_latency_csv
|
||||||
|
from .converter import to_unit, UNIT_FACTORS, UNIT_LABELS
|
||||||
|
from .helpers import human_count, percentile_table
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"load_latency_csv",
|
||||||
|
"to_unit",
|
||||||
|
"UNIT_FACTORS",
|
||||||
|
"UNIT_LABELS",
|
||||||
|
"human_count",
|
||||||
|
"percentile_table",
|
||||||
|
]
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
"""Unit conversion utilities for latency data."""
|
||||||
|
|
||||||
|
from typing import Dict
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
UNIT_FACTORS: Dict[str, float] = {
|
||||||
|
"ns": 1.0,
|
||||||
|
"us": 1e3,
|
||||||
|
"ms": 1e6,
|
||||||
|
"s": 1e9,
|
||||||
|
}
|
||||||
|
|
||||||
|
UNIT_LABELS: Dict[str, str] = {
|
||||||
|
"ns": "ns",
|
||||||
|
"us": "µs",
|
||||||
|
"ms": "ms",
|
||||||
|
"s": "s",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def to_unit(values_ns, unit: str) -> np.ndarray:
|
||||||
|
"""
|
||||||
|
Convert nanosecond values to the specified unit.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
values_ns: Values in nanoseconds
|
||||||
|
unit: Target unit (ns, us, ms, s)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Converted values as numpy array
|
||||||
|
"""
|
||||||
|
return np.asarray(values_ns, dtype=float) / UNIT_FACTORS[unit]
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
"""Drive detection module for disk-checker."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
from typing import Dict, List
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class DriveInfo:
|
||||||
|
"""Information about a block device."""
|
||||||
|
name: str # e.g., "sda", "sdb"
|
||||||
|
path: str # e.g., "/dev/sda"
|
||||||
|
size: str # e.g., "232.9G"
|
||||||
|
model: str # e.g., "CT250MX500SSD1"
|
||||||
|
type: str # e.g., "disk", "part"
|
||||||
|
mounted: bool # True if currently mounted
|
||||||
|
mount_point: str = "" # e.g., "/", "/home"
|
||||||
|
serial: str = "" # e.g., "2131G5F25137"
|
||||||
|
|
||||||
|
|
||||||
|
def get_mounted_devices() -> set:
|
||||||
|
"""
|
||||||
|
Get set of mounted device paths.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Set of device paths that are currently mounted
|
||||||
|
"""
|
||||||
|
mounted = set()
|
||||||
|
|
||||||
|
# Method 1: Parse lsblk with mount points
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["lsblk", "-J", "-o", "NAME,MOUNTPOINT"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True
|
||||||
|
)
|
||||||
|
data = json.loads(result.stdout)
|
||||||
|
for device in data.get("blockdevices", []):
|
||||||
|
mountpoint = device.get("mountpoint")
|
||||||
|
if mountpoint:
|
||||||
|
# Add both full path and device name
|
||||||
|
mounted.add(f"/dev/{device['name']}")
|
||||||
|
mounted.add(device["name"])
|
||||||
|
except (subprocess.CalledProcessError, json.JSONDecodeError):
|
||||||
|
print("lsblk failed to retrieve mounted devices information")
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Method 2: Parse /proc/mounts
|
||||||
|
try:
|
||||||
|
with open("/proc/mounts", "r") as f:
|
||||||
|
for line in f:
|
||||||
|
parts = line.split()
|
||||||
|
if len(parts) >= 2 and parts[0].startswith("/dev/"):
|
||||||
|
mounted.add(parts[0])
|
||||||
|
# Also add device name without /dev/
|
||||||
|
mounted.add(parts[0].replace("/dev/", ""))
|
||||||
|
except (FileNotFoundError, IOError):
|
||||||
|
print("/proc/mounts failed to retrieve mounted devices information")
|
||||||
|
pass
|
||||||
|
|
||||||
|
return mounted
|
||||||
|
|
||||||
|
|
||||||
|
def scan_drives() -> List[DriveInfo]:
|
||||||
|
"""
|
||||||
|
Scan host system for available block devices.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of DriveInfo objects for physical disks
|
||||||
|
"""
|
||||||
|
mounted_devices = get_mounted_devices()
|
||||||
|
drives = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Get block devices with lsblk
|
||||||
|
result = subprocess.run(
|
||||||
|
[
|
||||||
|
"lsblk",
|
||||||
|
"-J", # JSON output
|
||||||
|
"-d", # Only disks (not partitions)
|
||||||
|
"-b", # Show size in bytes
|
||||||
|
"-o", "NAME,SIZE,TYPE,MODEL,Serial,MOUNTPOINT"
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True
|
||||||
|
)
|
||||||
|
|
||||||
|
data = json.loads(result.stdout)
|
||||||
|
|
||||||
|
for device in data.get("blockdevices", []):
|
||||||
|
# Only include physical disks
|
||||||
|
if device.get("type") != "disk":
|
||||||
|
continue
|
||||||
|
|
||||||
|
name = device.get("name", "")
|
||||||
|
path = f"/dev/{name}"
|
||||||
|
|
||||||
|
# Check if mounted
|
||||||
|
is_mounted = False
|
||||||
|
#is_mounted = path in mounted_devices or name in mounted_devices
|
||||||
|
for md in mounted_devices:
|
||||||
|
is_mounted |= path in md
|
||||||
|
|
||||||
|
drive = DriveInfo(
|
||||||
|
name=name,
|
||||||
|
path=path,
|
||||||
|
size=device.get("size", "Unknown"),
|
||||||
|
model=device.get("model") or "",
|
||||||
|
type="disk",
|
||||||
|
mounted=is_mounted,
|
||||||
|
mount_point=device.get("mountpoint", ""),
|
||||||
|
serial=device.get("serial") or ""
|
||||||
|
)
|
||||||
|
drives.append(drive)
|
||||||
|
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
# If lsblk fails, return empty list
|
||||||
|
print("lsblk failed to parse drives data")
|
||||||
|
print(e)
|
||||||
|
pass
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
print("json decoder error")
|
||||||
|
print(e)
|
||||||
|
pass
|
||||||
|
|
||||||
|
return drives
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
"""CSV data extraction module for fio latency data."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
import pandas as pd
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
|
||||||
|
def load_latency_csv(path: str) -> pd.DataFrame:
|
||||||
|
"""
|
||||||
|
Parse wide-format fio JSON+ CSV into long-format DataFrame.
|
||||||
|
|
||||||
|
Transforms wide-format columns (e.g., read_clat_ns_count) into rows with
|
||||||
|
a metric identifier, making it easier to group and analyze by operation type.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
path: Path to CSV file generated by fio_jsonplus_clat2csv
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
DataFrame with columns: metric, latency_ns, count, cumulative, percentile
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If CSV format is invalid or contains no valid data
|
||||||
|
"""
|
||||||
|
df = pd.read_csv(path, skipinitialspace=True)
|
||||||
|
df.columns = [str(c).strip() for c in df.columns]
|
||||||
|
|
||||||
|
# Drop trailing unnamed / empty columns
|
||||||
|
df = df.loc[:, ~df.columns.str.match(r"^Unnamed")]
|
||||||
|
df = df.loc[:, df.columns != ""]
|
||||||
|
|
||||||
|
nsec_col = df.columns[0]
|
||||||
|
if "nsec" not in nsec_col.lower():
|
||||||
|
raise ValueError(
|
||||||
|
f"expected first column to be 'nsec', got {nsec_col!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Coerce the nsec column to numeric
|
||||||
|
df[nsec_col] = pd.to_numeric(df[nsec_col], errors="coerce")
|
||||||
|
df = df.dropna(subset=[nsec_col])
|
||||||
|
|
||||||
|
# Discover metric triples: <metric>_count / _cumulative / _percentile
|
||||||
|
metrics = {}
|
||||||
|
for col in df.columns:
|
||||||
|
if col == nsec_col:
|
||||||
|
continue
|
||||||
|
for suffix in ("_count", "_cumulative", "_percentile"):
|
||||||
|
if col.endswith(suffix):
|
||||||
|
base = col[: -len(suffix)]
|
||||||
|
metrics.setdefault(base, {})[suffix[1:]] = col
|
||||||
|
|
||||||
|
if not metrics:
|
||||||
|
raise ValueError("no *_count columns found in CSV")
|
||||||
|
|
||||||
|
frames = []
|
||||||
|
for base, cols in metrics.items():
|
||||||
|
count_col = cols["count"]
|
||||||
|
|
||||||
|
# Coerce count to numeric
|
||||||
|
counts = pd.to_numeric(df[count_col], errors="coerce")
|
||||||
|
|
||||||
|
# Keep only rows where the count column actually has a number
|
||||||
|
mask = counts.notna()
|
||||||
|
if not mask.any():
|
||||||
|
continue
|
||||||
|
|
||||||
|
sub = pd.DataFrame({
|
||||||
|
"latency_ns": df.loc[mask, nsec_col].values,
|
||||||
|
"count": counts[mask].astype(int).values,
|
||||||
|
})
|
||||||
|
if "cumulative" in cols:
|
||||||
|
sub["cumulative"] = pd.to_numeric(
|
||||||
|
df.loc[mask, cols["cumulative"]], errors="coerce"
|
||||||
|
).values
|
||||||
|
if "percentile" in cols:
|
||||||
|
sub["percentile"] = pd.to_numeric(
|
||||||
|
df.loc[mask, cols["percentile"]], errors="coerce"
|
||||||
|
).values
|
||||||
|
|
||||||
|
# read_clat_ns -> read_clat
|
||||||
|
sub["metric"] = base.replace("_ns", "")
|
||||||
|
frames.append(sub)
|
||||||
|
|
||||||
|
if not frames:
|
||||||
|
raise ValueError("no metric had any numeric count data")
|
||||||
|
|
||||||
|
out = pd.concat(frames, ignore_index=True)
|
||||||
|
out = out[out["count"] > 0].reset_index(drop=True)
|
||||||
|
return out
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
"""Helper functions for formatting and display."""
|
||||||
|
|
||||||
|
from typing import List
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from matplotlib.ticker import FuncFormatter
|
||||||
|
|
||||||
|
from .converter import to_unit, UNIT_LABELS
|
||||||
|
|
||||||
|
|
||||||
|
def human_count(x, _pos=None) -> str:
|
||||||
|
"""
|
||||||
|
Format large numbers with K/M/G suffixes for axis labels.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
x: Numeric value to format
|
||||||
|
_pos: Matplotlib tick position (unused)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted string (e.g., "1.5M", "3.2G")
|
||||||
|
"""
|
||||||
|
if x >= 1e9:
|
||||||
|
return f"{x/1e9:.1f}G"
|
||||||
|
if x >= 1e6:
|
||||||
|
return f"{x/1e6:.1f}M"
|
||||||
|
if x >= 1e3:
|
||||||
|
return f"{x/1e3:.0f}k"
|
||||||
|
return f"{x:.0f}"
|
||||||
|
|
||||||
|
|
||||||
|
def percentile_table(df: pd.DataFrame, metric: str, unit: str, pcts: List[float]) -> str:
|
||||||
|
"""
|
||||||
|
Generate a formatted table showing latency values at specified percentiles.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
df: Long-format DataFrame from load_latency_csv
|
||||||
|
metric: Metric name (e.g., 'read_clat')
|
||||||
|
unit: Display unit (ns, us, ms, s)
|
||||||
|
pcts: List of percentile values (e.g., [50, 90, 99])
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted string table
|
||||||
|
"""
|
||||||
|
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||||
|
if g.empty:
|
||||||
|
return f"(no data for metric {metric})"
|
||||||
|
|
||||||
|
counts = g["count"].values.astype(float)
|
||||||
|
total = int(counts.sum())
|
||||||
|
cdf = np.cumsum(counts) / total * 100.0
|
||||||
|
lat = to_unit(g["latency_ns"].values, unit)
|
||||||
|
|
||||||
|
lines = [
|
||||||
|
f"\nMetric: {metric} (total I/Os: {total:,})",
|
||||||
|
f" {'percentile':>12} {'latency':>14}",
|
||||||
|
]
|
||||||
|
|
||||||
|
for p in pcts:
|
||||||
|
idx = min(np.searchsorted(cdf, p), len(lat) - 1)
|
||||||
|
lines.append(f" {p:>11.2f}% {lat[idx]:>12.3f} {UNIT_LABELS[unit]}")
|
||||||
|
|
||||||
|
return "\n".join(lines)
|
||||||
@@ -0,0 +1,164 @@
|
|||||||
|
"""Tool orchestration module for running disk benchmarks."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import subprocess
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from enum import Enum
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, Optional
|
||||||
|
|
||||||
|
|
||||||
|
class ToolStatus(Enum):
|
||||||
|
"""Status of a benchmark job."""
|
||||||
|
PENDING = "pending"
|
||||||
|
RUNNING = "running"
|
||||||
|
COMPLETED = "completed"
|
||||||
|
FAILED = "failed"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class JobResult:
|
||||||
|
"""Result of a benchmark job."""
|
||||||
|
job_id: str
|
||||||
|
status: ToolStatus
|
||||||
|
output_path: Optional[Path] = None
|
||||||
|
error: Optional[str] = None
|
||||||
|
metadata: Dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class Orchestrator:
|
||||||
|
"""
|
||||||
|
Orchestrates disk benchmarking tools (fio, badblocks, etc.)
|
||||||
|
|
||||||
|
Provides async interfaces for running benchmarks and collecting results.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, work_dir: Path = Path(".")):
|
||||||
|
self.work_dir = work_dir
|
||||||
|
self._jobs: Dict[str, JobResult] = {}
|
||||||
|
|
||||||
|
async def run_fio(
|
||||||
|
self,
|
||||||
|
device: str,
|
||||||
|
job_file: Optional[Path] = None,
|
||||||
|
output_format: str = "jsonplus",
|
||||||
|
**kwargs,
|
||||||
|
) -> Path:
|
||||||
|
"""
|
||||||
|
Run fio benchmark and return path to output file.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
device: Block device path (e.g., '/dev/sda')
|
||||||
|
job_file: Optional fio job file
|
||||||
|
output_format: Output format (json, jsonplus, csv)
|
||||||
|
**kwargs: Additional fio options
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Path to output file
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
subprocess.CalledProcessError: If fio command fails
|
||||||
|
"""
|
||||||
|
job_id = f"fio_{asyncio.get_event_loop().time()}"
|
||||||
|
output_path = self.work_dir / f"{job_id}.{output_format}"
|
||||||
|
|
||||||
|
cmd = ["fio", "--output-format=jsonplus"]
|
||||||
|
if job_file:
|
||||||
|
cmd.extend(["--filename", str(device)])
|
||||||
|
else:
|
||||||
|
cmd.extend(["--name=test", "--rw=randread", "--bs=4k"])
|
||||||
|
|
||||||
|
cmd.extend(["--output", str(output_path)])
|
||||||
|
cmd.extend([f"--output-format={output_format}"])
|
||||||
|
|
||||||
|
for k, v in kwargs.items():
|
||||||
|
cmd.extend([f"--{k}={v}"])
|
||||||
|
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.RUNNING,
|
||||||
|
output_path=output_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
await asyncio.create_subprocess_exec(
|
||||||
|
*cmd,
|
||||||
|
stdout=asyncio.subprocess.PIPE,
|
||||||
|
stderr=asyncio.subprocess.PIPE,
|
||||||
|
)
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.COMPLETED,
|
||||||
|
output_path=output_path,
|
||||||
|
)
|
||||||
|
return output_path
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.FAILED,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def run_badblocks(
|
||||||
|
self,
|
||||||
|
device: str,
|
||||||
|
blocks_count: Optional[int] = None,
|
||||||
|
**kwargs,
|
||||||
|
) -> Path:
|
||||||
|
"""
|
||||||
|
Run badblocks scan on device.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
device: Block device path
|
||||||
|
blocks_count: Number of blocks to scan
|
||||||
|
**kwargs: Additional badblocks options
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Path to badblocks output file
|
||||||
|
"""
|
||||||
|
job_id = f"badblocks_{asyncio.get_event_loop().time()}"
|
||||||
|
output_path = self.work_dir / f"{job_id}.txt"
|
||||||
|
|
||||||
|
cmd = ["badblocks", "-w", "-s", "-p", "0", "-t", "RANDOM", "-o"]
|
||||||
|
cmd.extend([str(output_path), str(device)])
|
||||||
|
|
||||||
|
if blocks_count:
|
||||||
|
cmd.extend(["-b", str(blocks_count)])
|
||||||
|
|
||||||
|
for k, v in kwargs.items():
|
||||||
|
cmd.extend([f"-{k}", str(v)])
|
||||||
|
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.RUNNING,
|
||||||
|
output_path=output_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
await asyncio.create_subprocess_exec(
|
||||||
|
*cmd,
|
||||||
|
stdout=asyncio.subprocess.PIPE,
|
||||||
|
stderr=asyncio.subprocess.PIPE,
|
||||||
|
)
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.COMPLETED,
|
||||||
|
output_path=output_path,
|
||||||
|
)
|
||||||
|
return output_path
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
self._jobs[job_id] = JobResult(
|
||||||
|
job_id=job_id,
|
||||||
|
status=ToolStatus.FAILED,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def get_job_status(self, job_id: str) -> Optional[JobResult]:
|
||||||
|
"""Get status of a submitted job."""
|
||||||
|
return self._jobs.get(job_id)
|
||||||
|
|
||||||
|
def list_jobs(self) -> Dict[str, JobResult]:
|
||||||
|
"""List all jobs."""
|
||||||
|
return self._jobs.copy()
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
"""Tool detection module for disk-checker."""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
from typing import Dict, List
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ToolInfo:
|
||||||
|
"""Information about a benchmarking tool."""
|
||||||
|
name: str
|
||||||
|
available: bool
|
||||||
|
version: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
# Tools to scan for
|
||||||
|
TOOLS_TO_SCAN = [
|
||||||
|
"fio",
|
||||||
|
"badblocks",
|
||||||
|
"smartctl",
|
||||||
|
"iozone",
|
||||||
|
"hdparm",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def scan_tools() -> Dict[str, ToolInfo]:
|
||||||
|
"""
|
||||||
|
Scan host system for available benchmarking tools.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dictionary mapping tool names to ToolInfo objects
|
||||||
|
"""
|
||||||
|
results = {}
|
||||||
|
for tool_name in TOOLS_TO_SCAN:
|
||||||
|
available = shutil.which(tool_name) is not None
|
||||||
|
version = ""
|
||||||
|
if available:
|
||||||
|
# Try to get version (optional enhancement)
|
||||||
|
try:
|
||||||
|
import subprocess
|
||||||
|
result = subprocess.run(
|
||||||
|
[tool_name, "--version"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=2
|
||||||
|
)
|
||||||
|
version = result.stdout.split("\n")[0][:50]
|
||||||
|
except Exception:
|
||||||
|
# If we can't even run '--version', then
|
||||||
|
# a tool is not considered as available
|
||||||
|
available = False
|
||||||
|
results[tool_name] = ToolInfo(
|
||||||
|
name=tool_name,
|
||||||
|
available=available,
|
||||||
|
version=version
|
||||||
|
)
|
||||||
|
return results
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
"""Plotting module for disk latency visualization."""
|
||||||
|
|
||||||
|
from .plots import draw_histogram, draw_cdf, generate_plot
|
||||||
|
|
||||||
|
__all__ = ["draw_histogram", "draw_cdf", "generate_plot"]
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
"""Plotting module for disk latency visualization."""
|
||||||
|
|
||||||
|
import matplotlib
|
||||||
|
matplotlib.use("Agg") # Non-interactive backend for server environments
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
from matplotlib.ticker import FuncFormatter
|
||||||
|
import numpy as np
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Optional
|
||||||
|
|
||||||
|
from disk_checker.core.converter import to_unit, UNIT_LABELS
|
||||||
|
from disk_checker.core.helpers import human_count
|
||||||
|
|
||||||
|
|
||||||
|
def draw_histogram(
|
||||||
|
ax,
|
||||||
|
df,
|
||||||
|
metrics: List[str],
|
||||||
|
unit: str = "us",
|
||||||
|
log_y: bool = False,
|
||||||
|
log_x: bool = False,
|
||||||
|
normalize: bool = False,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
Draw a histogram showing the distribution of latencies for each metric.
|
||||||
|
|
||||||
|
Uses a step plot (where="post") to show the discrete nature of the data.
|
||||||
|
If normalize=True, Y-axis shows probability density instead of counts.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
ax: Matplotlib axis object
|
||||||
|
df: Long-format DataFrame from load_latency_csv
|
||||||
|
metrics: List of metric names to plot
|
||||||
|
unit: X-axis unit (ns, us, ms, s)
|
||||||
|
log_y: Use logarithmic Y scale
|
||||||
|
log_x: Use logarithmic X scale
|
||||||
|
normalize: Normalize to probability density
|
||||||
|
"""
|
||||||
|
for metric in metrics:
|
||||||
|
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||||
|
if g.empty:
|
||||||
|
continue
|
||||||
|
|
||||||
|
x = to_unit(g["latency_ns"].values, unit)
|
||||||
|
y = g["count"].values.astype(float)
|
||||||
|
|
||||||
|
if normalize:
|
||||||
|
s = y.sum()
|
||||||
|
if s > 0:
|
||||||
|
y = y / s
|
||||||
|
|
||||||
|
ax.step(x, y, where="post", lw=1.4, label=metric)
|
||||||
|
ax.fill_between(x, y, step="post", alpha=0.25)
|
||||||
|
|
||||||
|
ax.set_xlabel(f"Latency ({UNIT_LABELS[unit]})")
|
||||||
|
ax.set_ylabel("Probability" if normalize else "Number of I/Os")
|
||||||
|
ax.set_title("Completion latency distribution")
|
||||||
|
ax.grid(True, which="both", ls=":", alpha=0.5)
|
||||||
|
|
||||||
|
if not normalize:
|
||||||
|
ax.yaxis.set_major_formatter(FuncFormatter(human_count))
|
||||||
|
if log_y:
|
||||||
|
ax.set_yscale("log")
|
||||||
|
if log_x:
|
||||||
|
ax.set_xscale("log")
|
||||||
|
if len(metrics) > 1:
|
||||||
|
ax.legend(title="metric", fontsize=9)
|
||||||
|
|
||||||
|
|
||||||
|
def draw_cdf(ax, df, metrics: List[str], unit: str = "us", log_x: bool = False) -> None:
|
||||||
|
"""
|
||||||
|
Draw a Cumulative Distribution Function (CDF).
|
||||||
|
|
||||||
|
Shows the percentage of I/Os completed within each latency threshold.
|
||||||
|
Each metric gets its own line; Y-axis ranges from 0% to 100%.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
ax: Matplotlib axis object
|
||||||
|
df: Long-format DataFrame
|
||||||
|
metrics: List of metric names
|
||||||
|
unit: X-axis unit
|
||||||
|
log_x: Use logarithmic X scale
|
||||||
|
"""
|
||||||
|
for metric in metrics:
|
||||||
|
g = df[df["metric"] == metric].sort_values("latency_ns")
|
||||||
|
if g.empty:
|
||||||
|
continue
|
||||||
|
|
||||||
|
x = to_unit(g["latency_ns"].values, unit)
|
||||||
|
counts = g["count"].values.astype(float)
|
||||||
|
total = counts.sum()
|
||||||
|
if total <= 0:
|
||||||
|
continue
|
||||||
|
|
||||||
|
cdf = np.cumsum(counts) / total * 100.0
|
||||||
|
ax.plot(x, cdf, lw=1.6, label=f"{metric} (n={int(total):,})")
|
||||||
|
|
||||||
|
ax.set_xlabel(f"Latency ({UNIT_LABELS[unit]})")
|
||||||
|
ax.set_ylabel("Cumulative % of I/Os")
|
||||||
|
ax.set_title("Cumulative latency distribution")
|
||||||
|
ax.set_ylim(0, 100)
|
||||||
|
ax.grid(True, which="both", ls=":", alpha=0.5)
|
||||||
|
ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:.0f}%"))
|
||||||
|
|
||||||
|
if log_x:
|
||||||
|
ax.set_xscale("log")
|
||||||
|
ax.legend(fontsize=9)
|
||||||
|
|
||||||
|
|
||||||
|
def generate_plot(
|
||||||
|
df,
|
||||||
|
metrics: List[str],
|
||||||
|
output_path: Path,
|
||||||
|
unit: str = "us",
|
||||||
|
log_y: bool = False,
|
||||||
|
log_x: bool = False,
|
||||||
|
normalize: bool = False,
|
||||||
|
show_cdf: bool = False,
|
||||||
|
cdf_only: bool = False,
|
||||||
|
title: str = "",
|
||||||
|
dpi: int = 150,
|
||||||
|
) -> Path:
|
||||||
|
"""
|
||||||
|
Generate a complete plot with histogram and/or CDF.
|
||||||
|
|
||||||
|
Creates the appropriate subplot layout based on the options provided.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
df: Long-format DataFrame
|
||||||
|
metrics: Metrics to plot
|
||||||
|
output_path: Where to save the plot
|
||||||
|
unit: X-axis unit
|
||||||
|
log_y: Logarithmic Y scale
|
||||||
|
log_x: Logarithmic X scale
|
||||||
|
normalize: Normalize histogram
|
||||||
|
show_cdf: Show CDF below histogram
|
||||||
|
cdf_only: Show only CDF
|
||||||
|
title: Plot title
|
||||||
|
dpi: Output resolution
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Path to saved plot
|
||||||
|
"""
|
||||||
|
if cdf_only:
|
||||||
|
fig, ax = plt.subplots(figsize=(11, 6))
|
||||||
|
draw_cdf(ax, df, metrics, unit, log_x)
|
||||||
|
elif show_cdf:
|
||||||
|
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(11, 10))
|
||||||
|
draw_histogram(
|
||||||
|
ax1, df, metrics, unit, log_y, log_x, normalize
|
||||||
|
)
|
||||||
|
draw_cdf(ax2, df, metrics, unit, log_x)
|
||||||
|
else:
|
||||||
|
fig, ax = plt.subplots(figsize=(11, 6))
|
||||||
|
draw_histogram(ax, df, metrics, unit, log_y, log_x, normalize)
|
||||||
|
|
||||||
|
if title:
|
||||||
|
fig.suptitle(title, fontsize=13, y=0.995)
|
||||||
|
fig.tight_layout()
|
||||||
|
fig.savefig(output_path, dpi=dpi, bbox_inches="tight")
|
||||||
|
plt.close(fig)
|
||||||
|
return output_path
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
CLI entry point for disk-checker.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python -m disk_checker scan_latency.csv
|
||||||
|
disk-checker scan_latency.csv --metric read_clat --unit ms
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from disk_checker.core.extractor import load_latency_csv
|
||||||
|
from disk_checker.core.helpers import percentile_table
|
||||||
|
from disk_checker.plotting.plots import generate_plot
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Plot fio json+ wide-format latency CSV."
|
||||||
|
)
|
||||||
|
parser.add_argument("csv", help="CSV from fio_jsonplus_clat2csv")
|
||||||
|
parser.add_argument("--metric", default=None,
|
||||||
|
help="Metric to plot (e.g. read_clat). "
|
||||||
|
"If omitted, the metric with the most I/Os is used.")
|
||||||
|
parser.add_argument("--all-metrics", action="store_true",
|
||||||
|
help="Plot every metric that has data")
|
||||||
|
parser.add_argument("--unit", choices=["ns", "us", "ms", "s"], default="us",
|
||||||
|
help="X-axis latency unit (default: us)")
|
||||||
|
parser.add_argument("--log-y", action="store_true")
|
||||||
|
parser.add_argument("--log-x", action="store_true")
|
||||||
|
parser.add_argument("--cdf", action="store_true",
|
||||||
|
help="Add a CDF subplot below the histogram")
|
||||||
|
parser.add_argument("--cdf-only", action="store_true")
|
||||||
|
parser.add_argument("--normalize", action="store_true",
|
||||||
|
help="Normalize histogram to a probability density")
|
||||||
|
parser.add_argument("--percentiles", default="50,90,99,99.9,99.99")
|
||||||
|
parser.add_argument("--no-percentiles", action="store_true")
|
||||||
|
parser.add_argument("--output", "-o", default=None)
|
||||||
|
parser.add_argument("--dpi", type=int, default=150)
|
||||||
|
parser.add_argument("--title", default="")
|
||||||
|
parser.add_argument("--show", action="store_true")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if not Path(args.csv).is_file():
|
||||||
|
sys.exit(f"error: file not found: {args.csv}")
|
||||||
|
|
||||||
|
df = load_latency_csv(args.csv)
|
||||||
|
if df.empty:
|
||||||
|
sys.exit("error: no latency data in CSV")
|
||||||
|
|
||||||
|
available = sorted(df["metric"].unique())
|
||||||
|
print("Metrics with data:", ", ".join(available))
|
||||||
|
|
||||||
|
if args.all_metrics:
|
||||||
|
metrics = available
|
||||||
|
elif args.metric:
|
||||||
|
if args.metric not in available:
|
||||||
|
sys.exit(f"error: metric {args.metric!r} not in {available}")
|
||||||
|
metrics = [args.metric]
|
||||||
|
else:
|
||||||
|
totals = df.groupby("metric")["count"].sum().sort_values(ascending=False)
|
||||||
|
metrics = [totals.index[0]]
|
||||||
|
print(f"Auto-selected metric: {metrics[0]}")
|
||||||
|
|
||||||
|
output = args.output or f"{Path(args.csv).stem}_latency.png"
|
||||||
|
generate_plot(
|
||||||
|
df=df,
|
||||||
|
metrics=metrics,
|
||||||
|
output_path=Path(output),
|
||||||
|
unit=args.unit,
|
||||||
|
log_y=args.log_y,
|
||||||
|
log_x=args.log_x,
|
||||||
|
normalize=args.normalize,
|
||||||
|
show_cdf=args.cdf,
|
||||||
|
cdf_only=args.cdf_only,
|
||||||
|
title=args.title,
|
||||||
|
dpi=args.dpi,
|
||||||
|
)
|
||||||
|
print(f"Saved plot to {output}")
|
||||||
|
|
||||||
|
if not args.no_percentiles:
|
||||||
|
try:
|
||||||
|
pcts = [float(x) for x in args.percentiles.split(",") if x.strip()]
|
||||||
|
except ValueError:
|
||||||
|
sys.exit("error: --percentiles must be comma-separated numbers")
|
||||||
|
for m in metrics:
|
||||||
|
print(percentile_table(df, m, args.unit, pcts))
|
||||||
|
|
||||||
|
if args.show:
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
plt.show()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
"""Startup script for the web server."""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import uvicorn
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main entry point for the web server."""
|
||||||
|
parser = argparse.ArgumentParser(description="Start Disk Checker API server")
|
||||||
|
parser.add_argument("--host", default="127.0.0.1", help="Host to bind to")
|
||||||
|
parser.add_argument("--port", type=int, default=8000, help="Port to bind to")
|
||||||
|
parser.add_argument("--workers", type=int, default=1, help="Number of workers")
|
||||||
|
parser.add_argument("--reload", action="store_true", help="Enable auto-reload")
|
||||||
|
parser.add_argument("--work-dir", type=Path, default=Path("."), help="Work directory")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# Set work directory in environment
|
||||||
|
import os
|
||||||
|
os.environ["DISK_CHECKER_WORK_DIR"] = str(args.work_dir)
|
||||||
|
|
||||||
|
uvicorn.run(
|
||||||
|
"disk_checker.web.server:app",
|
||||||
|
host=args.host,
|
||||||
|
port=args.port,
|
||||||
|
workers=args.workers,
|
||||||
|
reload=args.reload,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
"""Web backend module for disk-checker API."""
|
||||||
|
|
||||||
|
from .web_routes import web_router
|
||||||
|
from .routes import api_router
|
||||||
|
|
||||||
|
__all__ = ["web_router", "api_router"]
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
"""Pydantic models for the web API."""
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from typing import List, Optional, Dict, Any
|
||||||
|
from enum import Enum
|
||||||
|
|
||||||
|
|
||||||
|
class ToolType(str, Enum):
|
||||||
|
"""Available benchmarking tools."""
|
||||||
|
FIO = "fio"
|
||||||
|
BADBLOCKS = "badblocks"
|
||||||
|
IO_URING = "io_uring"
|
||||||
|
|
||||||
|
|
||||||
|
class JobStatus(str, Enum):
|
||||||
|
"""Status of a benchmark job."""
|
||||||
|
PENDING = "pending"
|
||||||
|
RUNNING = "running"
|
||||||
|
COMPLETED = "completed"
|
||||||
|
FAILED = "failed"
|
||||||
|
|
||||||
|
|
||||||
|
class BenchmarkRequest(BaseModel):
|
||||||
|
"""Request to start a benchmark job."""
|
||||||
|
device: str = Field(..., description="Block device path (e.g., /dev/sda)")
|
||||||
|
tool: ToolType = ToolType.FIO
|
||||||
|
job_file: Optional[str] = None
|
||||||
|
options: Dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
class Config:
|
||||||
|
json_schema_extra = {
|
||||||
|
"example": {
|
||||||
|
"device": "/dev/sda",
|
||||||
|
"tool": "fio",
|
||||||
|
"options": {
|
||||||
|
"rw": "randread",
|
||||||
|
"bs": "4k",
|
||||||
|
"size": "1G",
|
||||||
|
"runtime": "60"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class JobStatusResponse(BaseModel):
|
||||||
|
"""Response with job status."""
|
||||||
|
job_id: str
|
||||||
|
status: JobStatus
|
||||||
|
output_path: Optional[str] = None
|
||||||
|
error: Optional[str] = None
|
||||||
|
created_at: Optional[str] = None
|
||||||
|
completed_at: Optional[str] = None
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
"""API routes for disk-checker web backend."""
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Request
|
||||||
|
|
||||||
|
from disk_checker.core.tool_scanner import ToolInfo
|
||||||
|
from disk_checker.core.drive_scanner import DriveInfo, scan_drives
|
||||||
|
|
||||||
|
api_router = APIRouter()
|
||||||
|
|
||||||
|
@api_router.get("/tools")
|
||||||
|
async def get_available_tools(request: Request):
|
||||||
|
"""
|
||||||
|
Get list of available benchmarking tools.
|
||||||
|
"""
|
||||||
|
tools = request.app.state.available_tools
|
||||||
|
return {
|
||||||
|
tool_name: {
|
||||||
|
"name" : tool.name,
|
||||||
|
"available": tool.available,
|
||||||
|
"version" : tool.version
|
||||||
|
}
|
||||||
|
for tool_name, tool in tools.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
@api_router.get("/drives")
|
||||||
|
async def get_available_drives(request: Request):
|
||||||
|
"""
|
||||||
|
Get list of available block devices
|
||||||
|
Mounted drives are marked as unavailable
|
||||||
|
"""
|
||||||
|
drives = scan_drives()
|
||||||
|
return {
|
||||||
|
"drives": [
|
||||||
|
{
|
||||||
|
"name": drive.name,
|
||||||
|
"path": drive.path,
|
||||||
|
"size": drive.size,
|
||||||
|
"model": drive.model,
|
||||||
|
"mounted": drive.mounted,
|
||||||
|
"mount_point": drive.mount_point,
|
||||||
|
"serial": drive.serial
|
||||||
|
}
|
||||||
|
for drive in drives
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
"""FastAPI application for disk-checker web API."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.staticfiles import StaticFiles
|
||||||
|
from fastapi.templating import Jinja2Templates
|
||||||
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from disk_checker.core.orchestrator import Orchestrator
|
||||||
|
from disk_checker.core.tool_scanner import scan_tools
|
||||||
|
# Import and include routers
|
||||||
|
from disk_checker.web.routes import api_router
|
||||||
|
from disk_checker.web.web_routes import web_router
|
||||||
|
|
||||||
|
|
||||||
|
# Get application root
|
||||||
|
BASE_DIR = Path(__file__).resolve().parent.parent.parent.parent
|
||||||
|
TEMPLATES_DIR = BASE_DIR / "templates"
|
||||||
|
STATIC_DIR = BASE_DIR / "static"
|
||||||
|
|
||||||
|
# Setup templates
|
||||||
|
templates = Jinja2Templates(directory=str(TEMPLATES_DIR))
|
||||||
|
|
||||||
|
# Setup logger
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI):
|
||||||
|
"""Application lifespan handler."""
|
||||||
|
# Startup: initialize orchestrator
|
||||||
|
#app.state.orchestrator = Orchestrator(work_dir=app.state.work_dir)
|
||||||
|
app.state.jobs = {}
|
||||||
|
|
||||||
|
# Scan for available benchmarking tools
|
||||||
|
app.state.available_tools = await asyncio.to_thread(scan_tools)
|
||||||
|
logger.info(f"Available tools: {list(app.state.available_tools.keys())}")
|
||||||
|
|
||||||
|
# Store templates and static_dir path in app.state
|
||||||
|
app.state.templates = templates
|
||||||
|
app.state.static_dir = STATIC_DIR
|
||||||
|
|
||||||
|
logger.info("Disk Checker API started")
|
||||||
|
|
||||||
|
yield
|
||||||
|
# Shutdown: cleanup
|
||||||
|
logger.info("Disk Checker API stopped")
|
||||||
|
|
||||||
|
|
||||||
|
app = FastAPI(
|
||||||
|
title="Disk Checker API",
|
||||||
|
description="API for disk benchmarking and latency visualization",
|
||||||
|
version="0.1.0",
|
||||||
|
lifespan=lifespan,
|
||||||
|
)
|
||||||
|
# Setup static files
|
||||||
|
app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")
|
||||||
|
# Include routers
|
||||||
|
app.include_router(api_router, prefix="/api/v1")
|
||||||
|
app.include_router(web_router)
|
||||||
|
|
||||||
|
# Add CORS (permissive by default)
|
||||||
|
app.add_middleware(
|
||||||
|
CORSMiddleware,
|
||||||
|
allow_origins=["*"],
|
||||||
|
allow_credentials=True,
|
||||||
|
allow_methods=["*"],
|
||||||
|
allow_headers=["*"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Simple health endpoint
|
||||||
|
@app.get("/health")
|
||||||
|
async def health_check():
|
||||||
|
"""Health check endpoint."""
|
||||||
|
return {"status": "ok", "version": "0.1.0"}
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
from fastapi import APIRouter, Request
|
||||||
|
from fastapi.templating import Jinja2Templates
|
||||||
|
from typing import Dict
|
||||||
|
|
||||||
|
web_router = APIRouter()
|
||||||
|
|
||||||
|
# Navigation configuration - NEW TABS GO HERE!
|
||||||
|
NAVIGATION = [
|
||||||
|
{"name": "home", "label": "Home", "url": "/", "icon": "bi-house"},
|
||||||
|
{"name": "benchmark", "label": "Benchmark", "url": "/benchmark", "icon": "bi-hdd"},
|
||||||
|
{"name": "jobs", "label": "Job Queue", "url": "/jobs", "icon": "bi-list-task"},
|
||||||
|
]
|
||||||
|
|
||||||
|
def get_active_tab(url: str) -> str:
|
||||||
|
"""Determine which navigation tab is active."""
|
||||||
|
for nav_item in NAVIGATION:
|
||||||
|
if nav_item["url"] == url:
|
||||||
|
return nav_item["name"]
|
||||||
|
return "home"
|
||||||
|
|
||||||
|
|
||||||
|
@web_router.get("/")
|
||||||
|
async def render_home(request: Request):
|
||||||
|
"""
|
||||||
|
Render the main page
|
||||||
|
"""
|
||||||
|
tools = request.app.state.available_tools
|
||||||
|
return request.app.state.templates.TemplateResponse(
|
||||||
|
name="index.html",
|
||||||
|
request=request,
|
||||||
|
context={
|
||||||
|
"tools":tools,
|
||||||
|
"navigation": NAVIGATION,
|
||||||
|
"active_tab": get_active_tab("/")
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
@web_router.get("/benchmark")
|
||||||
|
async def render_benchmark(request: Request):
|
||||||
|
"""Render the benchmark page."""
|
||||||
|
templates: Jinja2Templates = request.app.state.templates
|
||||||
|
|
||||||
|
return templates.TemplateResponse(
|
||||||
|
name="benchmark.html",
|
||||||
|
request=request,
|
||||||
|
context={
|
||||||
|
"navigation": NAVIGATION,
|
||||||
|
"active_tab": get_active_tab("/benchmark")
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@web_router.get("/jobs")
|
||||||
|
async def render_jobs(request: Request):
|
||||||
|
"""Render the job queue page."""
|
||||||
|
templates: Jinja2Templates = request.app.state.templates
|
||||||
|
|
||||||
|
return templates.TemplateResponse(
|
||||||
|
name="jobs.html",
|
||||||
|
request=request,
|
||||||
|
context={
|
||||||
|
"navigation": NAVIGATION,
|
||||||
|
"active_tab": get_active_tab("/jobs")
|
||||||
|
}
|
||||||
|
)
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
/* Basic styling for Disk Checker */
|
||||||
|
.container {
|
||||||
|
max-width: 1200px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card {
|
||||||
|
transition: transform 0.2s;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card:hover {
|
||||||
|
transform: translateY(-2px);
|
||||||
|
box-shadow: 0 4px 8px rgba(0,0,0,0.1);
|
||||||
|
}
|
||||||
|
|
||||||
|
.badge {
|
||||||
|
font-size: 0.9rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Drive cards styling */
|
||||||
|
.drive-card {
|
||||||
|
transition: all 0.3s ease;
|
||||||
|
height: 100%;
|
||||||
|
}
|
||||||
|
|
||||||
|
.drive-card.available {
|
||||||
|
border-left: 4px solid #28a745;
|
||||||
|
}
|
||||||
|
|
||||||
|
.drive-card.disabled {
|
||||||
|
border-left: 4px solid #6c757d;
|
||||||
|
opacity: 0.7;
|
||||||
|
}
|
||||||
|
|
||||||
|
.drive-card:hover:not(.disabled) {
|
||||||
|
transform: translateY(-2px);
|
||||||
|
box-shadow: 0 4px 8px rgba(0,0,0,0.1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Make disabled cards look unavailable */
|
||||||
|
.drive-card.disabled .card-title {
|
||||||
|
color: #6c757d;
|
||||||
|
}
|
||||||
|
|
||||||
|
.drive-card.disabled .card-text {
|
||||||
|
color: #6c757d;
|
||||||
|
}
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
/**
|
||||||
|
* Disk Checker - Main JavaScript
|
||||||
|
* Handles HTMX events and dynamic content rendering
|
||||||
|
*/
|
||||||
|
|
||||||
|
// Render drives after HTMX loads them
|
||||||
|
function renderDrives(data) {
|
||||||
|
const container = document.getElementById('drives-content');
|
||||||
|
if (!container) return;
|
||||||
|
|
||||||
|
container.innerHTML = '';
|
||||||
|
|
||||||
|
if (!data.drives || data.drives.length === 0) {
|
||||||
|
container.innerHTML = '<div class="col-12"><p class="text-muted">No drives found.</p></div>';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
data.drives.forEach(drive => {
|
||||||
|
const col = document.createElement('div');
|
||||||
|
col.className = 'col-md-4 mb-3';
|
||||||
|
|
||||||
|
const card = document.createElement('div');
|
||||||
|
card.className = `drive-card ${drive.mounted ? 'disabled' : 'available'}`;
|
||||||
|
|
||||||
|
card.innerHTML = `
|
||||||
|
<div class="card-body">
|
||||||
|
<h5 class="card-title">${drive.name}</h5>
|
||||||
|
<p class="card-text">
|
||||||
|
<strong>Size:</strong> ${drive.size}<br>
|
||||||
|
<strong>Model:</strong> ${drive.model || 'Unknown'}<br>
|
||||||
|
${drive.serial ? `<strong>Serial:</strong> ${drive.serial}<br>` : ''}
|
||||||
|
${drive.mount_point ? `<strong>Mounted at:</strong> ${drive.mount_point}<br>` : ''}
|
||||||
|
</p>
|
||||||
|
<div>
|
||||||
|
${drive.mounted
|
||||||
|
? '<span class="badge bg-secondary">Mounted - Unavailable</span>'
|
||||||
|
: '<span class="badge bg-success">Available for Benchmark</span>'
|
||||||
|
}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
|
||||||
|
col.appendChild(card);
|
||||||
|
container.appendChild(col);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Listen for HTMX events
|
||||||
|
document.addEventListener('htmx:afterRequest', function(evt) {
|
||||||
|
if (evt.detail.target.id === 'drives-content') {
|
||||||
|
renderDrives(JSON.parse(evt.detail.xhr.response));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Optional: Listen for HTMX load event as fallback
|
||||||
|
document.addEventListener('htmx:load', function(evt) {
|
||||||
|
if (evt.target.id === 'drives-content') {
|
||||||
|
// HTMX will load the content via hx-get
|
||||||
|
// The afterRequest event will handle rendering
|
||||||
|
}
|
||||||
|
});
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>Disk Checker - {% block title %}Home{% endblock %}</title>
|
||||||
|
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet">
|
||||||
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.10.0/font/bootstrap-icons.css">
|
||||||
|
<script src="https://unpkg.com/htmx.org@1.9.10"></script>
|
||||||
|
<link rel="stylesheet" href="/static/css/style.css">
|
||||||
|
{% block extra_css %}{% endblock %}
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<!-- Navigation Bar -->
|
||||||
|
<nav class="navbar navbar-expand-lg navbar-dark bg-dark">
|
||||||
|
<div class="container">
|
||||||
|
<!-- Logo / Brand -->
|
||||||
|
<a class="navbar-brand" href="/">
|
||||||
|
<i class="bi bi-hdd-stack"></i>
|
||||||
|
Disk Checker
|
||||||
|
</a>
|
||||||
|
|
||||||
|
<!-- Mobile toggle -->
|
||||||
|
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarNav">
|
||||||
|
<span class="navbar-toggler-icon"></span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<!-- Navigation Tabs -->
|
||||||
|
<div class="collapse navbar-collapse" id="navbarNav">
|
||||||
|
<ul class="navbar-nav">
|
||||||
|
{% for nav_item in navigation %}
|
||||||
|
<li class="nav-item">
|
||||||
|
<a class="nav-link {% if active_tab == nav_item.name %}active{% endif %}"
|
||||||
|
href="{{ nav_item.url }}">
|
||||||
|
<i class="bi {{ nav_item.icon }}"></i>
|
||||||
|
{{ nav_item.label }}
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
{% endfor %}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
<!-- Page Content -->
|
||||||
|
<div class="container mt-4">
|
||||||
|
{% block content %}{% endblock %}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script>
|
||||||
|
{% block extra_js %}{% endblock %}
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
|
||||||
|
{% block title %}Benchmark{% endblock %}
|
||||||
|
|
||||||
|
{% block content %}
|
||||||
|
<div class="row">
|
||||||
|
<div class="col-12">
|
||||||
|
<h2>Benchmark</h2>
|
||||||
|
<p class="text-muted">Configure and run disk benchmarks.</p>
|
||||||
|
|
||||||
|
<!-- Placeholder content - will be implemented later -->
|
||||||
|
<div class="card">
|
||||||
|
<div class="card-body">
|
||||||
|
<h5 class="card-title">Select Drive</h5>
|
||||||
|
<p class="card-text">Choose a drive to benchmark from the list below.</p>
|
||||||
|
|
||||||
|
<!-- Drive selection will go here -->
|
||||||
|
<div id="drive-selector"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{% endblock %}
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
|
||||||
|
{% block title %}Home{% endblock %}
|
||||||
|
|
||||||
|
{% block content %}
|
||||||
|
<div class="row">
|
||||||
|
<div class="col-12">
|
||||||
|
<!--Available tools section-->
|
||||||
|
<h2>Available Benchmarking Tools</h2>
|
||||||
|
|
||||||
|
<!-- Server-side rendered tools -->
|
||||||
|
{% if tools %}
|
||||||
|
<div class="row">
|
||||||
|
{% for tool_name, tool in tools.items() %}
|
||||||
|
<div class="col-md-4 mb-3">
|
||||||
|
<div class="card {% if tool.available %}border-success{% else %}border-secondary{% endif %}">
|
||||||
|
<div class="card-body">
|
||||||
|
<h5 class="card-title">{{ tool.name }}</h5>
|
||||||
|
<p class="card-text">
|
||||||
|
{% if tool.available %}
|
||||||
|
<span class="badge bg-success">Available</span>
|
||||||
|
{% if tool.version %}
|
||||||
|
<small class="text-muted">{{ tool.version }}</small>
|
||||||
|
{% endif %}
|
||||||
|
{% else %}
|
||||||
|
<span class="badge bg-secondary">Not Installed</span>
|
||||||
|
{% endif %}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
</div>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
<!--Available drives section-->
|
||||||
|
<h2 class="mt-5">Available Drives</h2>
|
||||||
|
<p class="text-muted">
|
||||||
|
<i class="bi bi-info-circle"></i>
|
||||||
|
Mounted drives are grayed out and cannot be benchmarked.
|
||||||
|
</p>
|
||||||
|
<div id="drives-list"
|
||||||
|
hx-get="/api/v1/drives"
|
||||||
|
hx-trigger="load"
|
||||||
|
hx-target="#drives-content"></div>
|
||||||
|
<div id="drives-content" class="row"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{% endblock %}
|
||||||
|
|
||||||
|
{% block extra_js %}
|
||||||
|
<script src="/static/js/main.js"></script>
|
||||||
|
{% endblock %}
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
|
||||||
|
{% block title %}Job Queue{% endblock %}
|
||||||
|
|
||||||
|
{% block content %}
|
||||||
|
<div class="row">
|
||||||
|
<div class="col-12">
|
||||||
|
<h2>Job Queue</h2>
|
||||||
|
<p class="text-muted">Monitor and manage benchmark jobs.</p>
|
||||||
|
|
||||||
|
<!-- Jobs list -->
|
||||||
|
<div id="jobs-list"
|
||||||
|
hx-get="/api/v1/jobs"
|
||||||
|
hx-trigger="load"
|
||||||
|
hx-target="#jobs-content">
|
||||||
|
<p>Loading jobs...</p>
|
||||||
|
</div>
|
||||||
|
<div id="jobs-content"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{% endblock %}
|
||||||
|
|
||||||
|
{% block extra_js %}
|
||||||
|
<script>
|
||||||
|
// Load jobs when page loads
|
||||||
|
document.addEventListener('htmx:afterRequest', function(evt) {
|
||||||
|
if (evt.detail.target.id === 'jobs-content') {
|
||||||
|
renderJobs(JSON.parse(evt.detail.xhr.response));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
function renderJobs(jobIds) {
|
||||||
|
const container = document.getElementById('jobs-content');
|
||||||
|
container.innerHTML = '';
|
||||||
|
|
||||||
|
if (!jobIds || jobIds.length === 0) {
|
||||||
|
container.innerHTML = '<div class="alert alert-info">No jobs in queue.</div>';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
jobIds.forEach(jobId => {
|
||||||
|
const div = document.createElement('div');
|
||||||
|
div.className = 'card mb-2';
|
||||||
|
div.innerHTML = `
|
||||||
|
<div class="card-body">
|
||||||
|
<h5 class="card-title">${jobId}</h5>
|
||||||
|
<p class="card-text text-muted">Job details will appear here...</p>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
container.appendChild(div);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
{% endblock %}
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Tests for disk-checker."""
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Tests for the unit conversion module."""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from disk_checker.core.converter import to_unit, UNIT_FACTORS, UNIT_LABELS
|
||||||
|
|
||||||
|
|
||||||
|
class TestToUnit:
|
||||||
|
"""Tests for to_unit function."""
|
||||||
|
|
||||||
|
def test_nanoseconds_to_microseconds(self):
|
||||||
|
"""Test conversion from ns to us."""
|
||||||
|
values = np.array([1000, 2000, 3000])
|
||||||
|
result = to_unit(values, "us")
|
||||||
|
expected = np.array([1.0, 2.0, 3.0])
|
||||||
|
np.testing.assert_array_equal(result, expected)
|
||||||
|
|
||||||
|
def test_nanoseconds_to_milliseconds(self):
|
||||||
|
"""Test conversion from ns to ms."""
|
||||||
|
values = np.array([1000000, 2000000])
|
||||||
|
result = to_unit(values, "ms")
|
||||||
|
expected = np.array([1.0, 2.0])
|
||||||
|
np.testing.assert_array_equal(result, expected)
|
||||||
|
|
||||||
|
def test_nanoseconds_to_seconds(self):
|
||||||
|
"""Test conversion from ns to s."""
|
||||||
|
values = np.array([1000000000, 2000000000])
|
||||||
|
result = to_unit(values, "s")
|
||||||
|
expected = np.array([1.0, 2.0])
|
||||||
|
np.testing.assert_array_equal(result, expected)
|
||||||
|
|
||||||
|
def test_same_unit_returns_same(self):
|
||||||
|
"""Test conversion to same unit."""
|
||||||
|
values = np.array([100, 200, 300])
|
||||||
|
result = to_unit(values, "ns")
|
||||||
|
np.testing.assert_array_equal(result, values)
|
||||||
|
|
||||||
|
def test_empty_array(self):
|
||||||
|
"""Test conversion with empty array."""
|
||||||
|
values = np.array([])
|
||||||
|
result = to_unit(values, "us")
|
||||||
|
assert len(result) == 0
|
||||||
|
|
||||||
|
def test_single_value(self):
|
||||||
|
"""Test conversion with single value."""
|
||||||
|
result = to_unit(1000, "us")
|
||||||
|
assert result == 1.0
|
||||||
|
|
||||||
|
|
||||||
|
class TestConstants:
|
||||||
|
"""Tests for module constants."""
|
||||||
|
|
||||||
|
def test_unit_factors(self):
|
||||||
|
"""Test UNIT_FACTORS contains all expected units."""
|
||||||
|
assert set(UNIT_FACTORS.keys()) == {"ns", "us", "ms", "s"}
|
||||||
|
assert UNIT_FACTORS["ns"] == 1.0
|
||||||
|
assert UNIT_FACTORS["us"] == 1e3
|
||||||
|
assert UNIT_FACTORS["ms"] == 1e6
|
||||||
|
assert UNIT_FACTORS["s"] == 1e9
|
||||||
|
|
||||||
|
def test_unit_labels(self):
|
||||||
|
"""Test UNIT_LABELS contains all expected units."""
|
||||||
|
assert set(UNIT_LABELS.keys()) == {"ns", "us", "ms", "s"}
|
||||||
|
assert UNIT_LABELS["ns"] == "ns"
|
||||||
|
assert UNIT_LABELS["us"] == "µs"
|
||||||
|
assert UNIT_LABELS["ms"] == "ms"
|
||||||
|
assert UNIT_LABELS["s"] == "s"
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
"""Tests for the data extraction module."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import pandas as pd
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from disk_checker.core.extractor import load_latency_csv
|
||||||
|
|
||||||
|
|
||||||
|
class TestLoadLatencyCsv:
|
||||||
|
"""Tests for load_latency_csv function."""
|
||||||
|
|
||||||
|
def test_load_valid_csv(self, sample_csv_path):
|
||||||
|
"""Test loading a valid fio CSV file."""
|
||||||
|
df = load_latency_csv(str(sample_csv_path))
|
||||||
|
assert not df.empty
|
||||||
|
assert "metric" in df.columns
|
||||||
|
assert "latency_ns" in df.columns
|
||||||
|
assert "count" in df.columns
|
||||||
|
|
||||||
|
def test_returns_long_format(self, sample_csv_path):
|
||||||
|
"""Test that output is in long format with metric column."""
|
||||||
|
df = load_latency_csv(str(sample_csv_path))
|
||||||
|
metrics = df["metric"].unique()
|
||||||
|
assert "read_clat" in metrics
|
||||||
|
|
||||||
|
def test_empty_csv_raises(self, empty_csv_path):
|
||||||
|
"""Test that empty CSV raises ValueError."""
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
load_latency_csv(str(empty_csv_path))
|
||||||
|
|
||||||
|
def test_invalid_format_raises(self, invalid_csv_path):
|
||||||
|
"""Test that invalid CSV format raises ValueError."""
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
load_latency_csv(str(invalid_csv_path))
|
||||||
|
|
||||||
|
def test_missing_nsec_column_raises(self, tmp_path):
|
||||||
|
"""Test that missing nsec column raises ValueError."""
|
||||||
|
csv_content = "col1,col2\n1,2\n"
|
||||||
|
path = tmp_path / "invalid.csv"
|
||||||
|
path.write_text(csv_content)
|
||||||
|
with pytest.raises(ValueError, match="expected first column to be 'nsec'"):
|
||||||
|
load_latency_csv(str(path))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_csv_path(tmp_path):
|
||||||
|
"""Create a sample CSV file for testing."""
|
||||||
|
csv_content = """nsec,read_clat_ns_count,read_clat_ns_cumulative,read_clat_ns_percentile
|
||||||
|
19791872,1,1,0.00016775708773695687
|
||||||
|
22413312,4,5,0.0008387854386847844
|
||||||
|
22675456,9,14,0.0023485992283173965
|
||||||
|
"""
|
||||||
|
path = tmp_path / "test.csv"
|
||||||
|
path.write_text(csv_content)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def empty_csv_path(tmp_path):
|
||||||
|
"""Create an empty CSV file for testing."""
|
||||||
|
path = tmp_path / "empty.csv"
|
||||||
|
path.write_text("nsec\n")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def invalid_csv_path(tmp_path):
|
||||||
|
"""Create an invalid CSV file for testing."""
|
||||||
|
path = tmp_path / "invalid.csv"
|
||||||
|
path.write_text("col1,col2\n1,2\n")
|
||||||
|
return path
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
"""Tests for helper functions."""
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from disk_checker.core.helpers import human_count, percentile_table
|
||||||
|
|
||||||
|
|
||||||
|
class TestHumanCount:
|
||||||
|
"""Tests for human_count function."""
|
||||||
|
|
||||||
|
def test_small_number(self):
|
||||||
|
"""Test formatting of small numbers."""
|
||||||
|
assert human_count(100) == "100"
|
||||||
|
assert human_count(50) == "50"
|
||||||
|
|
||||||
|
def test_thousands(self):
|
||||||
|
"""Test formatting of thousands."""
|
||||||
|
assert human_count(1500) == "2k"
|
||||||
|
assert human_count(2500) == "2k"
|
||||||
|
assert human_count(1000) == "1k"
|
||||||
|
assert human_count(999) == "999"
|
||||||
|
|
||||||
|
def test_millions(self):
|
||||||
|
"""Test formatting of millions."""
|
||||||
|
assert human_count(1500000) == "1.5M"
|
||||||
|
assert human_count(2750000) == "2.8M"
|
||||||
|
|
||||||
|
def test_billions(self):
|
||||||
|
"""Test formatting of billions."""
|
||||||
|
assert human_count(1500000000) == "1.5G"
|
||||||
|
assert human_count(2750000000) == "2.8G"
|
||||||
|
|
||||||
|
def test_zero(self):
|
||||||
|
"""Test formatting of zero."""
|
||||||
|
assert human_count(0) == "0"
|
||||||
|
|
||||||
|
def test_position_argument_ignored(self):
|
||||||
|
"""Test that _pos argument is ignored."""
|
||||||
|
assert human_count(1000, 0) == "1k"
|
||||||
|
|
||||||
|
|
||||||
|
class TestPercentileTable:
|
||||||
|
"""Tests for percentile_table function."""
|
||||||
|
|
||||||
|
def test_basic_percentiles(self):
|
||||||
|
"""Test basic percentile table generation."""
|
||||||
|
df = pd.DataFrame({
|
||||||
|
"metric": ["read_clat"] * 3,
|
||||||
|
"latency_ns": [1000, 2000, 3000],
|
||||||
|
"count": [1, 2, 1],
|
||||||
|
})
|
||||||
|
result = percentile_table(df, "read_clat", "us", [50, 90])
|
||||||
|
assert "Metric: read_clat" in result
|
||||||
|
assert "total I/Os: 4" in result
|
||||||
|
assert "percentile" in result
|
||||||
|
assert "latency" in result
|
||||||
|
|
||||||
|
def test_missing_metric(self):
|
||||||
|
"""Test percentile table with missing metric."""
|
||||||
|
df = pd.DataFrame({
|
||||||
|
"metric": ["read_clat"],
|
||||||
|
"latency_ns": [1000],
|
||||||
|
"count": [1],
|
||||||
|
})
|
||||||
|
result = percentile_table(df, "write_clat", "us", [50])
|
||||||
|
assert "(no data for metric write_clat)" in result
|
||||||
|
|
||||||
|
def test_unit_display(self):
|
||||||
|
"""Test that correct unit is displayed."""
|
||||||
|
df = pd.DataFrame({
|
||||||
|
"metric": ["read_clat"],
|
||||||
|
"latency_ns": [1000000], # 1ms in ns
|
||||||
|
"count": [1],
|
||||||
|
})
|
||||||
|
result = percentile_table(df, "read_clat", "ms", [50])
|
||||||
|
assert "ms" in result
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
"""Tests for the orchestrator module."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import pytest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import AsyncMock, patch
|
||||||
|
|
||||||
|
from disk_checker.core.orchestrator import Orchestrator, ToolStatus, JobResult
|
||||||
|
|
||||||
|
|
||||||
|
class TestOrchestrator:
|
||||||
|
"""Tests for Orchestrator class."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def orchestrator(self, tmp_path):
|
||||||
|
"""Create an orchestrator instance."""
|
||||||
|
return Orchestrator(work_dir=tmp_path)
|
||||||
|
|
||||||
|
def test_initial_state(self, orchestrator):
|
||||||
|
"""Test initial orchestrator state."""
|
||||||
|
assert orchestrator.list_jobs() == {}
|
||||||
|
|
||||||
|
def test_run_fio_creates_job(self, orchestrator, tmp_path):
|
||||||
|
"""Test that run_fio creates a job entry."""
|
||||||
|
with patch("asyncio.create_subprocess_exec") as mock_exec:
|
||||||
|
mock_exec.return_value = AsyncMock()
|
||||||
|
result = asyncio.get_event_loop().run_until_complete(
|
||||||
|
orchestrator.run_fio("/dev/sda")
|
||||||
|
)
|
||||||
|
|
||||||
|
jobs = orchestrator.list_jobs()
|
||||||
|
assert len(jobs) == 1
|
||||||
|
job = list(jobs.values())[0]
|
||||||
|
assert job.status == ToolStatus.COMPLETED
|
||||||
|
# Note: output_path may not exist in test environment without actual fio
|
||||||
|
|
||||||
|
def test_get_job_status(self, orchestrator, tmp_path):
|
||||||
|
"""Test retrieving job status."""
|
||||||
|
with patch("asyncio.create_subprocess_exec") as mock_exec:
|
||||||
|
mock_exec.return_value = AsyncMock()
|
||||||
|
result = asyncio.get_event_loop().run_until_complete(
|
||||||
|
orchestrator.run_fio("/dev/sda")
|
||||||
|
)
|
||||||
|
# result is the output path, job_id is derived from it
|
||||||
|
job_id = f"fio_{result.stem.split('_')[-1]}"
|
||||||
|
# Get the actual job_id from the jobs dict
|
||||||
|
jobs = orchestrator.list_jobs()
|
||||||
|
job_id = list(jobs.keys())[0]
|
||||||
|
|
||||||
|
job = orchestrator.get_job_status(job_id)
|
||||||
|
assert job is not None
|
||||||
|
assert job.status == ToolStatus.COMPLETED
|
||||||
|
|
||||||
|
def test_get_nonexistent_job(self, orchestrator):
|
||||||
|
"""Test getting status of non-existent job."""
|
||||||
|
job = orchestrator.get_job_status("nonexistent")
|
||||||
|
assert job is None
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
"""Tests for the plotting module."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
import matplotlib
|
||||||
|
matplotlib.use("Agg")
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
|
||||||
|
from disk_checker.plotting.plots import draw_histogram, draw_cdf, generate_plot
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
|
||||||
|
class TestDrawHistogram:
|
||||||
|
"""Tests for draw_histogram function."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_df(self):
|
||||||
|
"""Create a sample DataFrame for testing."""
|
||||||
|
return pd.DataFrame({
|
||||||
|
"metric": ["read_clat", "read_clat", "write_clat"],
|
||||||
|
"latency_ns": [1000, 2000, 1500],
|
||||||
|
"count": [1, 2, 1],
|
||||||
|
})
|
||||||
|
|
||||||
|
def test_draws_histogram(self, sample_df, tmp_path):
|
||||||
|
"""Test that histogram is drawn without errors."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_histogram(ax, sample_df, ["read_clat"])
|
||||||
|
assert ax.get_title() == "Completion latency distribution"
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
def test_multiple_metrics(self, sample_df, tmp_path):
|
||||||
|
"""Test drawing multiple metrics."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_histogram(ax, sample_df, ["read_clat", "write_clat"])
|
||||||
|
assert len(ax.collections) > 0
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
def test_log_scale(self, sample_df, tmp_path):
|
||||||
|
"""Test logarithmic scale."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_histogram(ax, sample_df, ["read_clat"], log_x=True, log_y=True)
|
||||||
|
assert ax.get_xscale() == "log"
|
||||||
|
assert ax.get_yscale() == "log"
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
|
||||||
|
class TestDrawCdf:
|
||||||
|
"""Tests for draw_cdf function."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_df(self):
|
||||||
|
"""Create a sample DataFrame for testing."""
|
||||||
|
return pd.DataFrame({
|
||||||
|
"metric": ["read_clat", "read_clat"],
|
||||||
|
"latency_ns": [1000, 2000],
|
||||||
|
"count": [1, 3],
|
||||||
|
})
|
||||||
|
|
||||||
|
def test_draws_cdf(self, sample_df, tmp_path):
|
||||||
|
"""Test that CDF is drawn without errors."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_cdf(ax, sample_df, ["read_clat"])
|
||||||
|
assert ax.get_title() == "Cumulative latency distribution"
|
||||||
|
assert ax.get_ylim() == (0, 100)
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
def test_log_scale(self, sample_df, tmp_path):
|
||||||
|
"""Test logarithmic X scale."""
|
||||||
|
fig, ax = plt.subplots()
|
||||||
|
draw_cdf(ax, sample_df, ["read_clat"], log_x=True)
|
||||||
|
assert ax.get_xscale() == "log"
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
|
||||||
|
class TestGeneratePlot:
|
||||||
|
"""Tests for generate_plot function."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_df(self):
|
||||||
|
"""Create a sample DataFrame for testing."""
|
||||||
|
return pd.DataFrame({
|
||||||
|
"metric": ["read_clat", "read_clat"],
|
||||||
|
"latency_ns": [1000, 2000],
|
||||||
|
"count": [1, 3],
|
||||||
|
})
|
||||||
|
|
||||||
|
def test_generates_histogram(self, sample_df, tmp_path):
|
||||||
|
"""Test generating histogram plot."""
|
||||||
|
output = tmp_path / "test.png"
|
||||||
|
result = generate_plot(
|
||||||
|
df=sample_df,
|
||||||
|
metrics=["read_clat"],
|
||||||
|
output_path=output,
|
||||||
|
)
|
||||||
|
assert result == output
|
||||||
|
assert output.exists()
|
||||||
|
|
||||||
|
def test_generates_cdf_only(self, sample_df, tmp_path):
|
||||||
|
"""Test generating CDF-only plot."""
|
||||||
|
output = tmp_path / "test_cdf.png"
|
||||||
|
result = generate_plot(
|
||||||
|
df=sample_df,
|
||||||
|
metrics=["read_clat"],
|
||||||
|
output_path=output,
|
||||||
|
cdf_only=True,
|
||||||
|
)
|
||||||
|
assert result == output
|
||||||
|
assert output.exists()
|
||||||
|
|
||||||
|
def test_generates_histogram_with_cdf(self, sample_df, tmp_path):
|
||||||
|
"""Test generating histogram with CDF below."""
|
||||||
|
output = tmp_path / "test_both.png"
|
||||||
|
result = generate_plot(
|
||||||
|
df=sample_df,
|
||||||
|
metrics=["read_clat"],
|
||||||
|
output_path=output,
|
||||||
|
show_cdf=True,
|
||||||
|
)
|
||||||
|
assert result == output
|
||||||
|
assert output.exists()
|
||||||
|
|
||||||
|
def test_custom_dpi(self, sample_df, tmp_path):
|
||||||
|
"""Test custom DPI setting."""
|
||||||
|
output = tmp_path / "test_dpi.png"
|
||||||
|
result = generate_plot(
|
||||||
|
df=sample_df,
|
||||||
|
metrics=["read_clat"],
|
||||||
|
output_path=output,
|
||||||
|
dpi=300,
|
||||||
|
)
|
||||||
|
assert result == output
|
||||||
|
assert output.exists()
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
"""Tests for the web backend."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import AsyncMock, patch
|
||||||
|
|
||||||
|
from disk_checker.web.server import app
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client():
|
||||||
|
"""Create test client."""
|
||||||
|
return TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_csv_content():
|
||||||
|
"""Sample CSV content for testing."""
|
||||||
|
return """nsec,read_clat_ns_count,read_clat_ns_cumulative,read_clat_ns_percentile
|
||||||
|
19791872,1,1,0.00016775708773695687
|
||||||
|
22413312,4,5,0.0008387854386847844
|
||||||
|
22675456,9,14,0.0023485992283173965
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_csv_path(tmp_path, sample_csv_content):
|
||||||
|
"""Create sample CSV file."""
|
||||||
|
path = tmp_path / "test.csv"
|
||||||
|
path.write_text(sample_csv_content)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def test_health_check(client):
|
||||||
|
"""Test health check endpoint."""
|
||||||
|
response = client.get("/health")
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json()["status"] == "ok"
|
||||||
|
|
||||||
|
|
||||||
|
def test_scan_csv(client, sample_csv_path):
|
||||||
|
"""Test CSV scan endpoint."""
|
||||||
|
with open(sample_csv_path, "rb") as f:
|
||||||
|
response = client.post("/api/v1/scan", files={"file": f})
|
||||||
|
assert response.status_code == 200
|
||||||
|
data = response.json()
|
||||||
|
assert "metrics" in data
|
||||||
|
assert len(data["metrics"]) > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_plot_endpoint(client, sample_csv_path):
|
||||||
|
"""Test plot generation endpoint."""
|
||||||
|
with open(sample_csv_path, "rb") as f:
|
||||||
|
response = client.post(
|
||||||
|
"/api/v1/plot",
|
||||||
|
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||||
|
data={"unit": "us"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.headers["content-type"] == "image/png"
|
||||||
|
|
||||||
|
|
||||||
|
def test_plot_with_options(client, sample_csv_path):
|
||||||
|
"""Test plot generation with options."""
|
||||||
|
with open(sample_csv_path, "rb") as f:
|
||||||
|
response = client.post(
|
||||||
|
"/api/v1/plot",
|
||||||
|
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||||
|
data={
|
||||||
|
"metric": "read_clat",
|
||||||
|
"unit": "ms",
|
||||||
|
"log_x": "true",
|
||||||
|
"cdf": "true",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_metric(client, sample_csv_path):
|
||||||
|
"""Test invalid metric returns error."""
|
||||||
|
with open(sample_csv_path, "rb") as f:
|
||||||
|
response = client.post(
|
||||||
|
"/api/v1/plot",
|
||||||
|
files={"file": (sample_csv_path.name, f, "text/csv")},
|
||||||
|
data={"metric": "invalid_metric", "unit": "us"},
|
||||||
|
)
|
||||||
|
# Note: endpoint returns 200 and auto-selects best metric when invalid
|
||||||
|
assert response.status_code == 200
|
||||||
Reference in New Issue
Block a user