1 Commits

Author SHA1 Message Date
max 0c77c19368 feat(repo): basic issue templates added 2026-09-30 21:34:23 +04:00
38 changed files with 331 additions and 2273 deletions
+97
View File
@@ -0,0 +1,97 @@
name: "Bug Report"
description: "Report a bug or unexpected behavior to help us improve"
title: "[BUG]: "
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
## ⚠️ Before You Begin
Please ensure you have:
- [ ] Searched existing issues to avoid duplicates
- [ ] Confirmed this is a bug, not a question or configuration problem
- type: textarea
id: description
attributes:
label: "📋 Description"
description: "Provide a clear and concise description of the bug"
placeholder: "What happened? What did you expect to happen instead?"
validations:
required: true
- type: textarea
id: reproduction
attributes:
label: "🔁 Steps to Reproduce"
description: "Step-by-step instructions to reproduce the issue"
placeholder: |
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
validations:
required: true
- type: textarea
id: expected
attributes:
label: "✅ Expected Behavior"
description: "What you expected to happen"
placeholder: "A clear description of what should happen..."
validations:
required: true
- type: textarea
id: actual
attributes:
label: "❌ Actual Behavior"
description: "What actually happened"
placeholder: "Include error messages, stack traces, or unexpected outcomes..."
validations:
required: true
- type: textarea
id: logs
attributes:
label: "📄 Logs / Screenshots"
description: |
Provide relevant logs, error messages, or screenshots.
For logs, please use a pastebin and share the URL.
**Remember to remove any sensitive information (API keys, passwords, etc.).**
placeholder: "Paste logs here or provide a Gist URL..."
validations:
required: false
- type: markdown
attributes:
value: |
---
## 🖥️ Environment Details
Please fill out the relevant information below.
- type: textarea
id: additional
attributes:
label: "📎 Additional Context"
description: "Any other information that might be relevant"
placeholder: |
- Database type and version (e.g., PostgreSQL 15, SQLite)
- Reverse proxy/CDN in use (e.g., Nginx, Cloudflare)
- Any custom configuration
- Related issues or PRs
validations:
required: false
- type: checkboxes
id: checklist
attributes:
label: "✅ Submission Checklist"
description: "Please confirm the following before submitting"
options:
- label: "I have searched for existing issues (open and closed) that report the same problem"
required: true
- label: "I have provided clear steps to reproduce the issue"
required: true
- label: "I have included relevant logs or error messages (with sensitive info removed)"
required: false
@@ -0,0 +1,99 @@
name: "Feature Request"
description: "Suggest a new feature or enhancement for this project"
title: "[FEATURE]: "
labels: ["enhancement"]
body:
- type: markdown
attributes:
value: |
## ⚠️ Before You Begin
Please ensure you have:
- [ ] Searched existing issues (open and closed) to avoid duplicates
- [ ] Reviewed the [project documentation](https://git.vmn.su/max/vmn-ansible/wiki) for existing functionality
- [ ] Confirmed this is a feature request, not a bug or configuration question
- type: textarea
id: problem
attributes:
label: "🎯 Problem Statement"
description: "What problem does this feature solve? What can't you do today?"
placeholder: |
I'm always frustrated when...
Currently, it's hard/impossible to...
This feature would help me because...
validations:
required: true
- type: textarea
id: solution
attributes:
label: "💡 Proposed Solution"
description: "Describe the solution you'd like to see"
placeholder: |
A clear description of what you want to happen.
If you have a specific API, UI, or implementation in mind, describe it here.
validations:
required: true
- type: textarea
id: screenshots
attributes:
label: "📸 Screenshots"
description: "If you can, provide screenshots or diagrams of the proposed feature"
placeholder: "Drag and drop images or paste links..."
validations:
required: false
- type: textarea
id: technical
attributes:
label: "⚙️ Technical Approach"
description: "If you have ideas about how to implement this, share them here"
placeholder: |
- Which components/modules would be affected?
- Any database schema changes?
- API design considerations?
validations:
required: false
- type: textarea
id: acceptance
attributes:
label: "✅ Acceptance Criteria"
description: "What does 'done' look like for this feature?"
placeholder: |
- [ ] Feature works in the UI
- [ ] API endpoints are documented
- [ ] Tests are added
- [ ] Documentation is updated
validations:
required: false
- type: dropdown
id: contribution
attributes:
label: "🤝 Are you willing to contribute this feature?"
description: "Knowing if you can help implement this helps us prioritize"
options:
- "Yes, I can contribute"
- "Yes, but I need guidance"
- "No, but I can test/review"
- "No, I'm just requesting"
validations:
required: true
- type: checkboxes
id: checklist
attributes:
label: "✅ Submission Checklist"
description: "Please confirm the following before submitting"
options:
- label: "I have searched for existing issues (open and closed) that request the same feature"
required: true
- label: "I have explained the problem this feature solves"
required: true
- label: "I have described a concrete solution or direction"
required: true
- label: "I have considered alternatives and explained why they fall short"
required: false
+134
View File
@@ -0,0 +1,134 @@
name: "Refactor Request"
description: "Propose a code restructuring or cleanup with no behavior change"
title: "[REFACTOR]: "
labels: ["refactoring"]
body:
- type: markdown
attributes:
value: |
## ⚠️ Before You Begin
Please ensure you have:
- [ ] Searched existing issues (open and closed) to avoid duplicates
- [ ] Reviewed the [project documentation](https://git.vmn.su/max/vmn-ansible/wiki) for architecture guidelines
- [ ] Confirmed this is a refactor (code restructuring with NO behavior change), not a bug fix or new feature
- type: textarea
id: problem
attributes:
label: "🎯 Problem Statement"
description: "What pain points does the current code cause? Why is this refactor needed?"
placeholder: |
The current implementation is difficult to maintain because...
This code has accumulated technical debt due to...
I'm constantly frustrated when working with this module because...
validations:
required: true
- type: textarea
id: scope
attributes:
label: "📂 Scope"
description: "Which files, modules, or components need refactoring? Be specific."
placeholder: |
- File paths: `src/auth/`, `internal/handler/`
- Component: `UserService` class
validations:
required: true
- type: textarea
id: current-state
attributes:
label: "🔍 Current State"
description: "Describe the current code structure, its issues, and why it's problematic"
placeholder: |
- The function `processUser()` is 500+ lines long
- Duplicated logic across 3 different files
- Mixed concerns (business logic + HTTP handling)
- Poor test coverage makes changes risky
- Naming is unclear and inconsistent
validations:
required: true
- type: textarea
id: proposed-changes
attributes:
label: "💡 Proposed Solution"
description: "What should the code look like after the refactor?"
placeholder: |
- Extract `processUser()` into smaller, focused functions
- Consolidate duplicated logic into a shared utility
- Separate business logic from HTTP handlers
- Introduce interfaces for better testability
- Rename variables and functions for clarity
validations:
required: true
- type: textarea
id: constraints
attributes:
label: "🚫 Constraints"
description: "What must NOT change? Public APIs, behavior, backward compatibility, performance characteristics?"
placeholder: |
- All public APIs must remain identical
- Database schema must not change
- External behavior must be identical
- Response formats must stay the same
- Performance must not degrade
validations:
required: true
- type: textarea
id: benefits
attributes:
label: "📈 Expected Benefits"
description: "What improvements will this refactor bring?"
placeholder: |
- Improved maintainability
- Better testability
- Reduced code duplication
- Clearer separation of concerns
- Easier onboarding for new contributors
validations:
required: false
- type: textarea
id: risks
attributes:
label: "⚠️ Risks & Mitigations"
description: "What could go wrong, and how will you mitigate it?"
placeholder: |
- Risk: Regression bugs
Mitigation: Comprehensive test coverage before and after
- Risk: Large diff making code review difficult
Mitigation: Break into smaller, incremental PRs
validations:
required: false
- type: dropdown
id: contribution
attributes:
label: "🤝 Are you willing to contribute this refactor?"
description: "Knowing if you can help implement this helps us prioritize"
options:
- "Yes, I can contribute"
- "Yes, but I need guidance"
- "No, but I can review"
- "No, I'm just suggesting"
validations:
required: true
- type: checkboxes
id: checklist
attributes:
label: "✅ Submission Checklist"
description: "Please confirm the following before submitting"
options:
- label: "I have searched for existing issues (open and closed) that request the same refactor"
required: true
- label: "I have clearly explained the problem this refactor solves"
required: true
- label: "I have described the scope of the refactor (specific files/modules)"
required: false
- label: "I have identified what must NOT change (APIs, behavior, backward compatibility)"
required: false
-49
View File
@@ -1,52 +1,3 @@
.vscode .vscode
__pycache__ __pycache__
test_data test_data
# Python
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Virtual environments
venv/
ENV/
env/
# IDE
.idea/
*.swp
*.swo
# Environment variables
.env
.env.local
# Test coverage
.htmlcov/
.coverage
.coverage.*
coverage.xml
# Pytest
.pytest_cache/
# Generated files
*.png
*.csv
data/
-78
View File
@@ -1,78 +0,0 @@
# Disk Checker
Disk benchmarking and latency visualization tool for fio and badblocks output.
## Features
- Parse fio JSON+ latency CSV files
- Generate histogram and CDF plots
- REST API for remote benchmarking
- Async job management for long-running benchmarks
## Installation
```bash
git clone <repository>
cd disk-checker
python -m venv venv
source venv/bin/activate
pip install -e ".[dev]"
```
## CLI Usage
```bash
# Basic usage
python -m disk_checker scan_latency.csv
# With options
python -m disk_checker scan_latency.csv --metric read_clat --unit ms --log-x
# Generate CDF
python -m disk_checker scan_latency.csv --cdf
# View all metrics
python -m disk_checker scan_latency.csv --all-metrics
```
## API Usage
```bash
# Start the server
python -m disk_checker.scripts.serve --host 0.0.0.0 --port 8000
# Access API docs at http://localhost:8000/docs
```
### API Endpoints
- `POST /api/v1/scan` - Upload CSV and get analysis
- `POST /api/v1/plot` - Generate and return plot image
- `POST /api/v1/benchmark` - Start a benchmark job
- `GET /api/v1/jobs/{id}` - Check job status
- `GET /health` - Health check
## Architecture
```
disk-checker/
├── src/disk_checker/
│ ├── core/ # Business logic (extractor, converter, helpers, orchestrator)
│ ├── plotting/ # Visualization (matplotlib)
│ ├── web/ # FastAPI backend
│ └── scripts/ # Entry points (cli, serve)
├── tests/ # Test suite
└── data/ # Sample data
```
## Testing
```bash
pytest
pytest --cov=disk_checker
pytest --cov=disk_checker --cov-report=html
```
## License
MIT
-53
View File
@@ -1,53 +0,0 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "disk-checker"
version = "0.1.0"
description = "Disk benchmarking and latency visualization tool"
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"pandas>=2.0",
"numpy>=1.24",
"matplotlib>=3.7",
"fastapi>=0.100",
"uvicorn[standard]>=0.23",
"python-multipart>=0.0.6",
]
[project.optional-dependencies]
dev = [
"pytest>=7.0",
"pytest-asyncio>=0.21",
"pytest-cov>=4.0",
"black>=23.0",
"ruff>=0.1.0",
"mypy>=1.0",
]
[project.scripts]
disk-checker = "disk_checker.scripts.cli:main"
disk-checker-serve = "disk_checker.scripts.serve:main"
[tool.hatch.build.targets.wheel]
packages = ["src/disk_checker"]
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_functions = ["test_*"]
[tool.black]
line-length = 100
target-version = ["py310"]
[tool.ruff]
line-length = 100
target-version = "py310"
[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true
-3
View File
@@ -1,3 +0,0 @@
"""Disk Checker - Disk benchmarking and latency visualization tool."""
__version__ = "0.1.0"
-6
View File
@@ -1,6 +0,0 @@
"""Allow running disk-checker as a module: python -m disk_checker"""
from disk_checker.scripts.cli import main
if __name__ == "__main__":
main()
-14
View File
@@ -1,14 +0,0 @@
"""Core business logic module."""
from .extractor import load_latency_csv
from .converter import to_unit, UNIT_FACTORS, UNIT_LABELS
from .helpers import human_count, percentile_table
__all__ = [
"load_latency_csv",
"to_unit",
"UNIT_FACTORS",
"UNIT_LABELS",
"human_count",
"percentile_table",
]
-32
View File
@@ -1,32 +0,0 @@
"""Unit conversion utilities for latency data."""
from typing import Dict
import numpy as np
UNIT_FACTORS: Dict[str, float] = {
"ns": 1.0,
"us": 1e3,
"ms": 1e6,
"s": 1e9,
}
UNIT_LABELS: Dict[str, str] = {
"ns": "ns",
"us": "µs",
"ms": "ms",
"s": "s",
}
def to_unit(values_ns, unit: str) -> np.ndarray:
"""
Convert nanosecond values to the specified unit.
Args:
values_ns: Values in nanoseconds
unit: Target unit (ns, us, ms, s)
Returns:
Converted values as numpy array
"""
return np.asarray(values_ns, dtype=float) / UNIT_FACTORS[unit]
-130
View File
@@ -1,130 +0,0 @@
"""Drive detection module for disk-checker."""
import json
import subprocess
from typing import Dict, List
from dataclasses import dataclass
from pathlib import Path
@dataclass
class DriveInfo:
"""Information about a block device."""
name: str # e.g., "sda", "sdb"
path: str # e.g., "/dev/sda"
size: str # e.g., "232.9G"
model: str # e.g., "CT250MX500SSD1"
type: str # e.g., "disk", "part"
mounted: bool # True if currently mounted
mount_point: str = "" # e.g., "/", "/home"
serial: str = "" # e.g., "2131G5F25137"
def get_mounted_devices() -> set:
"""
Get set of mounted device paths.
Returns:
Set of device paths that are currently mounted
"""
mounted = set()
# Method 1: Parse lsblk with mount points
try:
result = subprocess.run(
["lsblk", "-J", "-o", "NAME,MOUNTPOINT"],
capture_output=True,
text=True,
check=True
)
data = json.loads(result.stdout)
for device in data.get("blockdevices", []):
mountpoint = device.get("mountpoint")
if mountpoint:
# Add both full path and device name
mounted.add(f"/dev/{device['name']}")
mounted.add(device["name"])
except (subprocess.CalledProcessError, json.JSONDecodeError):
print("lsblk failed to retrieve mounted devices information")
pass
# Method 2: Parse /proc/mounts
try:
with open("/proc/mounts", "r") as f:
for line in f:
parts = line.split()
if len(parts) >= 2 and parts[0].startswith("/dev/"):
mounted.add(parts[0])
# Also add device name without /dev/
mounted.add(parts[0].replace("/dev/", ""))
except (FileNotFoundError, IOError):
print("/proc/mounts failed to retrieve mounted devices information")
pass
return mounted
def scan_drives() -> List[DriveInfo]:
"""
Scan host system for available block devices.
Returns:
List of DriveInfo objects for physical disks
"""
mounted_devices = get_mounted_devices()
drives = []
try:
# Get block devices with lsblk
result = subprocess.run(
[
"lsblk",
"-J", # JSON output
"-d", # Only disks (not partitions)
"-b", # Show size in bytes
"-o", "NAME,SIZE,TYPE,MODEL,Serial,MOUNTPOINT"
],
capture_output=True,
text=True,
check=True
)
data = json.loads(result.stdout)
for device in data.get("blockdevices", []):
# Only include physical disks
if device.get("type") != "disk":
continue
name = device.get("name", "")
path = f"/dev/{name}"
# Check if mounted
is_mounted = False
#is_mounted = path in mounted_devices or name in mounted_devices
for md in mounted_devices:
is_mounted |= path in md
drive = DriveInfo(
name=name,
path=path,
size=device.get("size", "Unknown"),
model=device.get("model") or "",
type="disk",
mounted=is_mounted,
mount_point=device.get("mountpoint", ""),
serial=device.get("serial") or ""
)
drives.append(drive)
except subprocess.CalledProcessError as e:
# If lsblk fails, return empty list
print("lsblk failed to parse drives data")
print(e)
pass
except json.JSONDecodeError as e:
print("json decoder error")
print(e)
pass
return drives
-88
View File
@@ -1,88 +0,0 @@
"""CSV data extraction module for fio latency data."""
from pathlib import Path
import pandas as pd
from typing import Optional
def load_latency_csv(path: str) -> pd.DataFrame:
"""
Parse wide-format fio JSON+ CSV into long-format DataFrame.
Transforms wide-format columns (e.g., read_clat_ns_count) into rows with
a metric identifier, making it easier to group and analyze by operation type.
Args:
path: Path to CSV file generated by fio_jsonplus_clat2csv
Returns:
DataFrame with columns: metric, latency_ns, count, cumulative, percentile
Raises:
ValueError: If CSV format is invalid or contains no valid data
"""
df = pd.read_csv(path, skipinitialspace=True)
df.columns = [str(c).strip() for c in df.columns]
# Drop trailing unnamed / empty columns
df = df.loc[:, ~df.columns.str.match(r"^Unnamed")]
df = df.loc[:, df.columns != ""]
nsec_col = df.columns[0]
if "nsec" not in nsec_col.lower():
raise ValueError(
f"expected first column to be 'nsec', got {nsec_col!r}"
)
# Coerce the nsec column to numeric
df[nsec_col] = pd.to_numeric(df[nsec_col], errors="coerce")
df = df.dropna(subset=[nsec_col])
# Discover metric triples: <metric>_count / _cumulative / _percentile
metrics = {}
for col in df.columns:
if col == nsec_col:
continue
for suffix in ("_count", "_cumulative", "_percentile"):
if col.endswith(suffix):
base = col[: -len(suffix)]
metrics.setdefault(base, {})[suffix[1:]] = col
if not metrics:
raise ValueError("no *_count columns found in CSV")
frames = []
for base, cols in metrics.items():
count_col = cols["count"]
# Coerce count to numeric
counts = pd.to_numeric(df[count_col], errors="coerce")
# Keep only rows where the count column actually has a number
mask = counts.notna()
if not mask.any():
continue
sub = pd.DataFrame({
"latency_ns": df.loc[mask, nsec_col].values,
"count": counts[mask].astype(int).values,
})
if "cumulative" in cols:
sub["cumulative"] = pd.to_numeric(
df.loc[mask, cols["cumulative"]], errors="coerce"
).values
if "percentile" in cols:
sub["percentile"] = pd.to_numeric(
df.loc[mask, cols["percentile"]], errors="coerce"
).values
# read_clat_ns -> read_clat
sub["metric"] = base.replace("_ns", "")
frames.append(sub)
if not frames:
raise ValueError("no metric had any numeric count data")
out = pd.concat(frames, ignore_index=True)
out = out[out["count"] > 0].reset_index(drop=True)
return out
-62
View File
@@ -1,62 +0,0 @@
"""Helper functions for formatting and display."""
from typing import List
import numpy as np
import pandas as pd
from matplotlib.ticker import FuncFormatter
from .converter import to_unit, UNIT_LABELS
def human_count(x, _pos=None) -> str:
"""
Format large numbers with K/M/G suffixes for axis labels.
Args:
x: Numeric value to format
_pos: Matplotlib tick position (unused)
Returns:
Formatted string (e.g., "1.5M", "3.2G")
"""
if x >= 1e9:
return f"{x/1e9:.1f}G"
if x >= 1e6:
return f"{x/1e6:.1f}M"
if x >= 1e3:
return f"{x/1e3:.0f}k"
return f"{x:.0f}"
def percentile_table(df: pd.DataFrame, metric: str, unit: str, pcts: List[float]) -> str:
"""
Generate a formatted table showing latency values at specified percentiles.
Args:
df: Long-format DataFrame from load_latency_csv
metric: Metric name (e.g., 'read_clat')
unit: Display unit (ns, us, ms, s)
pcts: List of percentile values (e.g., [50, 90, 99])
Returns:
Formatted string table
"""
g = df[df["metric"] == metric].sort_values("latency_ns")
if g.empty:
return f"(no data for metric {metric})"
counts = g["count"].values.astype(float)
total = int(counts.sum())
cdf = np.cumsum(counts) / total * 100.0
lat = to_unit(g["latency_ns"].values, unit)
lines = [
f"\nMetric: {metric} (total I/Os: {total:,})",
f" {'percentile':>12} {'latency':>14}",
]
for p in pcts:
idx = min(np.searchsorted(cdf, p), len(lat) - 1)
lines.append(f" {p:>11.2f}% {lat[idx]:>12.3f} {UNIT_LABELS[unit]}")
return "\n".join(lines)
-164
View File
@@ -1,164 +0,0 @@
"""Tool orchestration module for running disk benchmarks."""
import asyncio
import subprocess
from dataclasses import dataclass, field
from enum import Enum
from pathlib import Path
from typing import Dict, Optional
class ToolStatus(Enum):
"""Status of a benchmark job."""
PENDING = "pending"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
@dataclass
class JobResult:
"""Result of a benchmark job."""
job_id: str
status: ToolStatus
output_path: Optional[Path] = None
error: Optional[str] = None
metadata: Dict = field(default_factory=dict)
class Orchestrator:
"""
Orchestrates disk benchmarking tools (fio, badblocks, etc.)
Provides async interfaces for running benchmarks and collecting results.
"""
def __init__(self, work_dir: Path = Path(".")):
self.work_dir = work_dir
self._jobs: Dict[str, JobResult] = {}
async def run_fio(
self,
device: str,
job_file: Optional[Path] = None,
output_format: str = "jsonplus",
**kwargs,
) -> Path:
"""
Run fio benchmark and return path to output file.
Args:
device: Block device path (e.g., '/dev/sda')
job_file: Optional fio job file
output_format: Output format (json, jsonplus, csv)
**kwargs: Additional fio options
Returns:
Path to output file
Raises:
subprocess.CalledProcessError: If fio command fails
"""
job_id = f"fio_{asyncio.get_event_loop().time()}"
output_path = self.work_dir / f"{job_id}.{output_format}"
cmd = ["fio", "--output-format=jsonplus"]
if job_file:
cmd.extend(["--filename", str(device)])
else:
cmd.extend(["--name=test", "--rw=randread", "--bs=4k"])
cmd.extend(["--output", str(output_path)])
cmd.extend([f"--output-format={output_format}"])
for k, v in kwargs.items():
cmd.extend([f"--{k}={v}"])
self._jobs[job_id] = JobResult(
job_id=job_id,
status=ToolStatus.RUNNING,
output_path=output_path,
)
try:
await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
self._jobs[job_id] = JobResult(
job_id=job_id,
status=ToolStatus.COMPLETED,
output_path=output_path,
)
return output_path
except subprocess.CalledProcessError as e:
self._jobs[job_id] = JobResult(
job_id=job_id,
status=ToolStatus.FAILED,
error=str(e),
)
raise
async def run_badblocks(
self,
device: str,
blocks_count: Optional[int] = None,
**kwargs,
) -> Path:
"""
Run badblocks scan on device.
Args:
device: Block device path
blocks_count: Number of blocks to scan
**kwargs: Additional badblocks options
Returns:
Path to badblocks output file
"""
job_id = f"badblocks_{asyncio.get_event_loop().time()}"
output_path = self.work_dir / f"{job_id}.txt"
cmd = ["badblocks", "-w", "-s", "-p", "0", "-t", "RANDOM", "-o"]
cmd.extend([str(output_path), str(device)])
if blocks_count:
cmd.extend(["-b", str(blocks_count)])
for k, v in kwargs.items():
cmd.extend([f"-{k}", str(v)])
self._jobs[job_id] = JobResult(
job_id=job_id,
status=ToolStatus.RUNNING,
output_path=output_path,
)
try:
await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
self._jobs[job_id] = JobResult(
job_id=job_id,
status=ToolStatus.COMPLETED,
output_path=output_path,
)
return output_path
except subprocess.CalledProcessError as e:
self._jobs[job_id] = JobResult(
job_id=job_id,
status=ToolStatus.FAILED,
error=str(e),
)
raise
def get_job_status(self, job_id: str) -> Optional[JobResult]:
"""Get status of a submitted job."""
return self._jobs.get(job_id)
def list_jobs(self) -> Dict[str, JobResult]:
"""List all jobs."""
return self._jobs.copy()
-57
View File
@@ -1,57 +0,0 @@
"""Tool detection module for disk-checker."""
import shutil
from typing import Dict, List
from dataclasses import dataclass
@dataclass
class ToolInfo:
"""Information about a benchmarking tool."""
name: str
available: bool
version: str = ""
# Tools to scan for
TOOLS_TO_SCAN = [
"fio",
"badblocks",
"smartctl",
"iozone",
"hdparm",
]
def scan_tools() -> Dict[str, ToolInfo]:
"""
Scan host system for available benchmarking tools.
Returns:
Dictionary mapping tool names to ToolInfo objects
"""
results = {}
for tool_name in TOOLS_TO_SCAN:
available = shutil.which(tool_name) is not None
version = ""
if available:
# Try to get version (optional enhancement)
try:
import subprocess
result = subprocess.run(
[tool_name, "--version"],
capture_output=True,
text=True,
timeout=2
)
version = result.stdout.split("\n")[0][:50]
except Exception:
# If we can't even run '--version', then
# a tool is not considered as available
available = False
results[tool_name] = ToolInfo(
name=tool_name,
available=available,
version=version
)
return results
-5
View File
@@ -1,5 +0,0 @@
"""Plotting module for disk latency visualization."""
from .plots import draw_histogram, draw_cdf, generate_plot
__all__ = ["draw_histogram", "draw_cdf", "generate_plot"]
-162
View File
@@ -1,162 +0,0 @@
"""Plotting module for disk latency visualization."""
import matplotlib
matplotlib.use("Agg") # Non-interactive backend for server environments
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter
import numpy as np
from pathlib import Path
from typing import List, Optional
from disk_checker.core.converter import to_unit, UNIT_LABELS
from disk_checker.core.helpers import human_count
def draw_histogram(
ax,
df,
metrics: List[str],
unit: str = "us",
log_y: bool = False,
log_x: bool = False,
normalize: bool = False,
) -> None:
"""
Draw a histogram showing the distribution of latencies for each metric.
Uses a step plot (where="post") to show the discrete nature of the data.
If normalize=True, Y-axis shows probability density instead of counts.
Args:
ax: Matplotlib axis object
df: Long-format DataFrame from load_latency_csv
metrics: List of metric names to plot
unit: X-axis unit (ns, us, ms, s)
log_y: Use logarithmic Y scale
log_x: Use logarithmic X scale
normalize: Normalize to probability density
"""
for metric in metrics:
g = df[df["metric"] == metric].sort_values("latency_ns")
if g.empty:
continue
x = to_unit(g["latency_ns"].values, unit)
y = g["count"].values.astype(float)
if normalize:
s = y.sum()
if s > 0:
y = y / s
ax.step(x, y, where="post", lw=1.4, label=metric)
ax.fill_between(x, y, step="post", alpha=0.25)
ax.set_xlabel(f"Latency ({UNIT_LABELS[unit]})")
ax.set_ylabel("Probability" if normalize else "Number of I/Os")
ax.set_title("Completion latency distribution")
ax.grid(True, which="both", ls=":", alpha=0.5)
if not normalize:
ax.yaxis.set_major_formatter(FuncFormatter(human_count))
if log_y:
ax.set_yscale("log")
if log_x:
ax.set_xscale("log")
if len(metrics) > 1:
ax.legend(title="metric", fontsize=9)
def draw_cdf(ax, df, metrics: List[str], unit: str = "us", log_x: bool = False) -> None:
"""
Draw a Cumulative Distribution Function (CDF).
Shows the percentage of I/Os completed within each latency threshold.
Each metric gets its own line; Y-axis ranges from 0% to 100%.
Args:
ax: Matplotlib axis object
df: Long-format DataFrame
metrics: List of metric names
unit: X-axis unit
log_x: Use logarithmic X scale
"""
for metric in metrics:
g = df[df["metric"] == metric].sort_values("latency_ns")
if g.empty:
continue
x = to_unit(g["latency_ns"].values, unit)
counts = g["count"].values.astype(float)
total = counts.sum()
if total <= 0:
continue
cdf = np.cumsum(counts) / total * 100.0
ax.plot(x, cdf, lw=1.6, label=f"{metric} (n={int(total):,})")
ax.set_xlabel(f"Latency ({UNIT_LABELS[unit]})")
ax.set_ylabel("Cumulative % of I/Os")
ax.set_title("Cumulative latency distribution")
ax.set_ylim(0, 100)
ax.grid(True, which="both", ls=":", alpha=0.5)
ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:.0f}%"))
if log_x:
ax.set_xscale("log")
ax.legend(fontsize=9)
def generate_plot(
df,
metrics: List[str],
output_path: Path,
unit: str = "us",
log_y: bool = False,
log_x: bool = False,
normalize: bool = False,
show_cdf: bool = False,
cdf_only: bool = False,
title: str = "",
dpi: int = 150,
) -> Path:
"""
Generate a complete plot with histogram and/or CDF.
Creates the appropriate subplot layout based on the options provided.
Args:
df: Long-format DataFrame
metrics: Metrics to plot
output_path: Where to save the plot
unit: X-axis unit
log_y: Logarithmic Y scale
log_x: Logarithmic X scale
normalize: Normalize histogram
show_cdf: Show CDF below histogram
cdf_only: Show only CDF
title: Plot title
dpi: Output resolution
Returns:
Path to saved plot
"""
if cdf_only:
fig, ax = plt.subplots(figsize=(11, 6))
draw_cdf(ax, df, metrics, unit, log_x)
elif show_cdf:
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(11, 10))
draw_histogram(
ax1, df, metrics, unit, log_y, log_x, normalize
)
draw_cdf(ax2, df, metrics, unit, log_x)
else:
fig, ax = plt.subplots(figsize=(11, 6))
draw_histogram(ax, df, metrics, unit, log_y, log_x, normalize)
if title:
fig.suptitle(title, fontsize=13, y=0.995)
fig.tight_layout()
fig.savefig(output_path, dpi=dpi, bbox_inches="tight")
plt.close(fig)
return output_path
-97
View File
@@ -1,97 +0,0 @@
#!/usr/bin/env python3
"""
CLI entry point for disk-checker.
Usage:
python -m disk_checker scan_latency.csv
disk-checker scan_latency.csv --metric read_clat --unit ms
"""
import argparse
import sys
from pathlib import Path
from disk_checker.core.extractor import load_latency_csv
from disk_checker.core.helpers import percentile_table
from disk_checker.plotting.plots import generate_plot
def main():
parser = argparse.ArgumentParser(
description="Plot fio json+ wide-format latency CSV."
)
parser.add_argument("csv", help="CSV from fio_jsonplus_clat2csv")
parser.add_argument("--metric", default=None,
help="Metric to plot (e.g. read_clat). "
"If omitted, the metric with the most I/Os is used.")
parser.add_argument("--all-metrics", action="store_true",
help="Plot every metric that has data")
parser.add_argument("--unit", choices=["ns", "us", "ms", "s"], default="us",
help="X-axis latency unit (default: us)")
parser.add_argument("--log-y", action="store_true")
parser.add_argument("--log-x", action="store_true")
parser.add_argument("--cdf", action="store_true",
help="Add a CDF subplot below the histogram")
parser.add_argument("--cdf-only", action="store_true")
parser.add_argument("--normalize", action="store_true",
help="Normalize histogram to a probability density")
parser.add_argument("--percentiles", default="50,90,99,99.9,99.99")
parser.add_argument("--no-percentiles", action="store_true")
parser.add_argument("--output", "-o", default=None)
parser.add_argument("--dpi", type=int, default=150)
parser.add_argument("--title", default="")
parser.add_argument("--show", action="store_true")
args = parser.parse_args()
if not Path(args.csv).is_file():
sys.exit(f"error: file not found: {args.csv}")
df = load_latency_csv(args.csv)
if df.empty:
sys.exit("error: no latency data in CSV")
available = sorted(df["metric"].unique())
print("Metrics with data:", ", ".join(available))
if args.all_metrics:
metrics = available
elif args.metric:
if args.metric not in available:
sys.exit(f"error: metric {args.metric!r} not in {available}")
metrics = [args.metric]
else:
totals = df.groupby("metric")["count"].sum().sort_values(ascending=False)
metrics = [totals.index[0]]
print(f"Auto-selected metric: {metrics[0]}")
output = args.output or f"{Path(args.csv).stem}_latency.png"
generate_plot(
df=df,
metrics=metrics,
output_path=Path(output),
unit=args.unit,
log_y=args.log_y,
log_x=args.log_x,
normalize=args.normalize,
show_cdf=args.cdf,
cdf_only=args.cdf_only,
title=args.title,
dpi=args.dpi,
)
print(f"Saved plot to {output}")
if not args.no_percentiles:
try:
pcts = [float(x) for x in args.percentiles.split(",") if x.strip()]
except ValueError:
sys.exit("error: --percentiles must be comma-separated numbers")
for m in metrics:
print(percentile_table(df, m, args.unit, pcts))
if args.show:
import matplotlib.pyplot as plt
plt.show()
if __name__ == "__main__":
main()
-32
View File
@@ -1,32 +0,0 @@
"""Startup script for the web server."""
import argparse
import uvicorn
from pathlib import Path
def main():
"""Main entry point for the web server."""
parser = argparse.ArgumentParser(description="Start Disk Checker API server")
parser.add_argument("--host", default="127.0.0.1", help="Host to bind to")
parser.add_argument("--port", type=int, default=8000, help="Port to bind to")
parser.add_argument("--workers", type=int, default=1, help="Number of workers")
parser.add_argument("--reload", action="store_true", help="Enable auto-reload")
parser.add_argument("--work-dir", type=Path, default=Path("."), help="Work directory")
args = parser.parse_args()
# Set work directory in environment
import os
os.environ["DISK_CHECKER_WORK_DIR"] = str(args.work_dir)
uvicorn.run(
"disk_checker.web.server:app",
host=args.host,
port=args.port,
workers=args.workers,
reload=args.reload,
)
if __name__ == "__main__":
main()
-6
View File
@@ -1,6 +0,0 @@
"""Web backend module for disk-checker API."""
from .web_routes import web_router
from .routes import api_router
__all__ = ["web_router", "api_router"]
-52
View File
@@ -1,52 +0,0 @@
"""Pydantic models for the web API."""
from pydantic import BaseModel, Field
from typing import List, Optional, Dict, Any
from enum import Enum
class ToolType(str, Enum):
"""Available benchmarking tools."""
FIO = "fio"
BADBLOCKS = "badblocks"
IO_URING = "io_uring"
class JobStatus(str, Enum):
"""Status of a benchmark job."""
PENDING = "pending"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
class BenchmarkRequest(BaseModel):
"""Request to start a benchmark job."""
device: str = Field(..., description="Block device path (e.g., /dev/sda)")
tool: ToolType = ToolType.FIO
job_file: Optional[str] = None
options: Dict[str, Any] = Field(default_factory=dict)
class Config:
json_schema_extra = {
"example": {
"device": "/dev/sda",
"tool": "fio",
"options": {
"rw": "randread",
"bs": "4k",
"size": "1G",
"runtime": "60"
}
}
}
class JobStatusResponse(BaseModel):
"""Response with job status."""
job_id: str
status: JobStatus
output_path: Optional[str] = None
error: Optional[str] = None
created_at: Optional[str] = None
completed_at: Optional[str] = None
-45
View File
@@ -1,45 +0,0 @@
"""API routes for disk-checker web backend."""
from fastapi import APIRouter, Request
from disk_checker.core.tool_scanner import ToolInfo
from disk_checker.core.drive_scanner import DriveInfo, scan_drives
api_router = APIRouter()
@api_router.get("/tools")
async def get_available_tools(request: Request):
"""
Get list of available benchmarking tools.
"""
tools = request.app.state.available_tools
return {
tool_name: {
"name" : tool.name,
"available": tool.available,
"version" : tool.version
}
for tool_name, tool in tools.items()
}
@api_router.get("/drives")
async def get_available_drives(request: Request):
"""
Get list of available block devices
Mounted drives are marked as unavailable
"""
drives = scan_drives()
return {
"drives": [
{
"name": drive.name,
"path": drive.path,
"size": drive.size,
"model": drive.model,
"mounted": drive.mounted,
"mount_point": drive.mount_point,
"serial": drive.serial
}
for drive in drives
]
}
-80
View File
@@ -1,80 +0,0 @@
"""FastAPI application for disk-checker web API."""
import asyncio
from pathlib import Path
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
from fastapi.templating import Jinja2Templates
from fastapi.middleware.cors import CORSMiddleware
from contextlib import asynccontextmanager
import logging
from disk_checker.core.orchestrator import Orchestrator
from disk_checker.core.tool_scanner import scan_tools
# Import and include routers
from disk_checker.web.routes import api_router
from disk_checker.web.web_routes import web_router
# Get application root
BASE_DIR = Path(__file__).resolve().parent.parent.parent.parent
TEMPLATES_DIR = BASE_DIR / "templates"
STATIC_DIR = BASE_DIR / "static"
# Setup templates
templates = Jinja2Templates(directory=str(TEMPLATES_DIR))
# Setup logger
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Application lifespan handler."""
# Startup: initialize orchestrator
#app.state.orchestrator = Orchestrator(work_dir=app.state.work_dir)
app.state.jobs = {}
# Scan for available benchmarking tools
app.state.available_tools = await asyncio.to_thread(scan_tools)
logger.info(f"Available tools: {list(app.state.available_tools.keys())}")
# Store templates and static_dir path in app.state
app.state.templates = templates
app.state.static_dir = STATIC_DIR
logger.info("Disk Checker API started")
yield
# Shutdown: cleanup
logger.info("Disk Checker API stopped")
app = FastAPI(
title="Disk Checker API",
description="API for disk benchmarking and latency visualization",
version="0.1.0",
lifespan=lifespan,
)
# Setup static files
app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")
# Include routers
app.include_router(api_router, prefix="/api/v1")
app.include_router(web_router)
# Add CORS (permissive by default)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Simple health endpoint
@app.get("/health")
async def health_check():
"""Health check endpoint."""
return {"status": "ok", "version": "0.1.0"}
-65
View File
@@ -1,65 +0,0 @@
from fastapi import APIRouter, Request
from fastapi.templating import Jinja2Templates
from typing import Dict
web_router = APIRouter()
# Navigation configuration - NEW TABS GO HERE!
NAVIGATION = [
{"name": "home", "label": "Home", "url": "/", "icon": "bi-house"},
{"name": "benchmark", "label": "Benchmark", "url": "/benchmark", "icon": "bi-hdd"},
{"name": "jobs", "label": "Job Queue", "url": "/jobs", "icon": "bi-list-task"},
]
def get_active_tab(url: str) -> str:
"""Determine which navigation tab is active."""
for nav_item in NAVIGATION:
if nav_item["url"] == url:
return nav_item["name"]
return "home"
@web_router.get("/")
async def render_home(request: Request):
"""
Render the main page
"""
tools = request.app.state.available_tools
return request.app.state.templates.TemplateResponse(
name="index.html",
request=request,
context={
"tools":tools,
"navigation": NAVIGATION,
"active_tab": get_active_tab("/")
}
)
@web_router.get("/benchmark")
async def render_benchmark(request: Request):
"""Render the benchmark page."""
templates: Jinja2Templates = request.app.state.templates
return templates.TemplateResponse(
name="benchmark.html",
request=request,
context={
"navigation": NAVIGATION,
"active_tab": get_active_tab("/benchmark")
}
)
@web_router.get("/jobs")
async def render_jobs(request: Request):
"""Render the job queue page."""
templates: Jinja2Templates = request.app.state.templates
return templates.TemplateResponse(
name="jobs.html",
request=request,
context={
"navigation": NAVIGATION,
"active_tab": get_active_tab("/jobs")
}
)
-70
View File
@@ -1,70 +0,0 @@
/* Basic styling for Disk Checker */
.container {
max-width: 1200px;
}
.card {
transition: transform 0.2s;
}
.card:hover {
transform: translateY(-2px);
box-shadow: 0 4px 8px rgba(0,0,0,0.1);
}
.badge {
font-size: 0.9rem;
}
/* Drive selector cards */
.drive-selector-card {
transition: all 0.2s ease;
}
.drive-selector-card .btn-check:checked + .btn {
background-color: #e7f1ff;
border-color: #0d6efd;
box-shadow: 0 0 0 0.2rem rgba(13, 110, 253, 0.25);
}
.drive-selector-card .btn {
border-radius: 0.5rem;
padding: 1rem;
}
.drive-selector-card .btn:hover {
background-color: #f8f9fa;
}
.drive-selector-card .btn-check:checked + .btn:hover {
background-color: #e7f1ff;
}
/* General drive cards */
.drive-card {
transition: all 0.3s ease;
height: 100%;
}
.drive-card.available {
border-left: 4px solid #28a745;
}
.drive-card.disabled {
border-left: 4px solid #6c757d;
opacity: 0.7;
}
.drive-card:hover:not(.disabled) {
transform: translateY(-2px);
box-shadow: 0 4px 8px rgba(0,0,0,0.1);
}
/* Make disabled cards look unavailable */
.drive-card.disabled .card-title {
color: #6c757d;
}
.drive-card.disabled .card-text {
color: #6c757d;
}
-209
View File
@@ -1,209 +0,0 @@
/**
* Disk Checker - Main JavaScript
* Handles HTMX events and dynamic content rendering
*/
// Global state
let selectedDrive = null;
// Render drives after HTMX loads them
function renderDrives(data) {
const container = document.getElementById('drives-content');
if (!container) return;
container.innerHTML = '';
if (!data.drives || data.drives.length === 0) {
container.innerHTML = '<div class="col-12"><p class="text-muted">No drives found.</p></div>';
return;
}
data.drives.forEach(drive => {
const col = document.createElement('div');
col.className = 'col-md-4 mb-3';
const card = document.createElement('div');
card.className = `drive-card ${drive.mounted ? 'disabled' : 'available'}`;
card.innerHTML = `
<div class="card-body">
<h5 class="card-title">${drive.name}</h5>
<p class="card-text">
<strong>Size:</strong> ${drive.size}<br>
<strong>Model:</strong> ${drive.model || 'Unknown'}<br>
${drive.serial ? `<strong>Serial:</strong> ${drive.serial}<br>` : ''}
${drive.mount_point ? `<strong>Mounted at:</strong> ${drive.mount_point}<br>` : ''}
</p>
<div>
${drive.mounted
? '<span class="badge bg-secondary">Mounted - Unavailable</span>'
: '<span class="badge bg-success">Available for Benchmark</span>'
}
</div>
</div>
`;
col.appendChild(card);
container.appendChild(col);
});
}
// Render drive selector for benchmark tab (available drives only)
function renderDriveSelector(data) {
const container = document.getElementById('drive-cards');
if (!container) return;
container.innerHTML = '';
if (!data.drives || data.drives.length === 0) {
container.innerHTML = '<div class="col-12"><p class="text-muted">No drives found.</p></div>';
return;
}
// Filter to show only available drives
const availableDrives = data.drives.filter(drive => !drive.mounted);
if (availableDrives.length === 0) {
container.innerHTML = '<div class="col-12"><div class="alert alert-warning">No available drives. All drives are mounted.</div></div>';
return;
}
availableDrives.forEach(drive => {
const col = document.createElement('div');
col.className = 'col-md-4 mb-3';
const cardWrapper = document.createElement('div');
cardWrapper.className = 'drive-selector-card';
cardWrapper.dataset.drivePath = drive.path;
// Radio input (hidden)
const radio = document.createElement('input');
radio.type = 'radio';
radio.name = 'selected_drive';
radio.value = drive.path;
radio.id = `drive-${drive.name}`;
radio.className = 'btn-check';
radio.setAttribute('autocomplete', 'off');
// Label as card
const label = document.createElement('label');
label.htmlFor = `drive-${drive.name}`;
label.className = 'btn btn-outline-primary w-100 text-start';
label.innerHTML = `
<div class="d-flex justify-content-between align-items-start">
<div>
<h6 class="mb-1">${drive.name}</h6>
<small class="text-muted">${drive.size} ${drive.model || ''}</small>
</div>
<div class="text-end">
<small class="text-success">Available</small>
</div>
</div>
${drive.serial ? `<small class="text-muted d-block mt-1">SN: ${drive.serial}</small>` : ''}
`;
cardWrapper.appendChild(radio);
cardWrapper.appendChild(label);
col.appendChild(cardWrapper);
container.appendChild(col);
// Add click handler to show selected drive info
radio.addEventListener('change', function() {
if (this.checked) {
selectDrive(drive);
}
});
});
}
// Handle drive selection
function selectDrive(drive) {
selectedDrive = drive;
console.log('Selected drive:', drive);
// Update UI to show selection
const infoDiv = document.getElementById('selected-drive-info');
if (infoDiv) {
infoDiv.innerHTML = `
<div class="alert alert-success">
<strong>Selected:</strong> ${drive.name} (${drive.path})
<br><small>${drive.size} - ${drive.model || 'Unknown model'}</small>
</div>
`;
}
// Show clear selection button
const clearContainer = document.getElementById('clear-selection-container');
if (clearContainer) {
clearContainer.style.display = 'block';
}
// Enable benchmark configuration section
const configDiv = document.getElementById('benchmark-config');
if (configDiv) {
configDiv.classList.remove('opacity-50');
configDiv.querySelector('.card-body').innerHTML = `
<p>Drive <strong>${drive.name}</strong> selected. Configure benchmark parameters below.</p>
<!-- Benchmark options will go here -->
`;
}
}
// Handle drive deselection (clear selection)
function clearDriveSelection() {
selectedDrive = null;
console.log('Cleared drive selection');
// Clear selected drive info
const infoDiv = document.getElementById('selected-drive-info');
if (infoDiv) {
infoDiv.innerHTML = '';
}
// Hide clear selection button
const clearContainer = document.getElementById('clear-selection-container');
if (clearContainer) {
clearContainer.style.display = 'none';
}
// Disable benchmark configuration section
const configDiv = document.getElementById('benchmark-config');
if (configDiv) {
configDiv.classList.add('opacity-50');
configDiv.querySelector('.card-body').innerHTML = `
<p class="text-muted">Select a drive above to configure benchmark parameters.</p>
`;
}
// Uncheck all radio buttons
const radios = document.querySelectorAll('input[name="selected_drive"]');
radios.forEach(radio => { radio.checked = false; });
}
// Listen for clear selection button click
document.addEventListener('click', function(evt) {
if (evt.target.id === 'clear-selection-btn' || evt.target.closest('#clear-selection-btn')) {
clearDriveSelection();
}
});
// Listen for HTMX events
document.addEventListener('htmx:afterRequest', function(evt) {
// Render drives in home page
if (evt.detail.target.id === 'drives-content') {
renderDrives(JSON.parse(evt.detail.xhr.response));
}
// Render drive selector in benchmark page
if (evt.detail.target.id === 'drive-cards') {
renderDriveSelector(JSON.parse(evt.detail.xhr.response));
}
});
// Listen for HTMX load event as fallback
document.addEventListener('htmx:load', function(evt) {
if (evt.target.id === 'drive-cards') {
// HTMX will load the content via hx-get
// The afterRequest event will handle rendering
}
});
-53
View File
@@ -1,53 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Disk Checker - {% block title %}Home{% endblock %}</title>
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.10.0/font/bootstrap-icons.css">
<script src="https://unpkg.com/htmx.org@1.9.10"></script>
<link rel="stylesheet" href="/static/css/style.css">
{% block extra_css %}{% endblock %}
</head>
<body>
<!-- Navigation Bar -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark">
<div class="container">
<!-- Logo / Brand -->
<a class="navbar-brand" href="/">
<i class="bi bi-hdd-stack"></i>
Disk Checker
</a>
<!-- Mobile toggle -->
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarNav">
<span class="navbar-toggler-icon"></span>
</button>
<!-- Navigation Tabs -->
<div class="collapse navbar-collapse" id="navbarNav">
<ul class="navbar-nav">
{% for nav_item in navigation %}
<li class="nav-item">
<a class="nav-link {% if active_tab == nav_item.name %}active{% endif %}"
href="{{ nav_item.url }}">
<i class="bi {{ nav_item.icon }}"></i>
{{ nav_item.label }}
</a>
</li>
{% endfor %}
</ul>
</div>
</div>
</nav>
<!-- Page Content -->
<div class="container mt-4">
{% block content %}{% endblock %}
</div>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script>
{% block extra_js %}{% endblock %}
</body>
</html>
-55
View File
@@ -1,55 +0,0 @@
{% extends "base.html" %}
{% block title %}Benchmark{% endblock %}
{% block content %}
<div class="row">
<div class="col-12">
<h2>Benchmark</h2>
<p class="text-muted">Configure and run disk benchmarks.</p>
<!-- Drive Selection Section -->
<div class="card mb-4">
<div class="card-header">
<h5 class="mb-0">Select Drive to Benchmark</h5>
</div>
<div class="card-body">
<p class="text-muted">Choose an available drive from the list below.</p>
<!-- Drive selector container -->
<div id="drive-selector"
hx-get="/api/v1/drives"
hx-trigger="load"
hx-target="#drive-cards"
hx-swap="innerHTML">
</div>
<!-- Drive cards will be rendered here -->
<div id="drive-cards" class="row g-3"></div>
<!-- Selected drive display -->
<div id="selected-drive-info" class="mt-3"></div>
<div id="clear-selection-container" class="mt-2" style="display: none;">
<button id="clear-selection-btn" class="btn btn-sm btn-outline-secondary">
<i class="bi bi-x-circle"></i> Clear Selection
</button>
</div>
</div>
</div>
<!-- Benchmark Configuration (placeholder for next feature) -->
<div class="card" id="benchmark-config">
<div class="card-header">
<h5 class="mb-0">Benchmark Configuration</h5>
</div>
<div class="card-body">
<p class="text-muted">Select a drive above to configure benchmark parameters.</p>
</div>
</div>
</div>
</div>
{% endblock %}
{% block extra_js %}
<script src="/static/js/main.js"></script>
{% endblock %}
-53
View File
@@ -1,53 +0,0 @@
{% extends "base.html" %}
{% block title %}Home{% endblock %}
{% block content %}
<div class="row">
<div class="col-12">
<!--Available tools section-->
<h2>Available Benchmarking Tools</h2>
<!-- Server-side rendered tools -->
{% if tools %}
<div class="row">
{% for tool_name, tool in tools.items() %}
<div class="col-md-4 mb-3">
<div class="card {% if tool.available %}border-success{% else %}border-secondary{% endif %}">
<div class="card-body">
<h5 class="card-title">{{ tool.name }}</h5>
<p class="card-text">
{% if tool.available %}
<span class="badge bg-success">Available</span>
{% if tool.version %}
<small class="text-muted">{{ tool.version }}</small>
{% endif %}
{% else %}
<span class="badge bg-secondary">Not Installed</span>
{% endif %}
</p>
</div>
</div>
</div>
{% endfor %}
</div>
{% endif %}
<!--Available drives section-->
<h2 class="mt-5">Available Drives</h2>
<p class="text-muted">
<i class="bi bi-info-circle"></i>
Mounted drives are grayed out and cannot be benchmarked.
</p>
<div id="drives-list"
hx-get="/api/v1/drives"
hx-trigger="load"
hx-target="#drives-content"></div>
<div id="drives-content" class="row"></div>
</div>
</div>
{% endblock %}
{% block extra_js %}
<script src="/static/js/main.js"></script>
{% endblock %}
-54
View File
@@ -1,54 +0,0 @@
{% extends "base.html" %}
{% block title %}Job Queue{% endblock %}
{% block content %}
<div class="row">
<div class="col-12">
<h2>Job Queue</h2>
<p class="text-muted">Monitor and manage benchmark jobs.</p>
<!-- Jobs list -->
<div id="jobs-list"
hx-get="/api/v1/jobs"
hx-trigger="load"
hx-target="#jobs-content">
<p>Loading jobs...</p>
</div>
<div id="jobs-content"></div>
</div>
</div>
{% endblock %}
{% block extra_js %}
<script>
// Load jobs when page loads
document.addEventListener('htmx:afterRequest', function(evt) {
if (evt.detail.target.id === 'jobs-content') {
renderJobs(JSON.parse(evt.detail.xhr.response));
}
});
function renderJobs(jobIds) {
const container = document.getElementById('jobs-content');
container.innerHTML = '';
if (!jobIds || jobIds.length === 0) {
container.innerHTML = '<div class="alert alert-info">No jobs in queue.</div>';
return;
}
jobIds.forEach(jobId => {
const div = document.createElement('div');
div.className = 'card mb-2';
div.innerHTML = `
<div class="card-body">
<h5 class="card-title">${jobId}</h5>
<p class="card-text text-muted">Job details will appear here...</p>
</div>
`;
container.appendChild(div);
});
}
</script>
{% endblock %}
-1
View File
@@ -1 +0,0 @@
"""Tests for disk-checker."""
-68
View File
@@ -1,68 +0,0 @@
"""Tests for the unit conversion module."""
import numpy as np
import pytest
from disk_checker.core.converter import to_unit, UNIT_FACTORS, UNIT_LABELS
class TestToUnit:
"""Tests for to_unit function."""
def test_nanoseconds_to_microseconds(self):
"""Test conversion from ns to us."""
values = np.array([1000, 2000, 3000])
result = to_unit(values, "us")
expected = np.array([1.0, 2.0, 3.0])
np.testing.assert_array_equal(result, expected)
def test_nanoseconds_to_milliseconds(self):
"""Test conversion from ns to ms."""
values = np.array([1000000, 2000000])
result = to_unit(values, "ms")
expected = np.array([1.0, 2.0])
np.testing.assert_array_equal(result, expected)
def test_nanoseconds_to_seconds(self):
"""Test conversion from ns to s."""
values = np.array([1000000000, 2000000000])
result = to_unit(values, "s")
expected = np.array([1.0, 2.0])
np.testing.assert_array_equal(result, expected)
def test_same_unit_returns_same(self):
"""Test conversion to same unit."""
values = np.array([100, 200, 300])
result = to_unit(values, "ns")
np.testing.assert_array_equal(result, values)
def test_empty_array(self):
"""Test conversion with empty array."""
values = np.array([])
result = to_unit(values, "us")
assert len(result) == 0
def test_single_value(self):
"""Test conversion with single value."""
result = to_unit(1000, "us")
assert result == 1.0
class TestConstants:
"""Tests for module constants."""
def test_unit_factors(self):
"""Test UNIT_FACTORS contains all expected units."""
assert set(UNIT_FACTORS.keys()) == {"ns", "us", "ms", "s"}
assert UNIT_FACTORS["ns"] == 1.0
assert UNIT_FACTORS["us"] == 1e3
assert UNIT_FACTORS["ms"] == 1e6
assert UNIT_FACTORS["s"] == 1e9
def test_unit_labels(self):
"""Test UNIT_LABELS contains all expected units."""
assert set(UNIT_LABELS.keys()) == {"ns", "us", "ms", "s"}
assert UNIT_LABELS["ns"] == "ns"
assert UNIT_LABELS["us"] == "µs"
assert UNIT_LABELS["ms"] == "ms"
assert UNIT_LABELS["s"] == "s"
-72
View File
@@ -1,72 +0,0 @@
"""Tests for the data extraction module."""
import pytest
import pandas as pd
from pathlib import Path
from disk_checker.core.extractor import load_latency_csv
class TestLoadLatencyCsv:
"""Tests for load_latency_csv function."""
def test_load_valid_csv(self, sample_csv_path):
"""Test loading a valid fio CSV file."""
df = load_latency_csv(str(sample_csv_path))
assert not df.empty
assert "metric" in df.columns
assert "latency_ns" in df.columns
assert "count" in df.columns
def test_returns_long_format(self, sample_csv_path):
"""Test that output is in long format with metric column."""
df = load_latency_csv(str(sample_csv_path))
metrics = df["metric"].unique()
assert "read_clat" in metrics
def test_empty_csv_raises(self, empty_csv_path):
"""Test that empty CSV raises ValueError."""
with pytest.raises(ValueError):
load_latency_csv(str(empty_csv_path))
def test_invalid_format_raises(self, invalid_csv_path):
"""Test that invalid CSV format raises ValueError."""
with pytest.raises(ValueError):
load_latency_csv(str(invalid_csv_path))
def test_missing_nsec_column_raises(self, tmp_path):
"""Test that missing nsec column raises ValueError."""
csv_content = "col1,col2\n1,2\n"
path = tmp_path / "invalid.csv"
path.write_text(csv_content)
with pytest.raises(ValueError, match="expected first column to be 'nsec'"):
load_latency_csv(str(path))
@pytest.fixture
def sample_csv_path(tmp_path):
"""Create a sample CSV file for testing."""
csv_content = """nsec,read_clat_ns_count,read_clat_ns_cumulative,read_clat_ns_percentile
19791872,1,1,0.00016775708773695687
22413312,4,5,0.0008387854386847844
22675456,9,14,0.0023485992283173965
"""
path = tmp_path / "test.csv"
path.write_text(csv_content)
return path
@pytest.fixture
def empty_csv_path(tmp_path):
"""Create an empty CSV file for testing."""
path = tmp_path / "empty.csv"
path.write_text("nsec\n")
return path
@pytest.fixture
def invalid_csv_path(tmp_path):
"""Create an invalid CSV file for testing."""
path = tmp_path / "invalid.csv"
path.write_text("col1,col2\n1,2\n")
return path
-77
View File
@@ -1,77 +0,0 @@
"""Tests for helper functions."""
import pandas as pd
import pytest
from disk_checker.core.helpers import human_count, percentile_table
class TestHumanCount:
"""Tests for human_count function."""
def test_small_number(self):
"""Test formatting of small numbers."""
assert human_count(100) == "100"
assert human_count(50) == "50"
def test_thousands(self):
"""Test formatting of thousands."""
assert human_count(1500) == "2k"
assert human_count(2500) == "2k"
assert human_count(1000) == "1k"
assert human_count(999) == "999"
def test_millions(self):
"""Test formatting of millions."""
assert human_count(1500000) == "1.5M"
assert human_count(2750000) == "2.8M"
def test_billions(self):
"""Test formatting of billions."""
assert human_count(1500000000) == "1.5G"
assert human_count(2750000000) == "2.8G"
def test_zero(self):
"""Test formatting of zero."""
assert human_count(0) == "0"
def test_position_argument_ignored(self):
"""Test that _pos argument is ignored."""
assert human_count(1000, 0) == "1k"
class TestPercentileTable:
"""Tests for percentile_table function."""
def test_basic_percentiles(self):
"""Test basic percentile table generation."""
df = pd.DataFrame({
"metric": ["read_clat"] * 3,
"latency_ns": [1000, 2000, 3000],
"count": [1, 2, 1],
})
result = percentile_table(df, "read_clat", "us", [50, 90])
assert "Metric: read_clat" in result
assert "total I/Os: 4" in result
assert "percentile" in result
assert "latency" in result
def test_missing_metric(self):
"""Test percentile table with missing metric."""
df = pd.DataFrame({
"metric": ["read_clat"],
"latency_ns": [1000],
"count": [1],
})
result = percentile_table(df, "write_clat", "us", [50])
assert "(no data for metric write_clat)" in result
def test_unit_display(self):
"""Test that correct unit is displayed."""
df = pd.DataFrame({
"metric": ["read_clat"],
"latency_ns": [1000000], # 1ms in ns
"count": [1],
})
result = percentile_table(df, "read_clat", "ms", [50])
assert "ms" in result
-57
View File
@@ -1,57 +0,0 @@
"""Tests for the orchestrator module."""
import asyncio
import pytest
from pathlib import Path
from unittest.mock import AsyncMock, patch
from disk_checker.core.orchestrator import Orchestrator, ToolStatus, JobResult
class TestOrchestrator:
"""Tests for Orchestrator class."""
@pytest.fixture
def orchestrator(self, tmp_path):
"""Create an orchestrator instance."""
return Orchestrator(work_dir=tmp_path)
def test_initial_state(self, orchestrator):
"""Test initial orchestrator state."""
assert orchestrator.list_jobs() == {}
def test_run_fio_creates_job(self, orchestrator, tmp_path):
"""Test that run_fio creates a job entry."""
with patch("asyncio.create_subprocess_exec") as mock_exec:
mock_exec.return_value = AsyncMock()
result = asyncio.get_event_loop().run_until_complete(
orchestrator.run_fio("/dev/sda")
)
jobs = orchestrator.list_jobs()
assert len(jobs) == 1
job = list(jobs.values())[0]
assert job.status == ToolStatus.COMPLETED
# Note: output_path may not exist in test environment without actual fio
def test_get_job_status(self, orchestrator, tmp_path):
"""Test retrieving job status."""
with patch("asyncio.create_subprocess_exec") as mock_exec:
mock_exec.return_value = AsyncMock()
result = asyncio.get_event_loop().run_until_complete(
orchestrator.run_fio("/dev/sda")
)
# result is the output path, job_id is derived from it
job_id = f"fio_{result.stem.split('_')[-1]}"
# Get the actual job_id from the jobs dict
jobs = orchestrator.list_jobs()
job_id = list(jobs.keys())[0]
job = orchestrator.get_job_status(job_id)
assert job is not None
assert job.status == ToolStatus.COMPLETED
def test_get_nonexistent_job(self, orchestrator):
"""Test getting status of non-existent job."""
job = orchestrator.get_job_status("nonexistent")
assert job is None
-134
View File
@@ -1,134 +0,0 @@
"""Tests for the plotting module."""
import pytest
from pathlib import Path
from unittest.mock import MagicMock, patch
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from disk_checker.plotting.plots import draw_histogram, draw_cdf, generate_plot
import pandas as pd
class TestDrawHistogram:
"""Tests for draw_histogram function."""
@pytest.fixture
def sample_df(self):
"""Create a sample DataFrame for testing."""
return pd.DataFrame({
"metric": ["read_clat", "read_clat", "write_clat"],
"latency_ns": [1000, 2000, 1500],
"count": [1, 2, 1],
})
def test_draws_histogram(self, sample_df, tmp_path):
"""Test that histogram is drawn without errors."""
fig, ax = plt.subplots()
draw_histogram(ax, sample_df, ["read_clat"])
assert ax.get_title() == "Completion latency distribution"
plt.close(fig)
def test_multiple_metrics(self, sample_df, tmp_path):
"""Test drawing multiple metrics."""
fig, ax = plt.subplots()
draw_histogram(ax, sample_df, ["read_clat", "write_clat"])
assert len(ax.collections) > 0
plt.close(fig)
def test_log_scale(self, sample_df, tmp_path):
"""Test logarithmic scale."""
fig, ax = plt.subplots()
draw_histogram(ax, sample_df, ["read_clat"], log_x=True, log_y=True)
assert ax.get_xscale() == "log"
assert ax.get_yscale() == "log"
plt.close(fig)
class TestDrawCdf:
"""Tests for draw_cdf function."""
@pytest.fixture
def sample_df(self):
"""Create a sample DataFrame for testing."""
return pd.DataFrame({
"metric": ["read_clat", "read_clat"],
"latency_ns": [1000, 2000],
"count": [1, 3],
})
def test_draws_cdf(self, sample_df, tmp_path):
"""Test that CDF is drawn without errors."""
fig, ax = plt.subplots()
draw_cdf(ax, sample_df, ["read_clat"])
assert ax.get_title() == "Cumulative latency distribution"
assert ax.get_ylim() == (0, 100)
plt.close(fig)
def test_log_scale(self, sample_df, tmp_path):
"""Test logarithmic X scale."""
fig, ax = plt.subplots()
draw_cdf(ax, sample_df, ["read_clat"], log_x=True)
assert ax.get_xscale() == "log"
plt.close(fig)
class TestGeneratePlot:
"""Tests for generate_plot function."""
@pytest.fixture
def sample_df(self):
"""Create a sample DataFrame for testing."""
return pd.DataFrame({
"metric": ["read_clat", "read_clat"],
"latency_ns": [1000, 2000],
"count": [1, 3],
})
def test_generates_histogram(self, sample_df, tmp_path):
"""Test generating histogram plot."""
output = tmp_path / "test.png"
result = generate_plot(
df=sample_df,
metrics=["read_clat"],
output_path=output,
)
assert result == output
assert output.exists()
def test_generates_cdf_only(self, sample_df, tmp_path):
"""Test generating CDF-only plot."""
output = tmp_path / "test_cdf.png"
result = generate_plot(
df=sample_df,
metrics=["read_clat"],
output_path=output,
cdf_only=True,
)
assert result == output
assert output.exists()
def test_generates_histogram_with_cdf(self, sample_df, tmp_path):
"""Test generating histogram with CDF below."""
output = tmp_path / "test_both.png"
result = generate_plot(
df=sample_df,
metrics=["read_clat"],
output_path=output,
show_cdf=True,
)
assert result == output
assert output.exists()
def test_custom_dpi(self, sample_df, tmp_path):
"""Test custom DPI setting."""
output = tmp_path / "test_dpi.png"
result = generate_plot(
df=sample_df,
metrics=["read_clat"],
output_path=output,
dpi=300,
)
assert result == output
assert output.exists()
-89
View File
@@ -1,89 +0,0 @@
"""Tests for the web backend."""
import pytest
from fastapi.testclient import TestClient
from pathlib import Path
from unittest.mock import AsyncMock, patch
from disk_checker.web.server import app
@pytest.fixture
def client():
"""Create test client."""
return TestClient(app)
@pytest.fixture
def sample_csv_content():
"""Sample CSV content for testing."""
return """nsec,read_clat_ns_count,read_clat_ns_cumulative,read_clat_ns_percentile
19791872,1,1,0.00016775708773695687
22413312,4,5,0.0008387854386847844
22675456,9,14,0.0023485992283173965
"""
@pytest.fixture
def sample_csv_path(tmp_path, sample_csv_content):
"""Create sample CSV file."""
path = tmp_path / "test.csv"
path.write_text(sample_csv_content)
return path
def test_health_check(client):
"""Test health check endpoint."""
response = client.get("/health")
assert response.status_code == 200
assert response.json()["status"] == "ok"
def test_scan_csv(client, sample_csv_path):
"""Test CSV scan endpoint."""
with open(sample_csv_path, "rb") as f:
response = client.post("/api/v1/scan", files={"file": f})
assert response.status_code == 200
data = response.json()
assert "metrics" in data
assert len(data["metrics"]) > 0
def test_plot_endpoint(client, sample_csv_path):
"""Test plot generation endpoint."""
with open(sample_csv_path, "rb") as f:
response = client.post(
"/api/v1/plot",
files={"file": (sample_csv_path.name, f, "text/csv")},
data={"unit": "us"},
)
assert response.status_code == 200
assert response.headers["content-type"] == "image/png"
def test_plot_with_options(client, sample_csv_path):
"""Test plot generation with options."""
with open(sample_csv_path, "rb") as f:
response = client.post(
"/api/v1/plot",
files={"file": (sample_csv_path.name, f, "text/csv")},
data={
"metric": "read_clat",
"unit": "ms",
"log_x": "true",
"cdf": "true",
},
)
assert response.status_code == 200
def test_invalid_metric(client, sample_csv_path):
"""Test invalid metric returns error."""
with open(sample_csv_path, "rb") as f:
response = client.post(
"/api/v1/plot",
files={"file": (sample_csv_path.name, f, "text/csv")},
data={"metric": "invalid_metric", "unit": "us"},
)
# Note: endpoint returns 200 and auto-selects best metric when invalid
assert response.status_code == 200