add all files

controls-rework/docs/historian-readme.md

@@ -0,0 +1,94 @@
+# LASUCA Historian Notes
+
+## SQL Server History Database
+- **Server:** `192.168.0.13\SQLEXPRESS`
+- **Database:** `history`
+- **Credentials:** username `opce`, password `opcelasuca`
+
+## Core Tables
+- `dbo.archive`
+  - Columns: `TimeStamp` (`datetime`), `ID` (`int`), `Value` (`float`/`numeric`), `Quality` (`int`)
+  - Holds raw samples from the PLC historian.
+- `dbo.id_names`
+  - Columns: `idnumber` (`varchar`), `name` (`varchar`), `description` (`varchar`)
+  - Maps numeric IDs to human-readable tag names.
+
+### Joining Archive to Tag Metadata
+`id_names.idnumber` is stored as text while `archive.ID` is an integer. Cast the `idnumber` column when joining:
+
+```sql
+FROM dbo.archive AS a
+INNER JOIN dbo.id_names AS n
+    ON a.ID = CAST(n.idnumber AS INT)
+```
+
+Trending endpoints such as `trends/live/realtime_data.php` rely on this join to look up tags by name:
+
+```php
+$sql = "SELECT a.TimeStamp, a.Value
+        FROM dbo.archive AS a
+        INNER JOIN dbo.id_names AS n ON a.ID = CAST(n.idnumber AS INT)
+        WHERE n.name = :tag_name";
+```
+
+## Downtime View (Zero-Value Islands)
+
+```sql
+CREATE OR ALTER VIEW dbo.v_downtime_zero_segments AS
+WITH ordered AS (
+    SELECT
+        a.ID,
+        n.name,
+        a.TimeStamp,
+        a.Value,
+        CASE
+            WHEN a.Value = 0
+             AND (LAG(a.Value) OVER (PARTITION BY a.ID ORDER BY a.TimeStamp) <> 0
+                  OR LAG(a.Value) OVER (PARTITION BY a.ID ORDER BY a.TimeStamp) IS NULL)
+            THEN 1 ELSE 0
+        END AS start_flag,
+        CASE
+            WHEN a.Value = 0
+             AND (LEAD(a.Value) OVER (PARTITION BY a.ID ORDER BY a.TimeStamp) <> 0
+                  OR LEAD(a.Value) OVER (PARTITION BY a.ID ORDER BY a.TimeStamp) IS NULL)
+            THEN LEAD(a.TimeStamp) OVER (PARTITION BY a.ID ORDER BY a.TimeStamp)
+            ELSE NULL
+        END AS next_good_timestamp
+    FROM dbo.archive AS a
+    INNER JOIN dbo.id_names AS n
+        ON a.ID = CAST(n.idnumber AS INT)
+),
+grouped AS (
+    SELECT
+        *,
+        SUM(start_flag) OVER (
+            PARTITION BY ID
+            ORDER BY TimeStamp
+            ROWS UNBOUNDED PRECEDING
+        ) AS group_id
+    FROM ordered
+    WHERE Value = 0
+)
+SELECT
+    ID,
+    name,
+    MIN(TimeStamp) AS downtime_start,
+    COALESCE(MAX(next_good_timestamp), MAX(TimeStamp)) AS downtime_end,
+    DATEDIFF(SECOND, MIN(TimeStamp), COALESCE(MAX(next_good_timestamp), MAX(TimeStamp))) AS downtime_seconds
+FROM grouped
+GROUP BY ID, name, group_id;
+```
+
+Usage example (minimum 5 minutes):
+
+```sql
+SELECT *
+FROM dbo.v_downtime_zero_segments
+WHERE name IN ('MCCOUTPUT', 'TONS_PER_HOUR')
+  AND downtime_start >= '2025-10-15T00:00:00'
+  AND downtime_end   <= '2025-10-16T00:00:00'
+  AND downtime_seconds >= 5 * 60
+ORDER BY name, downtime_start;
+```
+
+This view expects SQL Server 2012 or later (window functions with `LAG/LEAD`).

controls-rework/docs/nl2sql.md

@@ -0,0 +1,157 @@
+# NL2SQL — Natural Language Search for Historian Data
+
+This document captures the ongoing work to add conversational, open-ended search capabilities to the LASUCA controls dashboard. The goal is to let operators ask questions in plain English (e.g., *"Show me Boiler 3 steam for the last 24 hours"*) and receive historian data without needing to know the underlying SQL.
+
+---
+
+## 1. Background
+
+### Current state
+| Layer | Description |
+|-------|-------------|
+| **Data store** | SQL Server historian on `192.168.0.13\SQLEXPRESS`, database `history`. Core tables: `dbo.archive` (timestamped values), `dbo.id_names` (tag metadata). |
+| **Search UI** | PHP forms (`opcsearch.php`, trend pages) with pre-built queries driven by dropdowns (tag picker, date range). |
+| **RAG infrastructure** | Vector DB already in use for memory/context; SQL agents consume context files to help generate queries. |
+
+### Objective
+Replace or augment the rigid form-based search with a conversational interface that:
+1. Accepts natural-language questions.
+2. Translates them to valid SQL via an LLM agent.
+3. Executes the query safely (read-only, row-limited).
+4. Returns results in a friendly format (table, chart link, plain-English summary).
+
+---
+
+## 2. Architecture overview
+
+```
+┌──────────────┐      ┌───────────────────┐      ┌─────────────────┐
+│  Chat UI     │─────▶│  nl_query.php     │─────▶│  SQL Agent      │
+│  (browser)   │      │  (orchestrator)   │      │  (Python/LLM)   │
+└──────────────┘      └───────────────────┘      └────────┬────────┘
+                                                          │
+       ┌──────────────────────────────────────────────────┘
+       ▼
+┌────────────────┐   ┌─────────────────┐   ┌─────────────────┐
+│ Schema context │   │ Few-shot store  │   │ Tag index       │
+│ (schema.md)    │   │ (vector DB)     │   │ (id_names embed)│
+└────────────────┘   └─────────────────┘   └─────────────────┘
+       │                      │                     │
+       └──────────────────────┴─────────────────────┘
+                              │
+                              ▼
+                    ┌─────────────────┐
+                    │ SQL Server      │
+                    │ (history DB)    │
+                    └─────────────────┘
+```
+
+### Key components
+
+| Component | Purpose | Status |
+|-----------|---------|--------|
+| **Schema context** | Compact description of tables, columns, types, relationships, and sample rows. Gives the LLM structural knowledge. | Export created; cleanup in progress. |
+| **Few-shot store** | Curated pairs of `(user_query, sql_template)` embedded in vector DB. Retriever surfaces the most relevant examples at query time. | Seeding strategy defined; logging table created. |
+| **Tag index** | `id_names` rows embedded so fuzzy tag references resolve to exact names. | Planned. |
+| **Query log** | Captures form-based searches to bootstrap few-shot examples and track usage patterns. | Table schema created (`dbo.controls_query_log`). |
+| **Orchestrator endpoint** | PHP (or Python) service that accepts NL prompt, calls agent, validates SQL, executes, returns results. | Not started. |
+| **Chat UI** | Lightweight front-end alongside existing forms. | Not started. |
+
+---
+
+## 3. Query logging
+
+To seed few-shot examples from real usage, we created a logging table in SQL Server.
+
+**Location:** `sql/create_query_log.sql`
+
+### Table: `dbo.controls_query_log`
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | INT (identity) | Primary key. |
+| `user_query` | NVARCHAR(1000) | Natural-language description derived from form inputs. |
+| `sql_template` | NVARCHAR(MAX) | Parameterized SQL with placeholders (`:tag`, `:start`, `:end`). |
+| `params` | NVARCHAR(MAX) | JSON blob of actual parameter values. |
+| `source_form` | NVARCHAR(100) | Originating form (e.g., `opcsearch`, `trend`). |
+| `user_session` | NVARCHAR(100) | Optional session/user ID. |
+| `execution_ms` | INT | Query runtime in milliseconds. |
+| `row_count` | INT | Rows returned. |
+| `created_at` | DATETIME2 | UTC timestamp (defaults to `SYSUTCDATETIME()`). |
+
+### Helper view: `dbo.vw_controls_query_templates`
+
+Aggregates distinct `sql_template` values with usage counts and a sample `user_query` for each—useful when exporting few-shot pairs.
+
+### Deduplication strategy
+
+Form queries are repetitive (same SQL template, different parameters). When seeding the vector DB:
+1. Group by `sql_template`.
+2. Pick one representative `user_query` per template (or synthesize a canonical description).
+3. Embed the `user_query`; attach `sql_template` as metadata.
+
+---
+
+## 4. Safety guardrails
+
+| Guardrail | Implementation |
+|-----------|----------------|
+| **Read-only connection** | Agent uses a login with `db_datareader` only; no write permissions. |
+| **Row limit injection** | Generated SQL wrapped with `SELECT TOP 1000 ...` or validated to include a cap. |
+| **Table/column allow-list** | Agent context explicitly lists permitted objects; LLM instructed to reject queries outside scope. |
+| **Syntax validation** | Run `SET PARSEONLY ON` or use `sqlparse` before execution. |
+| **Audit log** | Every generated query stored in `query_log` with execution stats. |
+
+---
+
+## 5. Next steps
+
+### Immediate (in progress)
+- [x] Create `controls_query_log` table on `lasucaai` database (192.168.0.16)
+- [x] Build logging helper (`includes/log_query.php`)
+- [x] Instrument `opcsearch.php` to log searches
+- [ ] Deploy to production and gather data for a few days
+
+### Short-term (after data collection)
+- [ ] Review `vw_controls_query_templates` — analyze distinct query patterns
+- [ ] Clean up schema export (`docs/historian-readme.md`) — remove noise, add sample rows
+- [ ] Seed tag index — embed `id_names` into vector DB for fuzzy tag resolution
+- [ ] Export few-shot pairs — write script to pull user_query ↔ sql_template pairs from log
+- [ ] Instrument other search forms (trends, boiler averages) if needed
+
+### Medium-term (build the agent)
+- [ ] Build orchestrator endpoint (`nl_query.php` or Python sidecar)
+- [ ] Integrate schema context + few-shot retrieval + tag index
+- [ ] Add SQL validation step (syntax check before execution)
+- [ ] Create chat UI — text input alongside existing forms
+- [ ] Implement safety guardrails (read-only connection, row limits, allow-list)
+
+### Long-term (iterate & improve)
+- [ ] Review agent failures and add corrective few-shot pairs
+- [ ] Add result summarization (natural-language answers, not just tables)
+- [ ] Consider chart generation from NL queries
+- [ ] Expand to other databases if needed
+
+---
+
+## 6. File reference
+
+| File | Purpose |
+|------|---------|
+| `sql/create_query_log.sql` | Creates `dbo.controls_query_log` table and `vw_controls_query_templates` view (run on `lasucaai` DB). |
+| `includes/log_query.php` | PHP helper to log searches; connects to 192.168.0.16/lasucaai. |
+| `docs/historian-readme.md` | Existing historian schema documentation. |
+| `docs/nl2sql.md` | This document. |
+
+---
+
+## 7. Database connections
+
+| Purpose | Server | Database | User |
+|---------|--------|----------|------|
+| Historian (read) | 192.168.0.13\SQLEXPRESS | history | opce |
+| AI/Logging (write) | 192.168.0.16 | lasucaai | lasucaai |
+
+---
+
+*Last updated: December 16, 2025*

controls-rework/docs/personal-dashboards.md

@@ -0,0 +1,71 @@
+# Personal Dashboard Rollout
+
+## Table: `user_tag_dashboards`
+
+Purpose: capture per-user widget selections and configuration for upcoming personal dashboards. One row represents a single widget/tag on a personal page.
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `id` | INT UNSIGNED PK | Auto-increment identifier. |
+| `user_id` | INT | References `members.member_id`; cascades on delete/update. |
+| `dashboard_key` | VARCHAR(64) | Allows multiple saved pages per user (e.g., `default`, `boilers`, `lab`). |
+| `widget_type` | VARCHAR(32) | `trend`, `stat`, `gauge`, etc. Defaults to `trend`. |
+| `panel_size` | VARCHAR(16) | Layout hint (`full`, `half`, `card`). |
+| `position_index` | SMALLINT UNSIGNED | Zero-based ordering for rendering widgets. |
+| `tag_name` | VARCHAR(255) | Historian/OPC tag identifier. |
+| `display_label` | VARCHAR(255) | Optional friendly label shown on the UI. |
+| `series_color` | VARCHAR(32) | Optional color override (hex/RGB). |
+| `preferred_axis` | VARCHAR(16) | Left/right axis preference for trend widgets. |
+| `update_interval_ms` | INT UNSIGNED | Polling cadence in milliseconds (default 5000). |
+| `time_window_minutes` | SMALLINT UNSIGNED | Historical span for trend charts (default 180). |
+| `rollup_function` | VARCHAR(16) | Server-side aggregation hint (`raw`, `avg`, `min`, `max`, `sum`, `delta`). |
+| `scale_min` / `scale_max` | DECIMAL(14,4) | Optional manual bounds for chart scaling. |
+| `threshold_low` / `threshold_high` | DECIMAL(14,4) | Alert thresholds for future highlighting. |
+| `is_active` | TINYINT(1) | Soft delete flag; inactive rows stay archived. |
+| `config_json` | LONGTEXT | Free-form JSON for extended configuration (annotations, alert styles, etc.). |
+| `last_viewed_at` | DATETIME | Updated when a user loads the dashboard. |
+| `created_at` / `updated_at` | TIMESTAMP | Audit columns managed by MySQL. |
+
+### Indexing
+- `uniq_user_dashboard_tag` ensures each user/dashboard/widget/tag combo is unique.
+- `idx_user_dashboard` supports fast lookups while rendering the personal page.
+
+> **FK note:** the `user_id` column is left signed (no `UNSIGNED`) so it matches the existing `members.member_id` definition in production. MySQL requires the parent and child column types to be identical (size and sign) for foreign keys to succeed.
+
+### Seed Checklist
+1. Run `sql/create_user_tag_dashboards.sql` in the LASUCA MySQL schema.
+2. Insert starter rows for the pilot account (use your `members.member_id`).
+3. Verify `SELECT * FROM user_tag_dashboards WHERE user_id = ?;` returns the seeded tags.
+
+### Implemented
+- Read endpoint: `data/personal/get_user_tags.php` returns active rows for the authenticated user and accepts an optional `dashboard` query parameter (defaults to `default`).
+- Prototype personal page: `personal/dashboard.php` renders saved widgets, surfaces metadata, and links out to the live trend view.
+
+## Table: `dashboard_row_catalog`
+
+Purpose: master reference of displayable rows (friendly labels, units, rendering hints) that users can pin to their personal dashboard.
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `id` | INT UNSIGNED PK | Auto-increment identifier. |
+| `row_key` | VARCHAR(64) | Stable identifier used by code/user selections. Unique. |
+| `friendly_name` | VARCHAR(255) | Picker label shown to users. |
+| `source_type` | VARCHAR(32) | Data origin (`historian`, `mysql`, `computed`, etc.). |
+| `source_id` | VARCHAR(128) | Identifier consumed by the source system (historian ID, OPC tag, etc.). |
+| `unit` | VARCHAR(32) | Unit of measure (psi, %, °F, rpm, etc.). |
+| `is_level_indicator` | TINYINT(1) | `1` = render with a level indicator/UI instead of plain numeric output. |
+| `value_precision` | TINYINT UNSIGNED | Optional decimal precision hint. |
+| `display_template` | VARCHAR(128) | Optional PHP include or template key for rendering the row. |
+| `requires_include` | VARCHAR(128) | Name of prerequisite include (e.g., `items.php`). |
+| `sort_order` | SMALLINT UNSIGNED | Default order when listing catalog entries. |
+| `metadata_json` | LONGTEXT | Extra configuration (thresholds, grouping, color hints). |
+| `created_at` / `updated_at` | TIMESTAMP | Audit columns managed by MySQL. |
+
+Seed checklist mirrors the prior pattern:
+1. Run `sql/create_dashboard_row_catalog.sql` in the LASUCA MySQL schema.
+2. Insert catalog rows for each displayable metric (include unit/level flag).
+3. Verify lookups via `SELECT * FROM dashboard_row_catalog ORDER BY sort_order, friendly_name;`.
+
+### Next Steps
+- Prototype a personal dashboard page that hydrates from the endpoint and renders the tags in slot order.
+- Build write endpoints/UI for managing selections, expose threshold alerts, and support multiple dashboards via `dashboard_key`.