Initial commit

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-22 20:43:59 +00:00
commit 6c21525b79
23 changed files with 36533 additions and 0 deletions
@@ -0,0 +1,730 @@
+# RV50x Template Manager — Docker Edition
+
+A containerized web-based tool for managing configuration templates on Sierra Wireless AirLink RV50x modems. Runs as a full Docker stack including the web application, NocoDB, and PostgreSQL — start and stop the entire thing with a single command.
+
+---
+
+## Table of Contents
+
+1. [Stack Overview](#stack-overview)
+2. [Requirements](#requirements)
+3. [Project Structure](#project-structure)
+4. [First-Time Setup](#first-time-setup)
+5. [The .env File](#the-env-file)
+6. [Building and Starting](#building-and-starting)
+7. [NocoDB Setup](#nocodb-setup)
+8. [Updating the App](#updating-the-app)
+9. [Managing the Stack](#managing-the-stack)
+10. [Using Portainer](#using-portainer)
+11. [Migrating to a New Machine](#migrating-to-a-new-machine)
+12. [Reconnecting to a New NocoDB Instance](#reconnecting-to-a-new-nocodb-instance)
+13. [Data and Volumes](#data-and-volumes)
+14. [Troubleshooting](#troubleshooting)
+
+---
+
+## Stack Overview
+
+The stack runs three containers:
+
+| Container | Image | Purpose | Default Port |
+|---|---|---|---|
+| `rv50x-manager` | Built from `Dockerfile` | FastAPI web app + Playwright | Your choice |
+| `rv50x-nocodb` | `nocodb/nocodb:latest` | NocoDB UI and API | 8090 |
+| `rv50x-postgres` | `postgres:16-alpine` | PostgreSQL database for NocoDB | Internal only |
+
+All three containers communicate over a private internal Docker network. PostgreSQL is never exposed to the host — only NocoDB can reach it. The web app talks to NocoDB via the internal hostname `nocodb`.
+
+```
+Your Browser
+     │
+     ▼
+rv50x-manager (YOUR_PORT) ──→ nocodb (8090) ──→ postgres (internal)
+     │
+     ▼
+Your Modems (port 443, via Playwright)
+```
+
+---
+
+## Requirements
+
+- Docker Engine 24+ or Docker Desktop
+- Docker Compose v2 (`docker compose` or `docker-compose`)
+- Network access from the Docker host to your modems on port 443
+- Portainer (optional, but recommended for easy management)
+
+---
+
+## Project Structure
+
+```
+/opt/rv50x-manager/
+├── app.py                  ← FastAPI backend
+├── index.html              ← Web UI
+├── requirements.txt        ← Python dependencies
+├── Dockerfile              ← Container build instructions
+├── docker-compose.yml      ← Stack definition
+├── .env                    ← Your secrets and config (never commit this)
+├── .env.example            ← Template for .env
+└── .dockerignore           ← Files excluded from the Docker image
+```
+
+**Not needed in the Docker folder** (kept separately if you want CLI access):
+- `download.py` / `upload.py` — standalone CLI scripts
+- `modems.csv` — legacy device list
+- `.venv/` — Python virtual environment
+
+---
+
+## First-Time Setup
+
+### Step 1 — Install Docker
+
+**Fedora / RHEL / Rocky:**
+```bash
+sudo dnf install -y docker docker-compose-plugin
+sudo systemctl start docker
+sudo systemctl enable docker
+sudo usermod -aG docker $USER   # log out and back in after this
+```
+
+**Debian / Ubuntu:**
+```bash
+sudo apt-get install -y docker.io docker-compose-plugin
+sudo systemctl start docker
+sudo systemctl enable docker
+sudo usermod -aG docker $USER
+```
+
+Verify Docker is working:
+```bash
+docker run hello-world
+```
+
+### Step 2 — Copy project files to the host machine
+
+Create the install directory and set permissions, then copy your files:
+
+```bash
+# Create the directory and give your user ownership
+sudo mkdir -p /opt/rv50x-manager
+sudo chown $USER:$USER /opt/rv50x-manager
+
+# Copy all project files into it
+cp app.py index.html requirements.txt Dockerfile \
+   docker-compose.yml .env.example .dockerignore \
+   /opt/rv50x-manager/
+
+cd /opt/rv50x-manager
+```
+
+### Step 3 — Create your .env file
+
+```bash
+cd /opt/rv50x-manager
+cp .env.example .env
+nano .env    # or use any text editor
+```
+
+See [The .env File](#the-env-file) section below for what to put in each field.
+
+### Step 4 — Set your port
+
+Open `/opt/rv50x-manager/docker-compose.yml` and find this line under `rv50x-manager`:
+
+```yaml
+ports:
+  - "YOUR_PORT:8000"
+```
+
+Replace `YOUR_PORT` with the port number you want to use on the host machine. For example:
+
+```yaml
+ports:
+  - "8001:8000"
+```
+
+The app will then be accessible at `http://host-ip:8001`.
+
+### Step 5 — Build and start
+
+```bash
+cd /opt/rv50x-manager
+docker-compose up -d
+```
+
+This builds the `rv50x-manager` image and starts all three containers. The first build takes a few minutes because it downloads the Playwright base image and installs Chromium.
+
+### Step 6 — Set up NocoDB
+
+See [NocoDB Setup](#nocodb-setup) below.
+
+### Step 7 — Update .env with NocoDB IDs
+
+After importing your data into NocoDB, update `.env` with the real IDs and restart the manager:
+
+```bash
+docker-compose restart rv50x-manager
+```
+
+### Step 8 — Open the app
+
+```
+http://your-docker-host-ip:YOUR_PORT
+```
+
+---
+
+## The .env File
+
+Create this file by copying `.env.example` and filling in your values. It is read automatically by `docker-compose`. **Never commit this file to version control** — it contains passwords and API tokens.
+
+```bash
+# ── NocoDB connection ──────────────────────────────────────────────────────
+#
+# Use "http://nocodb:8080" to connect to the NocoDB container in this stack.
+# Use your external NocoDB URL if you prefer to point at an existing instance.
+NOCODB_URL=http://nocodb:8080
+
+# Your NocoDB API token.
+# Get it from: NocoDB → Profile (bottom-left) → Team & Settings → API Tokens
+# Tokens look like: eWU_ilelaCtNy1JzC7vf41DokkqFOovcLHM0zVml
+NOCODB_TOKEN=your-api-token-here
+
+# These three IDs come from the NocoDB browser URL after you import your data.
+# The URL structure is:
+#   http://host:8090/{org_id}/{BASE_ID}/{TABLE_ID}/{VIEW_ID}/table-name
+# Leave as placeholder for now — update after NocoDB setup (Step 6 above).
+NOCODB_BASE_ID=your-base-id-here
+NOCODB_TABLE_ID=your-table-id-here
+NOCODB_VIEW_ID=your-view-id-here
+
+# ── PostgreSQL ─────────────────────────────────────────────────────────────
+# Internal password used between NocoDB and PostgreSQL containers only.
+# You will never need to type this manually — make it long and strong.
+POSTGRES_PASSWORD=SomeLongStrongPassword123!
+
+# ── NocoDB JWT secret ──────────────────────────────────────────────────────
+# Any long random string used to sign NocoDB authentication tokens.
+# Generate one with: openssl rand -hex 32
+NC_JWT_SECRET=paste-a-long-random-string-here
+
+# ── Playwright timeouts (optional) ────────────────────────────────────────
+# Uncomment and adjust if your modems are slow to respond. Values in ms.
+# PAGE_TIMEOUT=90000
+# DOWNLOAD_TIMEOUT=120000
+# UPLOAD_TIMEOUT=120000
+# MAX_RETRIES=3
+```
+
+### Generating secrets
+
+```bash
+# Generate a strong JWT secret
+openssl rand -hex 32
+
+# Generate a strong password
+openssl rand -base64 24
+```
+
+---
+
+## Building and Starting
+
+### Build the image
+
+```bash
+docker-compose build
+```
+
+Only needed when `app.py`, `index.html`, or `requirements.txt` change. Skipped automatically if you just change `.env`.
+
+### Start the stack
+
+```bash
+docker-compose up -d
+```
+
+The `-d` flag runs containers in the background (detached mode). Without it the logs stream to your terminal and the stack stops when you close the terminal.
+
+### Check that everything is running
+
+```bash
+docker-compose ps
+```
+
+You should see all three containers with status `running` or `healthy`.
+
+### View logs
+
+```bash
+# All containers
+docker-compose logs -f
+
+# Just the web app
+docker-compose logs -f rv50x-manager
+
+# Just NocoDB
+docker-compose logs -f rv50x-nocodb
+```
+
+### Stop the stack
+
+```bash
+docker-compose stop
+```
+
+Stops all containers. Data is preserved in volumes. Start again with `docker-compose up -d`.
+
+### Remove containers (keep data)
+
+```bash
+docker-compose down
+```
+
+Removes containers but keeps all named volumes (your modem data, templates, downloads). Safe to run before a rebuild.
+
+### Remove everything including data ⚠
+
+```bash
+docker-compose down -v
+```
+
+**This deletes all volumes** including your NocoDB database and all template files. Only use this if you want a completely clean slate.
+
+---
+
+## NocoDB Setup
+
+After starting the stack for the first time, NocoDB needs to be configured before the web app can use it.
+
+### Step 1 — Open NocoDB
+
+```
+http://your-docker-host-ip:8090
+```
+
+### Step 2 — Create your account
+
+On first launch NocoDB will prompt you to create an admin account. Use a strong password and note the credentials.
+
+### Step 3 — Create a new base
+
+Click **+ New Base** and name it something like `Cell Modems`.
+
+### Step 4 — Import your modem data from CSV
+
+1. Export your current modem data from your existing NocoDB instance: **toolbar → Download → CSV**
+2. In the new NocoDB, click **+ Add or import** → **Import from CSV**
+3. Upload the CSV file
+4. NocoDB will auto-detect all columns including `hostname`, `ip_address`, `dept`, `password`, etc.
+5. Confirm the import
+
+### Step 5 — Recreate filtered views
+
+The CSV import brings the data but not the views. Recreate them manually:
+
+**Electric view:**
+1. In the left sidebar, click **+ Add View** → **Grid**
+2. Name it `Electric`
+3. Click **Filter** → **+ Add Filter**
+4. Set: `dept` `is` `ELEC`
+
+**Gas & Water view:**
+1. Click **+ Add View** → **Grid**
+2. Name it `Gas & Water`
+3. Click **Filter** → **+ Add Filter**
+4. Set: `dept` `is` `GW`
+
+### Step 6 — Get the IDs from the browser URL
+
+Navigate to your Cell Modems table. The URL will look like:
+
+```
+http://host:8090/abc123def/BASE_ID_HERE/TABLE_ID_HERE/VIEW_ID_HERE/cell-modems
+```
+
+Copy `BASE_ID_HERE`, `TABLE_ID_HERE`, and `VIEW_ID_HERE` (use the All view ID).
+
+### Step 7 — Generate an API token
+
+1. Click your profile avatar (bottom-left)
+2. Go to **Team & Settings → API Tokens**
+3. Click **Add Token**, give it a name like `rv50x-manager`
+4. Copy the token
+
+### Step 8 — Update .env and restart
+
+```bash
+cd /opt/rv50x-manager
+nano .env
+# Update NOCODB_TOKEN, NOCODB_BASE_ID, NOCODB_TABLE_ID, NOCODB_VIEW_ID
+
+docker-compose restart rv50x-manager
+```
+
+### Step 9 — Verify the connection
+
+Open the web app and check that the device groups show the correct counts. If devices appear, the connection is working.
+
+---
+
+## Updating the App
+
+When `app.py` or `index.html` change:
+
+```bash
+# Stop the stack
+docker-compose stop
+
+# Rebuild the manager image (NocoDB and Postgres don't need rebuilding)
+docker-compose build rv50x-manager
+
+# Start everything back up
+docker-compose up -d
+```
+
+When only `.env` changes (NocoDB IDs, token, timeouts):
+
+```bash
+docker-compose restart rv50x-manager
+```
+
+When `requirements.txt` changes (new Python packages):
+
+```bash
+docker-compose build rv50x-manager
+docker-compose up -d
+```
+
+---
+
+## Managing the Stack
+
+### Start individual services
+
+```bash
+docker-compose up -d postgres    # start just postgres
+docker-compose up -d nocodb      # start just nocodb
+docker-compose up -d rv50x-manager  # start just the web app
+```
+
+### Restart a single service
+
+```bash
+docker-compose restart rv50x-manager
+```
+
+### Rebuild and restart a single service
+
+```bash
+docker-compose up -d --build rv50x-manager
+```
+
+### Execute a command inside a running container
+
+```bash
+# Open a shell in the web app container
+docker exec -it rv50x-manager bash
+
+# Check Python packages installed
+docker exec rv50x-manager pip list
+
+# Test NocoDB connection from inside the container
+docker exec rv50x-manager curl -s http://nocodb:8080/api/v1/health
+```
+
+### Access the PostgreSQL database directly
+
+```bash
+docker exec -it rv50x-postgres psql -U nocodb -d nocodb
+```
+
+---
+
+## Using Portainer
+
+Portainer gives you a browser-based UI to manage the entire stack without needing SSH.
+
+### Install Portainer (if not already installed)
+
+```bash
+docker volume create portainer_data
+docker run -d \
+  -p 9000:9000 \
+  --name portainer \
+  --restart=always \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  -v portainer_data:/data \
+  portainer/portainer-ce:latest
+```
+
+Then open `http://host-ip:9000` and create your admin account.
+
+### Deploy the stack via Portainer
+
+1. Go to **Stacks → + Add Stack**
+2. Name it `rv50x`
+3. Paste the contents of `docker-compose.yml` into the editor
+4. Scroll down to **Environment Variables**
+5. Click **+ Add an environment variable** for each line in your `.env` file:
+
+| Name | Value |
+|---|---|
+| `NOCODB_URL` | `http://nocodb:8080` |
+| `NOCODB_TOKEN` | your token |
+| `NOCODB_BASE_ID` | your base ID |
+| `NOCODB_TABLE_ID` | your table ID |
+| `NOCODB_VIEW_ID` | your view ID |
+| `POSTGRES_PASSWORD` | your password |
+| `NC_JWT_SECRET` | your secret |
+
+6. Click **Deploy the stack**
+
+### Start and stop via Portainer
+
+- **Stacks → rv50x → Start** — starts all containers
+- **Stacks → rv50x → Stop** — stops all containers
+- Individual containers: **Containers** list → click the start/stop icons
+
+### Update the stack via Portainer
+
+1. **Stacks → rv50x → Editor**
+2. Paste updated `docker-compose.yml`
+3. Click **Update the stack**
+
+For code changes (`app.py`, `index.html`), you need to rebuild the image first from the command line:
+
+```bash
+docker-compose build rv50x-manager
+```
+
+Then update the stack in Portainer to pick up the new image.
+
+---
+
+## Migrating to a New Machine
+
+### Step 1 — Export your NocoDB data
+
+Before migrating, export your modem data as CSV from the current NocoDB instance so you can reimport it on the new machine.
+
+In NocoDB: **Cell Modems table → toolbar → Download → CSV**
+
+### Step 2 — Copy project files
+
+From the old machine:
+
+```bash
+scp app.py index.html requirements.txt Dockerfile docker-compose.yml \
+    .env.example .dockerignore youruser@new-host:/tmp/rv50x-transfer/
+```
+
+**Do not copy `.env`** over an insecure connection — recreate it manually on the new machine.
+
+On the new machine, move files into place:
+
+```bash
+sudo mkdir -p /opt/rv50x-manager
+sudo chown $USER:$USER /opt/rv50x-manager
+cp /tmp/rv50x-transfer/* /opt/rv50x-manager/
+```
+
+### Step 3 — Copy template files (optional)
+
+If you want to keep your existing XML templates, downloaded configs, and upload files:
+
+```bash
+# Export volumes from the old machine
+cd /opt/rv50x-manager
+
+docker run --rm \
+  -v rv50x-manager_xml_templates:/data \
+  -v $(pwd):/backup \
+  alpine tar czf /backup/xml_templates.tar.gz -C /data .
+
+docker run --rm \
+  -v rv50x-manager_template_downloads:/data \
+  -v $(pwd):/backup \
+  alpine tar czf /backup/template_downloads.tar.gz -C /data .
+
+docker run --rm \
+  -v rv50x-manager_template_uploads:/data \
+  -v $(pwd):/backup \
+  alpine tar czf /backup/template_uploads.tar.gz -C /data .
+
+# Copy archives to new machine
+scp xml_templates.tar.gz template_downloads.tar.gz template_uploads.tar.gz \
+    youruser@new-host:/opt/rv50x-manager/
+```
+
+### Step 4 — Set up the new machine
+
+```bash
+# Install Docker (see First-Time Setup above)
+
+# Files should already be in /opt/rv50x-manager from Step 2
+cd /opt/rv50x-manager
+
+# Create fresh .env with new passwords and secrets
+cp .env.example .env
+nano .env
+
+# Set your port
+nano docker-compose.yml
+
+# Build and start
+docker-compose up -d
+```
+
+### Step 5 — Restore template files (if copied)
+
+```bash
+cd /opt/rv50x-manager
+
+# Restore xml_templates
+docker run --rm \
+  -v rv50x-manager_xml_templates:/data \
+  -v $(pwd):/backup \
+  alpine tar xzf /backup/xml_templates.tar.gz -C /data
+
+# Restore template_downloads
+docker run --rm \
+  -v rv50x-manager_template_downloads:/data \
+  -v $(pwd):/backup \
+  alpine tar xzf /backup/template_downloads.tar.gz -C /data
+
+# Restore template_uploads
+docker run --rm \
+  -v rv50x-manager_template_uploads:/data \
+  -v $(pwd):/backup \
+  alpine tar xzf /backup/template_uploads.tar.gz -C /data
+```
+
+### Step 6 — Set up NocoDB on the new machine
+
+Follow the [NocoDB Setup](#nocodb-setup) section — import your CSV, recreate views, get new IDs.
+
+### Step 7 — Update .env with new NocoDB IDs
+
+```bash
+nano .env
+docker-compose restart rv50x-manager
+```
+
+---
+
+## Reconnecting to a New NocoDB Instance
+
+If your NocoDB database is corrupted, rebuilt, or moved to a new server:
+
+### Step 1 — Get the new connection details
+
+1. Open the new NocoDB in your browser
+2. Import your modem data from CSV
+3. Recreate the Electric and Gas & Water filtered views
+4. Go to **Profile → Team & Settings → API Tokens** → create a new token
+5. Copy the base ID, table ID, and view ID from the browser URL
+
+### Step 2 — Test the connection
+
+```bash
+curl -H "xc-token: YOUR_NEW_TOKEN" \
+  "http://new-nocodb-host:8090/api/v1/db/data/noco/BASE_ID/TABLE_ID?limit=1"
+```
+
+A successful response returns JSON with a `list` array containing your first modem row.
+
+### Step 3 — Update .env
+
+```bash
+cd /opt/rv50x-manager
+nano .env
+```
+
+Update these values:
+
+```bash
+NOCODB_URL=http://new-host:8090        # or http://nocodb:8080 if using the stack
+NOCODB_TOKEN=your-new-token
+NOCODB_BASE_ID=your-new-base-id
+NOCODB_TABLE_ID=your-new-table-id
+NOCODB_VIEW_ID=your-new-view-id
+```
+
+### Step 4 — Restart the manager
+
+```bash
+cd /opt/rv50x-manager
+docker-compose restart rv50x-manager
+```
+
+Or in Portainer: **Stacks → rv50x → rv50x-manager → Restart**
+
+---
+
+## Data and Volumes
+
+All persistent data lives in Docker named volumes. They survive `docker-compose down` and rebuilds, and are only deleted with `docker-compose down -v`.
+
+| Volume | Contents | Maps to container path |
+|---|---|---|
+| `rv50x_template_manager_postgres_data` | NocoDB database | `/var/lib/postgresql/data` |
+| `rv50x_template_manager_nocodb_data` | NocoDB config and uploads | `/usr/app/data` |
+| `rv50x_template_manager_template_downloads` | Downloaded modem configs + reports | `/data/template_downloads` |
+| `rv50x_template_manager_template_uploads` | Staged XML files for upload + reports | `/data/template_uploads` |
+| `rv50x_template_manager_xml_templates` | Your XML builder templates | `/data/xml_templates` |
+
+### List all volumes
+
+```bash
+docker volume ls | grep rv50x
+```
+
+### Back up a volume
+
+```bash
+# Backs up the xml_templates volume to a tar file in the current directory
+docker run --rm \
+  -v rv50x_template_manager_xml_templates:/data \
+  -v $(pwd):/backup \
+  alpine tar czf /backup/xml_templates_backup.tar.gz -C /data .
+```
+
+### Back up the NocoDB database
+
+```bash
+cd /opt/rv50x-manager
+docker exec rv50x-postgres \
+  pg_dump -U nocodb nocodb > nocodb_backup_$(date +%Y%m%d).sql
+```
+
+### Restore the NocoDB database
+
+```bash
+cd /opt/rv50x-manager
+docker exec -i rv50x-postgres \
+  psql -U nocodb nocodb < nocodb_backup_20260413.sql
+```
+
+---
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `docker-compose: command not found` | Using older Docker without compose plugin | Use `docker compose` (space not hyphen) or install `docker-compose-plugin` |
+| Build fails with "no space left on device" | Docker image cache full | Run `docker system prune` to free space |
+| `rv50x-manager` exits immediately after start | App crash on startup — bad .env values | Run `docker-compose logs rv50x-manager` to see the error |
+| NocoDB shows "Service Unavailable" | Postgres not ready yet | Wait 30s and refresh — healthcheck retries handle this automatically |
+| Web app shows 0 devices | NocoDB IDs wrong or token invalid | Test with curl (see Reconnecting section), update .env, restart manager |
+| `ERR_BASE_NOT_FOUND` in curl test | Wrong NOCODB_BASE_ID | Re-read the ID from the NocoDB browser URL |
+| `ERR_AUTHENTICATION_REQUIRED` | Wrong or expired token | Regenerate token in NocoDB → API Tokens |
+| Modems show in All but not Electric/Gas & Water | Wrong dept field values | Check actual `dept` values in NocoDB — must be exactly `ELEC` and `GW` |
+| Playwright / Chromium crashes | Missing system library | The Playwright base image should have everything — check `docker-compose logs rv50x-manager` |
+| Template upload times out | Modem slow or unreachable | Increase `UPLOAD_TIMEOUT` in `.env`, restart manager |
+| Can't reach NocoDB at port 8090 | Port conflict or firewall | Change the left port in `docker-compose.yml` under `nocodb:` ports |
+| Can't reach web app | Port conflict or firewall | Change `YOUR_PORT` in `docker-compose.yml` |
+| `permission denied` on docker commands | User not in docker group | Run `sudo usermod -aG docker $USER` then log out and back in |
+| Volume data missing after `docker-compose down` | Used `down -v` by mistake | Data is gone — restore from backup or re-import CSV |