Troubleshooting

Cause The absdb-server process is not running, or is listening on a different port.

Fix Start the server and verify it is listening:

# Start the server
./bin/absdb-server

# Verify it is running (should return {"status":"ok"})
curl http://localhost:8080/health

# Check what port it is on
./bin/absdb-server --port 5432   # if you need port 5432

Cause The host firewall (iptables/ufw/firewalld) is blocking incoming connections to port 5433.

Fix

# Ubuntu/Debian — allow port 5433
sudo ufw allow 5433/tcp

# RHEL/CentOS — allow port 5433
sudo firewall-cmd --permanent --add-port=5433/tcp
sudo firewall-cmd --reload

Cause Server is not binding to the right interface. By default it may bind to 127.0.0.1 only.

Fix Start server with --bind 0.0.0.0 to accept remote connections:

./bin/absdb-server --bind 0.0.0.0 --port 5433

Cause Absolute DB uses trust authentication by default — the connecting OS user must have a matching database user.

Fix Create the user or connect as an existing user:

# Connect as the default admin user first
psql -h localhost -p 5433 -U absdb

# Then create your user
CREATE USER myapp;

# Or just use -U to specify a user that exists
psql -h localhost -p 5433 -U absdb -d mydb

Cause JWT token has expired, the signing key doesn't match, or the clock skew between issuer and server exceeds the configured tolerance.

Fix Verify the token with your identity provider, ensure server time is synced (NTP), and check the --jwt-issuer and --jwt-audience config flags match your IdP settings.

Cause LDAP server address, bind DN, or port is incorrect. Port 636 (LDAPS) may need TLS configured.

Fix Verify LDAP connectivity independently, then check your --ldap-server, --ldap-bind-dn, and --ldap-port configuration.

Cause GCC build toolchain is not installed.

Fix

# Ubuntu/Debian
sudo apt update && sudo apt install -y gcc make build-essential git

# RHEL/Fedora
sudo dnf install -y gcc make git

# macOS
xcode-select --install
# or: brew install gcc make

Cause Building in debug mode, or cross-compiling for a different target. Release build requires make release (not make all).

Fix

# Always use make release for production binaries
make clean && make release

# Expected sizes (stripped, -Os + LTO):
# bin/absdb        ~154 KB (CLI)
# bin/absdb-server ~230 KB (server)
# bin/absdb-lite   ~154 KB (embedded)

Cause A platform-specific compiler version may emit additional warnings. Absolute DB is tested with GCC 12+ and Clang 15+.

Fix Upgrade your compiler to a supported version, or report the warning at GitHub Issues.

Cause The planner may not be using the index if selectivity estimates are off, or if the query bypasses the index.

Fix

-- Check query plan
EXPLAIN SELECT * FROM orders WHERE customer_id = 42;

-- Check currently running queries
SELECT * FROM absdb_active_queries;

-- Check advisor recommendations
SELECT * FROM absdb_advisor_recommendations;

Cause Very high vector dimensions (>1024) or too many vectors without quantisation reduce search speed.

Fix Enable Product Quantisation (PQ) for large-scale vector sets, or use Matryoshka truncation to reduce effective dimensionality:

-- Create HNSW index with PQ quantisation
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops)
WITH (quantization = 'pq', pq_segments = 96);

-- Or reduce dims with Matryoshka truncation
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops)
WITH (dims = 256);  -- truncate from 1536 to 256

Cause Buffer pool is configured too large for the available RAM.

Fix Reduce the buffer pool size:

# Set buffer pool to 25% of available RAM (example: 4 GB system)
./bin/absdb-server --buffer-pool-mb 1024

# Check idle RAM usage (should be ~4 MB minimal config)
./bin/absdb-server --buffer-pool-mb 64  # minimal test

Cause WAL retention window is too large, or WAL archiving to object storage is failing.

Fix Reduce the PITR retention window or fix the archive target:

# Reduce WAL retention (default keeps 7 days)
./bin/absdb-server --wal-retention-hours 24

# Move WAL to a separate disk
./bin/absdb-server --wal-dir /mnt/fast-ssd/absdb-wal

Cause Power loss or kill -9 during a write. Absolute DB uses CRC-32C checked WAL records with "Re-Read Before Shutdown" safety, but an abrupt crash may leave a partial record.

Fix The server performs automatic WAL recovery on startup. If recovery fails, restore from your last clean backup and replay WAL archives from that point.

Cause Raft internal port 9091 is blocked by firewall, or the --cluster-peers addresses are wrong.

Fix

# Allow Raft port on each node
sudo ufw allow 9091/tcp   # Raft consensus
sudo ufw allow 9092/tcp   # C-RAID replication

# Start with correct peer list
./bin/absdb-server \
  --cluster-peers node1:9091,node2:9091,node3:9091 \
  --node-id node3

Cause Network bandwidth between leader and replica is insufficient, or replica disk is slower than leader write rate.

Fix Check absdb_replication_status virtual table for lag metrics. Upgrade network or move replica to faster storage. C-RAID auto-balancer will detect and re-mirror from a faster node if the replica is consistently 3x slower.

Cause Certificate has expired, CN/SAN doesn't match the hostname, or client doesn't support TLS 1.3.

Fix

# Check certificate validity
openssl s_client -connect localhost:5433 -starttls postgres 2>&1 | grep -E 'subject|expire|verify'

# Check expiry
openssl x509 -in server.crt -noout -dates

# Renew certificate and reload (zero-downtime hot rotation supported)
# Then restart or send SIGHUP to reload certs

Cause Wrong key format, environment variable not set before server start, or key was issued for a different edition.

Fix

# Licence key format
# Professional:  PRO-XXXX-XXXX-XXXX-XXXX
# Enterprise:    ENT-XXXX-XXXX-XXXX-XXXX

# Set via environment variable (set BEFORE starting server)
export ADB_LICENSE_KEY=PRO-XXXX-XXXX-XXXX-XXXX
./bin/absdb-server

# Or pass as flag
./bin/absdb-server --license-key PRO-XXXX-XXXX-XXXX-XXXX

# Verify active edition
curl http://localhost:8080/health | grep edition

Cause Consumer is processing too slowly, or the 100 MB ring buffer is filling up. If the buffer overflows, the oldest unacknowledged events are overwritten.

Fix Speed up the consumer, or scale out to multiple consumers. Check the ACK cursor position:

-- Check CDC cursor status
SELECT * FROM _cdc_cursors;

-- Check ring buffer usage
SELECT * FROM absdb_stats WHERE metric LIKE 'cdc_%';

Cause The WAL archive does not contain the required segment for the target recovery time. This happens if WAL archiving was interrupted or the retention window was shorter than the restore target.

Fix Restore from the most recent full backup that is older than your target time, then apply available WAL segments from that point. Configure longer WAL retention to prevent this in future.

Cause DQ_LEVEL expectations are set to QUARANTINE and incoming data is failing quality checks.

Fix Review the quarantined rows and either fix the source data pipeline or adjust the DQ thresholds:

-- Review quarantined rows
SELECT * FROM _absdb_dq_quarantine LIMIT 50;

-- Check quality scores by table
SELECT * FROM _absdb_dq_profiles;

-- Get overall quality score
SELECT absdb_dq_score('orders');  -- returns 0-100