Case study 07
Troubleshooting & Performance Engineering
My method
- Reproduce: capture the exact traceback.
- Measure: query the real data with SQL.
- Isolate: decide whether the cause is code, data or configuration.
- Fix: make the smallest correct change.
- Guard: add a regression test.
Incidents
| Symptom | Evidence & root cause | Fix | Verified |
|---|---|---|---|
| Field dashboard crashed with an RPC error | It looked like bad data, but a SQL check returned 0 bad rows. The real cause was a child_of [0] domain when a rep had no custody location. | Guarded the domain in code | Yes |
| ~500 false late-arrival violations in HR | Penalty logic ignored split shifts | Shift-aware calculation (new module version) | Yes |
| RPC error after older code was re-deployed | A stored filter referenced a field the older code no longer had | Restored the field; retested on a throwaway database | Yes |
| Database selector crashed on a fresh environment | A stripped add-ons copy in addons_path shadowed core modules | Corrected the add-ons path | Yes |
| Contracts failing to save for dozens of employees | Legacy payslip data from an earlier major version | Data/code fix | Yes |
| Caps exceeded and commission paid twice under parallel requests | Reproduced with a dedicated concurrency test harness | Locking plus regression tests | Yes |
| Database restore failed on an embeddings table | The vector extension was missing in PostgreSQL | Installed the extension and ran CREATE EXTENSION | Needs confirmation |
Lesson I now enforce
In one test run, the database filter also matched production and wrote a few cache rows into it. I documented the incident and adopted a rule: test runs must use an exact, non-production database filter.
Not claimed
I have no evidence of OOM or 502/504 incidents, so none are described. Worker and memory tuning appears in my configuration notes only.