From f251903874dd331de6b98fcb70de58d65176d980 Mon Sep 17 00:00:00 2001 From: Julian Tabel Date: Thu, 5 Feb 2026 19:06:33 +0100 Subject: [PATCH] Update seeding refactor bean to use PokeAPI/api-data JSON instead of CSV Co-Authored-By: Claude Opus 4.6 --- ...eding-to-use-pokeapi-csv-data-via-git-s.md | 45 ++++++++++--------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/.beans/nuzlocke-tracker-ya2a--refactor-seeding-to-use-pokeapi-csv-data-via-git-s.md b/.beans/nuzlocke-tracker-ya2a--refactor-seeding-to-use-pokeapi-csv-data-via-git-s.md index 798845c..d2358dc 100644 --- a/.beans/nuzlocke-tracker-ya2a--refactor-seeding-to-use-pokeapi-csv-data-via-git-s.md +++ b/.beans/nuzlocke-tracker-ya2a--refactor-seeding-to-use-pokeapi-csv-data-via-git-s.md @@ -3,19 +3,23 @@ title: Refactor seeding to use PokeAPI CSV data via git submodule status: draft type: task +priority: normal created_at: 2026-02-05T18:01:09Z -updated_at: 2026-02-05T18:01:09Z +updated_at: 2026-02-05T18:06:04Z --- ## Summary -Replace the current seeding approach (which uses the `pokebase` Python library to hit the PokeAPI REST API, then writes intermediate JSON files) with direct CSV parsing from the [PokeAPI/pokeapi](https://github.com/PokeAPI/pokeapi) repository's `data/v2/csv/` directory, pulled in as a git submodule. +Replace the current seeding approach (which uses the `pokebase` Python library to hit the PokeAPI REST API, then writes intermediate JSON files) with reading static JSON data from the [PokeAPI/api-data](https://github.com/PokeAPI/api-data) repository, pulled in as a git submodule. + +The `api-data` repo contains a static copy of the full PokeAPI output as JSON files at `data/api/v2/{endpoint}/{id}/index.json`, mirroring the REST API structure exactly. ## Motivation - **Eliminates network dependency**: No more hitting the PokeAPI REST API (or running a local instance) during seed generation -- **Faster**: Reading local CSVs is instant vs. hundreds of HTTP requests (even with pokebase caching) -- **More data available**: The CSVs contain the complete dataset, not just what we query for +- **Faster**: Reading local JSON files is instant vs. hundreds of HTTP requests (even with pokebase caching) +- **Minimal code change**: The JSON structure matches the API responses, so parsing logic stays similar to the current `fetch_pokeapi.py` +- **More data available**: The full dataset is available locally, not just what we query for - **Version-pinnable**: The git submodule can be pinned to a specific commit for reproducible builds - **Removes `pokebase` dependency**: One less runtime/dev dependency to maintain @@ -28,29 +32,26 @@ Replace the current seeding approach (which uses the `pokebase` Python library t ## Proposed Approach -1. **Add git submodule**: Add `https://github.com/PokeAPI/pokeapi` as a git submodule (e.g., at `backend/pokeapi-data/` or a top-level `data/pokeapi/` directory) -2. **Write CSV parser**: Create a new module that reads the relevant CSVs directly. Key CSV files include: - - `pokemon.csv`, `pokemon_species.csv`, `pokemon_types.csv` — Pokemon data - - `locations.csv`, `location_areas.csv`, `location_names.csv` — Location/route data - - `encounters.csv`, `encounter_slots.csv`, `encounter_methods.csv` — Encounter data - - `versions.csv`, `version_groups.csv`, `version_names.csv` — Game/version data - - `pokemon_evolution.csv`, `evolution_chains.csv`, `evolution_triggers.csv` — Evolution data - - `types.csv`, `type_names.csv` — Type data - - `regions.csv` — Region data -3. **Replace `fetch_pokeapi.py`**: The new CSV parser replaces the API-fetching script. It should produce the same (or equivalent) output that `loader.py` expects, or `loader.py` should be updated to accept the new data format. +1. **Add git submodule**: Add `https://github.com/PokeAPI/api-data` as a git submodule with `--depth 1` (e.g., at `data/pokeapi/` or `backend/pokeapi-data/`) +2. **Rewrite `fetch_pokeapi.py`**: Replace API calls with local JSON file reads from the submodule. The data lives at `data/api/v2/{endpoint}/{id}/index.json`. Key endpoints: + - `pokemon/{id}/` and `pokemon-species/{id}/` — Pokemon data & names + - `type/{id}/` — Type data + - `region/{id}/` — Region data with location refs + - `location/{id}/` — Locations with area refs + - `location-area/{id}/` — Location areas with encounter data + - `version/{id}/` and `version-group/{id}/` — Game/version data + - `evolution-chain/{id}/` — Evolution chain data +3. **Keep the same output format**: The rewritten script should still produce the same intermediate JSON files (`games.json`, `pokemon.json`, `firered.json`, etc.) so `run.py` and `loader.py` remain unchanged. 4. **Keep the override mechanism**: `evolution_overrides.json` should still work for manual corrections. -5. **Remove intermediate JSON files**: The generated JSON files in `seeds/data/` can be removed from version control since data now comes from the submodule. -6. **Remove `pokebase` dependency**: Remove from `pyproject.toml` / `requirements.txt`. -7. **Update documentation**: Update any setup/dev docs and the seed run command instructions. +5. **Remove `pokebase` dependency**: Remove from `pyproject.toml` / `requirements.txt`. +6. **Update documentation**: Update any setup/dev docs and the seed run command instructions. ## Checklist -- [ ] Add PokeAPI repo as a git submodule -- [ ] Identify and document all needed CSV files from the PokeAPI data -- [ ] Write CSV parsing module to replace `fetch_pokeapi.py` -- [ ] Update `run.py` and/or `loader.py` to work with the new data source +- [ ] Add `PokeAPI/api-data` repo as a git submodule (shallow clone) +- [ ] Rewrite `fetch_pokeapi.py` to read local JSON files from the submodule instead of calling the API +- [ ] Verify output JSON files match the current format (so `run.py`/`loader.py` stay unchanged) - [ ] Preserve evolution override mechanism -- [ ] Remove intermediate JSON seed data files from version control - [ ] Remove `pokebase` dependency - [ ] Test that seeding produces equivalent results - [ ] Update dev setup docs / seed run instructions \ No newline at end of file