changelog.md for @fin.cx/opendata

2025-11-07 - 3.5.0 - feat(stocks) 
 Add provider fetch limits, intraday incremental fetch, cache deduplication, and provider safety/warning improvements 
 
 Add maxRecords and defaultIntradayLimit to IProviderConfig to control maximum records per request and default intraday limits. 
 CoinGecko provider: enforce maxRecords when processing historical data, warn when large historical/intraday results are returned without explicit limits, preserve priority mappings when rebuilding the coin cache, and improve cache load logging. 
 Marketstack provider: make safety maxRecords configurable, apply a configurable default intraday limit, warn when no explicit limit is provided, and ensure effective limits are applied to returned results. 
 StockPriceService: always attempt incremental fetch for intraday requests without a date to fetch only new data since the last cached timestamp and fall back to full fetch when necessary. 
 StockPriceService: deduplicate price arrays by timestamp before caching and after merges to avoid duplicate timestamps and reduce cache bloat. 
 Introduce StockDataService for unified access to prices and fundamentals with automatic enrichment (market cap, P/E, price-to-book) and caching improvements. 
 Various cache/TTL improvements and safer default behaviors for intraday, historical and live data to improve performance and memory usage. 
 
 2025-11-06 - 3.4.0 - feat(stocks) 
 Introduce unified stock data service, new providers, improved caching and German business data tooling 
 
 Add StockDataService: unified API to fetch price + fundamentals with automatic enrichment and batch support 
 Introduce BaseProviderService abstraction and refactor provider management, caching and retry logic 
 Enhance StockPriceService: unified getData, discriminated union request types, data-type aware TTLs and smarter cache keys 
 Add Marketstack provider with intraday/EOD selection, pagination, OHLCV and exchange filtering 
 Add CoinGecko provider with robust rate-limiting, coin ID resolution and crypto support (current, historical, intraday) 
 Add SEC EDGAR fundamentals provider: CIK lookup, company facts parsing, rate limiting and caching 
 Improve FundamentalsService: unified fetching, caching and enrichment helpers (enrichWithPrice, enrichBatchWithPrices) 
 Enhance Yahoo provider and other provider mappings for better company metadata and market state handling 
 Add German business data tooling: JsonlDataProcessor for JSONL bulk imports, HandelsRegister browser automation with download handling and parsing 
 Expose OpenData entry points: DB init, JSONL processing and Handelsregister integration; add readme/docs and usage examples 
 
 2025-11-02 - 3.3.0 - feat(stocks/CoinGeckoProvider) 
 Add CoinGecko provider for cryptocurrency prices, export and tests, and update documentation 
 
 Implemented CoinGeckoProvider with rate limiting, coin-id resolution, and support for current, batch, historical and intraday price endpoints 
 Added unit/integration tests for CoinGecko: test/test.coingecko.node+bun+deno.ts 
 Exported CoinGeckoProvider from ts/stocks/index.ts 
 Updated README and readme.hints.md with CoinGecko usage, provider notes and examples 
 Added .claude/settings.local.json with webfetch and bash permissions required for testing and CI 
 
 2025-11-01 - 3.2.2 - fix(handelsregister) 
 Correct screenshot path handling in HandelsRegister and add local tool permissions 
 
 ts/classes.handelsregister.ts: Replace string concatenation for screenshot path with a template literal and explicit string assertion to ensure the path is formed correctly for page.screenshot() and avoid type issues. 
 Add .claude/settings.local.json: Introduce local Claude settings that grant specific tool permissions used during development and testing (bash commands, web fetches, pnpm build, tstest, etc.). 
 
 2025-11-01 - 3.2.1 - fix(stocks/providers/provider.secedgar) 
 Improve SEC EDGAR provider networking and error handling, update plugin path import, bump dev deps and add/refresh tests and lockfile 
 
 SEC EDGAR provider: switch from SmartRequest to native fetch for ticker list and company facts, add AbortController-based timeouts, handle gzip automatically, improve response validation and error messages, and keep CIK/ticker-list caching 
 Improve timeout and rate-limit handling in SecEdgarProvider (uses native fetch + explicit timeout clear), plus clearer logging on failures 
 Update ts/plugins import to use node:path for Node compatibility 
 Bump devDependencies: @git.zone/tsrun to ^1.6.2 and @git.zone/tstest to ^2.7.0; bump @push.rocks/smartrequest to ^4.3.4 
 Add and refresh comprehensive test files (node/bun/deno variants) for fundamentals, marketstack, secedgar and stockdata services 
 Add deno.lock (dependency lock) and a local .claude/settings.local.json for CI/permissions 
 
 2025-11-01 - 3.2.0 - feat(StockDataService) 
 Add unified StockDataService and BaseProviderService with new stockdata interfaces, provider integrations, tests and README updates 
 
 Introduce StockDataService: unified API to fetch prices and fundamentals with automatic enrichment and caching 
 Add IStockData and IStockDataServiceConfig interfaces to define combined price+fundamentals payloads and configuration 
 Implement BaseProviderService abstraction to share provider registration, health, stats and caching logic 
 Add classes.stockdataservice.ts implementing batch/single fetch, enrichment, caching, health checks and provider stats 
 Export new stockdata module and classes from ts/stocks/index.ts 
 Add comprehensive tests: test/test.stockdata.service.node.ts to cover setup, provider registration, fetching, caching, enrichment, health and error handling 
 Update README with Unified Stock Data API examples, usage, and documentation reflecting new unified service 
 Minor infra: add .claude/settings.local.json permissions for local tooling and web fetch domains 
 
 2025-11-01 - 3.1.0 - feat(fundamentals) 
 Add FundamentalsService and SEC EDGAR provider with caching, rate-limiting, tests, and docs updates 
 
 Introduce FundamentalsService to manage fundamentals providers, caching, retry logic and provider statistics 
 Add SecEdgarProvider to fetch SEC EDGAR company facts (CIK lookup, company facts parsing) with rate limiting and local caches 
 Expose fundamentals interfaces and services from ts/stocks (exports updated) 
 Add comprehensive tests for FundamentalsService and SecEdgarProvider (new test files) 
 Update README with new Fundamentals module documentation, usage examples, and configuration guidance 
 Implement caching and TTL handling for fundamentals data and provider-specific cache TTL support 
 Add .claude/settings.local.json (local permissions) and various test improvements 
 
 2025-10-31 - 3.0.0 - BREAKING CHANGE(stocks) 
 Unify stock provider API to discriminated IStockDataRequest and add company name/fullname enrichment 
 
 Replace legacy provider methods (fetchPrice/fetchPrices) with a single fetchData(request: IStockDataRequest) on IStockProvider — providers must be migrated to the new signature. 
 Migrate StockPriceService to the unified getData(request: IStockDataRequest) API. Convenience helpers getPrice/getPrices now wrap getData. 
 Add companyName and companyFullName fields to IStockPrice and populate them in provider mappings (Marketstack mapping updated; Yahoo provider updated to support the unified API). 
 MarketstackProvider: added buildCompanyFullName helper and improved mapping to include company identification fields and full name formatting. 
 YahooFinanceProvider: updated to implement fetchData and to route current/batch requests through the new unified request types; historical/intraday throw explicit errors. 
 Updated tests to exercise the new unified API, company-name enrichment, caching behavior, and provider direct methods. 
 Note: This is a breaking change for external providers and integrations that implemented the old fetchPrice/fetchPrices API. Bump major version. 
 
 2025-10-31 - 2.1.0 - feat(stocks) 
 Add unified stock data API (getData) with historical/OHLCV support, smart caching and provider enhancements 
 
 Introduce discriminated union request types (IStockDataRequest) and a unified getData() method (replaces legacy getPrice/getPrices for new use cases) 
 Add OHLCV fields (open, high, low, volume, adjusted) and metadata (dataType, fetchedAt) to IStockPrice 
 Implement data-type aware smart caching with TTLs (historical = never expire, EOD = 24h, live = 30s, intraday matches interval) 
 Extend StockPriceService: new getData(), data-specific cache keys, cache maxEntries increased (default 10000), and TTL-aware add/get cache logic 
 Enhance Marketstack provider: unified fetchData(), historical date-range retrieval with pagination, exchange filtering, batch current fetch, OHLCV mapping, and intraday placeholder 
 Update Yahoo provider to include dataType and fetchedAt (live data) and maintain legacy fetchPrice/fetchPrices compatibility 
 Add/adjust tests to cover unified API, historical retrieval, OHLCV presence and smart caching behavior; test setup updated to require explicit OpenData directory paths 
 Update README to document v2.1 changes, migration examples, and new stock provider capabilities 
 
 2025-10-31 - 2.0.0 - BREAKING CHANGE(OpenData) 
 Require explicit directory paths for OpenData (nogit/download/germanBusinessData); remove automatic .nogit creation; update HandelsRegister, JsonlDataProcessor, tests and README. 
 
 Breaking: OpenData constructor now requires a config object with nogitDir, downloadDir and germanBusinessDataDir. The constructor will throw if these paths are not provided. 
 Removed automatic creation/export of .nogit/download/germanBusinessData from ts/paths. OpenData.start now ensures the required directories exist. 
 HandelsRegister API changed: constructor now accepts downloadDir and manages its own unique download folder; screenshot and download paths now use the configured downloadDir. 
 JsonlDataProcessor now accepts a germanBusinessDataDir parameter and uses it when ensuring/storing data instead of relying on global paths. 
 Updated tests to provide explicit path configuration (tests now set testNogitDir, testDownloadDir, testGermanBusinessDataDir and write outputs accordingly) and to use updated constructors and qenv usage. 
 Documentation updated (README) to document the breaking change and show examples for required directory configuration when instantiating OpenData. 
 Added .claude/settings.local.json for local permissions/config used in development/CI environments. 
 
 2025-10-11 - 1.7.0 - feat(stocks) 
 Add Marketstack provider (EOD) with tests, exports and documentation updates 
 
 Add MarketstackProvider implementation (ts/stocks/providers/provider.marketstack.ts) providing EOD single and batch fetching, availability checks and mapping to IStockPrice. 
 Export MarketstackProvider from ts/stocks/index.ts so it is available via the public API. 
 Add comprehensive Marketstack tests (test/test.marketstack.node.ts) covering registration, health checks, single/batch fetches, caching, ticker/market validation, provider stats and sample output. 
 Update README with Marketstack usage examples, configuration, API key instructions and provider/health documentation. 
 Bump dev dependency @git.zone/tstest to ^2.4.2 in package.json. 
 Add project helper/config files (.claude/settings.local.json, .serena/project.yml and .serena/.gitignore) to support CI/tooling. 
 
 2025-09-24 - 1.6.1 - fix(stocks) 
 Fix Yahoo provider request handling and bump dependency versions 
 
 Refactored Yahoo Finance provider to use SmartRequest.create() builder and await response.json() for HTTP responses (replaces direct getJson usage). 
 Improved batch and single-price fetching to use the SmartRequest API, keeping User-Agent header and timeouts. 
 Added a compile-time type-check alias to ensure IStockPrice matches tsclass.finance.IStockPrice. 
 Bumped development and runtime dependency versions (notable bumps include @git.zone/tsbuild, @git.zone/tstest, @push.rocks/qenv, @push.rocks/smartarchive, @push.rocks/smartdata, @push.rocks/smartfile, @push.rocks/smartlog, @push.rocks/smartpath, @push.rocks/smartrequest, @tsclass/tsclass). 
 Added .claude/settings.local.json to grant local CI permissions for a few Bash commands. 
 
 2025-07-12 - 1.6.0 - feat(readme) 
 Revamp documentation and package description for enhanced clarity 
 
 Restructured README to highlight real-time stock data and German business data, streamlining quick start and advanced examples 
 Updated package.json description to better reflect library capabilities 
 Added .claude/settings.local.json to define permissions for external tools 
 Refined code examples in tests and documentation for improved clarity and consistency 
 
 2025-04-09 - 1.5.3 - fix(test) 
 Await file writes in Handelsregister tests to ensure all downloads complete before test end 
 
 Replaced array.map with await Promise.all to properly await asynchronous file writes in test/test.handelsregister.ts 
 Improved robustness of asynchronous operations in test suite 
 
 2025-04-09 - 1.5.2 - fix(readme) 
 Improve .env configuration code block formatting in documentation 
 
 Wrap the .env variables block in triple backticks for clarity 
 Ensure consistency in the Markdown styling of code snippets 
 
 2025-04-09 - 1.5.1 - fix(core) 
 No changes detected in project files or documentation. This commit is a placeholder to record that nothing was updated. 
 2025-04-09 - 1.5.0 - feat(documentation) 
 Enhance project metadata and documentation with comprehensive usage examples, updated descriptions, and improved keywords. 
 
 Updated npmextra.json and package.json to refine the project description and keyword list. 
 Expanded readme.md with detailed sections on environment setup, CRUD operations, bulk JSONL processing, and advanced Handelsregister integrations. 
 Included advanced workflow examples and error handling strategies in the documentation. 
 Adjusted test cases (e.g. in test/test.handelsregister.ts) to reflect changes in company name usage. 
 
 2025-04-08 - 1.4.6 - fix(tests & jsonl) 
 Improve test structure and refine JSONL parsing for incomplete data 
 
 Refactored test files to remove redundant get-specific-company tests in test.ts and added missing tests in test.handelsregister.ts 
 Updated JSONL data processor to conditionally parse remaining data when available 
 
 2025-04-05 - 1.4.5 - fix(metadata) 
 Update repository, bugs, and homepage URLs to code.foss.global 
 
 Repository URL updated from gitlab.com to code.foss.global 
 Bugs URL updated from gitlab.com to code.foss.global 
 Homepage URL updated to code.foss.global 
 
 2025-04-05 - 1.4.4 - fix(dependencies & tests) 
 Update dependency versions and refine test search query 
 
 Bumped versions for several dependencies in package.json, including @git.zone/tsbuild, @git.zone/tsbundle, @git.zone/tstest, @push.rocks/tapbundle, @push.rocks/smartdata, @push.rocks/smartfile, @push.rocks/smartpromise, @push.rocks/smartrequest, and @tsclass/tsclass 
 Updated test file to replace the search query 'Volkswagen' with 'LADR' 
 Re-enabled the build initial data test by removing tap.skip 
 
 2025-01-07 - 1.4.3 - fix(test) 
 Corrected index value in test for fetching specific company data 
 
 Updated the index from 8 to 7 for the germanParsedRegistration fetch in test 
 
 2025-01-07 - 1.4.2 - fix(core) 
 Fix concurrency and download handling in HandelsRegister class and adjust test cases 
 
 Improved the clickFindButton function to include an argument for results limit. 
 Enhanced the downloadFile function to rename and ensure files are correctly handled. 
 Updated searchCompany method to allow specifying a limit on the number of search results. 
 Adjusted test cases to select specific test data indices and output test files to a dedicated directory. 
 
 2025-01-04 - 1.4.1 - fix(core) 
 Fix issues with JSONL data processing and improve error handling in business record validation 
 
 Fixed JSONL data processing by adding concurrent processing for each JSON line to enhance performance. 
 Added validation logic in BusinessRecord class to ensure that the mandatory fields are checked. 
 Adjusted environment variable loading in OpenData class to ensure correct database initialization. 
 Included missing dependencies and exports in the project files to ensure proper functionality. 
 
 2025-01-04 - 1.4.0 - feat(HandelsRegister) 
 Add file download functionality to HandelsRegister 
 
 Implemented file download feature in the HandelsRegister class. 
 Configured pages in Puppeteer to allow downloads and set download paths. 
 Parsed German registration information with more robust error handling. 
 Added specific methods for downloading and handling 'SI' and 'AD' files. 
 
 2025-01-03 - 1.3.1 - fix(HandelsRegister) 
 Refined HandelsRegister functionality for better error handling and response capture. 
 
 Improved parsing logic in parseGermanRegistration function. 
 Enhanced navigateToPage and clickFindButton methods with error messages for clarity. 
 Implemented a new responseListener to handle and log HTTP responses correctly. 
 
 2025-01-03 - 1.3.0 - feat(core) 
 Enhanced data handling capabilities and improved company search functionalities. 
 
 Updated business record handling to support more registration types. 
 Improved search capabilities for fetching company data with refined registration type matching. 
 Added robust logging for JSONL data processing with early exit on successful parse. 
 Reorganized test cases to include specific company data retrieval. 
 
 2025-01-02 - 1.2.1 - fix(BusinessRecord) 
 Add missing field registrationType to BusinessRecord data 
 
 Introduced the 'registrationType' field to the BusinessRecord data schema with possible values 'HRA' or 'HRB'. 
 
 2025-01-02 - 1.2.0 - feat(core) 
 Integrate Handelsregister search for company data retrieval 
 
 Added support for searching company data via Handelsregister. 
 Replaced GermanBusinessData functionality with JsonlDataProcessor. 
 Included smartbrowser dependency for handling web requests to Handelsregister. 
 
 2025-01-01 - 1.1.5 - fix(GermanBusinessData) 
 Add console log for total records processed at the end of the stream. 
 
 Ensure that the number of records processed is logged at the end of data stream processing. 
 
 2024-12-31 - 1.1.4 - fix(documentation) 
 Update description and keywords in package.json 
 
 Corrected the package description to reflect the focus on managing, accessing, and updating open data with MongoDB integration. 
 Expanded the keywords in the package metadata to include data integration and processing terms. 
 Improved README.md with more extensive setup, usage, and introduction of the library's functionalities. 
 
 2024-12-31 - 1.1.3 - fix(core) 
 Added missing license file for project completeness. 
 
 Introduced a LICENSE file to the project, ensuring clarity on software usage permissions. 
 
 2024-12-31 - 1.1.2 - fix(GermanBusinessData) 
 Ensure unique ID generation for BusinessRecord 
 
 Added generation of a new ID for each BusinessRecord in GermanBusinessData. 
 Ensures each business record has a unique identifier. 
 
 2024-12-31 - 1.1.1 - fix(dependencies) 
 Update dependencies and devDependencies to newer versions. 
 
 @git.zone/tsbuild from ^2.1.25 to ^2.2.0 
 @git.zone/tsbundle from ^2.0.5 to ^2.1.0 
 @git.zone/tsrun from ^1.2.46 to ^1.3.3 
 @git.zone/tstest from ^1.0.84 to ^1.0.90 
 @push.rocks/tapbundle from ^5.0.15 to ^5.5.4 
 @types/node from ^20.9.0 to ^22.10.2 
 @push.rocks/qenv from ^6.0.4 to ^6.1.0 
 @push.rocks/smartarchive from ^4.0.19 to ^4.0.39 
 @push.rocks/smartdata from ^5.0.33 to ^5.2.10 
 @push.rocks/smartfile from ^11.0.0 to ^11.0.23 
 @push.rocks/smartpath from ^5.0.11 to ^5.0.18 
 @push.rocks/smartpromise from ^4.0.3 to ^4.0.4 
 @push.rocks/smartrequest from ^2.0.21 to ^2.0.23 
 @push.rocks/smartstream from ^3.0.30 to ^3.2.5 
 
 2024-12-31 - 1.1.0 - feat(core) 
 Enhanced data handling and retrieval features, improved usage documentation 
 
 Updated package description and added project keywords in package.json. 
 Extended readme with detailed usage examples and class structures. 
 Added getBusinessRecordByName function for dynamic business record retrieval. 
 
 2024-07-02 - 1.0.3 - fix(core) 
 No new changes detected. Preparing for patch release. 
 2023-11-14 - 1.0.1 to 1.0.2 - General Updates 
 Minor version updates and fixes. 
 
 fix(core): update