changelog.md for @fin.cx/opendata

2025-11-07 - 3.5.0 - feat(stocks)

Add provider fetch limits, intraday incremental fetch, cache deduplication, and provider safety/warning improvements

• Add maxRecords and defaultIntradayLimit to IProviderConfig to control maximum records per request and default intraday limits.
• CoinGecko provider: enforce maxRecords when processing historical data, warn when large historical/intraday results are returned without explicit limits, preserve priority mappings when rebuilding the coin cache, and improve cache load logging.
• Marketstack provider: make safety maxRecords configurable, apply a configurable default intraday limit, warn when no explicit limit is provided, and ensure effective limits are applied to returned results.
• StockPriceService: always attempt incremental fetch for intraday requests without a date to fetch only new data since the last cached timestamp and fall back to full fetch when necessary.
• StockPriceService: deduplicate price arrays by timestamp before caching and after merges to avoid duplicate timestamps and reduce cache bloat.
• Introduce StockDataService for unified access to prices and fundamentals with automatic enrichment (market cap, P/E, price-to-book) and caching improvements.
• Various cache/TTL improvements and safer default behaviors for intraday, historical and live data to improve performance and memory usage.

2025-11-06 - 3.4.0 - feat(stocks)

Introduce unified stock data service, new providers, improved caching and German business data tooling

• Add StockDataService: unified API to fetch price + fundamentals with automatic enrichment and batch support
• Introduce BaseProviderService abstraction and refactor provider management, caching and retry logic
• Enhance StockPriceService: unified getData, discriminated union request types, data-type aware TTLs and smarter cache keys
• Add Marketstack provider with intraday/EOD selection, pagination, OHLCV and exchange filtering
• Add CoinGecko provider with robust rate-limiting, coin ID resolution and crypto support (current, historical, intraday)
• Add SEC EDGAR fundamentals provider: CIK lookup, company facts parsing, rate limiting and caching
• Improve FundamentalsService: unified fetching, caching and enrichment helpers (enrichWithPrice, enrichBatchWithPrices)
• Enhance Yahoo provider and other provider mappings for better company metadata and market state handling
• Add German business data tooling: JsonlDataProcessor for JSONL bulk imports, HandelsRegister browser automation with download handling and parsing
• Expose OpenData entry points: DB init, JSONL processing and Handelsregister integration; add readme/docs and usage examples

2025-11-02 - 3.3.0 - feat(stocks/CoinGeckoProvider)

Add CoinGecko provider for cryptocurrency prices, export and tests, and update documentation

• Implemented CoinGeckoProvider with rate limiting, coin-id resolution, and support for current, batch, historical and intraday price endpoints
• Added unit/integration tests for CoinGecko: test/test.coingecko.node+bun+deno.ts
• Exported CoinGeckoProvider from ts/stocks/index.ts
• Updated README and readme.hints.md with CoinGecko usage, provider notes and examples
• Added .claude/settings.local.json with webfetch and bash permissions required for testing and CI

2025-11-01 - 3.2.2 - fix(handelsregister)

Correct screenshot path handling in HandelsRegister and add local tool permissions

• ts/classes.handelsregister.ts: Replace string concatenation for screenshot path with a template literal and explicit string assertion to ensure the path is formed correctly for page.screenshot() and avoid type issues.
• Add .claude/settings.local.json: Introduce local Claude settings that grant specific tool permissions used during development and testing (bash commands, web fetches, pnpm build, tstest, etc.).

2025-11-01 - 3.2.1 - fix(stocks/providers/provider.secedgar)

Improve SEC EDGAR provider networking and error handling, update plugin path import, bump dev deps and add/refresh tests and lockfile

• SEC EDGAR provider: switch from SmartRequest to native fetch for ticker list and company facts, add AbortController-based timeouts, handle gzip automatically, improve response validation and error messages, and keep CIK/ticker-list caching
• Improve timeout and rate-limit handling in SecEdgarProvider (uses native fetch + explicit timeout clear), plus clearer logging on failures
• Update ts/plugins import to use node:path for Node compatibility
• Bump devDependencies: @git.zone/tsrun to ^1.6.2 and @git.zone/tstest to ^2.7.0; bump @push.rocks/smartrequest to ^4.3.4
• Add and refresh comprehensive test files (node/bun/deno variants) for fundamentals, marketstack, secedgar and stockdata services
• Add deno.lock (dependency lock) and a local .claude/settings.local.json for CI/permissions

2025-11-01 - 3.2.0 - feat(StockDataService)

Add unified StockDataService and BaseProviderService with new stockdata interfaces, provider integrations, tests and README updates

• Introduce StockDataService: unified API to fetch prices and fundamentals with automatic enrichment and caching
• Add IStockData and IStockDataServiceConfig interfaces to define combined price+fundamentals payloads and configuration
• Implement BaseProviderService abstraction to share provider registration, health, stats and caching logic
• Add classes.stockdataservice.ts implementing batch/single fetch, enrichment, caching, health checks and provider stats
• Export new stockdata module and classes from ts/stocks/index.ts
• Add comprehensive tests: test/test.stockdata.service.node.ts to cover setup, provider registration, fetching, caching, enrichment, health and error handling
• Update README with Unified Stock Data API examples, usage, and documentation reflecting new unified service
• Minor infra: add .claude/settings.local.json permissions for local tooling and web fetch domains

2025-11-01 - 3.1.0 - feat(fundamentals)

Add FundamentalsService and SEC EDGAR provider with caching, rate-limiting, tests, and docs updates

• Introduce FundamentalsService to manage fundamentals providers, caching, retry logic and provider statistics
• Add SecEdgarProvider to fetch SEC EDGAR company facts (CIK lookup, company facts parsing) with rate limiting and local caches
• Expose fundamentals interfaces and services from ts/stocks (exports updated)
• Add comprehensive tests for FundamentalsService and SecEdgarProvider (new test files)
• Update README with new Fundamentals module documentation, usage examples, and configuration guidance
• Implement caching and TTL handling for fundamentals data and provider-specific cache TTL support
• Add .claude/settings.local.json (local permissions) and various test improvements

2025-10-31 - 3.0.0 - BREAKING CHANGE(stocks)

Unify stock provider API to discriminated IStockDataRequest and add company name/fullname enrichment

• Replace legacy provider methods (fetchPrice/fetchPrices) with a single fetchData(request: IStockDataRequest) on IStockProvider — providers must be migrated to the new signature.
• Migrate StockPriceService to the unified getData(request: IStockDataRequest) API. Convenience helpers getPrice/getPrices now wrap getData.
• Add companyName and companyFullName fields to IStockPrice and populate them in provider mappings (Marketstack mapping updated; Yahoo provider updated to support the unified API).
• MarketstackProvider: added buildCompanyFullName helper and improved mapping to include company identification fields and full name formatting.
• YahooFinanceProvider: updated to implement fetchData and to route current/batch requests through the new unified request types; historical/intraday throw explicit errors.
• Updated tests to exercise the new unified API, company-name enrichment, caching behavior, and provider direct methods.
• Note: This is a breaking change for external providers and integrations that implemented the old fetchPrice/fetchPrices API. Bump major version.

2025-10-31 - 2.1.0 - feat(stocks)

Add unified stock data API (getData) with historical/OHLCV support, smart caching and provider enhancements

• Introduce discriminated union request types (IStockDataRequest) and a unified getData() method (replaces legacy getPrice/getPrices for new use cases)
• Add OHLCV fields (open, high, low, volume, adjusted) and metadata (dataType, fetchedAt) to IStockPrice
• Implement data-type aware smart caching with TTLs (historical = never expire, EOD = 24h, live = 30s, intraday matches interval)
• Extend StockPriceService: new getData(), data-specific cache keys, cache maxEntries increased (default 10000), and TTL-aware add/get cache logic
• Enhance Marketstack provider: unified fetchData(), historical date-range retrieval with pagination, exchange filtering, batch current fetch, OHLCV mapping, and intraday placeholder
• Update Yahoo provider to include dataType and fetchedAt (live data) and maintain legacy fetchPrice/fetchPrices compatibility
• Add/adjust tests to cover unified API, historical retrieval, OHLCV presence and smart caching behavior; test setup updated to require explicit OpenData directory paths
• Update README to document v2.1 changes, migration examples, and new stock provider capabilities

2025-10-31 - 2.0.0 - BREAKING CHANGE(OpenData)

Require explicit directory paths for OpenData (nogit/download/germanBusinessData); remove automatic .nogit creation; update HandelsRegister, JsonlDataProcessor, tests and README.

• Breaking: OpenData constructor now requires a config object with nogitDir, downloadDir and germanBusinessDataDir. The constructor will throw if these paths are not provided.
• Removed automatic creation/export of .nogit/download/germanBusinessData from ts/paths. OpenData.start now ensures the required directories exist.
• HandelsRegister API changed: constructor now accepts downloadDir and manages its own unique download folder; screenshot and download paths now use the configured downloadDir.
• JsonlDataProcessor now accepts a germanBusinessDataDir parameter and uses it when ensuring/storing data instead of relying on global paths.
• Updated tests to provide explicit path configuration (tests now set testNogitDir, testDownloadDir, testGermanBusinessDataDir and write outputs accordingly) and to use updated constructors and qenv usage.
• Documentation updated (README) to document the breaking change and show examples for required directory configuration when instantiating OpenData.
• Added .claude/settings.local.json for local permissions/config used in development/CI environments.

2025-10-11 - 1.7.0 - feat(stocks)

Add Marketstack provider (EOD) with tests, exports and documentation updates

• Add MarketstackProvider implementation (ts/stocks/providers/provider.marketstack.ts) providing EOD single and batch fetching, availability checks and mapping to IStockPrice.
• Export MarketstackProvider from ts/stocks/index.ts so it is available via the public API.
• Add comprehensive Marketstack tests (test/test.marketstack.node.ts) covering registration, health checks, single/batch fetches, caching, ticker/market validation, provider stats and sample output.
• Update README with Marketstack usage examples, configuration, API key instructions and provider/health documentation.
• Bump dev dependency @git.zone/tstest to ^2.4.2 in package.json.
• Add project helper/config files (.claude/settings.local.json, .serena/project.yml and .serena/.gitignore) to support CI/tooling.

2025-09-24 - 1.6.1 - fix(stocks)

Fix Yahoo provider request handling and bump dependency versions

• Refactored Yahoo Finance provider to use SmartRequest.create() builder and await response.json() for HTTP responses (replaces direct getJson usage).
• Improved batch and single-price fetching to use the SmartRequest API, keeping User-Agent header and timeouts.
• Added a compile-time type-check alias to ensure IStockPrice matches tsclass.finance.IStockPrice.
• Bumped development and runtime dependency versions (notable bumps include @git.zone/tsbuild, @git.zone/tstest, @push.rocks/qenv, @push.rocks/smartarchive, @push.rocks/smartdata, @push.rocks/smartfile, @push.rocks/smartlog, @push.rocks/smartpath, @push.rocks/smartrequest, @tsclass/tsclass).
• Added .claude/settings.local.json to grant local CI permissions for a few Bash commands.

2025-07-12 - 1.6.0 - feat(readme)

Revamp documentation and package description for enhanced clarity

• Restructured README to highlight real-time stock data and German business data, streamlining quick start and advanced examples
• Updated package.json description to better reflect library capabilities
• Added .claude/settings.local.json to define permissions for external tools
• Refined code examples in tests and documentation for improved clarity and consistency

2025-04-09 - 1.5.3 - fix(test)

Await file writes in Handelsregister tests to ensure all downloads complete before test end

• Replaced array.map with await Promise.all to properly await asynchronous file writes in test/test.handelsregister.ts
• Improved robustness of asynchronous operations in test suite

2025-04-09 - 1.5.2 - fix(readme)

Improve .env configuration code block formatting in documentation

• Wrap the .env variables block in triple backticks for clarity
• Ensure consistency in the Markdown styling of code snippets

2025-04-09 - 1.5.1 - fix(core)

No changes detected in project files or documentation. This commit is a placeholder to record that nothing was updated.

2025-04-09 - 1.5.0 - feat(documentation)

Enhance project metadata and documentation with comprehensive usage examples, updated descriptions, and improved keywords.

• Updated npmextra.json and package.json to refine the project description and keyword list.
• Expanded readme.md with detailed sections on environment setup, CRUD operations, bulk JSONL processing, and advanced Handelsregister integrations.
• Included advanced workflow examples and error handling strategies in the documentation.
• Adjusted test cases (e.g. in test/test.handelsregister.ts) to reflect changes in company name usage.

2025-04-08 - 1.4.6 - fix(tests & jsonl)

Improve test structure and refine JSONL parsing for incomplete data

• Refactored test files to remove redundant get-specific-company tests in test.ts and added missing tests in test.handelsregister.ts
• Updated JSONL data processor to conditionally parse remaining data when available

2025-04-05 - 1.4.5 - fix(metadata)

Update repository, bugs, and homepage URLs to code.foss.global

• Repository URL updated from gitlab.com to code.foss.global
• Bugs URL updated from gitlab.com to code.foss.global
• Homepage URL updated to code.foss.global

2025-04-05 - 1.4.4 - fix(dependencies & tests)

Update dependency versions and refine test search query

• Bumped versions for several dependencies in package.json, including @git.zone/tsbuild, @git.zone/tsbundle, @git.zone/tstest, @push.rocks/tapbundle, @push.rocks/smartdata, @push.rocks/smartfile, @push.rocks/smartpromise, @push.rocks/smartrequest, and @tsclass/tsclass
• Updated test file to replace the search query 'Volkswagen' with 'LADR'
• Re-enabled the build initial data test by removing tap.skip

2025-01-07 - 1.4.3 - fix(test)

Corrected index value in test for fetching specific company data

• Updated the index from 8 to 7 for the germanParsedRegistration fetch in test

2025-01-07 - 1.4.2 - fix(core)

Fix concurrency and download handling in HandelsRegister class and adjust test cases

• Improved the clickFindButton function to include an argument for results limit.
• Enhanced the downloadFile function to rename and ensure files are correctly handled.
• Updated searchCompany method to allow specifying a limit on the number of search results.
• Adjusted test cases to select specific test data indices and output test files to a dedicated directory.

2025-01-04 - 1.4.1 - fix(core)

Fix issues with JSONL data processing and improve error handling in business record validation

• Fixed JSONL data processing by adding concurrent processing for each JSON line to enhance performance.
• Added validation logic in BusinessRecord class to ensure that the mandatory fields are checked.
• Adjusted environment variable loading in OpenData class to ensure correct database initialization.
• Included missing dependencies and exports in the project files to ensure proper functionality.

2025-01-04 - 1.4.0 - feat(HandelsRegister)

Add file download functionality to HandelsRegister

• Implemented file download feature in the HandelsRegister class.
• Configured pages in Puppeteer to allow downloads and set download paths.
• Parsed German registration information with more robust error handling.
• Added specific methods for downloading and handling 'SI' and 'AD' files.

2025-01-03 - 1.3.1 - fix(HandelsRegister)

Refined HandelsRegister functionality for better error handling and response capture.

• Improved parsing logic in parseGermanRegistration function.
• Enhanced navigateToPage and clickFindButton methods with error messages for clarity.
• Implemented a new responseListener to handle and log HTTP responses correctly.

2025-01-03 - 1.3.0 - feat(core)

Enhanced data handling capabilities and improved company search functionalities.

• Updated business record handling to support more registration types.
• Improved search capabilities for fetching company data with refined registration type matching.
• Added robust logging for JSONL data processing with early exit on successful parse.
• Reorganized test cases to include specific company data retrieval.

2025-01-02 - 1.2.1 - fix(BusinessRecord)

Add missing field registrationType to BusinessRecord data

• Introduced the 'registrationType' field to the BusinessRecord data schema with possible values 'HRA' or 'HRB'.

2025-01-02 - 1.2.0 - feat(core)

Integrate Handelsregister search for company data retrieval

• Added support for searching company data via Handelsregister.
• Replaced GermanBusinessData functionality with JsonlDataProcessor.
• Included smartbrowser dependency for handling web requests to Handelsregister.

2025-01-01 - 1.1.5 - fix(GermanBusinessData)

Add console log for total records processed at the end of the stream.

• Ensure that the number of records processed is logged at the end of data stream processing.

2024-12-31 - 1.1.4 - fix(documentation)

Update description and keywords in package.json

• Corrected the package description to reflect the focus on managing, accessing, and updating open data with MongoDB integration.
• Expanded the keywords in the package metadata to include data integration and processing terms.
• Improved README.md with more extensive setup, usage, and introduction of the library's functionalities.

2024-12-31 - 1.1.3 - fix(core)

Added missing license file for project completeness.

• Introduced a LICENSE file to the project, ensuring clarity on software usage permissions.

2024-12-31 - 1.1.2 - fix(GermanBusinessData)

Ensure unique ID generation for BusinessRecord

• Added generation of a new ID for each BusinessRecord in GermanBusinessData.
• Ensures each business record has a unique identifier.

2024-12-31 - 1.1.1 - fix(dependencies)

Update dependencies and devDependencies to newer versions.

• @git.zone/tsbuild from ^2.1.25 to ^2.2.0
• @git.zone/tsbundle from ^2.0.5 to ^2.1.0
• @git.zone/tsrun from ^1.2.46 to ^1.3.3
• @git.zone/tstest from ^1.0.84 to ^1.0.90
• @push.rocks/tapbundle from ^5.0.15 to ^5.5.4
• @types/node from ^20.9.0 to ^22.10.2
• @push.rocks/qenv from ^6.0.4 to ^6.1.0
• @push.rocks/smartarchive from ^4.0.19 to ^4.0.39
• @push.rocks/smartdata from ^5.0.33 to ^5.2.10
• @push.rocks/smartfile from ^11.0.0 to ^11.0.23
• @push.rocks/smartpath from ^5.0.11 to ^5.0.18
• @push.rocks/smartpromise from ^4.0.3 to ^4.0.4
• @push.rocks/smartrequest from ^2.0.21 to ^2.0.23
• @push.rocks/smartstream from ^3.0.30 to ^3.2.5

2024-12-31 - 1.1.0 - feat(core)

Enhanced data handling and retrieval features, improved usage documentation

• Updated package description and added project keywords in package.json.
• Extended readme with detailed usage examples and class structures.
• Added getBusinessRecordByName function for dynamic business record retrieval.

2024-07-02 - 1.0.3 - fix(core)

No new changes detected. Preparing for patch release.

2023-11-14 - 1.0.1 to 1.0.2 - General Updates

Minor version updates and fixes.

• fix(core): update