@push.rocks/smartdata

An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.

readme.md for @push.rocks/smartdata

npm version

The ultimate TypeScript-first MongoDB ODM β€” type-safe decorators, real-time change streams, Lucene-powered search, distributed leader election, and cursor streaming. Built for modern applications that demand performance, correctness, and developer experience.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

🌟 Why SmartData?

πŸ“¦ Installation

pnpm add @push.rocks/smartdata

🚦 Requirements

Note: SmartData uses TC39 Stage 3 decorators (the standard). Make sure experimentalDecorators is not set in your tsconfig.json. Bun is not currently supported as it doesn't implement TC39 decorators yet.

🎯 Quick Start

1️⃣ Connect to Your Database

import { SmartdataDb } from '@push.rocks/smartdata';

const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://localhost:27017/myapp',
  mongoDbName: 'myapp',
  mongoDbUser: 'username',
  mongoDbPass: 'password',
});

await db.init();
console.log(db.status); // 'connected'

2️⃣ Define Your Data Models

import {
  SmartDataDbDoc,
  Collection,
  unI,
  svDb,
  index,
  searchable,
} from '@push.rocks/smartdata';

@Collection(() => db)
class User extends SmartDataDbDoc<User, User> {
  @unI()
  public id!: string;

  @svDb()
  @searchable()
  public username!: string;

  @svDb()
  @searchable()
  @index({ unique: false })
  public email!: string;

  @svDb()
  public status!: 'active' | 'inactive' | 'pending';

  @svDb()
  public tags!: string[];

  @svDb()
  public createdAt: Date = new Date();

  constructor(username: string, email: string) {
    super();
    this.username = username;
    this.email = email;
  }
}

3️⃣ CRUD Operations

// ✨ Create
const user = new User('johndoe', 'john@example.com');
user.status = 'active';
user.tags = ['developer', 'typescript'];
await user.save();

// πŸ” Read β€” fully type-safe filters
const foundUser = await User.getInstance({ username: 'johndoe' });
const activeUsers = await User.getInstances({ status: 'active' });

// ✏️ Update
foundUser.email = 'newemail@example.com';
await foundUser.save();

// πŸ—‘οΈ Delete
await foundUser.delete();

πŸ”₯ Features

🎯 Type-Safe Query Filters

SmartData provides a rich, type-safe filtering system supporting all MongoDB operators with full IntelliSense:

// Comparison operators
const adults = await User.getInstances({
  age: { $gte: 18, $lt: 65 },
});

// Array operators
const experts = await User.getInstances({
  tags: { $all: ['typescript', 'mongodb'] },
  skills: { $size: 5 },
});

// Logical operators
const complex = await Order.getInstances({
  $and: [
    { status: 'active' },
    { $or: [{ priority: 'high' }, { value: { $gte: 1000 } }] },
  ],
});

// Deep nested object queries
const users = await User.getInstances({
  profile: {
    settings: {
      notifications: { email: true },
    },
  },
});

// Dot notation
const sameUsers = await User.getInstances({
  'profile.settings.notifications.email': true,
});

// Regex patterns
const gmailUsers = await User.getInstances({
  email: { $regex: '@gmail\\.com$', $options: 'i' },
});

Security: The $where operator is automatically blocked to prevent NoSQL injection. Unknown operators trigger warnings.

πŸ”Ž Lucene-Powered Search

Mark fields with @searchable() to enable a built-in search engine with automatic compound text indexing:

@Collection(() => db)
class Product extends SmartDataDbDoc<Product, Product> {
  @unI() public id!: string;
  @svDb() @searchable() public name!: string;
  @svDb() @searchable() public description!: string;
  @svDb() @searchable() public category!: string;
  @svDb() public price!: number;
}

// Simple text search across all @searchable fields
const results = await Product.search('laptop');

// Field-specific search
const electronics = await Product.search('category:Electronics');

// Wildcard
const matches = await Product.search('Mac*');

// Boolean operators (AND, OR, NOT)
const query = await Product.search('laptop AND NOT gaming');

// Phrase search
const exact = await Product.search('"MacBook Pro"');

// Range queries
const midRange = await Product.search('price:[100 TO 500]');

// Combined with MongoDB filters and post-fetch validation
const affordable = await Product.search('laptop', {
  filter: { price: { $lte: 1500 } },
  validate: async (p) => p.price > 0,
});

πŸ“‘ Real-Time Change Streams

Watch for database changes with RxJS subjects and EventEmitter support:

const watcher = await User.watch(
  { status: 'active' },
  {
    fullDocument: 'updateLookup',
    bufferTimeMs: 100, // optional: buffer changes via RxJS
  },
);

// RxJS subscription
watcher.changeSubject.subscribe((user) => {
  console.log('User changed:', user);
});

// Or EventEmitter style
watcher.on('change', (user) => {
  console.log('User changed:', user);
});

// Clean up
await watcher.close();

πŸ”„ Cursor Streaming

Process large datasets without memory pressure:

const cursor = await User.getCursor(
  { status: 'active' },
  {
    modifier: (c) => c.sort({ createdAt: -1 }).limit(10000),
  },
);

// Iterate one-by-one
await cursor.forEach(async (user) => {
  await processUser(user);
});

// Or collect into an array
const users = await cursor.toArray();

// Always close when done
await cursor.close();

πŸ” Transactions

Ensure atomic consistency across multiple operations:

const session = db.startSession();

try {
  await session.withTransaction(async () => {
    const sender = await User.getInstance({ id: 'user-1' }, { session });
    sender.balance -= 100;
    await sender.save({ session });

    const receiver = await User.getInstance({ id: 'user-2' }, { session });
    receiver.balance += 100;
    await receiver.save({ session });
  });
} finally {
  await session.endSession();
}

πŸ’Ύ EasyStore β€” Type-Safe Key-Value Storage

Built on top of SmartData collections, EasyStore provides simple key-value persistence:

interface AppConfig {
  apiKey: string;
  features: { darkMode: boolean; notifications: boolean };
  limits: { maxUsers: number };
}

const config = await db.createEasyStore<AppConfig>('app-config');

// Write
await config.writeKey('features', { darkMode: true, notifications: false });

// Read
const features = await config.readKey('features');
// TypeScript knows: features.darkMode is boolean βœ…

// Read all
const all = await config.readAll();

// Write multiple keys
await config.writeAll({ apiKey: 'new-key', limits: { maxUsers: 500 } });

// Delete a key
await config.deleteKey('features');

// Wipe the store
await config.wipe();

🌐 Distributed Coordination

Built-in leader election using MongoDB for coordination, integrating with @push.rocks/taskbuffer:

import { SmartdataDistributedCoordinator } from '@push.rocks/smartdata';

const coordinator = new SmartdataDistributedCoordinator(db);

// Start coordination β€” automatic heartbeat and leader election
await coordinator.start();

// Fire distributed task requests
const result = await coordinator.fireDistributedTaskRequest({
  submitterId: 'instance-1',
  requestResponseId: 'unique-id',
  taskName: 'process-payments',
  taskVersion: '1.0.0',
  taskExecutionTime: Date.now(),
  taskExecutionTimeout: 30000,
  taskExecutionParallel: 1,
  status: 'requesting',
});

// Graceful shutdown with leadership handoff
await coordinator.stop();

🎨 Custom Serialization

Transform data on its way in and out of MongoDB:

@Collection(() => db)
class Doc extends SmartDataDbDoc<Doc, Doc> {
  @svDb({
    serialize: (set) => Array.from(set),
    deserialize: (arr) => new Set(arr),
  })
  public tags!: Set<string>;

  @svDb({
    serialize: (date) => date?.toISOString(),
    deserialize: (str) => (str ? new Date(str) : null),
  })
  public scheduledAt!: Date | null;
}

🎣 Lifecycle Hooks

Add custom logic before and after save/delete:

@Collection(() => db)
class Order extends SmartDataDbDoc<Order, Order> {
  @unI() public id!: string;
  @svDb() public items!: Array<{ product: string; quantity: number; price: number }>;
  @svDb() public totalAmount!: number;

  async beforeSave() {
    this.totalAmount = this.items.reduce((s, i) => s + i.price * i.quantity, 0);
  }

  async afterSave() {
    await notificationService.orderUpdated(this.id);
  }

  async beforeDelete() {
    if (this.totalAmount > 0) throw new Error('Cannot delete non-zero orders');
  }

  async afterDelete() {
    await cache.delete(`order:${this.id}`);
  }
}

πŸ—οΈ Indexing

@Collection(() => db)
class HighPerformanceDoc extends SmartDataDbDoc<HighPerformanceDoc, HighPerformanceDoc> {
  @unI()
  public id!: string; // Unique index

  @index()
  public userId!: string; // Regular index

  @index({ sparse: true })
  public deletedAt?: Date; // Sparse index β€” only indexes docs where field exists

  @index({ expireAfterSeconds: 86400 })
  public sessionToken!: string; // TTL index β€” auto-expires after 24h
}

πŸ”§ Connection Options

const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://localhost:27017',
  mongoDbName: 'myapp',
  mongoDbUser: 'admin',
  mongoDbPass: 's3cret',

  // Connection pool tuning (all optional)
  maxPoolSize: 100,           // Max connections (default: 100)
  maxIdleTimeMS: 300000,      // Close idle connections after 5min (default)
  serverSelectionTimeoutMS: 30000, // Timeout for server selection
  socketTimeoutMS: 30000,     // Socket timeout to prevent hung operations
});

πŸ“š Decorators Reference

Decorator Target Description
@Collection(dbGetter) Class Binds a document class to a MongoDB collection
@managed(managerGetter?) Class Like @Collection but controlled by a manager instance
@unI() Field Marks as unique index + saveable
@svDb(options?) Field Marks field as saveable, with optional serialize/deserialize
@index(options?) Field Creates a regular MongoDB index
@searchable() Field Enables Lucene-style text search on this field
@globalSvDb() Field Marks field as globally saveable across all doc types

πŸ“š API Reference

Core Classes

Class Description
SmartdataDb Database connection, session management, EasyStore factory
SmartDataDbDoc<T, TImpl> Base class for all document models
SmartdataCollection<T> Underlying collection manager (usually accessed indirectly)
SmartdataDbCursor<T> Cursor for streaming large result sets
SmartdataDbWatcher<T> Change stream watcher with RxJS + EventEmitter
SmartdataDistributedCoordinator Leader election and distributed task coordination
EasyStore<T> Type-safe key-value store backed by a collection

Key Static Methods on SmartDataDbDoc

Method Description
getInstances(filter, opts?) Find multiple documents
getInstance(filter, opts?) Find a single document (or null)
getCursor(filter, opts?) Get a streaming cursor
getCount(filter?) Count matching documents
watch(filter, opts?) Watch for real-time changes
search(query, opts?) Lucene-style full-text search
forEach(filter, fn) Iterate all matches with a callback
getNewId(length?) Generate a class-prefixed unique ID
createSearchFilter(luceneQuery) Convert Lucene query to MongoDB filter
getSearchableFields() List all @searchable() fields

Key Instance Methods on SmartDataDbDoc

Method Description
save(opts?) Insert or update the document
delete(opts?) Delete the document
updateFromDb() Refresh fields from the database
saveDeep(savedMap?) Recursively save referenced documents
createSavableObject() Serialize to a plain object for persistence
createIdentifiableObject() Extract unique index fields for filtering

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the LICENSE file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.

changelog.md for @push.rocks/smartdata

2026-03-26 - 7.1.3 - fix(deps)

bump development dependencies for tooling and Node types

2026-03-24 - 7.1.2 - fix(docs)

refresh project guidance for TC39 decorators, build configuration, and dependency compatibility

2026-03-24 - 7.1.1 - fix(build)

update build and test tooling configuration, migrate project config to .smartconfig.json, and align TypeScript typings

2026-02-26 - 7.1.0 - feat(config)

normalize npmextra.json to namespaced keys and add CI/release configuration

2026-02-26 - 7.0.16 - fix(mongodb)

set default socketTimeoutMS to 30000ms in MongoClient options to prevent hung operations from holding connections

2025-12-01 - 7.0.15 - fix(classes.doc)

Avoid emitting instance fields for collection and manager to preserve decorator-defined prototype getters

2025-11-28 - 7.0.14 - fix(classes.collection)

Centralize TC39 decorator metadata initialization and use context.metadata in class decorators

2025-11-28 - 7.0.13 - fix(classes.doc)

Remove noisy debug logging from decorators and serialization logic

2025-11-28 - 7.0.12 - fix(collection)

Ensure TC39 decorator metadata is initialized on both original and decorated constructors/prototypes and add debug logging

2025-11-28 - 7.0.9 - fix(classes.collection)

Fix closure bug in Collection decorator by defining collection getter on original constructor and prototype

2025-11-28 - 7.0.8 - fix(classes.collection)

Fix closure issue in managed decorator so Class.collection/instance.collection resolve correctly

2025-11-28 - 7.0.7 - fix(decorators)

Fix decorator metadata initialization and Lucene query transformation

2025-11-28 - 7.0.6 - fix(classes.collection)

Guard against missing collection before attaching document constructor in Collection decorator

2025-11-28 - 7.0.5 - fix(package)

Add package exports entry and remove legacy main/typings fields

2025-11-28 - 7.0.4 - fix(decorators)

Add Symbol.metadata polyfill and import it at entry to ensure decorator metadata is available

2025-11-28 - 7.0.3 - fix(build)

Bump devDependency @git.zone/tsbuild to ^3.1.2

2025-11-28 - 7.0.2 - fix(collectionfactory)

Simplify CollectionFactory.getCollection: remove unnecessary IIFE and instantiate collection only when dbArg is SmartdataDb

2025-11-27 - 7.0.1 - fix(build)

Update build tooling and TypeScript compilation target

2025-11-27 - 7.0.0 - BREAKING CHANGE(mongodb)

Upgrade dependencies: bump mongodb to ^7.0.0 and @git.zone/tstest to ^3.1.3

2025-11-17 - 6.0.0 - BREAKING CHANGE(decorators)

Migrate to TC39 Stage 3 decorators and refactor decorator metadata handling; update class initialization, lucene adapter fixes and docs

2025-11-17 - 5.16.7 - fix(classes.collection)

Improve Deno and TypeScript compatibility: Collection decorator _svDbOptions forwarding and config cleanup

2025-11-17 - 5.16.6 - fix(classes)

Add Deno compatibility, prototype-safe decorators and safe collection accessor; bump a few deps

2025-11-16 - 5.16.5 - fix(watcher)

Update dependencies, tooling and watcher import; add .serena cache ignore

2025-08-18 - 5.16.4 - fix(classes.doc (convertFilterForMongoDb))

Improve filter conversion: handle logical operators, merge operator objects, add nested filter tests and docs, and fix test script

2025-08-18 - 5.16.3 - fix(docs)

Add local Claude settings and remove outdated codex.md

2025-08-18 - 5.16.2 - fix(readme)

Update README: clarify examples, expand search/cursor/docs and add local Claude settings

2025-08-12 - 5.16.1 - fix(core)

Improve error handling and logging; enhance search query sanitization; update dependency versions and documentation

2025-04-25 - 5.16.0 - feat(watcher)

Enhance change stream watchers with buffering and EventEmitter support; update dependency versions

2025-04-24 - 5.15.1 - fix(cursor)

Improve cursor usage documentation and refactor getCursor API to support native cursor modifiers

2025-04-24 - 5.15.0 - feat(svDb)

Enhance svDb decorator to support custom serialization and deserialization options

2025-04-23 - 5.14.1 - fix(db operations)

Update transaction API to consistently pass optional session parameters across database operations

2025-04-23 - 5.14.0 - feat(doc)

Implement support for beforeSave, afterSave, beforeDelete, and afterDelete lifecycle hooks in document save and delete operations to allow custom logic execution during these critical moments.

2025-04-22 - 5.13.1 - fix(search)

Improve search query parsing for implicit AND queries by preserving quoted substrings and better handling free terms, quoted phrases, and field:value tokens.

2025-04-22 - 5.13.0 - feat(search)

Improve search query handling and update documentation

2025-04-22 - 5.12.2 - fix(search)

Fix handling of quoted wildcard patterns in field-specific search queries and add tests for location-based wildcard phrase searches

2025-04-22 - 5.12.1 - fix(search)

Improve implicit AND logic for mixed free term and field queries in search and enhance wildcard field handling.

2025-04-22 - 5.12.0 - feat(doc/search)

Enhance search functionality with filter and validate options for advanced query control

2025-04-22 - 5.11.4 - fix(search)

Implement implicit AND logic for mixed simple term and field:value queries in search

2025-04-22 - 5.11.3 - fix(lucene adapter and search tests)

Improve range query parsing in Lucene adapter and expand search test coverage

2025-04-21 - 5.11.2 - fix(readme)

Update readme to clarify usage of searchable fields retrieval

2025-04-21 - 5.11.1 - fix(doc)

Refactor searchable fields API and improve collection registration.

2025-04-21 - 5.11.0 - feat(ts/classes.lucene.adapter)

Expose luceneWildcardToRegex method to allow external usage and enhance regex transformation capabilities.

2025-04-21 - 5.10.0 - feat(search)

Improve search functionality: update documentation, refine Lucene query transformation, and add advanced search tests

2025-04-18 - 5.9.2 - fix(documentation)

Update search API documentation to replace deprecated searchWithLucene examples with the unified search(query) API and clarify its behavior.

2025-04-18 - 5.9.1 - fix(search)

Refactor search tests to use unified search API and update text index type casting

2025-04-18 - 5.9.0 - feat(collections/search)

Improve text index creation and search fallback mechanisms in collections and document search methods

2025-04-17 - 5.8.4 - fix(core)

Update commit metadata with no functional code changes

2025-04-17 - 5.8.3 - fix(readme)

Improve readme documentation on data models and connection management

2025-04-14 - 5.8.2 - fix(classes.doc.ts)

Ensure collection initialization before creating a cursor in getCursorExtended

2025-04-14 - 5.8.1 - fix(cursor, doc)

Add explicit return types and casts to SmartdataDbCursor methods and update getCursorExtended signature in SmartDataDbDoc.

2025-04-14 - 5.8.0 - feat(cursor)

Add toArray method to SmartdataDbCursor to convert raw MongoDB documents into initialized class instances

2025-04-14 - 5.7.0 - feat(SmartDataDbDoc)

Add extended cursor method getCursorExtended for flexible cursor modifications

2025-04-07 - 5.6.0 - feat(indexing)

Add support for regular index creation in documents and collections

2025-04-06 - 5.5.1 - fix(ci & formatting)

Minor fixes: update CI workflow image and npmci package references, adjust package.json and readme URLs, and apply consistent code formatting.

2025-04-06 - 5.5.0 - feat(search)

Enhance search functionality with robust Lucene query transformation and reliable fallback mechanisms

2025-04-06 - 5.4.0 - feat(core)

Refactor file structure and update dependency versions

2025-03-10 - 5.3.0 - feat(docs)

Enhance documentation with updated installation instructions and comprehensive usage examples covering advanced features such as deep queries, automatic indexing, and distributed coordination.

2025-02-03 - 5.2.12 - fix(documentation)

Remove license badge from README

2025-02-03 - 5.2.11 - fix(documentation)

Updated project documentation for accuracy and added advanced feature details

2024-09-05 - 5.2.10 - fix(smartdata.classes.doc)

Fix issue with array handling in convertFilterForMongoDb function

2024-09-05 - 5.2.9 - fix(smartdata.classes.doc)

Fixed issue with convertFilterForMongoDb to handle array operators.

2024-09-05 - 5.2.8 - fix(smartdata.classes.doc)

Fix key handling in convertFilterForMongoDb function

2024-09-05 - 5.2.7 - fix(core)

Fixed issue with handling filter keys containing dots in smartdata.classes.doc.ts

2024-06-18 - 5.2.6 - Chore

Maintenance Release

2024-05-31 - 5.2.2 - Bug Fixes

Fixes and Maintenance

2024-04-15 - 5.1.2 - New Feature

Enhancements and Bug Fixes

2024-04-14 - 5.0.43 - New Feature

New Feature Addition

2024-03-30 - 5.0.41 - Bug Fixes

Improvements and Fixes

2023-07-10 - 5.0.20 - Chore

Organizational Changes

2023-07-21 - 5.0.21 to 5.0.26 - Fixes

Multiple Fix Releases

2023-07-21 - 5.0.20 - Chore

Organizational Changes

2023-06-25 - 5.0.14 to 5.0.19 - Fixes

Multiple Fix Releases

2022-05-17 - 5.0.0 - Major Update

Breaking Changes

2022-05-18 - 5.0.2 - Bug Fixes

Bug Fixes

2022-05-17 - 5.0.1 - Chore

Testing Improvements

2022-05-17 to 2022-11-08 - 5.0.8 to 5.0.10

Multiple Fix Releases

2021-11-12 - 4.0.17 to 4.0.20

Multiple Fix Releases

2021-09-17 - 4.0.10 to 4.0.16

Multiple Fix Releases

2021-06-09 - 4.0.1 to 4.0.9

Multiple Fix Releases

2021-06-06 - 4.0.0 - Major Update

Major Release

2021-05-17 - 3.1.56 - Chore

Maintenance Release

2020-09-09 - 3.1.44 to 3.1.52

Multiple Fix Releases

2020-06-12 - 3.1.26 to 3.1.28

Multiple Fix Releases

2020-02-18 - 3.1.23 to 3.1.25

Multiple Fix Releases

2019-09-11 - 3.1.20 to 3.1.22

Multiple Fix Releases

2018-07-10 - 3.0.5 - New Feature

Added Feature

2018-07-08 - 3.0.1 - Chore

Dependencies Update

2018-07-08 - 3.0.0 - Major Update

Refactor and Cleanup

2018-01-16 - 2.0.7 - Breaking Change

Big Changes

2018-01-12 - 2.0.0 - Major Release

Core Updates